Shooting 360 VR with GoPro Odyssey and Google Jump VR
This past fall Mystery Box produced and finished a 360 VR short action film in collobration with our good friends over at Team Supertramp and we ended up filming with the GoPro Odyssey using Google’s Jump VR system. We have filmed with several other 360 video rigs in the past (Freedom 360, Nokia Osmo, Fly 360, among others) we were excited to see what this new setup could and couldn’t do, and how it would affect the story we were looking to tell.
When the opportunity to film with the GoPro Odyssey and to use the Google Jump VR system for processing came up and we jumped on it. We really went into it blind, we wanted to spend a day in demo testing out shoots, but scheduling wouldn't allow time for prep. But we dove right in and problem solved as we went.
Unlike every other 360 camera we’d tested, the GoPro Odyssey and Google Jump VR got us inspired by the medium. It gave us far better results than the other systems we’d used. The workflow had its inconvenient moments and the system has its limitations, but we finished the project wanting to do more with the rig, rather than the usual sentiment of “never again”.
Which may make you wonder: how is it different, and why does it matter?
The Rig
The GoPro Odyssey Rig
The GoPro Odyssey is essentially a ring of 16 GoPro Hero 4 Blacks, all tied together with a unified power source and custom firmware that links their controls. The cameras are aligned vertically, and shoot in 2.7K 4:3 aspect ratio at 30fps (29.97fps). Each camera has an effective coverage of about 45 degrees horizontally (around the center axis of the rig), and 120 degrees vertically.
If you look at those numbers you’ll notice a couple of “wait, what?” questions.
First, yes, it doesn’t do a full 360 x 180 sphere, there are cones of occlusion on the top and bottom of the rig. There’s a screw mount in the center of the rig that you can add another GoPro to if you want to fill in the missing gap on the top, but if you’re in the normal planar orientation it’s actually pretty rare for a viewer to look up when watching the video. If you’re hanging the rig upside down and flying it, you’ll probably want to throw one there for the downward looking orientation. Otherwise, the screw mount on the top is perfect for mounting a spatial microphone like the Zoom H2N and getting ambisonic audio as well. But I’ll get to that in a minute.
But what about the horizontal angle of view - 45 degrees per camera? 360 degrees / 16 cameras is 22.5 degrees per camera.
True. Unless you’re shooting in 3D.
That’s probably the biggest draw of the Odyssey rig, one that makes the VR experience that much better than other rigs we’ve used. Yes, you’re limited to a torroidal field of view, but when viewed on a Google Cardboard, Oculus or other head mounted VR display, the 3D immersion really sells the final image.
How does it capture 3D? Parallax, of course. That’s right: the same concept that creates problems and stitching issues on most rigs allows for 3D on the GoPro Odyssey.
In case this is the first time you’re encountering parallax, briefly, parallax is the concept that viewing the same scene from different positions renders slight differences in the relative positions of objects. For human vision, the interocular distance (distance between the eyes) creates a point of view difference in the image coming from each eye that our brain uses to provide the bulk of our depth perception. There are other cues for perceiving depth, but none is as important as the parallax information.
GoPro Odyssey Single Camera Field of View. Notice the amount of overlap between adjascent cameras
The part of the Camera's Field of View actually used in the final image. The outer horizontal edges of each frame are discarded
Each camera within the Odyssey rig sees a 90 degree arc on the horizontal axis. Usually, the outer 22.5 degrees (ish - these angles are approximate because of how the stitching process works) are discarded because this is where the greatest parallax and distortion occurs. The center 45 degrees is then cut into 22.5 degree segments, with each adjacent camera around the rig forming a left-eye / right-eye pair.
How the center 45° of each GoPro Camera is split into a Left and Right eye, to form stereo pairs between adjacent cameras.
The 16 “left eye” arcs of 22.5 degrees are then blended into one equirectangular image and the 16 “right eyes” arcs are blended into a second. The two equirectangular images are combined into a single over-under 3D video file which you can edit and work with.
Left and Right eye image slices from all 16 GoPros in the Odyssey form the slices that are combined into the left and right eye stitched equirectangular images.
Google’s Jump VR system handles the stitching, which makes our control-freak workflow specialist a little nervous. You upload the footage from all 16 of the cameras, Google’s servers stitch it and you get it back a couple of days later.
How can you tell if everything’s worked right? You can’t.
Weakpoints
With the Odyssey rig is that there is no way to do playback, which is a rarity in digital filmmaking. Using it reminds us of when we used to shoot film, where our ability to replay previous clips was limited, but in this case it’s worse. With a 360 degree field of view, there’s nowhere for the director to hide and see how the take went.
In other words, there’s no way to see what the camera’s seeing, review what’s been shot, or stand to the side and watch the performances from there. So how are we supposed to know if we got the shot?
During a tutorial from Google, we were shown images of directors hiding underneath the tripod to watch the action. Unfortunately, this wouldn’t work for us since we had the camera mounted on a cart, that blocked the view of anyone hiding underneath, and statically sitting under the tripod made it really tough to turn around and follow the action through the whole shot.
Nowhere to hide in the cart!
Our solution? We simply added more GoPros! We dialed in the settings of two GoPro Hero4 Blacks to match the ISO and frame rate as the Odyssey, and live streamed/captured the action from two mounting points underneath the Odyssey’s chassis. This gave us almost the full 360 degrees of view as footage we were able to play back, show actors/clients, and review several times to give notes and make adjustments.
Bottom Mounted GoPros for live monitoring, capture and playback.
Image from the Reference GoPro side by side with the image from the Odyssey (Uncolored Stitch). The reference GoPro was used to stream live to a tablet and give us playback.
Odyssey's battery hidden below the rig - notice the size!
Another fairly major limitation of the Odyssey rig is that in order to power 16 GoPro’s you need a hefty quad v-mount battery bank. When coming from the Fly360 (which is about the size of a baseball) or the Freedom 360 which is about the size of a volleyball, this was a huge jump. The Odyssey rig itself is large and cumbersome, but the battery was even heavier. The battery being separated from the rig proves challenging to keep it hidden whenever you want to move the camera or attach it to anything. Once again, most directors place it under the tripod.
On the plus side though, the large battery enabled us to film all day, and since we were pushing the rig around on a cart the whole shoot, the size and weight didn’t pose a huge issue to us, this time anyway. Maybe we should always have a cart? (Problem solved!)
Maybe it's Us?
Because the rig is still new, there were some challenges we ran into while using it that that should go away with time. The first was how careful you have to be when using the camera. The cameras are controlled by a daisy chained set of network cables that run from camera 1 to camera 16, and everything is controlled by GoPro #1. This creates a pretty substantial delay every time we powered on, rolled the camera, cut the camera, or made adjustments to the settings; we have to take things slowly and wait 2-3 seconds betweens actions as camera 1 synced with the other 15 cameras. Rolling on the rig, for instance involves pushing record on GoPro #1, which beeps to confirm, followed by a slight delay and then a chorus of beeps from all of the GoPros which signals the actual start of recording. Once we got used to the delayed pace it became second nature
A second issue we ran into was that we had to power down the whole system (including the battery) after each take. This took some getting used to, as after each “CUT!” it would take 3 delayed button pushes to power down the system (stop recording, power off Odyssey, power off battery). This might have been because each of our takes were several minutes in length; we don’t know what would have happened if we were using shorter takes. But what we do know is that the one time we did try and squeeze two takes in one power cycle, things went a little haywire. The rig locked up and we had to do a hard reboot in order to resync the cameras; several of the cameras didn’t finalize their takes and had to rebuild the footage when the rig was repowered before we could start again.
Those hiccups aside the end product was absolutely outstanding. The GoPro Odyssey delivers an 8K x 8K image which enables an amazingly immersive and detailed 3D experience.
4K by 4K Output from GoPro Odyssey and Google Jump VR
One unmarketed advantage to having a 16-lensed 360 camera is that should any of the cameras fail, you are still more than covered for the full 360 field of view. Because we were working with an untested (brand new) camera (as in, showed up from the manufacturer a day or two before we were to use it), we didn’t know until after Google got back to us with our footage that 3 of the 16 cameras were dead on arrival, and their footage was fully corrupted. This was when the extra field of view overlapping came in to save the day, as Google Jump VR was able to stitch the remaining camera angles together for a seamless image (with only one VFX shot to fix a glitch). We lost out on the 8K resolution and had to master at 4K 3D, but being able to recover from a 3 camera failure was amazing.
Another advantage to having 16 cameras is that the auto exposure settings are better tuned for the 360 experience. Several times we had the Bear push the cart from outdoors to indoors. The front facing cameras would adjust to the darker space, while the rear cameras would remain at their outdoor exposure until they were mostly inside the room. In many ways, this makes its autoexposure superior to that of single lens 360 cameras, since they will only be able to have one exposure setting no matter where the viewer is looking: with the GoPro Odyssey, the exposure is tuned for wherever you are looking.
Autoexposure differences between the camera - the ones facing outdoors have autoexposed darker than the ones facing the interior.
Audio
Just a quick note about audio before we wrap this up. The Zoom H2n is a fantastic recorder for capturing ambisonic / spatial audio, and setting it up was a breeze. Any sound within about 20 feet of the rig was captured with surprising clarity. We mounted it on the top of the rig so that the audio center and alignment matched the Odyssey's alignment:
But we didn't end up using it.
Okay, that's a little lie. We did use the audio it captured, but not as spatial audio. We ended up having to convert all of the ambisonic audio into stereo because of the amount of sound design we added to the piece. Budgetary and time constraints meant that we had to do that part of the project quickly and with our internal team instead of outsourcing. In order to make the sound design ambisonic we would have needed an expensive audio plugin for Pro Tools (which we don't normally use internally) and double or triple the amount of time used. That, and there's still no go way of putting music in the space (other than dead center of the space, which effectively makes it monophonic). Next time we'll leave larger considerations for sound design in the ambisonic space, to better accompany the quality of the VR video experience.
Last Thoughts
Looking back on this project, we were pretty ambitious in pushing the limits of the 360 experience. We wanted to move the camera (handbook said not to), go from dark to light spaces several times (also discouraged) and get super close to the camera (BIG NO NO - you’re supposed stay about 1m away). In all of these scenarios, the GoPro Odyssey and Google Jump post processing held up incredibly well, and inspired us to do future action packed 360 shoots.
Our tiny cast and crew.
We see the GoPro Odyssey 360 as the best VR camera that’s commercially available, for most applications. It’s not perfect - far from it - but despite it’s flaws, it’s still a fantastic system and one that we’d recommend using for any project looking to shoot in VR today.
This is how the GoPro Odyssey makes us feel.
Written by Samuel Bilodeau, Head of Technology and Post Production