I spent the Labor Day weekend learning how to add text and 2D video into an immersive video scene. I’m doing this because I have this vision for a film/app for the Apple Vision Pro and I thought I could 1. prototype a large part of this experience in video and 2. I could also make immersive videos like this and learn how to tell stories in this medium. 

So to begin with, I came up with the requirements for my project.

• I had to use existing video. I wasn’t going to waste time shooting new things or trying to have this video mean anything or stand on it’s own. I’m just working on learning the technical part.
• I want to include text in the 3D space.
• I want to include 2D video in the 3D space.

There’s more I want to learn — immersive audio & 3D video in the 3D space — but I’ll save that for another time. Knowing how to add text and 2D video is most of what I need and it allows me to work on storytelling in parallel with the other technical skills. 

• Apple Vision Pro: download this video and view it in Spatialgen.
• Meta Quest: use this version (to view in 3D, put the video in fullscreen, then put the browser in fullscreen, finally choose VR 180 Left/Right as the projection mode)
• You can also watch it on YouTube.

(more…)

I spent the weekend learning how to composite text and 2D video into a 180 degree, 3D video. I mostly have it all working. Once I figure out a couple of issues I’ll write something up.

Shooting and editing immersive video is not very straightforward. Thankfully, Hugh Hou has made a million tutorials on this subject. I’ve been experimenting with the techniques he’s outlined to try to find what works best for me. This week I tried Mistika VR again and I found that it doesn’t do a noticeably better job than other methods. It’s also really expensive. I’ll have to miss out on it’s ability to reframe 180 degree video into rectangular 16 x 9 video. The Canon VR utility is easy to use and does a decent job (though my copy refuses to recognize my purchase and limits me to 2 min exports – ugh).

Today I used the Canon VR Utility to stitch my footage from the Canon R5C and export it to ProRes.

I then took that ProRes file and added the Canon LUT and did noise reduction in Davinci Resolve.

Then I exported left and right eye views and combined and compressed them with Mike Swanson’s Spatial Video Tool.

Well, I did that whole workflow for a 10 second test clip. I’m now exporting 5 min clips which will take about 12 hours (!!!).

I also recorded ambisonic audio for this but I don’t know what to do with it yet. I’ll have to work on that later.

You can also skip the Canon VR Utility and do that part in Davinci Resolve using the Karta VR plugin. I’ve done that before and it works well but I’m unclear if I can export left and right views without first exporting a side-by-side video that then gets chopped into left and right. I guess that’ll be another experiment.

Issue 4 is ready to go. Next, I have print mailing labels, write some notes, and stuff envelopes. I’m aiming to get these in the mail by Monday at the latest. I’ll also have a few copies with me at Zine Camp on the 21st.

I’ve still got one more image to create but I can get the printing started today.

I finished re-writing issue 4 for the third time today. It’s finished now. I cut a couple of things that I’ll put on the issue’s page. Other than the covers I have to create new illustrations. I’ll start working on those tomorrow. Only one side will be 4 colors. The others will be just 2 so it shouldn’t take as long to print as I anticipated.