VR machinima

Go down

VR machinima

Post by AndrewGoodnight on Sun Dec 13, 2015 5:28 pm

Hey folks, thought I'd put up a post about the R&D I'm doing into making VR machinima which I presented at the last artgames. I'm in London for Dec/Jan so won't likely be able to make it to the next meetup but I'm keen to follow everyone's projects so please do post them up on this forum.

For those who weren't there, to give you some background, I've been working as a freelance filmmaker in Melbourne for the last 10 years mainly focused on live action productions, however a passion of mine has always been machinima filmmaking. I've dabbled with various machinima projects over this time as you can see on my youtube channel, but a couple of years ago I decided to start taking it more seriously and produced a high production value web series called Optimism Bias. It was the project I presented at the very first artgames meetup and there is a write up I've done for it in the projects section of this forum.

So, VR machinima

VR's pretty cool right? One thing I noticed when producing my last machinima show is that I was a little to late to the game. machinima was hip and cool back in the early millenium, now youtube is awash with so much content it's nigh on impossible to break through the noise floor and reach any kind of audience.

But VR, VR is new ad hip now. There is a metric shit ton of VR hardware about to hit the market but there's not enough content. There's no shortage of games being developed for VR, but if the predictions floating about are true, VR headsets are going to be as common as smart phones within the next five years. And not everyone likes playing games. Some people want passive experiences or semi-interactive experiences, and that's where VR film making comes in. There is a fairly high barrier to entry with live action VR filmmaking, being the cost of the cameras. Even at it's lowest with the GoPro rigs etc, it's still a fairly costly investment, and the image quality just isn't there, especially as resolutions within headsets are only going to get higher.

And that's just for the hardware. Live action film making is just plain expensive, whether it be 360 degree or conventional. I'm still paying off the last feature film I made! It's one of the reasons I chose to start telling my own stories using the medium of machinima, because it essentially just costs time assuming you have a half decent computer and a few other less than decent computers as well.

So the next logical conclusion is, how do you make 360 degree machinima?

As far as I am aware, there isn't yet any narrative 360 degree machinima in existence. Narrative being the operative word. I've seen a number of in game 360 degree experiences but they are essentially just fly through's of levels and a way of marketing a game. I'm talking about a story driven VR experience that takes place within a game engine, much like Optimism Bias but where the audience can look around at everything that's happening rather than being guided by the editing.

Firstly I'm just going to focus on visuals. I haven't even begun to think about how 360 degree audio could be implemented into a project like this, so that is a hurdle for another day. Visually, it all comes down to display and capture.


How do you display a game in 360 degrees? The whole thing about field of view (FOV) in games is that it restricts what is being rendered in a scene to what the player is looking at, and saves a lot of resources by the graphics card not having to render the things that are outside the player's FOV. So the first and most obvious solution is to set a games FOV to 360 degrees, screen capture a scene and then unwrap that recorded image in post-production (After Effects) At the moment because I'm travelling I'm very limited to the hardware available to me (just my laptop) so it's not possible to test these theories just yet, but I have seen tests other people have done which I assume is using this method. What seems to be the main issue however is resolution. If you set a games FOV to 360 and are playing on a 1080p screen, that image is going to lose a lot of resolution to be stretched around a 360 degree sphere. Below is a video of Skyrim which I believe has been recorded in this manner. Whether or not the bad stretched image graphical artifacts are as a result of low resolution, poor handling in post production, or just an inevitable result of the engine not being designed to operate in this way, I just don't know. Theoretically, I wonder if you were gaming on a 4K/5K screen, and then captured that content and wrapped it around a 360 degree sphere whether or not these issues would be resolved (pun intended), but once again all my hardware is still in Melbourne so it's hard to test these theories. Needless to say recording a game environment in 360 degrees as a 5K video file would require a hefty amount of GPU/CPU grunt.

Scripting and Stitching

Another method which I have seen quite a lot of is scripting and stitching. A lot of games these days allow you to script content for modding etc. Skyrim is heavily modable and allows you to script a camera path within the engine that can be reproduced 100% accurately over and over again. So for example, rather than recording actions in 360 degrees, you record an action as a script, which can be repeated exactly the same way over and over. Below is a 360 degree video that is a good example of this. The way it was created was by scripting a movement for the camera to go starting at the top of a hill and moving down throgh the landscape. This movement was then repeated over and over again withe POV being changed each time, and a different camera angle being recorded in 4K for as many times that were required to achieve full 360 coverage. Imagine this process like sides on a dice. The more sides the dice has, the less the finished result is going to have ugly visible seems between the video layers.

Another video that uses it in a more similar fashion to what I am intending is a 360 degree replay of matches in Counter Strike Go. CS:GO allows spectators to record "demos". Demos aren't actually screen recordings, they are a recording of everything that happened in a match that can be payed back afterwards using the game engine. They always play back the exact same way and can be viewed from any angle after the fact. This function was originally created for players to be able to analyse a match afterwards for training and entertainment purposes, but this function can also be used for generating 360 machinima videos. In this video, the creator recorded a demo of a match and then replayed the demo 16 different times screen recording different perspectives (sides of a dice) and then manually stitched those videos together into a 360 degree video. There are sometimes very visible issues with the resulting stitched video, though whether or not this is inevitable or simply due to the creators skill level is yet to be determined.

This does seem like a more feasible workflow for generating VR machima, due to less load on the capture computer, though would take a significant amount of time. In this situation, it would be great from a directors perspective because directing the story is just a matter of getting the choreography right, and you can record a "demo" multiple times without putting any load on the system, and then only record video of a scene once the desired choreography is achieved. This then means a lot of work needing to be done after a demo has been successfully recorded to then capture that recorded demo from all the camera angles required, but ideally this process could be scripted. An example of someone who has done this to good effect is in a 360 degree Dota video I found. This video has been recorded as a demo, and then the creator scripted the capturing and the stitching of the recorded video to be completely automated. No doubt this is a lot of work for a PC to do, but not requiring any human input means it can be done in down time.

Lens Mapping

Another potential avenue I found for generating 360 degree machinima is lens mapping. I first encountered this when watching a video on youtube about a mod that had been made for Quake called Blinky. The concept of the mod is to map a 360 degree view around the player a map that visual data onto a rectangular screen, so a player could at any point see when someone was behind them. Quite an advantage for a multiplayer environment for someone prepared to train their eye to interpret the visuals and no where an opponent is. An unintentional application for this mod of course is to see if it can be used to make 360 degree videos. There are multiple ways in which the 360 degree FOV can be projected onto a rectangular display, the question is can any one of those possibilities be unwrapped and projected onto a sphere without graphical artifacts?


Once again, in order for 360 video display to be feasible, the recorded content would need to be captured at a very high resolution. However if we look at the above image, and consider if we could map six sides of a cube to 6x 720p images, if the projected lens was captured on a 4K monitor, theoretically there would be enough resolution for a 360 degree video with the area of interest (AOI) resolution being 720p (see below) With a 5K monitor that resolution would be increased further.

Currently this Blinky mod is only compatible with Quake, but it works, so there's no reason to believe this kind of technology couldn't be implemented into other game engines in order to screen capture 360 degree videos in game.

So that's where I'm up to at the moment. I need to get my hands on some hardware before I can really continue researching this much further. Whether I buy stuff or ship stuff over from Oz or use someone else's facility is yet to be determined. It's all just food for thought at the moment. I've got some great story ideas that I think would work well as a VR experience, but getting the tech nailed down is a pretty big hurdle. Please give me feedback and ideas. This stuff sparked a lot of interesting discussion at the last artgames and it was great to be talking with people that 100% understand what I'm trying to achieve. Hoping to continue that discussion here.

Cheers, Andrew.


Posts : 3
Join date : 2015-11-18

View user profile

Back to top Go down

Back to top

Permissions in this forum:
You cannot reply to topics in this forum