Game Engines are not just for games

As graphics cards and processors continue to become more powerful with each new generation, the line between the virtual and physical world is getting thinner. In 2020 Microsoft Flight Simulator gave players the opportunity to fly over a scale recreation of planet earth, travelling to different countries to view amazing architecture and the seven wonders of the world. In the same year, Valve set a new standard for virtual reality simulations with Half-Life: Alyx and the Index Knuckles controllers. Gaming and game development has led to some of the most advanced technology in the world, and other sectors are noticing.

Filmmaking has made progress in leaps and bounds since the beginning of the 1890s to the present. The progress of motion picture complexity has been driving the film industry’s continuous pursuit for technological evolution. Technological advancements in filmmaking techniques, cameras, animation, sound recording, and editing have expanded the creative potential of filmmakers by providing an increased arsenal of tools to utilize. The evolution of animation techniques to computer-generated imagery has opened a world of possibilities that previously only existed in the imaginations of filmmakers and animators. James Cameron’s influence in this industry and specific focus has been profound. His creation of the movie Avatar showcases the tremendous impact he has made on the industry and the technological evolution of motion pictures. The 2009 blockbuster ‘Avatar’ utilized 60% computer-generated imagery (CGI), with most of the character animation filmed with motion-capture techniques using live actors. The other 40% of the film used traditional live-action imagery techniques.

As technology has improved and the creative possibilities expanded, the degree to which the entertainment industry is reliant on CGI has rapidly increased. As a result, the entertainment industry has arguably been almost entirely dependant on CGI for over a decade now. Modern CGI has become widely recognised as an art and an impactful part of film making, with award such as the Oscars now recognising the contribution of VFX studios. In recent years, audiences have experienced breath-taking sequences from the Marvel Cinematic Universe and the stunning de-aging of actors such as Robert De Niro and Joe Pesci in The Irishman. Modern advancements have also provided film makers with greater flexibility such as the replacement of actors/characters in post-production without returning to principal photography, such as Tig Notaro being added to Zack Snyder’s Army of the Dead.

While current approaches have produced spectacular cinematic sequences, they typically require significant pre-visualisation work to determine all the shots required for the CGI to be seamlessly composited onto the live-action footage. A new approach that is growing in popularity is the use of game engines such as Unity3D and Unreal Engine which are heavily optimised for rendering high visual fidelity assets in real-time. Disney combined the real-time rendering power of Unreal Engine with an immersive and massive 20’ high by 270-degree semi-circular LED video wall and ceiling with a 75’- diameter performance space, where the practical set pieces were combined with digital extensions on the screens. The LED walls were used to display fully digital 3D environments that were rendered and edited in real-time during the shoot. As the digital environments could be rendered at high resolution in real-time, the production team were able to use pixel-accurate tracking and perspectivecorrect 3D imagery. The environments were lit and rendered from the perspective of the camera to provide parallax in real-time, as if the camera were really capturing the physical environment with accurate interactive light on the actors and practical sets. This gave the production team the ability to make creative choices driven by principal photography and achieve real-time in camera compositing on set.

The production of The Mandalorian used game engine technology to remove the need for a green screen and achieve in camera compositing. However, this is not the only option for leveraging game engine technology for a cinematic production. The 2019 remake of the Lion King used the Unity3D game engine and VR to explore the virtual set and discuss creative choices such as the placement of the sun, where the camera should be setup and the best lenses to capture a particular shot. The principal photography phase of the production was completed in a 25-foot square area called the ‘Volume’. The Lion King team had a custom setup that combined the HTC Vive sensor system and US Digital encoders. It meant the crew could use and, more importantly, track conventional camera equipment including Steadicam stabilizers, cranes, and wheeled dollies. A drone pilot was even hired to operate a virtual quadcopter and capture aerial shots in a believable manner. The crew were able to view the virtual cameras view of the digital world rendered in real-time using the Unity3D engine. The three-dimensional movement of the camera by operators moving around the virtual stage was tracked and reflected inside the virtual world, allowing the team to ‘shoot’ the master scenes in Unity.

Once the camerawork was finalised, the animation studio was sent the 3D camera tracking data to be translated into software packages such as Maya to produce the final version of the movie that had production quality assets and animation.

The use of game engine technology outside of games development has seen significant growth over the past few years and part of that growth is likely due to the growth of game engines themselves. The first-generation Unreal Engine was release in 1998 and the Unity3D game engine was announced and released in June 2005. Since release, both engines have been continuously developed with each version adding new features and improved functionality. In the last few years, both engines have added a plethora of features for visual effects and animation such as Timeline & Cinemachine in Unity3D and the Sequencer & Virtual Camera System in Unreal Engine.

At the end of 2021, Unity announced the acquisition of Weta Digital which included its artist tools, core pipeline, intellectual property, and engineering. The aim of the purchase being to combine the deep development and real-time knowledge of Unity with Weta Digital to bring their powerful VFX solutions into the Unity3D engine. The result of this continuous development and innovation by both Unity and Unreal is that both are now far more than game engines, they are 3D visualisation software suites.

Of course, these visualisation suites can be used outside of games and films, they can be used wherever a visualisation of data is needed. Sectors currently making use of the technology include automotive, transportation, manufacturing, architecture, engineering, construction, government, and aerospace. For example, Unity Robotics Simulation enables users to prototype, test and train robots in a high-fidelity realistic simulation before deploying them to the real world. Unity VisualLive overlays BIM and CAD files onto job sites using augmented reality, allowing users to visualise designs and collaborate in real-time.

The power of modern game engines/visualisation suites like Unity and Unreal unlock a wide range of possibilities. It’s time to get creative and see how these technologies could help your business.

How can SOLVD help?

If you would like to discuss your specific requirements or have any questions about Immersive Technologies, Game Engines and Human-Computer Interaction we can work with you to identify and integrate the optimal solutions for your business.

You can email the University of Wolverhampton SOLVD team solvd@wlc.ac.uk or visit www.wlv.ac.uk/solvd.

The SOLVD project supports Telford & Wrekin and Shropshire businesses with the adoption of digital technologies to improve productivity and growth. Eligible businesses can access 12 hours of fully funded support with academic experts.

Blog by: Dr Adam Grant Worrallo - Research Fellow in Digital Innovation at The University of Wolverhampton.

References

https://hackernoon.com/game-engines-arent-just-for-gaming-anymore-nb2a34k0

https://www.youtube.com/watch?v=gUnxzVOs3rk 

https://filmstudies2270.wordpress.com/animation/the-evolution-of-animation-to-cgi-computer-generated-imagery-and-the-impact-of-james-camerons-avatar/

https://www.engadget.com/2019-07-29-lion-king-remake-vfx-mpc-interview.html

https://ascmag.com/articles/making-the-lion-king Lion King Production Photos -

https://ascmag.com/articles/making-the-lion-king

https://blog.unity.com/news/welcome-weta-digital