Compared to traditional green screens, this solution presents a virtual, three-dimensional space that combines the real and the virtual, and allows for the synthesis of special effects and backgrounds through real-time filming, greatly simplifying the production process and reducing filming costs. Furthermore, the images are more realistic and the color reproduction is more accurate.

Virtual filming is essentially an emerging application based on XR technology. It can be applied not only to filmmaking but also to variety shows, live broadcasts, studio broadcasts, television series, and other filming applications. The immersive nature of large LED screens creates a sci-fi feel that's difficult to achieve in real life. "Birthday" premiered at the National Association of Broadcasters (NAB) in April of this year, generating significant buzz within the industry. It was the world's first film entirely produced using virtual production. The director explained that the film used three props and took four days to shoot, requiring no post-production special effects. The live action footage looked exactly like the finished film. This demonstrates the global growth of virtual filming and the disruptive impact this technology has on related industries. From relying on post-production synthesis, to the current common practice of setting up tracks and installing numerous sensors to capture the changes in depth and scale of people in three-dimensional scenes, and finally to today's virtual shooting solutions driven by LED display + XR technology, we combine real-life characters and scenery with virtual 3D scenes in real time, shooting them in an integrated manner through digital cameras, maintaining the consistency of optical lens parameters and virtual lens coordinates, and the consistency of real-world and virtual world coordinates, presenting realistic visual relationships and perspective effects.

Virtual Shooting System Framework

For virtual filming, all the elements required for a shot, including the setting, characters, and lighting, are integrated into the computer. The overall virtual filming solution primarily consists of a video playback system, a virtual scene rendering system, a virtual scene production system, a positioning tracking system, an LED control system, and peripheral hardware (cameras and LED screens).

Virtual production has many advantages over traditional production

Traditional production setup is time-consuming and labor-intensive, with many uncontrollable factors. Repeated adjustments in post-production are difficult, and the shooting environment and process are irreversible. Some aspects can be processed through editing and special effects, but some cannot be processed and have to be re-shot, increasing costs and cycles.

Traditional production process

Virtual shooting and production process

Core technology of the solution

The essence of this solution is to ensure that the scene follows the subject's movements. We aim to achieve a near-realistic shooting experience when the subject moves freely within the matte area. Therefore, positioning and tracking technology is particularly important. Behind these amazing technologies lies the development of software and hardware, as well as the continuous upgrading and application of artificial intelligence technology.

This solution uses Xvisio Technology's SeerSense® product as a positioning and tracking device. Through VSLAM positioning and tracking, the camera is bound to a virtual camera within the 3D scene, allowing the scene to follow the person during shooting.

The Xvisio SeerSense® series can be used as a scene tracking system for 3D virtual filming. Simply connect it to a camera lens. As the camera lens moves, it can directly obtain the real-time position and motion parameters of a person in real space. Then, by calculating and analyzing this data, a perfect virtual-real composite image can be obtained in real time.

Xvisio SeerSense® products powered by Intel chips

Virtual scene positioning technology VSLAM

This tracking system offers high accuracy, strong anti-interference capabilities, and can be used indoors and outdoors without markers. It requires minimal setup requirements, allows for a larger tracking area, and provides 6DOF data. Compared to existing products, it offers superior performance and a significant price-performance advantage. This solution also incorporates an Intel 10nm process Mini-PC for intelligent acceleration and media playback at the edge.

Core product specifications
Xvisio SeerSense® DS80

Mini-PC

Xvisio Technology is a high-tech startup focused on core XR spatial perception and interaction technologies (high-speed VSLAM technology) and XR product solutions. Founded in Silicon Valley in 2016 and based in Shanghai, Xvisio currently has R&D centers in Silicon Valley and Europe. It boasts a distinguished domestic and international R&D team and over 20 core technology IPs covering algorithms, hardware, software, systems, and applications. Its key products include the SeerSense® perception module and the SeerLens® series of XR glass, as well as core technology licensing and customized product development services. Its clients include many industry-leading companies, and its products are sold throughout Japan, the United States, Europe, and other regions.