Intelligent Cinematography
Everyone can be a Director
VirtualFilmStudio is dedicated to content creation specifically for the film industry, aiming to leverage artificial intelligence technology to enhance production efficiency across all aspects of film production. We firmly believe that the responsible and effective utilization of artificial intelligence technology will empower film practitioners to create superior artistic works, supporting them in their creative endeavors.
Extract various visual elements from existing video footage, including stage design, lighting and the arrangement of actors in scenes.
Assist creators in planning scripts and formulating the shooting plan, enabling efficient decision-making processes in film production.
Pay homage to the masters of film by replicating their iconic shots with distinct content, thereby showcasing our reverence for their artistic contributions.
Develop user-friendly tools that seamlessly integrate into existing workflows across various platforms to facilitate the content creation process.
Virtual Dynamic Storyboard (VDS) allows users storyboarding shots in virtual environments, where the filming staffs can easily test the settings of shots before the actual filming.
Virtual Dynamic Storyboard runs on a “propose-simulate-discriminate” mode: given a formatted story script and a camera script as input, it generates several character animation and camera movement proposals following predefined story and cinematic rules⚡️.
A Visualization Tool to Aid Scriptwriting based on a Large Movie Database.
A screenwriter types in a script and selects their story’s fixed and variable attributes. ScriptViz retrieves sequences for the input script. The screenwriter iterates on the script based on the proposed sequences. The visualization helps to enrich the details of the existing dialogue and write the unfinished dialogue.
Cinematic Behavior Transfer via NeRF-based Differentiable Filming.
Given a film shot, a series of visual continuous frames, containing complex camera movement and character motion, we present an approach that estimates the camera trajectory and character motion in a coherent global space. The extracted camera and characters' behavior can be applied to new 2D/3D content through our cinematic transfer pipeline. The 2D cinematic transfer replicates character motion, camera movement, and background with new characters.The 3D cinematic transfer result is similar to 2D results, but have more freedom than 2D, such as changing the lighting and environment.
Temporal and Contextual Transformer for Multi-Camera Editing of TV Shows.
The ability to choose an appropriate camera view among multiple cameras plays a vital role in TV shows delivery. But it is hard to figure out the statistical pattern and apply intelligent processing due to the lack of high-quality training data. To solve this issue, we first collect a novel benchmark on this setting with four diverse scenarios including concerts, sports games, gala shows, and contests, where each scenario contains 6 synchronized tracks recorded by different cameras. It contains 88-hour raw videos that contribute to the 14-hour edited videos. Based on this benchmark, we further propose a new approach temporal and contextual transformer that utilizes clues from historical shots and other views to make shot transition decisions and predict which view to be used. Extensive experiments show that our method outperforms existing methods on the proposed multi-camera editing benchmark.