Monday, February 05, 2024

 

A fully autonomous drone system for cinematography and wildlife monitoring

A fully autonomous drone system for cinematography and wildlife monitoring
CineMPC in action. This figure depicts two experiments where CineMPC 
manages both camera intrinsics and extrinsics to user instructions. 
The first row showcases the initial experiment, where CineMPC manages both 
the intrinsics and extrinsics to capture a target (the chair), adhering to the rule of 
thirds and maintaining focus. In the second row, two frames from the second 
experiment demonstrate the recording of two targets (car and bottle) meeting 
various requirements for image focus (initially focusing on the bottle and then on
 the car) and the positioning of targets in the image (placing them in 
accordance with the rule of thirds). The drone remains stationary throughout the 
entire execution of this experiment. Thus, all changes in recording are 
performed through manipulation of the focal length and focus distance of the 
camera, its intrinsics.
 Credit: Pueyo et al

Recent technological advances, such as increasingly sophisticated drones and cameras, have opened exciting new possibilities for cinematography. Most notably, film directors can now shoot scenes from a wide range of angles that were previously inaccessible and in far higher resolution.

Researchers at University of Zaragoza and Stanford University recently developed CineMPC, a new cinematographic system that relies on a fully autonomous drone that carries a cinematographic  to film multiple targets autonomously, while following a director's instructions. The platform modulates various drone and camera parameters to satisfy these instructions. The team's innovative system, outlined in IEEE Transactions on Robotics, could bring a wave of innovation to the  and other sectors that can benefit from high-quality video footage.

"Existing solutions for autonomous drone cinematography revealed a common oversight, namely, none provided  over camera intrinsic parameters (i.e., , aperture, focus distance)," Pablo Pueyo Ramon, co-author of the paper, told Tech Xplore.

"In cinematography, controlling these parameters is essential to achieve different artistic and technical goals, such as a desired depth of field (parts of the scene shown in focus or blurred), or iconic shots like the dolly-zoom or vertigo effect. CineMPC fills this gap, determining autonomously the appropriate camera intrinsics capable of achieving a wide array of user-defined cinematographic instructions."

Pueyo and his colleagues have been developing innovative technologies for cinematography for some time now. The new system they developed, dubbed CineMPC, was first introduced in 2021.

CineMPC essentially consists of software that can be installed in any drone equipped with a controllable professional camera (e.g., a DSLR camera). As part of their recent study, the researchers broadened their software's functionalities and set out to assess its performance in a real-world setting.

Credit: Pablo Pueyo Ramon

"Besides including a better and more advanced control strategy, the new version of CineMPC also includes a perception module capable of identifying relevant information from the scene, like the actors and actresses, making it a 100% autonomous control solution," Pueyo explained. "Finally, we are now releasing the source code so that everybody can use it."

Pueyo and his colleagues have tested the improved version of CineMPC by applying it to a real drone for cinematography and filming various scenes with it. They found that their software achieved remarkable results, reliably estimating the relative poses of filmed targets and allowing users greater control over the footage captured by the drone, for instance adding unique effects, tracking specific people or objects, and so on.

"In our opinion, the implications of our study for cinematography alone are remarkable," Pueyo said. "We are enormously happy to offer filmmakers creative freedom, improved safety, and increased autonomy for real-time decision-making."

More information: Pablo Pueyo et al, CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene Composition, IEEE Transactions on Robotics (2024). DOI: 10.1109/TRO.2024.3353550

Pablo Pueyo et al, CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2023). DOI: 10.1109/IROS55552.2023.10342280

Journal information: IEEE Transactions on Robotics 

No comments:

Post a Comment