A fully autonomous drone system for cinematography and wildlife monitoring

by Ingrid Fadelli , Tech Xplore

CineMPC in action. This figure depicts two experiments where CineMPC 

manages both camera intrinsics and extrinsics to user instructions. 

The first row showcases the initial experiment, where CineMPC manages both 

the intrinsics and extrinsics to capture a target (the chair), adhering to the rule of 

thirds and maintaining focus. In the second row, two frames from the second 

experiment demonstrate the recording of two targets (car and bottle) meeting 

various requirements for image focus (initially focusing on the bottle and then on

 the car) and the positioning of targets in the image (placing them in 

accordance with the rule of thirds). The drone remains stationary throughout the 

entire execution of this experiment. Thus, all changes in recording are 

performed through manipulation of the focal length and focus distance of the 

camera, its intrinsics.

 Credit: Pueyo et al

Recent technological advances, such as increasingly sophisticated drones and cameras, have opened exciting new possibilities for cinematography. Most notably, film directors can now shoot scenes from a wide range of angles that were previously inaccessible and in far higher resolution.

Researchers at University of Zaragoza and Stanford University recently developed CineMPC, a new cinematographic system that relies on a fully autonomous drone that carries a cinematographic camera to film multiple targets autonomously, while following a director's instructions. The platform modulates various drone and camera parameters to satisfy these instructions. The team's innovative system, outlined in IEEE Transactions on Robotics, could bring a wave of innovation to the film industry and other sectors that can benefit from high-quality video footage.

"Existing solutions for autonomous drone cinematography revealed a common oversight, namely, none provided automatic control over camera intrinsic parameters (i.e., focal length, aperture, focus distance)," Pablo Pueyo Ramon, co-author of the paper, told Tech Xplore.

"In cinematography, controlling these parameters is essential to achieve different artistic and technical goals, such as a desired depth of field (parts of the scene shown in focus or blurred), or iconic shots like the dolly-zoom or vertigo effect. CineMPC fills this gap, determining autonomously the appropriate camera intrinsics capable of achieving a wide array of user-defined cinematographic instructions."

Pueyo and his colleagues have been developing innovative technologies for cinematography for some time now. The new system they developed, dubbed CineMPC, was first introduced in 2021.

CineMPC essentially consists of software that can be installed in any drone equipped with a controllable professional camera (e.g., a DSLR camera). As part of their recent study, the researchers broadened their software's functionalities and set out to assess its performance in a real-world setting.

Credit: Pablo Pueyo Ramon

"Besides including a better and more advanced control strategy, the new version of CineMPC also includes a perception module capable of identifying relevant information from the scene, like the actors and actresses, making it a 100% autonomous control solution," Pueyo explained. "Finally, we are now releasing the source code so that everybody can use it."

Pueyo and his colleagues have tested the improved version of CineMPC by applying it to a real drone for cinematography and filming various scenes with it. They found that their software achieved remarkable results, reliably estimating the relative poses of filmed targets and allowing users greater control over the footage captured by the drone, for instance adding unique effects, tracking specific people or objects, and so on.

"In our opinion, the implications of our study for cinematography alone are remarkable," Pueyo said. "We are enormously happy to offer filmmakers creative freedom, improved safety, and increased autonomy for real-time decision-making."

More information: Pablo Pueyo et al, CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene Composition, IEEE Transactions on Robotics (2024). DOI: 10.1109/TRO.2024.3353550

Pablo Pueyo et al, CineTransfer: Controlling a Robot to Imitate Cinematographic Style from a Single Example, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2023). DOI: 10.1109/IROS55552.2023.10342280

Journal information: IEEE Transactions on Robotics 

© 2024 Science X NetworkCineMPC: An algorithm to enable autonomous drone-based cinematography