New Oxford study uncovers explosion of accessible deepfake AI image generation models intended for the creation of non-consensual, sexualised images of women
Researchers from the Oxford Internet Institute (OII) at the University of Oxford have uncovered a dramatic rise in easily accessible AI tools specifically designed to create deepfake images of identifiable people, finding nearly 35,000 such tools available for public download on one popular globally accessible online platform, for example.
The study, led by Will Hawkins, a doctoral student at the OII, and accepted for publication at the ACM Fairness, Accountability, and Transparency (FAccT) conference, reveals these deepfake generators have been downloaded almost 15 million times since late 2022, primarily targeting women. The data point towards a rapid increase in AI-generated non-consensual intimate imagery (NCII).
Key findings:
- Massive scale: Nearly 35,000 publicly downloadable “deepfake model variants” were identified. These are models that have been fine-tuned to produce deepfake images of identifiable people, often celebrities. Other variants seek to generate less prominent individuals, with many based on social media profiles. They are primarily hosted on Civitai, a popular open database of AI models.
- Widespread use: Deepfake model variants have been downloaded almost 15 million times cumulatively since November 2022. Each variant downloaded could generate limitless deepfake images.
- Overwhelmingly targeting women: A detailed analysis revealed 96% of the deepfake models targeted identifiable women. Targeted women ranged from globally recognised celebrities to social media users with relatively small followings. Many of the most popular deepfake models target individuals from China, Korea, Japan, the UK and the US.
- Easily created: Many deepfake model variants are created using a technique called Low Rank Adaptation (LoRA), requiring as few as 20 images of the target individual, a consumer-grade computer, and 15 minutes of processing time.
- Intended to generate NCII: Many models carry tags such as ‘porn’, ‘sexy’ or ‘nude’ or descriptions signalling intent to generate Non-Consensual Intimate Imagery (NCII), despite such uses violating the hosting platforms' Terms of Service and being illegal in some countries including the UK.
“There is an urgent need for more robust technical safeguards, clearer and more proactively enforced platform policies, and new regulatory approaches to address the creation and distribution of these harmful AI models,” said Will Hawkins, lead author of the study.
The sharing of sexually explicit deepfake images was made a criminal offence in England and Wales under an amendment to the Online Safety Act in April 2023. The UK Government hopes to also make creating such images an offence as part of its Crime and Policing Bill, which is at currently at Committee Stage.
The results may be merely the tip of the iceberg, with this analysis conducted on only publicly available models on reputable platforms. Given the low cost for creating these models, more egregious deepfake content – for example child sexual abuse material – may also be increasingly widespread but not publicised or hosted on public platforms.
The study, ‘Deepfakes on Demand: the rise of accessible non-consensual deepfake image generators’ by Will Hawkins, Chris Russell and Brent Mittelstadt of the Oxford Internet Institute, will be available as a pre-print on arXiv from 7 May. It will be formally published as part of the ACM Fairness, Accountability, and Transparency (FAccT) peer-reviewed conference proceedings. The conference will be held from 23-26 June in Athens, Greece.
Notes for editors
About this research
The research was carried out through a meta-data analysis of thousands of publicly downloadable text-to-image model variants, focusing on Flux and Stable Diffusion models hosted on two popular repositories, Hugging Face and Civitai.
This study was reviewed and received approval from the University of Oxford’s Central University Research Ethics Committee, application ID 987316
Funding information
This research was supported by the Wellcome Trust (grant nr 223765/Z/21/Z), Sloan Foundation (grant nr G2021-16779), Department of Health and Social Care, EPSRC (grant nr EP/Y019393/1), and Luminate Group, as part of the Trustworthiness Auditing for AI project and Governance of Emerging Technologies research programme at the Oxford Internet Institute, University of Oxford.
During the course of this work, Will Hawkins held an employed position at Google DeepMind. Google did not fund or contribute to the research.
Contact
For more information and briefings, please contact: Anthea Milnes, Head of Communications / Veena McCoole, Media and Communications Manager.
T: +44 (0)1865 280527
M: +44 (0)7551 345493
E: press@oii.ox.ac.uk
About the Oxford Internet Institute (OII)
The Oxford Internet Institute (OII) is a multidisciplinary research and teaching department of the University of Oxford, dedicated to the social science of the Internet. Drawing from many different disciplines, the OII works to understand how individual and collective behaviour online shapes our social, economic and political world.
Since its founding in 2001, research from the OII has had a significant impact on policy debate, formulation and implementation around the globe, as well as a secondary impact on people’s wellbeing, safety and understanding.
Drawing on many different disciplines, the OII takes a combined approach to tackling society’s big questions, with the aim of positively shaping the development of the digital world for the public good.
About the University of Oxford
Oxford University has been placed number 1 in the Times Higher Education World University Rankings for the ninth year running, and number 3 in the QS World Rankings 2024. At the heart of this success are the twin-pillars of our ground-breaking research and innovation and our distinctive educational offer.
Oxford is world-famous for research and teaching excellence and home to some of the most talented people from across the globe. Our work helps the lives of millions, solving real-world problems through a huge network of partnerships and collaborations. The breadth and interdisciplinary nature of our research alongside our personalised approach to teaching sparks imaginative and inventive insights and solutions.
Through its research commercialisation arm, Oxford University Innovation, Oxford is the highest university patent filer in the UK and is ranked first in the UK for university spinouts, having created more than 300 new companies since 1988. Over a third of these companies have been created in the past five years. The university is a catalyst for prosperity in Oxfordshire and the United Kingdom, contributing £15.7 billion to the UK economy in 2018/19, and supports more than 28,000 full time jobs.
