New method forecasts computation, energy costs for sustainable AI models
The process of updating deep learning/AI models when they face new tasks or must accommodate changes in data can have significant costs in terms of computational resources and energy consumption. Researchers have developed a novel method that predicts those costs, allowing users to make informed decisions about when to update AI models to improve AI sustainability.
“There have been studies that focused on making deep learning model training more efficient,” says Jung-Eun Kim, corresponding author of a paper on the work and an assistant professor of computer science at North Carolina State University. “However, over a model’s life cycle, it will likely need to be updated many times. One reason is that, as our work here shows, retraining an existing model is much more cost effective than training a new model from scratch.
“If we want to address sustainability issues related to deep learning AI, we must look at computational and energy costs across a model’s entire life cycle – including the costs associated with updates. If you cannot predict what the costs will be ahead of time, it is impossible to engage in the type of planning that makes sustainability efforts possible. That makes our work here particularly valuable.”
Training a deep learning model is a computationally intensive process, and users want to go as long as possible without having to update the AI. However, two types of shifts can happen that make these updates inevitable. First, the task that the AI is performing may need to be modified. For example, if a model was initially tasked with only classifying digits and traffic symbols, you may need to modify the task to identify vehicles and humans as well. This is called a task shift.
Second, the data users provide to the model may change. For example, you may need to make use of a new kind of data, or perhaps the data you are working with is being coded in a different way. Either way, the AI needs to be updated to accommodate the change. This is called a distribution shift.
“Regardless of what is driving the need for an update, it is extremely useful for AI practitioners to have a realistic estimate of the computational demand that will be required for the update,” Kim says. “This can help them make informed decisions about when to conduct the update, as well as how much computational demand they will need to budget for the update.”
To forecast what the computational and energy costs will be, the researchers developed a new technique they call the REpresentation Shift QUantifying Estimator (RESQUE).
Essentially, RESQUE allows users to compare the dataset that a deep learning model was initially trained on to the new dataset that will be used to update the model. This comparison is done in a way that estimates the computational and energy costs associated with conducting the update.
Those costs are presented as a single index value, which can then be compared with five metrics: epochs, parameter change, gradient norm, carbon and energy. Epochs, parameter change and gradient norm are all ways of measuring the amount of computational effort necessary to retrain the model.
“However, to provide insight regarding what this means in a broader sustainability context, we also tell users how much energy, in kilowatt hours, will be needed to retrain the model,” Kim says. “And we predict how much carbon, in kilograms, will be released into the atmosphere in order to provide that energy.”
The researchers conducted extensive experiments involving multiple data sets, many different distribution shifts, and many different task shifts to validate RESQUE’s performance.
“We found that the RESQUE predictions aligned very closely with the real-world costs of conducting deep learning model updates,” Kim says. “Also, as I noted earlier, all of our experimental findings tell us that training a new model from scratch demands far more computational power and energy than retraining an existing model.”
In the short term, RESQUE is a useful methodology for anyone who needs to update a deep learning model.
“RESQUE can be used to help users budget computational resources for updates, allow them to predict how long the update will take, and so on,” Kim says.
“In the bigger picture, this work offers a deeper understanding of the costs associated with deep learning models across their entire life cycle, which can help us make informed decisions related to the sustainability of the models and how they are used. Because if we want AI to be viable and useful, these models must be not only dynamic but sustainable.”
The paper, “RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability,” will be presented at the Thirty-Ninth Association for the Advancement of Artificial Intelligence (AAAI) Conference on Artificial Intelligence, which will be held Feb. 25-Mar. 4 in Philadelphia, Penn. The first author of the paper is Vishwesh Sangarya, a graduate student at NC State.
Method of Research
Computational simulation/modeling
Subject of Research
Not applicable
Article Title
RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability
Understanding bias and discrimination in AI: Why sociolinguistics holds the key to better Large Language Models and a fairer world
The language ‘engines’ that power generative artificial intelligence (AI) are plagued by a wide range of issues that can hurt society, most notably through the spread of misinformation and discriminatory content, including racist and sexist stereotypes.
In large part these failings of popular AI systems, such as ChatGPT, are due to shortcomings with the language databases upon which they are trained.
To address these issues, researchers from the University of Birmingham have developed a novel framework for better understanding large language models (LLMs) by integrating principles from sociolinguistics – the study of language variation and change.
Publishing their research in Frontiers in AI, the experts argue that by accurately representing different ‘varieties of language’, the performance of AI systems could be significantly improved – addressing critical challenges in AI, including social bias, misinformation, domain adaptation, and alignment with societal values.
The researchers emphasise the importance of using sociolinguistic principles to train LLMs to better represent the diverse dialects, registers, and periods of which any language is composed – opening new avenues for developing AI systems that are more accurate and reliable, as well as more ethical and socially aware.
Lead author Professor Jack Grieve commented: “When prompted, generative AIs such as ChatGPT may be more likely to produce negative portrayals about certain ethnicities and genders, but our research offers solutions for how LLMs can be trained in a more principled manner to mitigate social biases.
“These types of issues can generally be traced back to the data that the LLM was trained on. If the training corpus contains relatively frequent expression of harmful or inaccurate ideas about certain social groups, LLMs will inevitably reproduce those biases resulting in potentially racist or sexist content.”
The study suggests that fine-tuning LLMs on datasets designed to represent the target language in all its diversity – as decades of research in sociolinguistics has described in detail – can generally enhance the societal value of these AI systems. The researchers also believe that by balancing training data from different social groups and contexts, it is possible to address issues around the amount of data required to train these systems.
“We propose that increasing the sociolinguistic diversity of training data is far more important than merely expanding its scale,” added Professor Grieve. “For all these reasons, we therefore believe there is a clear and urgent need for sociolinguistic insight in LLM design and evaluation.
“Understanding the structure of society, and how this structure is reflected in patterns of language use, is critical to maximizing the benefits of LLMs for the societies in which they are increasingly being embedded. More generally, incorporating insights from the humanities and the social sciences is crucial for developing AI systems that better serve humanity.”
ENDS
Notes to editor:
The University of Birmingham is ranked amongst the world’s top 100 institutions. Its work brings people from across the world to Birmingham, including researchers, teachers and more than 8,000 international students from over 150 countries.
‘The Sociolinguistic Foundations of Language Modelling’ – Jack Grieve, Sara Bartl, Matteo Fuoli, Jason Grafmiller, Weihang Huang, Alejandro Jawerbaum, Akira Murakami, Marcus Perlman, Dana Roemling, and Bodo Winter is published by Frontiers in AI.
Journal
Frontiers in Artificial Intelligence
Method of Research
Data/statistical analysis
Subject of Research
People
Article Title
The Sociolinguistic Foundations of Language Modelling
Article Publication Date
13-Jan-2025
UniTrento partners with Google's Quantum Artificial Intelligence Lab for research
A research team coordinated by the Department of Physics was able to work on the powerful computers of Google's Quantum Ai Lab to conduct a study on confinement in lattice gauge theory. The results have been published in Nature Physic
Università di Trento
image:
Philipp Hauke ©UniTrento – Ph. Alessio Coser
view moreCredit: Philipp Hauke ©UniTrento – Ph. Alessio Coser
Science is always looking for more computing power and more efficient tools capable of answering its questions. Quantum computers are the new frontier in data processing, as they use the quantum properties of matter, such as the superposition of states and entanglement, to perform very complex operations.
A research team coordinated by the Department of Physics of the University of Trento had the opportunity to test some hypotheses on confinement in ℤ2 lattice gauge theory on the quantum computers of Google's Quantum Artificial Intelligence Lab, in California. Their work was published in Nature Physics.
Gauge theories describe the fundamental forces in the standard model of particle physics and play an important role in condensed matter physics. The constituents of gauge theories, such as charged matter and electric gauge field, are governed by local gauge constraints, which lead to key phenomena that are not yet fully understood. In this context, quantum simulators may offer solutions that cannot be reached using conventional computers.
"At the end of 2019 – explains Philipp Hauke, professor of theoretical physics of fundamental interactions at UniTrento and corresponding author of the research – Google launched a call for projects exploring the potential of quantum computers. The University of Trento was among the eight winners worldwide, the only institution in the entire European Union."
The group led by Professor Hauke chose to work on a question that concerns elementary particles, in particular lattice gauge theory, according to which continuous spacetime is discretized typically into a hypercubic lattice of points. The question concerned the ways in which electrons, positrons and, in perspective, quarks and gluons interact to form particles and matter.
The research team wrote an algorithm that was sent to Google's powerful computers, which performed the computations remotely. These quantum supercomputers, located in Santa Barbara, use the quantum properties of matter to describe quantum objects in a very natural way, something that the classic "bits", based on the binary opposition between 1 and 2, cannot do.
"To give you an idea of the potential of these computers – continues Hauke – we can say that classical instruments, without further approximations, are able to correctly solve the dynamics of systems with a maximum of 40 particles. Quantum computers have the potential to process an exponentially greater number. To achieve this goal, however, it is necessary to work at the interface between fundamental physics and engineering. And that is where our research is located."
What about the future of this research? Hauke replies: "At the moment, our research is interesting for theoretical and experimental physics. In the future, however, it could have various applications, for example in the industrial sector for the study of new materials, or in the pharmaceutical sector for chemical compounds."
About the article
The results of the research have been published in Nature Physics. The article "Confinement in ℤ2 lattice gauge theory on a quantum computer" is available at: https://doi.org/10.1038/s41567-024-02723-6.
The authors of the study are Julius Mildenberger (Pitaevskii Bec Center, CNR-INO and Department of Physics of the University of Trento, Trento Institute for Fundamental Physics and Applications), Wojciech Mruczkiewicz (Google Quantum Ai), Jad C. Halimeh (Pitaevskii Bec Center, CNR-INO and Department of Physics of the University of Trento, Ludwig-Maximilians-Universität München and Munich Center for Quantum Science and Technology), Zhang Jiang (Google Quantum Ai) and Philipp Hauke (Pitaevskii Bec Center, CNR-INO and Department of Physics of the University of Trento, Trento Institute for Fundamental Physics and Applications).
Journal
Nature
Method of Research
Computational simulation/modeling
Subject of Research
Not applicable
Article Title
Confinement in a Z2 lattice gauge theory on a quantum computer
Article Publication Date
13-Jan-2025
Yu & Martin adapting mixed reality training programs to real-world scenes to enhance human-AI teaming in emergency responses
Lap Fai (Craig) Yu, Associate Professor, Computer Science, College of Engineering and Computing, and Joel Martin, Associate Professor, Kinesiology, College of Education and Human Development, received funding for the project: “EAGER: TaskDCL: Adapting Mixed Reality Training Programs to Real-World Scenes to enhance Human-AI Teaming in Emergency Responses.”
This EArly-concept Grant for Exploratory Research (EAGER) project funds research that intends to speed up the development of mixed reality and artificial intelligence (AI) technologies to help first responders, aiming to reduce training-related risks and casualties.
The research team will work with the Fairfax Fire and Rescue Department to explore how AI can be used in mixed reality tools to improve training and effectiveness for first responders. By adding virtual elements like fires, hazards, firefighters, robots, and people in need of rescue to real-life scenes, these mixed reality scenarios help first responders practice handling real-world challenges through interactive training.
The project will also involve a postdoctoral researcher and undergraduate students, including those from underrepresented groups in science and technology fields. The team will share their findings at conferences focused on mixed reality and training.
This EAGER project offers a novel interdisciplinary research perspective by integrating concepts and techniques from mixed reality, AI, human-computer interaction, and movement science to advance first responder training.
The researchers aim to devise a novel optimization-based generative framework for adapting mixed reality training scenarios to real scenes, which will offer ample training opportunities for first responders to practice, accomplishing different first-responder tasks (e.g., firefighting, search and rescue) via human-AI collaboration enabled by mixed reality headsets.
To carry out the research, the team will first investigate how AI techniques could be integrated with mixed reality devices to provide first response assistance. The team will then devise a generative framework based on optimization techniques for adapting mixed reality training scenarios to real scenes. They will conduct user studies to evaluate the performance gain brought about by the advanced mixed reality interfaces and the synthesized training scenarios.
Yu received $299,861 from the National Science Foundation for this research. Funding began in Jan. 2025 and will end in late Dec. 2026.
###
Arc Institute partners with NVIDIA to accelerate the future of computational biomedical research
Arc Institute
The Arc Institute today announced that it is working with NVIDIA to accelerate scientific research by developing and sharing powerful computational models and tools that advance biomedical discovery. The work brings together Arc’s biology researchers with NVIDIA's computing experts and both organizations' machine learning teams to advance the capabilities of the global biomedical research community.
“The convergence of biology and artificial intelligence holds exciting promise to transform the way we do science,” said Arc Co-Founder, Core Investigator, and Executive Director Silvana Konermann. “Our collaboration with NVIDIA will pioneer new approaches to exploring and engineering living systems, accelerating insights into complex human disease.”
“Generative AI has revolutionized our ability to model complex biology digitally, offering researchers a new instrument to scale science through machine learning,” said Anthony Costa, Director of Digital Biology at NVIDIA. “Combining Arc Institute's researchers and NVIDIA's AI experts, we are working to turn massive scientific datasets into invaluable scientific tools and insights.”
Arc’s biology and machine learning researchers are working with NVIDIA’s engineers to scale the potential of AI models to design, predict, and understand biology. This collaboration will build on Arc’s previous work with the Evo model, which is capable of both prediction and design at the level of DNA and across RNA and proteins in single-celled organisms.
“Machine learning provides a universal framework that harnesses data, compute, and scale to learn complex patterns from diverse data that would not be discernible with the human eye alone," said Arc Co-Founder and Core Investigator Patrick Hsu, and one of Evo’s lead researchers. “Evo is a foundation model for biology–a single unifying system that can work across different modalities and scales of complexity, from individual base pairs all the way up to genome-scale sequences.”
"By training these models on diverse biological data, we aim to discover emergent properties similar to those found in language, videos, and robotics,” said Brian Hie, one of Evo’s lead researchers and an Arc Institute Innovation Investigator in Residence. “We're enabling researchers to leverage complex generative models in ways that could unlock biological design at scales previously inaccessible to science."
Arc Institute is contributing to and sharing its upcoming models with global developers through the open-source NVIDIA BioNeMo Framework, a collection of accelerated computing tools for biomolecular research. These models will also be available as NVIDIA NIM microservices, a set of optimized, easy-to-use, portable AI microservices designed to unlock an entirely new scale of AI-driven biomolecular therapeutic design exploration.
As part of the collaboration, Arc biology and machine learning researchers can use BioNeMo via NVIDIA DGX Cloud on AWS, which gives them access to a high-performance, fully managed AI platform optimized for large-scale, distributed training with NVIDIA AI software and expertise.
Arc and NVIDIA will detail the first outputs of their partnership later this year.
###
The Arc Institute (@arcinstitute) is an independent nonprofit research organization located in Palo Alto, California, that aims to accelerate scientific progress and understand the root causes of complex diseases. Arc’s model gives scientists complete freedom to pursue curiosity-driven research agendas and fosters deep interdisciplinary collaboration.
