Friday, May 19, 2023

Researchers build mosquito testing arena to discover how they find us over long distances

Peer-Reviewed Publication

CELL PRESS

Human scent draws malaria mosquitoes to warmed targets mimicking human skin to guide mosquito behavior and host selection. 

IMAGE: THIS IS A GRAPHICAL ABSTRACT DEPICTING HOW HUMAN SCENT DRAWS MALARIA MOSQUITOES TO WARMED TARGETS MIMICKING HUMAN SKIN. view more 

CREDIT: GIRALDO AND RANKIN-TURNER ET AL

We know a lot about mosquito preferences up close, but how do mosquitoes find us from up to a hundred meters away? Using an ice-rink-sized outdoor testing arena in Zambia, researchers found that human body odor is critical for mosquito host-seeking behavior over long distances. The team also identified specific airborne body-odor components that might explain why some people are more attractive to mosquitoes than others. The work appears May 19 in the journal Current Biology.

Most studies of mosquito preference have been performed in confined laboratory settings that probably don’t represent a mosquito’s experience in the wild. To test how the African malaria mosquito Anopheles gambiae locates and chooses human hosts over a large and more realistic spatial scale, researchers from Johns Hopkins Bloomberg School of Public Health’s Malaria Research Institute and Macha Research Trust teamed up to build a 1,000 mtesting arena in Choma District, Zambia.

“This is the largest system to assess olfactory preference for any mosquito in the world,” says neuroscientist Diego Giraldo (@dags263), a postdoctoral fellow at Johns Hopkins Bloomberg School of Public Health, one of the study’s first authors. “And it’s a very busy sensory environment for the mosquitoes.”

The testing arena contained a ring of evenly spaced landing pads that were heated to human skin temperature (35ÂșC). Each night, the researchers released 200 hungry mosquitoes into the testing arena and monitored their activity using infrared motion cameras. Specifically, they took note of how often mosquitoes landed on each of the landing pads (which is a good sign that they’re ready to bite).

First, the team compared the relative importance of heat, CO2, and human body odor for attracting mosquitoes. They found that mosquitoes were not attracted to the heated landing pads unless they were also baited with CO2, but human body odor was a more attractive bait than CO2 alone.

Next, the team tested the mosquitoes’ choosiness. To do this, they had six people sleep in single-person tents surrounding the arena over six consecutive nights, and they used repurposed air conditioner ducting to pipe air from each tent—containing the aromas of its sleeping occupant—onto the heated landing pads. As well as recording the mosquitoes’ preferences, the researchers collected nightly air samples from the tents to characterize and compare the airborne components of body odor.

“These mosquitoes typically hunt humans in the hours before and after midnight,” says senior author and vector biologist Conor McMeniman (@McMenimanLab), assistant professor at Johns Hopkins Bloomberg School of Public Health and Johns Hopkins Malaria Research Institute. “They follow scent trails and convective currents emanating from humans, and typically they’ll enter homes and bite between around 10 PM and 2 AM. We wanted to assess mosquito olfactory preferences during the peak period of activity when they're out and about and active and also assess the odor from sleeping humans during that same time window.”

They found that, night after night, some people were more attractive to mosquitoes than others, and one of the volunteers, who had a strikingly different odor composition from the others, consistently attracted very few mosquitoes.

The team identified 40 chemicals that were emitted by all of the humans, though at different rates. “It’s probably a ratio-specific blend that they're following,” says analytical chemist Stephanie Rankin-Turner (@stephrankin2), a postdoctoral fellow at Johns Hopkins Bloomberg School of Public Health, the study’s other first author. “We don’t really know yet exactly what aspect of skin secretions, microbial metabolites, or breath emissions are really driving this, but we’re hoping we’ll be able to figure that out in the coming years.”

Though each person’s odor profile varied from night to night, the researchers found some stable patterns. People who were more attractive to mosquitoes consistently emitted more carboxylic acids, which are probably produced by skin microbes. In contrast, the person who was least attractive to mosquitoes emitted less carboxylic acids but approximately triple the amount of eucalyptol, a compound found in many plants; the researchers hypothesize that elevated levels of eucalyptol may be related to the person’s diet.

The researchers were surprised by how effectively the mosquitoes could locate and choose between potential human meals within the huge arena. “When you see something moved from a tiny laboratory space where the odors are right there, and the mosquitoes are still finding them in this big open space out in a field in Zambia, it really drives home just how powerful these mosquitoes are as host seekers,” says Rankin-Turner.

This research was supported by funding from the Johns Hopkins Malaria Research Institute, Bloomberg Philanthropies, and the Human Frontier Science Program.

Current Biology, Giraldo and Rankin-Turner et al. ‘Human scent guides mosquito thermotaxis and host selection under naturalistic conditions’ https://www.cell.com/current-biology/fulltext/S0960-9822(23)00532-8 DOI: 10.1016/j.cub.2023.04.050

Current Biology (@CurrentBiology), published by Cell Press, is a bimonthly journal that features papers across all areas of biology. Current Biology strives to foster communication across fields of biology, both by publishing important findings of general interest and through highly accessible front matter for non-specialists. Visit: http://www.cell.com/current-biology. To receive Cell Press media alerts, contact press@cell.com.

Pollinators are attracted to humidity, not just scent

Peer-Reviewed Publication

CORNELL UNIVERSITY

ITHACA, N.Y. -- Humidity is as important as scent in attracting pollinators to a plant, new Cornell-led research finds, advancing basic biology and opening new avenues to support agriculture.

In a study published May 7 in Current Biology, a team of Cornell researchers and colleagues at Harvard University and the Montgomery Botanical Center found that the weevil responsible for pollinating the plant Zamia furfuracea was just as sensitive to humidity as to scent.

“The world of plant-insect interactions was drastically changed by the work that was done on visual and scent cues,” said first author Shayla Salzman, a postdoctoral National Science Foundation fellow in the School of Integrative Plant Science Plant Biology Section, in the College of Agriculture and Life Sciences. “And now we’re just starting to realize how many other factors are playing a role in plant reproduction and impacting insect decision making, pollination and success.”

Co-authors include Robert Raguso, professor of neurobiology and behavior (CALS); Chelsea Specht, the Barbara McClintock Professor of Plant Biology (CALS); Ajinkya Dahake, a Ph.D. candidate in Raguso’s lab; and Will Kandalaft ’21.

Dahake was first author of a groundbreaking study published in 2022 in Nature Communications that found humidity was acting as a signal to encourage hawkmoths to pollinate the sacred datura flower (Datura wrightii). Taken together, the studies demonstrate that two very distantly related plants actively use humidity to encourage pollination, Dahake said.

“Prior to our research, humidity was seen as just an outcome of evaporation of nectar, a side note,” he said. “What we’ve found is that this is an active process of the flower, coming through specialized cells, and these organisms may even have evolved to privilege this humidity release, because it attracts pollinators.”

Until now, the study of pollination and plant-insect interactions has focused on visual and scent markers – senses that humans can also interpret. Insects, however, are far more adept than humans at sensing changes in humidity, carbon dioxide and temperature, Salzman said.

“Especially as climate change directly impacts exactly those things,” she said, “it’s crucial that we understand how insects utilize all of that information in their interactions with plants.”

For example, farmers and food distributors could use such information to encourage pollination of food crops or to direct insects away from stored foods and toward traps, Salzman said.

While humans need relatively large changes in humidity before we can sense a difference, insects can sense minuscule changes, Dahake said.

“Insects have specialized receptors that respond to very small changes in humidity: Even a change of 0.2% to 0.3% will cause a neuron to fire,” he said. “Even a 1 part per million change in carbon dioxide concentration will cause an insect neuron response. What does that mean behaviorally? We’re just beginning to scratch the surface.”

-30-

NJ Medicaid reforms tied to increased use of opioid addiction treatment

Peer-Reviewed Publication

RUTGERS UNIVERSITY

Reforms to New Jersey’s Medicaid program successfully spurred ongoing increases in buprenorphine prescriptions for the treatment of opioid addiction, according to a Rutgers analysis.

Although medications such as buprenorphine effectively combat opioid addiction, less than 30 percent of potential users receive them nationwide. New Jersey sought to increase prescription numbers with three Medicaid reforms that took effect in 2019. The reforms:

  • Removed prior authorization requirements for buprenorphine prescriptions
  • Increased reimbursement rates for in-office opioid addiction treatment
  • Established regional centers of excellence for addiction treatment

Medicaid records showed steady increases in buprenorphine prescriptions before 2019, but the rate of growth increased by 36 percent after the reforms took effect, and this increased rate of growth persisted until the end of the study period in December 2020.

A similar trend affected caregiver behavior. The percentage of caregivers prescribing buprenorphine had been rising before the reforms took effect, but it increased faster after their implementation.

New Jersey also experienced faster growth in buprenorphine prescriptions to Medicaid patients than did other states, the analysis reported, a strong indication that the boosts stemmed from the state’s reforms rather than some unrelated nationwide trend.

The only disappointment came from the study’s measurement of long-term usage. The percentage of buprenorphine prescriptions that remained active for more than 180 days didn’t rise during the study period.

“Usage remains far below the ideal where virtually everyone battling opioid addiction receives an effective medical treatment like buprenorphine,” said Peter Treitler, research project manager for the Rutgers Institute for Health, Health Care Policy and Aging Research and lead author of the analysis published in JAMA Network Open. “However, our analysis suggests these reforms may get us to that point years before we would have reached it under the previous policies.”

Buprenorphine — a once-daily prescription pill sold in the United States as Suboxone, Zubsolv and Sublocade — works in two ways. First, it binds to the same brain cells as drugs such as opium, heroin, morphine, oxycodone and fentanyl. Once it’s in place, those other drugs struggle to dislodge it, bind to the target cells and produce addictive highs. Second, it stimulates a milder effect that reduces cravings for those other drugs and prevents withdrawal symptoms.

Regulators once placed extra restrictions on buprenorphine prescriptions because they worried its mood-altering effects would create more addicts than it cured. These restrictions, which forced caregivers to justify each new buprenorphine prescription at length and investigated providers who prescribed the drug “too often,” led many to avoid prescribing the drug at all, said Treitler, who added Medicaid’s low reimbursement rates for office-based addiction treatment further reduced buprenorphine availability to poor patients.

Regulators have relaxed many restrictions on buprenorphine because opioid addiction has increased. Recent research has shown buprenorphine to be both safer and more effective than previously thought. The reforms to New Jersey’s Medicaid program further reduced barriers to medication usage among its patients.

“They looked at what obstacles were blocking the usage of a valuable drug in this particularly underserved patient population,” Treitler said. “They removed several of the biggest obstacles. And the results so far suggest they’re getting the desired results.”

Indeed, the positive numbers seen in initiatives such as the one undertaken by New Jersey’s Medicaid program may finally be affecting the most important number of all: overdose deaths. After several decades of speedy increases, overdoses nationwide rose by just 500 in 2022.

The trend was even better in New Jersey. Total overdose deaths fell by 232 from 2021 to 2022, and they were 93 lower in the first three months of 2023 than in the first three months of 2022.

U$A

Early data from MedStar Health suggests addressing social risk factors through screening, support services, and digital tools can support improved maternal health

Research presented at the American College of Obstetrics and Gynecology Annual Clinical and Scientific Meeting discusses MedStar Health’s efforts to address health disparities among birthing individuals

Reports and Proceedings

MEDSTAR HEALTH RESEARCH INSTITUTE

BALTIMORE – Screening for social risk factors in routine clinical care and offering patients tailored resources to address needs could help improve maternal health outcomes, according to three new studies. These findings were presented today by MedStar Health researchers at the 2023 American College of Obstetricians and Gynecologists (ACOG) Annual Clinical & Scientific Meeting (ACSM) and build on the health system’s commitment to improving the health and well-being of mothers and infants in Washington, D.C., through its D.C. Safe Babies Safe Moms initiative.

The District of Columbia has some of the highest maternal and infant mortality rates in the U.S. with disparities by race and place[1][2] and Black birthing individuals are at much greater risk of poor maternal health outcomes than their White counterparts, even when education and employment levels are similar.

“We know that there are so many factors at play when we look to the root causes of what our maternal health rates are today in this country. Racial inequity is one of the major drivers of disparate health outcomes,” says Angela D. Thomas, DrPH, executive lead for the D.C. Safe Babies Safe Moms initiative and co-author for these research projects. “By examining the role that other stressors caused by structural racism play in patients’ lives, we can begin to provide support where gaps exist and develop interventions that help our patients thrive.”

The first study[3] analyzed data collected from 930 patients who were screened for social risk factors (including food and housing insecurity, transportation, unmet legal needs, intimate partner violence, and trauma history) at their initial prenatal and postpartum visits, with most of these patients identifying as Black and receiving public-sponsored health insurance plans. Researchers found that 78% reported at least one social or environmental risk factor that could negatively impact their health.

The second study[4], led by Georgetown Law in partnership with MedStar Health Research Institute, researchers found that offering legal support to patients with unmet legal needs can be a powerful tool to help address maternal health disparities, particularly among birthing individuals who are a part of historically marginalized or underserved communities and are at a higher risk of poor health outcomes. For health systems to be successful in offering legal services or establishing a medical-legal partnership, study authors suggested a two-pronged approach:

  1. Make screening for unmet legal needs and referrals to legal support services a routine part of prenatal and postpartum appointments. This will likely require changes to operational workflows and, as researchers predict, integrations with the electronic health record.
  2. Train healthcare team members to recognize the importance of tackling social determinants of health as a part of clinical management, how to spot unmet legal needs of their patients, and how to effectively communicate with patients about how social and environmental factors impact their health.

“Overcoming barriers, especially ones rooted in the law, can cause tremendous amounts of stress, even in the best of circumstances. When you are pregnant and faced with daily struggles around housing, education, employment, legal status, and income supports, that stress can be tenfold and cause a domino effect on your health and other aspects of you and your family’s lives,” says lead author S. Roxana Richardson, Esq., director of Georgetown University Health Justice Alliance’s Perinatal Legal Assistance and Wellbeing (LAW) Project at MedStar Washington Hospital Center and lead author on the study. “The Perinatal LAW Project helps families tackle these issues head-on by adding lawyers to the care team and creating a low-barrier path to access justice through early legal intervention. It’s an honor to present our data about this program and continue to build the evidence base for medical-legal partnerships as a critical resource for this vulnerable patient population.”

The third study[5] tested how using digital technologies like chatbots could provide critically important follow-up outreach to recently discharged birthing individuals during the postpartum period starting 24 hours after discharge through 42 days post-delivery. The research team developed a chatbot that patients could interact with using logic to both offer educational content on postpartum recovery and newborn care as well as address barriers to care. The team then solicited feedback from users and monitored the usage of the chatbot. Overall, they found 61% of participants opened the chatbot at any point in time, suggesting that the technology could be an effective channel to deliver information to birthing individuals.

However, the research team also found that the users who opened the chatbot were more likely to be White and have private insurance, suggesting the need for further exploration on how to use chatbot technology to ensure more equitable delivery across patient populations at higher risk for poor outcomes.

“The chatbot tool is a promising way to proactively reach and engage patients during a crucial period for birthing individuals and those with newborns, particularly for those who might not otherwise reach out with questions or have access to their healthcare provider after they leave the hospital,” says Hannah Arem, Ph.D., scientific director of implementation science at MedStar Health Research Institute and co-author of the study. “While we’ve seen early success with content developed by medical experts, our next phase of this work will be to incorporate user feedback to better tailor the tool to what patients most at risk for adverse outcomes need and want.”

The MedStar Health team’s work was funded through an investment made by the A. James & Alice B. Clark Foundation to establish the D.C. Safe Babies Safe Moms initiative which seeks to address maternal and infant health disparities in the District of Columbia.

For more information, visit MedStarHealth.org/SBSM.

 


[1] Centers for Disease Control and Prevention. District of Columbia. CDC. Updated February 3, 2022. Accessed December 1, 2022. https://www.cdc.gov/nchs/pressroom/states/dc/

[2] Conduent Healthy Communities Institute. Infant Mortality Rate. DC Health Matters. Updated December 2019. Accessed December 1, 2022. https://www.dchealthmatters.org/indicators/index/view?indicatorId=9671&localeTypeId=27

[3] C. Laccay, H. Rogovin, H. Arem, T. Auguste, A.D. Thomas, L. Patchen. (2023, May 19-21.) Early Indicators that Universal Screening for Social Risk Factors is Essential in the Perinatal Period. [Conference presentation abstract]. American College of Obstetrics and Gynecology Annual Clinical and Scientific Meeting.

[4]R. Richardson, L. Patchen, D. Schille Jensen, L. Kessler, D. Perry, A.D. Thomas. (2023, May 19-21.) Integrating Lawyers into the Perinatal Healthcare Team: Initial Findings from an Innovative Medical-Legal Partnership. [Conference presentation abstract]. American College of Obstetrics and Gynecology Annual Clinical and Scientific Meeting.

[5]H. Arem, K. Ganacias, A. Danielson, L. Patchen, J. Rethy, T. Auguste. (2023; May 19-21.) Tailored, just-in-time education and resources for birthing individuals and newborn caregivers after hospital discharge: developing a chatbot experience. [Conference presentation abstract]. American College of Obstetrics and Gynecology Annual Clinical and Scientific Meeting.

 

New research finds mechanism that regulates PTSD in the female brain

“For it to be doing this function in the context of PTSD in females is very surprising” said Tim Jarome, an associate professor in the College of Agriculture and Life Sciences’ School of Animal Sciences.

Peer-Reviewed Publication

VIRGINIA TECH

Kayla Farrell, 

IMAGE: KAYLA FARRELL, PH.D CANDIDATE IN THE SCHOOL OF ANIMAL SCIENCES, WORKS IN TIM JAROME'S LAB. PHOTO BY LUKE HAYES FOR VIRGINIA TECH. view more 

CREDIT: VIRGINIA TECH

From humans to plants to single-cell organisms, there’s a protein that rules them all.

This protein does general housekeeping of the cells, regulating them through normal daily functions.

Virginia Tech researchers found that one specific form of this ubiquitous protein has a different function in the female brains – it helps regulate events in the memory that cause post-traumatic stress disorder (PTSD).

Watch video: https://video.vt.edu/media/1_lnhr71n4

“The protein is primarily thought of as a protein that marks other proteins to be destroyed,” said Tim Jarome, an associate professor in the College of Agriculture and Life Sciences’ School of Animal Sciences. “For it to be doing this function in the context of PTSD in females is very surprising.”

This protein, ubiquitin, even has its name originating from ubiquitous because of its presence across all walks of life. The form of this protein that the researchers could manipulate, called K-63, was selective in forming fear memories in the female brain.

“Oftentimes, molecules are found in the brain that are involved in forming these fear-based memories in both sexes, and this is the first time that we found one that's selectively involved in one sex,” Jarome said. “In particular, this was found in the sex that seems to be more likely to have PTSD. It's rare to find these mechanisms that are specific to one sex in terms of regulating the underlying factors that cause PTSD.”

The discovery could lead to the development of better therapeutic treatments. Kayla Farrell, a Ph.D. candidate in the School of Animal Sciences, was the project lead.

The research was published recently in Molecular Psychiatry in the Nature Portfolio of Journals. The research was supported by a grant from the National Institute of Mental Health.

PTSD is a complex disorder with a variety of therapeutic treatment options that include pharmacological approaches. By having a specific molecule to target, pharmacological approaches could be considered.

“Right now, treatment options are not very effective and the success rate isn’t very good,” Jarome said. “PTSD is not created equal among patients, and we know females are more likely to have it. The therapeutic approaches that we take to treat it might have to differ between males and females. This may be a mechanism in which we could specifically target treatment in females as a way to treat PTSD.”

 

Novel gene-editing strategy leverages unusual genetic alteration to block HIV spread in cells

Peer-Reviewed Publication

TEMPLE UNIVERSITY HEALTH SYSTEM

Novel Gene-Editing Strategy Leverages Unusual Genetic Alteration to Block HIV Spread in Cells 

IMAGE: DR. KAMEL KHALILI, DR. TRICIA BURDO, DR. RAFAL KAMINSKI AND TEMPLE HIV RESEARCHERS AND COLLABORATORS view more 

CREDIT: LEWIS KATZ SCHOOL OF MEDICINE AT TEMPLE UNIVERSITY

(Philadelphia, PA) – Genetic alterations that give rise to a rare, fatal disorder known as MOGS-CDG paradoxically also protect cells against infection by viruses. Now, scientists at the Lewis Katz School of Medicine at Temple University have harnessed this unusual protective ability in a novel gene-editing strategy aimed at eliminating HIV-1 infection with no adverse effects on cell mortality.

The new approach, described online April 28 in the journal Molecular Therapy – Nucleic Acids, is based on a combination of two gene-editing constructs, one that targets HIV-1 DNA and one that targets a gene called MOGS – defects in which cause MOGS-CDG. In cells from persons infected with HIV-1, the Temple researchers show that disrupting the virus’s DNA while also deliberately altering MOGS blocks the production of infectious HIV-1 particles. The discovery opens up new avenues in the development of a cure for HIV/AIDS.

Proper MOGS function is essential for glycosylation, a process by which some cellular proteins synthesized in the body are modified to make them stable and functional. Glycosylation, however, is leveraged by certain kinds of infectious viruses. In particular, viruses like HIV, influenza, SARS-CoV-2, and hepatitis C, which are surrounded by a viral envelope, rely on glycosylated proteins to enter host cells.

In the new study, lead investigators Kamel Khalili, PhD, Laura H. Carnell Professor and Chair of the Department of Microbiology, Immunology, and Inflammation, Director of the Center for Neurovirology and Gene Editing, and Director of the Comprehensive NeuroAIDS Center at the Lewis Katz School of Medicine, and Rafal Kaminski, PhD, Assistant Professor at the Center for Neurovirology and Gene Editing at the Lewis Katz School of Medicine designed a genetic approach to exclusively turn on CRISPR to impede MOGS gene expression through DNA editing within immune cells that harbor replication competent, HIV-1. Their novel approach is expected to avoid any impact on the health of uninfected cells that retain normal MOGS gene function. Stimulation of the apparatus in HIV-1 infected cells disrupted the glycan structure of the HIV-1 envelope protein, culminating in the production of non-infectious virus particles.

“This approach is conceptually very interesting,” said Dr. Khalili, who is also senior investigator on the new study. “By mitigating the ability of the virus to enter cells, which requires glycosylation, MOGS may offer another target, in addition to the integrated viral DNA for developing the next generation of CRISPR gene-editing technology for HIV elimination.”

Dr. Kaminski, Dr. Khalili, and Tricia H. Burdo, PhD, Professor and Vice Chair in the Department of Microbiology, Immunology, and Inflammation and the Center for Neurovirology and Gene Editing at Temple and an expert in the use of non-human primate models for HIV-1, have been working together to further assess the efficacy and safety of CRISPR-MOGS strategy in preclinical studies. In previous work, the team demonstrated that CRISPR-based technology can successfully remove viral DNA from the cells of infected non-human primates.

Other researchers who contributed to the study include Hong LiuChen ChenShuren Liao, and Shohreh Amini, Department of Microbiology, Immunology, and Inflammation, Center for Neurovirology and Gene Editing, Lewis Katz School of Medicine at Temple University; Danielle K. SohaiiConrad R.Y. Cruz, and Catherine M. Bollard, Center for Cancer and Immunology Research, Children’s National Health System, The George Washington University; Thomas J. Cradick and Jennifer Gordon, Excision Biotherapeutics, San Francisco, CA;  Anand MehtaStephane Grauzam, and James Dressman, Department of Cell and Molecular Pharmacology, Medical University of South Carolina; and Carlos Barrero and Magda Florez, Department of Pharmaceutical Sciences, School of Pharmacy, Temple University.

The research was supported in part by grants from the National Institutes of Health and the W.W. Smith Charitable Trust.

Editor’s Note:

Kamel Khalili is Co-Founder and Chief Scientific Consultant and holds equity in Excision BioTherapeutics, which has licensed the viral gene-editing technology from Temple University. Kamel Khalili and Rafal Kaminski are named inventors on patents that cover the viral gene-editing technology. Tricia Burdo serves on the Scientific Advisory Board and holds equity in Excision BioTherapeutics. These named researchers are employed by Temple University and conduct research activities sponsored by the company. Questions regarding their affiliations with Temple University may be directed to coisom@temple.edu.

Dr. Khalili has not received financial compensation from any other third parties for any aspects of this published work. Shohreh Amini is the spouse of Kamel Khalili.  

In addition to owning the viral gene-editing technology that Excision is licensing, Temple University also holds an equity interest in Excision. As a result of these interests, Temple University could ultimately potentially benefit financially from the outcome of this research. These interests have been reviewed and approved by Temple University in accordance with its Institutional Conflict of Interest policy. Questions about this can be directed to coitemple@temple.edu.

About the Lewis Katz School of Medicine

Founded in 1901, the Lewis Katz School of Medicine at Temple University attracts students and faculty committed to advancing individual and population health through culturally competent patient care, research, education, and service. The School confers the MD degree; MS and PhD degrees in Biomedical Science; the MA in Urban Bioethics; the MS in Physician Assistant studies; a certificate in Narrative Medicine; a non-degree post-baccalaureate program; several dual degree programs with other Temple University schools; continuing medical education programs; and in partnership with Temple University Hospital, 40 residency and fellowship programs for physicians. The School also manages a robust portfolio of publicly and privately funded transdisciplinary studies aimed at advancing the prevention, diagnosis, and treatment of disease -- with specialized research centers focused on heart disease, cancer, substance use disorder, metabolic disease, and other regional and national health priorities. To learn more about the Lewis Katz School of Medicine, please visit: medicine.temple.edu.

Evolutionary reinforcement learning promises further advances in machine learning

Peer-Reviewed Publication

INTELLIGENT COMPUTING

Key research areas in evolutionary reinforcement learning 

IMAGE: KEY RESEARCH AREAS IN EVOLUTIONARY REINFORCEMENT LEARNING. view more 

CREDIT: HUI BAI ET AL.

Evolutionary reinforcement learning is an exciting frontier in machine learning, combining the strengths of two distinct approaches: reinforcement learning and evolutionary computation. In evolutionary reinforcement learning, an intelligent agent learns optimal strategies by actively exploring different approaches and receiving rewards for successful performance. This innovative paradigm combines reinforcement learning's trial-and-error learning with evolutionary algorithms' ability to mimic natural selection, resulting in a powerful methodology for artificial intelligence development that promises breakthroughs in various domains.

A groundbreaking review article on evolutionary reinforcement learning was published Apr. 21 in Intelligent Computing, a Science Partner Journal. It sheds light on the latest advancements in the integration of evolutionary computation with reinforcement learning and presents a comprehensive survey of state-of-the-art methods.

Reinforcement learning, a subfield of machine learning, focuses on developing algorithms that learn to make decisions based on feedback from the environment. Remarkable examples of successful reinforcement learning include AlphaGo and, more recently, Google DeepMind robots that play soccer. However, reinforcement learning still faces several challenges, including the exploration and exploitation trade-off, reward design, generalization and credit assignment.

Evolutionary computation, which emulates the process of natural evolution to solve problems, offers a potential solution to the problems of reinforcement learning. By combining these two approaches, researchers created the field of evolutionary reinforcement learning.

Evolutionary reinforcement learning encompasses six key research areas:

  • Hyperparameter optimization: Evolutionary computing methods can be used for hyperparameter optimization. That is, they can automatically determine the best settings for reinforcement learning systems. Discovering the best settings manually can be challenging due to the multitude of factors involved, such as the learning speed of the algorithm and its inclination towards future rewards. Furthermore, the performance of reinforcement learning relies heavily on the architecture of the neural network employed, including factors like the number and size of its layers.
  • Policy search: Policy search entails finding the best approach to a task by experimenting with different strategies, aided by neural networks. These networks, akin to powerful calculators, approximate task execution and make use of advancements in deep learning. Since there are numerous task execution possibilities, the search process resembles navigating a vast maze. Stochastic gradient descent is a common method for training neural networks and navigating this maze. Evolutionary computing offers alternative “neuroevolution” methods based on evolution strategies, genetic algorithms and genetic programming. These methods can determine the best weights and other properties of neural networks for reinforcement learning.
  • Exploration: Reinforcement learning agents improve by interacting with their environment. Too little exploration can lead to poor decisions, while too much exploration is costly. Thus there is a trade-off between an agent’s exploration to discover good behaviors and an agent’s exploitation of the discovered good behaviors. Agents explore by adding randomness to their actions. Efficient exploration faces challenges: a large number of possible actions, rare and delayed rewards, unpredictable environments and complex multi-agent scenarios. Evolutionary computation methods address these challenges by promoting competition, cooperation and parallelization. They encourage exploration through diversity and guided evolution.
  • Reward shaping: Rewards are important in reinforcement learning, but they are often rare and hard for agents to learn from. Reward shaping adds extra fine-grained rewards to help agents learn better. However, these rewards can alter agents’ behavior in undesired ways, and figuring out exactly what these extra rewards should be, how to balance them and how to assign credit among multiple agents typically requires specific knowledge of the task at hand. To tackle the challenge of reward design, researchers have used evolutionary computation to adjust the extra rewards and their settings in both single-agent and multi-agent reinforcement learning.
  • Meta-reinforcement learning: Meta-reinforcement learning aims to develop a general learning algorithm that adapts to different tasks using knowledge from previous ones. This approach addresses the issue of requiring a large number of samples to learn each task from scratch in traditional reinforcement learning. However, the number and complexity of tasks that can be solved using meta-reinforcement learning are still limited, and the computational cost associated with it is high. Therefore, exploiting the model-agnostic and highly parallel properties of evolutionary computation is a promising direction to unlock the full potential of meta-reinforcement learning, enabling it to learn, generalize and be more computationally efficient in real-world scenarios.
  • Multi-objective reinforcement learning: In some real-world problems, there are multiple goals that conflict with each other. A multi-objective evolutionary algorithm can balance these goals and suggest a compromise when no solution seems better than the others. Multi-objective reinforcement learning methods can be grouped into two types: those that combine multiple goals into one to find a single best solution and those that find a range of good solutions. Conversely, some single-goal problems can be usefully broken down into multiple goals to make problem-solving easier.

Evolutionary reinforcement learning can solve complex reinforcement learning tasks, even in scenarios with rare or misleading rewards. However, it requires significant computational resources, making it computationally expensive. There is a growing need for more efficient methods, including improvements in encoding, sampling, search operators, algorithmic frameworks and evaluation.

While evolutionary reinforcement learning has shown promising results in addressing challenging reinforcement learning problems, further advancements are still possible. By enhancing its computational efficiency and exploring new benchmarks, platforms and applications, researchers in the field of evolutionary reinforcement learning can make evolutionary methods even more effective and useful for solving complex reinforcement learning tasks.