Researchers from Gwangju Institute of Science and Technology develop an intelligent observer for Esports

It utilizes an object detection algorithm that learns human spectating data to find engaging viewports

Peer-Reviewed Publication

GIST (GWANGJU INSTITUTE OF SCIENCE AND TECHNOLOGY)

 

IMAGE: AN ARTIFICIAL OBSERVER WILL INTERACT WITH COMMENTATORS WHILE ANALYZING REAL-TIME IN-GAME STATE FOR THE BEST SPECTATOR EXPERIENCE. view more 

CREDIT: KYUNG-JOONG KIM

Esports, already a billion-dollar industry, is growing, partly because of human game observers. They control the camera movement and show spectators the most engaging portions of the game screen. However, these observers might miss significant events occurring concurrently across multiple screens. They are also difficult to afford in small tournaments. Consequently, the demand for automatic observers has grown. Artificial observing methods can either be rule-based or learning-based. Both of them predefine events and their importance, necessitating extensive domain knowledge. Moreover, they cannot capture undefined events or discern changes in the significance of the events.

Recently, researchers from South Korea, led by Dr. Kyung-Jong Kim, Associate Professor in Gwangju Institute of Science and Technology, have proposed an approach to overcome these problems. “We have created an automatic observer using object detection algorithm, Mask R-CNN, to learn human spectating data,” explains Dr. Kim. Their findings were made available online on 10 October 2022 and published in Volume 213 Part B of Expert Systems with Applications journal.

The novelty lies in defining the object as the two-dimensional spatial area viewed by the spectator. In contrast, conventional object detection treats a single unit, for instance, a worker or a building, as the object. In this study, the researchers first collected StarCraft in-game human observation data from 25 participants. Next, the viewports—areas viewed by the spectator—were identified and labeled as “one.” The rest of the screen was filled with “zeroes.” While the in-game features are used as input data, the human observations constituted the target information.

The researchers then fed the data into the convolution neural network (CNN), which learnt the patterns of the viewports to find the “region of common interest” (ROCI)—the most exciting area for the spectators to watch. They then compared the ROCI Mask R-CNN approach with other existing methods quantitatively and qualitatively. The former evaluation showed that CNN’s predicted viewports were similar to the collected human observational data. Additionally, the ROCI-based method outperformed others in the long run during the generalization test, which involved different matchup races, starting locations, and playing maps. The proposed observer was able to capture the scenes of interest to humans. In contrast, it could not be done by behavior cloning—an imitation learning technique.

Dr. Kim points out the future applications of their work. “The framework can be applied to other games representing some of the overall game state, not only StarCraft. As services such as multi-screen transmission continue to grow in Esports, the proposed automatic observer will play a role in these deliverables. It will also be actively used in additional content developed in the future.”

 

***

 

Reference

DOI: https://doi.org/10.1016/j.eswa.2022.118979

 

Authors: Ho-Taek Joo1, Sung-Ha Lee2, Cheong-mok Bae1, Kyung-Joong Kim1,2

 

Affiliations:     

1School of Integrated Technology, Gwangju Institute of Science and Technology, 123 Cheomdangwagi-ro, Buk-gu, Gwangju, 61005, South Korea

2AI Graduate School, Gwangju Institute of Science and Technology, 123 Cheomdangwagi-ro, Buk-gu, Gwangju, 61005, South Korea

 

About the Gwangju Institute of Science and Technology (GIST)
The Gwangju Institute of Science and Technology (GIST) is a research-oriented university situated in Gwangju, South Korea. Founded in 1993, GIST has become one of the most prestigious schools in South Korea. The university aims to create a strong research environment to spur advancements in science and technology and to promote collaboration between international and domestic research programs. With its motto of “A Proud Creator of Future Science and Technology,” GIST has consistently received one of the highest university rankings in Korea.

Website: http://www.gist.ac.kr/

 

About the author
KYUNG-JOONG KIM (Member, IEEE) received the B.S., M.S., and Ph.D. degrees in computer science from Yonsei University, in 2000, 2002, and 2007, respectively. He worked as a Postdoctoral Researcher with the Department of Mechanical and Aerospace Engineering, Cornell University, in 2007. He is currently an Associate Professor with the School of Integrated Technology, Gwangju Institute of Science and Technology (GIST). His research interests include artificial intelligence, game, and robotics.

 

Funding Information:
This research was supported by the National Research Foundation of Korea (NRF) funded by the MSIT (2021R1A4A1030075).

---

JOURNAL

Expert Systems with Applications

DOI

10.1016/j.eswa.2022.118979 

METHOD OF RESEARCH

Computational simulation/modeling

SUBJECT OF RESEARCH

Not applicable

ARTICLE TITLE

Learning to automatically spectate games for Esports using object detection mechanism