Supertest evaluates performance of engineering students in Russia, the United States, India, China

A test developed jointly by HSE University Moscow and Stanford University assessed student performance in mathematics, physics, critical thinking

NATIONAL RESEARCH UNIVERSITY HIGHER SCHOOL OF ECONOMICS

Research News

 

IMAGE: IGOR CHIRIKOV, AFFILIATED RESEARCHER AT THE HSE INSTITUTE OF EDUCATION view more 

CREDIT: IGOR CHIRIKOV

A group of researchers representing four countries summed up the results of the Supertest, a large-scale study of the academic performance of engineering students in Russia, China, India, and the United States. It is the first study to track the progress of students in computer science and electrical engineering over the course of their studies with regard to their abilities in physics, mathematics, and critical thinking and compare the results among four countries. The article about study in Nature Human Behavior.

The HSE Institute of Education played a key role not only in collecting and analyzing data from Russia, but also in developing uniform assessment tools in mathematics and physics for all countries. The Institute conducted a full range of psychometric studies in addition to substantiating the quality of the measurements and the comparability of results across different countries.

The Supertest was initiated by Stanford University, HSE University Moscow, the Educational Testing Service (ETS), and partner universities in China and India. The study authors include Prashant Loyalka, an Associate Professor at Stanford University and a leading researcher at the HSE International Laboratory for Evaluating Practices and Innovations in Education; Igor Chirikov a senior researcher at the Center for Studies in Higher Education at UC Berkeley and an affiliated researcher of the HSE Institute of Education; and Elena Kardanova and Denis Federyakin , leading researchers at the Centre for Psychometrics and Measurements in Education at the HSE.

More than 30,000 undergraduate students participated in the study. The researchers collected a sample of students from elite and large universities, roughly equal in number for each country. In Russia, the sample included students from six Project 5-100 universities and 28 other universities. Their skill development was measured three times: upon entering university, at the end of their second year, and at the end of their studies.

The task of the specialists of the HSE Centre for Psychometrics and Measurements in Education was to develop tests that had questions that would be neutral for students of different countries and would yield adequately comparable results across different countries. 'Over the course of analyzing the test results, we have proven that we were able to achieve both tasks,' said Centre Director Elena Kardanova. 'Testing in different countries was conducted in accordance with the same rules, with the assistance of specially trained examiners. All students were offered the same incentives to participate. We additionally tested the sensitivity of the results to possible differences in student motivation.'

The Supertest showed that at the start of their studies, Russian students perform lower than Chinese students in mathematics and physics, but higher than students from India in mathematics. After two years of study, the gap between Russian and Chinese students narrows, while Indian students catch up with Russian students in mathematics.

One finding of the study that is cause for concern relates to engineering students' critical thinking skills. Initially, upon entering university, Russian engineering students outperform Indian students while performing lower than Chinese students. In terms of developing these skills over the course of their studies, students of all three countries perform lower than students in the United States. 'We found that, as the students progress in their studies, their critical thinking skills remain approximately the same in Russia and India, but significantly decrease in China. On the contrary, American students show improvement,' said Igor Chirikov. This is a serious problem, the researchers note, because technologies change rapidly, and in order to be able to master new ones, you need not only a firm grasp of the subject area, but, above all, skills of the 21st century.

Another unexpected result was the gradual decline of academic skills among engineering students in China. 'Students at Chinese universities have an extremely high level of skills upon enrollment, but over the course of their university studies this level decreases: this applies to physics, mathematics, and critical thinking. We observe this at both elite and large universities, albeit to different extents,' said Igor Chirikov. 'A possible explanation lies in the way undergraduate education is organized in China, where institutions put emphasis on lectures, and instructors are not as demanding as in Russia and India. As a result, students have less motivation to learn and are not held accountable for developing skills.'

The results of the Supertest provide insight into how graduates perform in the globally competitive market of future computer science and electronic engineering professionals. Each represented country is renowned for its engineering expertise, and it is the Chinese, American, Russian and Indian specialists who migrate to different countries that largely determine the technological progress around the world. The balance of power in this sphere of education can play a decisive role in who will win the technology race tomorrow.