Fujitsu Reports Boost to Todai Robot's College Exam Scores

Tokyo and Kawasaki, Japan, November 14, 2016

The National Institute of Informatics (NII), Fujitsu Laboratories Ltd., and Cybernet Systems Co., Ltd. today announced that, in cooperation with Nagoya University and the University of Tokyo, the "Can a robot get into the University of Tokyo?" AI project (also known as "Todai Robot"), launched by NII in 2011, has significantly surpassed its test scores from the previous fiscal year, having taken the essay-type mock test for the University of Tokyo's second scholastic aptitude test and the mock sheet type exam provided by the National Center for University Entrance Examinations (Center Test). This fiscal year, Todai Robot achieved a deviation value⁽¹⁾ of 76.2 in mathematics, and a deviation value of 59.0 in physics on a comprehensive mock exam.

The Todai Robot mathematics team, which is composed primarily of NII, Fujitsu Laboratories, and Nagoya University, took on the challenge of the University of Tokyo essay-type Entrance Exam Pre-Test⁽²⁾ from Yoyogi Seminar, a Japanese test-preparation company. In math (science course) after the text of the problems was input, the AI automatically handled everything else, from the interpretation of the problem text, to solving the equations and composing the answers, completely answering four out of six problems, achieving a score of 80/120 and a deviation value of 76.2. Last fiscal year, Todai Robot took the mock essay-type exam for the Sundai Prep School from the Surugadai Gakuen Educational Institute, scoring a deviation value of 44.3 (20 points) in math (science course). The mathematics team has been carrying out joint research centered on the three organizations since Fujitsu Laboratories joined in 2012 and Nagoya University joined in 2013. Within the response development process, the first half, namely natural language processing, was driven mainly by Nagoya University. The second half, the math processing, was handled primarily by Fujitsu Laboratories.

The Todai Robot physics team, made up of Fujitsu Laboratories, Cybernet Systems and the University of Tokyo, took on the challenge of the "Shinken Comprehensive Academic Mock Exam"⁽³⁾ Center Test from Benesse Corporation, as was the case last fiscal year, incorporating the technologies confirmed by the mathematics team. Todai Robot uses simulations to solve physics problems and humans were involved in the configuration of these simulations. However, this could be automated if existing natural language processing technologies and image processing technologies were used to understand text and images. From the logical form stage the problems were automatically solved using AI, and resulted in a score of 62/100 and a deviation value of 59.0. Compared with last fiscal year, this is an improvement of 12.5 points in the deviation value and 20 points in the test score. Until last fiscal year, research and development for the physics team was carried out primarily at NII, but beginning this fiscal year, Fujitsu Laboratories, Cybernet Systems and the University of Tokyo joined the project, and are carrying out joint research under a new structure centered on the three organizations.

Development Background

A university entrance exam problem is an integrated matter that requires various technologies, led by natural language processing used to analyze the problem's text. Taking this approach, and using points and deviation values, quantitatively assessing results is an achievable task. In applying AI to a university entrance exam problem with such characteristics, it is the Todai Robot project's goal to be an objective benchmark that can be an indicator to guide consideration of a question such as "what is the field in which it would be possible for AI to replace a human." The Todai Robot project was launched in April 2011, and has pursued the evaluation and review of each fiscal year's research results while also distilling issues with the technology. Up to this fiscal year one of its goals has been to score high Center Test marks.

For mathematics, the text of the problems, expressed in natural language and formulae that are easy for humans to understand, is converted by a computational program into an executable format, then the problem is solved by a computer algebra program ("solver"), and the answer is output in natural language that can easily be understood by humans. In addition, because the problems in the physics test have similar settings to real-world problems, it is thought that the results will lead to AI development necessary for cooperation between people and computers or robots. For example, the problems of text, image, and quantities for Todai Robot today, would be problems of verbal commands from a human, camera image recognition, and control values for motors, etc., for the robot of the future.

Issues

Mathematics

Until last fiscal year, problems were solved by a solver that used repeated transformations of mathematical formulae, called "quantifier elimination"⁽⁴⁾. This is a general-purpose method of solving problems presented by polynomial equations and inequalities with real variables, which can solve problems from a wide range of mathematical fields in university entrance exams. Because it could not be directly applied to problems involving trigonometric functions, some problems were left blank.

Physics

Until last fiscal year, research and development was carried out with the goal of building a system in which the AI could use language processing technology to analyze the problem text, convert it to a logical form, set the initial conditions of a physics simulation based on the problem, and answer the problem using the results of the simulation. In the development process until last fiscal year, the system reached a point where, if the initial conditions for the simulation were input by a human, it was confirmed that it could solve a portion of the entrance exam problems by replicating the physical phenomenon, but it had not achieved automation from the logical form, as human intervention was still necessary.

About the Technology

Mathematics

With regard to computer algebra, the mathematics team developed technology that could convert the types of problems involving trigonometric functions that often appear on entrance exams into a format where quantifier elimination could be applied to solve the problem, expanding the types of problems the solver could handle. In addition, by expanding the language processing functionality of the system, including development of the math formula analyzer and the inter-sentential relationship analyzer, expansion of its grammar and dictionary resources, and improvements to its syntactic analysis technology, the team expanded the boundaries of the problems it could solve autonomously.

As a result of these developments, the system was able to achieve a deviation value of 76.2 with a test score of 80/120 on the science course of the University of Tokyo Entrance Exam Pre-Test entirely automatically, without human intervention (Figure 2).

Physics

The physics team has developed technology that can build a conditional expression that expresses the circumstances of the problem from a logical form which is assumed to be possible to generate with language processing, and then use quantifier elimination to simplify the expression and automatically generate the initial conditions for the simulation. In addition, the team developed a dedicated solver which can also use information from images, for problems that are difficult to handle with the simulator, such as problems which add components to the entrance exam problem simulation, or equilibrium problems where the initial conditions for the simulation are the answer to the problem. Using these measures, in a challenge which expanded the boundaries of automation more than last fiscal year, the system achieved a deviation value of 59.0 with a test score of 62/100, a massive improvement over previous results (Figure 4).

Future Plans

The Todai Robot mathematics and physics teams aim to further improve the accuracy of each language processing step, both syntactic analyzer and inter-sentential relation analysis, to advance research and development on technology to "read while thinking" by devising a processing order for the intermediate stage between language processing and computer algebra, and by further combining with natural language processing and computer algebra, and to develop technology to correctly answer a wider variety of problems.

In a collaboration between academia and industry to promote research, joint research is being carried out by researchers across different specialty fields, such as natural language processing and computer algebra, from Nagoya University, Tokyo University of Science, the University of Tsukuba, and Rikkyo University. The Todai Robot project shows how this kind of cross-disciplinary research contributes to cultivating more well-rounded people with a depth of specialization and a breadth of knowledge, and more sophisticated problem-solving abilities.

Through the Todai Robot project, Fujitsu Laboratories will continue to work on the development of advanced language-processing technologies and mathematic techniques, and will use them in Human Centric AI Zinrai, Fujitsu's AI platform.

Cybernet Systems will apply the technology developed through this project to the development of Maple and MapleSim, products of its subsidiary - Maplesoft, with a goal of contributing to the industrial application of mathematics technology.

Related Websites

• "National Institute of Informatics, Fujitsu, and Nagoya University Significantly Enhance Robot Project Math Score" (press release dated November 14, 2015)
• "Fujitsu Supports Artificial Brain Project with Prep School Test Challenge" (press release dated November 25, 2013)
• "Fujitsu to Participate in Artificial Brain Project, 'Can a Robot Pass the University of Tokyo Entrance Exam?'" (press release dated September 10, 2012)