---

Profile

Shuji Suzuki
Born in 1987 in Saitama Prefecture. Researcher at Preferred Networks. After completing his doctoral program at the Graduate School of Information Science and Engineering at Tokyo Institute of Technology (now Institute of Science Tokyo), he worked at Fujitsu Laboratories before joining the company in 2017. He has been engaged in research and development in the fields of bio-healthcare and large-scale distributed deep learning. He currently leads pre-training development for the large language model “PLaMo” developed by the PFN group.

Kaizaburo Chubachi
Born in 1994 in Miyagi Prefecture. Engineer at Preferred Networks. After completing his master’s program at the Graduate School of Information Sciences at Tohoku University, he joined the company in 2019. He has been involved in projects such as the development of autonomous operation systems for chemical plants. He currently leads post-training development for the large language model “PLaMo” developed by the PFN group.

---

Inspired by Near-Future Anime: Entering the Worlds of Bio and Information Science

──What first sparked your interest in generative AI?

Suzuki: I have loved anime since I was young, and a major turning point for me was a TV anime I watched in kindergarten, Future GPX Cyber Formula. The machines in that series were equipped with AI for navigation, and I became fascinated with the idea of creating something like that myself.

In high school, I was torn between pursuing information science and medicine, but ultimately chose information science. In my third year of undergraduate studies, I encountered bioinformatics, and was deeply impressed by the fact that it allowed me to study both information engineering and medicine. That led me to begin research aimed at integrating biological science and information science.

──What kind of research did you conduct specifically?

Suzuki: The field of bioinformatics is quite broad, but my research focused on protein sequence analysis. To understand protein functions, it is important to identify characteristic sequences, so I worked on developing systems that could rapidly search for target sequences within vast datasets.

──You also worked at a manufacturer’s research institute for a period. What led you to join PFN afterward?

Suzuki: I had the opportunity to intern at PFN during the first year of my doctoral program, where I met one of the founders, Nishikawa. Later, when I heard from him that the company was serious about advancing work in the life science field, I felt that if I wanted to truly pursue the integration of biology and information science, this was the right place. That led me to decide to join the company in 2017. After joining, I was involved in projects such as developing systems for disease diagnosis and prediction using blood data.

From an Interest in Manufacturing to a Passion for Programming

──What sparked your interest in generative AI, Mr. Chubachi?

Chubachi: I had been interested in manufacturing from an early age, but after entering the engineering faculty, I became deeply engaged in programming. At first, I enjoyed developing smartphone applications and games, but in my third year as an undergraduate, I learned about AlphaGo, which sparked my interest in machine learning. A major turning point for me was becoming fascinated by the challenge of pursuing highly efficient algorithms.

──What led you to join PFN?

Chubachi: During my studies, I interned at several companies, but what ultimately drew me to PFN was the abundance of development resources, such as GPUs, and the well-equipped experimental environment.

──What kind of impression did you have of PFN at the time?

Chubachi: I had the impression that it was a “unique company” where top competitive programmers and Kaggle Grandmasters work. However, through my internship, I learned that the company had many clients in the industrial sector, and I was attracted by the opportunity to apply my programming skills to solving real-world problems. That led me to join PFN as a new graduate in 2019.

After joining the company, I spent about five years working on joint research projects with partner companies. One example was the development of autonomous operation systems for chemical plants, where I conducted research and development on building and optimizing predictive models to enable safe and efficient control of plants based on large volumes of sensor data.

Developing Generative AI Where Failure Is Not an Option: Its Challenges and Rewards

──Both of you have experience developing algorithms and systems for efficiently processing large volumes of data. At the time you joined the company, LLM projects had not yet begun, correct?

Suzuki: Since LLM development requires enormous computational resources, we initially considered it a field that was difficult to enter as a business area as of 2022, before the emergence of ChatGPT. However, the following year, when we learned that the GENIAC project would be launched, the company decided to pursue LLM development in earnest.

As I had been working on large-scale distributed learning in the bio domain, I was selected as a member of the project, as I was able to contribute to LLM development.

──Mr. Chubachi, what role are you responsible for in this project?

Chubachi: The development of LLMs can be broadly divided into two phases: large-scale “pre-training” and “post-training,” in which the foundation model is refined into a practical form after pre-training. Mr. Suzuki serves as the technical lead for pre-training model development, while I serve as the technical lead for post-training.

──We understand that PFN developed its LLM entirely from scratch. What was the most challenging aspect?

Suzuki: It is hard to single out just one thing—everything was challenging (laughs). But if I had to choose, it would be the difficulty of conducting trial and error, since pre-training LLMs requires enormous computational resources. For example, with “PLaMo-100B,” which we developed in the first cycle, we only had one opportunity to train the model during the development period, so failure was not an option.

In particular, improving Japanese-language performance requires large volumes of high-quality Japanese data. As a result, the most demanding part was the preparatory work, such as data collection and preprocessing.

Chubachi: In the post-training team, we were responsible for identifying issues in the pre-trained model and exploring solutions to address them. Compared to pre-training, we are able to iterate through trial and error, but the process of solving these problems was the most challenging aspect. At the same time, when we saw clear improvements resulting from the solutions we devised, it brought a great sense of accomplishment. Although it is demanding, I find it to be highly rewarding work.

LLM Development Is Not the Goal, but a Milestone Toward Solving Societal Challenges

──In the second phase of GENIAC, we understand that you are working on developing a new model with capabilities comparable to the 100B model. Could you tell us more about this initiative?

Suzuki: In the first phase, we confirmed that the quality and quantity of training data have a significant impact on model performance. Therefore, in the second phase, we first created a high-quality dataset of approximately 100 billion tokens in Japanese and English by leveraging our proprietary models, including PLaMo-100B. This is roughly equivalent to about 50 times the size of the Japanese version of Wikipedia.

Using this high-quality training data, we aim to develop a smaller model that surpasses the Japanese language processing performance of the 100B model while reducing computational costs and power consumption to less than one-tenth.

──Does that refer to the 30B model and the 8B model using MoE (Mixture of Experts)?

Suzuki: Initially, we planned to achieve our target by combining a 30B model with a model using MoE with 8B active parameters. However, during the training of a new 8B model, we had already surpassed the benchmark targets we had originally set.

As a result, after discussions with NEDO (New Energy and Industrial Technology Development Organization), we revised our approach and decided to aim to achieve our targets with a standalone 8B model without using MoE.

──So you are already seeing performance exceeding your initial targets. Could you also share your future goals?

Suzuki: As for PLaMo’s flagship model, we have already begun offering it as a service under the name “PLaMo Prime” since last December. However, we believe that LLMs should not be limited to chat-based AI, but should also serve as a foundation for creating problem-solving AI tailored to specific industries. Currently, PFN is also working on developing domain-specific generative AI trained with specialized knowledge in fields such as finance and bio.

Chubachi: In the future, we hope to develop AI that contributes not only to the fields we are currently working on, but also to areas such as disaster prevention and public administration, thereby benefiting society as a whole in Japan. There is no doubt that a time will come when generative AI is utilized in all aspects of society. To achieve this, we intend to continue refining our technologies.

Suzuki: One of PFN’s unique strengths is that, in addition to developing generative AI, the PFN group is also engaged in hardware development, including AI chips. For us, developing domestically developed LLMs is not the end goal, but rather a milestone along the way. We aim to continue advancing our technologies while constantly considering how AI can be applied in the real world.

GENIAC Top Page