Experts at OpenAI Trained a Neural Network to Play Minecraft

The neural network was trained on 70 000 hours of various gaming videos, supplemented by a database of 2,000 hours of video clips in which contractors performed specific gaming tasks, while also recording keyboard and mouse inputs.

Data about the keys that players pressed was used for markup. As part of the training, the AI used an emulation of a standard mouse and keyboard and thereby learned how to process the video, guess keystrokes and record them.

The importance of the Minecraft project is that it demonstrates the effectiveness of a new technique used by OpenAI to train AI models - called Video PreTraining (VPT) - which the company says could accelerate the development of "general computer-using agents".

To fine-tune the underlying model, the team then plugs in smaller datasets designed for task-specific training. In this context, OpenAI used videos of players performing actions in an early game, such as cutting down trees and building craft tables, which is said to have made a "significant improvement" in the reliability with which the model could perform these tasks.

To encourage further experimentation in this area, OpenAI is partnering with the MineRL NeurIPS competition, providing its contractor data and model code to contestants trying to use AI to solve complex Minecraft tasks. Grand prize: $100,000.