A General Goal-Conditioned Minecraft Model - Pantograph

At Pantograph, we're working on training fully general robotics models that can act autonomously for hours at a time. Especially in robotics, it's difficult to get diverse data at scale. Learning to act from internet video data could allow models to scale with compute, rather than being limited by small action datasets.In this work, we develop a simple method for learning goal-directed behavior through pretraining on internet-scale video.Usually, goal-directedness is taught in a post-training phase, which limits the extent to which it can generalize. Here, we learn goal-directedness during pretraining on internet-scale video, which greatly improves the models' ability to achieve complex goals.Video games are a useful testing ground for robotics. We're starting with Minecraft because it's open-ended, and supports commanding diverse, long-horizon goals.11This model can only act within the one game, but we will be training larger models across a broader distribution of visual environments in the coming months.↩ Our largest model, which we call Pan, is a 4B parameter model that can fight mobs, explore to find specific objects, complete challenging platforming environments, and build structures as commanded. We find it is able to generalize to quite diverse environments that it has never seen before.Reinforcement learning is interesting in part because it can produce models that are superhuman. If we can do reinforcement learning at scale on video, we may be able to produce models that are superhuman in many more areas all at once.#Reinforcement learning from observation-only dataWe view internet-scale videos as reinforcement learning trajectories that include only observations.22This lens makes sense for some modalities and not others. Most text data, for example, can't be viewed as a written transcript of an agent interacting with an environment. It could likely work for audio, however.↩ It's difficult to do reinforcement learning from this kind of data because it lacks rewards and actions, but it turns out there are simple solutions for both.Goal-conditioning is a simple way of side-stepping the need to specify a reward function. The idea is to use what happens later in a video as the goal for the earlier part, a trick sometimes called hindsight relabeling,33Marcin Andrychowicz et al., Hindsight Experience Replay, NeurIPS 2017.

↩ because it takes what may have been a failure at one goal, and relabels it as a success for whatever actually happens. In principle, you could use any description of what happens later in a video as the goal, but in this work we focus on using frames of video as goals.Although videos don't contain actions, it's still possible to learn functions from video that only depend on states. This includes value functions,44Typically, value functions are tied to a specific policy. In this very general setting, where any number of policies may be acting in any kind of environment, it makes more sense to use in-context learning to infer the tendencies and level of skill of the policies in a specific video. This way a single model can be trained that can act as a generic value function, capable of evaluating any policy's behavior.↩ which in goal-conditioned RL represent roughly the probability that a goal frame is achieved in the agent's future, as well as the next frame distribution for a goal-conditioned policy.55Both of these can be modeled in a number of ways, including likelihood-based models, energy-based models, and contrastive learning. We're excited about future work that could study the trade-offs between these different approaches.↩After this action-agnostic pretraining, a smaller dataset containing actions can be used to produce an agent capable of acting in the environment.66Given a strong value function, learning to act is a much easier problem (Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018). A value function effectively converts the very difficult multi-step decision making problem of reinforcement learning into a single-step decision problem, which is generally much more tractable.↩We think that this lens of viewing video as RL trajectories is fruitful, and there are many ideas yet to be explored. If this is the kind of work you're interested in, join us!Selected rollouts from Pan-4b in various environments. On the top are the rollout videos, and on the bottom are the goal images that the model is commanded with.#TrainingWe pretrained a family of goal-conditioned models from scratch on about 500k hours of diverse videos of Minecraft gameplay. Afterwards, we post-trained them on about 2k hours of contractor trajectories including both videos and action sequences.

The pretraining is action-agnostic, and the action space only comes in during post-training. (See Appendix for more details.)#EvaluationWe created a suite of 104 evaluation environments. Each environment consists of an initial Minecraft save world and a goal save world. We load the goal save world once per rollout and render a single goal image from it as the model's prompt. We use a mix of programmatic and language model-based graders to assess model performance, depending on the environment.#ComparisonsWe compared to STEVE-1, an open source goal-conditioned policy for Minecraft. STEVE-1's goal conditioning comes from post-training on a contractor dataset with action labels, on top of VPT, a model pretrained by OpenAI using an inverse dynamics model to label internet videos with estimated actions.88Shalev Lifshitz et al., STEVE-1: A Generative Model for Text-to-Behavior in Minecraft, 2023; and Bowen Baker et al., Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos, 2022.↩We also trained a VLA baseline on the same post-training dataset of contractor trajectories that we used for Pan. This VLA model was initialized from Gemma 4, one of the strongest open-source language models with support for video inputs.77Google DeepMind, Gemma 4 model card, 2026.↩ In total, much more compute went into Gemma 4 than into the models we trained, though our models have likely seen more Minecraft data through their pretraining.#ResultsPerformance by categoryPan-4BSTEVE-1VLAMean score across environments in each category, weighting every environment equally. Error bars are 95th percentile bootstrap confidence intervals.The models trained with goal-conditioned pretraining are better able to accomplish their goals in our evaluation worlds, including for varied and off distribution goals. We find that model size matters most for complicated, semantic things like mechanisms, and dextrous tasks like building. Our models learn some mechanisms (buckets, flint and steel, crops, fishing) and not others (levers, ender pearls, animal taming).

We are excited to see what capabilities emerge from further scale-ups.#BasicsWe begin simply: if we command a goal a little bit in front of the model, does it care about the goal frame at all? Will it stay there? We find that the Pan models walk to the target position and oscillate around it for the remainder of the rollout, seeming to try to line up their view with the goal frame perfectly. We call this a "goal dance", and it is a useful indicator that the model thinks it has achieved the goal. STEVE and the VLA are less consistent on these simple tasks, tending to wander off rather than stay on the goal, though they do accomplish them sometimes.Walk forward to villageModelScoreError bars show 95% CIPan-4B85.7%STEVE-116.5%VLA18.7%EnvironmentsIn Minecraft, building a tower underneath you is almost as basic as walking forward, and all of the models are able to do it. Our models again demonstrate more faithful goal following: whereas STEVE and the VLA tower up way beyond the commanded image, the Pan models frequently stop at the commanded height.Small towerModelScoreError bars show 95% CIPan-4B70.8%STEVE-16.7%VLA14.9%EnvironmentsWhat about simple but unusual goals, like looking at a target made out of wool, or swimming through a pool towards a gold panel? This is easier to do, but less common in the training distribution. Here, the difference between model families is larger: Pan models are much more goal directed.Wool targetModelScoreError bars show 95% CIPan-4B99.4%STEVE-11.0%VLA3.2%Environments#BuildingWe test building ability in various settings, varying how much of the final structure we show in the goal frame, what materials are used, and how much is built at the start of the eval. At this level of difficulty, STEVE and the VLA are almost entirely incapable of producing sensible actions.

Within our model family, we find a significant difference from scale: Pan-4B scores 34.7%95% CI[32.0%, 37.4%], Pan-2B 26.9%95% CI[24.4%, 29.4%], and Pan-200M 13.9%95% CI[12.1%, 15.8%].Add wood floorModelScoreError bars show 95% CIPan-4B38.3%STEVE-15.0%VLA4.1%EnvironmentsWe find that our models have difficulty recovering from mistakes. They will sometimes break wood or stone with their bare hands, but with only 30 seconds they can only do this so many times (these models don't have access to hotbar switching). Another failure mode is that the models sometimes wander around, seeming to look for the structure already built elsewhere in the world. It would be interesting to train larger models with much longer context windows, to see if they eventually give up looking and start building the structure themselves.#MechanismsTo play Minecraft competently, you must know about dozens of complex and specific mechanisms. Here we find significant returns to scale. On a diverse set of 30 environments testing many different mechanisms, Pan-4B scores 24.4%95% CI[22.3%, 26.5%], compared with 15.1%95% CI[13.5%, 16.9%] for the next-best model, Pan-2B—a margin of 9.3%95% CI[6.5%, 12.1%]. We find that Pan-4B significantly outperforms all other models tested.

Some notable successes:Hoe three blocksModelScoreError bars show 95% CIPan-4B85.4%STEVE-18.3%VLA8.3%EnvironmentsAnd failures:Tame wolfModelScoreError bars show 95% CIPan-4B12.5%STEVE-18.3%VLA0.0%Environments#CombatWe command Pan to fight by putting it in a narrow hallway, with a monster or two in between it and a gold panel. Pan-2B and Pan-4B have learned an effective general-purpose strategy for fighting monsters, first hitting, then backing up and waiting, then going forward and hitting again. This allows Pan to kill creepers and zombies effectively, even in tight quarters. Pan does not use a shield well, and often fails to kill skeletons.Creeper hallwayModelScoreError bars show 95% CIPan-4B45.3%STEVE-144.9%VLA21.0%Environments#Difficult EnvironmentsWe construct some environments to be out-of-distribution by including substantial hand-made structures.The following environment is called gauntlet. Pan has never seen this construction before, but still moves effectively, and the largest model sometimes completes it (note that grading on these envs is continuous; Pan is the only model to complete them):GauntletModelScoreError bars show 95% CIPan-4B51.2%STEVE-121.4%VLA6.4%Environments#Reward HackingWe've found some interesting instances of reward hacking, where the model finds a way to reproduce the goal image that wasn't what we intended. In one, rather than riding a horse over some fences to reach a gold panel, it sneaks around the fences, and jumps in order to reproduce the higher viewpoint it would have had were it on the horse. In another, rather than building a path from one side of a bridge to another, it falls through the gap but then manages to climb back up the other side of the ravine and reach the goal.Note that unlike the reward hacking that occurs with online RL methods, this model was trained with purely offline data, and has never seen these environments before.