GPT-6 Astra Tops the First GameLab Cognitive Leaderboard
Games are one of the best ways to measure cognitive skills.
People design games to challenge particular abilities. A memory game is built around remembering where things are, a puzzle around planning several moves ahead, and a card game around weighing risk and reading an opponent.
At GameLab, we released our first cognitive leaderboard, using diverse games to evaluate the cognitive capabilities of today's frontier AI models and quantify their cognitive profiles.
We started by defining the cognitive skills we wanted to measure. The table below lists the seven on this leaderboard and how we define each one.
| Cognitive skill | What it measures |
|---|---|
| Strategic Decision Making | Choosing actions under uncertainty or competition, weighing tradeoffs, anticipating others' moves. |
| Physical Reasoning | Mental simulation of physical properties, such as gravity, trajectory, momentum, and material interactions. |
| Memory | Retrieving and integrating information from previous states to inform current or future actions. |
| Spatial Reasoning | Understanding object relation in space, including rotation, navigation, and distances. |
| Planning | Simulating future states and evaluating the outcomes of a sequence of actions. |
| General Knowledge | Broad range of information about various subjects (such as language, science, history, geography, and pop culture) that is gathered from context outside of the game itself. |
| Visual Processing | Identify, match, and categorize objects based on visual properties. |
We then annotated Arkadium’s games, using a shared rubric to rate how strongly each game demands every skill on the list, and picked the game that tests each skill most directly. Take Block Champ: players drop pieces of different shapes onto a grid, and any row or column that fills up clears. Every move comes down to seeing where a shape will fit and how it will sit against the pieces already on the board. Spatial Reasoning is the skill the game leans on most, so it stands for Spatial Reasoning on the leaderboard. A model’s score on each game becomes its score for that game’s skill. The figure below shows the seven games we chose and the skill each one measures.
Gin Rummy
Strategic Decision Making
8 Ball Pool
Physical Reasoning
Memory Game
Memory
Block Champ
Spatial Reasoning
Connect The Dots
Planning
Daily Crossword
General Knowledge
Tile Match Flowers
Visual ProcessingThe leaderboard
The chart below ranks the eight models by their overall score, the average of their seven skill scores.
GPT-6 Astra finishes first overall. It has the top score in four of the seven games, more than any other model. No model is strong at everything: Grok 4.7 solved every Connect The Dots puzzle but scored 9.3 in Tile Match, the one game played from a picture. Below the top three the field is close: Kimi K3, GLM 5.2 and Gemini 3.1 Pro are within six points of each other, with Kimi K3 in fourth.
The spider graph below shows each model’s cognitive profile across the seven skills; click a model’s name to switch between them.
Even the best model makes illegal moves
The game refuses any move that breaks the rules, and the AI has to try again. No model avoided this completely. GPT-6 Astra, the top model, still made a few illegal moves, and most other models made far more. For instance, in Tile Match Flowers, about one in five of Kimi K3’s moves was refused, and in the Memory Game about one in seven of GLM 5.2’s were.
The replay below shows one of Astra’s illegal moves, from a game of Block Champ, where the goal is to fit pieces onto a grid and clear full rows and columns.
Astra had a two-square piece left in its tray and aimed it at a gap in the lower half of the board. Before moving, it wrote that it had checked both squares, found them empty, and that the piece would complete a row. In fact the upper square was already filled. When the piece lands, the visualizer marks both squares with red crosses and outlines the board in red: the game has refused the move. On its next try Astra put the same piece somewhere else, and this time it completed a column, which cleared.
What stands out is that Astra did not skip the check. It said it had looked at those exact squares, and it got the answer wrong. Its reasoning sounded careful, but it misread the board.
How we evaluate models on games
Each model plays the games on its own, connected to them through its API. Before the first move, it receives the rules of the game and the format for describing a move.
Every turn then follows the same loop. The game sends the model its current state in symbolic form: the board as a grid of symbols and numbers, or the cards as a list. The one exception is Tile Match Flowers, where the model receives a screenshot of the board instead. The model replies with the action it wants to take and a short explanation of why. If the action breaks the rules, the game refuses it and the model tries again. If it is legal, the game plays it and sends back the new state, and the loop repeats until the game ends. The diagram below walks through one turn, using a simplified Block Champ board.
Every state, action and explanation is recorded, so any game can be replayed move by move. Each model played every game several times. Its score for a game is the average over those matches, and its overall score is the average of its seven game scores.
Next steps
This is the first version of the GameLab cognitive leaderboard. Next, we will add more cognitive skills, such as Temporal Coordination, Social Reasoning and World-Model Learning, along with games that test them. We will also use more than one game for each skill, so that every score reflects the skill rather than a single game, starting with a trivia game for General Knowledge. And we will keep running more matches for every model, adding new models as they are released.