Memory — Concentration — is the childhood game of turning cards over two at a time and remembering what you saw.
Your Mission
Twenty-two tiles sit in a fixed pattern on a 7×7 board and form eleven matching pairs. Clear all eleven.
This Gamelab ruleset plays the Level 10 board with the client's 150-second countdown switched off; the episode ends at the turn cap instead.
How a turn works
- Exactly one tile is flipped per turn.
- The first flip reveals a picture and the tile stays face up.
- The second flip is compared immediately: a match clears both tiles and scores, a mismatch turns both face down again and hides their pictures.
- Cleared tiles stay cleared.
The notepad
The model has no memory between turns and the board tells it nothing about a face-down tile. The only thing carried forward is a short notes field it writes itself and receives back the next turn.
Anything it saw and failed to write down is gone. That makes the game as much a test of note-keeping as of recall.
Memory Game as an Evaluation Tool
Memory turns an ordinary limitation of language models into the subject of the benchmark. The model is stateless between turns, so it cannot rely on remembering the board — it has to externalise what it has seen into a note and then trust and maintain that note.
The optimal policy is simple to state and demanding to execute: if two known tiles match, take them; otherwise flip a tile you have never seen. Models fail here by re-flipping tiles they already know, by letting the notes drift out of step with the board, or by dropping an entry that was still needed. The score gap between a model that keeps a disciplined notepad and one that does not is very wide.
Harness and Structure
The benchmark runs Arkadium's Memory environment as a game server behind the Gamelab harness.
Each prompt contains:
- The board, showing which positions hold tiles and which are cleared
- Any tile currently face up from the previous flip
- The notes the model wrote on its last turn, handed back verbatim
- The score and the turn number
The model answers with one JSON action naming the tile to flip and the notes to carry forward. The notes field is mandatory: it is the model's entire memory, and it replaces the previous note rather than appending to it.
Illegal flips — a cleared tile, a position with no tile, the tile already face up — are rejected and recorded separately.
Evaluation
Episodes run to an 80-turn cap on seeded boards, so the same picture layout is used for every model.
Performance is read from pairs cleared, the final score, how many flips were spent re-examining already-seen tiles, and the number of rejected actions.
Notes
- Imperfect information: a face-down tile reveals nothing until it is flipped.
- Externalised memory: the notes field is the only state that survives a turn.
- Self-consistency: notes must stay aligned with a board that changes underneath them.
- Exploration versus exploitation: known pairs should be taken before new tiles are turned.
- Deterministic once dealt: the seed fixes the layout, so runs are reproducible.
- Text-only interaction: pictures are named in text rather than shown.
Contact Us
Do you want to know more about the project?