To play Gin Rummy, two players begin the game. Each player gets 10 cards from a regular 52-card deck. One card is laid face-up to begin the discard pile, while the rest wind up in the stockpile. Your target is to create "melds." These can be runs (three or more consecutive cards in the same suit) or sets (three or four cards of the same rank). Deadwood are cards not fitting into a meld. You have to dispose of these by creating new melds to simply discard them. Players alternately pick a mystery card from the stockpile or the top card from the discard pile. They discard one card after drawing. Every card has a point value. Face cards are worth 10 points, number cards are worth their face value, and aces are worth one point. You may 'knock' at the end of your turn when you have 10 or fewer deadwood points. 'Knocking' means declaring that you are about to end the game. The two players then reveal their hands. Your opponent may "lay off" deadwood cards appropriate for your melds. You score the point difference if your deadwood count is less than your opponent's. If not, your opponent undercuts you and scores the difference, plus a bonus. "Going gin" is the term used if you manage to use all your cards in melds. Your opponent cannot lay off any deadwood, and you gain a 20-point bonus. The game goes on until one player accumulates 100 points. Bonuses for winning and the total number of hands won make up the final score.
Gin Rummy as an Evaluation Tool
Gin Rummy is a benchmark for imperfect-information reasoning, memory, opponent modeling, and long-horizon strategic planning. Unlike games such as Chess, where the entire state is visible, Gin Rummy requires models to make decisions while reasoning about hidden cards, inferred opponent intentions, and future scoring consequences.
Success depends on a model's ability to maintain a belief state over unseen cards, identify emerging meld opportunities, evaluate discard risk, and determine the optimal timing for knocking or going gin. Models must continuously balance offensive objectives (completing melds) against defensive considerations (preventing the opponent from completing theirs). Because information is revealed incrementally through draws and discards, the benchmark provides a compact test of memory, uncertainty tracking, risk calibration, and adaptive decision-making.
Harness and Structure
The Gin Rummy environment is hosted on Arkadium's game infrastructure and managed through a universal text-based evaluation harness.
Each player receives a private hand while sharing visibility of the discard pile, public scoring information, and gameplay history. Models interact through structured prompts that contain:
- Current hand composition
- Top discard pile card
- Historical draw and discard actions
- Round score information
- Legal action set
On every turn, the model must decide whether to draw from the stock pile or discard pile, select a card to discard, and determine whether to knock, continue play, or declare gin when appropriate.
The harness uses a prompt-parse-validation pipeline:
- Prompt Builder: Presents the complete observable game state while withholding hidden opponent cards.
- Response Parser: Accepts structured JSON actions describing draw and discard decisions. Invalid card references, malformed responses, and illegal actions are rejected.
- Game Manager: Enforces Gin Rummy rules, computes deadwood values, manages scoring, and resolves hand outcomes.
No information beyond the visible game state is provided to the model.
Evaluation
Models compete in large-scale head-to-head matches against all other benchmark participants.
In addition to win rate, the evaluation tracks several interpretable behavioral metrics:
- Deadwood efficiency at hand completion
- Knock timing quality
- Gin frequency
- Draw source selection accuracy
- Discard risk exposure
- Opponent card denial effectiveness
These metrics provide insight into how models reason under uncertainty, not merely whether they win.
Notes
- Imperfect information: Opponent hands remain hidden throughout gameplay.
- Memory and inference: Models must infer likely opponent holdings from observed actions and public information.
- Long-horizon planning: Early discards often have consequences many turns later.
- Risk calibration: Aggressive knocking may secure points quickly but can expose a model to undercuts.
- Information leakage: Every discard reveals information to the opponent and can be exploited strategically.
- No external tools: Models operate entirely from the provided prompt and their internal reasoning.
- Text-only interaction: Gameplay is conducted through textual state descriptions and structured actions.
Contact Us
Do you want to know more about the project?