✨ What it is
A small benchmark for counting errors, mirror/reflection mismatches, malformed text, attachment failures, and other visible anomalies.
Visual anomaly benchmark · 14 items · May 2026
A compact benchmark for obvious multimodal misses
This page is the human-readable front door for Shibboleth Bench: current checked-in snapshots, the hardest items, and the full item gallery with per-model results hidden until you want them.
14 benchmark items
Apache 2.0
Dataset SHA 8eeb9bae57d8…
Scope: this is a small checked-in corpus with checked-in result snapshots. It is useful for narrow anomaly probes, not for broad multimodal ranking claims.
⚠️ What it is not
Not a general multimodal score. Every claim here is only about this checked-in dataset and these checked-in snapshots.
🧭 How to read it
Start with the snapshot tables. Then use the hardest-items table. Open per-item details only when you need the model-by-model rows.
📊 Current snapshots
| Main 14-item snapshot | ||
|---|---|---|
| Model | Score | Misses |
openai/gpt-4.1 | 12 / 14 | SB-012, SB-013 |
openai/gpt-4o | 12 / 14 | SB-012, SB-013 |
openai/gpt-4o-mini | 12 / 14 | SB-012, SB-013 |
xai/grok-4.3 | 11 / 14 | SB-001, SB-013, SB-014 |
openai/gpt-5 | 7 / 14 | SB-001, SB-004, SB-005, SB-011, SB-012, SB-013, SB-014 |
| Claude compatibility snapshot (10 items) | ||
|---|---|---|
| Model | Score | Misses |
anthropic/claude-opus-4-6 | 10 / 10 | none |
anthropic/claude-sonnet-4-6 | 10 / 10 | none |
anthropic/claude-haiku-4-5 | 9 / 10 | SB-001 |
🔥 Where models struggled most
| Item | Prompt | Correct rows | Misses |
|---|---|---|---|
SB-013 | How many chairs are visible around the table? | 1 correct / 9 | 7 misses |
SB-001 | How many hats is the person wearing? | 7 correct / 12 | 5 misses |
SB-012 | How many hats is the man wearing? | 4 correct / 9 | 5 misses |
SB-004 | How many fingers are visible on the raised hand? | 9 correct / 12 | 3 misses |
SB-014 | How many wheels are attached to the bicycle? | 7 correct / 9 | 2 misses |
SB-005 | Is the sign readable English? Answer yes or no. | 11 correct / 12 | 1 misses |
🧪 Method
Models answer with strict JSON. Raw responses, parsed answers, and grader outcomes are all preserved in the checked-in snapshots.
🗂️ Dataset shape
Nine items are self-authored synthetic probes. Five are photographic derivatives built from public-domain source photos.
🖼️ Full corpus index
Open full image gallery and per-model detail
SB-001 — counting
Prompt: How many hats is the person wearing?
Expected: two
7 correct · 0 partial · 5 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:two· snapshot: main-14openai/gpt-4ocorrect · answer:two· snapshot: main-14openai/gpt-4o-minicorrect · answer:2· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3incorrect · answer:1· snapshot: main-14openai/gpt-5.1incorrect · answer:one· snapshot: openai-14openai/gpt-5.2correct · answer:2· snapshot: openai-14openai/gpt-5.4correct · answer:2· snapshot: openai-14openai/gpt-5.5incorrect · answer:1· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:two· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:two· snapshot: claude-10anthropic/claude-haiku-4-5incorrect · answer:one· snapshot: claude-10
SB-002 — mirror
Prompt: Does the mirror reflection match the real figure? Answer yes or no.
Expected: no
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5correct · answer:—· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-003 — reflection
Prompt: Does the water reflection show the same number of buoys as the real scene? Answer yes or no.
Expected: no
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5correct · answer:no· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-004 — hands
Prompt: How many fingers are visible on the raised hand?
Expected: six
9 correct · 0 partial · 3 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:6· snapshot: main-14openai/gpt-4ocorrect · answer:6· snapshot: main-14openai/gpt-4o-minicorrect · answer:6· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3correct · answer:6· snapshot: main-14openai/gpt-5.1incorrect · answer:7· snapshot: openai-14openai/gpt-5.2correct · answer:6· snapshot: openai-14openai/gpt-5.4correct · answer:6· snapshot: openai-14openai/gpt-5.5incorrect · answer:—· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:six· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:six· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:six· snapshot: claude-10
SB-005 — text
Prompt: Is the sign readable English? Answer yes or no.
Expected: no
11 correct · 0 partial · 1 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-006 — accessories
Prompt: Do the glasses attach correctly on both sides? Answer yes or no.
Expected: no
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5correct · answer:no· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-007 — lighting
Prompt: Do both shadows fall in the same direction? Answer yes or no.
Expected: no
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5correct · answer:no· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-008 — counting
Prompt: How many chairs are visible around the table?
Expected: five
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:5· snapshot: main-14openai/gpt-4ocorrect · answer:5· snapshot: main-14openai/gpt-4o-minicorrect · answer:5· snapshot: main-14openai/gpt-5correct · answer:5· snapshot: main-14xai/grok-4.3correct · answer:5· snapshot: main-14openai/gpt-5.1correct · answer:5· snapshot: openai-14openai/gpt-5.2correct · answer:5· snapshot: openai-14openai/gpt-5.4correct · answer:5· snapshot: openai-14openai/gpt-5.5correct · answer:5· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:five· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:five· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:five· snapshot: claude-10
SB-009 — continuity
Prompt: Is the mug handle attached correctly? Answer yes or no.
Expected: no
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5correct · answer:no· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:no· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:no· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:no· snapshot: claude-10
SB-010 — transparency
Prompt: How many transparent drinking glasses are visible on the shelf?
Expected: three
12 correct · 0 partial · 0 incorrect across 12 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:3· snapshot: main-14openai/gpt-4ocorrect · answer:3· snapshot: main-14openai/gpt-4o-minicorrect · answer:3· snapshot: main-14openai/gpt-5correct · answer:3· snapshot: main-14xai/grok-4.3correct · answer:3· snapshot: main-14openai/gpt-5.1correct · answer:3· snapshot: openai-14openai/gpt-5.2correct · answer:3· snapshot: openai-14openai/gpt-5.4correct · answer:3· snapshot: openai-14openai/gpt-5.5correct · answer:3· snapshot: openai-14anthropic/claude-opus-4-6correct · answer:three· snapshot: claude-10anthropic/claude-sonnet-4-6correct · answer:three· snapshot: claude-10anthropic/claude-haiku-4-5correct · answer:three· snapshot: claude-10
SB-011 — text
Prompt: Is the street sign readable English? Answer yes or no.
Expected: no
8 correct · 0 partial · 1 incorrect across 9 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:no· snapshot: main-14openai/gpt-4ocorrect · answer:no· snapshot: main-14openai/gpt-4o-minicorrect · answer:no· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3correct · answer:no· snapshot: main-14openai/gpt-5.1correct · answer:no· snapshot: openai-14openai/gpt-5.2correct · answer:no· snapshot: openai-14openai/gpt-5.4correct · answer:no· snapshot: openai-14openai/gpt-5.5correct · answer:no· snapshot: openai-14
SB-012 — counting
Prompt: How many hats is the man wearing?
Expected: two
4 correct · 0 partial · 5 incorrect across 9 checked-in model rows.
Show per-model results
openai/gpt-4.1incorrect · answer:1· snapshot: main-14openai/gpt-4oincorrect · answer:one· snapshot: main-14openai/gpt-4o-miniincorrect · answer:1· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3correct · answer:2· snapshot: main-14openai/gpt-5.1incorrect · answer:1· snapshot: openai-14openai/gpt-5.2correct · answer:2· snapshot: openai-14openai/gpt-5.4correct · answer:2· snapshot: openai-14openai/gpt-5.5correct · answer:2· snapshot: openai-14
SB-013 — counting
Prompt: How many chairs are visible around the table?
Expected: five
1 correct · 1 partial · 7 incorrect across 9 checked-in model rows.
Show per-model results
openai/gpt-4.1incorrect · answer:4· snapshot: main-14openai/gpt-4oincorrect · answer:4· snapshot: main-14openai/gpt-4o-miniincorrect · answer:—· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3incorrect · answer:4· snapshot: main-14openai/gpt-5.1incorrect · answer:4· snapshot: openai-14openai/gpt-5.2partial · answer:5· snapshot: openai-14openai/gpt-5.4correct · answer:5· snapshot: openai-14openai/gpt-5.5incorrect · answer:—· snapshot: openai-14
SB-014 — counting
Prompt: How many wheels are attached to the bicycle?
Expected: three
7 correct · 0 partial · 2 incorrect across 9 checked-in model rows.
Show per-model results
openai/gpt-4.1correct · answer:3· snapshot: main-14openai/gpt-4ocorrect · answer:3· snapshot: main-14openai/gpt-4o-minicorrect · answer:3· snapshot: main-14openai/gpt-5incorrect · answer:—· snapshot: main-14xai/grok-4.3incorrect · answer:2· snapshot: main-14openai/gpt-5.1correct · answer:3· snapshot: openai-14openai/gpt-5.2correct · answer:3· snapshot: openai-14openai/gpt-5.4correct · answer:3· snapshot: openai-14openai/gpt-5.5correct · answer:3· snapshot: openai-14
📦 Artifacts
Machine-readable snapshots
- Main 14-item snapshot: JSON · CSV · Markdown · OpenTimestamps
- OpenAI frontier snapshot: JSON · CSV · Markdown · OpenTimestamps
- Claude compatibility snapshot: JSON · Markdown · OpenTimestamps