System One: Hill-Climbing Multimodal Embeddings from 77% Zero-Shot to 99% Calibrated Classification in 30 μs
Most production perception and routing decisions do not need an autoregressive LLM generating tokens. Pair a single frozen multimodal embedding model with tiny calibrated linear readouts, statistical confidence gates, and leave-one-group-out novelty floors—and you get deterministic classifiers that fit in seconds, ship as 241 KB binary files, reject 100% of out-of-domain inputs, and execute in 30 microseconds on CPU.
.system1 artifact1. Fast System 1 Intuition, Not System 2 Reasoning
When you ask a vision-language model a routine classification question—what fruit is on this conveyor?, is this support ticket urgent?, does this snippet contain personal data?—you pay for hundreds of autoregressive decoding steps, non-deterministic output formatting, and uncalibrated verbal confidence.
systemone takes the opposite stance: freeze a strong multimodal encoder
(google/embeddinggemma-2, or EG2, producing 768-dimensional L2-normalized vectors for text and images)
and treat every downstream decision as a pure, calibrated readout over that single vector.
- Embed once, decide many times. One forward pass yields a 768-d vector. Running ten different calibrated classifiers on that vector costs a few hundred microseconds total—zero extra model calls.
- Abstention is a first-class contract. Every prediction returns a calibrated probability
p, an explicitansweredboolean, and anunfamiliarflag. Below the confidence threshold or novelty floor, the readout refuses to guess and hands control back to the caller. - Statistical proof over vibes. A classifier clears its gate only when cross-fitted and held-out accuracy meet a Wilson lower bound (default: 95% accuracy at 95% confidence) with expected calibration error (ECE) near its sampling noise floor.
- Portable binary artifacts. A compiled
.system1file packs a short JSON header (labels, temperature, threshold, novelty floor, embedder canary hash) followed by contiguous float32 matrices. No PyTorch, no ONNX runtime—just a matrix-vector dot product in NumPy, Rust, or 50 lines of JavaScript.
.system1 readout adds ~30 μs of float32 arithmetic and produces a calibrated decision with an explicit abstention contract.2. Rung 0: Zero-Shot EG2 on Fruits 360 (77.4% Top-1)
To stress-test systemone on real visual inputs—not toy text prompts—we built a four-split
evaluation harness over the Fruits 360 turntable dataset (100×100 RGB images):
- Train-25 (1,000 images): 25 canonical fruit classes, 40 evenly-spaced turntable frames per class.
- Test-25 (1,000 images): held-out test split for the exact same 25 classes (40 frames/class).
- Unseen-20 (400 images): completely unseen cultivars from those species (e.g., Granny Smith apples, Meyer lemons, different pear/cherry/plum varieties) to measure real domain shift.
- OOD-5 (100 images): five non-fruit vegetables (white cabbage, corn, ginger, white onion, red potato) that the classifier must refuse to label as fruit.
Starting purely zero-shot—embedding class name prompts with EG2 and taking the nearest text prototype by cosine similarity—we tested 18 combinations of prompt templates, EG2 task prefixes, and text-centroid centering on the exact same 1,000 Test-25 image embeddings:
| Zero-Shot Configuration | 25c Top-1 | 25c Top-3 | 15c Core Top-1 | Median Cos Margin |
|---|---|---|---|---|
"a close-up photo of a(n) fresh {c}" (best) |
77.4% | 89.3% | 84.7% | 0.0385 |
"a photo of a(n) {c}, a type of fruit" + Classification prefix |
77.1% | 88.7% | 84.0% | 0.0213 |
| 5-template ensemble + text-centroid centering | 76.8% | 89.3% | 85.2% | 0.0944 (2.6×) |
Bare class name "{c}" + text-centroid centering |
76.6% | 89.6% | 85.7% | 0.0801 |
Bare class name "{c}" (raw) |
76.0% | 89.3% | 85.3% | 0.0338 |
Pluralized "{plural}" (e.g. "apples", "grapes") |
72.2% | 87.2% | 85.5% | 0.0314 |
| Non-zero-shot ceiling: 5-fold nearest image centroid on same vectors | 99.7% | — | — | — |
Three empirical findings jumped out immediately:
- Prompt engineering barely moves accuracy. All seven singular templates landed inside a
2.3 ppband (75.1%–77.4%). Bare class names already hit76.0%, and EG2 task prefixes stayed within0.8 ppof rawprompt=None. Only pluralizing hurt (-5.2 pp), because every image shows a single specimen. - Centering text prototypes expands margins 2.6×. Subtracting the mean text vector (
normalize(T - mean(T))) leaves Top-1 flat while stretching the median Top-1 vs Top-2 cosine gap from0.034to0.095—giving downstream calibration much cleaner separation. - The representations were already near-perfect. A 5-fold nearest-image-centroid probe on the exact same 768-d vectors hit 99.7%. Ten of the 25 classes already scored 100% zero-shot (apple, blackberry, cherry, lemon, orange, pear, pineapple, raspberry, strawberry, watermelon).
So where did the 22.3-point gap come from? Inspecting the actual 100×100 crops exposed four dataset artifacts where cropping removes scale, flesh, or bunch context:
| True Class | Zero-Shot Confusion | Errors (of 40) | Visual Cause (Verified on Crops) |
|---|---|---|---|
| Grapefruit | → orange | 40 / 40 (0% Top-1, 100% Top-3) | Whole uncut orange-skinned citrus sphere; no pink flesh or size reference. |
| Grape | → blueberry / avocado / plum | 40 / 40 (0% Top-1) | Single detached dark spherical berry scaled to 100×100, not a bunch. |
| Papaya | → kiwi / pear / avocado | 40 / 40 (0% Top-1) | Small unripe greenish-brown uncut oval. |
| Mango | → avocado / kiwi | 26 / 40 (32.5% Top-1) | All-green unripe oval silhouette closely resembling an avocado. |
Excluding just those four scale-stripped classes, zero-shot EG2 already averaged 90.6% Top-1 and 99.0% Top-3 across the remaining 21 fruits. The visual signal was sitting right there in the frozen 768-d vectors; text descriptions alone simply couldn't describe "a single detached blueberry-sized dark grape on a studio turntable."
3. Climbing the Ladder: Pair Probes, the Modality-Gap Trap, and Few-Shot Scaling
Rather than fine-tuning the vision backbone, we climbed the systemone readout ladder step by step, measuring both in-domain Test-25 and cultivar-shifted Unseen-20 at every rung.
Rung 1a: Targeted Pair Probes (77.4% → 84.8%)
First, we fitted five lightweight binary logistic probes solely on the top zero-shot confusion pairs. Standalone on held-out Test-25,
grapefruit vs orange, grape vs blueberry, and mango vs avocado each hit 100.0%
(Wilson LB95 0.954); pomegranate vs apple hit 91.25%. Folding them in as tie-breakers over the top zero-shot candidates lifted overall Top-1 by +7.4 pp to 84.8% while keeping Unseen-20 intact at 59.3%.
Rung 1b: The Modality-Gap Trap (77.4% → 16.0%)
Why did teaching four classes break the other twenty-one? The modality gap. In joint image-text embedding spaces, image-to-image cosine similarities cluster tightly around 0.95–0.98, whereas image-to-text cosine similarities sit around 0.65–0.75.
As soon as four classes hold visual centroids and twenty-one hold text centroids, the four visual prototypes act like gravitational wells—every test photo scores higher against an image centroid of the wrong fruit than against the text centroid of the right fruit.
Lesson: never mix raw visual and text prototypes in uncalibrated cosine space; either ground all classes in the same modality or use a joint discriminative readout with a calibrated text prior.
Rung 2: Few-Shot Scaling (1 to 40 Images per Class)
Once all 25 classes receive even a single labelled image, accuracy rockets upward. We swept k ∈ {1, 2, 3, 5, 10, 20, 40}
evenly-spaced turntable frames per class across centroid, visual-only logistic, and text-prior regularized logistic readouts (3 random seeds per k):
k). Just 2 images/class clears 95% Top-1 (100% Top-3); 10 images/class hits 99.1%. Notice how anchoring the readout with a text prior protects +6–10 pp on completely unseen cultivars (Unseen-20) compared to visual-only fits.Here is the complete hill-climbing progression from zero-shot to the compiled binary artifacts:
| Rung / Method | Train Images | Test-25 Top-1 | Test-25 Top-3 | Unseen-20 Top-1 | OOD Rej |
|---|---|---|---|---|---|
Rung 0: Zero-Shot EG2 (T5_closeup) |
0 | 77.4% | 89.3% | 59.0% | — |
| Rung 1a: Zero-Shot + 5 Pair Probes | ~400 (5 pairs) | 84.8% | 89.3% | 59.3% | — |
| Rung 1b: Surgical Teach Worst-4 (Modality Gap) | 20 (4 classes) | 16.0% | 16.0% | 12.0% | — |
| Rung 2 (k=1): Logistic + Text Prior | 25 (1/class) | 89.0% | 99.0% | 57.3% | — |
| Rung 2 (k=2): Logistic + Text Prior | 50 (2/class) | 95.0% | 100.0% | 57.1% | — |
| Rung 2 (k=10): Logistic (Visual) | 250 (10/class) | 99.1% | 100.0% | 50.8% | — |
Rung 4: Compiled .system1 Logistic + Prior (C=64) |
1,000 (40/class) | 99.0% | 100.0% | 54.5% | 100% |
Rung 4: Compiled .system1 Hybrid (β=0.35 / β=0.80) |
1,000 + text | 99.0% / 87.7% | 100.0% | 56.5% / 61.3% | 100% |
At k = 40, 23 of 25 classes hit a clean 100.0% Top-1, and the remaining 10 errors out of 1,000 images
are isolated to two tricky angles (3 foreshortened end-on bananas, 7 dark pomegranates). Top-3 accuracy is a flat 100.0%.
4. Calibration, Wilson Gates, and Stopping OOD Hallucinations
Accuracy alone is a trap. In real software, the expensive failures happen when a classifier encounters something outside its training
world and outputs 99.9% confidence anyway.
systemone addresses this with two orthogonal mechanisms stored directly inside the .system1 binary:
-
Cross-fitted temperature scaling: We fit a single scalar temperature
Tby minimizing negative log-likelihood on out-of-fold predictions. On Test-25, raw uncalibrated cosine softmax (T = 20) had an ECE of0.603and answered0%of rows at0.90. After temperature calibration (T = 2.56on the compiled logistic head), ECE drops to0.0072(essentially at the0.0052finite-sample noise floor) with Brier score0.0100. -
Leave-one-group-out novelty gating: Every compiled artifact embeds the 25 normalized class centroids (
NU) and a scalarnovelty_floorset to the 5th percentile of out-of-group training cosine similarity (0.948). At inference time, if an input's maximum cosine to every class centroid falls below the floor, the classifier marks itunfamiliar = trueand abstains—regardless of how sharp the softmax logit looks.
p = 1.000 for papaya). The novelty floor rejects 100/100 OOD items while holding Test-25 answered accuracy at 100.0%.Look at how starkly the novelty gate changes real-world behavior on our 100 OOD vegetables:
- Without novelty gate:
50%of OOD vegetables cleared thep ≥ 0.90threshold—Potato Red 1scoredp = 1.000for papaya andGinger 1scored0.982for papaya. - With novelty gate (
floor = 0.948): 100 out of 100 OOD vegetables were flaggedunfamiliar(0.0%false answers). On held-out Test-25, the classifier answered 94.7% of images (947 / 1000) at 100.0% answered accuracy (947 / 947correct, Wilson 95% lower bound0.9959). - Honest domain-shift detection: On the 400 unseen cultivars (Granny Smith, Meyer Lemon, etc.), the studio-trained logistic head saw its median cosine drop from
0.978to0.937and automatically flagged66.3%as unfamiliar rather than silently over-trusting a narrow studio boundary.
5. Sub-30 μs CPU Readouts & Edge Deployment on Cloudflare Workers AI
Because a compiled .system1 file is just an 8-byte header length, a JSON manifest, and three contiguous float32 buffers
(W of shape 25×768, b of shape 25, and novelty centroids NU of shape 25×768),
the entire 25-class model weighs 241 KB and needs zero ML dependencies at runtime.
| Pipeline Stage | Median Latency | Throughput | Share of Total |
|---|---|---|---|
| Image read + base64 encode (100×100 JPEG) | 0.032 ms | — | 0.04% |
| Frozen EG2 embed (single image, local GPU, keep-alive) | 72.3 ms | 13.8 img/s | 99.92% |
Frozen EG2 embed (batched, bs = 16) |
66.3 ms / img | 15.1 img/s | — |
.system1 pure CPU readout (single image) |
29.8 μs (0.030 ms) | ~33,500 img/s | 0.04% |
.system1 batch CPU readout (1,000 images) |
1.88 ms (1.88 μs / img) | ~531,000 img/s | — |
A Real-World Edge Surprise: Chat-Template Shift on Workers AI
To make this demo publicly interactive with zero home-server dependency, we deployed it natively on
Cloudflare Workers AI using @cf/google/embeddinggemma-2 (~290–420 ms, ~0.04 neurons/image)
and loaded the binary .system1 files from R2 into a 50-line JavaScript parser inside the edge Worker.
During live verification, systemone's embedder canary caught a subtle cross-provider mismatch immediately:
Cloudflare Workers AI wraps multimodal inputs inside Gemma's chat template, shifting image embeddings by roughly 0.85
cosine relative to raw non-templated server vectors. Moving a probe trained on raw vectors onto Workers AI without refitting would have
silently tripped the novelty floor.
Because fitting a .system1 probe takes seconds, we refit and calibrated all three readouts directly in Workers AI's EG2 space:
| Workers AI Artifact | Size | Readout | Test-25 Top-1 / Top-3 | Gate@0.90 Acc | ECE | Unseen-20 | OOD Rej |
|---|---|---|---|---|---|---|---|
| Hybrid (Default) | 241 KB | 28.0 μs | 97.6% / 100% | 90.7% @ 98.79% | 0.0110 | 61.3% | 100% |
| Logistic (Studio) | 241 KB | 27.8 μs | 98.6% / 100% | 91.8% @ 99.56% | 0.0062 | 60.0% | 100% |
| Open-Domain Zero-Shot | 164 KB | 44.8 μs | 71.9% / 87.8% | 38.6% @ 91.45% | 0.0630 | 55.8% | 97% |
Even better: because all three .system1 artifacts operate on the exact same 768-d embedding vector,
the edge Worker runs all three readouts on every single request in under 100 μs combined—letting
you flip between Studio Logistic, Hybrid, and Open-Domain Zero-Shot in the browser with zero extra API calls.
6. Minimal API
In Python, fitting a calibrated probe with an explicit abstention contract takes a dozen lines and depends only on NumPy:
from system1 import embed, probe, Task
about_food = probe("Is this about food or eating?", task=Task.SEARCH_QUERY, examples={
"yes": ["order a pizza", "what's for dinner", "recipe for banana bread", ...],
"no": ["set a timer for 10 minutes", "the server is down", "play jazz", ...],
})
vec = embed("grab a sandwich on the way home", Task.SEARCH_QUERY)
res = about_food(vec, threshold=0.90)
if res.answered:
handle(res.pick, confidence=res.p)
else:
escalate(res.ranked, unfamiliar=res.unfamiliar)
Try the Live Edge Classifier
Upload any photo from your phone or desktop—or test the zero-shot traps (grapefruit, single grape, green mango), unseen cultivars (Granny Smith, Meyer lemon), and OOD vegetables (white onion, red potato)—with live threshold and novelty sliders.