System One: Hill-Climbing Multimodal Embeddings to 99% in 30 µs — darmousseh.com
November 21, 2026 · 10 min read · systemone · EmbeddingGemma 2 · Edge ML

System One: Hill-Climbing Multimodal Embeddings from 77% Zero-Shot to 99% Calibrated Classification in 30 μs

Most production perception and routing decisions do not need an autoregressive LLM generating tokens. Pair a single frozen multimodal embedding model with tiny calibrated linear readouts, statistical confidence gates, and leave-one-group-out novelty floors—and you get deterministic classifiers that fit in seconds, ship as 241 KB binary files, reject 100% of out-of-domain inputs, and execute in 30 microseconds on CPU.

77.4% → 99.0%25-class Top-1 hill-climb (100% Top-3)
100.0%Answered acc (947/947, Wilson LB95 0.9959)
29.8 μsSingle-image CPU readout (1.88 μs batched)
100%OOD vegetable rejection (0/100 false answers)
241 KBSelf-contained binary .system1 artifact

Want to test it before reading? Snap or upload a photo—or click curated traps and OOD vegetables—running live on Cloudflare Workers AI.

Launch Live Demo →

1. Fast System 1 Intuition, Not System 2 Reasoning

When you ask a vision-language model a routine classification question—what fruit is on this conveyor?, is this support ticket urgent?, does this snippet contain personal data?—you pay for hundreds of autoregressive decoding steps, non-deterministic output formatting, and uncalibrated verbal confidence.

systemone takes the opposite stance: freeze a strong multimodal encoder (google/embeddinggemma-2, or EG2, producing 768-dimensional L2-normalized vectors for text and images) and treat every downstream decision as a pure, calibrated readout over that single vector.

Input Image JPEG / PNG / Cam resized ≤ 512 px FROZEN ENCODER EmbeddingGemma 2 x ∈ R⁷⁶⁸, ‖x‖₂ = 1 72 ms local GPU · ~290 ms edge .SYSTEM1 ARTIFACT (241 KB) Linear Readout + Gate p = softmax(T · (Wx + b)) ~30 μs CPU · max cos(x, NU) ANSWERED p ≥ 0.90 and cos ≥ floor UNCERTAIN (Defer) familiar, but p < threshold UNFAMILIAR (OOD) max cos < novelty floor
Figure 1. The frozen encoder handles 99.96% of the compute; the binary .system1 readout adds ~30 μs of float32 arithmetic and produces a calibrated decision with an explicit abstention contract.

2. Rung 0: Zero-Shot EG2 on Fruits 360 (77.4% Top-1)

To stress-test systemone on real visual inputs—not toy text prompts—we built a four-split evaluation harness over the Fruits 360 turntable dataset (100×100 RGB images):

Starting purely zero-shot—embedding class name prompts with EG2 and taking the nearest text prototype by cosine similarity—we tested 18 combinations of prompt templates, EG2 task prefixes, and text-centroid centering on the exact same 1,000 Test-25 image embeddings:

Zero-Shot Configuration 25c Top-1 25c Top-3 15c Core Top-1 Median Cos Margin
"a close-up photo of a(n) fresh {c}" (best) 77.4% 89.3% 84.7% 0.0385
"a photo of a(n) {c}, a type of fruit" + Classification prefix 77.1% 88.7% 84.0% 0.0213
5-template ensemble + text-centroid centering 76.8% 89.3% 85.2% 0.0944 (2.6×)
Bare class name "{c}" + text-centroid centering 76.6% 89.6% 85.7% 0.0801
Bare class name "{c}" (raw) 76.0% 89.3% 85.3% 0.0338
Pluralized "{plural}" (e.g. "apples", "grapes") 72.2% 87.2% 85.5% 0.0314
Non-zero-shot ceiling: 5-fold nearest image centroid on same vectors 99.7% — — —

Three empirical findings jumped out immediately:

  1. Prompt engineering barely moves accuracy. All seven singular templates landed inside a 2.3 pp band (75.1%–77.4%). Bare class names already hit 76.0%, and EG2 task prefixes stayed within 0.8 pp of raw prompt=None. Only pluralizing hurt (-5.2 pp), because every image shows a single specimen.
  2. Centering text prototypes expands margins 2.6×. Subtracting the mean text vector (normalize(T - mean(T))) leaves Top-1 flat while stretching the median Top-1 vs Top-2 cosine gap from 0.034 to 0.095—giving downstream calibration much cleaner separation.
  3. The representations were already near-perfect. A 5-fold nearest-image-centroid probe on the exact same 768-d vectors hit 99.7%. Ten of the 25 classes already scored 100% zero-shot (apple, blackberry, cherry, lemon, orange, pear, pineapple, raspberry, strawberry, watermelon).

So where did the 22.3-point gap come from? Inspecting the actual 100×100 crops exposed four dataset artifacts where cropping removes scale, flesh, or bunch context:

True Class Zero-Shot Confusion Errors (of 40) Visual Cause (Verified on Crops)
Grapefruit → orange 40 / 40 (0% Top-1, 100% Top-3) Whole uncut orange-skinned citrus sphere; no pink flesh or size reference.
Grape → blueberry / avocado / plum 40 / 40 (0% Top-1) Single detached dark spherical berry scaled to 100×100, not a bunch.
Papaya → kiwi / pear / avocado 40 / 40 (0% Top-1) Small unripe greenish-brown uncut oval.
Mango → avocado / kiwi 26 / 40 (32.5% Top-1) All-green unripe oval silhouette closely resembling an avocado.

Excluding just those four scale-stripped classes, zero-shot EG2 already averaged 90.6% Top-1 and 99.0% Top-3 across the remaining 21 fruits. The visual signal was sitting right there in the frozen 768-d vectors; text descriptions alone simply couldn't describe "a single detached blueberry-sized dark grape on a studio turntable."

3. Climbing the Ladder: Pair Probes, the Modality-Gap Trap, and Few-Shot Scaling

Rather than fine-tuning the vision backbone, we climbed the systemone readout ladder step by step, measuring both in-domain Test-25 and cultivar-shifted Unseen-20 at every rung.

Rung 1a: Targeted Pair Probes (77.4% → 84.8%)

First, we fitted five lightweight binary logistic probes solely on the top zero-shot confusion pairs. Standalone on held-out Test-25, grapefruit vs orange, grape vs blueberry, and mango vs avocado each hit 100.0% (Wilson LB95 0.954); pomegranate vs apple hit 91.25%. Folding them in as tie-breakers over the top zero-shot candidates lifted overall Top-1 by +7.4 pp to 84.8% while keeping Unseen-20 intact at 59.3%.

Rung 1b: The Modality-Gap Trap (77.4% → 16.0%)

Negative Result Worth Knowing: What happens if you "surgically teach" just the 4 broken classes using 5–10 image embeddings while keeping the other 21 classes as text prototypes? Accuracy collapses from 77.4% to 16.0% (840 errors out of 1,000).

Why did teaching four classes break the other twenty-one? The modality gap. In joint image-text embedding spaces, image-to-image cosine similarities cluster tightly around 0.95–0.98, whereas image-to-text cosine similarities sit around 0.65–0.75. As soon as four classes hold visual centroids and twenty-one hold text centroids, the four visual prototypes act like gravitational wells—every test photo scores higher against an image centroid of the wrong fruit than against the text centroid of the right fruit. Lesson: never mix raw visual and text prototypes in uncalibrated cosine space; either ground all classes in the same modality or use a joint discriminative readout with a calibrated text prior.

Rung 2: Few-Shot Scaling (1 to 40 Images per Class)

Once all 25 classes receive even a single labelled image, accuracy rockets upward. We swept k ∈ {1, 2, 3, 5, 10, 20, 40} evenly-spaced turntable frames per class across centroid, visual-only logistic, and text-prior regularized logistic readouts (3 random seeds per k):

100% 90% 80% 70% 60% 50% 40% k = 0 k = 1 k = 2 k = 3 k = 5 k = 10 k = 20 k = 40 Training Images per Class (Fruits 360) Zero-shot baseline (77.4%) 89.0% 95.0% 99.1% 99.0% (vis 99.3%) 59.8% Unseen-20 (with text prior) 50.0% Unseen-20 (visual-only drops ~9 pp) Test-25: Logistic + Text Prior Test-25: Logistic Visual Unseen-20: + Text Prior Unseen-20: Visual
Figure 2. Few-shot scaling across 25 fruit classes (3 seeds per k). Just 2 images/class clears 95% Top-1 (100% Top-3); 10 images/class hits 99.1%. Notice how anchoring the readout with a text prior protects +6–10 pp on completely unseen cultivars (Unseen-20) compared to visual-only fits.

Here is the complete hill-climbing progression from zero-shot to the compiled binary artifacts:

Rung / Method Train Images Test-25 Top-1 Test-25 Top-3 Unseen-20 Top-1 OOD Rej
Rung 0: Zero-Shot EG2 (T5_closeup) 0 77.4% 89.3% 59.0% —
Rung 1a: Zero-Shot + 5 Pair Probes ~400 (5 pairs) 84.8% 89.3% 59.3% —
Rung 1b: Surgical Teach Worst-4 (Modality Gap) 20 (4 classes) 16.0% 16.0% 12.0% —
Rung 2 (k=1): Logistic + Text Prior 25 (1/class) 89.0% 99.0% 57.3% —
Rung 2 (k=2): Logistic + Text Prior 50 (2/class) 95.0% 100.0% 57.1% —
Rung 2 (k=10): Logistic (Visual) 250 (10/class) 99.1% 100.0% 50.8% —
Rung 4: Compiled .system1 Logistic + Prior (C=64) 1,000 (40/class) 99.0% 100.0% 54.5% 100%
Rung 4: Compiled .system1 Hybrid (β=0.35 / β=0.80) 1,000 + text 99.0% / 87.7% 100.0% 56.5% / 61.3% 100%

At k = 40, 23 of 25 classes hit a clean 100.0% Top-1, and the remaining 10 errors out of 1,000 images are isolated to two tricky angles (3 foreshortened end-on bananas, 7 dark pomegranates). Top-3 accuracy is a flat 100.0%.

4. Calibration, Wilson Gates, and Stopping OOD Hallucinations

Accuracy alone is a trap. In real software, the expensive failures happen when a classifier encounters something outside its training world and outputs 99.9% confidence anyway.

systemone addresses this with two orthogonal mechanisms stored directly inside the .system1 binary:

  1. Cross-fitted temperature scaling: We fit a single scalar temperature T by minimizing negative log-likelihood on out-of-fold predictions. On Test-25, raw uncalibrated cosine softmax (T = 20) had an ECE of 0.603 and answered 0% of rows at 0.90. After temperature calibration (T = 2.56 on the compiled logistic head), ECE drops to 0.0072 (essentially at the 0.0052 finite-sample noise floor) with Brier score 0.0100.
  2. Leave-one-group-out novelty gating: Every compiled artifact embeds the 25 normalized class centroids (NU) and a scalar novelty_floor set to the 5th percentile of out-of-group training cosine similarity (0.948). At inference time, if an input's maximum cosine to every class centroid falls below the floor, the classifier marks it unfamiliar = true and abstains—regardless of how sharp the softmax logit looks.
ABSTAIN · UNFAMILIAR FAMILIAR ZONE Novelty Floor = 0.948 OOD Vegetables median 0.877 · 100/100 rejected Unseen Cultivars median 0.937 · 66% flagged Held-Out Test-25 median 0.978 · 947/947 right (100%) 0.82 0.87 0.92 0.97 1.00
Figure 3. Nearest-centroid cosine separation. Softmax probabilities alone misclassified 50% of OOD vegetables with high confidence (a red potato scored p = 1.000 for papaya). The novelty floor rejects 100/100 OOD items while holding Test-25 answered accuracy at 100.0%.

Look at how starkly the novelty gate changes real-world behavior on our 100 OOD vegetables:

5. Sub-30 μs CPU Readouts & Edge Deployment on Cloudflare Workers AI

Because a compiled .system1 file is just an 8-byte header length, a JSON manifest, and three contiguous float32 buffers (W of shape 25×768, b of shape 25, and novelty centroids NU of shape 25×768), the entire 25-class model weighs 241 KB and needs zero ML dependencies at runtime.

Pipeline Stage Median Latency Throughput Share of Total
Image read + base64 encode (100×100 JPEG) 0.032 ms — 0.04%
Frozen EG2 embed (single image, local GPU, keep-alive) 72.3 ms 13.8 img/s 99.92%
Frozen EG2 embed (batched, bs = 16) 66.3 ms / img 15.1 img/s —
.system1 pure CPU readout (single image) 29.8 μs (0.030 ms) ~33,500 img/s 0.04%
.system1 batch CPU readout (1,000 images) 1.88 ms (1.88 μs / img) ~531,000 img/s —

A Real-World Edge Surprise: Chat-Template Shift on Workers AI

To make this demo publicly interactive with zero home-server dependency, we deployed it natively on Cloudflare Workers AI using @cf/google/embeddinggemma-2 (~290–420 ms, ~0.04 neurons/image) and loaded the binary .system1 files from R2 into a 50-line JavaScript parser inside the edge Worker.

During live verification, systemone's embedder canary caught a subtle cross-provider mismatch immediately: Cloudflare Workers AI wraps multimodal inputs inside Gemma's chat template, shifting image embeddings by roughly 0.85 cosine relative to raw non-templated server vectors. Moving a probe trained on raw vectors onto Workers AI without refitting would have silently tripped the novelty floor.

Because fitting a .system1 probe takes seconds, we refit and calibrated all three readouts directly in Workers AI's EG2 space:

Workers AI Artifact Size Readout Test-25 Top-1 / Top-3 Gate@0.90 Acc ECE Unseen-20 OOD Rej
Hybrid (Default) 241 KB 28.0 μs 97.6% / 100% 90.7% @ 98.79% 0.0110 61.3% 100%
Logistic (Studio) 241 KB 27.8 μs 98.6% / 100% 91.8% @ 99.56% 0.0062 60.0% 100%
Open-Domain Zero-Shot 164 KB 44.8 μs 71.9% / 87.8% 38.6% @ 91.45% 0.0630 55.8% 97%

Even better: because all three .system1 artifacts operate on the exact same 768-d embedding vector, the edge Worker runs all three readouts on every single request in under 100 μs combined—letting you flip between Studio Logistic, Hybrid, and Open-Domain Zero-Shot in the browser with zero extra API calls.

6. Minimal API

In Python, fitting a calibrated probe with an explicit abstention contract takes a dozen lines and depends only on NumPy:

from system1 import embed, probe, Task

about_food = probe("Is this about food or eating?", task=Task.SEARCH_QUERY, examples={
    "yes": ["order a pizza", "what's for dinner", "recipe for banana bread", ...],
    "no":  ["set a timer for 10 minutes", "the server is down", "play jazz", ...],
})

vec = embed("grab a sandwich on the way home", Task.SEARCH_QUERY)
res = about_food(vec, threshold=0.90)

if res.answered:
    handle(res.pick, confidence=res.p)
else:
    escalate(res.ranked, unfamiliar=res.unfamiliar)

Try the Live Edge Classifier

Upload any photo from your phone or desktop—or test the zero-shot traps (grapefruit, single grape, green mango), unseen cultivars (Granny Smith, Meyer lemon), and OOD vegetables (white onion, red potato)—with live threshold and novelty sliders.

Open Fruit Classifier Demo →
← All posts Explore all demos →