SnapDash / Inside the build
How a coffee mug becomes a point
Two friends can race from different kitchens. Follow a single photo through the on-device recognition models and see exactly what the other player receives.

On this page
System trace
Trace one find through a race
Party host and players’ phones
The host shares one list of item IDs. Each phone looks them up in its own catalog and starts its clock when the target cards turn over.
- Reveal the target · Party host and players’ phones
The host shares one list of item IDs. Each phone looks them up in its own catalog and starts its clock when the target cards turn over. Kitchen category · coffee mug on the shared list
- Read the photo · On-device vision
The phone checks the whole frame and crops for the target. A difficult image can trigger a closer search before the verifier returns a verdict. Local image evidence · candidate label and rank
- Credit the find · Local match store
An accepted mug advances the round once. The store records the item and elapsed time, then computes points and combo. One credited mug · elapsed time on the finder’s phone
- Update the rival · Supabase Realtime room
The finder sends cumulative progress. The other phone can update the scoreboard while the pixels stay on the finder’s device. Player ID · find count · finish time when complete
What a rival needs to know
A coffee mug is one of five targets in a SnapDash Party race. One player finds hers on the kitchen counter. Across town, her brother sees the first segment of her lane fill in. His phone needs to know she found one item. The label that convinced the classifier, the number of points on her screen and the view into her kitchen can all stay on her phone. Those boundaries determine who can settle a disagreement at the finish. SnapDash on the App Store
Architecture diagram
Where the photo stops and the score travels
A Party room supplies the shared assignment. One phone owns the pixels, the recognition evidence and its local score. The other phones receive progress.
Recognize the photo. The phone checks the frame and centered crops. An unresolved match can enter a more detailed sweep. The verdict either credits the target or gives the player a reason to try the camera again.
Report the find. An accepted item changes the local match once. The room broadcasts the player’s cumulative count and, when the list is complete, elapsed finish time. Rival scoreboards consume these small updates.
A common assignment and two local clocks
One phone acts as host. It picks five items and broadcasts the category ID, the ordered item IDs, its own player ID and a proposed start time. Each device looks up the items in its bundled catalog, resets the finds and moves into the reveal phase. Sending IDs keeps the packet small and gives both players the same assignment. It also creates a dependency between app versions. The lookup filters out unknown IDs, so differing catalogs could leave two players with different finish lines.
The reveal screen flips its cards and calls beginHunt on the same beat. That transition changes the phase to hunting and records the current time on the device. Although the host’s message carries a timestamp, this Party flow starts its clock at the local reveal. A find time therefore measures elapsed play on the finder’s handset. It includes the time spent processing the picture before credit is recorded. A slower phone or a reveal that starts late matters when comparing two close finishes.
Four views first, nine more if needed
The camera gives the verifier a local image reference and the target. For a mug, the catalog accepts several labels, including “cup,” and the target policy lists nearby objects such as a measuring cup and teapot. Those lists let the game ask a narrower question than “what is in this kitchen?”
The verifier runs two bundled EfficientNet models on four views of the scene. They inspect the full photo, a center square and two tighter center crops. For each group of views it keeps the highest target-label score and the rank of that label in the view that produced it. A tiny object deep inside a crop can score differently from the same object in the full frame. Pooling all crops into one average would erase the evidence the game needs to tell someone to step closer.
If the labels suggest an early acceptance, MobileCLIP can compare the two center crops against text representations of the target and competing objects. An unresolved center scan triggers a three-by-three grid of overlapping detail crops and the remaining semantic checks. When the optional semantic model or target embeddings are unavailable, the classifier can use the label evidence alone. This makes the common well-framed snap cheaper while retaining a more expensive path for a small mug beside the sink. It also means that an ambiguous “cup” prediction can face different scrutiny depending on which evidence is available.
Four verdicts from the same mug prompt
I passed fixed inputs into the app’s real decision functions. I chose the scores below to isolate the policy. Each row describes a synthetic case.
| Evidence supplied to the decision function | Returned verdict | What changes in the round |
|---|---|---|
| “Cup” ranks first at 0.50. The semantic crop scores the mug at 0.30 and its strongest competitor at 0.18. | Accepted, perfect | The camera can submit one credited find. |
| Identical label and frame quality. The semantic crop now scores the mug at 0.16 and a competitor at 0.35. | Uncertain | The item remains open. The game asks for a clearer view. |
| The full frame has no mug evidence. One detail crop gives it a weak fourth-place score. | Too small | The player gets a move-closer prompt. |
| No candidate in any view. The frame is dark. | Too dark | The player gets a lighting prompt. |
A camera or model exception follows a separate retry path. This separation gives the team a practical debugging tool. If a real mug repeatedly fails, saved test evidence can reveal whether the model missed it entirely, the relative comparison blocked it or the verdict selected an unhelpful hint. Measuring recognition accuracy still calls for consenting, labeled photos across devices and lighting conditions.
How 160 points becomes one find
Consider that first, perfect mug recorded 11 seconds into the round. I ran the scoring function with the mug’s difficulty of one and an empty combo. It returned 100 base points + 40 for speed + 20 for a perfect find = 160. The combo advanced to one, yielding zero combo bonus. The match store records the item ID, elapsed time, points and combo, and rejects a second record with the same item ID. The camera can attach a re-encoded local thumbnail. A thumbnail failure can leave the credited entry without an image.
The message to the room is smaller still. It contains the player ID, cumulative finds and an optional finish time. In this example its values mean “this player, one find, still hunting.” At five finds the phone sends the latest elapsed find time. Supabase Realtime relays that message, and the receiving phone replaces that rival’s displayed progress. The 160-point celebration affects the finder’s score without moving her two spaces along the race track. SnapDash’s photo and multiplayer privacy policy
Settling a race after the link drops
A cumulative count repairs some lost messages naturally. A rival who missed “two” can receive “three” and draw the right lane. A lost final message is different. Rejoining a room announces presence again, but presence carries roster details and readiness. It carries no final count. Each handset evaluates the finish using the progress it has received, and a later result from a rival may arrive after the local result has been settled.
I would give each round an identity and retain one latest progress record per player until settlement. Rejoining could fetch that record, and a sequence number would make delayed or repeated updates harmless. The acceptance test would disconnect one device before its fifth find, reconnect after the other device shows a result, and compare the final counts, elapsed times and winner on both. That tests whether the two phones can agree while the photos remain on the devices that took them.
Review notes
Corey Crooks built SnapDash. This walkthrough draws on the product and its implementation.
- Traced the Party start and progress payloads, reveal timer, model crop schedule, target policy, match store and end-of-round evaluator in the app source.
- Executed decideVerdict and clipDecision on four controlled inputs and computeFindBonuses on a first find at 11 seconds. The App Store listing and privacy policy document the public product.
The fixtures test policy and scoring. Camera accuracy, inference time across iPhones, and two-phone recovery at the finish remain unmeasured here.