Skip to content
Claude AI Implementation

Reading warehouse box labels with a phone and Claude

November 17, 2026
TwinStack Team
Back to Case Studies
Reading warehouse box labels with a phone and Claude

TwinStack built a mobile app that finds box labels with the phone camera and has Claude read them into a CSV file. Reading costs about $0.002 per label on Claude Sonnet 5.5, measured on real crops. The working Android prototype was delivered on 20 October 2026.

At a glance

IndustryWarehousing and logistics
DeliveredWorking Android prototype, 20 October 2026
Claude productsClaude API (Messages API with structured output)
ModelsClaude Sonnet 5.5 (default), Claude Opus 5.5 (most accurate)
Tech stackFlutter (Android and iPhone from one codebase), Google ML Kit, Python test tool
Cost per labelAbout $0.002 on Sonnet 5.5

The challenge

The client needed to record the details printed on warehouse box labels: item number, barcode, description, quantity, PO number, lot or date, and case position. Typing these by hand while walking along a rack is slow and error-prone.

Proving the idea first

Before building the app, TwinStack built a small Python test tool that sent frames from a walk-through video to Claude. Claude returned structured data for each box it saw, and repeat sightings of the same box were merged. On two sample videos (24 and 17 seconds), the tool found 15 boxes for about $0.27 in total API cost.

The test also showed that moving video is a poor source, because labels are small and often blurry. That finding shaped the app: capture a sharp, close crop of each label instead of sending video.

The solution

The app splits the work so that Claude only ever sees good pictures of labels:

On the phone (free, offline) Claude (per call, online)
Watches the camera and finds label-shaped areas Reads the printed text on each label crop
Saves a label only when it is in frame, large enough, steady and sharp Returns item number, barcode digits, description, quantity, PO, lot or date, and case info
Straightens and crops the label, and skips labels already saved Says "unreadable" instead of guessing when a crop is bad
Reads QR codes and barcodes exactly (Google ML Kit) Fills extra columns the user defines, and suggests new ones

The user walks the rack, reviews the captured crops, taps Read with Claude, confirms the results and exports a CSV file.

How Claude is used

  • Batched, structured requests. Crops are sent in batches (8 per call by default) with a strict structured output format, so results always come back as clean data.
  • Strict rules against guessing. Claude is told to read exactly what is printed, never to guess or complete digits it can't see, to leave a value empty if it isn't legible, and never to copy values from one image to another.
  • Confidence and cross-checks. Each result carries a confidence level (high, medium or low). When Claude's barcode reading matches the phone's own scan, the card is marked "Matches scan", which is strong confirmation.
  • User-defined columns. Users can add a column such as "expiry date" for Claude to fill. Claude can also suggest new columns for useful facts it finds, such as weight or country of origin.
  • Cost in view. The app shows an estimate before sending, asks for confirmation above $0.50, and shows the real cost afterwards.
  • Model choice. Sonnet 5.5 is faster and cheaper, and Opus 5.5 is the most accurate.

Quality

  • 162 automated tests pass, covering label detection, the Claude request and reply handling (against a mock server, including retries and bad replies), CSV export and every main screen.
  • The app was installed and run on a real Android phone (Redmi Note 11), where it analysed about 10 to 16 frames per second.
  • All photos and data stay on the phone, apart from the crops the user chooses to send to Claude.

Results

  • Label details are captured into a clean CSV without typing.
  • Claude reads each label for about $0.002, because the phone filters out everything except sharp label crops.
  • Barcode readings can be verified against an exact on-device scan.

Next steps

The prototype still needs to be validated in a real depot, on a full rack. A production rollout would also move the Claude API key from the app to a server.

Want Claude working inside your business?

TwinStack builds Claude agents, skills and integrations that fit how your team already works, with people kept in the loop where judgement matters.

Talk to Us