GitHub - awlevin/typesafe-computer-use: Computer use for about $0.0002 a step: OCR the screen, classify the next action with TypeSafe, click. macOS.

typesafe-computer-use drives a Mac toward a goal you type in plain English, for about a fiftieth of a cent per step. It never sends a screenshot to a big model. Instead it reads the screen deterministically, asks a small classifier which action comes next, and only calls a writing model when a text field genuinely needs free text. clicker "go to techcrunch and take me to the checkout page for the cheapest tickets to their next upcoming event" --act Why Frontier-model computer use is capable and expensive: every step ships a screenshot and waits several seconds for a plan. Most steps do not need a plan. They need one choice from a short list, made quickly and cheaply, with a confidence number you can gate on. TypeSafe sells exactly that: a decision model that answers a Choice over up to 255 options with a full probability distribution and a calibrated confidence, in a few hundred milliseconds, with free output tokens. This project is a computer-use loop built around it. Measured on the same screenshot and goal, one decision each: typesafe (jev) Claude Opus 5, bare screenshot multiplier input tokens 4,882 4,785 same cost per decision $0.0002 $0.032 155x cheaper cost per decision, realistic loop with history $0.0002 $0.035 to $0.08 170x to 390x cheaper cost per 12-step task $0.003 $0.40 to $0.90 130x to 300x cheaper model latency 0.13 to 0.38 s 5.2 s 14x to 40x faster end-to-end step, with capture and OCR about 1.5 s about 5.5 s 3.7x faster The honest caveat: the big model read the event dates off the pixels and compared them unaided. The classifier needed the date parsing described below. Every piece of reasoning the frontier model does for free has to be rebuilt here as deterministic state. Install macOS 14 or newer, Python 3.12 or newer, uv. git clone https://github.com/awlevin/typesafe-computer-use cd typesafe-computer-use uv sync cp .env.example .env # fill in the keys variable required purpose TYPESAFE_API_KEY yes every decision ANTHROPIC_API_KEY no type_text, writer-proposed URLs, and the final answer CLICKER_EMAIL no enables the type_email action CLICKER_BROWSER no defaults to Google Chrome CLICKER_WRITER_MODEL no defaults to claude-haiku-4-5 CLICKER_ANSWER_MODEL no reads the last screen for the final answer; defaults to claude-sonnet-5 Grant your terminal Screen Recording and Accessibility in System Settings > Privacy & Security. Without the first, captures are wallpaper. Without the second, synthetic clicks are silently dropped, and --act refuses to start. uv run clicker "open the Playground" # dry run: one step, prints what it would do uv run clicker "open the Playground" --act # drives the machine, up to 100 steps uv run clicker "log in" --act --steps 20 --delay 3 # longer and slower uv run clicker-inspect "any goal" # 3-2-1, capture, open the annotated screen + payload Clear the terminal first. It is on screen, so its text is OCR input. Stopping a live run. Ctrl-C when the terminal has focus, or slam the mouse into the top-left corner of the screen from any app. The loop also stops itself on done or none, on confidence under --min-confidence (0.4), after two consecutive no-ops, or at --steps. The answer. When the loop stops itself, the writer reads the screen it stopped on and prints the result: the information the goal asked for, or where things stand and the next step when the screen does not hold it. A dry run that would have acted, and an aborted run, print no answer. How a step works screencapture ─► Vision OCR ─► merge lines into blocks ─► drop lines echoing the goal accessibility ─► actionable elements (role, label, frame), pruned to the display, the labelled pressable ones it pruned kept as off-screen controls │ └─► one numbered list of items, each carrying its source │ accessibility ─► focused field (role, label, placeholder, value, frame) AppleScript ─► frontmost app and pid, active tab URL clock ─► local date and time dates.py ─► "dated 2026-10-13 (in 27 days)" on any block containing a date, "near a line dated ..." on its neighbours │ ▼ one TypeSafe request, three Choices, four with off-screen controls ┌────────────────────────────────────────────────────────────┐ │ kind : click_item | use_browser | type_text | scroll… │ │ item : which item (used only for click_item) │ │ site : which website (used only for use_browser) │ │ offscreen : which hidden control (only for press_offscreen)│ └────────────────────────────────────────────────────────────┘ │ ▼ deterministic action ─► wait ─► next step Items carry where they came from: ocr for a text block, ax for a control the app declared, ax+ocr when both found the same thing. An ax item reads as button 'Share' (top-right) in the criteria, so the classifier can tell a real control from a line of text. Splitting the decision into three questions keeps screen noise out of the action choice. Every stall found while building this came from two options that meant the same thing. Confidence measures concentration, so overlapping options always read as doubt. Keep the action set mutually exclusive. OCR cost Vision is about two thirds of a step, and it charges by the amount of text rather than the number of pixels, so the only real saving is reading less of the screen. Crop. Each step reads the frontmost window with an 8 pt margin, plus the menu bar strip over the same columns, clamped to the display. Text on the desktop and in background windows is noise to the decision. Clipping the strip to the window's width is what makes the crop pay on a full-height window. The cost: the clock and the menu extras to the right of the window go unread. They stay clickable through the accessibility tree. Reuse. The capture is compared with the previous one at 1/8 scale, in 256 px tiles. Unchanged tiles keep the lines they produced last step. The changed tiles are clustered into blobs, sides and corners counting as touching, and each blob becomes a rectangle read on its own. Scattered change is the ordinary case, a clock digit plus one repaint, and one rectangle around both would span the display. Each rectangle grows until no known line straddles its edge, because a crop through a line returns the half it can see; ones that meet after growing merge, and more than four merge by closest pair down to four. Past 60% changed tiles, past 60% of the region in summed rectangle area, or on an app switch or a window move, the whole region is read instead. The timing line says how much was read, and in how many pieces: ocr 0.31s (22% of screen, 2 rects). A replay (--image) always reads the whole image and never reuses, so an offline repro matches the original run. Accessibility tree OCR cannot see an icon. The accessibility tree can, so each step also walks the frontmost process for labelled, on-screen controls. Coverage is uneven, measured on ten apps on one Mac: Finder 100% of on-screen controls labelled, Chrome 88%, Slack 85%, Notion 68%, Spotify 0 (its CEF shell exposes three window buttons and nothing else). Terminals expose the grid as one text area. So AX is a bonus source, never a replacement. Labels live in AXDescription for web and Electron, AXTitle for AppKit, and a short AXValue otherwise. A decorative image takes the label of the control around it; a list row takes it from a shallow AXStaticText. Frames lie, so the walk prunes hard: skip any subtree whose real frame misses the display (Notes reports rows 200 screens down, Chrome parks scrolled-out nodes above the viewport) skip any node under 4 pt wide or tall (Chromium clamps scrolled-out web nodes to slivers) skip AXMenu subtrees, which are thousands of zero-sized items behind a closed menu skip nameless AXGroup layout boxes, even pressable ones stop at 4000 nodes or 0.6 s and say so Walks measured here: Finder 152 controls in 0.08 s, Chrome 172 in 0.59 s. The assistive handshake attributes (AXManualAccessibility, AXEnhancedUserInterface) are unsupported on this macOS, so nothing relies on them. Off-screen controls AXPress does not need an element to be visible. Notes selects a row parked thousands of points below the display, Chromium delivers a click to a link it clamped to a 1 px sliver because the page is scrolled past it, and an auto-hidden Dock hands over all 37 of its items from 5 pt below the bottom edge. So the same walk keeps the labelled, pressable nodes it pruned, and offers them as a separate capped list rather than mixing them into the items: nothing on the capture points at them, and a mouse click would land somewhere else entirely. The list is deduplicated by role and label, drops any label the visible items already carry, and stops at 120 controls, after which those subtrees are pruned as before, so the walk costs what it always did. It is offered only when it is not empty, as a press_offscreen action plus an offscreen question, and the step log counts it next to ax=. A refusal is the end of it: there is no pixel to fall back on, so it reads as a no-op. What a walk finds depends on the app, and the node and time caps bind first on a big tree: Notes and Chrome spend all 4000 nodes on what is already on screen and report nothing hidden. Action space key does click_item press the element through the accessibility tree when the item came from it, so the press lands on the control rather than on whatever covers it; a mouse click at the center of the box otherwise, and as the fallback when the press is refused press_offscreen AXPress a labelled control the app exposes but does not show, chosen from the off-screen list; offered only when that list is not empty, and a refusal counts as a no-op since there is no pixel to fall back on use_browser go to the browser, showing the website the site answer names: none brings it forward on the page already open there, a SITES catalog key opens that URL through AppleScript open location, and other opens a URL the writer proposes type_text