Over The Shoulder / macOS
An assistant that follows your screen and conversation
OTSC helps while you work in an editor, cloud console or browser-based SSH session. It reads screenshots and listens to separate microphone and system-audio channels, then suggests code, explanations, diffs or diagrams. You continue to operate the tools; the assistant does not need access to the remote machine’s shell or files.
Python / AppKit · Screen and audio processing
How OTSC handles a changing retry policy
A colleague adds a requirement while you are editing a webhook worker. The assistant needs to incorporate it without losing the earlier safety constraint. Step through the conversation and resulting code proposal.
Screen and conversation
Colleague: The receiver also returns 429 with Retry-After. Do not retry earlier than it asks. You: The worker parses that header into seconds. Honor it on any retryable response. If it asks for more than ten seconds, stop and leave the item queued. Keep the original replay-safety check and retry limit.
Respect the receiver’s Retry-After delay
The changed requirement adds 429 and an optional server delay. A 30-second delay must stop this attempt; clamping it to ten would violate the receiver’s instruction. The worker retains the item for a later attempt.
Expected behavior
- 429, Retry-After 7 → wait 7 seconds
- 503, Retry-After 30 → stop; keep queued
- 429, Retry-After 0 → retry immediately
- 429, unsafe → stop
def retry_delay(status, attempt, *, safe_to_retry,
retry_after=None):
# Keep the replay-safety check and the three-retry limit.
if not safe_to_retry or not 1 <= attempt <= 3:
# None means stop; zero remains a valid immediate retry.
return None
# Include rate limiting alongside server errors.
if status != 429 and not 500 <= status < 600:
# Other responses are outside this retry policy.
return None
# The caller parses Retry-After into seconds, or passes None.
if retry_after is not None:
# Honor a valid delay; stop if it exceeds our ten-second limit.
return retry_after if 0 <= retry_after <= 10 else None
# Use exponential delay only when the server gives no delay.
return 2 ** (attempt - 1)Python policy helper. The worker supplies parsed header values and owns cancellation, queue persistence and the total time budget. This is not a complete HTTP retry client.
Comments explain the decisions. The desktop app also provides a clean copy without teaching comments.
Authored illustration, informed by the app’s task-continuity tests. The code is checked locally; these are not recorded model responses. This page captures no screen or audio and makes no model calls.
Capture and answers run independently
Screen reading and audio transcription continue while an answer is being generated. A context worker records new constraints, resolves corrections and keeps track of visible code fragments. The answer uses a fixed snapshot of that context.
When the task changes, a planning pass can change the output sections: code and an explanation for one task, a checklist or diagram for another. Repeated observations that add no useful information should keep the existing answer. You can hold an older artifact on screen while capture continues.
Code read from a screenshot
The app keeps the excerpt, its source observation and known line positions. A spoken description cannot establish file contents. If you explicitly connect a local project, verified file snapshots provide a stronger basis for diffs.
A proposed edit
The host calculates the diff against the observed or selected source. It shows what would be added or removed without applying the edit. Missing optional file metadata should not discard an otherwise useful answer.
Questions from another person
A question about replay safety belongs in the task, even if it comes from system audio. Channel labels help distinguish participants, but are not proof of speaker identity.
A source-mapping failure found by evaluation
In the native workflow study, OCR read a function correctly but included the editor’s line-number gutter. The context worker returned the code without those numbers. A literal-source check rejected the mismatch, so the user received a code proposal without its companion diff.
The fix records exact source regions and derives code and line positions from the gutter. A fresh replay recovered the observed file and diff for both failing screenshots. Those were two synthetic screens; they do not establish general OCR accuracy.
The same workflow study exercised changing requirements, another participant’s question, duplicate suppression and a delayed answer arriving after a task reset. It also distinguished an answer being generated from that answer becoming visible in the currently selected pane.
Desktop controls
The output pane can stay visible with a nearly transparent background and opaque text. Click-through, history controls and an optional MIDI keypad let you move through suggestions without taking over normal keyboard shortcuts.
Controller mappings and firmware ↗