Over The Shoulder / macOS
An assistant that follows your screen and conversation
OTSC helps while you work in an editor, cloud console or browser-based SSH session. It reads screenshots and listens to separate microphone and system-audio channels, then suggests code, explanations, diffs or diagrams. You continue to operate the tools; the assistant does not need access to the remote machine’s shell or files.
Python / AppKit · Screen and audio processing
How OTSC handles a changing retry policy
A colleague adds a requirement while you are editing a webhook worker. The assistant needs to incorporate it without losing the earlier safety constraint. Step through the conversation and resulting code proposal.
Screen and conversation
Screen: a webhook worker uses `status >= 500` to decide whether to retry. You: Limit this to three retries. Start with a one-second delay. Colleague: A timeout does not prove the receiver rejected the request. We cannot replay arbitrary POSTs. You: This worker already supplies safe_to_retry after checking the endpoint’s replay policy. Use that flag.
Handle server errors without replaying unsafe requests
The proposal keeps replay safety separate from HTTP status. It accepts 500–599, stops after three retries and returns the delay to the worker. It does not send requests or sleep inside the helper.
Expected behavior
- 503, retry 1, safe → wait 1 second
- 503, retry 4, safe → stop
- 503, retry 1, unsafe → stop
- 600, retry 1, safe → stop
def retry_delay(status, attempt, *, safe_to_retry):
# The caller must establish that repeating the request is safe.
if not safe_to_retry or not 1 <= attempt <= 3:
# None means stop, rather than retry immediately.
return None
# Retry server errors; a 429 policy has not been agreed yet.
if not 500 <= status < 600:
# Leave successful responses and other errors to the caller.
return None
# Space the three permitted retries 1, 2, then 4 seconds apart.
return 2 ** (attempt - 1)Python policy helper. The worker supplies parsed header values and owns cancellation, queue persistence and the total time budget. This is not a complete HTTP retry client.
Comments explain the decisions. The desktop app also provides a clean copy without teaching comments.
Authored illustration, informed by the app’s task-continuity tests. The code is checked locally; these are not recorded model responses. This page captures no screen or audio and makes no model calls.
Capture and answers run independently
Screen reading and audio transcription continue while an answer is being generated. A context worker records new constraints, resolves corrections and keeps track of visible code fragments. The answer uses a fixed snapshot of that context.
When the task changes, a planning pass can change the output sections: code and an explanation for one task, a checklist or diagram for another. Repeated observations that add no useful information should keep the existing answer. You can hold an older artifact on screen while capture continues.
Code read from a screenshot
The app keeps the excerpt, its source observation and known line positions. A spoken description cannot establish file contents. If you explicitly connect a local project, verified file snapshots provide a stronger basis for diffs.
A proposed edit
The host calculates the diff against the observed or selected source. It shows what would be added or removed without applying the edit. Missing optional file metadata should not discard an otherwise useful answer.
Questions from another person
A question about replay safety belongs in the task, even if it comes from system audio. Channel labels help distinguish participants, but are not proof of speaker identity.
A source-mapping failure found by evaluation
In the native workflow study, OCR read a function correctly but included the editor’s line-number gutter. The context worker returned the code without those numbers. A literal-source check rejected the mismatch, so the user received a code proposal without its companion diff.
The fix records exact source regions and derives code and line positions from the gutter. A fresh replay recovered the observed file and diff for both failing screenshots. Those were two synthetic screens; they do not establish general OCR accuracy.
The same workflow study exercised changing requirements, another participant’s question, duplicate suppression and a delayed answer arriving after a task reset. It also distinguished an answer being generated from that answer becoming visible in the currently selected pane.
Desktop controls
The output pane can stay visible with a nearly transparent background and opaque text. Click-through, history controls and an optional MIDI keypad let you move through suggestions without taking over normal keyboard shortcuts.
Controller mappings and firmware ↗