Over The Shoulder Coder / Mac desktop app
An AI collaborator for work in progress.
The code is on your screen. A requirement comes up in conversation. Someone challenges an assumption. OTSC brings those signals into one evolving task and helps create the next version of the work.
Python · AppKit · Screen and audio context · Model orchestration
FOLLOW THE TASK
New information earns a new answer.
Step through a small coding task. A new requirement changes the proposal; a repeated screen and an acknowledgment do not.
WHAT CHANGED
Screen: The same code is still visible.
Other participant: Okay, that makes sense.Current output retained
No new requirement, unanswered question, or meaningful code change. Keep version 2 visible instead of generating another answer or history entry.
# Average the available readings, including real zeros.
def average(values):
# Exclude missing readings without dropping zero.
samples = [v for v in values if v is not None]
# Check whether any usable readings remain.
if not samples:
# Keep the agreed empty-input result.
return 0
# Divide by the number of available readings.
return sum(samples) / len(samples)Teaching comments explain the proposal; the desktop app also offers clean copy.
Interactive illustration with fixed, self-authored inputs and responses. This page does not run the desktop app, capture your screen or audio, or call a model.
THE SYSTEM
Keep context moving. Keep output under control.
Screen readings + separate microphone and system-audio transcripts
Source-linked notes, constraints, and observed file fragments
Decide whether to answer and whether the output plan needs to change
Quick guidance, then deeper code, diffs, explanations, or images
Screen reading, transcription, and context building run independently. Each answer uses a fixed snapshot, so late work cannot overwrite a different task. Browsing an older artifact freezes that view without freezing the rest of the system.
ENGINEERING DECISIONS
Useful assistance has to respect what it knows.
An observed fragment stays a fragment
Visible code, spoken descriptions, inferred notes, and verified files are different evidence. The app preserves their sources and completeness instead of inventing a full repository from a screenshot.
The task chooses the output
A planning pass defines the instructions and named output sections. An algorithm might need code and complexity analysis; a design task might need an image and component explanations. Meaningful task changes can revise that structure.
Proposals you can inspect
Source-backed diffs show current and proposed code with line numbers. Teaching comments stay separate from clean copy. The app proposes changes; it does not apply them to your project.
Conversation affects the work
Questions and suggestions from other people can change the artifact or warrant a reply. Microphone and system audio remain separate channels; those channel roles are context cues, not verified voice identification.
Controls that leave your shortcuts alone
A MIDI keypad handles history, output types, capture, and window placement without taking over editor shortcuts. Click-through makes the pane background transparent while keeping its text readable.
RELIABILITY WORK
A good draft is no use if it never arrives.
Recorded-input evaluation exposed a delivery defect: optional file metadata could cause an otherwise usable answer to be discarded. Separating those validation paths made all 18 stored deep drafts deliverable, up from 9, with their original code preserved.
That comparison measures delivery behavior, not whether every answer was correct. Perception, task understanding, and usefulness are evaluated separately, with source media, rejected drafts, and provisional reference answers available in the local review interface.
A later 48-trial planning experiment tested combining context building and plan review. It reduced model calls but made fresh answers slower when the background worker was busy. The independent architecture remains the default.
Built around the work already in front of you.
OTSC is an actively developed Mac desktop application. Its source is public; recorded evaluation traces remain private.