Qwen TTRPG / Local model training and serving
Training an AI to take its turn.
A player asks a question, changes course, or challenges an assumption. The next response has to meet that moment and carry the story forward. I built a local pipeline to turn conversational context into training examples, fine-tune Qwen, and test what the resulting model actually does.
Qwen3.8-27B · PyTorch · Axolotl · QLoRA · FSDP2 · llama.cpp
THE MODEL WORK
Three jobs. One shared base.
I fine-tuned task adapters for recognizing game actions, suggesting the next response, and answering questions from supplied rules. Training runs on two 24 GB GPUs with CPU offload. At inference time, the adapters share a quantized base model and are selected per request.
Review responses, preserve their context, and reserve independent sources for evaluation.
Train small LoRA weight updates while keeping the large base model frozen.
Reload the exact tensors, compare behavior, and retain failures for review.
Route requests to task adapters and measure time to first text and completion.
INSIDE A TRAINING EXAMPLE
The question belongs with the answer.
THE TRAINING DECISION
Loss belongs to the response
The model sees the full exchange. Only the assistant's response and end-of-turn token contribute to the training loss. Required context and target text are never clipped to squeeze an example into the window.
Prompt · excluded from training loss
Narrator: The ferry is still tied to the dock. Player: I ask the operator whether we can leave before the storm.
Completion + end token · included in training loss
The operator checks the gathering clouds. ‘We can leave now, if you are ready.’ She waits for your answer.
Conceptual loss mask. The real build verifies boundaries using the model's own tokenizer.
Fixed, self-authored fictional illustration. It contains no source conversation or model-generated result and makes no inference calls.
ENGINEERING DECISIONS
Follow the evidence all the way to inference.
The data pipeline is part of the model
The companion Conversational Dataset Formatter preserves speaker and source provenance, binds reviews to exact response text, and checks for shared sources and repeated targets across training and evaluation splits. Completion-only masks keep the learning objective tied to the intended response.
Training has to fit the machine
QLoRA limits trainable parameters; FSDP2 and CPU offload distribute training across the available hardware. Activation checkpointing trades extra computation for memory. The launcher checks GPU availability and records the configuration used for each run.
A saved adapter must survive the handoff
Portable export removes checkpoint-wrapper names without altering tensor values. Strict reload compares names, shapes, and loaded values. GGUF conversion checks that attention-head permutations preserve the low-rank weight update.
Shared weights still have a scheduling cost
Each inference request enables its task adapter and explicitly disables the others. Reusing one base saves model memory, but different adapter configurations can queue separately. Context length and concurrent requests still compete for GPU memory.
EVALUATION
An improvement has to survive a comparison.
Base and adapted models receive matched inputs and generation settings. The harness records completion status, time to first visible text, total latency, token usage, and explicit response checks. A separate A/B review shuffles candidate labels independently of execution order.
Lower reference loss does not establish a better response. Review considers relevance, continuity, participant agency, and unsupported facts. The original continuation is one possible answer; a good alternative may use different words.
The published tools have automated CI checks and have been exercised with the real local tokenizer and model service. The included synthetic cases test the harness; they are not a representative quality benchmark. Training material, fine-tuned weights, and private comparison reports are not distributed.
Two repositories. One path from data to deployment.
Use the formatter to prepare reviewed examples, then train, evaluate, and serve adapters with Qwen TTRPG. Both run locally with your own material and model files. Cloning the code does not download my fine-tuned weights.