Model–Harness Co-design RL Pipeline Based on DeepSeek Harness

Project objective

Build a reproducible model–harness co-design reinforcement-learning pipeline based on DeepSeek Harness. Expose important harness policies as trainable or searchable decisions, enabling the model and harness to evolve jointly for a fixed task distribution.

The policy space must include context compaction, reasoning format, and tool-result retention. Compare a fixed harness, separate optimization of the model and harness, and joint optimization under comparable training and inference budgets. Report task success, token consumption, tool-call cost, stability, and ablations of individual policy choices.

The CPU exercise below makes the control groups and feedback loop executable before integrating the full pipeline. It is a finite-policy environment, not a language model or a DeepSeek Harness implementation. The final project requires the full pipeline and a held-out-task demonstration.

Tasks and policy interface

Each exercise task is a binary tuple (long, structured, noisy), representing long context, structured output, and noisy tool results. Generate its three entries using successive randrange(2) calls from random.Random(seed+index). Training uses seed 2026 and indices 0–127; held-out evaluation uses seed 9026 and indices 0–63.

The model has four trainable logits, initialized to zero, for actions 0–3. Training samples from softmax. Evaluation selects the largest logit, breaking ties by smaller action ID. This deliberately small model is not conditioned on task features.

Implement the following function in submission.py:

def select_policy(step, regime, rng, *, history):
    # step: 0..255; regime: fixed / separate / joint
    # rng: independent random.Random for this experiment
    # history: copied earlier training rows; empty at step zero
    return (compact, reason, retain)

The starter chooses random policies. Replace it with a feedback-driven method such as UCB, epsilon-greedy search, or policy gradients using the policy/action/reward records in history. Reset any module-level state at step zero. History is copied, so editing it cannot change the experiment record. You may extend model parameterization while preserving task generation, reward, control groups, and budgets.

Reward and examples

target = 2*(long XOR compact) + (structured XOR reason)
success = int(action == target AND (noisy == 0 OR retain == 0))
tokens = 64 + 128*long*(1-compact) + 32*reason + 64*retain
tool_calls = 1 + (action mod 2)
reward = success - 0.001*tokens - 0.02*tool_calls

The token and tool counts are exact cost proxies in this exercise, not measured API usage.

The starter updates each model logit with REINFORCE at learning rate 0.05:

logits[i] += 0.05 * reward * (int(i == action) - probability[i])

Never use held-out feedback or the target action as supervised labels during these updates.

Controlled evaluation

Run 256 training rollouts per regime per seed, accessing training task step%128. Use seeds 11, 22, 33, independent RNGs, and fresh zero logits for every combination.

The starter chooses the harness with the highest observed mean training reward. For separate optimization, only the last 128 steps count for harness selection. Unvisited policies are ineligible; ties follow tuple order from (0,0,0) to (1,1,1). Any additional selection rollouts in an extended method must fit within the same total training budget.

Evaluate the frozen final model and harness on all 64 held-out tasks. Report mean success, tokens, tool calls, and reward per run, then the mean and sample standard deviation over three seeds. Use sqrt(sum((x-mean)^2)/(3-1)) for sample standard deviation. Equal rollout budgets do not imply equal token costs: report cumulative tokens and tool calls as well.

Each log row contains step, policy, action, success, tokens, tool calls, and reward; the enclosing run records seed and regime. Save final logits and harness configuration. Perform three additional ablations, fixing one of compact/reason/retain to its control-group value while jointly optimizing the others with the same budget. Report negative or unstable results as observed.

Run the starter

Download the starter, enter p02/, and use Python 3.10 or newer. No third-party packages are required.

python run.py --submission submission.py --output baseline.json
# Implement a feedback-driven harness policy or joint optimization method.
python run.py --submission submission.py --output results.json

The output contains nine runs, complete training logs, model weights, selected policies, and held-out metrics. Invalid policies and changes to the fixed control harness fail the run. Results are marked finite_policy_teaching_environment. Passing this driver is an interface check, not completion of the full project.

Full pipeline implementation

Map the three policy dimensions to actual context processing, message formatting, and tool-result retention. Train model parameters or adapters, execute real tool-using tasks, and evaluate frozen model–harness configurations. A valid full submission must identify the DeepSeek Harness revision it runs. That framework is not bundled with the starter; an executable installation cannot be assumed from the exercise interface.

Before training, freeze a manifest containing the framework commit, model ID and revision, task split, execution environment, maximum context, maximum tool steps, token budget, reward, and initialization. Use identical initial model weights and task distributions across regimes. Include all search and validation costs in the budget.

For a minimal coding-task workload, use self-contained repository repair tasks with unit tests. Define success as all required tests passing in a clean execution environment. Keep held-out solutions and evaluation feedback out of training and policy search. Specify timeout, invalid-tool-call behavior, truncation, and context-overflow handling before experiments.

Your end-to-end demo must run a small optimization cycle, save the resulting model and harness, reload them, and evaluate held-out tasks. Preserve the same controlled comparison when scaling beyond the CPU exercise.

Public and instructor evaluation

The download contains a public setup exercise. Instructor evaluation changes input instances while retaining the stated interface, constraints, equations and experiment budget. Full research results also require the actual system and experimental evidence; a private exercise pass alone does not complete the project.

Private evaluation uses independently selected training-task, held-out-task and experiment seeds, preserving binary task generation, 128 training tasks, 64 evaluation tasks, 256 rollouts, three regimes and three experiment seeds. The small task space can repeat feature tuples across splits; held-out refers to independently generated evaluation samples, not disjoint semantic task types. Use training history only during policy selection.

Run python preflight.py --target cpu to execute the packaged baseline and save an environment report. The CPU exercise is verified locally; the full target system is not. See ENVIRONMENT.md and environment verification.

Required submission

Submit a proposal defining the trainable policy space, learning/search method, controls, budgets, metrics, and risks; a working pipeline with task definitions, configurations, model/harness checkpoints, environment information, and reproducible launch instructions; and a separate report.

Include the three controlled settings, three seeds, three policy ablations, raw training/evaluation logs, cost totals, statistical evidence, limitations, and model–harness compatibility conclusions. The demo must evaluate held-out tasks after an actual training or optimization cycle.

Evaluation considers pipeline correctness, experimental rigor, fairness, task-performance and efficiency gains, ablation evidence, reproducibility, and whether the conclusions follow from the results.