Full-stack Inference Optimization Harness for Coding-Agent Workloads

Project objective

Build an automated optimization harness for a serving-engine baseline, a fixed model, and target hardware. The harness must diagnose bottlenecks, propose and apply optimizations, execute controlled experiments, analyze results, and detect regressions. Improve serving cost-effectiveness without violating the specified service-quality constraints.

The final system must replay a coding-agent workload with long inputs and a KV-cache hit rate above 90%. Evaluate disaggregated prefill and decode as the primary deployment, and compare it with a colocated baseline under the same model and hardware budget. Isolate the contribution of major optimizations and examine interactions across system layers.

The runnable CPU exercise below establishes the replay and experiment interface. Completing that exercise alone does not complete the project: the final deliverable requires a working serving system, a verifiable demo, and measured performance.

Workload and optimization interface

Implement the starter interface in submission.py:

def optimize(baseline: dict, evaluate):
    # evaluate(config) returns metrics; at most four calls in the CPU exercise.
    return best_config, experiment_log

Return every evaluation result, unchanged and in call order, in experiment_log. The selected configuration must have been evaluated. The driver independently replays it. Use the unmodified workload.py and run.py to check this exercise; extend the interface as needed for the real serving harness, documenting all changes.

The CPU exercise is a deterministic queueing model. Colocated execution has one first-come-first-served queue; disaggregated execution has separate prefill and decode queues. These modes have different simulated resource models, so their speed ratio is not a claim about performance at equal GPU cost. There is no continuous batching in this simulator.

Each load level generates 128 requests with the following fields:

{"id": 8, "arrival_ms": 16000.0, "session": 0,
 "input_tokens": 4096, "cached_tokens": 3968, "output_tokens": 128}

For request i, session=i%8. Requests 0–7 warm the eight sessions: arrival time is i*2000 ms, with zero cached tokens. For requests 8–127, arrival time is 16000+(i-8)*1000/offered_qps ms, and 3968 tokens are cached. The measurement-phase hit rate is 3968/4096=96.875%. Disabling the cache makes the effective cached count zero. Each request produces exactly 128 tokens.

All inputs are generated by workload.generate; the CPU exercise requires no model weights, dataset, or API key. A real service must measure actual cached tokens rather than infer hits from repeated text.

Semantics and example

Let a be arrival time, p=(input_tokens-effective_cached_tokens)/4096*1000, and d=1000/2048. All times below are milliseconds. Initial queue availability is zero; equal arrival times are processed by increasing request ID.

colocated:
    first = max(a, busy) + p + d
    finish = first + 127*d
    busy = finish

disaggregated:
    prefill_end = max(a, prefill_end) + p
    first = max(prefill_end + 8, decode_end) + d
    finish = first + 127*d
    decode_end = finish

TTFT = first - a
TPOT = (finish - first) / 127

For an isolated request arriving at zero with 4096 input tokens, 3968 cached tokens, and 128 output tokens:

This example explains the equations. The first generated request has a cold cache. Additional transfer latency can increase isolated-request latency while separate queues reduce blocking under load.

Correctness and evaluation

For the exercise, scan offered QPS {1,2,4,8,16,32}. Exclude requests 0–7 from statistics but preserve their queue effects. Compute p99 using nearest rank: sort 120 values and take entry ceil(0.99*120), counting from one.

The default experiment SLOs are p99 TTFT ≤ 2000 ms and p99 TPOT ≤ 50 ms. Effective QPS is 120 / ((last_completion - arrival_of_request_8)/1000). The objective is the largest effective QPS among passing load levels; return zero if none pass. These are default task settings, not measured service characteristics.

Include baseline, disaggregated execution, and a cache-disabled ablation; the fourth evaluation can verify a result. Explain the diagnosis, selection rule, and regression rule. The starter evaluates only the baseline and therefore does not satisfy the optimization-loop requirement.

The real-service evaluation must fix model, precision, total hardware, arrival sequence, prewarm procedure, and output length across comparisons. Apply the same SLOs or declare and justify another fixed SLO before running experiments. Verify output validity and completeness; count failed and timed-out requests. Use at least three independent runs, report all load levels, and provide at least two optimization ablations plus one interaction experiment.

Run the starter

Download and extract the starter, enter p01/, and use Python 3.10 or newer. Only the standard library is required.

python run.py --submission submission.py --output baseline.json
# Implement diagnosis, search, validation, and regression checks.
python run.py --submission submission.py --output results.json

Exit code zero indicates that the exercise interface and log checks passed. The JSON contains kind=simulation_only, baseline, candidate, and experiments. Invalid configurations, fabricated logs, and an exceeded experiment budget fail the run.

Build the serving system

An unverified deployment starting point is Linux, Python 3.12, two identical CUDA GPUs with at least 24 GiB each, Qwen/Qwen2.5-Coder-0.5B-Instruct, and vLLM. The supplied materials do not contain this serving engine or a tested disaggregation connector. The following installation is a starting recipe, not a verified environment; validate it on the target GPUs and freeze the resolved dependencies:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install vllm
python -m pip freeze > requirements.lock.txt
vllm serve Qwen/Qwen2.5-Coder-0.5B-Instruct --tensor-parallel-size 2 --enable-prefix-caching --max-model-len 8192

Record the downloaded model revision and file hashes. This command starts a colocated baseline. Implement the disaggregated deployment using the installed engine's KV-transfer interface and separate prefill/decode workers. Its connector, transport, orchestration, and failure handling are part of your implementation. A simulator mode switch does not implement this deployment.

Construct eight sessions with distinct code prefixes. Tokenize each prefix once and retain exactly 3968 token IDs; append 128 varying input token IDs for each request. Generate exactly 128 output tokens at temperature zero. Record token arrival times from streaming responses and actual cache counters. Preserve the token sequence and EOS handling across comparisons.

The harness must save a machine-readable experiment record containing configuration, code/version identifiers, workload seed, hardware, output validation, cache hits, QPS, TTFT, TPOT, failures, and runtime cost. A candidate is eligible only if correctness and SLO checks pass. Re-run the selected candidate and reject it when the repeated result violates these checks. Include unsuccessful candidates in the log.

Public and instructor evaluation

The download contains a public setup exercise. Instructor evaluation changes input instances while retaining the stated interface, constraints, equations and experiment budget. Full research results also require the actual system and experimental evidence; a private exercise pass alone does not complete the project.

Private replay uses 96–192 requests, the same eight cold warmup sessions, input lengths in {4096,6144,8192}, 128 uncached measurement tokens, output lengths in {64,128,256}, and offered rates within 1–32 QPS. Warmup timing, fixed hardware rates, SLOs, nearest-rank percentile and four-call budget are unchanged. For variable output lengths, replace 127 in the example equations with output_tokens-1. Select configurations from callback feedback rather than memorized public scores.

Run python preflight.py --target cpu to execute the packaged baseline and save an environment report. The CPU exercise is verified locally; the full target system is not. See ENVIRONMENT.md and environment verification.

Required submission

Submit a technical proposal describing the architecture, optimization loop, rationale, experiment plan, and risks; a code ZIP containing the complete implementation, configurations, locked environment, raw logs, and launch commands; and a separate final report.

Provide an end-to-end demo that replays the workload, applies at least one optimization, checks service correctness and SLO compliance, and reports before/after results. The report must include baseline and optimized QPS, TTFT/TPOT, GPU-seconds/request or an explicitly defined cost metric, ablations, cross-layer interactions, reproducibility instructions, limitations, and failed approaches. Label simulated results separately from device measurements.

Evaluation considers functional correctness, completeness and automation of the loop, improvement under SLOs, cross-layer analysis, reproducibility, and the clarity of the technical justification.