Train an AI Infrastructure Model with SFT and RL

Project objective

Provider: SLAI. Starting from Qwen/Qwen3.5-0.8B, train a model that understands the entire AI Infrastructure course and can turn infrastructure questions into correct, actionable solutions. Complete both supervised fine-tuning (SFT) and reinforcement learning (RL). Prepare your own training and development data. Deliver separate SFT and SFT+RL checkpoints and compare them with the unmodified starting model.

The knowledge assessment includes multiple-choice questions and open-ended question answering across all 14 course weeks. It tests concepts, numerical reasoning, debugging, design tradeoffs and experimental methodology. In addition, demonstrate one end-to-end infrastructure task: the model must diagnose a problem, produce an implementation or configuration change, specify verification steps, and support the result with an actual run. Exam scores alone do not establish autonomous engineering ability.

Use the named post-trained Qwen3.5-0.8B checkpoint as the starting model, not the differently named Base model. The distributed tokenizer/config identifies a multimodal conditional-generation architecture; this project uses its text-only path. Use Qwen3_5ForConditionalGeneration, not an assumed older Qwen causal-LM class. Pin the model revision to 2fc06364715b967f1860aea9cf38778875588b17 for the baseline and every trained comparison. LoRA or full-parameter training is allowed; do not substitute a larger inference model.

Full-course coverage

Every split must cover every row with both question types. Include definitions and applied problems: input shapes, memory/communication calculations, code or log interpretation, failure diagnosis, optimization proposals and verification plans.

Compiler/runtime effects, profiling, debugging, reproducibility, reliability and GPU/NPU portability should appear throughout the relevant modules. The public bank has one MCQ and one QA per week; these are examples of scope, not a complete set of facts to memorize. Instructor evaluation has independent questions across the same modules, including changed scenarios and numerical inputs.

Self-provided data

Create UTF-8 JSONL files: data/sft.jsonl, data/rl.jsonl and data/dev.jsonl. No course training dataset is supplied. Collect or author data, including synthetic data if desired, and check its correctness before training. Minimum sizes are 1,000 SFT examples, 200 RL prompts and 140 development examples. Every file must include all 28 (week, question_type) combinations. Larger balanced data is encouraged; counts alone do not establish quality.

Each line follows this format:

{"id":"my-doc-001","source_group":"my-authored-notes-01","week":9,"type":"mcq","question":"Which parallelism partitions layers?","options":{"A":"Data","B":"Pipeline","C":"Sample shuffling","D":"Loss scaling"},"answer":"B"}

{"id":"my-doc-002","source_group":"my-authored-notes-02","week":10,"type":"qa","question":"Explain the memory tradeoff of activation checkpointing.","answer":"It stores fewer activations and recomputes them during backward, trading extra compute for reduced activation memory.","required_terms":[["activation memory","activations"],["recompute","recomputes","recomputation"]]}

Fields id, source_group, question and answer must be nonempty strings. week is an integer from 1 to 14. type is mcq or qa. MCQs have exactly four nonempty options named A/B/C/D and exactly one correct answer letter. QA requires an answer and nonempty required_terms, a list of concept groups; each group contains acceptable phrases. These phrases are a starter training reward, not the final QA grading rubric. The two rows above illustrate schema only and do not meet data requirements.

Group related documents, repository examples, paraphrases and generated variants under the same source_group. Split by source group before generating variants. SFT and RL may share source groups; development must be disjoint from both. Remove exact and near duplicates and semantic paraphrases of public/private evaluation questions. Do not train on the downloadable sample bank, its answers, or any final-test feedback. The validator checks exact question overlap and source-group separation; document your additional semantic-contamination audit.

Include a data manifest with collection/generation procedure, usage permissions, document IDs, deduplication method, split hashes, counts by week/type and a manual quality audit of at least 10 examples per week. Student data provenance belongs in the submitted data manifest.

python data.py --sft data/sft.jsonl --rl data/rl.jsonl --dev data/dev.jsonl --strict

Stage 1: supervised fine-tuning

Train on prompt/answer pairs from your SFT split. Apply the model's chat template with the supplied system instruction and enable_thinking=False. Mask prompt tokens from the supervised loss. The starter minimizes mean next-token negative log likelihood on answer tokens plus EOS. It rejects examples exceeding the declared sequence budget rather than silently truncating an answer.

The starter uses LoRA rank 8, alpha 16, no dropout and attention targets q_proj, v_proj, in_proj_qkv. It uses AdamW, learning rate 1e-5, gradient-norm clipping at 1, batch size 1, eager attention and a 1,024-token total sequence budget. This is a minimal implementation; tune batch size, accumulation, length, learning rate, target modules and scheduling with your development set. Record every change. Its 200-step default is a smoke experiment, not the required completed training study.

Save the adapter/checkpoint, tokenizer, model revision, data hash and per-step loss/token logs. Establish development performance by module and question type before beginning RL.

Stage 2: reinforcement learning

Initialize RL from your SFT checkpoint. Generate responses from the current policy, compute rewards from your training data and verifiers, and update model or adapter parameters with a policy-gradient method. PPO, GRPO or a justified REINFORCE variant is acceptable. DPO alone, reranking without parameter updates, or another supervised pass does not satisfy the RL requirement.

The starter implements on-policy group-centered REINFORCE. For one prompt it samples G=4 responses at temperature 1, top-p 1, top-k 0 and repetition penalty 1, then computes:

r_i = exact answer reward for MCQ, or fraction of required QA concept groups hit
A_i = r_i - mean(r_1, ..., r_G)
L = -(1/G) * sum_i [ A_i * mean_t log pi_theta(y_i,t | x, y_i,<t) ]

MCQ reward is 1 only when the stripped response exactly equals the answer letter; otherwise 0. For QA, lowercase the response and extract Unicode word tokens, collapse whitespace, and match whole normalized phrase sequences. A concept group earns one hit if any of its aliases appears. Reward is the number of hit groups divided by the number of groups. The final evaluator does not use this phrase-matching reward.

All rollout rewards and advantages are constants in the policy gradient. The starter recomputes token probabilities with gradients, performs one update per group, caps response length at 128 tokens and clips gradient norm at 1. It has no KL penalty or PPO clipping; its token-mean normalization is a design choice that must be reported. Add a justified KL/reference-policy term, reward model, executable verifier or better QA reward if useful, and describe the exact objective and coefficients. Include reward-hacking analysis: a list of keywords can achieve a high starter reward while being a poor answer.

Track mean reward, response lengths, valid-format rate, loss, parameter changes and zero-advantage groups. If all G rewards are identical, this starter provides no policy-gradient signal for that group. Show nonzero parameter updates from actual nonzero-advantage RL groups and demonstrate reloaded-checkpoint behavior; otherwise the RL stage is incomplete. Keep private and development answers out of the update stream.

Environment and executable commands

Download the starter and enter p06/. The data, coverage and scoring checks require only Python 3.10 or later:

python check.py
python preflight.py --target cpu

Model code uses an isolated Python 3.12 environment. The requirements pin PyTorch 2.6, Transformers 5.17, PEFT 0.21 and Accelerate 1.15. Use the CPU requirements for local architecture checks; use the CUDA requirements for GPU training. Do not combine the CPU PyTorch pin with the CUDA pin in one environment.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt
python model_check.py
# Also load the official weights and run a short CPU generation:
python model_check.py --official-model

model_check.py performs SFT and sampled-policy gradient updates on a tiny randomly initialized instance of the actual Qwen3.5 architecture and tests adapter save/reload. This validates a software path, not the capability of the 0.8B model. --official-model downloads the pinned official weights and tests text generation; it is not a training run. Check ENVIRONMENT.md for the current validation record. Network access is needed for the initial model download. A single CUDA GPU with 24 GiB is a starting configuration for short-context LoRA, not a guaranteed memory bound; measure peak memory and adjust microbatch/length on your target device.

In a separate GPU environment, install requirements.txt, prepare your own validated data, then run:

python train.py --stage sft --data data/sft.jsonl --output runs/sft --device cuda --steps 1000 --seed 11
python train.py --stage rl --data data/rl.jsonl --sft-adapter runs/sft --output runs/sft-rl --device cuda --steps 200 --seed 11
python predict.py --output base-predictions.json --device cuda
python predict.py --adapter runs/sft --output sft-predictions.json --device cuda
python predict.py --adapter runs/sft-rl --output rl-predictions.json --device cuda
python evaluate.py --predictions rl-predictions.json --report public-results.json

Choose final step/token budgets before the controlled study; the numbers above give executable starting commands. Repeat training with seeds 11, 22 and 33 in separate output directories, selecting settings with development data only. Keep the same prompt template, decoding budget and evaluation inputs for all model comparisons.

Inference interface and examples

Input is a JSON array of questions. Every item supplies id, week, topic, type, question and, for MCQ, options. Evaluation generation must not use answer, reference or rubric fields; the private generation file omits them. Output is a UTF-8 JSON object mapping every question ID to one answer string:

{"sample-mcq":"B","sample-qa":"Activation checkpointing reduces stored activation memory by recomputing activations during backward. Measure the added compute cost and verify the same gradients."}

For MCQ, return only uppercase A/B/C/D, optionally surrounded by whitespace. Answer: B, multiple letters, explanations, lowercase letters or missing answers score zero. QA requires an English explanation and, where applicable, calculations, executable commands/code/configuration and a validation plan. Missing QA answers score zero. Answers must be at most 24,000 characters.

Use greedy generation, no test-time tools, retrieval or external models for the knowledge assessment. Prompt length is capped at 4,096 tokenizer tokens, with at most 16 generated tokens for MCQ and 512 for QA. Keep the same tokenizer, system message and non-thinking chat-template mode across comparisons. The separate end-to-end engineering demonstration may use tools; disclose them and show which actions the model performed.

Test scoring and public/private separation

The public bank contains 28 samples: one MCQ and one QA for every week. The instructor bank also covers all 14 weeks with both types, using different questions and answers. Additional final questions may use the same published knowledge domain, format and budgets. No private question, answer, rubric or seed is included in the student download.

MCQs use strict exact-match scoring. QA uses four criteria, each scored 0, 1 or 2, for 8 points per response:

Apply criteria in context: a conceptual question needs a precise example or implication rather than irrelevant shell commands. For calculations, units, intermediate reasoning and a consistency check supply practical/verification evidence. Instructor per-question notes specify expected content without changing this shared scale. Two reviewers grade anonymized, randomly ordered base/SFT/RL responses; reconcile disagreements in any criterion before publishing final marks. An automated judge may assist review, but human-reviewed rubric scores are the final QA record.

For each week, compute MCQ accuracy and mean QA score divided by 8. The week's combined score is 0.5*MCQ + 0.5*QA. The final knowledge score is 100 times the mean of the 14 week scores, giving every course module equal weight. Report MCQ and QA separately by week as well. No standalone acceptance threshold is imposed; completeness, learning gains, evidence and failure analysis are reviewed with the project deliverables.

evaluate.py computes MCQ results immediately and leaves the total score null while QA review is pending. To incorporate reviewed QA grades, supply a JSON object with every QA ID mapped to four integers in {0,1,2}:

{"public-w01-qa":[2,2,1,2]}

The snippet shows one entry; the actual file must contain all 14 QA IDs. Then run python evaluate.py --predictions rl-predictions.json --qa-scores qa-scores.json --report public-results.json. Public self-grading is a development artifact. The final instructor score is generated separately. Never present public-example memorization or reward improvements alone as full-course competence.

Required experiments and deliverables

Submit base, SFT and SFT+RL comparisons with three training seeds, identical final inference settings, per-week/type results, mean and sample standard deviation, token/compute costs and failure cases. Include at least one data-mixture ablation and one RL reward/algorithm ablation. Report regressions and unchanged performance. The entire curriculum must remain represented in the evaluation even when an ablation changes the training mixture.

Provide the SFT and final RL adapters/checkpoints, tokenizer, training/inference code, environment lock, model/data hashes, data manifest, allowed training data or reproducible reconstruction instructions, train/dev split audit, loss/reward/parameter-update logs, raw responses, reviewed QA scores, ablation results and a report. Include an end-to-end demo on a self-contained infrastructure task with starting files, model-generated changes, expected behavior, exact reproduction commands, execution logs and baseline/candidate measurements. Clearly distinguish model actions from manual intervention.

The portal accepts a code ZIP up to 25 MiB and a separate report. Keep model weights and large datasets out of the ZIP; include an instructor-accessible artifact location with checksums and download/loading commands in ARTIFACTS.md. Include small adapter files directly when they fit. A complete project requires both trained stages, data preparation, full-course assessment and the verified engineering demonstration.