Status: current · sub-project 1 (engine) implemented, validated on threadripper02 (RTX PRO 6000, sm_120) 2026-10-07 · specs for all 19 ops; the sm_120 deep run ships tuned/*.sm_120.txt · Tuning tab built 2026-10-09 · the constants tuner (sub-project 2) is specified at interface level only
The tuner fills the selection tables and the tuning constants in tiers. The user picks a tier (preview, coarse or deep) per run. Measurements that cannot change a table are skipped, and a per-cell ledger ensures that a cheaper run never overwrites a better result. The current tuner is described in batchlas_tune; the flat selection it feeds is in Flat kernel selection §6. The sm_120 tables come from the engine ("Engine: the sm_120 deep run").
| Question | Decision |
|---|---|
| Main purpose | Initial fill of every table for a machine. Incremental retune and exploration come second. |
| Speed versus rigor | The user picks a tier per run. Budgets come from a cost model printed by --plan. More GPUs shard the work. |
| Grid reduction | Only preview is sparse. Coarse and deep use the full lattice, because sparse grids misrank 4-13% of cells. |
| Precedence | Higher fidelity wins per cell. A lower tier fills gaps and replaces stale results, never current ones. |
| Op scope | All 19 routed ops, plus the tuning_params.hh constants, under the same tiers, ledger and UI. The header is generated. |
| Result store | benchmarks/results/tuning/ under Git LFS, so precedence holds across machines and clones. |
| Raw per-rep timings | Kept for deep runs only (deferred, see open risks). |
| UI | A Tuning tab in benchviz serve, driving batchlas_tune. |
| Worker | One persistent worker per GPU, with a fresh-process audit. See persistent workers. |
| # | Sub-project | Depends on | State |
|---|---|---|---|
| 1 | Engine: tiers, racing, adaptive grid, persistent worker, ledger, table generation, on the 4 existing specs | none | implemented |
| 2 | Coverage: specs for the other 15 ops; constants tuner on the engine; tuning_params.hh generated | 1 | specs implemented; constants tuner: interface below |
| 3 | benchviz Tuning tab: four views | 1 | built 2026-10-09; interface and as-built notes below |
Sub-projects 2 and 3 can proceed in parallel once the spec interface, ledger schema and progress protocol are frozen.
Every tier runs the same algorithm, so preview results are low-fidelity data of the same kind as deep results.
| Parameter | preview | coarse | deep |
|---|---|---|---|
| Starting lattice | every 2nd point of each choice.hh axis, ends kept | full lattice | full lattice |
| Bisection | index space until adjacent, then geometric to hi/lo < 1.1; also between cells when a runner-up is within 10% | geometric to hi/lo < 1.1 | geometric to hi/lo < 1.1 |
| Refinement cap (x round-0 cells, per op and dtype) | 3.0 | 1.0 | 2.0 |
| Race reps, min / max per candidate | 3 / 6 | 4 / 12 | 6 / 16 |
| Elimination confidence | 0.80 | 0.90 | 0.98, plus a reversed confirmation round |
| Fresh-process audit sample | 2% of cells | 2% | 10% |
The tie margin is 3% in every tier. The lattices nest (preview ⊂ coarse ⊂ deep). An op whose grid is not a lattice is subsampled by a stable hash of the cell key, so subsets still nest.
Each cell goes through three steps:
Across cells:
Ledger. One file per (op, device): benchmarks/results/tuning/ledger/<op>.<device>.jsonl (Git LFS), every line tagged with its dtype, appended under flock by every dtype job and process. The layout, the line kinds and the dirty-run rule are in Tuning data layout; it replaced one directory per (op, dtype, device) with one file per run on 2026-10-11.
| Record | Fields |
|---|---|
| run | dtype, run id, tier, host, device key and name, batchlas git sha (-dirty), argv, worker mode per op and dtype, date, keys, candidates |
| cell | run id, tier, key fields, round (0 = starting lattice); per candidate: family hash, status (ok, skipped, bad, error, eliminated), reason, median ms, interval, reps; the ranked tie set |
| audit | run id, cell key, persistent and fresh-process medians and winners, verdict |
Results merge by device key (sm_89, sm_120); the host is recorded.
Hashes. Kernel hash v2, which ignores comments and whitespace (kernel hashes). A spec's kernel-sources block maps each family to its files, plus a common set (the op's shared headers) that every family depends on; src/select/ is not hashed. Each spec lists its src/ops/<op>/<op>.cc, where can_run lives, as a common kernel-deps entry, so an edit to what can run stales every cell. kernel-deps adds files outside the source list (// family: <name> "path" ... or // common "path" ...). The posv coupling is explicit: posv's families list the potrf and trsm families they call.
Which record counts, table generation: Tuning data layout (the ledger file, generation and the check).
Import. batchlas_tune --import-raw turns a schema-1 raw sweep into a deep run. Its hashes are legacy:<kernels> unless the raw file's op hash equals today's, so an old sweep imports stale.
Worker. Each GPU gets one long-lived batchlas_tune_impl --worker, started by the driver under the existing CUDA_VISIBLE_DEVICES fence and per-GPU flock. Cells arrive on a pipe and come back as cell records.
The sticky carve-out. The SLM carve-out attribute is sticky per CUfunction (benchmarks/factor_bench.cc header), so an earlier larger launch can make a later one succeed that fails in a fresh process. Two defences:
Result. No carve-out mismatch was found (threadripper02, GPU 1, 2026-10-07, tune_race_gpu_tests). For potrf float and double, posv float and trsm float, at batch 512, a worker raced the larger cell then the smaller one at or just below the 48 KB hole, and a fresh child raced the smaller one. Every candidate's feasibility agreed. potrf pads requests out of the hole and its kernels carry no static shared (potrf: the 48 KB launch hole). The guard stays in the test suite. The mismatch path is covered by injected verdicts in tune_tests.
The worker decision rests on this measurement (threadripper02, sm_120, GPU 1, other GPUs idle, one measuring process at a time, 2026-10-07). The cell is potrf float uplo=L,n=16,batch=8192, run as 20 sequential batchlas_tune_impl --cell children after a warm-up child.
| Configuration | median s | p90 s |
|---|---|---|
| time, 1 arm, reps 1, warm 0 | 0.488 | 0.494 |
| probe, 1 arm, no timing | 0.488 | 0.492 |
| time, 6 float arms, reps 3, warm 0.2 | 1.802 | 1.809 |
The 0.49 s is process start, SYCL and CUDA init, static init, JIT-cache load and verification launches. It is derived as wall time minus in-child kernel time, and is a lower bound because the JIT cache was warm. Against a raced preview cell that is 37-60% of the measuring time. The old driver ran three children per cell, about 1.5 s of start-up per cell, so per-cell processes would add about 70% to the preview estimate.
Decision: one persistent worker per GPU.
Implemented in refine_all_axes (tools/tune/grid.cc), shared by the driver and tune_replay. Each cell's raced medians arrive as a RefineCell.
The rule must be two-sided: requiring rule 1 at both ends dropped the trsm flips, whose crossover sits near one end of the bracket, and failed every tier in replay.
GPU run (potrf float, preview, one GPU, persistent worker): 90 lattice cells and 90 refinement cells in 7 rounds (90, 46, 23, 12, 6, 2, 1), 219 s wall. Refinement converged well inside the cap of
Tests follow the agent guide's "guards that cannot fail" rules (Agent environment guide §8).
Holdout mode (the default) races on pass-1 reps and scores against pass-2 medians. Pass 1 and pass 2 disagree by more than 3% on about 1.5% of float cells (1.15% double). An oracle that times every cell and picks from all pass-1 reps scores that same floor, so the verdict is the excess over it.
| Tier | dtype | Bound | Measured (lattice + refined) | Excess, race / table | Verdict |
|---|---|---|---|---|---|
| preview | float | 5% | 2197 (616 + 1581) | 0.41% / 3.89% | PASS |
| preview | double | 5% | 1763 (585 + 1178) | 0.21% / 4.25% | PASS |
| coarse | float | 1% | 4427 (3480 + 947) | 0.03% / 0.04% | PASS |
| coarse | double | 1% | 4066 (3219 + 847) | 0.03% / 0.12% | PASS |
| deep | float | 0.2% | 4427 (3480 + 947) | 0.01% / 0.09% | PASS |
| deep | double | 0.2% | 4068 (3219 + 849) | -0.02% / 0.07% | PASS |
The estimated cost for trsm float on sm_120 is 0.57 GPU-h for preview, 0.86 for coarse and 0.86 for deep. The estimate is est_gpu_h = [measure_s + live_candidates × (warm_topup_s + verify_s) + cell_overhead_s per cell] / 3600, with warm_topup_s 0.2, verify_s 0.05 and cell_overhead_s 0 for a persistent worker. Per-candidate warm-up dominates; the summed rep time is 0.07 GPU-h.
Alternatives (same-sample replay, table misrank over all raw cells, float / double):
| Alternative | Result | Verdict |
|---|---|---|
| Stride 2 start, geometric bisection to 1.25 | 7.9% / 9.7% on 820 / 712 cells | rejected |
| Stride 2 start, index bisection, no margin | 6.7% / 5.8% on 1412 / 1096 cells | rejected |
| Stride 2 start, index bisection, margin 0.10 | 3.6% / 4.3% on 2818 / 2363 cells | preview |
| Stride 4 start, any refinement | 7-13% on 501-1812 cells | rejected |
| Full lattice, geometric bisection to 1.25 | 2.5% / 1.4% on 4021 / 3725 cells | rejected |
| Full lattice, geometric bisection to 1.1, coarse race | 0.11% / 0.05% on 4413 / 4094 cells | coarse, deep |
| No bisection on the full lattice | 6.4% / 6.1% on 3480 / 3219 cells | rejected |
| Shrunk trsm grid: batch stride 2 | 4.3% / 4.8% on 2920 / 2546 cells | rejected |
| Shrunk trsm grid: one batch value | 13.5% / 7.2% on 916 / 814 cells | rejected |
| Shrunk trsm grid: q stride 2 | 1.5% / 3.0% on 2964 / 2435 cells | rejected |
| Shrunk trsm grid: order stride 2 | 4.7% / 6.9% on 1911 / 1837 cells | rejected |
| Shrunk trsm grid: q, batch and order stride 2 | 7.8% / 9.7% on 824 / 716 cells | rejected |
Batch is the costliest axis to shrink, because the winner flips along it. No shrunken grid meets the 1% coarse bound. The full lattice does.
The tab reads ledgers and tables directly and drives batchlas_tune --plan, --progress-fd and --status. It keeps benchviz's server, SSE stream, gpu_guard.sh, campaign store and figure style (benchmarks/benchviz/style.py).
Built on 2026-10-09 as /tuning in benchviz serve (benchmarks/benchviz/tuning.py, tuning.html; user guide in benchmarks/benchviz/README.md "Tuning tab"). Where it departs from the interface above:
| Interface above | As built | Why |
|---|---|---|
| reads --status | reads the ledgers through scripts/sweep_to_table.py (read_ledger, best record, staleness, ledger_rows, parse_table, nearest) | the views show exactly what sweep_to_table.py would write next and need no build |
| status matrix: staleness | per kernel family: the table header's hash against hash v2 of the sources now, the keys the ledger holds stale in it per dtype, and stale_kernel_warnings; per dtype, the table's rows and behind when its header does not name the newest run | one table per (op, device) carries the hashes; a row-exact check per dtype (the --check rule, header hashes) is in the op view, where it costs one ledger_rows |
| launch: tiers | preview, coarse, deep, and custom with listed cells (--remeasure-keys); --allow-dirty on request; --device-key when no GPU is visible | the tuner's flags since the tuning data layout (Tuning data layout) |
| gpu_guard.sh, campaign store | the tuner's own guard (--allow-idle-foreign, --guard-wait); runs in benchviz_runs/_tuning/<run>/ (run.json, events.jsonl, run.log, exit, remeasure-keys.txt) | the tuner already guards every GPU it uses; a run is not a campaign |
| figure style | interactive SVG | the map is clicked, not printed |
Since the tuning data layout the tab reads one ledger and one table per (op, device) and filters each by dtype. Loading all 39 ledger files takes about 13 s (threadripper02, 2026-10-11); the server warms this at start.
The diff's two picks are the same section 5.4 lookup: in the shipped table, and in the rows sweep_to_table.py would write now. Each takes the first entry the cell's record did not refuse (skipped, that is can_run false). The predicted speedup is the ratio of the two picks' medians in that one record. A race calls candidates within 3% a tie and keeps the earlier one, so a changed pick can read as low as 0.971×. Against main's tables (2026-10-09), gemm double on sm_120 changes 2918 picks, geomean 1.08× where both are timed.
Two engine additions came with it: every progress event carries t (Unix seconds), and --plan emits one plan_job event per (op, dtype) (tools/tune/README.md). ledger_rows now compares a row only with keys in its own and the adjacent floor(index / stride) buckets. All 70 sm_120 ledger tables came out byte-identical, in 8.4 s instead of 126.6 s.
Status: as built 2026-10-08, after every op got a spec. The problems come from the spec writers' preview runs on threadripper02 (sm_120).
Strict pins. vendor is both the vendor family's spelling and a routing class word, and a class word that cannot serve the shape falls back to Auto with a warning. A tuner arm pinned to vendor where can_run refuses it therefore timed Auto and credited the time to the vendor (seen on spmm cdouble N/N nrhs=1 and on gesvd). The tuner's pin is now strict (select::ScopedPin(op, word, select::StrictPin{}), used only in cell_runner.hh): a class word that nothing in its class can serve throws std::invalid_argument, as a spelling does, and vendor must resolve to the candidate spelled vendor. The arm is skipped, reason pin refused (strict): .... Library callers and BATCHLAS_<OP>_ROUTE keep the fallback. spmm's spec-level workaround is gone; gesvd's stays, because there can_run admits a shape whose vendor call faults the device (known defect 15), which no pin rule can see.
Gross losers. After every round from the second on, whatever min_reps says, the race eliminates an arm whose lower median paired ratio to the leader exceeds 4.0 (kGrossLoserRatio, race.hh; the lower of the two middle ratios, so at two rounds both must exceed it). One stalled round never kills an arm. It is recorded eliminated with the median of the rounds it ran. geqrf's vendor (geqrfBatched, one CTA per matrix) is tens to hundreds of times slower than blocked on large cells, and before this rule each of its timed rounds cost minutes.
--max-dim N (default 2048 for tiered runs, 0 = off) is a maintainer rule: no cell is measured with a matrix dimension above N, whatever its bytes. OpSpec::dims returns the extents a cell allocates (default: every integer key but batch; geqrf overrides it because its m is derived from form, n and aspect), and plan_round plans a larger cell as skip:dim before the byte cap, so refinement never measures one either.
Until 2026-10-08 an arm eliminated at more than 10x the winner's median was not timed at cells beyond that cell in the same run, op and dtype (larger along refine_key(), or every other integer key at least as large with more bytes), and was recorded skipped, reason dominated:<key of the losing cell>. It was removed after it fired thousands of times in the sm_120 deep run (trsm cdouble vendor 569 cells, symm double vendor 660; also gemm, syrk, syr2k, trmm, geqrf):
What stays is the in-race gross-loser rule (ratio > 4 over two rounds), which only uses data from the same cell. A ledger record that still holds a dominated: skip is partly stale: the skipped arm's family counts as stale whatever its hash, so the next run re-races exactly those arms against the stored winner and runner-up at the stored tier and writes a merged record (stale_candidates, ledger.cc, and stale_families in sweep_to_table.py; shared fixture tests/data/ledger_dominated).
The sm_120 deep run died with cannot query GPU 3 with nvidia-smi while the maintainer reset GPU 0: nvidia-smi fails for every GPU for a moment during a reset. A failed guard query (--query-compute-apps or the utilization sample) is now retried after 5 s, doubling to 60 s, until --guard-wait (default 300 s) of retrying has passed, with one line on stderr per retry; only then does the run stop (query_with_backoff, tune_core.hh).
An arm benched for confirmed worker errors (two consecutive cells in which the worker and the fresh child both reported error) races alone in a fresh child per cell. If it then errors in 5 consecutive such cells (kDropAfterErrors, worker.hh), it is dropped for the rest of that op and dtype in this run: error, reason dropped after 5 consecutive errors, not run. One clean cell resets the count. The case is the cdouble gemv vendor, which segfaults inside cuBLASLt on every shape on sm_120 (known defect 13) and cost two worker crashes and a retry per cell, about +3 h on a deep gemv cdouble run.
A multi-op run measures op by op in a stable topological order of op_dependencies() (schedule.cc): an op follows every op whose table its timings depend on through a public entry point, and input order breaks ties. all starts from every registered spec in the canonical order, which gives
gemm, trsm, syr2k, gemv, spmm, trmm, symm, syrk, potrf, getrf, getrs, geqrf, ormqr, getri, posv, gesv, orgqr, syev, gesvd.
| op | after |
|---|---|
| trmm, symm | gemm (expand calls the public gemm) |
| getrf | gemm, trsm |
| getrs | trsm |
| geqrf | gemm |
| ormqr | gemm, trmm |
| getri | trsm, getrf |
| posv | potrf, trsm |
| gesv | getrf, getrs |
| orgqr | ormqr |
| syev | gemm, trmm, syr2k, geqrf, ormqr |
| gesvd | gemm, trmm, syr2k |
A dependency not in the run holds nothing back. The hashes still do not follow a dependency's table: retuning gemm changes trmm's expand times without staling trmm's cells.
In a multi-op run, an (op, dtype) pair the spec refuses (symm, syrk and syr2k are real-only) is skipped with one printed == note: skipping <op> <dtype>: ... line. A single-op run still refuses it.
The driver used to shard each round round-robin over the GPUs and join every GPU at the end of each (op, dtype) round, so one GPU finishing gemm float's largest round-2 cells left the other three idle; over 19 ops, 4 dtypes and several rounds those tails cost hours. Now each job's round is one queue sorted by (per-item footprint, bytes), and one thread per GPU (gpu_loop in tiered_driver.cc) pops the next cell of the earliest job, in op order, that has queued cells. Every GPU therefore still runs ascending footprints within a job's round, and the carve-out restart rule is unchanged; moving to another job usually costs a restart, which the summary line and the schedule progress event count. A job's next round is planned by whichever thread records its last cell, never earlier, because refinement reads that round's results; until then the job waits for its stragglers while idle GPUs work on later jobs. Pipelining does not break the op order's dependencies: during one run a composed op times against the library's embedded tables, not the ledger, so an earlier op's unfinished rounds change nothing it measures. Host tests (TuneTieredPipeline.*) check no wait behind a straggler, ascending footprints per GPU and round, planning only after the round's last record, and records identical to a one-GPU run.
Added 2026-10-09. Every cell times and verifies at one leading dimension (ld_pad, normally 0), so a candidate that derives ld from n verifies, wins and ships. The deep sm_120 run was suspected of shipping exactly that for the syev vendor family. That suspicion turned out to be wrong (known defects, the LOBPCG entry), but nothing in the tuner could have caught such a defect if it had been real.
The rule. For each (op, dtype) in a tiered run, the driver asks every candidate's first verified cell (status ok, or eliminated after passing verification) for one extra verification at ld_pad + 3 (kLdAuditPad, tools/tune/spec.hh). The check runs in the same worker or child, straight after the timed cell, as a verify-mode cell, which is a fresh problem built by the spec at the padded ld. LdAuditBook (schedule.hh) records which arms have a verdict (pass or fail). A candidate that the cell did not verify, or whose pin refused the padded shape there (skipped), stays wanted for its next cell.
On a failure. If the padded verification comes back bad or error, or throws, the arm is bad in that cell's record with the reason ld audit: ld_pad +3: .... The failure condemns the kernel, not the cell, so the arm is bad in every cell of that (op, dtype):
The driver prints ld audit FAILED and ld audit: <arm> marked bad in N recorded cells, emits ld_audit and ld_retract progress events, and the run summary gives one line per (op, dtype) with passed, skipped and failed counts and lists the failures (event ld_audit_summary). skipped counts arms whose pin refused the padded shape at every cell that asked. The carve-out audit (above) leaves out an arm that failed the ld audit, because its fresh child re-verifies at the natural ld, where the arm passes. Counting that as a worker mismatch would send the rest of the run to fresh children for nothing.
Limits. The batch stride is padded only where a spec derives it from ld (most use ld * cols). gemm's packed layout ignores ld_pad. A spec that takes no ld_pad audits nothing. The audit is a once-per-candidate check, not a sweep: a defect that appears only at some shapes can still get through.
Evidence. Host tests TuneLdAudit.* cover the child-side fold, the book, the request and outcome round trip, the driver's first-cell rule and reporting, and the carve-out interplay. A deliberate break of each part turns exactly its own test red. GPU check (threadripper02, GPU 2): with vendor syev's run calls passing rows() as lda, a preview syev float jobz=V n=12:18 batch=2:256 sweep reported ld audit FAILED: vendor ... bad residual on its first cell and 5 passed, 0 skipped, 1 failed. With the break removed, the same sweep reported 6 passed, 0 skipped, 0 failed.
Added 2026-10-09 for the cells of the sm_120 deep run that the box's other user damaged. --remeasure-keys FILE takes one <op> <dtype> <key> line per cell, the key in the ledger's own name=value,... form and field order. Each named cell counts as missing, whatever the ledger holds: plan_round gives it the reason remeasure, every runnable candidate, the running tier and a fresh record, not a merge. The stored record's round carries over, so a re-measured refinement midpoint stays a refinement record for the cap and the margin hedge. The driver appends the named cells to round 0 when they are off the lattice. Round 0 is never budgeted, so the named cells always run, but so does every other cell round 0 plans: with stale records (a kernel change) that is the whole stale lattice, whatever --budget. --only-named makes round 0 exactly the named cells, recorded at the run's tier, with no lattice and no refinement; tier custom always behaves so. A named cell is measured past --max-dim, because the deep run measured some cells before the cap existed, but not past --cap-gib. A key whose fields do not match the op's, or an op and dtype the run does not tune, stops the run. The newer record wins in best_records (same tier, later date or run id), so the tables pick up the re-measure with no other change. Host tests: TuneRemeasure.* (a deliberate break of the planner, of the round-0 append or of --only-named turns them red).
Status: measured 2026-10-08/09 on threadripper02 (4x RTX PRO 6000 Blackwell, sm_120, CUDA 13.2, DPC++ /opt/dpcpp-cuda), GPUs 0-3, on a box shared with another user. The tables it produced ship as tuned/*.sm_120.txt; the ledger is benchmarks/results/tuning/ledger/.
What ran. batchlas_tune all --tier deep --dtype float,double,cfloat,cdouble --devices 0,1,2,3 covered all 19 ops and every dtype each op instantiates: 70 (op, dtype) jobs, matrix dimensions up to 2048. It ran from 2026-10-08 06:23 to 2026-10-09 04:54 CEST, with four starts. The first ran 11.5 h and wrote 56,144 cells, until a geqrf cell with an 8192-wide matrix timed out its worker. The second died after 3 minutes and 345 cells when nvidia-smi failed during a GPU reset. The third ran 4.2 h (32,912 cells) and the fourth 4.9 h (35,323 cells), when the maintainer stopped it. That is about 20.6 h of measuring and 124,724 cell records. On 2026-10-09 two follow-up runs went into the same ledger. One re-measured 250 cells that the other user's jobs had damaged ("Engine: re-measuring named cells"): 2.5 min on 4 GPUs. The other re-ran gemv in full (8,515 cells in 33.5 min on 4 GPUs), because the defect-13 fix below edited src/ops/gemv/gemv.cc and so staled every gemv record; that run (20261009T042647-threadripper02-422873) was -dirty, so every reader drops it, and gemv was re-measured at deep on 2026-10-11 (8,558 cells, run 20261011T204301-threadripper02-39013). The campaign wrote 133,489 cell records, the dirty run's included. On 2026-10-12 the 19 sm_120 ledgers hold 210,505 cell records: 157,444 measured (the 2026-10-11 samples and re-measures included) and 53,061 transcribed.
Refinement coverage. Every job's starting lattice is complete. Refinement is not: geqrf, ormqr, getri, posv, gesv, orgqr, syev, gesvd and complex getrs got little of it before the stop (a resumed run wanted 4,072 more refinement cells for the 15 ops of the re-measure run alone), so their crossovers sit on lattice spacing.
Interference. Cells where every candidate stalled (about 2.3 ms or 7-11 ms, more than twice both log-neighbours, which agree on the winner) were screened over each table's first log key: 83 strict outliers (over 2x both neighbours) and 252 soft ones (over 2x their log-interpolated time). The 250 non-gemv soft cells were re-measured at deep; afterwards the screen found 16 strict and 154 soft outliers, 153 of them among the cells just re-measured. The cells left reproduce, so they are real behaviour. Most are the gesvd jacobi cliff at m = 33, where the sub-group solver stops fitting, and spmm and syrk cells within a few microseconds of their neighbours.
What the run found, and what was fixed. Each fix has its own section or defect entry:
1.18.0
