Rigor Log
Every entry here is a moment I was about to be wrong in public.
Each entry records what the first metric said, what a second and independent metric said, the fix, and the file it can be checked in.
- 01silentsurrogate-trust-auditREPORT.md, section 2.4
The test was wrong and the solver was right
First metric saidA test asserted that the residual of an exact reference step stays below 1e-3. It measured 5.3e-3, so the solver looked broken.
Second metric saidRefinement showed the residual falls by exactly 4.00x per halving of the timestep, which is second-order convergence. I derived the closed-form law it should follow and it matched to within 1%: predicted 3.95e-3 against 3.9625e-3 measured. The solver was right and my threshold was a number with no theoretical basis.
FixReplace the arbitrary threshold with two stronger tests: that the floor converges at order 2, and that its magnitude matches the derived law.
LessonWhen a test fails, one of two things is wrong and it is not always the code. No tolerance was loosened to make anything pass.
- 02silentSkyScoutscripts/check-robot-descriptions.py
Parsing cleanly proves almost nothing
First metric saidThe robot description files parsed without error, and the bundled SDF validator passed them.
Second metric saidThree separate faults were live at once: a mesh URI using a scheme the consumer could not resolve, a mesh pointing at a file that was never modelled, and link and joint names that collide. The second was hidden by the first, because an unresolvable scheme is silently skipped by the existence check.
FixWrite a dedicated integrity checker and run it before and after every export. It exits non-zero on error, so it cannot be ignored.
LessonA file that parses is a file the parser understood, not a file that is correct. The name collision is legal in URDF and invalid in SDF, so even a valid file can be wrong for its destination.
- 03silentSkyScoutdocs/design/, and the repo checkers in scripts/
Nine design changes, nine passes, one interfering hinge
First metric saidA clearance check measured the distance between the hinge parts and passed. It passed on nine straight design changes.
Second metric saidA boolean intersection test, asking whether the two solids actually share volume, found the hinges had been interfering the whole time.
FixReplace the distance check with a solid overlap test on the real geometry.
LessonNine passes of a weak test is not nine pieces of evidence. It is one blind spot, repeated nine times.
- 04silentAdversarial Swarm Defence Simconfig.yaml, gan.generator.output_scale
The critic was winning for the wrong reason
First metric saidThe GAN was training. The critic separated real swarms from generated ones and the losses moved.
Second metric saidIt was separating them by magnitude alone. Real normalised features span about -2.78 to +5.47, and a bare Tanh output only reaches -1 to +1, under 40% of that range. The critic never had to look at how a swarm moves.
FixScale the generator output by 6.0 so it can reach the real support, and record the reason next to the value.
LessonA discriminator that is winning tells you it found a difference, not that it found the difference you care about. Run 1 diverged and mode collapsed because of it.
- 05caughtAdversarial Swarm Defence Simconfig.yaml, gan.discriminator.minibatch_std
Two proven techniques made it worse
First metric saidMinibatch standard deviation and DiffAugment are established GAN stabilisers with published results behind them, so adding them should have delayed mode collapse.
Second metric saidI measured when collapse actually arrived. With them, epoch 55. Without them, epoch 145. On a 560-sample dataset of 10 features by 99 timesteps they were a clear net negative.
FixDisable both, and leave the code in the repository with the measurement written beside the switch so the decision can be revisited at a larger data scale.
LessonA technique proven on image GANs with far more data is not proven on mine. Measuring the thing I was about to assume cost one run and saved the rest.
- 06caughtsurrogate-trust-auditPREREGISTRATION.md and results/tables/
The deliberately dumb baseline beat everything
First metric saidTrust signals for a neural PDE surrogate, a fast learned stand-in for a physics solver, looked like they were detecting when the surrogate was wrong.
Second metric saidA matched-capacity null model that sees only the input, registered as the primary comparison before any results, beat every trust signal tested.
FixReport the negative result, publish the code and the full audit trail, and do not quietly retire the null.
LessonRegister the dumb baseline before you run anything. Choose it afterwards and you will choose a weak one without noticing you did.
- 07caughtA Lead Time Is Not a DetectionPaper, pre-registered firing rule: 5 sigma over 3 checkpoints
weight_norm fired on zero of eight seeds
First metric saidweight_norm is cited as an early-warning signal for grokking, with a reported lead time.
Second metric saidUnder a detection rule fixed before I looked at the results, weight_norm fired on 0 of 8 seeds. The Fourier signal, under the same rule, led on 8 of 8 by a median of about 2,600 steps.
FixReport the signal as failing the rule, rather than loosening the rule until it passes.
LessonA lead time you can only produce after choosing the threshold is not a detection. That sentence became the title of the paper.
- 08silentA Lead Time Is Not a DetectionPaper, section on predictor screening
A second predictor that only worked in hindsight
First metric saidembed_eff_rank looked like it fired early, which would have given me a second working predictor and a much nicer paper.
Second metric saidUnder the same pre-registered rule it fired on 5 of 8 seeds, but after grokking had already happened. An early warning that arrives late is not an early warning.
FixDrop it from the claimed detectors, and say in the paper why it was dropped.
LessonA predictor that only fires once you already know the answer is a plot, not a predictor. Losing it made the paper smaller and correct.
- 09silentSkyScoutbrain/b2_step1_imu.json
The gyroscope was in the wrong frame, 187x wrong
First metric saidPosition error over the run looked fine, so the gyroscope integration looked fine.
Second metric saidA test aimed only at the frame convention, which axis set the angular rates are measured in, came back 187 times off.
FixCorrect the frame convention, then keep the isolated test in the suite so the same mistake cannot come back quietly.
LessonA healthy end-to-end number can hide a wrong sign or a wrong frame. Test the convention on its own, where nothing else can compensate for it.
TODO: confirm- Add the exact before and after numbers, and name which frame was wrong.
- 10silentsurrogate-trust-auditREPORT.md, section 2.2
A float32 wavenumber hiding inside a float64 field
First metric saidSpectral derivatives looked fine at a loose tolerance, and the field was float64 throughout.
Second metric saidA test at a 1e-10 tolerance in float64 failed. torch.fft.rfftfreq follows the default dtype, so it handed back float32 wavenumbers even for a float64 field, and squaring them cost two significant digits.
FixBuild the wavenumbers at float64 and cast to the complex dtype at the end.
LessonPrecision loss does not raise anything. Accuracy went from about 1e-5 to 3.3e-13 once the tolerance was tight enough to notice.
- 11silentSkyScoutbrain/b5_a1_rover_costmap.json
Drone-safe is not rover-safe
First metric saidThe drone had mapped the region and the hazard map scored well against ground truth, so the routes it handed down should be drivable.
Second metric saidRunning the rover traversability model over the same cells found 888 that the rover cannot cross and the drone can fly over without noticing. 427 of them fail on step height alone, which a slope-only check cannot see.
FixGive the rover its own cost map with its own limits, and plan over that rather than over the drone view.
LessonA map is safe for the vehicle it was built for. The drone had zero cells blocked that the rover could pass, which is the asymmetry: the aerial view is optimistic in exactly one direction.
- 12caughtSkyScoutbrain/b2_step4_correct.json
Proving the filter was right, not just quiet
First metric saidDead reckoning drifted 10.9 m in 14.4 s. After the ESKF went in the drift was gone, so the filter looked correct.
Second metric saidI checked whether the estimated sensor biases converged to the true biases I had injected into the simulation. They did.
FixMake bias convergence the pass criterion, not the drift number.
LessonA small position error can come from a filter cancelling two separate mistakes against each other. Recovering a hidden value you planted yourself is much harder to fake.
- 13silentSkyScoutbrain/b3_step3_truth.json
The controller flew the right shape 3.4x too fast
First metric saidThe drone followed the survey path and the path-following error looked acceptable.
Second metric saidComparing the commanded speed against the intended survey speed showed the lookahead controller was flying 3.4 times too fast.
FixCorrect the lookahead so the commanded speed matches the survey speed.
LessonTracking error tells you the shape is right. It says nothing about the pace, and a survey flown too fast is a survey that missed things.
- 14caughtinklesstest_inkless.py
A test that forbids reading the clock
First metric saidThe output was correct and the test suite passed, so the build looked reproducible.
Second metric saidOne call to the clock anywhere in the engine would make every build different, and no ordinary test would catch it because nothing else would change. A test parses the engine source into a syntax tree and asserts that time, datetime, calendar, random and uuid are never imported.
FixKeep the syntax-tree test in the suite, and pass the date in through a flag so the program never asks what today is.
LessonReproducibility is a property you enforce, not one you claim. A rebuild in an empty virtualenv is byte-identical by SHA-256 because the engine has no way to be otherwise.
- 15silentThis sitesrc/data/assets.ts
The asset check that quietly said no
First metric saidThe build passed, every page rendered, and the resume and portrait checks reported that neither file was there.
Second metric saidBoth files were sitting in public/. The check resolved its directory from import.meta.url, which points into dist/ once Vite has bundled the module, so it was testing a path that never exists. The rendered HTML still said "Photo to add" with the photo in place.
FixResolve from process.cwd(), and throw if the public directory itself is missing so a bad path fails the build instead of failing quietly.
LessonA check that can only answer no is not a check. This one had been wrong since the day I wrote it and the build never once complained.