Rigor Log

Every entry here is a moment I was about to be wrong in public.

Each entry records what the first metric said, what a second and independent metric said, the fix, and the file it can be checked in.

15entries
6projects
10would have shipped silently
  1. 01silent
    surrogate-trust-auditREPORT.md, section 2.4

    The test was wrong and the solver was right

    First metric said

    A test asserted that the residual of an exact reference step stays below 1e-3. It measured 5.3e-3, so the solver looked broken.

    Second metric said

    Refinement showed the residual falls by exactly 4.00x per halving of the timestep, which is second-order convergence. I derived the closed-form law it should follow and it matched to within 1%: predicted 3.95e-3 against 3.9625e-3 measured. The solver was right and my threshold was a number with no theoretical basis.

    Fix

    Replace the arbitrary threshold with two stronger tests: that the floor converges at order 2, and that its magnitude matches the derived law.

    Lesson

    When a test fails, one of two things is wrong and it is not always the code. No tolerance was loosened to make anything pass.

  2. 02silent
    SkyScoutscripts/check-robot-descriptions.py

    Parsing cleanly proves almost nothing

    First metric said

    The robot description files parsed without error, and the bundled SDF validator passed them.

    Second metric said

    Three separate faults were live at once: a mesh URI using a scheme the consumer could not resolve, a mesh pointing at a file that was never modelled, and link and joint names that collide. The second was hidden by the first, because an unresolvable scheme is silently skipped by the existence check.

    Fix

    Write a dedicated integrity checker and run it before and after every export. It exits non-zero on error, so it cannot be ignored.

    Lesson

    A file that parses is a file the parser understood, not a file that is correct. The name collision is legal in URDF and invalid in SDF, so even a valid file can be wrong for its destination.

  3. 03silent
    SkyScoutdocs/design/, and the repo checkers in scripts/

    Nine design changes, nine passes, one interfering hinge

    First metric said

    A clearance check measured the distance between the hinge parts and passed. It passed on nine straight design changes.

    Second metric said

    A boolean intersection test, asking whether the two solids actually share volume, found the hinges had been interfering the whole time.

    Fix

    Replace the distance check with a solid overlap test on the real geometry.

    Lesson

    Nine passes of a weak test is not nine pieces of evidence. It is one blind spot, repeated nine times.

  4. 04silent
    Adversarial Swarm Defence Simconfig.yaml, gan.generator.output_scale

    The critic was winning for the wrong reason

    First metric said

    The GAN was training. The critic separated real swarms from generated ones and the losses moved.

    Second metric said

    It was separating them by magnitude alone. Real normalised features span about -2.78 to +5.47, and a bare Tanh output only reaches -1 to +1, under 40% of that range. The critic never had to look at how a swarm moves.

    Fix

    Scale the generator output by 6.0 so it can reach the real support, and record the reason next to the value.

    Lesson

    A discriminator that is winning tells you it found a difference, not that it found the difference you care about. Run 1 diverged and mode collapsed because of it.

  5. 05caught
    Adversarial Swarm Defence Simconfig.yaml, gan.discriminator.minibatch_std

    Two proven techniques made it worse

    First metric said

    Minibatch standard deviation and DiffAugment are established GAN stabilisers with published results behind them, so adding them should have delayed mode collapse.

    Second metric said

    I measured when collapse actually arrived. With them, epoch 55. Without them, epoch 145. On a 560-sample dataset of 10 features by 99 timesteps they were a clear net negative.

    Fix

    Disable both, and leave the code in the repository with the measurement written beside the switch so the decision can be revisited at a larger data scale.

    Lesson

    A technique proven on image GANs with far more data is not proven on mine. Measuring the thing I was about to assume cost one run and saved the rest.

  6. 06caught
    surrogate-trust-auditPREREGISTRATION.md and results/tables/

    The deliberately dumb baseline beat everything

    First metric said

    Trust signals for a neural PDE surrogate, a fast learned stand-in for a physics solver, looked like they were detecting when the surrogate was wrong.

    Second metric said

    A matched-capacity null model that sees only the input, registered as the primary comparison before any results, beat every trust signal tested.

    Fix

    Report the negative result, publish the code and the full audit trail, and do not quietly retire the null.

    Lesson

    Register the dumb baseline before you run anything. Choose it afterwards and you will choose a weak one without noticing you did.

  7. 07caught
    A Lead Time Is Not a DetectionPaper, pre-registered firing rule: 5 sigma over 3 checkpoints

    weight_norm fired on zero of eight seeds

    First metric said

    weight_norm is cited as an early-warning signal for grokking, with a reported lead time.

    Second metric said

    Under a detection rule fixed before I looked at the results, weight_norm fired on 0 of 8 seeds. The Fourier signal, under the same rule, led on 8 of 8 by a median of about 2,600 steps.

    Fix

    Report the signal as failing the rule, rather than loosening the rule until it passes.

    Lesson

    A lead time you can only produce after choosing the threshold is not a detection. That sentence became the title of the paper.

  8. 08silent
    A Lead Time Is Not a DetectionPaper, section on predictor screening

    A second predictor that only worked in hindsight

    First metric said

    embed_eff_rank looked like it fired early, which would have given me a second working predictor and a much nicer paper.

    Second metric said

    Under the same pre-registered rule it fired on 5 of 8 seeds, but after grokking had already happened. An early warning that arrives late is not an early warning.

    Fix

    Drop it from the claimed detectors, and say in the paper why it was dropped.

    Lesson

    A predictor that only fires once you already know the answer is a plot, not a predictor. Losing it made the paper smaller and correct.

  9. 09silent
    SkyScoutbrain/b2_step1_imu.json

    The gyroscope was in the wrong frame, 187x wrong

    First metric said

    Position error over the run looked fine, so the gyroscope integration looked fine.

    Second metric said

    A test aimed only at the frame convention, which axis set the angular rates are measured in, came back 187 times off.

    Fix

    Correct the frame convention, then keep the isolated test in the suite so the same mistake cannot come back quietly.

    Lesson

    A healthy end-to-end number can hide a wrong sign or a wrong frame. Test the convention on its own, where nothing else can compensate for it.

    TODO: confirm
    • Add the exact before and after numbers, and name which frame was wrong.
  10. 10silent
    surrogate-trust-auditREPORT.md, section 2.2

    A float32 wavenumber hiding inside a float64 field

    First metric said

    Spectral derivatives looked fine at a loose tolerance, and the field was float64 throughout.

    Second metric said

    A test at a 1e-10 tolerance in float64 failed. torch.fft.rfftfreq follows the default dtype, so it handed back float32 wavenumbers even for a float64 field, and squaring them cost two significant digits.

    Fix

    Build the wavenumbers at float64 and cast to the complex dtype at the end.

    Lesson

    Precision loss does not raise anything. Accuracy went from about 1e-5 to 3.3e-13 once the tolerance was tight enough to notice.

  11. 11silent
    SkyScoutbrain/b5_a1_rover_costmap.json

    Drone-safe is not rover-safe

    First metric said

    The drone had mapped the region and the hazard map scored well against ground truth, so the routes it handed down should be drivable.

    Second metric said

    Running the rover traversability model over the same cells found 888 that the rover cannot cross and the drone can fly over without noticing. 427 of them fail on step height alone, which a slope-only check cannot see.

    Fix

    Give the rover its own cost map with its own limits, and plan over that rather than over the drone view.

    Lesson

    A map is safe for the vehicle it was built for. The drone had zero cells blocked that the rover could pass, which is the asymmetry: the aerial view is optimistic in exactly one direction.

  12. 12caught
    SkyScoutbrain/b2_step4_correct.json

    Proving the filter was right, not just quiet

    First metric said

    Dead reckoning drifted 10.9 m in 14.4 s. After the ESKF went in the drift was gone, so the filter looked correct.

    Second metric said

    I checked whether the estimated sensor biases converged to the true biases I had injected into the simulation. They did.

    Fix

    Make bias convergence the pass criterion, not the drift number.

    Lesson

    A small position error can come from a filter cancelling two separate mistakes against each other. Recovering a hidden value you planted yourself is much harder to fake.

  13. 13silent
    SkyScoutbrain/b3_step3_truth.json

    The controller flew the right shape 3.4x too fast

    First metric said

    The drone followed the survey path and the path-following error looked acceptable.

    Second metric said

    Comparing the commanded speed against the intended survey speed showed the lookahead controller was flying 3.4 times too fast.

    Fix

    Correct the lookahead so the commanded speed matches the survey speed.

    Lesson

    Tracking error tells you the shape is right. It says nothing about the pace, and a survey flown too fast is a survey that missed things.

  14. 14caught
    inklesstest_inkless.py

    A test that forbids reading the clock

    First metric said

    The output was correct and the test suite passed, so the build looked reproducible.

    Second metric said

    One call to the clock anywhere in the engine would make every build different, and no ordinary test would catch it because nothing else would change. A test parses the engine source into a syntax tree and asserts that time, datetime, calendar, random and uuid are never imported.

    Fix

    Keep the syntax-tree test in the suite, and pass the date in through a flag so the program never asks what today is.

    Lesson

    Reproducibility is a property you enforce, not one you claim. A rebuild in an empty virtualenv is byte-identical by SHA-256 because the engine has no way to be otherwise.

  15. 15silent
    This sitesrc/data/assets.ts

    The asset check that quietly said no

    First metric said

    The build passed, every page rendered, and the resume and portrait checks reported that neither file was there.

    Second metric said

    Both files were sitting in public/. The check resolved its directory from import.meta.url, which points into dist/ once Vite has bundled the module, so it was testing a path that never exists. The rendered HTML still said "Photo to add" with the photo in place.

    Fix

    Resolve from process.cwd(), and throw if the public directory itself is missing so a bad path fails the build instead of failing quietly.

    Lesson

    A check that can only answer no is not a check. This one had been wrong since the day I wrote it and the build never once complained.