Surrogate Trust Audit

The finding

Partial differential equations describe how physical things change: heat spreading, fluid moving, a wave travelling. Solving them properly is slow, so people train a neural network to stand in for the solver. It is much faster, and sometimes it is wrong in ways the output does not show.

Three signals are meant to tell you when to stop trusting it: the PDE residual of the surrogate’s own rollout, disagreement across a seed ensemble, and a frozen-feature embedding probe borrowed from the interatomic-potential literature.

Before running the grid I registered the comparison I would judge all three against: a null model of matched capacity that sees only the input, and never sees the physics, the residual or the solver. On the six most shifted cells, all three indicators lost to it, and every confidence interval sat below zero.

Why it matters

A null model is a deliberately unintelligent baseline. If it matches your clever method, the clever method is not using the information you think it is.

Losing to an input-only null means these indicators are largely reading which inputs tend to be hard, not detecting that this particular prediction is wrong. That is a real pattern, and it is useful, but it is a prior dressed up as a diagnosis.

Registering the null first is what makes that statement clean. Chosen afterwards, I would have picked a weaker baseline without noticing I was doing it.

Method

Three PDE families, advection, heat and Burgers. Two architectures, an FNO and a PINN. Six regimes running from in-distribution out to severe shift. That is 36 cells, each with 1,360 test samples.

The whole study rests on the true error being worth measuring against, so every solver was verified first.

observed expected tolerance

Results

Family

supervisedunsupervised

input-only null model trust indicator spread across the six cells

The preregistered test compares each indicator class against the null on the six most shifted cells. All three return the same verdict, recorded in the results table as “fails: null better”:

The PDE residual loses by 0.286, with a 95% interval from -0.333 to -0.235. Seed-ensemble disagreement loses by 0.694. The embedding probe, the closest of the three, still loses by 0.077 with an interval from -0.119 to -0.043.

Measured without shift, the same indicators look useful, which is why the shifted case is the one worth testing.

What I checked that could have broken it

The true error is only meaningful if the reference solvers are right, so each one was refined on a grid and its convergence order measured against the order the scheme should have. All four passed inside a tolerance of 0.15: upwind at 0.993 against an expected 1.0, Lax-Wendroff at 2.000, Crank-Nicolson at 2.000, and the Burgers spectral scheme at 1.956.

The Burgers convergence study uses an exact Cole-Hopf solution rather than a fine-grid rerun of the same scheme, because comparing a scheme against itself cannot see an error it makes consistently.

The harder problem is on the surrogate side, and it limits the result rather than supporting it. The PINN barely learns: measured against simply predicting no change at all, its validation error averages 0.957, where 1.0 would mean it learned nothing useful. The FNO, by comparison, sits at 0.0002. Half of the confirmatory cells therefore describe a model close to the identity map, and I report that rather than dropping those cells.

Limits

The PINN trainability problem above is the largest one. With half the confirmatory cells describing a model at or near the identity map, the confirmatory comparison is weaker than the cell count suggests.

“Reference-free” is not quite true of the embedding probe, which is fitted on data that the deployment setting would not have.

The residual indicator’s noise floor is tied to the same axis as the shift, so part of what it tracks is the regime rather than the error.

Three PDE families at one resolution and one width. Nothing here says the same holds for larger surrogates, other equations, or operator learning at scale.