Guards

A guard is a count of violations that must stay zero: how many predictions are not finite, how many points left the manifold, whether the loss crossed a ceiling. A run computes it with everything else, sums it as it sums a counter, and stops at the step where it is not zero, with the guard's name, its metadata, the count, the step and the line it is written on.

(guard "geometry/off-manifold" (count-where (> (abs (constraint-error x)) 1e-3)) :layer layer)
(assert-finite "attention/scores" scores :layer layer)
(metric "train/loss" loss :below 20.0)
(diagnostic "geometry/state-radius-max" (max r) :reduce :max :layer layer :below 50.0)
(metric "lr/effective" lr :above 0.0)

form	counts
`(guard "name" count :key value ...)`	`count`, a numeric scalar; it returns `count`
`(assert-finite "name" x :key value ...)`	the elements of `x` that are NaN or infinite, tested and summed in f32, inside a `bf16` region too; it returns `x`
`:below b` on `metric` or `diagnostic`	1 when the value is not strictly below `b`, a NaN included
`:above a` on `metric` or `diagnostic`	1 when the value is not strictly above `a`, a NaN included
`(count-where mask)`	not a guard: how many elements of the bool `mask` are true, as an f32 scalar summed in f32, the count `guard` reads

The keywords after the value are metadata, as for metric. A bound's guard has the name and metadata of its metric or diagnostic, so (metric "train/loss" loss :below 20.0) is that metric plus a guard train/loss. Each bound is a guard, so :above 0.0 :below 20.0 makes two guards of that one name, which count into one guard/train/loss series. A bound compares in f32 against the number as written. A count is a number, not a bool: (guard "x" (> loss 9.0)) is E-CONTRACT-020, and (count-where (> loss 9.0)) is the count.

Where they run

where	when it counts
`objective`, `prepare`, the model	every microbatch of every step
`evaluate`, the model in a pass	every batch of the pass
an optimizer body	every parameter's update, every step
a diagnostic's bound	only when the run selects the diagnostic, and only on sampled steps
a generator, an initializer, the curriculum	refused, `E-CONTRACT-019`: none of them has a step to stop; a diagnostic's bound in a generator is dropped with the diagnostic
a `tap-gradient` lambda	only a diagnostic's bound, since a gradient exists only on sampled steps; anything else is `E-CONTRACT-017`

A bound is checked on every value its graph computes, not on the number the series reports: a step whose two microbatch losses are 19 and 21 under :below 20.0 counts 1, though it reports their mean, 20. On a diagnostic the same holds for every parameter of an optimizer group, so :above on a :reduce :max fails when any value is not above the bound, though the max is, and likewise :below on a :reduce :min.

When a run stops

A training step sums each guard over its microbatches and every rank, so every GPU stops at the same step. The step's update may already be committed when the total is known; the run stops at that step, writes no checkpoint for it, and the last checkpoint stands. An evaluation pass sums its batches and stops the run after the pass.

An optimizer's guard is read before anything of the step is committed, with the finite guard. It counts each parameter's update on its own, and the error names the first parameter whose update counted, with that update's count. In one step, a training guard's error comes first, since it names the cause, then the finite guard's, then an optimizer guard's. The failure is that of the step not yet taken, the step its annotations carry.

sexpgpu run: E-GUARD-001: guard geometry/off-manifold layer=3 counted 12 at step 4107, expected 0; it is written at runs/lorentz.sx:41:5

The run fails like any other: the status file says failed with the error, as does the stream's closing failed: <error> annotation, and an annotation guard <name> failed marks the step with the guard, its metadata, its count and its source.

What they report

Each guard reports guard/<name> at every step it is computed, with its metadata and the runtime's split, pass and stage, and an optimizer's with its group's param, summed over the group's parameters: a flat zero, until it is not. A count past what an f32 holds is left out of the series, and the error carries it whole. Each guard and metadata is one series, counted in the reporting rate like any metric. explain lists every guard with the graphs it runs in and its source. Guards never reach the curriculum, the summary or the progress line.

What they cost

A training or evaluation guard's count is one more value in the read the step already makes for its counters: no new synchronization point and nothing between microbatches, though on CUDA it can add a small transfer. An optimizer's guard is read with the finite guard's results before the update is committed; with :guard-finite off, it adds that read. A file without guards pays nothing.