# Guards A guard is a count of violations that must stay zero: how many predictions are not finite, how many points left the manifold, whether the loss crossed a ceiling. A run computes it with everything else, sums it as it sums a counter, and stops at the step where it is not zero, with the guard's name, its metadata, the count, the step and the line it is written on. ```lisp (guard "geometry/off-manifold" (count-where (> (abs (constraint-error x)) 1e-3)) :layer layer) (assert-finite "attention/scores" scores :layer layer) (metric "train/loss" loss :below 20.0) (diagnostic "geometry/state-radius-max" (max r) :reduce :max :layer layer :below 50.0) (metric "lr/effective" lr :above 0.0) ``` | form | counts | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `(guard "name" count :key value ...)` | `count`, a numeric scalar; it returns `count` | | `(assert-finite "name" x :key value ...)` | the elements of `x` that are NaN or infinite, tested and summed in f32, inside a `bf16` region too; it returns `x` | | `:below b` on `metric` or `diagnostic` | 1 when the value is not strictly below `b`, a NaN included | | `:above a` on `metric` or `diagnostic` | 1 when the value is not strictly above `a`, a NaN included | | `(count-where mask)` | not a guard: how many elements of the bool `mask` are true, as an f32 scalar summed in f32, the count `guard` reads | The keywords after the value are metadata, as for [`metric`](https://sx.041.io/docs/metrics.md#reporting-from-the-experiment). A bound's guard has the name and metadata of its metric or diagnostic, so `(metric "train/loss" loss :below 20.0)` is that metric plus a guard `train/loss`. Each bound is a guard, so `:above 0.0 :below 20.0` makes two guards of that one name, which count into one `guard/train/loss` series. A bound compares in f32 against the number as written. A count is a number, not a bool: `(guard "x" (> loss 9.0))` is `E-CONTRACT-020`, and `(count-where (> loss 9.0))` is the count. ## Where they run | where | when it counts | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `objective`, `prepare`, the model | every microbatch of every step | | `evaluate`, the model in a pass | every batch of the pass | | an optimizer body | every parameter's update, every step | | a diagnostic's bound | only when the run [selects](https://sx.041.io/docs/diagnostics.md#turning-diagnostics-on) the diagnostic, and only on [sampled](https://sx.041.io/docs/diagnostics.md#when-they-compute) steps | | a generator, an initializer, the curriculum | refused, `E-CONTRACT-019`: none of them has a step to stop; a diagnostic's bound in a generator is dropped with the diagnostic | | a `tap-gradient` lambda | only a diagnostic's bound, since a gradient exists only on sampled steps; anything else is `E-CONTRACT-017` | A bound is checked on every value its graph computes, not on the number the series reports: a step whose two microbatch losses are 19 and 21 under `:below 20.0` counts 1, though it reports their mean, 20. On a diagnostic the same holds for every parameter of an optimizer group, so `:above` on a `:reduce :max` fails when any value is not above the bound, though the max is, and likewise `:below` on a `:reduce :min`. ## When a run stops A training step sums each guard over its microbatches and every rank, so every GPU stops at the same step. The step's update may already be committed when the total is known; the run stops at that step, writes no checkpoint for it, and the last checkpoint stands. An evaluation pass sums its batches and stops the run after the pass. An optimizer's guard is read before anything of the step is committed, with the [finite guard](https://sx.041.io/docs/checkpoints.md#the-finite-guard). It counts each parameter's update on its own, and the error names the first parameter whose update counted, with that update's count. In one step, a training guard's error comes first, since it names the cause, then the finite guard's, then an optimizer guard's. The failure is that of the step not yet taken, the step its annotations carry. ```console sexpgpu run: E-GUARD-001: guard geometry/off-manifold layer=3 counted 12 at step 4107, expected 0; it is written at runs/lorentz.sx:41:5 ``` The run fails like any other: the status file says `failed` with the error, as does the stream's closing `failed: ` annotation, and an annotation `guard failed` marks the step with the guard, its metadata, its count and its source. ## What they report Each guard reports `guard/` at every step it is computed, with its metadata and the runtime's `split`, `pass` and `stage`, and an optimizer's with its group's `param`, summed over the group's parameters: a flat zero, until it is not. A count past what an f32 holds is left out of the series, and the error carries it whole. Each guard and metadata is one series, counted in the [reporting rate](https://sx.041.io/docs/metrics.md#reporting-rate) like any metric. `explain` lists every guard with the graphs it runs in and its source. Guards never reach the curriculum, the summary or the progress line. ## What they cost A training or evaluation guard's count is one more value in the read the step already makes for its counters: no new synchronization point and nothing between microbatches, though on CUDA it can add a small transfer. An optimizer's guard is read with the finite guard's results before the update is committed; with `:guard-finite` off, it adds that read. A file without guards pays nothing. Related: [diagnostics](https://sx.041.io/docs/diagnostics.md), [metrics](https://sx.041.io/docs/metrics.md), [checkpoints](https://sx.041.io/docs/checkpoints.md#the-finite-guard), [diagnostic codes](https://sx.041.io/docs/errors.md). --- SexpGPU documentation. Every page: https://sx.041.io/llms.txt