# Generation inside the step

`defgenerate` makes records on the device, inside each training step,
before the step's training graph reads them: a student sampling its own
rows, a frozen teacher labelling them, a pool scored and cut down to the
rows worth a gradient. It is a loop of ordinary graphs the runtime runs; no
op carries a subgraph, and nothing is differentiated through it.

```lisp
(defgenerate :trips 6
             :init (lambda (batch ctx) (list :row ... :stride ...))
             :step (lambda (state trip ctx) (list :row ... :stride ...))
             :finish (lambda (state ctx) (list :sample ... :label ...)))
```

| keyword | meaning |
|---|---|
| `:trips` | how many times `:step` runs, 1 or more |
| `:init` | the batch, as `prepare` receives it, and the step's context, to the state: a plist of tensors, a keyword before each |
| `:step` | the state, the trip as an `i64` scalar and the context, to the next state, with the keys, shapes and dtypes `:init` made |
| `:finish` | optional: the last state and the context, once, to any plist of tensors |

Each training microbatch runs `:init` on its batch, `:step` `:trips` times
and `:finish` once, the state staying on the device. What `:finish`
returned, or the last state without one, joins the batch: `prepare` reads
each entry with `(field batch :name)` as it reads a loader field, so a
generated name may not be a field of the training loader or of any pass's
loader. An [evaluation pass](https://sx.041.io/docs/evaluation.md) whose loader has the training
loader's fields and batch size runs the generator on its batches too; a
pass whose loader differs does not, and its `prepare` reading a generated
name is `E-CONTRACT-005`.

The state has static shapes. A sequence written one position per trip is a
fixed-size tensor written with `where` at the trip's column; a cache is the
same. The fixture `crates/cli/tests/fixtures/tiny-generate.sx` has the
student write each row greedily, one position per trip, and `:finish` label
every position by the rule it is learning:

```lisp
:step (lambda (state trip ctx)
        (let* ((row (getf state :row))
               (at (+ (cast trip :i32) 2))
               (column (iota [1 seq] :axis 1))
               (guess (sum (where (= column (- at 1)) (argmax (model row)) 0)
                           :axes [1]
                           :keepdims true)))
          (list :row (where (= column at) guess row)
                :stride (getf state :stride))))
```

## What a generator may read

- The batch's fields in `:init`, the state in `:step` and `:finish`, and
  every parameter, [frozen](https://sx.041.io/docs/models.md#frozen-and-pretrained-parameters) or not: the model `model` binds,
  or any model reachable from it, runs inside a generator as it does in
  `objective`, at the parameters the step started from.
- The context: the builtin keys of [ctx](https://sx.041.io/docs/curriculum.md#ctx) only, and the
  trip as `:step`'s second argument. A counter is not among them:
  `prepare` declares counters, and it is traced after the generator, so
  `(getf ctx :tokens)` is nil there. A generator that reads `:microbatch`,
  `:records` or `:stage-records` keeps the run unstacked, since those move
  between the microbatches one stacked call holds.
- A draw is a pure function of the seed, its place and its salt; see
  [random draws](https://sx.041.io/docs/determinism.md#random-draws). An unsalted draw is the same
  at every step and microbatch. Salted with
  `(+ (* (getf ctx :step) 65536) (getf ctx :microbatch))` it is new for
  every microbatch, and the `:microbatch` keeps the run unstacked; a salt
  of `:step` alone is new every step and stacks.

## What a generator may not do

- Report: a `metric` inside it is `E-GEN-004`; return the value in the
  state and report it from `objective`. A `diagnostic` inside it, which a
  model it runs may well call, is never selected and is dropped, as an
  unselected diagnostic is anywhere.
- Declare a counter: `counter` belongs to `prepare` (`E-CONTRACT-010`).
- Change the state's keys, shapes or dtypes from trip to trip
  (`E-GEN-002`), or return anything but a plist of tensors (`E-GEN-001`).
  Under `:bf16` a state entry keeps the dtype it was made with, so a cache
  can be `bf16` from trip to trip.

## A phase of the step

- **Stacking.** A run that stacks microbatches stacks the generator with
  the same degree: each copy has its own batch and state, the parameters
  and the step's context scalars are shared, and the bits are those of the
  microbatches one at a time. The cost model prices a step's generation
  and training calls together; see [devices](https://sx.041.io/docs/devices.md#memory).
- **Memory.** Each generator graph is a phase of the memory plan: its live
  set beside the batch `:init` read and the state the trips carry, and
  what it made lives until the training call that reads it. The report's
  `memory` line shows the largest, `generation 0.4 GiB`.
- **Signals.** The loop looks for a signal before every trip; a stop
  abandons the step in flight, and the resume takes it again. See
  [checkpoints](https://sx.041.io/docs/checkpoints.md#signals).
- **Reports.** `sexpgpu explain` lists `generate.init`, `generate.step` and
  `generate.finish` with the graphs and the trips under the training step;
  the [lowering report](https://sx.041.io/docs/explain.md#the-lowering-report) has a `generate`
  row, `generate stacked` when the run stacks, and a `generation` line.
- **Determinism.** A generator is part of the file, so it adds nothing to
  a checkpoint's identity, and the trip is an input like the step; see
  [determinism](https://sx.041.io/docs/determinism.md).

Related: [the run file](https://sx.041.io/docs/run-file.md), [diagnostic codes](https://sx.041.io/docs/errors.md#generation).

---

SexpGPU documentation. Every page: https://sx.041.io/llms.txt