Macros and symbols

defmacro receives unevaluated forms and returns a form, which is evaluated in the caller's lexical environment. Macros are unhygienic by default, as in Common Lisp: capture is allowed and sometimes the point. Gensyms protect introduced bindings; module identities protect a library's private helpers.

(defmacro twice (x)
  (once-only (x) `(+ ,x ,x)))          ; x is evaluated once

(defmacro/g! twice-explicit (x)
  `(let ((,g!value ,x)) (+ ,g!value ,g!value)))

The helpers

In the prelude unless noted:

form	behavior
`(gensym [prefix])`	a fresh uninterned symbol on every call, even across compilations in one process; a builtin
`(with-gensyms (name...) body...)`	binds each name to a fresh symbol while the body builds code
`(once-only (arg...) body...)`	in a macro body, generates bindings that evaluate each argument once, in order
`(defmacro/g! name lambda-list body...)`	each distinct `g!name` in the body becomes a fresh symbol per expansion
`(macroexpand-1 form [env])`	expands the outer macro call once; returns the form and an expansion flag
`(macroexpand form [env])`	expands the outer call until it is not a macro; the same two values
`(intern string [env])`	the symbol visible under that name in `env`, or one of this module
`(macrolet ((name ll body...)...) body...)`	local macros; expansion bodies see the enclosing environment

Expansion inspection does not evaluate the result or expand subforms. macroexpand defaults to its own lexical environment; &environment receives the caller's. Put (macroexpand-1 '(twice (f y))) in a file and sexpgpu eval prints the expansion.

Symbols and capture

A symbol belongs to its module, or is a fresh symbol from gensym. Equal spelling does not make two modules' private symbols equal.
eq and equal compare identity. symbol-name returns the printable name, which is not the identity. Gensyms print as #:name; that is not reader syntax.
An unbound ordinary symbol falls back to the prelude by name, so a module may define its own when without changing anyone else's.
A symbol a macro writes belongs to the macro's module. It is looked up there and then in the prelude, never in the file the macro expands in, so a library macro calls the library's helpers and the caller imports the macro alone. A name only the caller defines is E-EVAL-003, naming the macro's module; pass it to the macro as an argument instead.
Unquoted arguments keep their original identities and bindings.
Importing a symbol shares its identity, and because functions and variables share a namespace, a let of an imported function name captures it.

;;; lib/schedules.sx
(defun clamp01 (x) (max 0 (min 1 x)))
(defmacro ramp (from to)
  `(lambda (step) (clamp01 (/ (- step ,from) (- ,to ,from)))))

;;; runs/a.sx
(require "../lib/schedules.sx" ramp)    ; not clamp01
(defvar warmup (ramp 0 100))            ; (warmup 50) is 0.5

This is how Common Lisp packages behave: a symbol the library wrote is the library's symbol, and the caller's same-spelled one is another symbol. Capture happens between symbols that are the same: in a macro and its caller in one file, and through a name the caller imported from the library. A let of it in a library macro binds the library's it, which the caller's (+ it 1) cannot see; the fix of the E-EVAL-003 says so. The prelude is the one difference: Common Lisp shares first between packages, here each module's first is its own symbol that falls back to the prelude, so a caller's flet of first does not reach the macro's. Intentional capture across a module boundary uses intern with the caller's environment:

(defmacro aif (test then else &environment caller)
  (let ((it (intern "it" caller)))
    `(let ((,it ,test)) (if ,it ,then ,else))))

(aif 7 (+ it 1) 0) ; 8

Generated declarations

Definitions register when their form executes. A macro can generate defun, defmacro, defmodel, defoptimizer, defvar, defknobs and defvariant, and another file can import the top-level names it produced.

(defmacro define-width (name width)
  `(defvariant ,name (width ,width)))

(define-width wide 512)
(defknobs (width 128))

Each form in a top-level progn resolves its symbols after the forms before it ran, exactly like separate top-level forms.
Variants, generated or not, come before the first defknobs; see knobs.
Local definitions stay local; imported names are not re-exported.
Duplicate definitions and conflicting imports fail at the second binding, E-EVAL-036.

Names in the model and the optimizer

Gensym bindings never name persistent things. A model bound with a gensym gets no path segment; a parameter declared with a gensym gets a stable param.N, and a gensym optimizer state a stable state.N, in declaration order. Give persistent parts ordinary names or (named "segment" ...); see models and optimizers.

Diagnostics through macros

Generated code keeps the macro call, the definition and the template origin. The notes survive into generated closures and show up when a later trace fails, so an error in expanded code still points at the call you wrote. When the error is raised in library code the expansion reached, a helper of the library or of sexpgpu/nn, the primary span moves to the macro call and the library line becomes the note "raised by this expression", as it does for a model application; the trace names the helper.

A worked example

A macro generates a model constructor, aif from above captures it on purpose, and a momentum optimizer step keeps its state under a gensym:

(defknobs (degree 3 "number of polynomial coefficients"))

(defmacro define-polynomial (name)
  `(defmodel ,name (degree &key (policy :high))
     (defparam coefficients (zeros [degree]) :numerics policy)
     (lambda (x)
       (let ((result (zeros-like x)))
         (dotimes (i degree result)
           (setq result
                 (+ result
                    (* (sum (slice coefficients 0 i (+ i 1))) (pow x i)))))))))

(define-polynomial polynomial)
(defvar model (aif degree (polynomial it) (error "degree is required")))

(defmacro momentum-step (p g rate)
  (with-gensyms (velocity next)
    `(progn
       (defstate ,velocity (zeros-like ,p))
       (let ((,next (+ (* 0.8 ,velocity) ,g)))
         (optimizer-update (- ,p (* ,rate ,next)) ',velocity ,next)))))