Mathematical shaders as live Lisp objects
Source mathematics and machine form are different views #S4H8DR
Luv has two shader languages with deliberately different jobs. The literal
language in hal/vulkan/spir-v/instructions.lisp names every result id and every SPIR-V instruction. It
is a good machine form, an assembler input, and a debugging view. It is a bad
place to state that illumination is ambient light plus a sun contribution or
that fog linearly combines lit albedo with sky colour.
The expression language in hal/shader/language.lisp states those relationships directly. It
does not replace either backend. The luv/shader system owns the typed graph
and open lowering protocol. Sibling luv/spir-v lowers it into the existing
spir-v-module, spir-v-function-definition, spir-v-basic-block, and
instruction objects, after which lower-spir-v and assemble emit binary;
luv/msl lowers the same graph directly into a structured MSL document.
typed shader specification
declarations + let* bindings + expressions
/ \
lower :spir-v / \ lower MSL target
v v
shader-lowering -- provenance maps msl-document
|
v
spir-v-module -- function -- basic block -- CLOS instruction occurrences
|
| lower-spir-v / assemble
v
SPIR-V wordsThis separation preserves the assembler as an inspectable truth rather than hiding it behind a string-generating compiler. It also makes the source and machine views objects that McCLIM can present and relate.
Mentioned in: Storage buffers and bit fields enter the shader language
The block pipeline is the first real client #B9M2FX
The vertex and fragment shaders used by the playable block world live in
luvcraft/shaders.lisp. Its complete material calculation is ordinary
Lisp-shaped source:
The vertex method is a CLOS graph too. Its built-in position output and
shared frame uniform block are first-class declaration objects; member
references lower to AccessChain and Load instructions with retained
provenance. Uniform blocks currently admit aligned vec4 lanes only, matching
the renderer's actual camera ABI instead of pretending to provide general
std140 packing. The old handwritten block vertex module has been deleted.
The public renderer API did not change. block-world-fragment-module still
returns a structured SPIR-V module and block-world-fragment-shader still
returns assembled words. block-world-fragment-lowering exposes the richer
result when a tool wants provenance.
The current material uses chromatic directional light: cool open-sky fill on
faces turned away from the sun and warm light on sun-facing faces. This keeps
snow faces legible against one another instead of flattening the foreground
into scalar white illumination. It still preserves the atlas, mesh-baked
ambient occlusion, and fog. Both block shaders are checked by spirv-val
through make shader-validate.
Mentioned in: One arithmetic medium, many clients
Methods are live definition identity #M3D7QK
shader-specification-for is an ordinary generic function specialized on a
stable render role and stage. define-shader-method parses the typed graph
once per method definition and returns that durable object cheaply. Evaluating
the same role/stage method again uses normal CLOS replacement identity instead
of accumulating anonymous versions.
A narrow MOP dependent observes add-method and remove-method. Its callback
only increments a revision under a mutex; it never calls the generic function,
compiles, or enters Vulkan while the generic function is being mutated. The
block-world frame callback notices a pending vertex or fragment revision and
builds a complete pair of specifications, lowerings, SPIR-V modules, shader
modules, and one render pipeline on the renderer's owning thread.
Only a fully created candidate is published. A failed edit records a diagnostic and leaves the last installed pipeline rendering. A successful swap publishes the new artifact before retiring the old pipeline and both shader modules through the backend's submission-aware destruction path. Stopping the demo unsubscribes all method dependents and destroys the installed artifacts.
The initial roles are :block-surface/:vertex,
:block-surface/:fragment, and :block-crosshair/:fragment. These are render roles, not block kinds: the
current chunk mesh combines block materials into one atlas-backed draw, so
pretending there is already one GPU pipeline per block kind would encode a
false ownership model.
Mentioned in: What the presentation stack became
Expressions, declarations, and bindings are objects #C6E3LA
A shader-specification owns interface declarations, resources, ordered
shader-binding objects, and output assignments. Every literal, reference,
and call is a shader-expression with:
- an inferred
shader-type; - its original source form;
- expression-specific operands or target;
- and, for a
let*value, its lexical result name.
The parser understands float scalars, vec2=/=vec3=/=vec4, component
swizzles, arithmetic, dot, sample, mix, and the GLSL.std.450 family
currently needed by the block world: min, max, clamp, smoothstep,
step, abs, sqrt, expt, and normalize. Texture resources now include
ordinary colour textures and depth textures; a depth-texture sample still
lowers through the same image-sampling path and becomes scalar shadow math only
after source-level swizzling. Type inference rejects an unknown name, an
out-of-range swizzle, an invalid product, a mismatched output, or a malformed
resource declaration as a shader-language-error carrying the source form,
reason, and details.
Declarations and expressions may opt into the backend-neutral quantity
experiment #9P358P with :quantity, :dimension, :unit, and :affine-p.
That metadata is separate from shader-type: ordinary arithmetic derives a
semantic specification while valid annotated and unannotated graphs still
lower to the same SPIR-V. quantity constructs semantic constants,
assume-quantity loudly marks raw external meaning, and interpret names only
a compatible already-derived result. Conversely, representation visibly
exposes the raw machine value of checked source for an opaque ABI calculation;
it requires a semantic operand and stops quantity propagation until a later
explicit assumption. None of these boundaries emits an instruction.
#VC7VFY and completed work #SQR8KR make component structure first-class too.
A homogeneous vector can publish named axes, while :components describes a
heterogeneous representation such as vec3(xy: texture point, z: occlusion).
Only exact declared swizzles recover quantities; undeclared projections and
arithmetic on the packed whole fail. Texture resources can likewise publish
sample-channel layouts, so RGB and alpha acquire meaning at the sample rather
than through a later cast. The production luvcraft shaders carry these
meanings through shadow filtering, lighting, sky, fog, and final colour without
changing the representation or emitted module.
let* is semantic. Lowering evaluates bindings in source order, interns
types and constants deterministically, preserves useful lexical names in
result ids, and emits instruction instances into a CLOS basic block. A
shader-lowering retains two EQ maps:
- expression object to the instruction occurrences it caused; and
- instruction occurrence to the source expressions that explain it.
Those maps are intentionally occurrence-based. “This multiply in this basic
block came from this expression object” is more useful for a live tool than a
global table saying that the f-mul opcode exists.
Mentioned in: Generated MSL remains a view of semantic objects, The stylesheet as Lisp definitions
Shader functions are graphs; source abstractions are syntax #RO74NL
Slug exposed a missing layer in the language. The core operators were typed
objects and a complete shader could have top-level let* bindings, but the
only reusable source vocabulary was define-shader-abstraction. Its method
ran as host Lisp before parsing and returned a newly constructed S-expression.
That is appropriate for a syntactic transformation such as expanding a
host-known sample stencil. It is a poor representation of an ordinary
mathematical function: the author has to manage quotation, invented binding
names, and source construction outside the typed graph.
define-shader-function now fills that gap. Its body is ordinary shader
source, not Lisp that returns shader source:
At each call site the parser first parses the actual arguments, then parses the
function's lexical let* against those typed values. The result is an
inspectable shader-function-call retaining its live definition, argument
expressions, local binding objects, typed result, quantity meaning, and source
call. Local values are hoisted in deterministic dependency order for the
single entry block. SPIR-V and direct MSL lowering inline the result graph and
associate its emitted occurrences with both the inner expressions and the call.
The definition macro is deliberately thin: it records the unevaluated body at
an EQL-specialized definition coordinate. Re-evaluating that coordinate
changes fresh parses and increments the shared shader-source revision; already
parsed graphs remain durable objects. A live SLY check redefined one function
from addition to subtraction: the existing call still exposed +, a fresh
parse exposed -, and the source revision advanced. Names can also migrate
from an old source abstraction to a typed function without leaving the old
EQL expansion methods active. This is definition syntax in the same sense as
defun, not a request for the author to program a source rewriter.
At that milestone function definition did not itself add runtime control flow. The subsequent data-driven Slug renderer added typed texel loads, integer data, shared conditionals, and a structured counted fold as separate language features. That separation remains useful: function source and application do not secretly own looping, while a function body can use the same lexical and control-flow vocabulary as any other expression body.
The separation above is between ordinary language and syntactic rewriting,
not between a permanently small “arithmetic language” and a richer shader
language. let*, reusable calls, conditionals, and well-structured iteration
are ordinary program structure. A CPU realization and a shader realization
may choose different representations and effects, but should not acquire two
unrelated accounts of function source, lexical scope, arity, or recursion.
Mentioned in: One fixed contour proves luv's pixel-stage mathematics
Referenced from code: Define a reusable typed shader expression with ordinary source syntax. The body is parsed at each call site, so argument types and quantity meanings
flow through the same operator protocol as handwritten shader expressions.
LET* is lexical inside the function. The definition macro records source; its
body does not execute as Lisp and does not return generated S-expressions.
#RO74NLdefmacro define-shader-function language.lisp:1938 ↗
One source language can have target-owned effects #8RQOF0
The extracted arithmetic frontend already owns literals, references, calls,
lexical bindings, quantity boundaries, unit conversion, and a checked
arithmetic-function-definition. Shader expressions subclass those common
objects. The first define-shader-function implementation nevertheless
introduced a second function-definition protocol: an arithmetic definition is
a graph parsed once from declared parameters, while a shader definition stores
an unparsed body and specializes it against represented arguments at each call.
The distinction in checking time is real; the duplicate meaning of “function”
is not desirable.
The intended convergence is one inspectable function-source and application protocol with target realizations. Common definitions retain semantic parameter contracts and can be compiled to ordinary Lisp. A shader realization adds concrete scalar/vector types, resource effects, stage legality, inlining, SSA names, and instruction provenance. Texture loads, derivatives, and output assignments remain shader operations, but they live in the same extensible body language instead of forcing a separate account of lexical structure. A future CPU realization may define useful meanings for some of those operations without making them part of the portable core.
Structured control flow belongs at this shared source level too. Slug's full pixel calculation is naturally a counted fold over a band of curve indexes: four accumulators advance together, conditionals admit eligible roots, and a sorted geometric bound terminates the fold early. SPIR-V realizes that as structured blocks and phi values; MSL realizes it as an ordinary loop. The source object should say what the loop computes, not encode either target's control-flow machinery.
Mentioned in: The reference's shape, with its constants as knobs, Full Slug is a structured fold over indexed outline data
DONE Share function application across arithmetic and shader realizations #YZS04W
Intent: make one portable arithmetic definition callable both by compiled Lisp and from a shader body, sharing source identity, parameter normalization, arity, lexical bindings, recursion rejection, and application provenance.
Evidence needed: move Slug's two scalar root-eligibility functions to the
common definition protocol, execute them through the Lisp realization, consume
the same definitions inside slug-axis-contribution, and retain identical
SPIR-V, MSL, and Metal proof pixels. Keep resource operations and represented
shader typing target-owned.
Done when: there is one source definition for each eligibility calculation, both CPU and shader call graphs contain an inspectable application object, live redefinition is observed by fresh realizations, and the old independent shader definition is unnecessary for these portable functions.
The resulting arithmetic-function-source and
arithmetic-function-call objects now own this shared meaning. Ordinary Lisp
lowers applications to lexical let* forms; shader parsing specializes the
same source against represented values and emits the existing typed call
provenance. The Slug root-eligibility helpers are
define-lisp-arithmetic-function definitions used directly by both the CPU
table oracle and the pixel shader. Publishing shared source also retires a
stale shader-only definition of the same name and advances the live shader
source revision.
The first shared control-flow node follows the same rule. counted-fold
carries one arbitrary state value through a runtime count. The Lisp target
emits dotimes, direct Metal emits a local and for loop, and SPIR-V emits
header/body/continue/merge blocks with OpPhi and OpLoopMerge. The generated
module passes spirv-val and the generated Metal 4 source compiles with the
Metal compiler. One vector state is enough to carry Slug's four coverage and
weight accumulators without making multiple assignment part of the portable
core.
The real Slug traversal established the lexical rule at the next level of
composition: a shared or shader-specialized reusable function called from a
fold may introduce its own let* bindings, and those values are defined within
the current iteration. MSL emits readable locals inside the for body;
SPIR-V emits values in the loop body and restricts load reuse to the owning
basic block. This is one account of function application and scope across the
arithmetic Lisp, MSL, and SPIR-V realizations, not a special Slug lowering.
Mentioned in: Teach the shared language Slug's structured band fold
Semantic maps are not merely matrix-shaped values #4XAF9Z
The first matrix feature is deliberately a map protocol rather than a generic
mat4 shader type. A shader-map-definition names a domain type and quantity
and the behavior of application. A projective definition separately names
its homogeneous representation and the type, semantic layout, and remap of
its sampling projection. Dense coefficients remain ordinary represented
shader values. This separates “four rows happen to be stored here” from
“this operation accepts an affine world point in cells and maps it into light
space.” The broader quantity argument and mp-units/Moppe comparison lives in
#SM4DFB.
The production :world-to-light definition has two explicit operations.
Calling
constructs a shader-map-application denoting the homogeneous vec4 after
four row dot products. The shadow pass uses that value directly.
project-sample consumes the same application, divides by w, applies the
declared remap, and returns a virtual shader-map-projection: xy is affine
:shadow-uv and z is affine :shadow-depth. Lowering caches represented
components, so selecting both fields performs the map once without constructing
an intermediate sample vector or homogeneous vector. Both paths preserve the
exact SPIR-V previously emitted by their two handwritten arithmetic spellings.
Map names use an open EQL-specialized lookup protocol, and lowering dispatches
on definition class. An unknown map, incompatible domain, non-affine point,
wrong row count, semantic row, or sampling projection of an unrelated raw
vec4 is a source error. Adding linear, affine, normal, or other map families
therefore means adding semantic definitions and lowering methods, not weakening
failed cases to unannotated values.
McCLIM presents the relationship #P7V5CK
mcclim/shader-lab.lisp defines a real application frame on luv's McCLIM
backend. Its material header renders the generated atlas as selectable block
cards and offers live definition tabs for block geometry, block surface, and
crosshair methods. Clicking a tab recompiles the current method while retaining the
last good lowering on error. Its left pane renders declarations and nested mathematical
expressions. Its right pane renders CLOS functions, basic blocks, and literal
SSA instruction forms. Materials, definitions, expressions, instructions,
and blocks are distinct McCLIM presentation types.
Selecting an expression highlights its associated instructions. Selecting an instruction keeps that machine occurrence selected while following the reverse provenance map to highlight its source expression. The detail pane reports the source form, inferred type, semantic meaning, and association count. Packed declarations and samples display lane-to-quantity maps, while construction, assumption, interpretation, and raw representation remain distinct selectable expression nodes. Homogeneous projective applications and their sampling projections are selectable nodes too: the former renders its map name, semantic point, and dense represented rows; the latter renders the explicit projection relationship. The presentation objects are the compiler objects themselves; there is no parallel UI tree whose identities can drift.
In the durable SLY image:
The workbench runs run-frame-top-level in its own McCLIM process; SDL
translates native events, while the frame command loop resolves presentation
gestures. Refresh and health checks queue an event through that ordinary
command loop and require a bounded acknowledgement. The structured report
also checks frame state, process liveness, mirror count, canvas state, and
mirror/event-handler ownership. A timeout returns :unresponsive with a
best-effort frame-thread backtrace rather than silently assuming a live thread
means a healthy window. Startup is bounded too. Closing the helper interrupts
the process with frame-exit and lets the usual disown/native teardown path
release its method dependents and native canvas.
The standalone workbench and luvcraft cannot presently be open at the same time: the Cocoa host deliberately supports one durable native canvas. Runtime pipeline status is already an optional definition-entry input for a future texture-backed workbench inside luvcraft, but the current standalone window primarily reports source compilation state. Refusing unsupported second windows is part of keeping the owning McCLIM frame responsive.
What this does not decide #F2Q9WU
This is a coherent first language, not a commitment to grow a second Common Lisp inside Common Lisp. Control flow, user functions, general matrix values and composition, integer and boolean types, storage buffers, derivative operations, and stage-specific validation should arrive only with shaders that require them. The projective-map slice exists because two production shaders already supply that requirement.
The useful constraints for the next experiments are already visible:
- source expressions remain CLOS objects with source provenance;
- lexical Lisp order remains observable in the lowered block;
- SPIR-V instructions and basic blocks remain the literal backend objects;
- lowering errors name the mathematical source form;
- tools consume the same expression and instruction occurrences as the compiler; and
- method notifications are revisions, while GPU publication stays on the renderer's ownership boundary;
- a failed edit preserves the last known good pipeline and records why; and
- a shader change is not accepted on pretty source alone: assemble it, validate it, and run the real renderer.
(generic-function name specialized-lambda-list options &body body)Define a shader-producing method with ordinary DEFMETHOD identity. Calling the method reparses its small source form so changes to source-level abstractions participate in live rebuilding. Method replacement remains the role/stage identity watched by the MOP; abstraction revisions are tracked separately by live…
(role stage)Return the current durable shader specification for ROLE and STAGE.
Select and reorder vector components by a designator such as :XYZ or :RGB.
Multiplication and scalar scaling.
Addition over compatible quantities.
The inner product of two vectors.
Interpolate compatible quantities by a scalar amount.
One inspectable environment light, packed into raw per-frame GPU lanes.
Sample a two-dimensional texture through a sampler at a UV coordinate.
(name parameters &body body)Define a reusable typed shader expression with ordinary source syntax. The body is parsed at each call site, so argument types and quantity meanings flow through the same operator protocol as handwritten shader expressions. LET* is lexical inside the function. The definition macro records source; its body does not…
((name symbol) (type (eql 'arithmetic-operator)))Logical conjunction of tests and raw truth values.
(name)Return the live typed shader function named by NAME, or NIL.
(name parameters body)Apply a named projective map to a semantic affine point.
(&key specification specifications pipelines
(server-path '(:luv-gpu))
(startup-timeout 10.0)
(title "Luvcraft material and shader workbench"))Open a live-definition, expression, and SSA workbench on luv's backend. PIPELINES may contain a running demo's live shader artifacts; their installed revision and last diagnostic then appear alongside the source definition.
(frame &key (timeout 1.0))(frame &key (timeout 0.5))(frame)Intent: let one real shader calculation test whether dimensions, named quantity spaces, tensor order, and affine points belong in a reusable arithmetic graph. Keep SPIR-V resources and lowering shader-specific; keep semantic arithmetic in a backend-neutral layer that a later scalar or SIMD kernel could also consume.…
A GPU vector is a representation shape, not necessarily one mathematical vector. Similar vector registers can have different semantic structure: The second and third values are semantic products: operational packing puts several related values in one register without claiming that arbitrary lane combinations form…
Intent: make homogeneous vector quantities and heterogeneous packed GPU tuples different semantic structures. Let a whole vector already carry quantity meaning, let exact swizzles recover explicitly declared component meanings, and make every undeclared projection or packed-whole arithmetic operation fail at parse…
A tensor quantity and a map answer related but different questions. A tensor quantity says that a scalar, vector, or higher-order representation carries a quantity specification. A map says what source space it accepts, what target space it produces, and what kind of application is lawful. Four vec4 rows alone…
Slug exposed a missing layer in the language. The core operators were typed objects and a complete shader could have top-level let* bindings, but the only reusable source vocabulary was define-shader-abstraction. Its method ran as host Lisp before parsing and returned a newly constructed S-expression. That is…
=> (:responsive 1 :open)