luv

Workshop wiki

mathematical-shaders.org

Mathematical shaders as live Lisp objects

Source mathematics and machine form are different views #S4H8DR

Luv has two shader languages with deliberately different jobs. The literal language in hal/vulkan/spir-v/instructions.lisp names every result id and every SPIR-V instruction. It is a good machine form, an assembler input, and a debugging view. It is a bad place to state that illumination is ambient light plus a sun contribution or that fog linearly combines lit albedo with sky colour.

The expression language in hal/shader/language.lisp states those relationships directly. It does not replace either backend. The luv/shader system owns the typed graph and open lowering protocol. Sibling luv/spir-v lowers it into the existing spir-v-module, spir-v-function-definition, spir-v-basic-block, and instruction objects, after which lower-spir-v and assemble emit binary; luv/msl lowers the same graph directly into a structured MSL document.

typed shader specification
  declarations + let* bindings + expressions
                 /                           \
   lower :spir-v /                             \ lower MSL target
               v                               v
shader-lowering -- provenance maps         msl-document
               |
               v
spir-v-module -- function -- basic block -- CLOS instruction occurrences
               |
               | lower-spir-v / assemble
               v
           SPIR-V words

This separation preserves the assembler as an inspectable truth rather than hiding it behind a string-generating compiler. It also makes the source and machine views objects that McCLIM can present and relate.

The block pipeline is the first real client #B9M2FX

The vertex and fragment shaders used by the playable block world live in luvcraft/shaders.lisp. Its complete material calculation is ordinary Lisp-shaped source:

define-shader-methodshader-specification-forblock-world-fragment-specification
role
eql:block-surface
stage
eql:fragment
:stage:fragment:inputs
uv-shade-input:vec3:location0
normal-input:vec3:location1
fog-input:vec4:location2
:outputs
color-output:vec4:location0
:resources
block-atlas:texture-2d:set0:binding0
block-sampler:sampler:set0:binding1
let*
uv-shadeuv-shade-input
uv
swizzleuv-shade:xy
ao
swizzleuv-shade:z
normalnormal-input
sun
vec30.300.860.40
hemisphere
*
+
dotnormalsun
1.0
0.5
shade-light
vec30.480.580.76
sun-light
vec31.020.960.82
directional-light
mixshade-lightsun-lighthemisphere
light
*directional-lightao
albedo
swizzle
sampleblock-atlasblock-sampleruv
:rgb
fog-statefog-input
sky
swizzlefog-state:rgb
fog
swizzlefog-state:w
lit
*albedolight
fogged
mixskylitfog
rgba
vec4fogged1.0
set-outputcolor-outputrgba

The vertex method is a CLOS graph too. Its built-in position output and shared frame uniform block are first-class declaration objects; member references lower to AccessChain and Load instructions with retained provenance. Uniform blocks currently admit aligned vec4 lanes only, matching the renderer's actual camera ABI instead of pretending to provide general std140 packing. The old handwritten block vertex module has been deleted.

The public renderer API did not change. block-world-fragment-module still returns a structured SPIR-V module and block-world-fragment-shader still returns assembled words. block-world-fragment-lowering exposes the richer result when a tool wants provenance.

The current material uses chromatic directional light: cool open-sky fill on faces turned away from the sun and warm light on sun-facing faces. This keeps snow faces legible against one another instead of flattening the foreground into scalar white illumination. It still preserves the atlas, mesh-baked ambient occlusion, and fog. Both block shaders are checked by spirv-val through make shader-validate.

Methods are live definition identity #M3D7QK

shader-specification-for is an ordinary generic function specialized on a stable render role and stage. define-shader-method parses the typed graph once per method definition and returns that durable object cheaply. Evaluating the same role/stage method again uses normal CLOS replacement identity instead of accumulating anonymous versions.

A narrow MOP dependent observes add-method and remove-method. Its callback only increments a revision under a mutex; it never calls the generic function, compiles, or enters Vulkan while the generic function is being mutated. The block-world frame callback notices a pending vertex or fragment revision and builds a complete pair of specifications, lowerings, SPIR-V modules, shader modules, and one render pipeline on the renderer's owning thread.

Only a fully created candidate is published. A failed edit records a diagnostic and leaves the last installed pipeline rendering. A successful swap publishes the new artifact before retiring the old pipeline and both shader modules through the backend's submission-aware destruction path. Stopping the demo unsubscribes all method dependents and destroys the installed artifacts.

The initial roles are :block-surface/:vertex, :block-surface/:fragment, and :block-crosshair/:fragment. These are render roles, not block kinds: the current chunk mesh combines block materials into one atlas-backed draw, so pretending there is already one GPU pipeline per block kind would encode a false ownership model.

Expressions, declarations, and bindings are objects #C6E3LA

A shader-specification owns interface declarations, resources, ordered shader-binding objects, and output assignments. Every literal, reference, and call is a shader-expression with:

The parser understands float scalars, vec2=/=vec3=/=vec4, component swizzles, arithmetic, dot, sample, mix, and the GLSL.std.450 family currently needed by the block world: min, max, clamp, smoothstep, step, abs, sqrt, expt, and normalize. Texture resources now include ordinary colour textures and depth textures; a depth-texture sample still lowers through the same image-sampling path and becomes scalar shadow math only after source-level swizzling. Type inference rejects an unknown name, an out-of-range swizzle, an invalid product, a mismatched output, or a malformed resource declaration as a shader-language-error carrying the source form, reason, and details.

Declarations and expressions may opt into the backend-neutral quantity experiment #9P358P with :quantity, :dimension, :unit, and :affine-p. That metadata is separate from shader-type: ordinary arithmetic derives a semantic specification while valid annotated and unannotated graphs still lower to the same SPIR-V. quantity constructs semantic constants, assume-quantity loudly marks raw external meaning, and interpret names only a compatible already-derived result. Conversely, representation visibly exposes the raw machine value of checked source for an opaque ABI calculation; it requires a semantic operand and stops quantity propagation until a later explicit assumption. None of these boundaries emits an instruction.

#VC7VFY and completed work #SQR8KR make component structure first-class too. A homogeneous vector can publish named axes, while :components describes a heterogeneous representation such as vec3(xy: texture point, z: occlusion). Only exact declared swizzles recover quantities; undeclared projections and arithmetic on the packed whole fail. Texture resources can likewise publish sample-channel layouts, so RGB and alpha acquire meaning at the sample rather than through a later cast. The production luvcraft shaders carry these meanings through shadow filtering, lighting, sky, fog, and final colour without changing the representation or emitted module.

let* is semantic. Lowering evaluates bindings in source order, interns types and constants deterministically, preserves useful lexical names in result ids, and emits instruction instances into a CLOS basic block. A shader-lowering retains two EQ maps:

Those maps are intentionally occurrence-based. “This multiply in this basic block came from this expression object” is more useful for a live tool than a global table saying that the f-mul opcode exists.

Shader functions are graphs; source abstractions are syntax #RO74NL

Slug exposed a missing layer in the language. The core operators were typed objects and a complete shader could have top-level let* bindings, but the only reusable source vocabulary was define-shader-abstraction. Its method ran as host Lisp before parsing and returned a newly constructed S-expression. That is appropriate for a syntactic transformation such as expanding a host-known sample stencil. It is a poor representation of an ordinary mathematical function: the author has to manage quotation, invented binding names, and source construction outside the typed graph.

define-shader-function now fills that gap. Its body is ordinary shader source, not Lisp that returns shader source:

define-shader-functionweighted-square
valuescale
let*
shifted
+value0.25
scaled
*shiftedscale
scaled

At each call site the parser first parses the actual arguments, then parses the function's lexical let* against those typed values. The result is an inspectable shader-function-call retaining its live definition, argument expressions, local binding objects, typed result, quantity meaning, and source call. Local values are hoisted in deterministic dependency order for the single entry block. SPIR-V and direct MSL lowering inline the result graph and associate its emitted occurrences with both the inner expressions and the call.

The definition macro is deliberately thin: it records the unevaluated body at an EQL-specialized definition coordinate. Re-evaluating that coordinate changes fresh parses and increments the shared shader-source revision; already parsed graphs remain durable objects. A live SLY check redefined one function from addition to subtraction: the existing call still exposed +, a fresh parse exposed -, and the source revision advanced. Names can also migrate from an old source abstraction to a typed function without leaving the old EQL expansion methods active. This is definition syntax in the same sense as defun, not a request for the author to program a source rewriter.

At that milestone function definition did not itself add runtime control flow. The subsequent data-driven Slug renderer added typed texel loads, integer data, shared conditionals, and a structured counted fold as separate language features. That separation remains useful: function source and application do not secretly own looping, while a function body can use the same lexical and control-flow vocabulary as any other expression body.

The separation above is between ordinary language and syntactic rewriting, not between a permanently small “arithmetic language” and a richer shader language. let*, reusable calls, conditionals, and well-structured iteration are ordinary program structure. A CPU realization and a shader realization may choose different representations and effects, but should not acquire two unrelated accounts of function source, lexical scope, arity, or recursion.

defmacro define-shader-function language.lisp:1938
defmacrodefine-shader-function
nameparameters&bodybody

Define a reusable typed shader expression with ordinary source syntax.

The body is parsed at each call site, so argument types and quantity meanings flow through the same operator protocol as handwritten shader expressions. LET* is lexical inside the function. The definition macro records source; its body does not execute as Lisp and does not return generated S-expressions. #RO74NL

let*
function-name
gensym"FUNCTION-NAME"
documentation
and
stringp
firstbody
firstbody
forms
ifdocumentation
restbody
body
`
progn
defmethodshader-function-definition-for
,function-name
eql',name
declare
ignore,function-name
load-time-value
make-shader-function-definition',name',parameters',forms
,@
whendocumentation`
setf
documentation',name'shader-function
,documentation
',name

One source language can have target-owned effects #8RQOF0

The extracted arithmetic frontend already owns literals, references, calls, lexical bindings, quantity boundaries, unit conversion, and a checked arithmetic-function-definition. Shader expressions subclass those common objects. The first define-shader-function implementation nevertheless introduced a second function-definition protocol: an arithmetic definition is a graph parsed once from declared parameters, while a shader definition stores an unparsed body and specializes it against represented arguments at each call. The distinction in checking time is real; the duplicate meaning of “function” is not desirable.

The intended convergence is one inspectable function-source and application protocol with target realizations. Common definitions retain semantic parameter contracts and can be compiled to ordinary Lisp. A shader realization adds concrete scalar/vector types, resource effects, stage legality, inlining, SSA names, and instruction provenance. Texture loads, derivatives, and output assignments remain shader operations, but they live in the same extensible body language instead of forcing a separate account of lexical structure. A future CPU realization may define useful meanings for some of those operations without making them part of the portable core.

Structured control flow belongs at this shared source level too. Slug's full pixel calculation is naturally a counted fold over a band of curve indexes: four accumulators advance together, conditionals admit eligible roots, and a sorted geometric bound terminates the fold early. SPIR-V realizes that as structured blocks and phi values; MSL realizes it as an ordinary loop. The source object should say what the loop computes, not encode either target's control-flow machinery.

DONE Share function application across arithmetic and shader realizations #YZS04W

Intent: make one portable arithmetic definition callable both by compiled Lisp and from a shader body, sharing source identity, parameter normalization, arity, lexical bindings, recursion rejection, and application provenance.

Evidence needed: move Slug's two scalar root-eligibility functions to the common definition protocol, execute them through the Lisp realization, consume the same definitions inside slug-axis-contribution, and retain identical SPIR-V, MSL, and Metal proof pixels. Keep resource operations and represented shader typing target-owned.

Done when: there is one source definition for each eligibility calculation, both CPU and shader call graphs contain an inspectable application object, live redefinition is observed by fresh realizations, and the old independent shader definition is unnecessary for these portable functions.

The resulting arithmetic-function-source and arithmetic-function-call objects now own this shared meaning. Ordinary Lisp lowers applications to lexical let* forms; shader parsing specializes the same source against represented values and emits the existing typed call provenance. The Slug root-eligibility helpers are define-lisp-arithmetic-function definitions used directly by both the CPU table oracle and the pixel shader. Publishing shared source also retires a stale shader-only definition of the same name and advances the live shader source revision.

The first shared control-flow node follows the same rule. counted-fold carries one arbitrary state value through a runtime count. The Lisp target emits dotimes, direct Metal emits a local and for loop, and SPIR-V emits header/body/continue/merge blocks with OpPhi and OpLoopMerge. The generated module passes spirv-val and the generated Metal 4 source compiles with the Metal compiler. One vector state is enough to carry Slug's four coverage and weight accumulators without making multiple assignment part of the portable core.

The real Slug traversal established the lexical rule at the next level of composition: a shared or shader-specialized reusable function called from a fold may introduce its own let* bindings, and those values are defined within the current iteration. MSL emits readable locals inside the for body; SPIR-V emits values in the loop body and restricts load reuse to the owning basic block. This is one account of function application and scope across the arithmetic Lisp, MSL, and SPIR-V realizations, not a special Slug lowering.

Semantic maps are not merely matrix-shaped values #4XAF9Z

The first matrix feature is deliberately a map protocol rather than a generic mat4 shader type. A shader-map-definition names a domain type and quantity and the behavior of application. A projective definition separately names its homogeneous representation and the type, semantic layout, and remap of its sampling projection. Dense coefficients remain ordinary represented shader values. This separates “four rows happen to be stored here” from “this operation accepts an affine world point in cells and maps it into light space.” The broader quantity argument and mp-units/Moppe comparison lives in #SM4DFB.

The production :world-to-light definition has two explicit operations. Calling

project-point:world-to-lightworld-positionshadow-row-xshadow-row-yshadow-row-zshadow-row-w

constructs a shader-map-application denoting the homogeneous vec4 after four row dot products. The shadow pass uses that value directly. project-sample consumes the same application, divides by w, applies the declared remap, and returns a virtual shader-map-projection: xy is affine :shadow-uv and z is affine :shadow-depth. Lowering caches represented components, so selecting both fields performs the map once without constructing an intermediate sample vector or homogeneous vector. Both paths preserve the exact SPIR-V previously emitted by their two handwritten arithmetic spellings.

Map names use an open EQL-specialized lookup protocol, and lowering dispatches on definition class. An unknown map, incompatible domain, non-affine point, wrong row count, semantic row, or sampling projection of an unrelated raw vec4 is a source error. Adding linear, affine, normal, or other map families therefore means adding semantic definitions and lowering methods, not weakening failed cases to unannotated values.

McCLIM presents the relationship #P7V5CK

mcclim/shader-lab.lisp defines a real application frame on luv's McCLIM backend. Its material header renders the generated atlas as selectable block cards and offers live definition tabs for block geometry, block surface, and crosshair methods. Clicking a tab recompiles the current method while retaining the last good lowering on error. Its left pane renders declarations and nested mathematical expressions. Its right pane renders CLOS functions, basic blocks, and literal SSA instruction forms. Materials, definitions, expressions, instructions, and blocks are distinct McCLIM presentation types.

Selecting an expression highlights its associated instructions. Selecting an instruction keeps that machine occurrence selected while following the reverse provenance map to highlight its source expression. The detail pane reports the source form, inferred type, semantic meaning, and association count. Packed declarations and samples display lane-to-quantity maps, while construction, assumption, interpretation, and raw representation remain distinct selectable expression nodes. Homogeneous projective applications and their sampling projections are selectable nodes too: the former renders its map name, semantic point, and dense represented rows; the latter renders the explicit projection relationship. The presentation objects are the compiler objects themselves; there is no parallel UI tree whose identities can drift.

In the durable SLY image:

asdf:load-system:mcluv
defparameter*shader-lab*
multiple-value-bind
statusreport
liststatus
mcluv:shader-lab-health-report-mirror-countreport
mcluv:shader-lab-health-report-canvas-statereport

=> (:responsive 1 :open)

The workbench runs run-frame-top-level in its own McCLIM process; SDL translates native events, while the frame command loop resolves presentation gestures. Refresh and health checks queue an event through that ordinary command loop and require a bounded acknowledgement. The structured report also checks frame state, process liveness, mirror count, canvas state, and mirror/event-handler ownership. A timeout returns :unresponsive with a best-effort frame-thread backtrace rather than silently assuming a live thread means a healthy window. Startup is bounded too. Closing the helper interrupts the process with frame-exit and lets the usual disown/native teardown path release its method dependents and native canvas.

The standalone workbench and luvcraft cannot presently be open at the same time: the Cocoa host deliberately supports one durable native canvas. Runtime pipeline status is already an optional definition-entry input for a future texture-backed workbench inside luvcraft, but the current standalone window primarily reports source compilation state. Refusing unsupported second windows is part of keeping the owning McCLIM frame responsive.

What this does not decide #F2Q9WU

This is a coherent first language, not a commitment to grow a second Common Lisp inside Common Lisp. Control flow, user functions, general matrix values and composition, integer and boolean types, storage buffers, derivative operations, and stage-specific validation should arrive only with shaders that require them. The projective-map slice exists because two production shaders already supply that requirement.

The useful constraints for the next experiments are already visible: