luv

Workshop wiki

temporal-antialiasing.org

Temporal reconstruction in the Luft atelier

Luft's temporal antialiasing is deliberately a renderer experiment rather than a new engine-wide temporal framework. Both backends draw a jittered scene and write current-to-previous motion beside it before the lens and presentation pass. Vulkan resolves those inputs with Luft's inspectable shader and two history images; Metal gives colour, depth, and motion to a MetalFX temporal scaler that owns its opaque history.

The work also answers two architectural questions the small greenfield renderer made cheap to ask. The renderer wanted explicit owners for a frozen frame and for every extent-sized surface, plus revisions on published scene products. The mathematical shader language already had all the expressions the Vulkan resolve needed. The portable GPU layer grew only the concrete vocabulary both consumers earned: ordered colour attachments, a two-half-float motion format, and a temporal-scaler command whose Metal implementation exposes the native texture contract and ordering fence.

TAA reconstructs samples; it is not a blur #DOEDL3

One rasterized frame samples each pixel once. Luft moves that sample to a different subpixel on each frame with an eight-sample Halton(2,3) sequence, then reprojects the last resolved picture so samples from different frames describe the same world points. A static camera therefore gathers eight positions within a pixel instead of repeatedly gathering its centre. A moving camera can keep the old evidence only where motion says where it went and the new neighbourhood says it is still plausible.

This ordering matters. The scene is jittered before rasterization; temporal resolve happens in linear floating-point colour; the photographic lens and the conversion to the final sRGB attachment happen afterward. Applying a screen blur first would only soften one aliased sample. Accumulating after the lens would feed its chromatic dispersion and focus blur back into history.

One frozen view defines one frame #VATCML

The fly camera is a mutable interactive object. Reading it independently in the surface, sky, and resolve setup would let one encoded frame contain several moments. luft.render::capture-frame-view instead copies position, basis, projection, and jitter into a frame-view once at the frame boundary. That immutable value is current; the value published by the last completed encoding is previous.

luft.render::temporal-jitter returns clip-space offsets, so one pixel of displacement is 2/width by 2/height. Every terrain vertex adds the current offset times homogeneous W to its clip position. The fullscreen sky does not move its triangle; it subtracts the same offset while reconstructing the view ray, which makes the sky and geometry samples belong to one jittered camera.

The shared frame block grew append-only from nineteen to twenty-six vec4 lanes. Its established camera, lighting, material, lens, and deformation ABI did not move. The suffix carries the previous view, current and previous jitter, inverse extent, history validity, and blend weight.

The frame graph has three passes and one ping-pong edge #4I4Y3Z

flowchart TB
  S["jittered surface + sky"] --> C["RGBA16F current colour"]
  S --> M["RG16F current-to-previous motion"]
  S --> D["D32 visibility"]
  C --> R["temporal resolve"]
  M --> R
  A["old RGBA16F history, Vulkan"] --> R
  R --> B["resolved RGBA16F"]
  B -. "swap roles next frame" .-> A
  B --> P["lens or plain present"]
  P --> O["output attachment"]
ProductFormatMeaning
Current colourRGBA16FThis frame's jittered, pre-presentation colour; alpha is current focus metadata
MotionRG16FUnjittered current-to-previous displacement in UV units
DepthD32Surface visibility; consumed by MetalFX and retained for the Vulkan experiment
Resolved colourRGBA16FVulkan ping-pongs explicit history A/B; MetalFX owns native history behind one output
OutputRenderer formatLens-processed or plainly presented result, normally sRGB

luft.render::encode-frame keeps this semantic order on both backends. Vulkan records three render passes and prepares textures between attachment and sampled states. Metal ends the surface pass by signaling the scaler's fence, encodes MetalFX on the Metal 4 command encoder, then makes the lens pass wait on the same fence before sampling the resolved colour. A cold or invalid frame resets the selected resolver; only after the whole sequence has encoded does Luft publish the frozen view.

Motion points from the current sample to its old address #C7WIN4

Let q_t be the unjittered UV position of a world point under the current view, q_{t-1} its position under the previous view, and j_t the current clip-space jitter. The motion attachment stores

v = q(t-1) - q(t)

without jitter. The current raster sample lies at u=q_t+j_t/2, so resolve addresses history at

u + v + (j(t-1) - j(t))/2 = q(t-1) + j(t-1)/2.

Keeping the two terms separate makes their conventions inspectable: luft.render.shaders::point-motion owns reprojection and the resolve shader owns the sampling pattern. Static terrain writes motion by projecting its interpolated world point through both frozen views; the sky projects the same world direction from each camera origin. Motion is loaded with exact texel-load rather than filtered at silhouettes, where blending foreground and background velocities would invent a third surface.

Ordinary and temporal surface fragments are paired from one source body. The ordinary form advertises only colour location zero; the temporal form adds motion at location one. Every Luft material -- flat, rounded, chamfered, field, ink, paper, stock, and sky -- therefore follows the same motion convention on both Vulkan and Metal.

Resolve admits old colour through a shaped neighbourhood #C4ED2V

luft.render.shaders:temporal-resolve-fragment-shader gathers the current 3×3 neighbourhood, converts it to YCoCg, and forms the componentwise minimum and maximum. It bilinearly samples reprojected history and clips that colour to the box before blending. YCoCg makes this a box in luminance and two chroma axes rather than three unrelated RGB axes, so clipping changes hue less readily.

The configured history share is 0.9. It falls as screen motion rises and as the clipped old luminance differs from the centre; an out-of-frame address or an invalid history makes it zero. This is rejection rather than a claim that old colour is true. Alpha is never accumulated: Luft's lens reads alpha as this frame's focus distance metadata, so resolve copies it from the current centre.

History is a semantic cache #OWG6ZD

A texture from the preceding submission is not necessarily history of the same picture. Luft reuses it only when all of these remain true:

The key contains the resolved numbers, not merely *light* and *material* names, so hand-tuning an atelier special invalidates history too. luft.render:refresh-scene increments a publication revision, which catches rebuilding the same scene object instead of relying on identity as a change detector. Resize, buffer re-publication, style changes, and teleports all produce one current-only frame before accumulation resumes.

Frame resources are one extent-sized ownership cohort #T7RQTI

TAA made the old collection of renderer texture slots too easy to publish halfway through a resize. A luft.render::frame-surfaces now owns the final colour and depth, optional current colour and motion, either two Vulkan histories or a MetalFX scaler and resolved output, their views, and the bind groups that depend on them. luft.render::make-frame-surfaces constructs a complete candidate; resize publishes it as one value, resets history, and then retires the old cohort. Destruction walks the opposite dependency order: bind groups, views, textures, then the longer-lived layouts and device resources.

Scene storage growth follows the same rule. Sites, cell bits, cell stock slots, and their bind group form one generation. If any buffer is too small, luft.render::upload-scene builds and fills replacements for all three, creates the candidate bind group, publishes them together, and only then retires the old generation. Buffers that already fit are updated in place; capacities double on growth so editing does not rebuild the cohort for every new face.

The shader language was ready; both HALs needed MRT #CZGBC9

No new mathematical shader operation was required. The language already represented several location outputs, texture sampling, exact texel-load, uniform and storage resources, vector min=/=max=/=clamp, derivatives, and source abstractions. A small Luft-local macro makes ordinary/temporal shader pairs; putting that policy into the generic graph would have confused a renderer variant with a new mathematical idea.

The GPU layer did need to tell the truth about a fragment stage with more than one result. :rg16-float now names two half-float motion lanes, including its four-byte texel size and native Vulkan and Metal formats. Vulkan render pipelines, compatible-render-pass caching, attachment descriptions and references, blend states, framebuffers, clear values, and pass validation all consume an ordered target list. Metal pipeline compilation and render-pass attachment setup now do the same. The generic descriptors still describe a one-target pass without a special case at their callers.

This is the useful boundary: multiple render targets and a format are portable GPU vocabulary; temporal rejection, ping-pong history, and camera-cut policy belong to the renderer.

Two backend resolves share one semantic boundary #D7GZA6

The frozen current/previous views, jitter sequence, current colour, depth, motion meaning, invalidation decision, and extent cohort are backend-neutral semantic inputs. luft.render::temporal-resolve-kind selects an implementation at the device boundary: Vulkan compiles Luft's shader resolve; Metal creates a fixed-extent gpu-temporal-scaler and does not compile that shader at all.

The scaler is a narrow capability, not a portable imitation of MetalFX. Its descriptor fixes input and output extents and the four formats. Creation asks MetalFX for the exact usage of each texture before Luft constructs the extent cohort. Each frame supplies colour, conventional depth, normalized current-to-previous motion, pixel-space jitter, and the semantic reset bit. Metal owns the neutral exposure texture and the fence required to order its render, scaler, and post stages; destruction retires the scaler before the textures and views on which it depends.

This leaves renderer policy in Luft and native reconstruction in the backend. The default full effect list is (:sky :lens :taa) on both platforms, but neither backend has to pretend that the other's history representation is its own.

DONE MetalFX resolves Luft's temporal inputs #NL5J0J

Intent. Make Darwin's default Luft renderer accumulate the same jittered colour and motion contract as Vulkan while letting MetalFX own reconstruction, history, and its required synchronization.

Evidence. The Metal renderer test creates a real RGBA16F + D32 + RG16F Metal4FX scaler, renders repeated frames, observes reset on the first frame and history use on the second, and invalidates on material values, same-object scene publication, and a camera cut. A 640 by 400 all-style demo compiled every temporal MRT pipeline and produced a MetalFX-resolved PNG. The same test reframes the renderer, proves a new extent-owned scaler is published cold, and then renders and tears down that replacement successfully.

Done when. Darwin enables :taa by default, queries MetalFX texture usage, orders surface, scaler, and lens work with the native fence, preserves Vulkan's shader path, and survives real rendering, readback, resize-owned destruction, and semantic history resets. Those conditions now hold.

defmethod create gpu.lisp:1016
defmethodcreate
descriptortemporal-scaler-descriptor

Create one synchronous Metal4FX temporal scaler for a fixed extent.

The scaler publishes the exact native texture contract that Luft uses to construct its temporal surface cohort. #NL5J0J

ensure-live-metal-objectdevice:create-temporal-scaler
let*
input-size
canonical-texture-extent
temporal-scaler-descriptor-input-sizedescriptor
descriptor:create-temporal-scaler
output-size
canonical-texture-extent
temporal-scaler-descriptor-output-sizedescriptor
descriptor:create-temporal-scaler
color-format
temporal-scaler-descriptor-color-formatdescriptor
depth-format
temporal-scaler-descriptor-depth-formatdescriptor
motion-format
temporal-scaler-descriptor-motion-formatdescriptor
output-format
temporal-scaler-descriptor-output-formatdescriptor
nativefenceexposurescaler
completed-pnil
unless
and
=1
thirdinput-size
=1
thirdoutput-size
reject-metal-gpu-requestdescriptor:invalid-temporal-scaler-extent
unwind-protect
progn
log-event:metal"begin Metal4FX temporal scaler ~{~Dx~D~}"
subseqinput-size02
multiple-value-setq
nativefence
luv.metal:new-metal-4-temporal-scaler
metal-native-objectdevice
metal-device-compilerdevice
metal-resource-pixel-formatcolor-formatdescriptor
metal-resource-pixel-formatdepth-formatdescriptor
metal-resource-pixel-formatmotion-formatdescriptor
metal-resource-pixel-formatoutput-formatdescriptor
firstinput-size
secondinput-size
firstoutput-size
secondoutput-size
unless
andnativefence
error'metal-gpu-error:operation:create-temporal-scaler:reason:unsupported-or-creation-failed:detailsdescriptor
multiple-value-bind
native-color-usagenative-depth-usagenative-motion-usagenative-output-usage
let
color-usage
portable-metal-texture-usagenative-color-usagedescriptor:color
depth-usage
portable-metal-texture-usagenative-depth-usagedescriptor:depth
motion-usage
portable-metal-texture-usagenative-motion-usagedescriptor:motion
output-usage
portable-metal-texture-usagenative-output-usagedescriptor:output
setfexposure
createdevice
make-texture-descriptor:label"MetalFX neutral exposure":size'
11
:dimensions:2d:format:r16-float:usage'
:copy-dst:texture-binding
write-texture
make-texture-copy:textureexposure
make-array'
11
:element-type'
unsigned-byte16
:initial-element#x3c00
make-texture-data-layout:bytes-per-row2
'
11
setfscaler
make-instance'metal-gpu-temporal-scaler:label
gpu-descriptor-labeldescriptor
:native-objectnative:devicedevice:fencefence:exposure-textureexposure:input-sizeinput-size:output-sizeoutput-size:color-formatcolor-format:depth-formatdepth-format:motion-formatmotion-format:output-formatoutput-format:color-usagecolor-usage:depth-usagedepth-usage:motion-usagemotion-usage:output-usageoutput-usage
completed-ptnativenilfencenilexposurenil
log-event:metal"complete Metal4FX temporal scaler"
scaler
unlesscompleted-p
whenexposure
ignore-errors

DONE First Vulkan temporal resolve #SL4ICD

Intent. Give the atelier stable subpixel edges under a still or flying camera, with motion for every surface style and without reviving a second terrain geometry path.

Evidence. The renderer test renders real Vulkan RGBA16F + RG16F multiple targets, proves the first frame rejects history and the second uses it, observes the ping-pong index, and invalidates on a same-object scene refresh, an actual lighting value change, and a camera teleport. Shader tests prove ordinary fragments expose one output and all temporal variants expose two. An all-style probe compiles every temporal module and pipeline and renders two accumulated frames.

Done when. Vulkan's full Luft renderer jitters geometry and sky, writes current-to-previous motion, resolves bounded history, presents it through the lens boundary, and survives scene publication and camera cuts. Those conditions now hold.

What the first resolve deliberately leaves open #3CW6GA

The first pass is a coherent baseline, not the last word in temporal quality. The next evidence should come from moving silhouettes and animated materials, where colour clipping alone has the least information.

TODO Make the Vulkan disocclusion test depth-aware #1MZC4T

Intent. Sample the retained current depth in Luft's shader resolve, choose the frontmost velocity from a small neighbourhood at silhouettes, and reject history whose reprojected depth no longer describes the visible surface. This is velocity dilation and disocclusion testing, one connected improvement rather than two unrelated filters. MetalFX already receives depth under #NL5J0J; this mark concerns the inspectable Vulkan implementation.

Evidence. Current motion is exact at its own pixel and the YCoCg box rejects many stale colours, but Vulkan discards depth after visibility and cannot yet distinguish a newly revealed background from the foreground that occupied its old address.

Done when. A camera orbit around the stock and field studio pieces leaves no foreground-coloured trail on newly exposed sky or wall, with a capture or pixel test that fails when depth rejection is disabled.

After that proof, a reactive mask for changing highlights and procedural detail, higher-quality history sampling, and previous positions for genuinely animated geometry are local quality experiments. Today a deformation change correctly invalidates the whole history; it does not pretend to have per-object motion that the scene model cannot yet express.