Fil-C on Nix

Filnix wiki

dashboard-architecture.md

The Tagflow campaign viewer

The viewer renders public, read-only HTML from the existing campaign database. It uses Tagflow aa07b0d, htmx 4.0.0 and Tailwind 4.3.3. This follows the library's dashboard example and uses its reading contracts directly. The controller, workers, stored evidence, queue policy and binary-cache publishers are independent of this change.

Ownership

Module Responsibility
experiment/dashboard/resources.py Canonical URLs, explicit reading options, routes and links derived from one schema
data.py One read-only SQLite transaction per representation; campaign ownership and evidence scope
base.py, views.py Tagflow HTML, navigation and literal Tailwind utility lists
logview.py Bounded byte-cursor readers, source selection and log controls
app.py Starlette routes, conditional representations, SSE revision hints and Hypercorn hosting
static/reader.js Scroll following, pause anchors, navigation scroll reset and old fragment bookmarks
experiment/web.py Retained read-only JSON API, raw downloads and controller health for external clients
experiment/python.nix Pinned Tagflow and Python runtime; no compiler build dependency
deploy/dashboard/ Locked Tailwind tooling; generated CSS is checked in

Application JavaScript neither fetches JSON nor constructs content nodes. Package metadata, diagnostics, status, dependency neighborhoods and log records all arrive as escaped server-rendered HTML. Assets are served locally with content-versioned URLs. Node is needed to regenerate CSS, not to run the installed viewer.

Reading policies

Every page is a normal URL under /campaigns/{id}. Activity, packages, batches, dependencies, individual packages and individual batch logs have full-page HTML representations. Links and GET forms work without JavaScript. Enhanced navigation uses hx.navigate to select #workspace from that same response. There is no HX-Request response variant, viewer cookie or server-side viewer session. Browser history retains the visited URL and scroll position.

Activity counts/current work, timeline/recent batches, package details, batch status and dependency neighborhoods refresh independently with hx.refresh and outerMorph. The campaign SSE connection sits inside the workspace but outside these replacing regions. It emits named revision hints, which trigger normal HTML requests. Reconnection announces the current revision; a five-second poll also picks up phase/heartbeat changes that do not append campaign events. transport=poll uses a three-second poll without SSE. Finished resources stop refreshing.

Activity follows by default, with an explicit Pause/Follow control for its timeline and recent batches. Concurrent attempts occupy separate timeline rows. Inventories and the full batch table are held snapshots with a visible timestamp and refresh notice. All matching rows are present in the document, including all 13,772 inventory attributes if selected. Browser Find works across them. The notice offers an explicit refresh; progress does not continually reorder the list. Package rows show name, version, description and result, with Built muted and Tested emphasized. Timings belong to the batch page.

Persisted semantic browsing

Catalog reads call classification_index(db, campaign) once in bulk and add classifications to each row, including the compatibility /api/packages JSON. This index contains all provider scores/provenance but omits full source documents to avoid loading every source document when browsing the inventory. Package detail reads annotations(db, campaign, candidate_id), including that package's exact evidence. These return only current cached annotations; freshness and pre-migration empty reads belong to the classification backend. HTML GETs perform no classification/model calls or writes. No interpretation changes factual states, checks, blockers, roles, platforms, scheduling or required outputs.

Inventory's optional facet=kind:label intersects its existing state filter. Package, diagnostic and patch labels occupy separate namespaces; colons in labels are retained. A probability of at least 0.8 from any provider suggests a facet for browsing only. Counts are distinct packages within the state-filtered inventory, not provider votes. No facet leaves the ordinary lists/default unchanged. The GET form, held snapshot refresh/update links and CSV URL retain the selected facet; CSV's columns remain unchanged. Unknown/bookmarked facets yield an empty result rather than silently clearing the selection. Navigation still uses full HTML responses, normal URLs and browser history without UI JSON fetching.

Detail labels these interpretations tentative and distinguishes package, diagnostic and patch sources. Every provider/model comparison stays visible, including probabilities below the browsing threshold. Each annotation has its timestamp, question version, evidence hash and an expandable escaped rendering of the actual named JSON source input. Threshold-qualified labels link back to the inventory facet. Missing cached annotations explicitly say unknown / not classified; a missing suggestion never implies a negative factual result.

The Tagflow boundary hashes the exact rendered HTML. The application exposes that validator as a weak ETag because gzip can change the wire representation, and uses Cache-Control: public, no-cache. Matching conditional requests return 304. Rendering still happens before hashing. Gzip reduces repeated markup in large lists; lists are deliberately not virtualized. Each representation is internally consistent, although separate refreshed regions can show adjacent revisions.

Logs

Logs retain the existing aligned raw-byte offsets and 256 KiB read windows. A cursor requests the next full HTML page and selects only its new records and next cursor. Replacing that cursor avoids duplicate append on retry. hx.read_cursor uses recurring requests plus click-to-retry and this:drop, so another tick does not abort a slow request. A source picker and capture metadata refresh separately without rereading the raw log. Those automatic reads wait while the source form or options menu is being used, so they cannot close the menu or reset a selection. Open diagnostic sections also survive morph updates.

Invalid offsets and the 2,000-record display cap replace the requesting reader. hx.recover_reader(closest="#log-reader") resolves relative to that requester: a late response cannot reacquire a newer reader with the same global ID. Recovery responses are not cached. The full raw download remains available.

Scrolling away from the end pauses using the first visible byte offset and its pixel position. Manual Pause uses the same anchor. Resume returns to the current tail. Paused cursors can load newer output explicitly. Earlier output, source, wrapping, font size and window search are ordinary URL options. Live logs advance to the next batch at EOF; a specific batch URL stays with that batch. Following never invents successful results from the text stream.

The common font is 14 px sans serif; output is consistently 12 px monospace by default, with 14/16 px options. There is no terminal escape interpretation or untrusted HTML rendering. Search highlights escaped text within the loaded window.

Development and verification

npm --prefix deploy/dashboard ci
npm --prefix deploy/dashboard run build
nix build --impure --file experiment/default.nix --no-link --print-out-paths
# Python environment including the HTTP test client:
nix build --impure --file experiment/python.nix --arg testing true -o result-dashboard-python
PYTHONPATH=.:tests result-dashboard-python/bin/python -m unittest discover -s tests -p 'test_dashboard.py'
PYTHONPATH=.:tests result-dashboard-python/bin/python -m unittest discover -s tests -p 'test_experiment*.py'
# A separate Chromium debugging endpoint must be listening on 127.0.0.1:9228:
node tests/dashboard-browser.mjs http://127.0.0.1:8788 results/tagflow

The browser regression uses the two real inventory campaign IDs, without writing to either campaign. It checks full lists, mobile fonts and widths, package/detail history with scroll restoration, batch filters, dependencies, live log cursors, pause/resume, failed/slow cursor requests and visible recovery, options staying open across refreshes, live/paused timeline sampling, duplicate offsets, desktop rendering, no-JavaScript pages and the absence of UI JSON requests. Inspect its captured images as well as the assertions. Unit tests cover escaping, exact full-page/cache parity, ownership, validation, reader reset headers, bounded cursors, completion, CSV, refresh contracts, overlapping timeline rows and test evidence scoped to its campaign.

Dependency nodes say Ready when their required outputs are available; this does not claim they were compiled in this campaign. The evidence tooltip distinguishes a store observation from inference through a consumer reaching a build phase. Batch headers distinguish Running, Finished and Finished with errors, and remain visible while scrolling the build list. All observed derivations are shown, with active builds and failures first. An ended activity says Awaiting result while the batch still runs: Nix's activity stop event does not establish success. Once the batch finishes, immutable per-derivation results distinguish Built, Tested, Failed and Result not recorded. A later retry cannot rewrite that history. Build failures and evaluation errors have separate filters, so the overview counts lead to the matching rows; All failures combines the failure categories. Failed refreshes display a notice until that reader successfully recovers.

The controller captures build_outcomes inside the attempt's existing result JSON after checking output validity and recording test evidence. On startup it fills older results from retained facts only when their evidence still belongs to that attempt, plus immutable test records. This migration is idempotent; overwritten or absent observations remain unknown. The web service does not write these facts or query Nix to reconstruct historical results. Batch outcomes and individual outcomes are separate: a batch can finish with errors while many of its dependencies built successfully.

Blocked packages distinguish their own planning/build history from the failed dependency. Each blocker links directly to the batch that owns its evidence, with a date, campaign provenance and the intervening dependency chain. Derivation facts are shared by exact store path, so an earlier campaign's evidence may still block a new campaign; its log URL belongs to that earlier campaign. Different derivations of the same named package do not share this evidence. Blocked/failed states do not imply an automatic retry. Cleared failures with queued candidates show Queued and label the previous log as history.

Planning logs and failures before a builder starts open unfiltered batch output. Only recorded build activities produce scoped build-log links. Scoped logs seek backwards in at most eight 256 KiB windows per request. If the match is earlier, an HTML search cursor continues automatically, with visible progress and a normal link for readers without JavaScript. Search requests replace only their owning reader, retry on failure, and pause while log controls are being edited. Empty bookmarked source filters retain their selected source and explain that no build output was recorded, with a link to the full batch output.

Build an immutable application, run a preview against the real read-only database under filnix-web, then select that store path with deploy/experiment/install and restart filnix-web. Existing workers keep their original executable and source; a viewer update does not rebuild packages. Verify /healthz, public navigation and captured mobile/log output after the switch.