LIM3D-XL

Overview #

LIM3D-XL watches how someone moves through a photographed scene and tells you where they are going and whether they should stop, looking at every object and every step before it answers. You give it a written sketch of the scene and the first few timed steps of the movement; it gives back the same readings as LIM3D, more often right, a little later. Use it unless you need the answer in single-digit milliseconds.

LIM3D-XL (Large Intention Model 3D, enhanced) is a scene-and-motion intention reader: given a Falcon scene sketch and the first seconds of an observed track through that scene, it returns what the mover is trying to do, which sketched object they are heading for, what they are doing right now, the next hazard on their line, a proceed / caution / stop call, the advice that follows from it, where the motion ends and in how many seconds. It is the same job, the same eight readings and the same label vocabularies as LIM3D; what changes is the middle of the network.

LIM3D summarises a scene by pooling its objects into one vector. LIM3D-XL keeps every sketch object and every observed step as its own token and runs six layers of self-attention over the set, so a relation such as “the cyclist is closing on the planter box two steps ahead” is something the model can represent directly instead of recovering from averages. It also watches twelve observed steps instead of six, accepts up to sixteen objects instead of twelve, hashes text into twice as many buckets, and was trained with the kind of noise a tracker produces — jittered positions and dropped frames — so a slightly wobbly track still reads cleanly.

It belongs to the intention family alongside ELIM, LIM and LIM3D. Its input comes from two upstream sources the host application already has: a vision model that emits the Falcon scene sketch, and a tracker or simulator that produces steps in the viewer frame (Waymark Extra traces are in that frame already). Its output is a small JSON record the host application folds into whatever it does next; a stop is meant to forbid a “keep going” reply.

Intended use #

• Reading the intent, target and next hazard of a person (or the viewer) moving through a sketched outdoor scene, from a short observed track.
• Deciding, before a responder answers a movement question, whether the answer may proceed, must carry a caution, or must stop.
• Estimating where a motion ends and how long it takes, in metres and seconds relative to the mover, for tracks with more than six steps or scenes with more than twelve objects.
• Tracks that come from a real tracker rather than a simulator, where positions jitter and frames drop.

Out of scope #

• Anything the sketch does not describe: the model never sees pixels, only the text the vision model produced.
• Precise geometry. Object positions are derived from band words (foreground / midground / background, left / centre / right), not from measured coordinates.
• Motion that is not a person walking, cycling or standing still; vehicles, animals and objects in flight were never simulated.
• Real-time gating at high frequency: one read costs on the order of 100 ms in the current runtime.

Choose LIM3D-XL when #

• The observed track is longer than six steps and the extra history matters; LIM3D only tokenises the last six.
• The scene holds thirteen to sixteen objects; LIM3D keeps only the first twelve of a sketch.
• The track comes from a real tracker with position jitter or missing frames, which LIM3D-XL was trained against and LIM3D was not.
• Target accuracy matters more than latency: LIM3D-XL scores 0.885 on target against LIM3D’s 0.726 on the same held-out scenes, and holds up under jittered and gappy tracks where LIM3D does not, at about seven times the forward cost (6.8 ms against 0.9 ms, batch 1 on a CPU).
• It is the default: a request to /v1/intention3d without a model field is answered by LIM3D-XL. Otherwise choose LIM3D, selected per request with model set to lim3d: it answers with the same readings, the same vocabularies and the same request shape at the lowest cost per read.

Architecture #

LIM3D-XL is a 36,153,641-parameter transformer encoder over a short, typed token sequence, followed by nine small linear heads. Its front half — how a sketch becomes object rows, how steps become motion features, how the text is hashed — is identical to LIM3D’s, so the two models share one feature pipeline and one runtime; only the encoder between the features and the heads differs.

Text features. The sketch and the rendered step lines are concatenated into one observed-text block, lower-cased, and hashed with FNV-1a into 32,768 buckets: character 3- and 4-grams of the last 360 characters plus word 1- and 2-grams of the whole text, capped at 2,048 features. An EmbeddingBag (32,768 × 768, mean-pooled) turns the bucket list into one 768-vector, which a linear layer with GELU projects to the 384-wide model dimension.

Step features. The last twelve steps each yield eleven numbers — time, x, y, z, velocity in x/y/z, speed, heading sine and cosine, and a valid flag — scaled and clipped to small ranges. Velocities are taken against the previous step in the full list, with the time delta floored at 0.1 s, so truncating to the window does not lose the incoming velocity of the first tokenised step. Shorter tracks are left-padded with zero rows and the padded rows are masked out of attention. Eleven global motion features summarise the whole track regardless of length: step count, elapsed time, total displacement, mean speed, current heading, whether the mover is a person, whether the last step arrived, and the current position.

Object features. The objects: line of the sketch is parsed into up to sixteen objects in sketch order. Each is given a role from a fixed name table with a keyword fallback (walk, block, over, road, goal, person, unknown), a radius per kind, and an approximate position from its band words: depth sets y (foreground 2.75 m, midground 7.0 m, background 16.0 m), lateral position sets x as a fraction of 2 × 0.58 × y, and “ground overhead” places the object at z = 5.0 m. Each object row holds nineteen numbers: the seven-way role one-hot, position, radius, distance from the mover, bearing sine and cosine relative to the mover’s heading, closing speed, clearance beyond the radius, forward and sideways offsets, and an is_mover flag marking the object that is the mover itself.

Tokens. One global token is the sum of the projected text vector and the projected global motion features; each step token is its projected eleven-vector plus a learned position embedding for its slot in the twelve-step window; each object token is its projected nineteen-vector plus a learned slot embedding. A learned type embedding (global / step / object) is added to each. The sequence [g, s_1 … s_12, o_1 … o_16] is at most 29 tokens; padded step slots and empty object slots are masked as attention keys.

Backbone. Six pre-LayerNorm blocks: x = x + proj(MHA(ln1(x))) with a fused 384 → 1,152 query/key/value projection, eight heads of dimension 48, scaled dot-product attention with key masking and no causal mask; then x = x + c_proj(GELU(c_fc(ln2(x)))) with the feed-forward at 384 → 1,536 → 384. A final LayerNorm produces the normalised sequence.

Heads. Intent (6), phase (8), action (3), hazard (7), advice (9), the three special targets (none / viewer / off_scene), endpoint (3 values in tens of metres) and ETA (1 value in tens of seconds) are linear layers on the normalised global token. The target head scores each object separately: a linear layer over the concatenation of that object’s output token and the global token gives one logit per object, and a softmax over 3 special + 16 object logits picks the target. The label sets are listed under Inputs & outputs and on /docs/vocabularies.

What is deliberately absent: there is no vision — the model never sees pixels, only the sketch text — no map and no memory across requests. The choice between LIM3D-XL and LIM3D is made by the request’s model field before the forward pass, not inside the network; the same 104 tensors answer every LIM3D-XL call.

Inputs & outputs #

Input #

A request carries a Falcon scene sketch, the observed steps of one mover, and optionally who the mover is and whether step notes should be shown to the text bag. Steps are in the viewer frame — approx m; x right, y ahead, z up; viewer at origin — with t in seconds from the start of the observation. The request below is one of the model’s own parity fixtures (a four-step track of the viewer walking through a small scene):

{
  "sketch": "kind: scene\nviewer: on trail looking forward\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: person with stroller (center background, colour grey stroller); planter box (center-left foreground, size medium); weeds in crack (left midground); maple tree (left background, ground on-ground); stair flight (left foreground, colour steel)\nrelations: weeds in crack in front of person with stroller; maple tree behind planter box; person with stroller behind stair flight; planter box in front of weeds in crack; weeds in crack behind stair flight; planter box in front of person with stroller; weeds in crack left of center; maple tree left of center\nrelative: weeds in crack closer than person with stroller; planter box closer than maple tree; stair flight closer than person with stroller",
  "steps": [
    {"t": 0.0, "x": -0.1, "y": 0.0, "z": 0.0, "heading": "ahead-left", "notes": ["weeds in crack ~9m ahead", "ahead: planter box ~2m", "ahead: stair flight ~3m", "ahead: maple tree ~20m"]},
    {"t": 1.5, "x": -2.2, "y": 0.7, "z": 0.0, "heading": "ahead-left", "notes": ["sidestep right around planter box", "weeds in crack ~7m ahead", "ahead-right: stair flight ~2m"]},
    {"t": 4.5, "x": -3.5, "y": 4.8, "z": 0.0, "heading": "ahead", "notes": ["weeds in crack ~3m ahead", "behind: stair flight ~3m"]},
    {"t": 6.0, "x": -4.2, "y": 6.9, "z": 0.0, "heading": "ahead", "notes": ["weeds in crack <1m ahead", "behind: stair flight ~6m"]}
  ],
  "mover": "you",
  "with_notes": true
}

Limits and framing:

• Sketch. Only the objects: line is parsed for geometry, as name (lateral depth[, colour …][, size …][, ground …]) entries separated by ; . Lateral is one of left, center-left, center, center-right, right; depth is foreground, midground or background. The first sixteen objects are kept; the rest are ignored. The full sketch text still feeds the hashed text bag. The sketch format is specified on /docs/formats.
• Steps. Up to 64 steps are accepted; steps whose t, x, y or z is not a finite number are dropped, and z defaults to 0. Only the last twelve become step tokens, but every step contributes to the global motion features and to the rendered text. A step may carry heading (ahead, ahead-left, right, …), free-text notes, arrive: true on an arrival line, and a target label for that arrival.
• Mover. "you" means the viewer at the origin. Any other value should match an objects: entry, whose row is then flagged is_mover and whose token carries the observed motion; an unmatched label leaves no object flagged and the read treats the mover as an unlisted person.
• Notes. When with_notes is true (the default whenever any step has notes) the notes are rendered into the step lines the text bag sees; setting it false reads the track from bare coordinates, which the model was trained to handle on half of its examples.
• Frame. Metres are approximate for the sketch objects: they are placed by band words, not measured. Step coordinates are whatever the tracker or simulator reports in the same viewer frame.

Output #

The response wraps the reading in the standard envelope. For the request above the final weights return:

{
  "ok": true,
  "engine": "lim3d",
  "frame": "approx m; x right, y ahead, z up; viewer at origin",
  "model": "lim3d-xl",
  "result": {
    "intent": "reach",
    "pIntent": 1.00,
    "target": "weeds in crack",
    "targetIndex": 2,
    "pTarget": 1.00,
    "phase": "walking",
    "hazard": "none",
    "pHazard": 1.00,
    "action": "proceed",
    "pAction": 1.00,
    "advice": "none",
    "endpoint": {"dx": -0.24, "dy": 0.71, "dz": 0.07},
    "eta": 0.48,
    "rule": "default-proceed",
    "model": "lim3d-xl",
    "summary": "You're heading for the weeds in crack, about 1m and 1s to go."
  }
}

Cut the same track after its first step — before the sidestep has happened — and the reading changes to what the training labels call the next event: hazard becomes obstacle_ahead, advice becomes sidestep_right, action becomes caution under the rule action-caution≥0.45, and the endpoint moves out to about 7 m and 6 s.

Vocabulary notes:

• intents — reach a sketched object; pass_ahead of the viewer; step_aside left or right; approach_viewer to a point about a metre in front of the camera; leave_scene out of the far end of the frame; loiter in place.
• phases describe the current step, not the goal: walking, sidestep, give_way, crossing, climbing, stopped, arrived, idle.
• hazards and advice are paired by the training labels: road_no_crosswalk → stop_at_edge; person_ahead → give_way; obstacle_ahead → sidestep_left / sidestep_right; road_crossing → use_crosswalk; elevation_change → climb_stairs / up_ramp; low_clearance → mind_overhead; none → none.
• target_special are the three non-object targets. In the response, targetIndex is the 0-based index into the objects: line for a real object and negative (−3, −2, −1 for none, viewer, off_scene) otherwise.
• roles are not an output; they are the seven object roles the feature pipeline assigns from object names, listed here because they shape what the model can distinguish.

Decision rule and confidence semantics:

1. stop when the action head says stop with p ≥ 0.50 (action-stop≥0.5), or when the hazard head says road_no_crosswalk with p ≥ 0.60 (hazard-road≥0.6).
2. caution when the action head says caution with p ≥ 0.45 (action-caution≥0.45), when any hazard other than none has p ≥ 0.60 (hazard≥0.6), or when the action head’s argmax is a stop that fell below 0.50 (weak-stop→caution).
3. proceed otherwise (default-proceed).

pAction is the probability of the raw action argmax before the rule runs, so a caution promoted from a weak stop can carry a pAction that belongs to stop; read rule to know which branch fired. pIntent, pTarget and pHazard are the softmax mass of the returned label; probabilities are estimates over a synthetic distribution and very confident values (as in the example) are common on in-distribution scenes. Endpoint is relative to the mover’s current position, not to the viewer, and eta is clamped at zero. Estimates, never measurements.

Training #

LIM3D-XL was trained from scratch on 2026-09-12 on Google Cloud Vertex AI (custom job lim3d-xl-20260912-182554, one a2-highgpu-1g machine with a single NVIDIA A100 40 GB) in 4,161 s — just under 70 minutes. No real user data was used: every example is synthetic. It was not warm-started from LIM3D; the two models share a feature pipeline and label set but no weights.

Data. The generator builds an outdoor scene (sidewalks, roads, crosswalks, planters, hedges, stairs, ramps, overpasses, doorways and up to a few people), renders it into a Falcon scene sketch, chooses a mover — a sketched person about 62 % of the time when one exists, otherwise the viewer — gives that mover a hidden goal, and simulates a walking trace toward it with events along the way (stop at a road edge, give way to a person, sidestep an obstacle, cross at a crosswalk, climb stairs or a ramp, duck under an overhang). Each full trace is cut at one or two random prefixes; the model sees the sketch and the steps before the cut, and the labels come from the hidden goal and from the first event after the cut. Hazard and advice are read off that next event; phase comes from the last observed line; action is stop if any future event is a stop-at-edge and otherwise follows the fixed hazard-to-action map; endpoint and ETA are the remainder of the trace. Half of the examples strip the step notes so the model also works from bare coordinates. For the XL profile the generator allows up to sixteen objects per scene (LIM3D’s generator rejects scenes with more than twelve) and keeps twelve steps of motion context. 800,000 scenes with up to two prefixes each yielded 1,375,601 examples after duplicate prefixes and degenerate cuts were dropped, generated across eight worker processes in 20,000-scene chunks with seed 5; 95 % (about 1.31 M) were used for training and 5 % (about 69 k) held out for validation.

Augmentation. Applied to half of the examples, on the observed steps only: each step has a 40 % chance of ±0.15 m jitter in x and y (rounded to 0.1 m), and a track of four or more steps has a 25 % chance of losing one random interior step. This is the tracker-style noise that separates LIM3D-XL’s training set from LIM3D’s; it does not model sensor drift, occlusion or wrong-person tracking.

Recipe. 16 epochs at batch size 2,048 — 639 optimiser steps per epoch, 10,224 in total — with AdamW at learning rate 1e-3 and weight decay 0.01, a 200-step linear warm-up, cosine decay to 5 % of the peak, and gradient-norm clipping at 1.0. The loss is a weighted sum over the nine heads: cross-entropy on intent, phase, hazard (× 1.25), advice and target, class-weighted cross-entropy on action (× 1.5, with stop weighted 2.0 and caution 1.3), and smooth-L1 on the endpoint (× 4.0) and ETA (× 2.0), both regressed in tenths of their unit. Validation ran after every epoch on the held-out split.

Not trained on. Real images, real trajectories or real people; indoor scenes; vehicles, animals or anything that is not a simulated person or the viewer walking; scenes with more than sixteen objects; tracks whose noise is worse than the augmentation above.

Evaluation #

Two evaluations stand behind the table. The rows marked trainer validation are the trainer’s own pass over the held-out 5 % validation split of the training distribution (up to 20,000 rows), recorded in the run’s run.json after the final epoch. Every other row comes from the evaluation programme of 2026-09-13, run offline on a CPU against the final weights and recorded in eval/lim3d-xl/2026-09-13/results.json, with the roll-up in REPORT-2026-09-13.md; the key path after the file name names the exact figure. Its held-out set is 20,000 fresh synthetic examples from generator seed 9001 (training used seeds 3 and 5), with no augmentation and scenes limited to twelve objects, so that LIM3D and LIM3D-XL score the same inputs. Accuracies are argmax-equals-label; stop recall and precision are those of the raw action head over the 1,297 true-stop examples; endpoint MAE is the L1 distance summed over dx, dy and dz in metres and endpoint L2 the Euclidean distance; ETA MAE is in seconds.

Held-out accuracy #

The fresh held-out figures reproduce the trainer’s to within about a point on every head, and the ordering against LIM3D is unchanged: LIM3D-XL leads on every head, most on target (0.885 against 0.726) and stop recall (0.894 against 0.823), and its endpoint error is 0.38 m lower. Per class (heldout.per_class), the weak spots are the same as LIM3D’s, only smaller: caution is read with recall 0.688 and precision 0.687, stop with recall 0.894 and precision 0.759, and among hazards road_crossing (recall 0.468 over 62 examples), obstacle_ahead (0.512) and person_ahead (0.583) are the hardest.

Robustness #

Each perturbation is applied to the observed steps of the same held-out examples; the labels stay those of the unperturbed prefix. Jitter adds Gaussian noise of the stated σ, in metres, to every step. Dropping removes one or two random interior steps and applies only where the prefix is long enough (n is the subset used). Truncation keeps only the first one, two or three observed steps while the labels still describe the full prefix, so it measures how far ahead the model can anticipate from less motion rather than accuracy in the usual sense.

Source: results.json, robustness.{condition}.metrics for the conditions jitter_0.15, jitter_0.3, jitter_0.5, drop_1, drop_2, truncate_1, truncate_2 and truncate_3; the clean row is heldout.metrics. The dropped-step rows sit above the clean full-set row because they are scored on the long-prefix subset only; against the clean score on that same subset (robustness.{condition}.clean_same_subset), one dropped step costs 0.030 of action accuracy and 0.99 m of endpoint MAE, and two cost 0.032 and 1.30 m. At σ = 0.5 m the target head loses 0.099 where LIM3D’s loses 0.171, and with two steps dropped the endpoint error grows by 1.30 m where LIM3D’s grows by 5.22 m: the jitter and dropped-frame augmentation shows clearly. Stop recall rises under dropped steps while stop precision falls, from 0.860 to 0.662 on that subset: a gappy track reads as slower, more hesitant motion and the model leans toward stop, a safe direction but not a gain in accuracy.

Calibration #

Expected calibration error of the max-softmax confidence over ten equal-width bins on the held-out set (calibration.{head}.ece): intent 0.006, phase 0.002, action 0.031, hazard 0.029, advice 0.028, target 0.017. The action head’s mean confidence is 0.939 against an accuracy of 0.907, so it is mildly over-confident; its reliability bins (calibration.action.bins) show that a confidence between 0.7 and 0.8 corresponds to an accuracy of about 0.66 and one between 0.8 and 0.9 to about 0.74, while readings above 0.9 — 16,216 of the 20,000 — are right 0.974 of the time. The decision-rule thresholds should be read with that in mind.

Latency #

Batch-1 forward pass in PyTorch on an arm64 workstation CPU with eight threads (latency.forward_batch1, 300 calls): 6.78 ms median, 7.26 ms p95; with featurisation included, 7.78 ms median and 8.23 ms p95. This is the network’s own cost. The pure-scalar in-process runtime measured about 105 ms per read (see Runtime & deployment), and the hosted route adds network time on top of whichever runtime serves it.

Real-distribution sketches #

The synthetic held-out set uses the generator’s own sketch grammar. To check the model on sketches from the distribution a vision model actually produces, 200 sketches from the spatial environment corpus that Waymark was trained on (348 prefixes; consistency) were parsed back into simulator scenes — object names to kinds, band words to metres sampled inside the band — given a hidden goal and simulated with the Waymark Extra simulator, so the sketch text is real while the labels are known. The mover is the viewer, because those sketches contain no people.

The two models agree on the target 0.816 of the time, on the action 0.914 and on the intent 0.948, and their endpoints lie within 1 m of each other 0.647 of the time (consistency.agreement). The real sketch grammar — two to four objects, no people, the viewer as mover — is not harder than the generator’s.

Known gaps #

Every ground truth above is synthetic. The held-out set, the perturbation sweeps and the real-sketch check all use simulated traces, and the real-sketch check changes only the sketch distribution; no real tracker output, no real photographs and no human-labelled scene have been evaluated, so these numbers bound what a real track can expect rather than measure it. The perturbations are synthetic too — Gaussian jitter rounded to 0.1 m and uniformly chosen dropped steps — and do not model drift, occlusion or identity switches between tracked people. Truncation keeps the labels of the full prefix, so its rows say how little motion the model can anticipate from, not how accurate it is: with one step visible, action accuracy falls to 0.707 and endpoint error to about 6 m, which is why a host should wait for two or three steps before trusting a reading. Stop recall of 0.894 still means about one simulated must-stop situation in ten is read as caution or proceed by the raw head, and stop precision of 0.759 means about one stop in four is a false alarm. Latency was measured in PyTorch, not in the scalar runtime that serves the route. Estimates, never measurements.

API #

LIM3D-XL is the model behind POST /v1/intention3d, the Falcon API’s one 3D intention route, and has served it since 2026-09-13. A request without a model field is answered by LIM3D-XL; a request with model set to lim3d is answered by LIM3D on the same route, with the same request shape, the same response shape, the same errors and the same quota. The response names the size that answered in model, and carries a warning only when LIM3D answered.

Route /v1/intention3d · quota bucket intention · body limit 256 KB.

POST /v1/intention3d HTTP/1.1
Authorization: Bearer fln_…
Content-Type: application/json

The base URL of the Falcon API is given on /docs/api.

Request body:

{
  "sketch": "kind: scene\nviewer: eye-level on sidewalk\nobjects: planter box (center foreground, colour wood); doorway (right midground); hedge (left midground, ground waist); concrete overpass (center background, ground overhead); shopping cart (center-right foreground)\nrelations: planter box in front of doorway",
  "steps": [
    {"t": 0.0, "x": 0.0, "y": 0.0, "z": 0.0, "heading": "ahead", "notes": ["doorway ~6m ahead"]},
    {"t": 1.5, "x": 1.6, "y": 1.5, "z": 0.0, "heading": "ahead-right", "notes": ["sidestep right around planter box"]}
  ],
  "mover": "you",
  "with_notes": true,
  "model": "lim3d-xl"
}

model is optional and takes lim3d-xl (the default) or lim3d. Response 200:

{
  "ok": true,
  "engine": "lim3d",
  "frame": "approx m; x right, y ahead, z up; viewer at origin",
  "model": "lim3d-xl",
  "result": {
    "intent": "reach",
    "pIntent": 1.00,
    "target": "doorway",
    "targetIndex": 1,
    "pTarget": 0.71,
    "phase": "sidestep",
    "hazard": "none",
    "pHazard": 0.87,
    "action": "caution",
    "pAction": 0.63,
    "advice": "none",
    "endpoint": {"dx": 1.02, "dy": 3.13, "dz": 0.99},
    "eta": 2.63,
    "rule": "action-caution≥0.45",
    "model": "lim3d-xl",
    "summary": "You're heading for the doorway, about 3m and 3s to go."
  }
}

The reading above is the final weights’ recorded output for that request, rounded to two decimals. It also shows the decision rule at work: the hazard head sees nothing on the line, but the action head puts 0.63 on caution during the sidestep, which clears the 0.45 threshold, so action is caution with no advice. engine is the fixed string lim3d, the name of the route; model, at the top level and again inside result, is the size that answered. When the request selects LIM3D, model is lim3d and the response carries a warning string, at the top level and inside result, saying that LIM3D is the low-latency option with weaker target identification and less robustness to missing steps; the exact sentence is shown on the LIM3D page. result is null with ok: true when the serving process has 3D intention reads switched off. The steps array is truncated to 64 entries before validation, and with_notes defaults to true when any step has notes.

Errors:

Quota bucket: intention (shared with ELIM, LIM and LIM3D; one request counts once whichever size answers). Body limit: 256 KB. GET /health reports lim3d: true when a 3D intention model is loaded and lists the loaded sizes in lim3dVariants, for example ["lim3d-xl", "lim3d"]. Envelope conventions, authentication, rate limits and the retry policy are on /docs/api.

Runtime & deployment #

The hosted endpoint has served this version since 2026-09-13: the Falcon API process loads the LIM3D-XL bundle beside the LIM3D bundle and answers /v1/intention3d with LIM3D-XL unless the request’s model field says otherwise. The Falcon API runs on 2 vCPU / 2 GiB and is kept warm with one minimum instance, so there is no wait for the service to start; LIM3D-XL answers through the API in well under a second.

LIM3D-XL runs in-process, in the same pure-scalar TypeScript runtime as LIM3D, with no network call and no accelerator. The loader reads kind, n_hash, traj_k, max_objects, n_layer and n_head from the weights’ meta.json, sees kind: "transformer", and builds the attention path instead of the pooling path; the same loader holds either bundle.

• Load behaviour. Weights load lazily on the first read and stay resident (~145 MB of float32 for the 144.6 MB bundle plus the parsed catalogue). A request that arrives while the load is in progress waits for it rather than failing. If the weights directory is missing or unreadable the process marks that size missing: the hosted route then falls back to LIM3D for requests that did not ask for a size, and answers 503 lim3d_unavailable only when neither bundle is loadable; a client should retry a 503 once after 2.5 s and then treat the model as absent.
• Selection. On the hosted route, selection is per request: model absent or lim3d-xl is answered by LIM3D-XL, model set to lim3d by LIM3D, and the response’s model field names which. In a host that embeds the runtime, one loader holds one bundle, chosen by the weights directory the host points it at; the two sizes are not run side by side inside one loader.
• Switched off. A serving process can keep 3D intention reads off altogether; the route then answers 200 with result: null and no weights are loaded.
• Cost per read. About 100 ms on a laptop CPU for a ten-token request in the scalar runtime (measured ~105 ms as a 20-call mean over the four-step, five-object fixture above), rising with token count; attention over up to 29 tokens through six layers, dominated by the 384 × 1,152 and 384 × 1,536 matrix products in scalar loops. The network itself costs 6.78 ms median at batch 1 in PyTorch on a workstation CPU (Evaluation). LIM3D answers the same request in about 6 ms in the scalar runtime and 0.93 ms in PyTorch.
• Concurrency. The forward pass is synchronous JavaScript: one read occupies the event loop for its duration, so a process serves reads one at a time. Run several processes for throughput; there is no shared state between them.

What the host must provide, when the weights are available to the host process (they are not distributed during the private preview): the two files of the weights bundle (lim3d.bin and meta.json) on local disk, a Falcon scene sketch, and steps in the viewer frame. Nothing else — no tokeniser download, no external service, no GPU. A host that calls the Falcon API instead provides only the request.

Limits & safety #

It does not see the image; it reads the sketch. Everything below follows from that, from the synthetic training data, and from the size of the network.

• Object positions are estimates from band words — three depth bands and five lateral bands — so a “midground” object is at 7 m whether it is 4 m or 12 m away, and a stated “~3m ahead” in a note is text to the model, not geometry.
• “Stop” means the model believes the next event on the line is a road edge with no crosswalk, as the simulator defines it. It does not mean the road is dangerous, that traffic is present, or that stopping is safe.
• It was trained only on simulated pedestrians and a simulated viewer. A track that belongs to a vehicle, a cyclist moving at road speed, an animal or a thrown object was never seen and will be read as if it were a walking person.
• Intent for a person other than the viewer is a guess from a few seconds of motion and a text description; it cannot know what that person will decide to do.
• The window is twelve tokenised steps, sixteen objects and 2,048 hashed text features; the seventeenth object and the thirteenth-oldest step are invisible except through the global summary and the text bag.
• The augmentation covers ±0.15 m jitter and single dropped frames, not sensor drift, identity swaps between tracked people, occlusion or gaps of several seconds.
• One read costs on the order of 100 ms in the current runtime — about twenty times LIM3D — which makes it a poor fit for gating every frame of a video track.
• The response names the size that answered in model, and a warning accompanies every LIM3D reading; a host that gates on action should also check model when it relies on LIM3D-XL’s robustness to gappy tracks.

Out of scope: acting as a collision or pedestrian-safety system; navigation or route planning; reading intent from a still image with no observed motion; scenes the sketch format cannot describe (indoor layouts, crowds, moving vehicles); and any claim about metres or seconds beyond “roughly, in a scene sketched this way”. Not a safety system. Estimates, never measurements.

Fixed weights per version; the model does not learn from requests.

Versions #

Compatibility: a major version bump changes the request or response contract — a field added, removed or renamed in the request or response object, a label added to or removed from any vocabulary, or a change to the token budget (twelve steps, sixteen objects) that alters which inputs are read. A minor bump is a retrain with the same contract and the same catalogues, for example more scenes, a different seed or a different augmentation mix; readings will move but every field and label keeps its meaning. A patch bump touches metadata or runtime only (the meta.json catalogue text, the loader, the summary sentence) with the tensors unchanged.

Current weights: version 1.0.0, lim3d.bin of 144,617,675 bytes (104 float32 tensors, SHA-256 9e1caf63f0f7be4640f7cf56a08f77dbb0b2cecbdbcc0e3501e736d6f0a03ede) with its meta.json of 28,315 bytes, both written 2026-09-12 from Vertex AI job lim3d-xl-20260912-182554. The catalogues in meta.json are the vocabularies listed on this page; the six parity fixtures embedded in it are the regression test for any runtime change.