LIM3D

Overview #

LIM3D watches how someone moves through a photographed scene and tells you where they are going and whether they should stop. You give it a written sketch of the scene and the first few timed steps of the person’s movement; it gives back what they are trying to do, what is in their way, a proceed, caution or stop call, and a rough guess at where and when the movement ends. Use it when a quick read matters more than the most careful one.

LIM3D (Large Intention Model 3D) is a spatial-intention classifier. Given a Falcon scene sketch and the first seconds of an observed track for one mover — the viewer, or a person named in the sketch — it says what that mover is trying to do: which object they are heading for, what they are doing right now, what is next on their line, and whether the right call is proceed, caution or stop.

The input is text plus numbers. The sketch is the structured scene description a vision model emits (objects with lateral and depth bands, relations, a viewer line); the track is a short list of timed x/y/z points in the viewer frame, produced by a tracker or by a simulator. The output is six labels — intent, target, phase, hazard, action, advice — with confidences, plus two regressions only a spatial model can give: where the motion ends, in metres relative to the mover, and how many seconds until it does.

LIM3D belongs to the intention family and is the spatial counterpart of LIM. LIM reads a conversation window and returns trajectory, action, harm and steer; LIM3D reads a scene and a motion and returns the same shape of answer for a body moving through space. Its action head — proceed, caution, stop — is the 3D analogue of allow, steer, abort, and a host treats it the same way: as an advisory check before it says anything about moving on. Positions are expressed in one approximate frame: approx m; x right, y ahead, z up; viewer at origin. Objects are placed from the sketch’s band words, never from measured depth, so every distance, endpoint and ETA LIM3D returns is a coarse estimate. Estimates, never measurements.

Intended use #

• Reading the intention behind a partial walk in a sketched scene: reaching a named object, passing through, stepping aside, approaching the viewer, leaving the scene, or loitering.
• Naming the next hazard on the mover’s line — a road with no crosswalk, a person or obstacle ahead, a crosswalk, stairs or a ramp, a low structure — and the manoeuvre that follows from it.
• Gating a host’s movement guidance: stop and caution tell the responder not to encourage the mover onward without a second check.
• Estimating an endpoint and an ETA for a walk that is already under way, for hosts that want a rough “about 5 m and 5 s to go”.

Out of scope #

• Anything the sketch does not contain. LIM3D never sees pixels; an object missing from or mislabelled in the sketch is invisible to it.
• Vehicles, cycling at speed, running indoors, or any motion that does not look like a person walking through an outdoor scene.
• Real tracker output with jitter, dropped frames or metric-accurate depth; the model was trained on clean simulated coordinates in band-derived metres.
• Physical safety decisions. stop means the model expects a road with no crosswalk in the sketch, not that a real road is busy or clear.

Choose LIM3D when #

• The lowest cost per read is what matters: a single-digit-millisecond read from a 66 MB bundle in-process, or the low-latency option on the hosted route, selected per request with model set to lim3d; choose LIM3D-XL, the default on that route, when accuracy on the target and hazard heads matters more — on the same held-out scenes it reads the target right 0.885 of the time against LIM3D’s 0.726.
• The tracks are clean — simulated, or lightly filtered — and rarely longer than six meaningful steps; choose LIM3D-XL when the input is raw tracker output with jitter and gaps, which its augmentation covers and LIM3D’s does not.
• Scenes have twelve or fewer objects; choose LIM3D-XL when sketches routinely carry more.
• The question is about a conversation rather than a scene: choose LIM or ELIM instead.

Architecture #

LIM3D has three readers, one fusion trunk and eight heads. Every layer is a plain linear map with GELU activations; there is no attention, no recurrence and no convolution. The whole network is 33 tensors, 16,516,905 parameters, of which about 12.6 M sit in the text embedding table and about 3.9 M in the dense layers.

Text bag. The sketch and the rendered trace are joined into one string — the sketch under a Spatial sketch: header, then an Observed (approx m; x right, y ahead, z up) block with one line per step, with the mover’s name appended to the header when the mover is not the viewer. The string is hashed the same way LIM hashes a conversation window: FNV-1a over character 3- and 4-grams of the last 360 characters plus word unigrams and bigrams over the whole text, into 16,384 buckets, capped at 2,048 features. A mean-pooled 768-wide embedding bag turns those indices into one vector. A typical example produces around 1,000 features.

Motion MLP. The last six steps become an 11-float row each — time, x, y, z, velocity, speed, heading sine and cosine, and a validity flag, all scaled and clipped — left-padded with zero rows when fewer than six steps exist. Eleven global features follow: step count, last time, displacement, mean speed, heading, whether the mover is a person, whether they have arrived, and the current position. The 77 floats pass through a 384-wide layer and one residual GELU block.

Object encoder. Each sketch object becomes a 19-float row: a seven-way role one-hot (walk, block, over, road, goal, person, unknown) and twelve geometric features relative to the mover — position, radius, distance, bearing sine and cosine, closing speed, clearance, forward and side offset, and an is-mover flag. Rows pass through a 192-wide layer and a residual block, then masked mean- and max-pooling collapse up to twelve objects into two 192-wide vectors. Object positions come from the band words alone: depth bands map to 2.75 m, 7 m and 16 m ahead; lateral bands to −0.55…0.55 of the width at that depth; overhead objects sit 5 m up. Roles come from a table of about fifty known names with a regex fallback, each with a radius between 0.2 m and 5 m; unknown names get role unknown and radius 0.6 m.

Fusion and heads. The four vectors concatenate to 1,536 floats, project to 1,024 and pass two residual GELU blocks. Six classification heads read the fused vector directly: intent (6), phase (8), action (3), hazard (7), advice (9) and target-special (3). The target head is permutation-equivariant: every object embedding is concatenated with the fused vector and scored by one shared linear map, then the three special logits and up to twelve object logits are softmaxed together as one 15-way decision, with absent object slots masked out. Two regression heads return the endpoint (three floats, in tens of metres) and the ETA (one float, in tens of seconds); the runtime multiplies both by ten. The full label sets are listed under Inputs & outputs and on /docs/vocabularies.

What is deliberately absent: no vision — it never sees pixels; no learned tokenizer — the text path is a fixed hash; no sequence model over steps — only the last six carry per-step features; no memory across calls.

Inputs & outputs #

Input #

The sketch is the Falcon scene format described on /docs/formats: a block of key: value lines. LIM3D parses only the objects: line for geometry. Entries are separated by ; , each of the form name (lateral depth[, colour …][, size …][, ground …]). Lateral is one of left, center-left, center, center-right, right; depth is one of foreground, midground, background; ground overhead forces the over role. The first twelve objects are kept in sketch order; the whole sketch, including relations and uncertainty lines, still reaches the text bag.

Steps are timed points in the frame approx m; x right, y ahead, z up; viewer at origin. Each step carries t (seconds), x, y and z (metres), and optionally a heading word (ahead, ahead-right, back-left, …), a notes array of short observations, an arrive flag and a target name. The runtime renders each step back into a trace line such as t=1.5s (1.6, 1.5, 0.0) ahead-right | note | note, with one decimal and notes only when with_notes is on; an arriving step renders as arrive t=…s (x, y, z).

A complete request body for the Falcon API, taken from the first parity fixture stored with the weights:

{
  "sketch": "kind: scene\nviewer: standing at curb looking along roadway\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: maple tree (center foreground, colour green foliage); two-lane road (center-right background, colour dark asphalt, size wider-than-frame); curb (center-right midground); ramp (center foreground, colour grey concrete, ground on-ground)\nrelations: maple tree next to ramp; maple tree in front of two-lane road; maple tree in front of curb; curb behind ramp; ramp in front of two-lane road; two-lane road behind curb\nrelative: maple tree closer than two-lane road; maple tree closer than curb; ramp closer than curb",
  "steps": [
    {
      "t": 0.0,
      "x": 0.0,
      "y": 0.0,
      "z": 0.0,
      "heading": "ahead",
      "notes": ["ahead: maple tree ~2m", "ahead: ramp ~2m", "ahead: curb ~9m"]
    }
  ],
  "mover": "you",
  "with_notes": true
}

Limits that shape the input:

• At most 12 sketch objects are placed; later entries are ignored for geometry.
• The Falcon API keeps the first 64 steps; only the last 6 contribute per-step motion features, the rest inform the global features and the trace text.
• Hashed text is capped at 2,048 features; character n-grams cover only the last 360 characters of the joined text, so a very long sketch pushes the trace out of the character window (word n-grams still cover it).
• Steps with a non-finite t, x, y or z are dropped before scoring.

Output #

The target head scores three special classes plus one slot per sketch object, so its label set is the three names above followed by the sketch’s own object names in order. targetIndex is the sketch index of the chosen object and is negative for the special classes.

The same fixture scored in-process returns:

{
  "intent": "pass_ahead",
  "pIntent": 1.0,
  "target": "none",
  "targetIndex": -3,
  "pTarget": 1.0,
  "phase": "walking",
  "hazard": "obstacle_ahead",
  "pHazard": 0.9994,
  "action": "caution",
  "pAction": 0.9519,
  "advice": "sidestep_left",
  "endpoint": { "dx": 0.25, "dy": 5.25, "dz": 0.21 },
  "eta": 4.65,
  "rule": "action-caution≥0.45",
  "model": "lim3d"
}

Decision rule. action is not the raw argmax of the action head. The runtime combines the action and hazard heads in a fixed order and records which branch fired in rule:

1. Action head says stop with p ≥ 0.50 → stop (action-stop≥0.5).
2. Hazard is road_no_crosswalk with p ≥ 0.60 → stop (hazard-road≥0.6).
3. Action head says caution with p ≥ 0.45 → caution (action-caution≥0.45).
4. Any hazard other than none with p ≥ 0.60 → caution (hazard≥0.6).
5. Action head says stop below 0.50 → caution (weak-stop→caution).
6. Otherwise → proceed (default-proceed).

Confidence semantics. pIntent, pTarget and pHazard are softmax probabilities of the chosen label within their head. pAction is the action head’s own confidence in its argmax and can disagree with action when a hazard branch fired; read rule before trusting it. Probabilities near 1.0 are common on synthetic-looking input and say nothing about calibration on real tracks; on the synthetic held-out set the action head’s mean confidence is 0.954 against an accuracy of 0.889 (see Evaluation). endpoint and eta are point estimates with no interval; treat 1–2 m and about 1 s as the working error (see Evaluation).

Examples #

A person in the sketch, one observed step, seven objects. The mover is matched to the dog walker entry by name; with_notes is off, so the notes are not rendered into the trace text.

{
  "sketch": "kind: scene\nviewer: standing at curb looking along roadway\ncapture: shot drone; orientation landscape, level; height_agl ~20-40m\nobjects: storefront (center-right midground); dog walker (center-left foreground, ground on-ground); trail path (center foreground, size medium, ground on-ground); turn lane (right background, colour white arrows); jogger (right foreground, colour bright top); asphalt roadway (center-right midground); median (center-right foreground, size medium)\nrelations: storefront next to asphalt roadway; asphalt roadway behind jogger; jogger next to median; turn lane behind dog walker; storefront behind jogger; trail path left of jogger; jogger right of center\nrelative: jogger closer than asphalt roadway; dog walker closer than turn lane; jogger closer than storefront\nuncertain: far end of path cut off",
  "steps": [
    { "t": 0.0, "x": -1.2, "y": 4.0, "z": 0.0, "heading": "ahead-right" }
  ],
  "mover": "dog walker",
  "with_notes": false
}

LIM3D reads intent reach (p 1.00), target storefront (p 0.30, index 0), phase walking, hazard none (p 0.95), action proceed (p 0.99), endpoint (6.52, 5.41, −0.04) and ETA 6.46 s. The Falcon API adds the summary “The dog walker is heading for the storefront, about 8m and 6s to go.” The low target probability is typical: with seven candidates and a single observed step the head spreads mass across the storefront, the turn lane and the roadway, and the top choice is a lean rather than a certainty.

Training #

LIM3D was trained on 2026-09-12 in a Google Cloud Vertex AI custom job (lim3d-20260912-171944) on one NVIDIA L4, size preset large, seed 3. The run took 1,493 s including data generation.

Data. The training set is fully synthetic. No real user data, real photographs or real tracks were used. A scene generator and walking simulator — a vendored copy of the one behind Waymark Extra — built 300,000 scenes with a person present in 60 % of them and extra hazards mixed in on top of the default scene: an additional road in 35 % (producing stop and cross events), a second person in 30 % (give-way events) and a stair flight or ramp in 20 % (climb events). Objects were shuffled and scenes with more than twelve objects rejected. When a person existed, that person was the mover 62 % of the time (50 % reaching a chosen object, 20 % approaching the viewer, 16 % leaving the scene, 14 % loitering); otherwise the viewer moved from the origin (72 % reaching an object, 14 % passing ahead, 8 % stepping aside, 6 % leaving). Each simulated trace was cut at one or two random prefixes, giving 515,943 examples; a 95/5 split left about 490,000 for training and about 25,800 for validation.

Labels. Every label comes from information the model cannot see: the hidden goal, and the simulator events after the cut. The first future event sets hazard and advice (a stop-at-edge event → road_no_crosswalk / stop_at_edge, a give-way event → person_ahead / give_way, a sidestep → obstacle_ahead / sidestep_left or _right, a crosswalk crossing → road_crossing / use_crosswalk, stairs → elevation_change / climb_stairs, a ramp → elevation_change / up_ramp, an overhead structure → low_clearance / mind_overhead). Phase comes from the last observed line. Action is stop if any future event is a stop at a road edge, otherwise the fixed hazard-to-action map (none → proceed; the rest → caution). Endpoint is the last point minus the current point; ETA is the last time minus the current time. Half of the examples strip the step notes so the model also works from bare coordinates. Every sketch had to parse back to the same objects in the same order or the scene was skipped.

Recipe. Trained from scratch, no warm start. AdamW, learning rate 2e-3, weight decay 0.01, batch 512, 12 epochs (about 11,500 optimiser steps); linear warm-up over the first 200 steps, then cosine decay to 5 % of peak; gradient clipping at global norm 1.0. The loss is a weighted sum: cross-entropy on intent, phase, advice and target at weight 1; action at 1.5 with class weights stop 2.0, caution 1.3, proceed 1.0; hazard at 1.25; smooth-L1 on endpoint/10 at 4.0 and on ETA/10 at 2.0. No input augmentation was applied for this size preset — the jitter and dropped-step augmentation used by LIM3D-XL was off.

Not trained on. Real tracker output, vehicles, indoor scenes, non-walking motion, scenes with more than twelve objects, and any text outside the sketch and trace grammar. After training, six parity fixtures — the hashed text head, the motion vector, the first object row, expected probabilities, endpoint and ETA — were written beside the weights and replayed in the TypeScript runtime to a tolerance of 2e-3.

Evaluation #

Two evaluations stand behind the table. The rows marked trainer validation are the trainer’s own pass over the held-out 5 % validation split of the training distribution (up to 20,000 rows), recorded in the run record of job lim3d-20260912-171944; the parity and scalar-runtime latency rows come from the runtime check. Every other row comes from the evaluation programme of 2026-09-13, run offline on a CPU against the final weights and recorded in eval/lim3d/2026-09-13/results.json, with the roll-up in REPORT-2026-09-13.md; the key path after the file name names the exact figure. Its held-out set is 20,000 fresh synthetic examples from generator seed 9001 (training used seed 3), with no augmentation and scenes limited to twelve objects, so that LIM3D and LIM3D-XL score the same inputs. Accuracies are argmax-equals-label; stop recall and precision are those of the raw action head over the 1,297 true-stop examples; endpoint MAE is the L1 distance summed over dx, dy and dz in metres and endpoint L2 the Euclidean distance; ETA MAE is in seconds.

Held-out accuracy #

The fresh held-out figures reproduce the trainer’s to within about a point on every head. Target remains the weakest head at 0.726 — roughly one target in four is wrong on synthetic scenes — against LIM3D-XL’s 0.885 on the same inputs, and the raw action head reads stop with recall 0.823 and precision 0.764. Per class (heldout.per_class), caution is the hardest action (recall 0.615, precision 0.622), and among hazards road_crossing (recall 0.339 over 62 examples), obstacle_ahead (0.451) and person_ahead (0.488) are read right less than half the time; none and road_no_crosswalk (0.786) carry the head.

Robustness #

Each perturbation is applied to the observed steps of the same held-out examples; the labels stay those of the unperturbed prefix. Jitter adds Gaussian noise of the stated σ, in metres, to every step. Dropping removes one or two random interior steps and applies only where the prefix is long enough (n is the subset used). Truncation keeps only the first one, two or three observed steps while the labels still describe the full prefix, so it measures how far ahead the model can anticipate from less motion rather than accuracy in the usual sense.

Source: results.json, robustness.{condition}.metrics for the conditions jitter_0.15, jitter_0.3, jitter_0.5, drop_1, drop_2, truncate_1, truncate_2 and truncate_3; the clean row is heldout.metrics. Two weaknesses show. First, jitter: trained without augmentation, LIM3D loses 0.100 of target accuracy already at σ = 0.15 m and 0.171 at σ = 0.5 m, where LIM3D-XL loses 0.031 and 0.099; action and hazard accuracy move by less than 0.02. Second, missing steps: against the clean score on the same long-prefix subset (robustness.{condition}.clean_same_subset), one dropped step adds 2.42 m of endpoint error and two add 5.22 m, against 0.99 m and 1.30 m for LIM3D-XL. The motion MLP reads absolute step positions and timing through a fixed six-step window, so a gap in the track distorts its velocity summary far more than the transformer’s. Stop recall rises under dropped steps while stop precision falls, from 0.893 to 0.542 on that subset: a gappy track reads as slower, more hesitant motion and the model leans toward stop, a safe direction but not a gain in accuracy. These two findings are what the warning string on the hosted route summarises.

Calibration #

Expected calibration error of the max-softmax confidence over ten equal-width bins on the held-out set (calibration.{head}.ece): intent 0.019, phase 0.003, action 0.065, hazard 0.069, advice 0.066, target 0.018. The action head’s mean confidence is 0.954 against an accuracy of 0.889, so it is over-confident, and about twice as much as LIM3D-XL’s (0.031); its reliability bins (calibration.action.bins) show that a confidence between 0.7 and 0.8 corresponds to an accuracy of about 0.56 and one between 0.8 and 0.9 to about 0.62, while readings above 0.9 — 17,105 of the 20,000 — are right 0.945 of the time. The target head is the best calibrated, because its confidence is low to begin with (mean 0.744 against accuracy 0.726).

Latency #

Batch-1 forward pass in PyTorch on an arm64 workstation CPU with eight threads (latency.forward_batch1, 300 calls): 0.93 ms median, 1.11 ms p95; with featurisation included, 1.82 ms median and 2.09 ms p95. LIM3D-XL costs 6.78 ms median on the same machine. The pure-scalar in-process runtime measured about 5.8 ms per read (the row above), and the hosted route adds network time on top of whichever runtime serves it.

Real-distribution sketches #

The synthetic held-out set uses the generator’s own sketch grammar. To check the model on sketches from the distribution a vision model actually produces, 200 sketches from the spatial environment corpus that Waymark was trained on (348 prefixes; consistency) were parsed back into simulator scenes — object names to kinds, band words to metres sampled inside the band — given a hidden goal and simulated with the Waymark Extra simulator, so the sketch text is real while the labels are known. The mover is the viewer, because those sketches contain no people.

The two models agree on the target 0.816 of the time, on the action 0.914 and on the intent 0.948, and their endpoints lie within 1 m of each other 0.647 of the time (consistency.agreement). The real sketch grammar — two to four objects, no people, the viewer as mover — is not harder than the generator’s; target accuracy is in fact higher here than on the synthetic set because these scenes hold fewer candidate objects.

Known gaps #

Every ground truth above is synthetic. The held-out set, the perturbation sweeps and the real-sketch check all use simulated traces, and the real-sketch check changes only the sketch distribution; no real tracker output, no real photographs and no human-labelled scene have been evaluated, so these numbers bound what a real track can expect rather than measure it. That bound is tighter for LIM3D than for LIM3D-XL: the jitter and dropped-step sweeps show a model that was trained on clean coordinates and reads noisy ones badly, and real trackers produce noise the sweeps do not model — drift, occlusion, identity switches between tracked people. Truncation keeps the labels of the full prefix, so its rows say how little motion the model can anticipate from, not how accurate it is: with one step visible, action accuracy falls to 0.695 and endpoint error to about 6 m, so a host should wait for two or three steps before trusting a reading. Stop recall of 0.823 means about one simulated must-stop situation in six is read as caution or proceed by the raw head; the hazard-road≥0.6 branch of the decision rule adds a second chance, but the combined recall has not been measured separately. Latency was measured in PyTorch, not in the scalar runtime that serves the route. Estimates, never measurements.

API #

LIM3D is served by the Falcon API at /v1/intention3d, the route that LIM3D-XL answers by default: a request whose body carries model set to lim3d is answered by LIM3D instead, with the same request shape, the same response shape, the same errors and the same quota. The response names the size that answered in model and, when it is LIM3D, carries a fixed warning string. One call scores one sketch plus one track and counts once against the intention preview quota bucket. Conventions shared by every route — the response envelope, authentication, rate limits, retries — are on /docs/api.

Route /v1/intention3d · quota bucket intention · body limit 256,000 bytes.

POST /v1/intention3d HTTP/1.1
Authorization: Bearer fln_xxxxxxxxxxxxxxxx
Content-Type: application/json

Request body (limit 256,000 bytes):

{
  "sketch": "kind: scene\nviewer: standing at curb looking along roadway\nobjects: maple tree (center foreground, colour green foliage); two-lane road (center-right background, colour dark asphalt, size wider-than-frame); curb (center-right midground); ramp (center foreground, colour grey concrete, ground on-ground)\nrelations: maple tree next to ramp; ramp in front of two-lane road",
  "steps": [
    { "t": 0.0, "x": 0.0, "y": 0.0, "z": 0.0, "heading": "ahead", "notes": ["ahead: maple tree ~2m", "ahead: ramp ~2m", "ahead: curb ~9m"] }
  ],
  "mover": "you",
  "with_notes": true,
  "model": "lim3d"
}

sketch and steps are required; steps must have at least one entry and is truncated to 64. mover defaults to "you". with_notes defaults to true when any step carries notes. model selects the size: lim3d for this model, lim3d-xl or absent for LIM3D-XL; any other value answers 400 bad_model.

Response 200:

{
  "ok": true,
  "engine": "lim3d",
  "frame": "approx m; x right, y ahead, z up; viewer at origin",
  "model": "lim3d",
  "warning": "lim3d is the low-latency option: in evaluation it identifies the target less often than lim3d-xl (0.73 vs 0.89 on held-out scenes) and its endpoint estimate degrades sharply when steps are missing. Prefer lim3d-xl unless the ~7 ms it costs per call matters.",
  "result": {
    "intent": "pass_ahead",
    "pIntent": 1.0,
    "target": "none",
    "targetIndex": -3,
    "pTarget": 1.0,
    "phase": "walking",
    "hazard": "obstacle_ahead",
    "pHazard": 0.9994,
    "action": "caution",
    "pAction": 0.9519,
    "advice": "sidestep_left",
    "endpoint": { "dx": 0.246, "dy": 5.246, "dz": 0.205 },
    "eta": 4.648,
    "rule": "action-caution≥0.45",
    "model": "lim3d",
    "warning": "lim3d is the low-latency option: in evaluation it identifies the target less often than lim3d-xl (0.73 vs 0.89 on held-out scenes) and its endpoint estimate degrades sharply when steps are missing. Prefer lim3d-xl unless the ~7 ms it costs per call matters.",
    "summary": "You're just passing straight through, about 5m and 5s to go. Watch: sidestep left around the obstacle."
  }
}

engine is the fixed string lim3d, the name of the route; model, at the top level and again inside result, is the size that answered. warning is present, at both levels, whenever LIM3D answered; it is one fixed sentence per version, quoting the held-out target accuracies of the two sizes (0.726 and 0.885, rounded) and the missing-step finding from the Evaluation chapter, and is absent when LIM3D-XL answered. frame is constant and is returned so a caller never has to guess the axis convention. summary is a plain-words rendering of the result in a casual register, safe to replace with the host’s own wording.

Errors:

Quota bucket: intention (shared with /v1/intention; one request counts once whichever size answers). Body limit: 256,000 bytes. GET /health reports "lim3d": true once a 3D intention model is loaded and lists the loaded sizes in lim3dVariants.

Runtime & deployment #

The hosted endpoint has served this version since 2026-09-13: the Falcon API process loads the LIM3D bundle beside the LIM3D-XL bundle and answers /v1/intention3d with LIM3D when the request selects it with model set to lim3d. The Falcon API runs on 2 vCPU / 2 GiB and is kept warm with one minimum instance, so there is no wait for the service to start; LIM3D answers through the API in well under a second.

LIM3D runs in-process: the runtime is pure scalar TypeScript with no native dependency and no network call. The weights are one float32 bundle of 33 tensors (66,068,508 bytes) plus a metadata file carrying the label catalogues, feature-shape constants, metrics and the six parity fixtures.

• Load behaviour. The first call reads both files from the model directory, builds the tensor map and logs a ready line with the parameter count and size preset; every later call reuses the loaded pack. If the directory is missing the runtime marks the model absent and in-process reads return null; the Falcon API answers 503 lim3d_unavailable only when neither size is loadable. A caller that sees 503 during a cold start should wait about 2.5 s and retry once.
• Selection. On the hosted route, selection is per request: model set to lim3d is answered by LIM3D, model absent or lim3d-xl by LIM3D-XL, and the response’s model field names which. In a host that embeds the runtime, the model directory is chosen with LIM3D_MODEL_DIR; one loader holds one bundle, and a deployment can switch 3D intention reads off altogether, in which case in-process reads return null and no weights are loaded.
• What the host provides. A sketch in the Falcon scene format, steps in the viewer frame, and the mover’s sketch label when the mover is not the viewer. The host decides how much of a track to show; a common choice is the first 60 % of a planned or observed path, with notes on, so the model has something to anticipate. A host that calls the Falcon API provides only the request.
• Resources. About 66 MB resident once loaded; single-digit milliseconds per read on a laptop CPU in the scalar runtime (0.93 ms median at batch 1 in PyTorch on a workstation CPU), rising with object count and text length; one synchronous forward pass per call, so throughput scales with the host’s worker count, not with threads inside the model.

Integration notes #

• Call LIM3D after the scene has been sketched and after at least one step exists; with zero steps the runtime returns null rather than guessing.
• Feed the responder summary (or your own rendering of intent, target and advice) as context, and gate on action: on stop, do not phrase the reply as encouragement to continue; on caution, surface advice.
• Keep with_notes consistent with the notes’ provenance. Notes from a simulator that names objects and distances help the text bag; free-form notes from another source may not.
• On the hosted route the two sizes are loaded side by side and the request’s model field selects; read model in the response to know which answered, and treat the presence of warning as the signal that the low-latency size did. In a host that embeds the runtime, one loader holds one bundle, chosen with LIM3D_MODEL_DIR.

Limits & safety #

It does not see the image; it reads the sketch and the observed points, and anything the sketch omits or mislabels is invisible to it.

• Object positions are reconstructed from band words alone (three depths, five lateral bands), so every distance, endpoint and ETA is a coarse estimate. Estimates, never measurements.
• Not a safety system. stop means the model expects a road with no crosswalk in the sketch, not that a real road is clear or busy; it must not be the sole gate for physical guidance.
• Predicting another person’s intention is a plausible read of a simulated scene, not knowledge of what they will do.
• Trained only on simulated walking-speed motion with clean coordinates and no augmentation; real tracker output — jitter, dropped frames, vehicles, non-walking motion — is out of distribution. Measured: target accuracy falls from 0.726 to 0.555 under σ = 0.5 m of jitter and endpoint error grows from 1.66 m to 6.03 m with two steps dropped, where LIM3D-XL holds 0.786 and 1.90 m.
• Only the last six steps carry per-step features, only twelve sketch objects are placed, and character n-grams cover only the last 360 characters of the joined text.
• Target is the weakest head (0.726 on the held-out set, against LIM3D-XL’s 0.885) and the raw action head misses about one true stop in six; the decision rule narrows but does not close that gap. The hosted route says so in a warning on every LIM3D reading.
• Probabilities are over-confident (action-head calibration error 0.065, mean confidence 0.954 against accuracy 0.889 on the held-out set); a pIntent of 1.0 on an unfamiliar scene is a symptom of that, not of certainty.

Out of scope: vehicles, indoor navigation, crowd flow, any scene without a Falcon-format sketch, metric localisation, and any decision where a wrong proceed could cause physical harm without a human or a sensor in the loop.

Fixed weights per version; the model does not learn from requests.

Versions #

Compatibility. A major version bump changes the input or output contract — the request fields, the label sets, the frame, or the meaning of action and rule. A minor bump is a retrain with the same contract (new weights, possibly new metrics, same fields and labels); a patch bump changes metadata or runtime only. Within 1.x, the label sets listed under Inputs & outputs, the 15-slot target head, the 77-float motion vector, the 12-object limit and the decision-rule thresholds are stable, so a host that reads action, rule and the label strings will not need changes between minor versions.

Current weights. Version 1.0.0 is the large size preset from Vertex AI job lim3d-20260912-171944 (2026-09-12): lim3d.bin, 66,068,508 bytes, 33 float32 tensors, 16,516,905 parameters, accompanied by its metadata file (catalogues, constants, metrics and six parity fixtures) and the run record. The base preset (10,493,481 parameters) is the trainer’s default but is not a published version.