LIM3D-XL

Overview #

LIM3D-XL watches how someone moves through a photographed scene and tells you where they are going and whether they should stop, looking at every object and every step before it answers. You give it a written sketch of the scene and the first few timed steps of the movement; it gives back the same readings as LIM3D, more often right, a little later. Use it unless you need the answer in single-digit milliseconds.

LIM3D-XL (Large Intention Model 3D, enhanced) is a scene-and-motion intention reader: given a Falcon scene sketch and the first seconds of an observed track through that scene, it returns what the mover is trying to do, which sketched object they are heading for, what they are doing right now, the next hazard on their line, a proceed / caution / stop call, the advice that follows from it, where the motion ends and in how many seconds. It is the same job, the same eight readings and the same label vocabularies as LIM3D; what changes is the middle of the network.

LIM3D summarises a scene by pooling its objects into one vector. LIM3D-XL keeps every sketch object and every observed step as its own token and runs six layers of self-attention over the set, so a relation such as “the cyclist is closing on the planter box two steps ahead” is something the model can represent directly instead of recovering from averages. It also watches twelve observed steps instead of six, accepts up to sixteen objects instead of twelve, hashes text into twice as many buckets, and was trained with the kind of noise a tracker produces — jittered positions and dropped frames — so a slightly wobbly track still reads cleanly.

It belongs to the intention family alongside ELIM, LIM and LIM3D. Its input comes from two upstream sources the host application already has: a vision model that emits the Falcon scene sketch, and a tracker or simulator that produces steps in the viewer frame (Waymark Extra traces are in that frame already). Its output is a small JSON record the host application folds into whatever it does next; a stop is meant to forbid a “keep going” reply.

Intended use #

• Reading the intent, target and next hazard of a person (or the viewer) moving through a sketched outdoor scene, from a short observed track.
• Deciding, before a responder answers a movement question, whether the answer may proceed, must carry a caution, or must stop.
• Estimating where a motion ends and how long it takes, in metres and seconds relative to the mover, for tracks with more than six steps or scenes with more than twelve objects.
• Tracks that come from a real tracker rather than a simulator, where positions jitter and frames drop.

Out of scope #

• Anything the sketch does not describe: the model never sees pixels, only the text the vision model produced.
• Precise geometry. Object positions are derived from band words (foreground / midground / background, left / centre / right), not from measured coordinates.
• Motion that is not a person walking, cycling or standing still; vehicles, animals and objects in flight were never simulated.
• Real-time gating at high frequency: one read costs on the order of 100 ms in the current runtime.

Choose LIM3D-XL when #

• The observed track is longer than six steps and the extra history matters; LIM3D only tokenises the last six.
• The scene holds thirteen to sixteen objects; LIM3D keeps only the first twelve of a sketch.
• The track comes from a real tracker with position jitter or missing frames, which LIM3D-XL was trained against and LIM3D was not.
• Target accuracy matters more than latency: LIM3D-XL scores 0.885 on target against LIM3D’s 0.726 on the same held-out scenes, and holds up under jittered and gappy tracks where LIM3D does not, at about seven times the forward cost (6.8 ms against 0.9 ms, batch 1 on a CPU).
• It is the default: a request to /v1/intention3d without a model field is answered by LIM3D-XL. Otherwise choose LIM3D, selected per request with model set to lim3d: it answers with the same readings, the same vocabularies and the same request shape at the lowest cost per read.

Specification #

Try it #

You send

A sketch of a trail with a stroller, a planter box and a stair flight, plus a short track of the viewer walking.

You get back

The viewer is heading for the weeds in the crack, about a metre away, with nothing in the way, so the call is proceed.

The same exchange as the API sees it:

{
  "sketch": "kind: scene\nviewer: on trail looking forward\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: person with stroller (center background, colour grey stroller); planter box (center-left foreground, size medium); weeds in crack (left midground); maple tree (left background, ground on-ground); stair flight (left foreground, colour steel)\nrelations: weeds in crack in front of person with stroller; maple tree behind planter box; person with stroller behind stair flight; planter box in front of weeds in crack; weeds in crack behind stair flight; planter box in front of person with stroller; weeds in crack left of center; maple tree left of center\nrelative: weeds in crack closer than person with stroller; planter box closer than maple tree; stair flight closer than person with stroller",
  "steps": [
    {"t": 0.0, "x": -0.1, "y": 0.0, "z": 0.0, "heading": "ahead-left", "notes": ["weeds in crack ~9m ahead", "ahead: planter box ~2m", "ahead: stair flight ~3m", "ahead: maple tree ~20m"]},
    {"t": 1.5, "x": -2.2, "y": 0.7, "z": 0.0, "heading": "ahead-left", "notes": ["sidestep right around planter box", "weeds in crack ~7m ahead", "ahead-right: stair flight ~2m"]},
    {"t": 4.5, "x": -3.5, "y": 4.8, "z": 0.0, "heading": "ahead", "notes": ["weeds in crack ~3m ahead", "behind: stair flight ~3m"]},
    {"t": 6.0, "x": -4.2, "y": 6.9, "z": 0.0, "heading": "ahead", "notes": ["weeds in crack <1m ahead", "behind: stair flight ~6m"]}
  ],
  "mover": "you",
  "with_notes": true
}

{
  "ok": true,
  "engine": "lim3d",
  "frame": "approx m; x right, y ahead, z up; viewer at origin",
  "model": "lim3d-xl",
  "result": {
    "intent": "reach",
    "pIntent": 1.00,
    "target": "weeds in crack",
    "targetIndex": 2,
    "pTarget": 1.00,
    "phase": "walking",
    "hazard": "none",
    "pHazard": 1.00,
    "action": "proceed",
    "pAction": 1.00,
    "advice": "none",
    "endpoint": {"dx": -0.24, "dy": 0.71, "dz": 0.07},
    "eta": 0.48,
    "rule": "default-proceed",
    "model": "lim3d-xl",
    "summary": "You're heading for the weeds in crack, about 1m and 1s to go."
  }
}

Limits & safety #

It does not see the image; it reads the sketch. Everything below follows from that, from the synthetic training data, and from the size of the network.

• Object positions are estimates from band words — three depth bands and five lateral bands — so a “midground” object is at 7 m whether it is 4 m or 12 m away, and a stated “~3m ahead” in a note is text to the model, not geometry.
• “Stop” means the model believes the next event on the line is a road edge with no crosswalk, as the simulator defines it. It does not mean the road is dangerous, that traffic is present, or that stopping is safe.
• It was trained only on simulated pedestrians and a simulated viewer. A track that belongs to a vehicle, a cyclist moving at road speed, an animal or a thrown object was never seen and will be read as if it were a walking person.
• Intent for a person other than the viewer is a guess from a few seconds of motion and a text description; it cannot know what that person will decide to do.
• The window is twelve tokenised steps, sixteen objects and 2,048 hashed text features; the seventeenth object and the thirteenth-oldest step are invisible except through the global summary and the text bag.
• The augmentation covers ±0.15 m jitter and single dropped frames, not sensor drift, identity swaps between tracked people, occlusion or gaps of several seconds.
• One read costs on the order of 100 ms in the current runtime — about twenty times LIM3D — which makes it a poor fit for gating every frame of a video track.
• The response names the size that answered in model, and a warning accompanies every LIM3D reading; a host that gates on action should also check model when it relies on LIM3D-XL’s robustness to gappy tracks.

Related models #

Latest versions #