LIM3D

Overview #

LIM3D watches how someone moves through a photographed scene and tells you where they are going and whether they should stop. You give it a written sketch of the scene and the first few timed steps of the person’s movement; it gives back what they are trying to do, what is in their way, a proceed, caution or stop call, and a rough guess at where and when the movement ends. Use it when a quick read matters more than the most careful one.

LIM3D (Large Intention Model 3D) is a spatial-intention classifier. Given a Falcon scene sketch and the first seconds of an observed track for one mover — the viewer, or a person named in the sketch — it says what that mover is trying to do: which object they are heading for, what they are doing right now, what is next on their line, and whether the right call is proceed, caution or stop.

The input is text plus numbers. The sketch is the structured scene description a vision model emits (objects with lateral and depth bands, relations, a viewer line); the track is a short list of timed x/y/z points in the viewer frame, produced by a tracker or by a simulator. The output is six labels — intent, target, phase, hazard, action, advice — with confidences, plus two regressions only a spatial model can give: where the motion ends, in metres relative to the mover, and how many seconds until it does.

LIM3D belongs to the intention family and is the spatial counterpart of LIM. LIM reads a conversation window and returns trajectory, action, harm and steer; LIM3D reads a scene and a motion and returns the same shape of answer for a body moving through space. Its action head — proceed, caution, stop — is the 3D analogue of allow, steer, abort, and a host treats it the same way: as an advisory check before it says anything about moving on. Positions are expressed in one approximate frame: approx m; x right, y ahead, z up; viewer at origin. Objects are placed from the sketch’s band words, never from measured depth, so every distance, endpoint and ETA LIM3D returns is a coarse estimate. Estimates, never measurements.

Intended use #

• Reading the intention behind a partial walk in a sketched scene: reaching a named object, passing through, stepping aside, approaching the viewer, leaving the scene, or loitering.
• Naming the next hazard on the mover’s line — a road with no crosswalk, a person or obstacle ahead, a crosswalk, stairs or a ramp, a low structure — and the manoeuvre that follows from it.
• Gating a host’s movement guidance: stop and caution tell the responder not to encourage the mover onward without a second check.
• Estimating an endpoint and an ETA for a walk that is already under way, for hosts that want a rough “about 5 m and 5 s to go”.

Out of scope #

• Anything the sketch does not contain. LIM3D never sees pixels; an object missing from or mislabelled in the sketch is invisible to it.
• Vehicles, cycling at speed, running indoors, or any motion that does not look like a person walking through an outdoor scene.
• Real tracker output with jitter, dropped frames or metric-accurate depth; the model was trained on clean simulated coordinates in band-derived metres.
• Physical safety decisions. stop means the model expects a road with no crosswalk in the sketch, not that a real road is busy or clear.

Choose LIM3D when #

• The lowest cost per read is what matters: a single-digit-millisecond read from a 66 MB bundle in-process, or the low-latency option on the hosted route, selected per request with model set to lim3d; choose LIM3D-XL, the default on that route, when accuracy on the target and hazard heads matters more — on the same held-out scenes it reads the target right 0.885 of the time against LIM3D’s 0.726.
• The tracks are clean — simulated, or lightly filtered — and rarely longer than six meaningful steps; choose LIM3D-XL when the input is raw tracker output with jitter and gaps, which its augmentation covers and LIM3D’s does not.
• Scenes have twelve or fewer objects; choose LIM3D-XL when sketches routinely carry more.
• The question is about a conversation rather than a scene: choose LIM or ELIM instead.

Specification #

Try it #

You send

A sketch of a kerbside scene with a tree, a ramp and a road, plus a few timed steps of someone walking ahead.

You get back

They are passing ahead, an obstacle is coming up, the call is caution, and the advice is to sidestep left.

The same exchange as the API sees it:

{
  "sketch": "kind: scene\nviewer: standing at curb looking along roadway\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: maple tree (center foreground, colour green foliage); two-lane road (center-right background, colour dark asphalt, size wider-than-frame); curb (center-right midground); ramp (center foreground, colour grey concrete, ground on-ground)\nrelations: maple tree next to ramp; maple tree in front of two-lane road; maple tree in front of curb; curb behind ramp; ramp in front of two-lane road; two-lane road behind curb\nrelative: maple tree closer than two-lane road; maple tree closer than curb; ramp closer than curb",
  "steps": [
    {
      "t": 0.0,
      "x": 0.0,
      "y": 0.0,
      "z": 0.0,
      "heading": "ahead",
      "notes": ["ahead: maple tree ~2m", "ahead: ramp ~2m", "ahead: curb ~9m"]
    }
  ],
  "mover": "you",
  "with_notes": true
}

{
  "intent": "pass_ahead",
  "pIntent": 1.0,
  "target": "none",
  "targetIndex": -3,
  "pTarget": 1.0,
  "phase": "walking",
  "hazard": "obstacle_ahead",
  "pHazard": 0.9994,
  "action": "caution",
  "pAction": 0.9519,
  "advice": "sidestep_left",
  "endpoint": { "dx": 0.25, "dy": 5.25, "dz": 0.21 },
  "eta": 4.65,
  "rule": "action-caution≥0.45",
  "model": "lim3d"
}

Limits & safety #

It does not see the image; it reads the sketch and the observed points, and anything the sketch omits or mislabels is invisible to it.

• Object positions are reconstructed from band words alone (three depths, five lateral bands), so every distance, endpoint and ETA is a coarse estimate. Estimates, never measurements.
• Not a safety system. stop means the model expects a road with no crosswalk in the sketch, not that a real road is clear or busy; it must not be the sole gate for physical guidance.
• Predicting another person’s intention is a plausible read of a simulated scene, not knowledge of what they will do.
• Trained only on simulated walking-speed motion with clean coordinates and no augmentation; real tracker output — jitter, dropped frames, vehicles, non-walking motion — is out of distribution. Measured: target accuracy falls from 0.726 to 0.555 under σ = 0.5 m of jitter and endpoint error grows from 1.66 m to 6.03 m with two steps dropped, where LIM3D-XL holds 0.786 and 1.90 m.
• Only the last six steps carry per-step features, only twelve sketch objects are placed, and character n-grams cover only the last 360 characters of the joined text.
• Target is the weakest head (0.726 on the held-out set, against LIM3D-XL’s 0.885) and the raw action head misses about one true stop in six; the decision rule narrows but does not close that gap. The hosted route says so in a warning on every LIM3D reading.
• Probabilities are over-confident (action-head calibration error 0.065, mean confidence 0.954 against accuracy 0.889 on the held-out set); a pIntent of 1.0 on an unfamiliar scene is a symptom of that, not of certainty.

Related models #

Latest versions #