Input formats

Overview #

The LIM and Waymark models exchange five small formats. Waymark, Waymark Extra and both 3D Intention models read the scene sketch; Waymark Extra writes the movement trace; LIM3D and LIM3D-XL read the trace as observed steps; Waymark Flight reads the airspace scene; ELIM, LIM and LIM Nano read the conversation window. All of them are plain UTF-8 text or JSON, and every sketch-based spatial quantity shares one frame; the airspace scene has its own, described in its section.

The frame sentence, reused verbatim by every route that returns coordinates, is:

approx m; x right, y ahead, z up; viewer at origin

• x is metres to the right of the viewer (negative = left).
• y is metres ahead of the viewer (depth).
• z is metres up, with z = 0 at the viewer’s feet.
• Every value is an estimate derived from band words in the sketch. Estimates, never measurements.

Scene sketch #

The scene sketch is the text a vision model emits for one image or clip. It is the only thing Waymark, Waymark Extra, LIM3D and LIM3D-XL know about the scene: none of them sees pixels. The Falcon API accepts the sketch as the sketch string on /v1/spatial and /v1/intention3d.

Line grammar #

A sketch is a sequence of lines, each key: value, in this fixed order. Only the first line is mandatory; the others appear when the vision model had something to say.

Allowed shot values are phone, drone, mv (moving vehicle or transit camera), camera and unknown. orientation is portrait, landscape or square, optionally followed by a pitch word (looking-down, level, looking-up). height_agl is a rough camera height above ground in words, e.g. eye-level ~1.5-1.8m, drone ~20-40m, low ~0.5m, or unknown.

Relations use six fixed phrasings: A left of B, A in front of B, A behind B, A above B, A next to B, A blocking B; the sketch producer is asked for three to eight of them. The relative: line carries phrases such as A closer than B or A wider than B and never carries invented metres, GPS or compass degrees.

Object grammar #

Each object entry is:

name (lateral depth, colour c, size s, ground g)

• name is the object label, e.g. street tree, two-lane road, doorway. When the object moves, the name or position may say so briefly.
• The first item inside the parentheses is the position: a lateral band word, a space, and a depth band word. Occlusion notes may follow in the same item.
• colour c, size s and ground g are optional and appear in that order, separated by , . Any of them may be absent.
• An object with no details at all is written as its bare name.

Allowed band and detail words:

The band words are code tokens and keep their spelling (center, colour) exactly as shown.

From bands to the viewer frame #

LIM3D and LIM3D-XL place every object in the frame from its band words alone, using the same rule at training and inference time:

• y from the depth word: foreground = 2.75 m, midground = 7.0 m, background = 16.0 m.
• x = frac × 2 × 0.58 × y, where frac comes from the lateral word: left = −0.55, center-left = −0.25, center = 0, center-right = 0.25, right = 0.55. In other words the frame is treated as 2 × 0.58 × y metres wide at depth y, and frac is the object’s offset as a fraction of that width.
• z = 5.0 m when the object is overhead, otherwise 0.
• A missing or unrecognised band word defaults to center and midground.

The resulting grid, in metres:

An object counts as overhead when its ground word is overhead, or when its name matches an overhead keyword (overpass, bridge, canopy, deck, awning, ceiling, beam, sign overhead). Each object also receives a role and a footprint radius from its name — walk (surfaces you can walk on), block (ground obstacles), over (overhead structures), road (vehicle surfaces), goal (destination-only features such as a doorway), person, or unknown — which the model uses as a one-hot feature. See Output vocabularies for the role list.

Only the first objects: line is read. LIM3D keeps the first 12 objects in sketch order; LIM3D-XL keeps the first 16.

Example #

A complete sketch as a sketch producer emits it (taken from the Waymark Extra training set):

kind: scene
viewer: at stair landing
capture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m
objects: turn lane (right foreground, colour asphalt); street tree (center-right foreground); bike lane (right midground); guardrail (right background, size medium); sidewalk (left foreground)
relations: sidewalk in front of bike lane; turn lane next to street tree; guardrail behind turn lane; sidewalk left of turn lane; bike lane behind turn lane; guardrail behind sidewalk; turn lane right of center; bike lane right of center
relative: sidewalk closer than bike lane; turn lane closer than guardrail; turn lane closer than bike lane
uncertain: exact clearance height unknown

Sent as JSON, the same sketch is one string with \n between lines:

{
  "sketch": "kind: scene\nviewer: at stair landing\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: turn lane (right foreground, colour asphalt); street tree (center-right foreground); bike lane (right midground); guardrail (right background, size medium); sidewalk (left foreground)\nrelations: sidewalk in front of bike lane; turn lane next to street tree; guardrail behind turn lane; sidewalk left of turn lane; bike lane behind turn lane; guardrail behind sidewalk; turn lane right of center; bike lane right of center\nrelative: sidewalk closer than bike lane; turn lane closer than guardrail; turn lane closer than bike lane\nuncertain: exact clearance height unknown",
  "question": "which way to the bike lane"
}

Movement trace #

Waymark Extra reads a sketch plus a movement goal and writes a timed trace of one mover walking through the scene. The prompt it was trained on is the sketch, a blank line, Move: followed by the goal, and a header line; the model continues from the header:

Spatial sketch:
…sketch lines…

Move: head to the bike lane
Trace (approx m; x right, y ahead, z up):

When the mover is a person in the scene rather than the viewer, the header ends for followed by the mover’s sketch label (Trace (approx m; x right, y ahead, z up) for pedestrian:).

Step lines #

Every trace line matches one pattern:

t=…s (x, y, z) heading | note | note
arrive t=…s (x, y, z) target | note | note

Formally, a line is ^(arrive )?t=T s (X, Y, Z) REST$ where T, X, Y, Z are decimal numbers (X, Y, Z may be negative) and REST is split on | . Values are written with one decimal place.

• t is seconds since the trace began; consecutive steps are typically 1.5 s apart.
• (x, y, z) is the mover’s position in the viewer frame. The viewer starts at (0.0, 0.0, 0.0); a person in the scene starts at that object’s band position.
• On an ordinary line the first item after the coordinates is a heading word: ahead, ahead-left, ahead-right, left, right, back-left, back-right, back, or still.
• On the final line the word arrive precedes t=, and the first item after the coordinates is the target label — the sketch object reached, or a free-walk label such as 4m ahead or 6m left.
• Every later item is a note. Notes are either surroundings (ahead: planter box ~4m, left: turn lane ~2m, bike lane ~7m ahead, street tree behind, tree canopy overhead) or events (sidestep left around street tree, give way right for pedestrian, stop at edge of service road, no crosswalk in sketch — don't cross here, cross two-lane road at crosswalk, overhead: concrete overpass ~5m up, climbing stair flight, up the ramp, climbed ~2m up). Distances in notes are rounded: <1m, ~Nm below 12 m, then to the nearest 5 m.
• A trace that ends with a still line and a stop at edge of … note has no arrive line: the mover stopped short of a road with no crosswalk in the sketch.

Waymark Extra also writes two-leg traces (“go to A, then B”): the first leg ends in a still line with at A and next: B notes, and the second leg continues from that point with its times shifted accordingly.

Example #

The trace for the sketch above and the goal head to the bike lane (a training row, mover you):

t=0.0s (0.0, 0.0, 0.0) ahead-right | bike lane ~7m ahead | ahead: turn lane ~2m | ahead: street tree ~3m | ahead-left: sidewalk ~3m
t=1.5s (2.0, 0.8, 0.0) ahead-right | sidestep left around street tree | bike lane ~5m ahead | ahead-left: turn lane ~2m
t=3.0s (2.5, 2.8, 0.0) ahead | bike lane ~3m ahead | left: turn lane ~2m
t=4.5s (3.0, 4.8, 0.0) ahead | bike lane ~1m ahead | left: street tree ~3m
arrive t=5.3s (3.2, 5.8, 0.0) bike lane | bike lane ahead | street tree behind

Observed steps (JSON) #

Parsed, each trace line becomes one step object. This is the form LIM3D and LIM3D-XL read, and the form /v1/intention3d accepts in its steps array:

The route keeps at most 64 steps and drops any step whose t, x, y or z is not a finite number. Alongside steps, the request carries the sketch, an optional mover (default "you"; otherwise the sketch label of the person moving) and an optional with_notes flag that tells the model whether to read the notes (default: true when any step has notes). Internally the steps are rendered back into trace lines under the header Observed (approx m; x right, y ahead, z up): — with for and the mover label when the mover is not the viewer — and appended to the sketch as the model’s text input.

{
  "sketch": "kind: scene\nviewer: standing at curb looking along roadway\nobjects: maple tree (center foreground, colour green foliage); two-lane road (center-right background, colour dark asphalt, size wider-than-frame); curb (center-right midground); ramp (center foreground, colour grey concrete, ground on-ground)\nrelations: maple tree next to ramp; ramp in front of two-lane road",
  "steps": [
    {
      "t": 0.0,
      "x": 0.0,
      "y": 0.0,
      "z": 0.0,
      "heading": "ahead",
      "notes": ["ahead: maple tree ~2m", "ahead: ramp ~2m", "ahead: curb ~9m"],
      "arrive": false
    }
  ],
  "mover": "you",
  "with_notes": true
}

A host that has a full simulated or tracked path usually sends only its opening portion — the first 60 % of the steps, rounded up, and never the arrival step — so that the model is asked what the mover is about to do rather than what it already did.

Airspace scene #

The airspace scene is the JSON Waymark Flight reads on /v1/airspace/read and /v1/airspace/plan: a volumes array of at most 24 cylinders or rings, each with a floor and a ceiling. It has its own frame, unrelated to the viewer frame above:

• Horizontal positions are nautical miles in a local east/north frame: x_nm east, y_nm north, with any origin the host chooses. Waymark Flight works relative to the mover, so the origin does not change the answer.
• Altitudes are feet MSL; times are seconds.
• A volume contains a point when inner_radius_nm ≤ distance < radius_nm and floor_ft ≤ altitude < ceiling_ft; a floor_ft of null means the volume starts at the surface.
• The host supplies every volume. The scene is not real airspace data unless the host made it so, and Waymark Flight is not certified for navigation.

A Class B area around a hand-built “Field A”, with its surface area and first shelf (illustrative dimensions):

[
  {"id": "B0", "kind": "class_b", "center": {"x_nm": 0, "y_nm": 0}, "radius_nm": 5, "floor_ft": null, "ceiling_ft": 10000, "airport": "Field A", "tier": 0},
  {"id": "B1", "kind": "class_b", "center": {"x_nm": 0, "y_nm": 0}, "radius_nm": 10, "inner_radius_nm": 5, "floor_ft": 3000, "ceiling_ft": 10000, "airport": "Field A", "tier": 1}
]

A track point is {t_s, x_nm, y_nm, alt_ft} in the same frame; a track has 1–12 points in ascending t_s, newest last. Start and goal points of a plan use {x_nm, y_nm, alt_ft}. The catalogues the scene uses — volume kinds, movers, tiers, event kinds and advisories — are on Output vocabularies.

Conversation window #

ELIM, LIM and LIM Nano read a short transcript rather than a single line. The window is built from the last three prior turns plus the current turn, each on its own line, tagged U: for the user and A: for the responder, and cut to 1,500 characters from the start:

U: remind me in 20 minutes
A: ok — 20 min.
U: is the message from my bank asking for a code a scam

• Turns are trimmed of surrounding whitespace; the tag is followed by one space.
• Older turns beyond the last three are not included. Responder turns that merely repeat a standard safety notice are dropped before the three-turn cut.
• The current turn is always the final U: line.

Over the Falcon API the text field of /v1/intention is the current turn only, so the window the model sees is a single U: line; a host that runs ELIM, LIM or LIM Nano in-process supplies the prior turns itself.

Where each format is used #

Routes, envelopes, limits and error codes are on API conventions. Every label a model can return is listed on Output vocabularies.