Waymark Extra
Movement traces from a scene sketch and a goal, with a plain-words summary
On this page
Overview #
Waymark Extra describes how someone would walk through a photographed scene. You give it a written sketch of the scene and a goal such as “walk to the door”, which can have a stop on the way or involve a person coming toward you; it gives back the walk step by step — where the person is at each moment, which way they face, what they do on the way — and a one-line summary you can show as it is.
Waymark Extra is the movement model of the Waymark series. It reads a Falcon scene sketch — the structured text a vision model emits for a photographed scene — together with a movement goal such as “walk to the stair flight, then over to the plaza paving”, and returns a timed trace of a person moving through that scene. Each line of the trace gives a time, a position in approximate metres, a heading, and short notes about what is ahead, beside or behind the mover. The Falcon API parses the trace into step objects and adds a plain-words summary built from the last step and the events along the way.
Where Waymark answers layout questions, Waymark Extra simulates walks of up to twelve steps. Its 1024-token window fits a long sketch and a long trace, and its training set covers two-leg routes with a waypoint, a person walking toward the viewer, a person heading off into the background, and a person who simply waits. Scenes in its data are busy — a second person, an extra road, a flight of stairs — so give-ways, road-edge stops and climbs are common rather than rare. It keeps Waymark’s layout question-and-answer ability in the same weights.
Everything it returns is an estimate reconstructed from the sketch’s depth and lateral bands. It never sees pixels, and its coordinates are approximate metres in a viewer-relative frame: x right, y ahead, z up, with the viewer at the origin.
Intended use #
- Turning a goal such as “walk to the doorway” into a plausible timed path through a sketched scene, with a summary a host can show or hand to its responder.
- Predicting where a named person in the sketch — a jogger, a cyclist, a dog walker — would plausibly end up, including approach toward the viewer, departure into the background and waiting.
- Producing steps in the viewer frame that an intention model such as LIM3D-XL can read for intent, hazards and advice when no tracker output is available.
- Two-leg routes with a stop on the way, and busy scenes with more than one person.
Out of scope #
- Navigation, accessibility or safety decisions; distances are estimates, not measurements.
- Predicting what a real person will actually do; a trace is a plausible simulation, not a forecast.
- Scenes or objects outside the fifty-kind sketch vocabulary, and sketches whose kind is not scene.
- Answering inside a tight latency budget: a trace takes seconds on a GPU, and a cold start takes minutes.
Choose Waymark Extra when #
- The question is about movement through the scene rather than static layout: how someone gets somewhere, and what they do on the way.
- The goal is a two-leg route, an approach toward the viewer, a walk into the background or a loiter, or the scene is long or busy.
- Only a layout question is being asked — choose Waymark, which is the smaller model built for exactly that and answers in well under a second.
- The host has an observed track rather than a goal — choose LIM3D-XL or LIM3D, which read steps and return intent, hazard and advice labels.
Architecture #
| Parameters | 354,650,112 |
|---|---|
| Layers | 24 pre-LN decoder blocks |
| Width | 1024 |
| Attention heads | 16 × 64 |
| MLP hidden size | 4096, GELU (tanh approximation) |
| Context window | 1024 tokens, learned absolute positions |
| Vocabulary | 50,304 (GPT-2 BPE, padded) |
| Embeddings | Token embedding tied to the output head; no bias on linear layers |
| Attention kernel | Fused causal scaled-dot-product attention |
| Training precision | bf16 autocast, fp32 master weights |
| Checkpoint | model.pt, 1,418,678,978 bytes (fp32), best step 1750 |
| Serving device | One NVIDIA L4 GPU, scale to zero |
Waymark Extra is a decoder-only transformer in the GPT-2-medium shape: 24 pre-LayerNorm blocks, a 1024-wide residual stream, 16 attention heads of 64 dimensions each, and a 4096-wide MLP with the tanh approximation of GELU. Token embeddings are tied to the output head, no linear layer carries a bias, and positions come from a learned table of 1024 entries — twice the window of Waymark, which is what allows a full sketch and a twelve-line trace to share one context. Attention uses the fused causal scaled-dot-product kernel, chosen to keep memory manageable at the longer block size.
Text is tokenised with the GPT-2 byte-pair vocabulary (50,257 tokens padded to 50,304). The model has no separate heads: the trace, its headings and its notes are all generated as text and parsed afterwards. The closed vocabularies that appear in that text — headings, relative-position words, event notes, mover labels and the two fixed target labels — are listed under Output and on the vocabularies page.
The model has no vision component: it never sees pixels, only the sketch a vision model wrote about them. It also has no planner and no map; obstacle avoidance and road rules exist only as patterns learned from the synthetic traces.
Comparison #
| Waymark | Waymark Extra | |
|---|---|---|
| Job | Layout question → one-line answer | Sketch + goal → trace and summary |
| Parameters | 123,999,744 | 354,650,112 |
| Layers × width | 12 × 768 | 24 × 1024 |
| Context window | 512 tokens | 1024 tokens |
| Trace length | — | up to 12 lines |
| Goal kinds | — | walk to, free walk, person to target, two-leg, toward viewer, leave, loiter |
| Warm start | — | none (from scratch) |
| Serving device | CPU | one L4 GPU |
| Generation cap | 96 tokens | 480 tokens |
| Falcon API route | /v1/spatial | /v1/move |
The two models read the same sketch grammar and share one frame. Waymark Extra also answers Waymark’s layout completion, so one service can serve both, but the Falcon API binds /v1/spatial to Waymark and /v1/move to Waymark Extra.
Inputs & outputs #
Input #
| Field | Type | Required | Description | Limit |
|---|---|---|---|---|
sketch | string | Yes | The Falcon scene sketch as text (kind, viewer, objects, relations, relative, uncertain lines). | At most 6,000 characters over the Falcon API; older sketch tokens are dropped when the prompt exceeds the 1024-token window minus the trace budget |
goal | string | Yes | The movement request in plain words, for example walk to the doorway, or the jogger heads off into the background. | At most 300 characters; goals shaped like the training templates work best |
mover | string | No | Who moves: omit for the viewer, or name a person label from the sketch (pedestrian, jogger, dog walker, person with stroller, cyclist). | One of the five person labels; cut to 40 characters |
max_new_tokens | integer | No | Generation budget for the trace. The default of 400 leaves room for twelve-step traces. | Clamped to 32–480 |
temperature | number | No | Sampling temperature. The default 0.2 decodes greedily, so the same request returns the same trace. | Clamped to 0–1; values at or below 0.25 are greedy, top-k is fixed at 40 |
The service builds one prompt from the request: the sketch under a “Spatial sketch:” header, a blank line, a “Move:” line carrying the goal, and the trace header “Trace (approx m; x right, y ahead, z up):”. When a mover other than the viewer is named, the header ends “for {mover}:” instead. The sketch format itself is documented on the input formats page.
{
"sketch": "kind: scene\nviewer: eye-level on path\ncapture: shot phone; orientation portrait, looking-down; height_agl eye-level ~1.5-1.8m\nobjects: roadway below (right midground); sidewalk (right midground, colour grey concrete); retaining wall (left foreground); ramp (center-left midground); concrete overpass (right background, colour grey concrete); crosswalk (left foreground); dog walker (center-right foreground)\nrelations: roadway below behind crosswalk; roadway below next to sidewalk; retaining wall in front of sidewalk; crosswalk left of retaining wall; ramp behind retaining wall; ramp left of sidewalk; roadway below right of center; crosswalk left of center\nrelative: crosswalk closer than roadway below; retaining wall closer than sidewalk; retaining wall closer than ramp\nuncertain: true distances not visible",
"goal": "the dog walker walks toward you",
"mover": "dog walker",
"max_new_tokens": 400,
"temperature": 0.2
}Limits on the input:
- Over the Falcon API
sketchis at most 6,000 characters andgoalat most 300; longer values answer413 text_too_long. - The prompt budget is the 1024-token window minus the smaller of
max_new_tokensand 512. A longer prompt loses its oldest sketch tokens first, so very long object lists are truncated from the top of the sketch. max_new_tokensis clamped to 32–480 and further capped so that prompt plus trace fit the window.movermust be one of the five person labels; the goal should refer to objects by the labels used in the sketch.- The frame is fixed: approx m; x right, y ahead, z up; viewer at origin. Depth bands map to roughly 1.5–4 m (foreground), 4–10 m (midground) and 10–22 m (background); the lateral spread assumes a field of view of about 60 degrees.
Output #
| Field | Type | Description |
|---|---|---|
engine | string | Always waymark-extra, so a host can tell which model answered the route. |
frame | string | Always the sentence approx m; x right, y ahead, z up; viewer at origin. |
mover | string | Echo of the mover, or you when none was given. |
trace | string | The trace as newline-separated lines in the Falcon trace grammar, cut at the first arrival or road-edge stop. |
steps | array | One object per parsed line: t, x, y, z, heading, target, notes[], arrive. Lines that do not match the grammar are skipped. |
summary | string | One or two plain sentences built from the last step and the events along the way, ready to show or to hand to a responder. |
heading 10 labels
aheadahead-leftahead-rightleftrightback-leftback-rightbackstillarrive
relation 8 labels
aheadahead-leftahead-rightleftrightbehindhereoverhead
event 15 labels
sidestepgive waystop at edgeno crosswalk in sketchcross at crosswalkoverheadclimbing stair flightup the rampclimbedatnextstanding stilllooking aroundwaitingchecking phone
mover 6 labels
youpedestrianjoggerdog walkerperson with strollercyclist
target 2 labels
youbackground
{
"ok": true,
"engine": "waymark-extra",
"frame": "approx m; x right, y ahead, z up; viewer at origin",
"mover": "dog walker",
"trace": "t=0.0s (0.9, 3.2, 0.0) back | you ~3m ahead | behind: sidewalk ~2m | left: roadway below ~2m | right: ramp ~2m\nt=1.5s (0.4, 1.8, 0.0) back | you ~1m ahead | ahead-right: retaining wall ~2m\narrive t=2.6s (0.1, 0.8, 0.0) you | retaining wall right | crosswalk right",
"steps": [
{"t": 0.0, "x": 0.9, "y": 3.2, "z": 0.0, "heading": "back", "notes": ["you ~3m ahead", "behind: sidewalk ~2m", "left: roadway below ~2m", "right: ramp ~2m"], "arrive": false},
{"t": 1.5, "x": 0.4, "y": 1.8, "z": 0.0, "heading": "back", "notes": ["you ~1m ahead", "ahead-right: retaining wall ~2m"], "arrive": false},
{"t": 2.6, "x": 0.1, "y": 0.8, "z": 0.0, "heading": "arrive", "target": "you", "notes": ["retaining wall right", "crosswalk right"], "arrive": true}
],
"summary": "The dog walker reaches the you in about 3s, ending ~0m straight and ~1m ahead of where the photo was taken."
}Each trace line follows one grammar, described in full on the input formats page:
t={t}s ({x}, {y}, {z}) {heading} | {note} | {note} ...
arrive t={t}s ({x}, {y}, {z}) {target label} | {note} | {note}How to read the result:
headingon an ordinary line is one of nine words: ahead, ahead-left, ahead-right, left, right, back-left, back-right, back, still. On the arrival line the parsed step carriesheading: "arrive"and the destination intarget;targetis absent on other lines.- A waypoint on a two-leg route is a
stillline whose notes readat {first stop}andnext: {second stop}; the times of the second leg continue from it. - A loiter trace is three to five
stilllines with a note such aswaitingorlooking aroundand no arrival line;arrivestays false on every step. - A walk toward the viewer ends with
target: "you"about 0.7–1.2 m ahead of the origin; a walk into the background ends withtarget: "background"at roughly 19–22 m. - When the viewer’s path crosses a road and the sketch contains no crosswalk, the trace ends on a
stillline notingstop at edge of {road}andno crosswalk in sketch — don't cross here; there is no arrival. - Distances in notes are rounded:
<1mbelow one metre, to the nearest metre below 12 m, and to the nearest 5 m beyond. Coordinates carry one decimal. summaryis assembled by the Falcon API from the last step (arrival target and time, or where the mover stopped) and up to four distinct event notes (sidestep,give way,stop at edge,cross,overhead,climb); it is a template, not generated text, and reads the target label verbatim, so a walk toward the viewer says “the you”.- There is no confidence score. Decoding is greedy at the default temperature, so a repeated request returns the same trace; a host that wants variety raises
temperature. - The
stepsarray can be shorter than the trace if a generated line does not match the grammar; the Falcon API answers502 move_failedwhen no step parses at all.
Examples #
A two-leg route for the viewer, taken from the training set. The first arrival is rewritten as a waypoint line and the second leg’s times continue from it:
{
"sketch": "kind: scene\nviewer: eye-level on path\nobjects: potted plant (center-left background, colour terracotta); bike lane (center-left midground, ground on-ground); roadway below (center-left foreground, colour dark asphalt, size large); sidewalk (center-right foreground, size medium); weeds in crack (left foreground); retaining wall (center-right background)\nrelations: weeds in crack in front of potted plant; weeds in crack in front of bike lane; roadway below in front of potted plant; sidewalk in front of potted plant; bike lane in front of potted plant; retaining wall behind potted plant; weeds in crack left of center\nrelative: weeds in crack closer than potted plant; weeds in crack closer than bike lane; roadway below closer than potted plant\nuncertain: true distances not visible",
"goal": "go to the potted plant first and then the roadway below"
}t=0.0s (0.0, 0.0, 0.0) ahead | potted plant ~15m ahead | ahead: roadway below ~2m | ahead: sidewalk ~2m | ahead: weeds in crack ~3m
t=2.5s (-0.8, 3.1, 0.0) ahead | potted plant ~9m ahead | left: weeds in crack ~1m
t=5.0s (-1.7, 6.2, 0.0) ahead | potted plant ~6m ahead | ahead: bike lane ~1m
t=7.5s (-2.5, 9.3, 0.0) ahead | potted plant ~3m ahead | behind: bike lane ~2m
t=9.8s (-3.3, 12.2, 0.0) still | at potted plant | next: roadway below
t=11.3s (-2.8, 10.3, 0.0) back | roadway below ~4m ahead | ahead: bike lane ~3m
t=12.8s (-2.3, 8.5, 0.0) back | roadway below ~3m ahead | ahead: bike lane <1m
t=14.3s (-1.7, 6.6, 0.0) back | roadway below <1m ahead | behind: bike lane ~1m
arrive t=14.8s (-1.6, 6.0, 0.0) roadway below | bike lane behind | weeds in crack aheadA loiter goal for a named person. The trace has no arrival line, every heading is still, and the position jitters within about 0.3 m:
{
"sketch": "kind: scene\nviewer: standing at curb looking along roadway\nobjects: shrub bed (right foreground); parking aisle (center foreground); dog walker (left foreground, size medium); curb (center-left foreground)\nrelations: dog walker left of curb; shrub bed right of parking aisle; dog walker left of shrub bed; curb left of shrub bed; parking aisle next to curb; dog walker left of parking aisle",
"goal": "the dog walker stays where they are",
"mover": "dog walker"
}t=0.0s (-2.4, 3.6, 0.0) still | checking phone | here
t=1.5s (-2.6, 3.5, 0.0) still | waiting | here
t=3.0s (-2.4, 3.9, 0.0) still | standing still | here
t=4.5s (-2.3, 3.6, 0.0) still | checking phone | hereThe second note on each loiter line is a generator artefact: the target label is empty for a loiter, so the note reads “ here” with nothing before it. Hosts should ignore notes that consist only of a relation word.
Training #
Waymark Extra was trained from scratch — it was not warm-started from Waymark — as a Vertex AI custom job started on 2026-09-12 using two NVIDIA A100 40 GB GPUs with DistributedDataParallel; the best checkpoint was fetched and its weights declared final on 2026-09-13.
The training file holds 138,200 rows. 120,000 are movement examples produced by the Falcon movement generator in its extended mode (seed 2027): scenes are drawn from the same fifty-kind vocabulary and ten viewer phrases as Waymark, a person is present in about 55 percent of scenes, and the extended mode adds a second person to 35 percent of scenes, an extra road to 30 percent and an extra stair flight or ramp to 20 percent. Thirty percent of movement examples use one of the extended goal kinds — toward the viewer, leave, loiter, or (twice as often) two-leg. The remaining 18,200 rows are the 2,275 Waymark layout question-and-answer rows repeated eight times, so the model keeps its layout skill while movement rows dominate. Measured on a 20,000-row sample, movement rows average 414 GPT-2 tokens (maximum 765; about one in five exceeds 512, none exceed 1024) and 5.19 trace lines (maximum 12). The whole set is estimated at 51.7 million tokens. Coverage measured on the first 21,800 movement rows: arrival line 88.7 percent, sidestep 18.8, overhead note 13.2, two-leg waypoint 12.1, crosswalk crossing 7.7, road-edge stop 6.6, give way 6.0, loiter 3.2, ramp 2.3, stair climb 2.2. Movers: the viewer in about 58 percent of rows and roughly 8 percent each for cyclist, dog walker, jogger, pedestrian and person with stroller.
No real user data was used. Every sketch, goal and trace is synthetic; the generator is a small stdlib program with no access to photographs, requests or traces from real people.
Recipe: the medium configuration above, bf16 autocast, AdamW with betas (0.9, 0.95) and weight decay 0.1, gradient clipping at 1.0, peak learning rate 3e-4 with 300 warm-up steps and cosine decay toward 3e-5, seed 29. Each optimiser step sees 128 sequences of 1024 tokens (16 per GPU with gradient accumulation of 4 on each of two ranks), that is 131,072 tokens per step. The schedule allowed 6,000 steps with evaluation every 250 steps on 24 validation batches and early stopping after three evaluations without improvement; both ranks evaluate the same validation slice and average the loss so the stopping decision is identical everywhere. Validation rows are every tenth row of the file by index. Training stopped at step 2,500 after 58 minutes of wall time — about 328 million tokens, roughly seven passes over the estimated 47-million-token training split — with the best checkpoint at step 1,750.
What it was not trained on: real photographs or pixels; real trajectories, GPS tracks or measured distances; sketches of kind document or map; objects outside the fifty scene kinds; and any goal outside the templated families listed above.
Evaluation #
| Metric | Value | Source |
|---|---|---|
| Endpoint error, mean (300 held-out scenes) | 1.84 m | results.json of the 2026-09-13 Waymark evaluation — 2-D distance between the last model step and the simulator’s; 4.95 m for the 124M preview |
| Endpoint error, median | 0.92 m | results.json of the 2026-09-13 Waymark evaluation — 53 % of traces end within 1 m; 17 % for the 124M preview |
| Opening-heading agreement | 0.92 | results.json of the 2026-09-13 Waymark evaluation — first heading word equals the simulator’s; 0.44 for the 124M preview |
| Two-leg goals completed | 25 / 25 | results.json of the 2026-09-13 Waymark evaluation — waypoint line then arrival at the second stop; 0 / 25 for the 124M preview |
| Parse rate · completion rate | 0.98 · 0.98 | results.json of the 2026-09-13 Waymark evaluation — every line matches the grammar; trace ends in an arrival, a stop or a loiter |
| Sidestep recall | 0.61 (36 / 59) | results.json of the 2026-09-13 Waymark evaluation — ground-truth sidestep notes whose type and label the model also emitted; 0.05 for the 124M preview |
| Judge score (third-party language model, 1–5) | 4.74 | results.json of the 2026-09-13 Waymark evaluation — 98 judged traces, mean of plausibility, obstacles and goal; 3.87 for the 124M preview |
| Generation time per trace | ~6.6 s | results.json of the 2026-09-13 Waymark evaluation — 300 traces on one NVIDIA L4, 28 tokens/s |
| Validation loss (best checkpoint, step 1750) | 0.166 | model-best.json checkpoint sidecar |
The trace-level figures come from the Waymark evaluation of 2026-09-13: 300 fresh synthetic scenes (generator seed 9001, none seen in training) across seven goal kinds — 141 viewer walks to an object, 73 walks by a named person, 31 free walks, 25 two-leg routes, 11 approaches toward the viewer, 11 loiters and 8 departures into the background — each simulated by the same generator that produced the training data, so every trace has a ground truth to compare against. Traces were generated greedily (temperature 0.2) on one NVIDIA L4. The same suite was run on the 124M preview model that preceded this version, and the source column of each row records that figure.
Against the simulator’s ground truth, the mean endpoint error is 1.84 m (median 0.92 m; 53 % of traces end within a metre), the opening heading agrees with the simulator’s 92 % of the time, and every one of the 25 two-leg routes produces a waypoint line before arriving at the second stop. Per goal kind, endpoint error is lowest on free walks (0.19 m) and approaches toward the viewer (0.40 m) and highest on departures into the background (4.20 m, eight scenes). Of 59 ground-truth sidesteps, 36 appear in the model’s notes; of 11 walks that the simulator stops at a road edge, the model stops 3 and arrives in the others. A third-party language model judged 98 traces on plausibility, obstacle handling and goal completion and scored them 4.74 out of 5 on average (plausibility 4.73, obstacles 4.66, goal 4.83); its lowest marks went to traces cut off mid-line without an arrival and to departures that walked the wrong way. Generation takes about 6.6 s per trace on the L4 at 28 tokens per second, plus about 11 s to load the checkpoint.
The validation loss is token cross-entropy over the held-out split — every tenth row of the mixed movement-plus-layout file, roughly 13,800 rows — measured on 24 batches at each evaluation. The best value, 0.166 at step 1750, is recorded in the model-best.json checkpoint sidecar; later evaluations were worse and training stopped at step 2500.
Known gaps. The ground truth is the generator, so the trace metrics say how well the model reproduces the simulator’s rules, not how those rules match real movement; no human evaluation and no measurement against real trajectories exist. Road-edge stops are the weakest behaviour: the model arrives across a road with no crosswalk in 8 of 11 such scenes. The judge is a language model reading trace text, not a listening or viewing study, and it saw 98 of the 300 traces. The synthetic data carries two defects that the model has learned faithfully: the side word in sidestep and give way notes is inverted relative to the coordinate motion (the coordinates are right, the word is wrong), and loiter lines carry a blank-label here note. Latency was measured on one GPU type with a warm service; a cold start adds one to two minutes.
API #
| Endpoint | Auth | Body limit | Description |
|---|---|---|---|
POST /v1/move | Bearer preview key (fln_) or session token (fls_) | 64 KB; sketch ≤ 6,000 characters, goal ≤ 300 | Scene sketch + goal (+ mover) → trace text, parsed steps and a plain-words summary. |
Waymark Extra is public on the Falcon API at POST /v1/move. Requests count against the spatial preview quota bucket, shared with /v1/spatial, and the request body is limited to 64 KB, with sketch at most 6,000 characters and goal at most 300. The route forwards the request to the GPU service, parses the trace into steps and adds the summary. The base URL, envelope, authentication, rate limits and retry guidance are documented in API conventions.
POST /v1/move HTTP/1.1
Authorization: Bearer fln_…
Content-Type: application/json{
"sketch": "kind: scene\nviewer: eye-level on sidewalk\nobjects: planter box (center foreground); doorway (right midground); hedge (left midground)\nrelations: planter box in front of doorway; hedge left of doorway\nrelative: planter box closer than doorway",
"goal": "walk to the doorway",
"mover": "you"
}{
"ok": true,
"engine": "waymark-extra",
"frame": "approx m; x right, y ahead, z up; viewer at origin",
"mover": "you",
"trace": "t=0.0s (0.0, 0.0, 0.0) ahead-right | doorway ~6m ahead | ahead: planter box ~2m | ahead-left: hedge ~5m\nt=1.5s (1.6, 1.2, 0.0) ahead-right | sidestep left around planter box | doorway ~5m ahead\nt=3.0s (2.3, 3.2, 0.0) ahead | doorway ~3m ahead | left: hedge ~4m\nt=4.5s (2.8, 5.2, 0.0) ahead | doorway ~1m ahead\narrive t=5.4s (3.0, 6.1, 0.0) doorway | planter box behind | hedge left",
"steps": [
{"t": 0.0, "x": 0.0, "y": 0.0, "z": 0.0, "heading": "ahead-right", "notes": ["doorway ~6m ahead", "ahead: planter box ~2m", "ahead-left: hedge ~5m"], "arrive": false},
{"t": 1.5, "x": 1.6, "y": 1.2, "z": 0.0, "heading": "ahead-right", "notes": ["sidestep left around planter box", "doorway ~5m ahead"], "arrive": false},
{"t": 3.0, "x": 2.3, "y": 3.2, "z": 0.0, "heading": "ahead", "notes": ["doorway ~3m ahead", "left: hedge ~4m"], "arrive": false},
{"t": 4.5, "x": 2.8, "y": 5.2, "z": 0.0, "heading": "ahead", "notes": ["doorway ~1m ahead"], "arrive": false},
{"t": 5.4, "x": 3.0, "y": 6.1, "z": 0.0, "heading": "arrive", "target": "doorway", "notes": ["planter box behind", "hedge left"], "arrive": true}
],
"summary": "You reach the doorway in about 5s, ending ~3m right and ~6m ahead of where the photo was taken. Along the way: sidestep left around planter box."
}The body takes sketch and goal (both required), and optional mover, max_new_tokens (default 400, clamped to 32–480) and temperature (default 0.2, clamped to 0–1). engine is always waymark-extra; mover echoes the request or reads you. The trace above is illustrative of the shape; the exact lines depend on the weights.
Errors:
| Status | Code | Meaning |
|---|---|---|
| 400 | bad_json | The body is not valid JSON or exceeds 64 KB. |
| 400 | sketch_required | sketch is missing or empty after trimming. |
| 400 | goal_required | goal is missing or empty after trimming. |
| 401 | invalid_credentials | Missing, malformed or revoked bearer credential. |
| 402 | payment_required | The credential’s owner is not in good standing with the preview. |
| 413 | text_too_long | sketch is over 6,000 characters or goal over 300. |
| 429 | quota_exceeded | The spatial bucket for the current UTC calendar month is exhausted; kind is spatial. |
| 503 | move_warming | The GPU service is still loading the checkpoint after a cold start; retry after the Retry-After interval (60 s). |
| 502 | move_failed | The GPU service failed after a valid request, or returned a trace with no parseable step; safe to retry once. |
| 503 | move_unavailable | The GPU service is not configured or is still warming up; retry once after 2.5 s, then wait for the cold start. |
GET /health reports waymarkExtra: true when a movement service is configured; a false predicts 503 move_unavailable on the route.
Runtime & deployment #
| Kind | GPU service |
|---|---|
| Resident | one container on a single NVIDIA L4 GPU, 16 GiB memory, 4 vCPU, shared with Waymark since 2026-09-16 |
| Serving | the Waymark GPU service it shares with Waymark, fronted by the Falcon API at POST /v1/move |
| Cold start | scales to zero when idle; a cold start copies the 1.4 GB checkpoint onto the GPU and takes about one to two minutes; a CUDA warm-up at start-up means the first trace after it is no slower than later ones |
| Concurrency | 2 requests per instance, at most 1 instance |
| Timeout | 180 s per request at the service; the Falcon API waits 60 s and retries once after 2.5 s |
Waymark Extra runs as a GPU service on Cloud Run with one NVIDIA L4, fronted by the Falcon API; a preview key never reaches the service directly. The service loads the checkpoint in a background thread as soon as it boots and answers 503 until the model is on the GPU; the Falcon API waits 2.5 s and retries once, with a 60 s overall timeout per attempt. Because the instance count scales to zero when idle, the first request after a quiet period pays a cold start of about one to two minutes that includes copying the 1.4 GB checkpoint onto the GPU; a host should expect a 503 move_warming (with a Retry-After of 60 s) during that window and retry after it. Since 2026-09-16 a CUDA warm-up runs at start-up, so the first trace after a cold start is no slower than later ones; the same service also holds Waymark as a second model. Measured through the Falcon API, a twelve-step trace takes about 5–10 s depending on route length (it took 11–14 s before the warm-up).
- Loading. Two models per instance since 2026-09-16 — Waymark Extra and Waymark — each loaded once at boot, warmed with a CUDA pass and resident for the life of the instance.
- Selection. There is no request field:
/v1/moveis bound to Waymark Extra and/v1/spatialto Waymark. Theenginefield in every response names the model that answered. - Concurrency. Two requests per instance and at most one instance, shared with Waymark, so a burst queues; a host that sends traces in parallel should expect the later ones to wait.
- Timeouts. 180 s per request at the service; 60 s per attempt at the Falcon API, with one retry after 2.5 s on a
503or a timeout.
What the host must provide:
- A vision model that emits the Falcon scene sketch for the image, or a sketch authored by other means.
- A goal in plain words and, when someone other than the viewer moves, a mover label that appears in the sketch.
- Retry handling for
503and502during a cold start, and a fallback whenstepscomes back short. - A responder or template for the reply, or the
summaryfield used as it is.
| Setting | Configured value |
|---|---|
| GPU | 1 × NVIDIA L4 |
| Memory / CPU | 16 GiB / 4 vCPU, CPU never throttled |
| Request timeout | 180 s at the service; 60 s per attempt at the Falcon API |
| Concurrency | 2 per instance |
| Instances | minimum 0, maximum 1 |
| Models resident | Waymark Extra and Waymark (since 2026-09-16) |
| Generation cap | 480 tokens |
Integration notes #
- Feed the
stepsarray, unchanged, to LIM3D-XL together with the same sketch when the host needs intent, hazard and advice labels for the simulated walk; both models use the frame approx m; x right, y ahead, z up; viewer at origin. - Summarise a trace from its last step, or show
summary:arrivetrue means the mover reachedtargetat timet;arrivefalse on the last step means the walk stopped early — at a road edge, or because the mover loitered. - Filter notes by prefix when extracting events:
sidestep,give way,stop at edge,no crosswalk,cross,overhead:,climb,up the ramp. Treat the side word insidestepandgive wayas unreliable and use the change inxinstead. - Keep
temperatureat the default when the host caches results; the output is deterministic there. - Warm the service ahead of an interactive session with one small request, so the user does not wait through the cold start.
Limits & safety #
It does not see the image; it reads the sketch. Anything the sketch omits, mislabels or misplaces, the trace inherits, and the model will not add objects, exits or paths the sketch did not mention.
- Distances and positions are approximate metres reconstructed from depth and lateral bands. Estimates, never measurements. Nothing in a trace is GPS, surveyed or scaled to the real scene.
- Traces are capped at twelve lines and 480 generated tokens; long sketches lose their oldest lines to make room.
- Only the viewer stops at a road edge; simulated persons cross roads without stopping, and a road counts as crossable only if a crosswalk appears somewhere in the sketch. Even for the viewer, the model stopped in only 3 of 11 evaluated road-edge scenes.
- Avoidance is a geometric sidestep of at most three detours per trace, not path planning; the side word in
sidestepandgive waynotes is inverted in the training data. - Walks toward the viewer end about one metre in front of the origin and walks into the background end at 19–22 m by construction of the training data, whatever the scene.
- Loiter traces have no arrival line and carry a blank-label
herenote on every line. - Where another person goes is a plausible simulation from templates, not a prediction of anyone’s behaviour.
- The model is sized for a GPU with a single instance that scales to zero: one busy cold start can delay every caller by a minute or two.
Out of scope: navigation, wayfinding for people with reduced sight or mobility, road-crossing decisions, and any use where a wrong distance or a missed obstacle has physical consequences. Not a navigation or safety system. Traces describe a synthetic world drawn from the sketch; the host should present them as illustrations of the scene, never as instructions.
Fixed weights per version; the model does not learn from requests.
Versions #
| Version | Date | Status | Note |
|---|---|---|---|
| 1.0.0 | Released | First released version. Trained from scratch on 2 × A100 with early stopping at step 2500; best validation loss 0.166 at step 1750. Serves behind POST /v1/move from 2026-09-13. |
Compatibility. A major version bump means the request or response contract changes — a new required field, a different trace grammar, a changed frame, or a different set of heading or event words — and a host must be updated to read it. A minor bump is a retrain with the same contract: traces may differ in content and quality but parse the same way. A patch bump changes only metadata or runtime settings such as the generation cap or the container size.
A 124M-parameter preview, warm-started from Waymark and trained on 2026-09-11 with a 512-token window and traces of up to seven steps, was never published; this version supersedes it, and the evaluation above records the preview’s figures beside its own.
Current weights: the best checkpoint of the 2026-09-12 Vertex AI job, step 1750, as recorded in the model-best.json sidecar (validation loss 0.166); the fetched checkpoint file is model.pt, 1,418,678,978 bytes in fp32, from the 24-layer, 1024-wide, 1024-token configuration. Weights are not distributed during the private preview; see Access.
Weights are not distributed during the private preview.