Waymark

Overview #

Waymark answers questions about where things are in a photographed scene. You give it a written sketch of the scene — the list of objects and where they sit, which a vision model produces from the image — and one question; it gives back a single short sentence in everyday words such as left, right, closer or behind. Use it when you need a quick layout answer rather than a route.

Waymark is a 124M-parameter decoder-only language model that answers layout questions about a scene it has never seen. It reads a structured text sketch — the kind of scene, where the viewer stands, a list of objects with viewer-relative positions, pairwise relations and anything uncertain — together with one short question, and returns a single sentence: what is on the left or right, what is closer or farther, what is in front of or behind what, and what is blocking what.

The input is text and the output is text. A vision model that emits the Falcon scene sketch does the seeing; Waymark does the reasoning about the sketch and keeps to its vocabulary of relative positions. It never reports metres, GPS coordinates or compass bearings, and it does not invent objects, doors or paths that the sketch does not mention. Answers are sampled from the model at a low temperature, so they are short, plain and slightly variable from call to call.

Since version 1.1.0 (2026-09-20) Waymark is one model for street and indoor scenes — homes, offices, schools, shops, restaurants, hotels, station concourses and garages as well as streets, paths and plazas. It was reached through an intermediate indoor finetune (published for one day as “Waymark Indoor”, now withdrawn) and then retrained on a balanced mix of street and indoor sketches; the request value "model": "waymark-indoor" still works on /v1/spatial as an alias for these weights.

Waymark is the default model of the Waymark series. Its sibling, Waymark Extra, reads the same sketch grammar and keeps Waymark's layout question-and-answer task, but adds movement: given a sketch and a goal it emits a timed, viewer-relative trace of a person moving through the scene. Waymark Extra is a larger model — 355M parameters with a 1,024-token window, trained from scratch — and the Falcon API binds it to /v1/move; Waymark is the model behind /v1/spatial.

Intended use #

• Answering “which side”, “what is closer”, “what is in front” and “is my view blocked” questions over a sketch produced by a vision model from a street-level scene.
• Turning a sketch of sidewalks, roadways, crossings, plants, stairs, ramps and overpasses into a one-line layout read that a responder can fold into a longer reply.
• Acting as the fallback answer when a larger responder cannot finish, since the sketch is already grounded and the answer is short.

Out of scope #

• Reading pixels: Waymark only ever sees the text sketch.
• Anything metric — distances in metres, heights, headings, sizes in real units.
• Documents, maps, receipts and other non-scene material; sketches of kind document are rejected before they reach the model.
• Navigation, accessibility or safety decisions, such as when to cross a road.

Choose Waymark when #

• The sketch is a plain street-level scene with objects, relations and a viewer line, and you want the smallest model that answers in the trained vocabulary.
• You need a layout answer in about a second, rather than a multi-second trace.
• You need a movement trace rather than a layout sentence — a goal such as “walk to the doorway” turned into timed, viewer-relative steps — or your sketches are long or busy, with a second person, two-leg routes or prompts past 512 tokens: choose Waymark Extra instead, with its 24 layers and 1,024-token window, on /v1/move.

Architecture #

Waymark is a GPT-style decoder built from twelve pre-LayerNorm transformer blocks. Each block applies causal self-attention over twelve heads of dimension 64, then a four-times-wider MLP (768 → 3,072 → 768) with a GELU activation; both sub-layers are residual. There are no biases on the linear layers, and the output head shares its weight matrix with the token embedding. Positions are learned absolute embeddings over a 512-token block, so every prompt — sketch plus question — has to fit in 511 tokens with room left for the answer.

Tokenisation is the standard GPT-2 byte-pair encoding with a vocabulary padded to 50,304 entries. The sketch is not parsed into fields; it is tokenised as ordinary text under a fixed prompt template, and the model learns the line structure (kind:, viewer:, objects:, relations:, relative:, uncertain:) from the training rows. The question follows on a User: line and the model continues after a fixed answer-role line; generation stops at the first of a new role line, a blank line or the end-of-text token.

There is one head — the language-model head — and no fixed label set. The answer is free text, but the training data confines it to a small vocabulary: five lateral bands (left, center-left, center, center-right, right), three depth bands (foreground, midground, background) and the relation words in front of, behind, closer than, next to, above, under, blocking. Decoding uses top-k 40 sampling; the Falcon API route asks for temperature 0.3, and a temperature at or below 0.25 switches the service to greedy argmax.

What is deliberately absent: there is no vision encoder and no image input — Waymark never sees pixels — and there is no numeric head, so it cannot output a distance, an angle or a size even when the sketch hints at one.

Comparison #

Both share the GPT-2 tokeniser, the sketch grammar and the layout prompt template, so a sketch written for Waymark is valid input for its sibling. The differences are the second task Waymark Extra was trained on, the richer sketches in its movement rows (capture, colour, size and ground attributes), and how much capacity it brings to long or unusual prompts.

Inputs & outputs #

Input #

A sketch is a short block of key: value lines, one per line, in this order: kind (always scene for Waymark), an optional viewer phrase, an optional capture line, objects, relations, relative and uncertain. Objects are separated by semicolons and carry their position in brackets as a lateral band and a depth band — doorway (right foreground). Relations are plain phrases such as hedge left of path or bike rack in front of doorway; the relative line holds A closer than B comparisons; the uncertain line lists anything the sketching model could not settle. The full grammar is documented under Input formats.

The frame is the viewer's: lateral bands run left, center-left, center, center-right, right across the field of view, and depth bands run foreground, midground, background away from the viewer. There are no coordinates, no units and no compass. The question is one short line; when it is omitted the route substitutes “what's the layout?”.

A complete request with a caller-supplied sketch:

{
  "sketch": "kind: scene\nviewer: eye-level on sidewalk\nobjects: doorway (right foreground); retaining wall (center background)\nrelations: doorway in front of retaining wall; doorway right of center; retaining wall behind doorway\nrelative: doorway closer than retaining wall",
  "question": "what's in front"
}

The same request on the media path, where the Falcon API produces the sketch first:

{
  "image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD...",
  "question": "which side is the roadway"
}

Limits: the JSON body may be at most 24 MB; a decoded image at most 6,000,000 bytes and a decoded video at most 15,000,000 bytes. Internally the sketch and question are assembled into a prompt of at most 511 GPT-2 tokens; a longer prompt is truncated from the front, which drops the header and the earliest sketch lines first. Training rows averaged 112 tokens and never exceeded 180, so ordinary sketches are far inside the budget.

Output #

Waymark returns the continuation after the answer-role line, trimmed at the first new role line, blank line or end-of-text token, with the end-of-text marker removed and whitespace collapsed. The route wraps that sentence in the Falcon envelope; since 2026-09-19 the envelope also names the model that answered:

{
  "ok": true,
  "engine": "waymark",
  "model": "waymark",
  "text": "doorway on your right foreground.",
  "from": "sketch"
}

On the media path the envelope also carries the sketch the model read, and from names the media kind:

{
  "ok": true,
  "engine": "waymark",
  "model": "waymark",
  "text": "Roadway on your right, past the curb; sidewalk runs ahead.",
  "sketch": "kind: scene\nviewer: eye-level on sidewalk facing traffic\nobjects: sidewalk (center foreground); curb (center-right midground); asphalt roadway (right midground)\nrelations: curb right of sidewalk; asphalt roadway right of curb\nrelative: sidewalk closer than asphalt roadway",
  "from": "image"
}

Decision-rule semantics: there is no confidence score and no ranked list. The answer is one sample from the model at temperature 0.3 with top-k 40, so repeated calls on the same sketch can phrase the answer differently, and a question outside the trained templates can draw a generic sentence rather than a refusal. Treat text as a grounded prior about the sketch — the sketch is the ground truth, the sentence is Waymark's reading of it — and let the responder or the host decide how much weight it carries. When engine is not "waymark", the sentence came from the third-party sketching model instead and Waymark's reading was unavailable.

Examples #

Question and answer pairs drawn from the training set, each over a sketch of the kind shown above:

Question forms the model saw most often, by count in the training file: “which way to the {X}” (160), “any stairs or ramps” (160), “any overpass?” (141), “don't invent meters” (102), “what's in front” (97), “is {X} closer than {Y}” (93), “is something blocking my view” (88), “what should I look at first” (83), “what plants do you see” (80), “what's closest” (80), “which side is the roadway” (80). Questions phrased close to these forms get the most specific answers.

Training #

Version 1.1.0 (2026-09-20) was retrained from an intermediate indoor finetune of Waymark 1.0.0 (three stages on 144,000 synthetic indoor rows over 21 environments, with answers computed exactly from each sketch) as one Vertex AI custom job on a single NVIDIA A100 (nine minutes of training): 1,500 steps of batch 32 sequences of 512 tokens, AdamW, peak learning rate 5e-5 with a 30-step warm-up and cosine decay, gradient clipping at 1.0, bf16 autocast, and cross-entropy on the answer tokens only (the prompt is masked out of the loss). The data is 108,825 synthetic rows: 60,000 indoor rows over 21 environments, 42,000 street rows whose answers are computed exactly from sketches written by Waymark's own scene generator (the question types of the original data — stairs or ramps, overpass, which side the roadway is on, plants, distances — plus the indoor generator's left, right, closest, compare, behind, where and layout questions), and the 2,275 original street rows three times, so 45 % of the rows are street scenes. A denial such as “No roadway listed in the sketch” is only generated when nothing in the sketch implies the category; an independent audit of the generator found no answer that contradicts its own sketch. Weight blends of Waymark 1.0.0 and Waymark Indoor 1.0.0 were tried first and rejected: no blend kept both skills. Every version-1.0.0 sentence below still describes the base these weights descend from.

Waymark 1.0.0 was trained on spatial-env.jsonl, a file of 2,275 synthetic rows, each holding a formatted sketch, a question and a target answer. The rows were template-generated: every sketch is of kind scene, drawn from 45 street-level object names (sidewalk, curb, asphalt roadway, bike lane, crosswalk, hedge, planter box, stair flight, ramp, pedestrian overpass, support pier and so on) and 10 viewer phrases (eye-level on sidewalk, eye-level approaching underpass, at stair landing, standing in parking lot, on trail looking forward, among others), with two to four objects per sketch (mean 3.27), a relations line in every row, a relative: line in every row and an uncertain: line in 1,101 rows. Questions come from 32 templates; answers run 10–232 characters (mean 62). The file holds 253,963 GPT-2 tokens in total, with rows averaging 112 tokens and none longer than 180. No real user data was used: no captured images, no real sketches, no real conversations.

Data preparation tokenises each row with the GPT-2 encoder, appends the end-of-text token, and routes every tenth row (by index) to the validation split — about 228 rows — with the rest forming roughly 229,000 training tokens. Training samples random 513-token windows from the concatenated token stream, so windows are not aligned to row boundaries and the model also learns to start a new example after end-of-text. No augmentation is applied.

Training ran on Vertex AI as two custom jobs on a single NVIDIA L4 (g2-standard-8). The first, on 2026-09-08, trained the small configuration from scratch and reached its best validation loss of 0.1505 at step 250; that step-250 checkpoint is the served weight file and is also the warm-start base for Waymark Extra. Its exact hyper-parameters were not recorded with the checkpoint. The second, job waymark-20260909-091849 on 2026-09-09, was a refine run warm-started from that checkpoint with the recipe committed alongside the trainer: AdamW (betas 0.9 and 0.95, weight decay 0.1), peak learning rate 5e-5 with a 50-step linear warm-up and cosine decay to a tenth of the peak, gradient-norm clipping at 1.0, bf16 autocast, batch 16 with gradient accumulation 8 (128 sequences of 512 tokens, about 65,000 tokens per step), at most 4,000 steps, evaluation every 250 steps over 20 validation batches, early stopping after two evaluations without improvement, seed 17. The objective in both runs is plain next-token cross-entropy over the whole prompt and answer; there is no separate answer-only loss. No metrics from the refine run are recorded in the repository, which is why the published numbers come from the first run's checkpoint.

Waymark was not trained on real photographs or on sketches produced from them, on sketches with capture, colour, size or ground attributes (those fields appear only in the Waymark Extra mix), on sketch kinds other than scene, on objects outside the 45 names, or on any goal-directed or movement questions.

Evaluation #

Version 1.1.0 was scored on 1,102 synthetic layout questions with the hospital benchmark's deterministic scorer, next to Waymark 1.0.0, the intermediate indoor finetune and six other candidates, all sampled as the service samples (at most 64 new tokens, temperature 0.35, top-k 40, one fixed seed per question). Seven sets: 100 street questions from the hospital benchmark's outdoor control set, 202 street questions in the question types Waymark 1.0.0 was trained on (stairs or ramps, overpass, roadway present, which side the roadway is on, plants, distances; the “is the X ahead” questions and 78 street items whose sketch contradicted the object list were excluded after an audit of the scorer), 200 hospital-scene questions, 200 over two indoor environments held out of training, 200 over training environments with fresh scenes, and 200 over five-band sketches. Version 1.1.0 is exact on 87 % of the street control questions (1.0.0: 16 %), on 96.5 % of the street question types (49.5 %), on 40.5 % of hospital questions (3 %), on 53 % in unseen indoor environments (4 %) and on 84.5 % in training environments (7 %); it is significantly better than the intermediate indoor finetune on six of the seven sets and never worse (exact McNemar test on paired questions). The street question types come from the same scene generator as the training rows, so that figure is in-distribution; the street control set was written by the benchmark's own generator. Nothing here measures sketches a vision model writes from real photographs, and nothing is clinically validated.

The 1.0.0 figures below are the ones recorded with that checkpoint:

The held-out set is every tenth row of the training file — about 228 rows, roughly a tenth of the tokens — tokenised the same way as the training split; the loss is mean next-token cross-entropy over random 512-token windows of that split, measured on 20 batches at each evaluation. The best value, 0.1505 at step 250, is the one recorded in the checkpoint sidecar alongside a training loss of 0.0834.

Known gaps: no task-level accuracy, exact-match rate, human rating or benchmark score exists for Waymark, and the validation rows are generated from the same templates as the training rows, so the loss reflects fluency on in-distribution sketches rather than generalisation to sketches produced by a vision model from real scenes. The gap between training loss and validation loss at the best step also suggests some memorisation of the small file. Any claim about answer quality on real-scene sketches is unmeasured.

API #

Waymark is exposed through the Falcon API route /v1/spatial. Authentication, the response envelope, retry guidance and the full error vocabulary are documented under API conventions; this section covers what is specific to Waymark.

Route /v1/spatial · quota bucket spatial · body limit 24 MB JSON body (base64 media inflates by about a third).

POST /v1/spatial HTTP/1.1
Authorization: Bearer fln_your_preview_key
Content-Type: application/json

Request, sketch path:

{
  "sketch": "kind: scene\nviewer: eye-level on sidewalk\nobjects: doorway (right foreground); retaining wall (center background)\nrelations: doorway in front of retaining wall; doorway right of center; retaining wall behind doorway\nrelative: doorway closer than retaining wall",
  "question": "what's in front"
}

Response:

{
  "ok": true,
  "engine": "waymark",
  "model": "waymark",
  "text": "doorway on your right foreground.",
  "from": "sketch"
}

When the body carries image or video instead of sketch, the Falcon API first runs server-side vision sketching (third-party model) at temperature 0 to produce a JSON sketch, formats it into the text grammar, then asks Waymark for the answer with the same decoding settings. If Waymark's answer is unavailable on that path, the sketching model answers from the sketch and engine reports its identifier. The response then includes the sketch field and from is "image" or "video". Sketch-path requests ask Waymark for at most 72 new tokens at temperature 0.3.

Errors #

Every successful call, on either path, counts one unit against the spatial preview quota bucket, tallied per UTC calendar month. The body limit is 24 MB of JSON; because base64 inflates media by about a third, the practical ceilings are the decoded sizes above. Bucket sizes and the shared envelope fields are listed under API conventions.

Runtime & deployment #

Since 2026-09-16 Waymark no longer has a service of its own: it runs on the same Cloud Run GPU service as Waymark Extra — one NVIDIA L4, 16 GiB of memory, 4 vCPU — loaded as a second model, and its /v1/complete answers behind the Falcon API's /v1/spatial. The service loads both checkpoints in a background thread at start-up; until the weights are resident, a completion request receives HTTP 503 with the error model still loading, and if loading fails the service reports HTTP 500 with the load error and stays unhealthy. The Falcon API's client for the service allows 45 s per request and, on a 503 or a timeout, waits 2.5 s and retries exactly once before surfacing spatial_failed. The service scales to zero when idle, so after a quiet period the first request pays a cold start of one to two minutes; during that window the route answers with its existing codes (503 waymark_unavailable, or 502 spatial_failed once the built-in retry is spent) and a host should wait and try again after it. Once warm it stays fast: measured through the Falcon API, an answer takes 0.6–1.1 s (it took 14–18 s on the retired 2 vCPU service), generating at most 96 new tokens through the 124M-parameter model on the GPU.

Selection is by route and, on /v1/spatial, by an optional request field. Since 1.1.0 the two accepted values, "model": "waymark" (the default) and "model": "waymark-indoor", select the same weights; the service loads the checkpoint once and answers under the requested name, and any other value answers 400 bad_model. /v1/move is bound to Waymark Extra. The engine field of every response names the family that answered and model names the model. The service picks CUDA when a GPU is visible and CPU otherwise; the production image runs on the GPU.

What the host must provide:

• A sketch in the documented grammar, or an image or video for the Falcon API to sketch first.
• One question per call, ideally phrased close to the trained templates.
• Retry handling for 503 and 502 beyond the single built-in retry, with a client timeout of at least 45 s, and patience through a one-to-two-minute cold start after a quiet period.
• A responder or presentation layer that treats text as a reading of the sketch, not as a verified fact about the scene.

Inside the container the service also answers GET /health (fields ok, model, ready, loading, error) and accepts a fully assembled prompt on POST /v1/complete with optional max_new_tokens (1–96, default 64) and temperature (default 0.35). That surface is internal to the Falcon API and is not part of the public contract.

Integration notes #

• Write sketches in the trained vocabulary. Position words outside the five lateral and three depth bands, or object names far from the 45 street-level names, put the model out of distribution.
• Keep the sketch honest about uncertainty. The training rows use an uncertain: line for anything unresolved (for example exact clearance height unknown), and answers over such rows lean to the safer reading.
• Prefer the trained question forms — “which side is the X”, “is X closer than Y”, “is something blocking my view”, “any stairs or ramps” — over long or compound questions.
• Ask one thing per call; the model returns one sentence and stops.
• Feed the sentence to a responder as a prior on left / right, depth and blocking. If the responder and the sentence disagree, the sketch wins and the reply should stay cautious.
• Do not parse the sentence for numbers. There are none by design, and any that appear are not grounded.
• On the media path, keep the returned sketch with the answer; it is the only record of what the model actually read.

Limits & safety #

It does not see the image; it reads the sketch. Everything Waymark can say about a scene is bounded by what the sketching model wrote down, and anything the sketch omits or gets wrong, Waymark inherits.

• No true distances or sizes: answers are relative — closer, farther, in front, behind — never metres, GPS coordinates or compass bearings. Estimates, never measurements.
• No pixels and no capture details: Waymark's training rows contain no capture, colour, size or ground attributes, so questions about shot type, colour, size or height off the ground are not backed by its training data even when the sketch carries those fields.
• Narrow vocabulary: 45 street-level object names, 10 viewer phrases and 32 question templates. Document, map or unknown sketch kinds, and objects outside the vocabulary, are out of distribution and can draw generic or wrong sentences.
• No guarantee against invention: the model was trained not to name objects, doors or paths absent from the sketch, but this is a property of the data, not a checked constraint on the output. On busy hospital sketches of eight to twelve objects, version 1.1.0 still substitutes a familiar name for an unfamiliar one in about four answers in ten.
• No confidence score: the answer is a single sample at temperature 0.3, so repeated calls can differ in wording and occasionally in substance.
• Context of 512 tokens: a prompt longer than 511 tokens is truncated from the front, silently dropping the sketch header and earliest lines.
• Not a navigation, accessibility or safety aid, and not a substitute for looking: it must not be used to decide when to cross a road or whether a passage is clear.

Out of scope: documents, receipts, tickets, labels and maps are rejected before Waymark is reached (the sketching step returns no scene, and the route answers 422 sketch_unavailable); goal-directed questions (“how do I get to the pier”) and anything requiring measurement belong to the siblings or to no model at all.

Fixed weights per version; the model does not learn from requests.

Versions #

Compatibility: a major version bump changes the input or output contract — the sketch grammar, the prompt template, the answer vocabulary or the shape of the /v1/spatial envelope — and a caller should re-read this page before upgrading. A minor bump is a retrain on the same contract (new weights, possibly a different loss, same fields), and a patch bump touches only metadata or the runtime. Within version 1.x a sketch written today stays valid input.

Current weights: model.pt, the step-1,500 checkpoint of the 2026-09-20 job (answer-only validation loss 0.1716), 496,039,138 bytes, float32 PyTorch checkpoint with the model-best.json sidecar. The 1.0.0 weights (step 250 of the 2026-09-08 run, validation loss 0.1505) are kept for rollback. Weights are not distributed during the private preview.

References #

• Input formats — the sketch grammar Waymark reads.
• API conventions — authentication, envelope, quotas and errors.
• Families — how the Waymark series relates to the LIM and Response series.
• Status & versioning — what Released, Trained and the other statuses mean.
• Waymark overview — the short page for this model.
• Waymark Extra — the sibling that walks a person through the scene, on /v1/move.