Tabulate Extra

Overview #

Tabulate Extra turns a data file into records. You give it the text of the file as you found it — a CSV with a two-line preamble from the tool that exported it, a tab-separated dump with a totals line at the bottom, a fixed-width report, a block of key–value pairs, a log, JSON lines, a Markdown table — and it gives back the file's format, its columns with a name, a type and a role, and its rows as JSON with the values converted: numbers as numbers, dates as ISO strings, yes/no as booleans, the junk lines left out.

It is the second Alternate of the Tabulate family: the same catalogues, executor and response as Tabulate, from the large size of the two networks (width 256, four layers), trained on the richer generator profile that adds what the first release got wrong on hand-written files — titled preambles (“Report: Q3 sales”), totals rows with a value in every column, non-English booleans (ja/nein, oui/non, sí/no), two-value categories that are not booleans, entirely empty columns, Swiss thousands separators and accountants' negatives in parentheses. A host selects it with model: "tabulate-extra".

Tabulate is two small networks around a deterministic executor. The layout network reads the first forty lines byte by byte and decides what kind of file this is, what separates the cells, how they are quoted, whether the first data line is a header, and which lines are preamble, comments, blanks or footers rather than data. The executor applies that plan to the whole file with ordinary parsing code. The typer network then reads a sample of each column's values, with the header name when there is one, and names the column's type and role, which decides how the executor converts the values. Everything runs in the host process; nothing is sent anywhere but the file's own text.

Intended use #

• Ingesting files people upload or paste: a spreadsheet export, a bank statement download, a sensor log, a configuration dump, without a parser per format.
• Turning a data file into JSON for a language model, a chart or a form, with the totals line and the “exported on” preamble already removed.
• Naming columns by type and role so a host can route them: amounts to arithmetic, dates to calendars, identifiers to lookups.

Out of scope #

• Binary or nested formats: spreadsheets, PDFs, XML, deeply nested JSON. Tabulate reads text with one record per line or per block.
• Files whose rows change shape half-way, or whose structure only becomes clear after the fortieth line; the plan is made from the head of the file.
• Semantics beyond the column: it does not join, deduplicate, validate against a schema or fill gaps.

Choose Tabulate Extra when #

• The files come from reports and statements rather than clean exports: a title line, a totals row, a European or accounting number style.
• Twenty megabytes of weights and twice the latency are acceptable.
• Otherwise pick Tabulate, the default, or Tabulate Nano for a device.

Architecture #

Both networks start from the same byte encoder. A string is read as UTF-8 bytes, truncated or padded to a fixed width (64 bytes for a line, 24 for a value), embedded byte by byte with a position embedding, passed through two GELU convolutions of kernel three, and pooled with a masked mean and a masked max over the real bytes; a linear projection gives the token. No tokeniser, no vocabulary, no language assumption: a delimiter is a byte like any other, and the convolutions learn what a quoted comma or a thousands separator looks like.

The layout network turns each of the first forty lines into one token, adds a projection of fourteen counted features (length, counts of the seven likely delimiter and quote characters, digit, letter and space ratios, a comment-prefix flag, a double-space flag, a blank flag) and a position embedding, prepends a learned [CLS] token, and runs three transformer layers with six heads over the lines. Lines past the end of a short file are masked out of attention. The [CLS] output feeds four classifiers — format, delimiter, quote, header — and every line's output feeds a role classifier, so the same pass both names the dialect and marks which lines to parse.

The typer network reads a column as a sequence: the header name (empty when the file has none) as the first token, then up to twelve values sampled evenly through the column, each through the byte encoder, then two transformer layers. The name token's output feeds the type and role classifiers. Sampling through the column rather than taking the first values keeps a late change of style, or a run of blanks at the top, from deciding the type.

The executor is code, not weights. For delimited formats it splits each data line with the planned delimiter and quote, honouring doubled quotes inside quoted cells; for fixed width it finds column bounds as the runs of at least two spaces shared by every line; for key–value files it groups lines into blocks at blank lines or repeated keys; for logs it matches timestamp, level, source and message; for JSON lines and Markdown tables it does the obvious. Conversion by type handles thousands separators in either convention, currency symbols and percent signs, ten date and nine datetime layouts, the common boolean pairs and null tokens; anything that does not fit its type is returned as the trimmed string rather than dropped.

Inputs & outputs #

Input #

One field carries the file. The service applies no cleaning beyond what the executor documents: a leading byte-order mark is dropped and CRLF and CR line ends are read as LF. The first forty lines decide the plan; the rest are parsed under it, so a file whose structure changes later in the file is parsed as if it did not.

{
  "text": "Export from Acme CRM\nsku,qty,price,shipped\nSKU-0041,12,\"$1,204.50\",yes\nSKU-0042,3,$96.00,no\n",
  "max_rows": 50
}

Limits: text is at most 256,000 characters and the JSON body at most 400,000 bytes; max_rows is clamped to 1–2,000 and defaults to 200. There is no frame, no units and no structured input beyond the text.

Output #

A successful response carries the plan, the columns and the rows. row_count counts every data row found; rows carries the first max_rows of them.

{
  "format": "csv",
  "delimiter": ",",
  "quote": "\"",
  "header": true,
  "columns": [
    { "name": "sku", "type": "identifier", "role": "id", "p_type": 0.98 },
    { "name": "qty", "type": "integer", "role": "quantity", "p_type": 0.99 },
    { "name": "price", "type": "currency", "role": "amount", "p_type": 0.97 },
    { "name": "shipped", "type": "boolean", "role": "status", "p_type": 0.99 }
  ],
  "rows": [
    { "sku": "SKU-0041", "qty": 12, "price": 1204.5, "shipped": true },
    { "sku": "SKU-0042", "qty": 3, "price": 96, "shipped": false }
  ],
  "row_count": 2,
  "skipped": { "preamble": 1, "comment": 0, "blank": 0, "footer": 0 },
  "probs": { "format": [0.97, 0.01, 0.01, 0, 0, 0, 0, 0, 0.01], "header": 0.99 }
}

The example is illustrative of the shape. There is one decision rule: the most probable class wins for every head, and the executor follows the plan exactly. A host that wants to second-guess a parse reads probs.format and probs.header: a flat format distribution or a header probability near one half is the signal to show the person the plan.

Training #

Tabulate Extra is trained from scratch on synthetic files; no real data file is used at any point. The generator draws a schema of one to ten typed columns, renders each column's values consistently in one of many dialects (eight date layouts, six datetime layouts, currency symbols before or after, thousands separators as commas, points or spaces, seven boolean pairs, several identifier shapes, null tokens), lays the rows out in one of nine formats, and then adds the things real files have: a preamble of one to three lines (including the sep=; hint some spreadsheets write), comment lines with #, // or ;;, blank lines, a totals or end-of-file footer, ragged rows, CRLF line ends and a byte-order mark. Every file carries its ground truth — format, delimiter, quote, header, a role per line, and each column's type and role — and the rows the executor must recover, which is checked: the executor alone recovers 99.5 % of files exactly from the true plan.

The layout network learns from one example per file (the first forty lines), with cross-entropy over the four file-level heads and, weighted double, the per-line role head. The typer learns from one example per column, with the header name shown 85 % of the time when the file has one and never when it has not, so that it can type a column from its values alone. Both train with AdamW, a cosine schedule with warm-up, bf16 autocast on the GPU, and a held-out 3 % of the generated examples for validation.

The released weights are the first run: the large size on 1.6 million synthetic files from the richer generator, about 8.6 million column examples, Vertex AI job tabulate-20260922-123626, 53 minutes on one A100.

Evaluation #

The held-out set is 3 % of the generated examples, never trained on; the end-to-end check is 2,000 fresh files from the generator with a different seed, parsed by the whole pipeline and compared row for row with the truth. Two figures are ceilings of the data rather than of the model: the quote character cannot be known for a file in which no cell needed quoting (such a file is identical with and without a quote rule), which accounts for most of the gap between quote accuracy and the rest, and the column role is a soft label the generator assigns at random among two or three candidates for many types. The figures that decide a parse — format, delimiter, header, line roles, types — are all above 97 %, and 99.0 % of held-out files come back with every row exactly right, against 99.2 % with the true plan, so the networks cost the executor 0.2 points.

The held-out set here is drawn from the richer generator, so the figures are not directly comparable with Tabulate's, which are measured on the standard one. On the eight hand-written files that found Tabulate's gaps, three of the four are closed: a title line with a colon is read as preamble, a totals row with a value in every column as footer, and German yes/no words as a boolean. Two misses remain: a yes/no column under a header that does not name a boolean (in_stock) was typed as a category, and an ISO datetime column with a Z suffix as a date. Synthetic held-out sets measure the generator's world; real files are only as well covered as that world is.

API #

Tabulate Extra is exposed through the shared Falcon API route /v1/parse, selected with model: "tabulate-extra". Requests count against the parse preview quota bucket, one unit per file. Envelope, authentication, rate limits and retry guidance are in API conventions.

POST /v1/parse HTTP/1.1
Authorization: Bearer $FALCON_API_KEY
Content-Type: application/json

{ "text": "Export from Acme CRM\nsku,qty,price,shipped\nSKU-0041,12,\"$1,204.50\",yes\nSKU-0042,3,$96.00,no\n", "model": "tabulate-extra", "max_rows": 50 }

{ "ok": true, "engine": "tabulate", "model": "tabulate-extra", "format": "csv", "delimiter": ",", "quote": "\"", "header": true, "columns": [ { "name": "sku", "type": "identifier", "role": "id", "p_type": 0.98 } ], "rows": [ { "sku": "SKU-0041", "qty": 12, "price": 1204.5, "shipped": true } ], "row_count": 2, "skipped": { "preamble": 1, "comment": 0, "blank": 0, "footer": 0 }, "probs": { "format": [0.97, 0.01, 0.01, 0, 0, 0, 0, 0, 0.01], "header": 0.99 } }

Route /v1/parse · quota bucket parse · body limit 400,000 bytes; text at most 256,000 characters.

The route runs the model in the API process; there is no separate service contract.

Runtime & deployment #

Tabulate runs inside the Falcon API process through a TypeScript twin of the trainer's networks and executor, loaded lazily from an 8.6 MB bundle on the first parse. A parse costs one layout pass over up to forty lines and one typer pass per column, in scalar JavaScript: tens of milliseconds for a typical file, more for a wide one. The executor's cost is linear in the file.

• The host provides the file's text; nothing else is read.
• Selection is a request field: model: "tabulate-extra" on /v1/parse; without it the route runs Tabulate.
• The twin is checked against fixtures the trainer writes into meta.json — whole files with their expected plan, probabilities, columns and converted rows — so a runtime change that drifts from the Python original fails the build.

Limits & safety #

Tabulate sees only the text it is given: not the file name, its extension, its declared encoding or its origin. It does not see the bytes past the first forty lines when it plans, and it does not see any other file.

• It is a guesser, not a validator: a plausible plan for a file it has never seen the like of can still be wrong, and the executor will follow it faithfully. The probabilities and the skipped-line counts are there to be shown.
• It knows the dialects its generator produces. A layout outside them — a multi-line quoted cell, a file with two tables, an encoding that is not UTF-8 — is parsed as the nearest thing it knows.
• Types are named from a sample of twelve values; a column that changes type half-way is typed by whichever values were sampled.
• Conversion is by type, not by locale: 1.234,56 and 1,234.56 both become 1234.56, which is right for most files and wrong for a file where the comma is a thousands separator in one column and a decimal mark in another.
• It reads text of at most 256,000 characters; a larger file has to be cut by the host.
• Its output can echo anything in the input, including personal data; the host decides what to keep and where it goes.

Out of scope: Tabulate does not judge the data. Nothing it returns says whether a value is correct, a row is a duplicate or a file is what it claims to be. Not a substitute for a schema, and no help with a file that is not text.

Fixed weights per version; the model does not learn from requests.

Versions #

Compatibility: a major version bump changes the request or response contract — a field added, removed or renamed, a format, type or role added to or removed from a catalogue — or the tensor shapes the twin reads (the forty-line and sixty-four-byte budgets, the twelve sampled values). A minor bump is a retrain with the same contract and catalogues, for example more files, a larger size or a richer generator; parses will change but every field and label keeps its meaning. A patch bump touches metadata or runtime only with the tensors unchanged.

Current weights: version 1.0.0, tabulate.bin of 20,369,556 bytes (123 float32 tensors, SHA-256 ed44472808473e7edaa2778154f3bec6fdf7e1a4054a498f3542b3dcee3199b4) with its meta.json of 24,535 bytes, both written 2026-09-22 from Vertex AI job tabulate-20260922-123626. The catalogues in meta.json are the label sets listed on this page; the eight parity fixtures embedded in it are the regression test for any runtime change. Weights are not distributed during the private preview; see Access and Status & versioning.