Tabulate Extra

Overview #

Tabulate Extra turns a data file into records. You give it the text of the file as you found it — a CSV with a two-line preamble from the tool that exported it, a tab-separated dump with a totals line at the bottom, a fixed-width report, a block of key–value pairs, a log, JSON lines, a Markdown table — and it gives back the file's format, its columns with a name, a type and a role, and its rows as JSON with the values converted: numbers as numbers, dates as ISO strings, yes/no as booleans, the junk lines left out.

It is the second Alternate of the Tabulate family: the same catalogues, executor and response as Tabulate, from the large size of the two networks (width 256, four layers), trained on the richer generator profile that adds what the first release got wrong on hand-written files — titled preambles (“Report: Q3 sales”), totals rows with a value in every column, non-English booleans (ja/nein, oui/non, sí/no), two-value categories that are not booleans, entirely empty columns, Swiss thousands separators and accountants' negatives in parentheses. A host selects it with model: "tabulate-extra".

Tabulate is two small networks around a deterministic executor. The layout network reads the first forty lines byte by byte and decides what kind of file this is, what separates the cells, how they are quoted, whether the first data line is a header, and which lines are preamble, comments, blanks or footers rather than data. The executor applies that plan to the whole file with ordinary parsing code. The typer network then reads a sample of each column's values, with the header name when there is one, and names the column's type and role, which decides how the executor converts the values. Everything runs in the host process; nothing is sent anywhere but the file's own text.

Intended use #

• Ingesting files people upload or paste: a spreadsheet export, a bank statement download, a sensor log, a configuration dump, without a parser per format.
• Turning a data file into JSON for a language model, a chart or a form, with the totals line and the “exported on” preamble already removed.
• Naming columns by type and role so a host can route them: amounts to arithmetic, dates to calendars, identifiers to lookups.

Out of scope #

• Binary or nested formats: spreadsheets, PDFs, XML, deeply nested JSON. Tabulate reads text with one record per line or per block.
• Files whose rows change shape half-way, or whose structure only becomes clear after the fortieth line; the plan is made from the head of the file.
• Semantics beyond the column: it does not join, deduplicate, validate against a schema or fill gaps.

Choose Tabulate Extra when #

• The files come from reports and statements rather than clean exports: a title line, a totals row, a European or accounting number style.
• Twenty megabytes of weights and twice the latency are acceptable.
• Otherwise pick Tabulate, the default, or Tabulate Nano for a device.

Specification #

Try it #

You send

“Export from Acme CRM ⏎ sku,qty,price,shipped ⏎ SKU-0041,12,"$1,204.50",yes ⏎ SKU-0042,3,$96.00,no”

You get back

Format csv with a header; columns sku (identifier), qty (integer), price (currency), shipped (boolean); two rows with 1204.5 and true as numbers and booleans; one preamble line skipped.

The same exchange as the API sees it:

{
  "text": "Export from Acme CRM\nsku,qty,price,shipped\nSKU-0041,12,\"$1,204.50\",yes\nSKU-0042,3,$96.00,no\n",
  "max_rows": 50
}

{
  "format": "csv",
  "delimiter": ",",
  "quote": "\"",
  "header": true,
  "columns": [
    { "name": "sku", "type": "identifier", "role": "id", "p_type": 0.98 },
    { "name": "qty", "type": "integer", "role": "quantity", "p_type": 0.99 },
    { "name": "price", "type": "currency", "role": "amount", "p_type": 0.97 },
    { "name": "shipped", "type": "boolean", "role": "status", "p_type": 0.99 }
  ],
  "rows": [
    { "sku": "SKU-0041", "qty": 12, "price": 1204.5, "shipped": true },
    { "sku": "SKU-0042", "qty": 3, "price": 96, "shipped": false }
  ],
  "row_count": 2,
  "skipped": { "preamble": 1, "comment": 0, "blank": 0, "footer": 0 },
  "probs": { "format": [0.97, 0.01, 0.01, 0, 0, 0, 0, 0, 0.01], "header": 0.99 }
}

Limits & safety #

Tabulate sees only the text it is given: not the file name, its extension, its declared encoding or its origin. It does not see the bytes past the first forty lines when it plans, and it does not see any other file.

• It is a guesser, not a validator: a plausible plan for a file it has never seen the like of can still be wrong, and the executor will follow it faithfully. The probabilities and the skipped-line counts are there to be shown.
• It knows the dialects its generator produces. A layout outside them — a multi-line quoted cell, a file with two tables, an encoding that is not UTF-8 — is parsed as the nearest thing it knows.
• Types are named from a sample of twelve values; a column that changes type half-way is typed by whichever values were sampled.
• Conversion is by type, not by locale: 1.234,56 and 1,234.56 both become 1234.56, which is right for most files and wrong for a file where the comma is a thousands separator in one column and a decimal mark in another.
• It reads text of at most 256,000 characters; a larger file has to be cut by the host.
• Its output can echo anything in the input, including personal data; the host decides what to keep and where it goes.

Related models #

Latest versions #