Tabulate

Overview #

Tabulate turns a data file into records. You give it the text of the file as you found it — a CSV with a two-line preamble from the tool that exported it, a tab-separated dump with a totals line at the bottom, a fixed-width report, a block of key–value pairs, a log, JSON lines, a Markdown table — and it gives back the file's format, its columns with a name, a type and a role, and its rows as JSON with the values converted: numbers as numbers, dates as ISO strings, yes/no as booleans, the junk lines left out.

It is the only member of the Tabulate series and family, and the only Falcon model whose output is a table rather than a label, a sentence or a sound. It does not summarise, transform or validate the data; it recovers the structure the file already has.

Tabulate is two small networks around a deterministic executor. The layout network reads the first forty lines byte by byte and decides what kind of file this is, what separates the cells, how they are quoted, whether the first data line is a header, and which lines are preamble, comments, blanks or footers rather than data. The executor applies that plan to the whole file with ordinary parsing code. The typer network then reads a sample of each column's values, with the header name when there is one, and names the column's type and role, which decides how the executor converts the values. Everything runs in the host process; nothing is sent anywhere but the file's own text.

Intended use #

• Ingesting files people upload or paste: a spreadsheet export, a bank statement download, a sensor log, a configuration dump, without a parser per format.
• Turning a data file into JSON for a language model, a chart or a form, with the totals line and the “exported on” preamble already removed.
• Naming columns by type and role so a host can route them: amounts to arithmetic, dates to calendars, identifiers to lookups.

Out of scope #

• Binary or nested formats: spreadsheets, PDFs, XML, deeply nested JSON. Tabulate reads text with one record per line or per block.
• Files whose rows change shape half-way, or whose structure only becomes clear after the fortieth line; the plan is made from the head of the file.
• Semantics beyond the column: it does not join, deduplicate, validate against a schema or fill gaps.

Choose Tabulate when #

• The input is a text data file of unknown dialect and the host needs rows now, in-process, without configuration.
• A wrong guess is cheap to see: the response carries the plan, its probabilities and the skipped lines, so a host can show them and let a person correct the delimiter or the header.
• Otherwise, when the format is already known and fixed, an ordinary parser configured once will be simpler.

Specification #

Try it #

You send

“Export from Acme CRM ⏎ sku,qty,price,shipped ⏎ SKU-0041,12,"$1,204.50",yes ⏎ SKU-0042,3,$96.00,no”

You get back

Format csv with a header; columns sku (identifier), qty (integer), price (currency), shipped (boolean); two rows with 1204.5 and true as numbers and booleans; one preamble line skipped.

The same exchange as the API sees it:

{
  "text": "Export from Acme CRM\nsku,qty,price,shipped\nSKU-0041,12,\"$1,204.50\",yes\nSKU-0042,3,$96.00,no\n",
  "max_rows": 50
}

{
  "format": "csv",
  "delimiter": ",",
  "quote": "\"",
  "header": true,
  "columns": [
    { "name": "sku", "type": "identifier", "role": "id", "p_type": 0.98 },
    { "name": "qty", "type": "integer", "role": "quantity", "p_type": 0.99 },
    { "name": "price", "type": "currency", "role": "amount", "p_type": 0.97 },
    { "name": "shipped", "type": "boolean", "role": "status", "p_type": 0.99 }
  ],
  "rows": [
    { "sku": "SKU-0041", "qty": 12, "price": 1204.5, "shipped": true },
    { "sku": "SKU-0042", "qty": 3, "price": 96, "shipped": false }
  ],
  "row_count": 2,
  "skipped": { "preamble": 1, "comment": 0, "blank": 0, "footer": 0 },
  "probs": { "format": [0.97, 0.01, 0.01, 0, 0, 0, 0, 0, 0.01], "header": 0.99 }
}

Limits & safety #

Tabulate sees only the text it is given: not the file name, its extension, its declared encoding or its origin. It does not see the bytes past the first forty lines when it plans, and it does not see any other file.

• It is a guesser, not a validator: a plausible plan for a file it has never seen the like of can still be wrong, and the executor will follow it faithfully. The probabilities and the skipped-line counts are there to be shown.
• It knows the dialects its generator produces. A layout outside them — a multi-line quoted cell, a file with two tables, an encoding that is not UTF-8 — is parsed as the nearest thing it knows.
• Types are named from a sample of twelve values; a column that changes type half-way is typed by whichever values were sampled.
• Conversion is by type, not by locale: 1.234,56 and 1,234.56 both become 1234.56, which is right for most files and wrong for a file where the comma is a thousands separator in one column and a decimal mark in another.
• It reads text of at most 256,000 characters; a larger file has to be cut by the host.
• Its output can echo anything in the input, including personal data; the host decides what to keep and where it goes.

Related models #

Latest versions #