Tabulate Nano

Overview #

Tabulate Nano turns a data file into records. You give it the text of the file as you found it — a CSV with a two-line preamble from the tool that exported it, a tab-separated dump with a totals line at the bottom, a fixed-width report, a block of key–value pairs, a log, JSON lines, a Markdown table — and it gives back the file's format, its columns with a name, a type and a role, and its rows as JSON with the values converted: numbers as numbers, dates as ISO strings, yes/no as booleans, the junk lines left out.

It is the Alternate of the Tabulate family: the same catalogues, the same executor and the same response as Tabulate, from the small size of the two networks (width 128, two layers) rather than the base size. A host selects it with model: "tabulate-nano".

Tabulate is two small networks around a deterministic executor. The layout network reads the first forty lines byte by byte and decides what kind of file this is, what separates the cells, how they are quoted, whether the first data line is a header, and which lines are preamble, comments, blanks or footers rather than data. The executor applies that plan to the whole file with ordinary parsing code. The typer network then reads a sample of each column's values, with the header name when there is one, and names the column's type and role, which decides how the executor converts the values. Everything runs in the host process; nothing is sent anywhere but the file's own text.

Intended use #

• Ingesting files people upload or paste: a spreadsheet export, a bank statement download, a sensor log, a configuration dump, without a parser per format.
• Turning a data file into JSON for a language model, a chart or a form, with the totals line and the “exported on” preamble already removed.
• Naming columns by type and role so a host can route them: amounts to arithmetic, dates to calendars, identifiers to lookups.

Out of scope #

• Binary or nested formats: spreadsheets, PDFs, XML, deeply nested JSON. Tabulate reads text with one record per line or per block.
• Files whose rows change shape half-way, or whose structure only becomes clear after the fortieth line; the plan is made from the head of the file.
• Semantics beyond the column: it does not join, deduplicate, validate against a schema or fill gaps.

Choose Tabulate Nano when #

• The parser has to run on a device or inside a memory budget where 2.5 MB of weights is welcome and 8.6 MB is not.
• The files are of the common dialects; the held-out figures say how much the smaller networks give up.
• Otherwise pick Tabulate, the default, or Tabulate Extra for the widest coverage.

Specification #

Try it #

You send

“Export from Acme CRM ⏎ sku,qty,price,shipped ⏎ SKU-0041,12,"$1,204.50",yes ⏎ SKU-0042,3,$96.00,no”

You get back

Format csv with a header; columns sku (identifier), qty (integer), price (currency), shipped (boolean); two rows with 1204.5 and true as numbers and booleans; one preamble line skipped.

The same exchange as the API sees it:

{
  "text": "Export from Acme CRM\nsku,qty,price,shipped\nSKU-0041,12,\"$1,204.50\",yes\nSKU-0042,3,$96.00,no\n",
  "max_rows": 50
}

{
  "format": "csv",
  "delimiter": ",",
  "quote": "\"",
  "header": true,
  "columns": [
    { "name": "sku", "type": "identifier", "role": "id", "p_type": 0.98 },
    { "name": "qty", "type": "integer", "role": "quantity", "p_type": 0.99 },
    { "name": "price", "type": "currency", "role": "amount", "p_type": 0.97 },
    { "name": "shipped", "type": "boolean", "role": "status", "p_type": 0.99 }
  ],
  "rows": [
    { "sku": "SKU-0041", "qty": 12, "price": 1204.5, "shipped": true },
    { "sku": "SKU-0042", "qty": 3, "price": 96, "shipped": false }
  ],
  "row_count": 2,
  "skipped": { "preamble": 1, "comment": 0, "blank": 0, "footer": 0 },
  "probs": { "format": [0.97, 0.01, 0.01, 0, 0, 0, 0, 0, 0.01], "header": 0.99 }
}

Limits & safety #

Tabulate sees only the text it is given: not the file name, its extension, its declared encoding or its origin. It does not see the bytes past the first forty lines when it plans, and it does not see any other file.

• It is a guesser, not a validator: a plausible plan for a file it has never seen the like of can still be wrong, and the executor will follow it faithfully. The probabilities and the skipped-line counts are there to be shown.
• It knows the dialects its generator produces. A layout outside them — a multi-line quoted cell, a file with two tables, an encoding that is not UTF-8 — is parsed as the nearest thing it knows.
• Types are named from a sample of twelve values; a column that changes type half-way is typed by whichever values were sampled.
• Conversion is by type, not by locale: 1.234,56 and 1,234.56 both become 1234.56, which is right for most files and wrong for a file where the comma is a thousands separator in one column and a decimal mark in another.
• It reads text of at most 256,000 characters; a larger file has to be cut by the host.
• Its output can echo anything in the input, including personal data; the host decides what to keep and where it goes.

Related models #

Latest versions #