CSV dialects, separators, and quoting
| Dialect | Field separator | Quote char | Escape rule | Defined by |
|---|
| RFC 4180 | , | " | double the quote: "" | IETF RFC 4180 (2005) |
| Excel CSV | , (locale-dependent) | " | double the quote | de facto, Microsoft Office |
| TSV (IANA) | \t | optional " | backslash or doubled quote | IANA text/tab-separated-values |
| TSV (Unix) | \t | none | tabs and newlines forbidden in fields | de facto, Unix tools |
| MySQL OUTFILE | , | " | backslash: \" | MySQL SELECT INTO OUTFILE |
| PostgreSQL COPY | , | " | double the quote | PostgreSQL COPY ... CSV |
| European semicolon | ; | " | double the quote | Excel, decimal-comma locales |
| Pipe-delimited | | | usually none | often forbid pipes in fields | de facto, mainframe and ETL |
The most common silent failure sits at the boundary between RFC 4180 (quote-doubled) and MySQL (backslash-escaped): a field holding a literal " reads fine in the dialect that wrote it and corrupts in the other.
| Separator | Hex | Decimal | Used by | Detected by Rowson |
|---|
| Comma | 0x2C | 44 | RFC 4180, Excel (en-US), MySQL, PostgreSQL | yes (priority 1) |
| Tab | 0x09 | 9 | IANA TSV, Unix TSV | yes (priority 2) |
| Semicolon | 0x3B | 59 | Excel (European locales) | yes (priority 3) |
| Pipe | 0x7C | 124 | mainframe, ETL, Apache Hive | yes (priority 4) |
Rowson detects the top four. Anything else, such as the ASCII unit separator (0x1F), a caret, or a colon, needs a pre-processing step that swaps the separator for a tab or comma before paste.
For quoting, RFC 4180, Excel, and PostgreSQL COPY all quote a field when it contains a delimiter, a quote, or a CR/LF, and write a literal quote as two adjacent quotes:
id,name,note
1,"Smith, John","She said ""hello""."
MySQL OUTFILE instead backslash-escapes (\"), which is not RFC 4180. Round-tripping through it and back into a default parser mis-reads any field with a literal backslash. IANA TSV backslash-escapes tab and newline inside fields; the looser Unix convention forbids them entirely and uses no quoting, which is what awk -F'\t', cut, and sort assume.
JSON output formats and where each is used
| Format | Specification | MIME type | Extension | Typical consumer |
|---|
| Array of objects | RFC 8259 / ECMA-404 | application/json | .json | JS apps, REST request bodies |
| JSONL | jsonlines.org | application/x-ndjson | .jsonl, .ndjson | BigQuery bq load, DuckDB, OpenAI fine-tuning |
| Array of arrays | RFC 8259 | application/json | .json | Chart libraries (Plotly, Highcharts) |
| Keyed object | RFC 8259 | application/json | .json | Lookup tables, ID-indexed cache loads |
Each format trades a property the others keep.
| Property | Array of objects | JSONL | Array of arrays | Keyed object |
|---|
| Preserves all rows when keys collide | yes | yes | yes | no |
| Preserves header names | yes | yes | first row only | yes |
| Parseable line by line | no | yes | no | no |
| Parseable as a single JSON value | yes | no | yes | yes |
| Round-trips back to CSV cleanly | yes | yes | yes | no (lossy) |
| Supports streaming insert | no | yes | no | no |
JSONL is the only line-orientable format; the other three require the consumer to buffer the whole document before parsing. For a multi-hundred-megabyte file destined for a data warehouse, JSONL is the only viable choice.
Encoding, headers, and parser libraries
| Encoding | BOM bytes | What modern parsers do |
|---|
| UTF-8 (no BOM) | none | parse as-is |
| UTF-8 with BOM | 0xEF 0xBB 0xBF | PapaParse 5.4.1+, Python csv 3.9+, Go 1.20+ strip it |
| UTF-16 LE/BE | 0xFF 0xFE / 0xFE 0xFF | most parsers fail; pre-decode to UTF-8 |
| Windows-1252 | none | decoded as Latin-1 unless the reader sniffs |
When an Excel-saved CSV reads back as mojibake, the cause is almost always Windows-1252 vs UTF-8; save as “CSV UTF-8 (Comma delimited)” instead.
| Header convention | First row contains | Common in |
|---|
| Headers present | column names | Excel exports, web downloads, API fixtures |
| No headers | data from row 1 | sensor logs, older database exports |
| Commented headers | #-prefixed rows above data | Unix tools, R read.csv(comment.char="#") |
Rowson assumes headers present. For headerless input, prepend a synthetic header row or pick array-of-arrays and let the consumer assign keys.
| Language | CSV library | JSON output |
|---|
| JavaScript / TS | PapaParse (Rowson uses this) | array of objects; auto-delimiter via delimiter: "" |
| Python | pandas.read_csv | to_json(orient="records"); add lines=True for JSONL |
| Go | encoding/csv (stdlib) | manual; pair with encoding/json |
| Java | OpenCSV, Apache Commons CSV | manual; pair with Jackson |
| Shell | csvkit (csvjson) | AoO, plus --stream for JSONL |