Reading CSV files

Options

Every option is optional. The defaults are readr’s.

The type codes are ? (guess), l (bool), i (int), d (double), c (string), D (date), T (datetime), and _ (skip). phibble reads f (factor) and n (number) columns by requesting c and parsing in PHP.

Result

[
    'names' => list<string>,
    'columns' => list<null|array{
        type: 'bool'|'int'|'float'|'string'|'date'|'datetime',
        values: list<bool|int|float|string|null>,
        failed: int,
    }>,
    'problems' => list<string>,
]

Values use phibble’s storage form. Bools, ints, and floats are PHP scalars and strings are strings. A date is an int count of days since 1970-01-01. A datetime is an int count of microseconds since the epoch in UTC. NA is null. A skipped column is null. failed counts values that did not parse under the column’s explicit or guessed type; those values are null. phibble turns nonzero counts into its column 'x': n values could not be parsed as type problems.

problems holds one entry per ragged row, such as row 7: expected 5 columns, got 4. The row number is the record’s position in the file, counting skipped lines and the header. A multi-line quoted field counts as one record. A file that ends inside a quoted field adds row N: unterminated quoted field at end of file.

Reading in chunks

Phit\CsvReader reads a file a chunk of rows at a time. It is the native reader behind phibble’s DataFrame::csvChunks() and fromCsvChunked(), which follow readr’s read_delim_chunked(). It holds the text of one chunk and one read block, so its memory follows the chunk size and not the file size.

$reader = new Phit\CsvReader(string $path, array $options = []);
$reader->names(): array;           // list<string>
$reader->next(int $rows): ?array;  // the next chunk, or null

The options are read_csv()’s. The constructor reads the header, so names() is available before the first chunk. It returns the header’s names, X1..Xn when col_names is false, or [] for an empty file. next() returns at most $rows rows in read_csv()’s result shape, with two more keys: position, the 1-based index of the chunk’s first row, and rows, the chunk’s row count. It returns null once no rows remain.

Chunks follow phibble’s chunked reader:

• Types not given in types are guessed from the first chunk, from guess_max of its rows sampled as above. Every later chunk is parsed as those types. phibble sets guess_max to the chunk size by default, as readr does. A value in a later chunk that does not fit its column’s type counts in that chunk’s failed.
• Problem rows are numbered from the start of the file. Each chunk carries the problems of its own rows.
• The unterminated-quote problem goes with the last chunk. After each chunk the reader looks past blank lines to see whether records remain.
• n_max caps the rows across all chunks.
• A file with no data rows gives no chunks. The one exception is an unterminated quote in the header: next() then returns a chunk with no rows that carries the problem.

Records

• Short rows are padded with NA. Long rows put the extra fields, joined by the delimiter, into the last column.
• A UTF-8 byte order mark at the start of the file is removed before tokenizing (when skip is 0), so a quoted first header loses its quotes. One at the start of the first record after skipped lines is also removed.
• LF and CRLF line endings are accepted. CR is read as the line ending when the first line break falls in a 4 MiB stretch of the file (counted from the first byte after skipped lines) that holds no LF. phibble reads 4 MiB blocks and decides from the first block with a line break, so the two agree. Otherwise a lone CR is data. Quoted fields can hold delimiters and line breaks, which are kept as written.
• A line is blank when its raw text holds nothing but spaces, tabs and carriage returns. That is the raw line and not the parsed fields, so a quoted empty string is a field holding data, and it does not depend on trim_ws or on the delimiter: a line of spaces is blank even when the delimiter is a space. Under skip_empty_rows a blank line is dropped; otherwise it is tokenized like any other line, so an empty line is one empty field and a wider data frame records it as one column short. A last line the file left unterminated is never dropped, so a\n is two rows and a\n \n is one.
• An unterminated quote at the end of the file closes the field there.
• Bytes are passed through unchanged. There is no encoding validation.
• An empty file, or one with only empty lines, returns no names and no columns. When the header is read and no data rows are, the names come back with empty columns typed string, following readr, which reads a header-only file as text. Four cases reach that state: n_max: 0, a header-only file, a skip that leaves no row after the header, and every row empty under skip_empty_rows. An explicit code in types still applies.

phibble tokenizes by vroom’s (readr 2’s) quote rules with a fast reader when it can (no comment, "" escapes, no backslash escapes, and a quote character) and with its own character-level reader otherwise. The native reader reproduces both. The two differ in small ways:

• In the fast reader, a field is quoted when its first byte is the quote. Inside it, every lone quote toggles quoting and is dropped, and a doubled quote is a literal quote, so "x "y" z" reads as x y z. Quotes in an unquoted field are literal.
• The character-level reader folds CRLF to LF everywhere, including inside quotes. A quote opens a quoted section only when the field so far is whitespace. Text after the comment prefix ends the record, so a line holding only spaces and a comment is a one-field record holding those spaces, unless no record has been read yet: before the first one, a comment prefix with only whitespace in front of it drops the line, as readr drops it.

Guessing

Values are trimmed and NA-mapped first. The guess uses every row when there are at most guess_max. Otherwise it uses guess_max - 1 evenly spaced rows (step = intdiv(rows, guess_max): rows 0, step, 2·step, and so on) plus the last row, as vroom does. The first rule that fits every sampled value wins.

1. bool: T F TRUE FALSE True False true false t f.
2. int: an optional sign, then digits with optional grouping marks between them, that fits in 64 bits. An unsigned leading zero (007) stays text; 0 and -07 are ints.
3. double: [+-]inf or [+-]nan (any case), or a decimal number with an optional decimal mark, exponent, and grouping marks. 1. and 1, qualify. The leading-zero rule applies: 0e5 and 00.5 stay text.
4. date: YYYY-MM-DD or YYYY/MM/DD (the separators may differ).
5. datetime: YYYY-MM-DD[(T| )HH[:MM[:SS[.fraction]]]][Z|±HH[[:]MM]].
6. Otherwise string.

An all-NA column with at least one data row is bool. phibble differs from readr in one rule: whole numbers guess as int, where readr guesses double. The date and datetime rules check only the shape. A column whose values look like dates guesses date even when a value is not a calendar date, and that value then fails to parse.

Parsing

The column’s type is applied to every row. A guessed numeric column allows the grouping mark. An explicitly typed one does not, as in phibble.

• bool also accepts 1 and 0.
• int accepts leading zeros and a + sign.
• double accepts PHP numeric strings (when the decimal mark is .), inf and nan, and the guessing grammar.
• date needs a valid calendar date in years 0001 to 9999.
• datetime needs a valid date, hours up to 23, and minutes and seconds up to 59. Fractions are truncated to microseconds. An offset converts to UTC and no offset means UTC. Offsets up to ±24:59 are accepted, as by PHP’s DateTimeImmutable. A date alone is midnight UTC.

Errors

A missing or unreadable file and invalid options throw \ValueError (not the statistical exception). Invalid options include an unknown key, a wrong value type, a multi-byte delimiter, quote, decimal mark, or grouping mark, an unknown type code, and a negative skip or n_max. The decimal and grouping marks must differ and cannot be a digit, sign, e, E, or newline.

Differences from readr

These follow phibble’s reader:

• Whole numbers guess as int.
• With a comment prefix, a value followed by a comment is trimmed (2 # x reads as 2).
• Date and datetime guessing checks the shape only.
• In a file whose only line ending is CR, readr skips a blank line only before the first record and keeps the interior and trailing ones; both readers read CR as the line ending and then apply the blank-line rule throughout.

Performance

On an Apple Silicon Mac with PHP 8.5, a 1,000,000-row file with int, float, string, date, and datetime columns (59 MB) reads in about 0.11 seconds, including building the PHP arrays. phibble’s csv_guess benchmark (1,000,000 rows) takes 0.12 seconds through DataFrame::fromCsv() with the extension and 1.0 seconds with phibble’s PHP reader.

Verification

tests/csv.php checks quoting, escapes, NA handling, trimming, every option, every type code, the guessing rules, datetimes, ragged rows, chunks, and the error cases.

tests/csv-conformance.php checks read_csv() against phibble’s reader without a phibble checkout. phibble generates tests/fixtures/csv-conformance.json from its own PHP reader with bin/generate-csv-conformance, and this repository commits a byte-identical copy. The file holds 108 cases: every option at its default and at one other value, all eight type codes, ragged rows, a BOM, CRLF and CR line endings, an unterminated quote, comments, an empty file, bytes that are not valid UTF-8, the four ways to read no data rows, and the blank-line rule in its eighteen shapes. Each case records the input, the options, and the names, types, values, failure counts, and problems phibble produces. The suite requires read_csv() to reproduce all of it, so a change in either reader fails in its own repository. It reads every case again at block sizes of 1, 2, 3, 5, 8, and 64 bytes, and through CsvReader in one chunk and in chunks of 1, 2, and 3 rows. One chunk must equal the whole read. Small chunks must give the same problems in order and, read as text, the same values.

The copy is pinned. tests/fixtures/PHIBBLE_REFERENCE names the phibble commit it came from, and bin/fetch-csv-fixtures copies the file from that commit, either from a local checkout given as its first argument or in PHIBBLE_REPO, or from a clone of the repository named in the pin. Refreshing the corpus means regenerating it in phibble, moving the pin, and running that script.

phibble’s reader does not report an unterminated quoted field and read_csv() does. Each case therefore carries a closed list of the extra problems read_csv() is expected to add, and the suite compares its problems with the union rather than filtering by pattern, so an unexpected problem fails.

tests/csv-phibble.php is the deep tier and runs when PHIBBLE_AUTOLOAD points at phibble’s vendor/autoload.php. It compares both readers through DataFrame::fromCsv() on the fixtures and on thousands of random files and option combinations, with the case count and the seed taken from CSV_FUZZ_CASES and CSV_FUZZ_SEED. It requires identical names, types, values, and warnings. With a phibble that has DataFrame::csvChunks(), it also compares CsvReader with phibble’s PHP chunked reader on random files, chunk sizes, and native block sizes (CSV_CHUNK_CASES of them). It requires the same chunks, positions, types, values, and per-chunk warnings, in the same order.