When a CSV Is Too Big for a Spreadsheet

Excel has a hard ceiling of 1,048,576 rows per sheet. It is not a performance guideline — it is the maximum, and a file with more rows does not partially open. In many versions it loads the first million and change, and whether it warns you clearly enough that anyone notices is a matter of luck. An analysis run against a silently truncated dataset is worse than one that failed outright.

In practice the usable limit arrives well before that. Long before a million rows, recalculation across a few formula columns turns every change into a wait, and memory use climbs to several times the file size because a spreadsheet holds a rich object model for every cell rather than just its value.

The deeper difference is not capacity, it is that you describe the result you want rather than the steps to produce it. A grid asks you to build the answer by hand — sort, filter, add a helper column, pivot, copy the result somewhere. A query states the question once.

SELECT country,
       COUNT(*)        AS orders,
       SUM(total)      AS revenue,
       AVG(total)      AS average_order
FROM 'orders.csv'
WHERE order_date >= '2026-01-01'
GROUP BY country
HAVING COUNT(*) > 100
ORDER BY revenue DESC;

That is a pivot table, a filter, a sort and a threshold in nine lines — and unlike the equivalent sequence of spreadsheet operations, it is written down. You can read it back in a month and know exactly what was counted, change one clause and re-run, or hand it to someone else as a precise description of the analysis.

This is the part that matters for work anyone depends on. A spreadsheet records the result; a query records the method. When a number is questioned, the query is the answer to how it was produced, and a chain of manual steps performed weeks ago is not.

CSV has no real standard, which is why files that look identical parse differently. The failure modes are worth recognising because they produce plausible-looking wrong answers rather than errors.

• →The separator is not always a comma. Locales that use a comma as the decimal mark commonly export semicolon-separated files, still named .csv.
• →Quoted fields can contain the delimiter, and quotes are escaped by doubling them. A parser that splits on commas without handling quoting will silently misalign every column after the first address field containing a comma.
• →Fields can contain newlines when quoted, so a line in the file is not necessarily a row in the data. Counting lines to count records is a habit that eventually produces a wrong total.
• →Encoding is not declared anywhere in the format. A file with accented characters can only be read correctly if you already know how it was written.
• →There is no type information. Every value is text until something decides otherwise, and what decides is the tool, using heuristics.

None of this makes CSV a bad interchange format — its universality is genuinely valuable. It does mean that opening one and trusting what appears is optimistic, and that a tool which lets you inspect and declare types explicitly is doing something useful rather than pedantic.

Parquet stores data by column rather than by row, and that one change explains most of its advantages for analysis.

A query that sums one column out of forty has to read the entire file when rows are stored together, because each row's forty values are interleaved. With columnar storage it reads only the column it needs and skips the rest — often a tenfold reduction in work before any computation happens.

Compression benefits from the same arrangement, because a single column holds values of one type that are often similar to each other — a column of country codes compresses far better as a block than the same values scattered through rows of mixed data. Parquet also stores per-chunk statistics, so a query filtered to one date range can skip whole sections of the file without decompressing them.

The trade-off is that you cannot open it in a text editor, and it is a poor choice for small files a human needs to read. For anything large enough that this guide is relevant, converting once to Parquet usually pays for itself immediately.

The reason this no longer requires installing a database is WebAssembly: a compilation target that lets code written in systems languages run in a browser at close to native speed. A full analytical database engine compiled to WebAssembly runs inside the page, reading files straight from your disk.

The practical consequence is that the awkward middle ground has largely disappeared. Files too big for a spreadsheet but not worth provisioning infrastructure for used to mean a request to a data team and a wait. Now the same query runs where the file already is.

The instinct with a new file is to start answering the question you came with. Profiling first is faster overall, because it surfaces the problems that would have quietly corrupted that answer.