swift

Intuition Lies: Lessons from a CSV Parser

Intuition Lies: Lessons from a CSV Parser — Part 1: Twenty Seconds

A CSV with 260 MB and 5.6 million rows: reading it the most intuitive way costs 11 seconds and 782 MB, but the culprit isn't what it seems.

Part 1 of 3

View all parts

• Part 1 Intuition Lies: Lessons from a CSV Parser — Part 1: Twenty Seconds
• Part 2 Intuition Lies: Lessons from a CSV Parser — Part 2: Correct, Then Fast
• Part 3 Intuition Lies: Lessons from a CSV Parser — Part 3: The 9%

I’m playing with static GTFS data from my city’s public transportation: a zip file with CSV files describing routes, stops, times, and lines.

They’re generally small files, just a few megabytes. All except one: stop_times.txt contains the scheduled arrival and departure times of every trip at each stop along the route.
It’s a 260 MB file.
5,617,443 rows to read and split into various fields.

trip_id,arrival_time,departure_time,stop_id,stop_sequence,stop_headsign,pickup_type,drop_off_type,shape_dist_traveled,timepoint
1#1-2,07:30:00,07:30:00,74761,1,,,,0,1
...
1#1-2,07:34:26,07:34:26,74763,2,,,,1167,0

Reading it the easiest way possible takes 11 seconds and 782 MB. But with a bit of work we can get down to 0.34 seconds and 250 MB.
Not bad, right?

That second bar isn’t a drawing error: 0.34 seconds against 10.97 is 3% of the first.

Between the two results are several hours of work, but above all a long series of choices. Unfortunately, what they have in common isn’t that they were right.
In fact, quite a few were wrong.
The most expensive mistake? Most of the effort on a phase that accounts for just 9% of the total.

But there’s something more important that ties them together: each one was made by trying to prove it false.
Every time something seemed obvious, I went looking for the thing that would prove the opposite: a benchmark, a test built to distinguish two different ways of failing. Sometimes even pen and paper.

When we get to cite Knuth, it’s almost always while cutting off the word that counts, premature:

«Premature optimization is the root of all evil».
- Knuth’s Optimization Principle

Except for Knuth the problem was never spending time optimizing, but knowing where to do it.

When writing code it’s too easy to trust the first solution that works.
It’s a shortcut we always take in life, because in most cases it’s a great way to save resources.

But here I tried to think differently.
This is the story of all those mistakes and where they took me.

It’s Not I/O

Reading a CSV file in itself is trivial:

let data = try String(contentsOf: url, encoding: .utf8)
for row in data.components(separatedBy: "\n") {
    let fields = row.components(separatedBy: ",")
}

The complete code from these articles can be download from Github.

Four lines of code, they’re easy to read, they work.
On a CSV of a few megabytes nobody would complain.

When we try it on something larger though, the times skyrocket: on the file above it takes 11 seconds using a whopping 782 MB of memory.
Not exactly efficient, is it? On a mobile phone it’s downright criminal.

The obvious question here is: where the hell does all this time go?
The answer that would seem obvious and natural is: reading 260 MB from disk.

And that’s what I thought too, except I was completely wrong.
Disproving this theory is easy: with a small test we measure the two main actions, reading the file and splitting it into rows and fields.

The test: the classic version, measuredThe complete code is on GitHub, along with the file used for measurements.

	Time
String(contentsOf:) alone	0.112s
+ split rows and fields	10.86s

Surprisingly reading the file accounts for just 1% of the time.
Everything else comes after, when those bytes become objects.

Why isn't it an I/O problem?

260 MB in 0.112 seconds is 2.3 GB/s. An NVMe SSD can do that, so in theory that file could have actually come from disk. But almost certainly it didn't.

The operating system keeps recently read file contents in RAM using the page cache.

This means reads of the same data at different times give very different times if done cold (from disk) or hot (from memory). We'll discover later that the difference is even 5x.

From here on all numbers are warm and in release builds (more on why the latter isn't a minor detail later, at my expense). Before each measurement there's a warm-up run that brings the file into cache. Not because it's the "true" number, but because it's what the code actually competes against, not the disk.

What happens is that components(separatedBy:) builds a String for every row and one for every field: over sixty million allocations, each one with its chunk of heap to request, fill, and eventually return.

The counterproof is to use split(separator:): it does the exact same work, but returns Substring—slices of the original string without making copies.

And the time drops to 7.26 seconds from 10.86.
A third less time just by stopping copying.
Same algorithm, same structure, everything the same.

Three Decisions Made Too Soon

At this point I knew only two things: that most of the time went into allocations, and that therefore I had to stop making them.
I knew nothing else.

Too bad it was also time to make decisions, and as always happens, it was also the worst time to make them.
The point is that most structural choices are made when you have the least information you’ll ever have, but they’re also the ones that will constrain you.

By the end I’d already made three in the first half hour.

1. Choose How to Read the File

There are two ways to do it and they’re very different.

Block Reading

You ask the system for 64 KB at a time, process them, and ask for another 64 KB. This way you avoid loading the entire file into memory. It’s fantastic because it works with any source: a file on disk, an HTTP response, a socket.
The price though is that every byte ends up getting copied: the kernel reads from disk and brings it into cache, then copies it to your buffer.
In our case we’re talking about two hundred sixty megabytes of copies that serve no purpose, if we’re only going to look at them.

File Mapping with mmap

The second way is mmap and the first time you encounter it, it seems like a damn trick. Instead of asking for data, you ask the operating system to pretend the file is already in memory: you tell it “map this file for me,” and it returns you a pointer. From that moment the file is one big array of bytes that you can index like any other.

The bytes, physically, aren’t there: when we read a position not yet available, the CPU will page fault and the kernel will go get the page from disk to put in memory.
There’s no copy: it’s the operating system that does the heavy lifting, page by page, as we walk through the file.

The problem is that mmap wants a real file: you can’t map a socket, because the bytes don’t exist yet and there’s nothing to map until they arrive.

In the end I decided to lean toward block reading, convinced it would be simpler to implement and more versatile.

2. Choose What to Index

Convinced that reading the whole thing would be inefficient, there was now the question of what to pull out from analyzing the file.
The idea was to scan it just once, at opening, and note the start of every row.
A clever idea for jumping where needed when requested.

The strong temptation here was to also note the start of every field on every row.
That means immediate access to any column, zero work after.

But with 5.6 million rows for ten columns there are a beautiful 56 million positions to keep in memory: hundreds of MB of indexes alone, for a file that weighs 260. I’d have had a map almost as big as the file itself. Damn little useful.

I decided I’d keep only the rows, and get the fields on-the-fly when they were requested.

3. Guess Headers or Not?

Now we’re inside the format: a CSV file can start with data or have a header row with column names.

The clever temptation was to “figure out” if there’s a header or not: it’s done through various heuristics.
For example, read the values of the first row: if they’re all strings (alphabetic text) and the rows below contain numbers or dates in the same columns, it’s very likely the first row is a header.
Refinements on this rule or variants of it can accurately estimate this thing.

The problem isn’t that it works poorly. It’s that when it’s wrong, it’s silently wrong: it eats a row of data mistaking it for a header, or invents columns called 1#1-2 and 07:30:00. And nobody notices until it’s too late.

No point in guessing; simpler to make it explicit in the API.
Anyone using the library already knows perfectly well if their file has a header.

The First Version

Ready to assemble a first version:

var offsets: [Int] = [0] // map of row start indices
var fileOffset = 0 // cursor for navigating the file

// This state must survive from one block to the next:
// a field can easily span the 64 KB boundary.
var atFieldStart = true
var fieldIsQuoted = false
var inQuotes = false

while true {
    let data = fileHandle.readData(ofLength: config.bufferSize) // read the buffer
    guard !data.isEmpty else { break }

    data.withUnsafeBytes { raw in
        // Associates a specific data type to a region of "raw" (untyped) memory,
        // returning a typed pointer that allows safe interaction for the compiler.
        let bytes = raw.bindMemory(to: UInt8.self)
        for i in 0..<bytes.count {
            let b = bytes[i]

            // ① First byte of a field. It's here, and only here, that a quote
            //    has the power to open a quoted field: later in the field
            //    it would be a character like any other.
            if atFieldStart {
                atFieldStart = false
                if b == quoteByte {
                    fieldIsQuoted = true
                    inQuotes = true
                    continue        // the opening quote is not content
                }
                fieldIsQuoted = false
            }

            // ② Inside a quoted field, every quote makes us enter or exit.
            if fieldIsQuoted && b == quoteByte {
                inQuotes.toggle()

            // Outside quotes separators really count.
            // Inside, they're data and should be ignored: that's what `!inQuotes` is for.
            } else if !inQuotes {

                // ③ A comma closes the field and opens another.
                if b == delimiterByte {
                    atFieldStart = true
                    fieldIsQuoted = false

                // ④ A newline closes the row: the next one starts at the byte after.
                } else if b == 0x0A {
                    atFieldStart = true
                    fieldIsQuoted = false
                    offsets.append(fileOffset + i + 1)
                }
            }
        }
    }

    fileOffset += data.count // advance through the file
}

offsets starts with [0] because the first row always starts at the beginning of the file: there’s no newline before it.

The three variables tell us, byte by byte, where we are inside the row:

• if we’re about to start a new field (atFieldStart)
• if that field is opened by a quote (fieldIsQuoted)
• whether right now we’re inside or outside the quotes (inQuotes)

In CSV, a newline doesn’t always mean end of row: inside a quoted field it’s a character like any other, and the reader needs to know that. We’ll come back to this since it’s considerably more treacherous than this.

They stay outside the while, and it’s not a style choice.
A quoted field can start at byte 65,000 and end in the next block: if the state reset every loop, the parser would forget it was inside quotes right in the middle of the field.

That’s the price of reading in blocks.

Then I ran the test from the terminal: swift test.

Twenty Seconds

… The test took twenty seconds, twice the four lines we started with: 1.8x slower than the code I was trying to beat.

My first instinct was to hunt for the culprit in those four ifs repeated for every byte, multiplied by two hundred sixty million bytes. They had to weigh on the count somehow.

I was already poised for a refactor, when I remembered I should run the test in release to have a useful benchmark:

swift test              21.24s
swift test -c release    0.38s

0.38s, fifty-five times smaller. Same code, same file, same machine.
The only difference is that in the first case the compiler doesn’t optimize anything.

But what's happening in debug?

In debug every access to bytes[i] carries with it a bounds check. The subscript of UnsafeBufferPointer doesn't get inlined, so it's a real function call for every byte. And the three state variables, captured by a closure, end up on the heap with an exclusive access check at runtime for every read and write.

That's four checks per byte, over two hundred sixty million bytes. All told, over two hundred clock cycles just to compare a byte with two constants.

In the end there was nothing to rewrite, it was just a parameter.
I had a parser that in release did 0.38 seconds.
Too bad it didn’t work at all on any file slightly harder than this one.

Part 1 of 3

View all parts

• Part 1 Intuition Lies: Lessons from a CSV Parser — Part 1: Twenty Seconds
• Part 2 Intuition Lies: Lessons from a CSV Parser — Part 2: Correct, Then Fast
• Part 3 Intuition Lies: Lessons from a CSV Parser — Part 3: The 9%

• #swift
• #craft

More on swift

• L'intuito mente: lezioni da un parser CSV — Parte 3: Il 9% Sep 2026
• L'intuito mente: lezioni da un parser CSV — Parte 2: Corretto, poi veloce Sep 2026
• Intuition Lies: Lessons from a CSV Parser — Part 3: The 9% Sep 2026