Note: This piece started as a pitch to LWN. The editors passed on it, so it’s going up here instead.

Peripheral Component Interconnect Express (PCIe) failures are often reported in a form that is precise, information-dense, and nearly unreadable: a Transaction Layer Packet (TLP) header dumped as four raw 32-bit words in an Advanced Error Reporting (AER) log. Understanding what those bytes mean requires knowing the TLP format — and the format just changed.

What a TLP header looks like

When a PCIe device causes an error, the hardware records the offending packet’s header in the AER Header Log register. A typical kernel log or lspci -vv output shows it as four DWORDs:

HeaderLog: 00000001 0000220f 01070000 9eece789

For PCIe 1.0 through 5.0, the layout of those bytes has been stable. DW0 — the first four bytes — encodes the packet type, traffic class, and payload length. Everything else follows from what DW0 says:

DW0:  0x00000001
      Byte 0: 0x00 = 0b00000000
        [7:5] FMT   = 000   → 3DW header, no data
        [4:0] TYPE  = 00000 → Memory Read Request
      Byte 1: 0x00  → TC=0, Attr[2]=0
      Byte 2: 0x00  → TD=0, EP=0 (not poisoned), Attr[1:0]=00
      Byte 3: 0x01  → Length = 1 DW (4 bytes)

The FMT and TYPE fields together form a matrix that determines both the packet kind and the header length (3DW or 4DW). Once you know the format, you know where every subsequent field lives:

That table is the core of traditional TLP parsing. The same FMT+TYPE matrix, the same field positions, the same bit extraction — unchanged from 2003 to 2019. Sixteen years of stability.

Then Flit Mode happened

PCIe 6.0 introduced Flit Mode to support PAM4 signaling at 64.0 GT/s. Instead of variable-length packets delimited by STP/END framing tokens, TLPs are packed into fixed 256-byte containers called flits, each with its own forward error correction. That is a data-link layer change, but it reached up into the TLP format itself.

DW0 in Flit Mode carries a different set of fields. The FMT+TYPE matrix — the thing that has been the entry point for TLP parsing since PCIe 1.0 — is gone. In its place, there are thirteen type codes in a flat enumeration and a new set of header fields:

Classic DW0:  FMT[7:5] | TYPE[4:0] | TC | Attr | Length
Flit DW0:     Type[4:0] | T9 | TC | OHC-A | TH | Attr | AT | Length

TLP Prefixes — the extension mechanism in non-Flit framing — are replaced by Optional Header Components (OHC). The type encoding is not a subset or superset of the old one; it is a different numbering. The same raw DW0 bytes decode to a different packet type depending on whether the link was using Flit framing or not.

A handful of representative DW0 byte 0 values, side by side, makes the divergence concrete:

The first row is the pathological case: a byte that has meant “Memory Read, 3DW header” since 2003 now means “no operation” on a Flit Mode link. The 0x04 row is almost as severe — a Type 0 Configuration Read in classic PCIe is not a defined type at all in Flit Mode. Other rows align by design — Memory Write at 0x40, the AtomicOps at 0x4C/0x4E, the routed Message-with-Data at 0x70, and the TLP Prefix at 0x8D — because the spec writers preserved semantic continuity where they could. But neither the alignments nor the divergences are something the bytes themselves announce; the framing context is what selects the table.

The header does not tell you which interpretation is correct — the framing context does. The worked example in the next section will run 04000001 through the tool as a Type 0 Configuration Read in classic mode; on a Flit Mode link, those same bytes would not decode as a defined type at all.

No one expects you to memorize this

The field-level detail in those tables is useful for understanding the problem, but nobody should be decoding TLP headers by hand for every AER event. The PCIe maintainers agree — which is why the kernel’s PCI documentation now points users at tlp-tool, whose binary is named rtlp-tool.

On March 23, Lukas Wunner posted a patch titled “Documentation: PCI: Document decoding of TLP Header in AER messages”, arguing that the TLP Header hints at the root cause of an error but is often ignored because of its seeming opaqueness. Wunner noted that the obvious alternative — wireshark TLP dissectors — expects complete TLPs rather than just headers, and cannot consume the hex format the kernel emits directly; tlp-tool was, in his words, “the most cut and dried solution out there.” Bjorn Helgaas applied the patch to pci/for-linus for the 7.0 merge window the same day, tweaking the commit log on the way in to note that the Header Log lives in the AER Capability — which may be present on any PCIe function, not just root ports. The result is modest but significant: the kernel’s PCIe AER documentation now recommends tlp-tool as a way to decode the TLP Header into human-readable form, and ships a worked example — piping a kernel commit URL through rtlp-tool --aer — to make the recommendation directly executable. Wunner’s cover letter also floated going one step further — pointing users at the tool from a printk_once() message when the first AER error fires — but kept the patch itself to documentation only, calling that “probably sufficient” for now.

The lowest-level entry point is to hand the tool the four header words directly:

rtlp-tool -i "04000001 00200a03 05010000 00050100"

That is useful when a header is already in hand. The two modes that matter for real debugging map to where Header Logs actually surface in the kernel: dmesg AER messages, and the HeaderLog: line inside lspci -vv output.

--aer: kernel log input. A TLP Header copied straight from dmesg goes through --aer:

echo 'TLP Header: 04000001 00200a03 05010000 00050100' | rtlp-tool --aer

The output makes the bit extraction explicit:

+----------+------------------+--------------------+
| TLP Type | ConfType0ReadReq | 3DW no Data Header |
+----------+------------------+--------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
|            | (bits) | (bits) |       |
+------------+--------+--------+-------+
| Fmt        | 0      | 3      | 0     |
| Type       | 3      | 5      | 4     |
| T9         | 8      | 1      | 0     |
| TC         | 9      | 3      | 0     |
| T8         | 12     | 1      | 0     |
| Attr_b2    | 13     | 1      | 0     |
| LN         | 14     | 1      | 0     |
| TH         | 15     | 1      | 0     |
| TD         | 16     | 1      | 0     |
| Ep         | 17     | 1      | 0     |
| Attr       | 18     | 2      | 0     |
| AT         | 20     | 2      | 0     |
| Length     | 22     | 10     | 1     |
+------------+--------+--------+-------+
+-----------------+------------------------------------------------+
| TLP:            | 3DW no Data Header                             |
+-----------------+------------------------------------------------+
| Req ID          | 0x20                                           |
| Tag             | 0xA                                            |
| First DW BE     | 0x3 (bytes 0-1)                                |
| Last DW BE      | 0x0 (none)                                     |
| Target BDF      | 05:00.1                                        |
| Bus             | 0x5                                            |
| Device          | 0x0                                            |
| Function        | 0x1                                            |
| Register Offset | 0x000                                          |
| Register Name   | Vendor ID / Device ID                          |
| Ext Reg Nr      | 0x0                                            |
| Reg Nr          | 0x0                                            |
| Operation       | Read Vendor ID / Device ID register at 05:00.1 |
+-----------------+------------------------------------------------+

Three things to read out of that. First, the type: ConfType0ReadReq — a Configuration Read Request, Type 0 (terminating at the addressed device, not forwarded), 3DW header with no data payload. Second, the requester: ID 0x0020, which decodes as Bus 0, Device 4, Function 0 — a Root Complex device, almost certainly a Root Port issuing the read. Third, the Operation line spells out the rest directly: a read of the Vendor ID / Device ID register on Function 1 of the target at 05:00.1. tlp-tool did not always name the register; earlier releases stopped at the raw register number and left the lookup to the reader. A register-name table for standard PCI config space now resolves the common cases — Vendor/Device ID, Command, Status, Class Code, and so on — to a name instead of a hex offset.

That target is the interesting part. Probing Function 1 with a config read of register 0 is exactly what the kernel does during PCI enumeration to discover whether a multi-function device exposes a second function. If Function 1 does not exist, the device responds with an Unsupported Request, and the kernel logs an AER event for it. What looks like a scary uncorrectable PCIe error in dmesg turns out to be a benign enumeration miss. Without the decode, all you would see is the AER summary; with it, you can tell this case apart from a real failure in seconds.

--lspci: device-attributed input. When the AER Header Log is read directly out of device capabilities, --lspci parses the entire lspci -vv block, picks up every non-zero HeaderLog, and annotates each TLP with the BDF of the device it came from:

lspci -vv -s 05:00.1 | rtlp-tool --lspci

+----------+------------------------------------------------------------+--------------------+
| TLP Type | ConfType0ReadReq                                           | 3DW no Data Header |
+----------+------------------------------------------------------------+--------------------+
| Source   | 05:00.1 Non-Volatile memory controller: Phison Electronics |                    |
+----------+------------------------------------------------------------+--------------------+
+------------+--------+--------+-------+
| Field Name | Offset | Length | Value |
|            | (bits) | (bits) |       |
+------------+--------+--------+-------+
| Fmt        | 0      | 3      | 0     |
| Type       | 3      | 5      | 4     |
| T9         | 8      | 1      | 0     |
| TC         | 9      | 3      | 0     |
| T8         | 12     | 1      | 0     |
| Attr_b2    | 13     | 1      | 0     |
| LN         | 14     | 1      | 0     |
| TH         | 15     | 1      | 0     |
| TD         | 16     | 1      | 0     |
| Ep         | 17     | 1      | 0     |
| Attr       | 18     | 2      | 0     |
| AT         | 20     | 2      | 0     |
| Length     | 22     | 10     | 1     |
+------------+--------+--------+-------+
+-----------------+------------------------------------------------+
| TLP:            | 3DW no Data Header                             |
+-----------------+------------------------------------------------+
| Req ID          | 0x20                                           |
| Tag             | 0xA                                            |
| First DW BE     | 0x3 (bytes 0-1)                                |
| Last DW BE      | 0x0 (none)                                     |
| Target BDF      | 05:00.1                                        |
| Bus             | 0x5                                            |
| Device          | 0x0                                            |
| Function        | 0x1                                            |
| Register Offset | 0x000                                          |
| Register Name   | Vendor ID / Device ID                          |
| Ext Reg Nr      | 0x0                                            |
| Reg Nr          | 0x0                                            |
| Operation       | Read Vendor ID / Device ID register at 05:00.1 |
+-----------------+------------------------------------------------+

The leading Source row is the practical advantage over --aer. When several PCIe devices report errors at once — a noisy enclosure, a flaky switch, an enumeration error storm after hotplug — a single lspci -vv capture aggregates everything, and --lspci decodes every Header Log and tells you which BDF produced each one. That distinction matters when you are trying to figure out whether one device is producing all the noise or several devices are independently misbehaving.

The two modes also differ in how they pick up Flit framing — and that is what the next section is about.

The Flit detection problem

On March 24, the day after the patch was applied, the linux-pci discussion surfaced a harder question. Maciej Grochowski — the author of this article and the maintainer of tlp-tool — pointed out an awkward PCIe 6.0 reality: Flit Mode is mandatory at 64.0 GT/s but supported at all link speeds, so a PCIe 6.x link operating below 64.0 GT/s may still be using Flit framing. You cannot simply check the link speed and assume the framing model — and this is not a hypothetical worry. The same message noted that the framing ambiguity is “already a concern among switch and device vendors working through the transition” to PCIe 6.x.

The raw TLP header bytes do not encode which framing produced them — the same bytes decode differently depending on whether they came from classic PCIe or from Flit Mode. The same-bytes-different-meaning ambiguity from the FMT+TYPE comparison above is no longer hypothetical; it is now a real condition in production logs.

Ilpo Järvinen then added the caveat that makes this a real debugging hazard. The Flit Mode Status bit in Link Status 2 is useful only while the link is up — which is not guaranteed in failure scenarios. The kernel tries to compensate by explicitly indicating Flit Mode in its log messages, but the spec creates an asymmetry the kernel cannot fully paper over: AER carries an explicit flag indicating which framing each TLP Log was captured under, while Downstream Port Containment’s (DPC) TLP logging path does not. Järvinen called the DPC omission “botched in the PCIe spec.” To work around it, the kernel saves off Link Status 2 contents and hopes the cached value is still valid when DPC brings the link down — “relatively likely to remain valid,” as he put it, “but fundamentally racy.”

That discussion led directly to tool changes. Grochowski replied that tlp-tool would add auto-detection of the kernel’s (Flit) suffix (introduced in kernel commit 7e077e6707b3, v6.15+) in --aer mode so that mixed Flit and non-Flit TLPs in the same log could be decoded correctly without forcing a global --flit switch. The --lspci mode would read the Flit+ indicator from LnkSta2 per device, while keeping --flit as a manual override for logs that lack markers. The v0.5.1 release notes reflect those changes.

Why this matters now

The good news first: PCIe 1.0 through 5.0 is stable. The header layout that was true in 2003 is still true on hardware shipping today, and tools that decode it work the same way they always have. If your debugging stays within that envelope, nothing has changed and nothing needs to change.

The harder news is what is now sharing the bus next to it. PCIe 6.0’s Flit Mode is the protocol-level shift this article has been about, but the broader pressure on PCIe is the proliferation of accelerators — GPUs, custom AI silicon, FPGAs, smart NICs — putting more devices, more lanes, and more error volume on every system. Users at the edge of that wave will spend more time staring at AER messages, not less; and the maintainers who triage incoming bug reports will receive more raw HeaderLog lines from people who do not yet read TLPs by eye.

tlp-tool is useful on both sides of that exchange. For a user filing a bug report, it turns an opaque four-DWORD line in dmesg into a structured report that names the requester, the target, and the nature of the access — the kind of detail that makes a bug actionable instead of a guess. For a maintainer reading that report, it removes a step: the bit extraction is already done, and the conversation can start at “this is a Type 0 Configuration Read of register 0 on Function 1” instead of “let me decode that for you.” Small per-report, large in aggregate.

The kernel maintainers pointing users at the tool is a quiet acknowledgment that the era of casually reading TLP headers by eye is ending — partly because the protocol made it harder, and partly because the population of people who now need to read them is much larger than it used to be.

But there’s a question we haven’t asked: what did the vendor do?

If there’s a newer firmware version, comparing old and new tells you exactly what changed – and whether the vendor agreed with your diagnosis. In this post, we’ll put two firmware versions side by side, decompile the same function from each, and see how a real fix differs from a four-byte patch. Then we’ll turn to a second technique: using call frequency statistics to automatically identify core utility functions in firmware where every name starts with FUN_.

Two Binaries, One Function

Suppose the vendor ships a firmware update. We now have two flat binaries:

$ ls -la firmware*.bin
-rw-r--r-- 1 user user  8496 Apr 25 firmware.bin       # v2.4.1-rc3 (buggy)
-rw-r--r-- 1 user user  8712 Apr 25 firmware_v2.bin    # v3.0.0 (fixed)

The new version is 216 bytes larger. That is suspicious – a NOP patch does not add bytes. Whatever changed, it was not a one-instruction fix.

The first thing we do with any new firmware version is the same thing we did in Part 2: strings. But this time, we diff the output:

$ diff <(strings firmware.bin | sort) <(strings firmware_v2.bin | sort)

< bridge routes deferred to bridge handler
---
> bridge
> bridge entries registered
> forwarding
> forwarding table updated
> registering route modules
< route: bridge flag set, skipping
> route: registering bridge entry
< status
< uptime
< v2.4.1-rc3
---
> v3.0.0

Several things jump out:

1. Version changed: v2.4.1-rc3 to v3.0.0 – a major version bump, not a point release
2. New strings: “forwarding”, “bridge”, “registering route modules” – these look like module names
3. Changed messages: “route: bridge flag set, skipping” became “route: registering bridge entry” – the bridge path is not skipping routes anymore
4. Removed string: “bridge routes deferred to bridge handler” is gone – the old deferral logic was removed entirely

And two lines that look like removed commands but are not: status and uptime disappear from the v3.0.0 output entirely. Did the vendor remove the status and uptime CLI commands? No – nm firmware_v2.elf still lists both handlers. What disappeared is the standalone string literal. In v2.4.1-rc3, the command name "status" happens to be a suffix of the help text "Show system status", and "uptime" a suffix of "Show system uptime". The linker’s mergeable string sections do suffix merging: if one string is a tail of another, the shorter one gets folded into the longer one’s bytes instead of getting its own copy. In v3.0.0 that merge just happened to trigger for these two commands, so strings – which only reports null-terminated runs – no longer sees "status" or "uptime" as independent strings. It is a linker artifact of that specific build, not a vendor change. Cross-version string diffs will occasionally show noise like this; the fix is to check nm before concluding a command was removed.

The string diff alone tells a story: the route engine was restructured from “check flags, skip bridge” to “register modules, dispatch independently.” Let’s confirm this by decompiling both versions.

Side-by-Side Decompilation

Here’s iterate_active_routes from the old firmware (v2.4.1-rc3). We saw this in Part 3 – the function that contains our priority inversion bug:

/* v2.4.1-rc3 — iterate_active_routes() — flag-checking loop */
void iterate_active_routes(int stack_idx)
{
    int i;
    int programmed = 0;
    int skipped = 0;

    for (i = 0; i < MAX_ROUTES; i++) {
        route_entry_t *entry = &g_route_table_entries[i];

        if (entry->flags == 0)
            continue;

        if (entry->flags & ROUTE_FLAG_BRIDGE) {        /* ← checks bridge FIRST */
            log_msg(MOD_ROUTE, "route: bridge flag set, skipping");
            skipped++;
            continue;                                    /* ← never reaches active check! */
        }

        if (entry->flags & ROUTE_FLAG_ACTIVE) {
            log_msg(MOD_ROUTE, "route: programming forwarding entry");
            programmed++;
        }
    }

    if (programmed == 0)
        log_msg(MOD_ROUTE, "WARNING: no routes programmed for stack");
    if (skipped > 0)
        log_msg(MOD_ROUTE, "bridge routes deferred to bridge handler");
}

The structure is a single loop with inline flag checks. The if/else ordering means BRIDGE (0x04) is checked before ACTIVE (0x01), and the continue means routes with both flags never reach the active check.

Now here’s the same function from v3.0.0:

/* v3.0.0 — iterate_active_routes() — table-driven dispatch */

typedef void (*route_handler_t)(route_entry_t *entry, int *count);

typedef struct {
    const char *name;
    route_handler_t handler;
    uint32_t route_mask;
} route_module_t;

static void handle_forwarding_routes(route_entry_t *entry, int *count) {
    if (entry->flags & ROUTE_FLAG_ACTIVE) {
        log_msg(MOD_ROUTE, "route: programming forwarding entry");
        (*count)++;
    }
}

static void handle_bridge_routes(route_entry_t *entry, int *count) {
    if (entry->flags & ROUTE_FLAG_BRIDGE) {
        log_msg(MOD_ROUTE, "route: registering bridge entry");
        (*count)++;
    }
}

const route_module_t g_route_modules[] = {
    { "forwarding", handle_forwarding_routes, ROUTE_FLAG_ACTIVE },
    { "bridge",     handle_bridge_routes,     ROUTE_FLAG_BRIDGE },
};

void iterate_active_routes(int stack_idx)
{
    int i, m;
    int counts[NUM_ROUTE_MODULES];

    for (m = 0; m < NUM_ROUTE_MODULES; m++)
        counts[m] = 0;

    for (i = 0; i < MAX_ROUTES; i++) {
        route_entry_t *entry = &g_route_table_entries[i];

        if (entry->flags == 0)
            continue;

        /* Dispatch to EVERY registered module — no flag priority */
        for (m = 0; m < NUM_ROUTE_MODULES; m++) {
            g_route_modules[m].handler(entry, &counts[m]);
        }
    }

    if (counts[0] == 0)
        log_msg(MOD_ROUTE, "WARNING: no routes programmed for stack");
    if (counts[0] > 0)
        log_msg(MOD_ROUTE, "forwarding table updated");
    if (counts[1] > 0)
        log_msg(MOD_ROUTE, "bridge entries registered");
}

This is a complete architectural rewrite. The function no longer checks flags inline. Instead:

1. Route handlers are registered in a module table (array of route_module_t)
2. Each handler is a separate function that decides independently whether to process an entry
3. The dispatch loop calls every handler for every non-empty entry
4. A route with both ACTIVE and BRIDGE flags gets processed by both handlers

There’s no if/else between the bridge check and the active check. There’s no continue that skips one path. The two handlers run independently, in separate functions, dispatched from a data table. The priority inversion that existed in v2.4.1-rc3 is structurally impossible in v3.0.0.

What Changed and Why It Matters

Let’s make the comparison precise:

The v2.4.1-rc3 bug was in the ordering of two if statements inside one function. Our NOP patch in Part 3 fixed the symptom by disabling the bridge branch. The vendor fixed the class of bug by making the ordering irrelevant.

This is a common pattern in firmware evolution. The first version implements something quickly with inline checks. A bug is found. The fix isn’t “swap two lines” – it’s “restructure the function so that the bug category can’t exist.” If you see a major version bump and a function that went from 20 lines to 60 lines, this is probably what happened.

For a reverse engineer, cross-version comparison gives you three things:

1. Confirmation: The vendor changed iterate_active_routes, confirming our bug diagnosis from Part 3
2. Understanding: The vendor didn’t just fix the symptom – they rewrote the dispatch architecture
3. Future-proofing: When analyzing the new version, we know to look for the module table instead of inline flag checks

The general technique: when you have two firmware versions and a known bug location, decompile the same function from both, and diff. The changes tell you exactly what the vendor considered broken and how they chose to fix it.

Call Graph Analysis: Naming the Unnamed

Let’s shift to a different problem. In Part 2, we found 35 functions by scanning for MIPS prologues. In Part 4, we saw what Ghidra produces when it decompiles them: port_validate renamed to FUN_80000c84, calling a still-unidentified helper, FUN_800002e4, twice along the way. Meaningful names replaced by addresses.

If you have debug symbols, nm gives you every name instantly. But vendor firmware ships stripped. You’re staring at 35 functions (or 4,426 in the real project) and need to figure out which is which.

Here’s the insight: not all functions are equal. Some functions are called once. Others are called from everywhere. The ones called from everywhere are utilities – and their call frequency alone tells you what they are.

Let’s count how many times each function is called as a JAL target in our firmware:

$ mipsel-linux-gnu-objdump -d firmware.elf \
    | grep -P '\tjal\t' \
    | sed 's/.*jal\s*[0-9a-f]*//' \
    | sort | uniq -c | sort -rn | head -10

    111 FUN_800002e4
     15 FUN_80000790
     15 FUN_80000530
     12 FUN_80000b74
      6 FUN_800002c4
      4 FUN_80000fec
      4 FUN_80000aa8
      2 FUN_80000d00
      2 FUN_80000c84
      2 FUN_80000280

(I’ve replaced the actual symbol names with Ghidra-style FUN_ addresses – in a real stripped binary, this is what you’d see. Note the \tjal\t filter: a plain grep 'jal' also matches jalr, the indirect-call-through-register instruction, and pollutes the count with a bogus entry. That distinction matters beyond cleanliness – v3.0.0’s table-driven dispatch calls each route handler through a function pointer, which compiles to jalr, not jal. A JAL-only call count is structurally blind to every call the new dispatch table makes; that dispatch traffic simply will not show up in this kind of analysis, no matter how the grep is tuned.)

The distribution is wildly uneven. FUN_800002e4 is called 111 times – more than all other functions combined. That’s not business logic. That’s infrastructure. Let’s figure out what it is.

From Counts to Names

FUN_800002e4 (111 calls): Called from nearly every function in the binary. It takes a single pointer argument. In the string cross-reference analysis from Part 2, every call to this function passes a string address. A function called 111 times that takes a string? That’s uart_puts – the serial output function.

FUN_80000790 (15 calls) and FUN_80000530 (15 calls): Both called frequently, both appear in the same functions as uart_puts. One takes a 32-bit integer and outputs hex characters (look for the 0123456789ABCDEF string reference). The other outputs decimal digits. These are uart_puthex and uart_putdec.

FUN_80000b74 (12 calls): Called by every subsystem initialization function. Takes two arguments: an integer (always a small constant like 1, 2, 3, 4) and a string pointer. The strings are log messages (“system starting”, “initializing ports”). This is log_msg – the logging function with a module ID and message string.

FUN_80000aa8 (4 calls): Takes three arguments: an integer, a string that ends in .c, and another integer. The strings are source filenames: route.c, flash.c, diag.c. A function that takes (code, "file.c", line_number) is an assert handler. This is fw_assert.

Five functions identified from call counts alone:

Call Frequency Analysis — Top Functions
========================================

Address       Calls  Identification      Evidence
──────────────────────────────────────────────────────────
FUN_800002e4    111  uart_puts           Every function, takes string arg
FUN_80000790     15  uart_putdec         Takes int, outputs decimal digits
FUN_80000530     15  uart_puthex         Takes int, refs "0123456789ABCDEF"
FUN_80000b74     12  log_msg             Takes (module_id, string), called by init fns
FUN_80000aa8      4  fw_assert           Takes (code, "file.c", line), halts on fatal
FUN_800002c4      6  uart_putchar        Takes single char, called by uart_puts

In a 35-function firmware, we just named 6 functions (17%) from call frequency and string cross-references. Those 6 functions account for 163 of 184 total calls (89%). Name the infrastructure and you understand most of the binary’s call graph.

The Heuristics at Scale

In our sample firmware with 35 functions, this is a nice exercise. In real vendor firmware with thousands of functions, it’s essential. The real project had 4,426 functions. Call graph analysis identified:

• fw_assert at the top with 497 inbound calls – unmistakable as the assert handler
• log_msg with 470 callers – the logging function with severity and module ID
• irq_disable/irq_enable with 345/277 callers – interrupt lock pairs (always appear together)
• mem_alloc with 330 callers – the firmware’s memory allocator
• spinlock_acquire/release with 240/204 callers – synchronization primitives

The heuristics that work at scale:

1. Highest call count + string argument → output function (print, log, puts)
2. Called with "file.c" + integer → assert or error handler
3. Always called in pairs (enable/disable, acquire/release) → lock/unlock primitives
4. Called at the start of many functions → initialization or lock acquisition
5. Called at the end of many functions → cleanup or lock release

Once you’ve named the top 10-15 utility functions, Ghidra’s decompiled output transforms. Instead of:

FUN_c0060934(0x8000, "psc_port.c", 342);
FUN_c0098e38(1, 0x53, 3, 0, 0, 0, 0);
FUN_c01714e0();

You read:

fw_assert(FATAL, "route.c", 342);
log_msg(ERROR, MOD_ROUTE, MSG_BRIDGE_FLAG);
irq_disable();

The code hasn’t changed. Your understanding of it has.

The Full Picture

Over five posts, we’ve built a complete firmware reverse engineering toolkit:

1. Part 1: Understood the flat binary format – no headers, no symbols, raw instructions at byte zero
2. Part 2: Found strings, traced cross-references, discovered hidden CLI commands
3. Part 3: Decompiled the route engine, found the priority inversion bug, patched four bytes
4. Part 4: Tried to recompile, hit 2,896 unresolved symbols, learned why firmware linking is hard
5. Part 5: Compared two firmware versions to understand architectural fixes, used call graphs to name functions automatically

Every technique came from a real project. Every pattern is something we found in production firmware. The domain was changed (network gateway instead of the actual device), but the methods are universal. Cross-version comparison works on any firmware with multiple releases. Call graph analysis works on any binary with function calls.

The firmware still doesn’t want you to read it. But now you have five ways to read it anyway.

What’s Next: A New Series

This series used an 8 KB sample firmware to keep things approachable. But real firmware is bigger, more complex, and full of patterns we haven’t touched yet – cryptographic algorithm identification from lookup tables, struct recovery from raw memory offsets, hardware peripheral mapping from MMIO access patterns, and the kind of runtime architecture (RTOS schedulers, pool allocators, rule engines) that makes embedded devices work.

For the next series – “Crypto Tables, Struct Ghosts, and Other Secrets Hidden in Firmware” – we’ve built a 38 KB firmware with AES-128 encryption, SHA-256 hashing, a cooperative RTOS, a three-mode pool allocator, and seven MMIO hardware blocks. We’ll also finally introduce radare2 and compare it head-to-head against Ghidra and Capstone.

If you’ve followed along this far, you have the foundation. The next series goes deeper.

References

• Ghidra – Decompiler used for cross-version comparison
• BinDiff – Google/Zynamics tool for binary diffing (automates what we did manually)
• Diaphora – Open-source Ghidra/IDA binary diffing plugin
• Capstone Engine – Disassembly framework used for call graph extraction
• MIPS Instruction Reference – JAL encoding for call target extraction
• Part 1: Headers, Symbols, and other things you won’t find
• Part 2: Opcodes, Prologues, and other hidden patterns
• Part 3: Decompilers, Annotations, and other ways to read the unreadable
• Part 4: Symbols, Scripts, and other linking nightmares

You cannot do that with a hex editor. You would need to recompile from source.

We have Ghidra’s decompiled C. We have the function names, the string references, the module structure. We have gcc and a linker script. How hard can it be?

Very.

The Illusion of Having Source Code

Ghidra produces C code that reads like source but is not source. Here is what Ghidra generates for port_validate:

undefined4 FUN_80000c84(int param_1)
{
    undefined4 uVar1;

    if ((param_1 < 0) || (7 < param_1)) {
        FUN_800002e4(s_ERROR_invalid_port_number_80001bb0);
        FUN_800002e4(&DAT_80001a88);
        uVar1 = 0xffffffff;
    }
    else if ((*(uint *)(param_1 * 0x20 + 0x80040200) & 1) == 0) {
        FUN_800002e4(s_ERROR_port_is_not_enabled_80001b94);
        FUN_800002e4(&DAT_80001a88);
        uVar1 = 0xffffffff;
    }
    else {
        uVar1 = 0;
    }
    return uVar1;
}

The structure is correct. The logic is correct. But look at what is missing:

• Types are wrong: undefined4 instead of int. No port_config_t struct.
• Names are gone: FUN_80000c84 instead of port_validate. param_1 instead of port_num.
• Data access is raw: *(uint *)(param_1 * 0x20 + 0x80040200) instead of g_ports[port_num].flags.
• Symbols are addresses: s_ERROR_invalid_port_number_80001bb0 and DAT_80001a88 are Ghidra-generated names that encode the binary address.
• No headers: No #include, no type definitions, no struct layouts.

You can fix all of this manually. Rename FUN_80000c84 to port_validate, change param_1 to port_num, define the port_config_t struct. But now multiply that by every function in the firmware.

In the real project, Ghidra decompiled 4,426 functions across 1.5 MB of code. We organized them into 191 C source files matching the original module structure. Every single file compiled individually with gcc -c. It looked like we were close to a complete build.

Then we tried to link.

The Linking Wall

$ mipsel-linux-gnu-ld -T linker.ld -o firmware.elf *.o
startup.o: undefined reference to `firmware_main'
main.o: undefined reference to `s_system_starting_80001ee0'
main.o: undefined reference to `s_initializing_ports_80001cf0'
route.o: undefined reference to `DAT_80040008'
cli.o: undefined reference to `s_Available_commands_800018b4'
port.o: undefined reference to `_DAT_B8000008'
... 2,896 more ...

2,896 unresolved symbols. Every single one needs to be resolved before the linker will produce a binary.

Why? When you compile with gcc -c, each source file compiles in isolation. The compiler trusts your extern declarations: “this function exists, this variable exists, the linker will find them.” Compilation succeeds because the compiler does not check. Linking fails because the linker does.

In userspace, the linker resolves symbols against libraries (libc.so, libm.so, etc.) and other object files. In firmware, there are no libraries. Every symbol must come from your own object files, your linker script, or it does not exist.

If you are following along with the sample firmware, you will not hit this wall yourself – it is a single translation unit that links cleanly from its own source. This wall belongs to the real project: 191 separately compiled object files, each trusting the others to supply the symbols it referenced. The sample reproduces the shape of the problem, not its scale.

The Symbol Taxonomy

In the real project, the 2,896 unresolved symbols broke into six categories, each needing a different resolution strategy:

Unresolved Symbol Breakdown
============================

Category           Count   Example                          Resolution
─────────────────────────────────────────────────────────────────────────
String literals    1,172   s_system_starting_80001ee0       Extract from binary
Const data           993   DAT_C0012345                     Extract from binary
MMIO registers       444   _DAT_B8000008                    PROVIDE() in linker script
RAM globals          107   DAT_80240100                     BSS/data section placement
Code labels           91   LAB_C0012345                     Switch table extraction
Compiler runtime      15   func_0xC0170000                  Extract from binary
Other                 74   builtin_strncpy, _gp_disp        Stubs + linker magic
─────────────────────────────────────────────────────────────────────────
Total              2,896

String Literals: The Biggest Category

1,172 symbols are strings. Ghidra names them after their address: s_system_starting_80001ee0 means “the string starting at address 0x80001EE0.” The string data lives in the original binary’s .rodata section, which objcopy happily included in the flat binary.

The fix: extract the strings from the original binary and assemble them into a linkable object file:

# Extract strings from firmware.bin at known addresses
# Generate assembly that the linker can resolve

with open('firmware.bin', 'rb') as f:
    binary = f.read()

print('.section .rodata')
for name, addr in string_symbols:
    offset = addr - BASE_ADDR
    string = read_cstring(binary, offset)
    print(f'.global {name}')
    print(f'{name}: .asciz "{escape(string)}"')

Assemble this into strings.o, add it to the link – 1,172 symbols resolved. This works because the strings are data, not code. We are literally pulling bytes out of the original binary and giving them labels.

MMIO Registers: Addresses Without Content

444 symbols are hardware register addresses – UART, GPIO, timer, system control. They look like _DAT_B8000008 (UART status register at 0xB8000008). These are not in the binary at all – they are memory-mapped I/O addresses that the CPU accesses at runtime.

The linker does not need content for these. It just needs to know the address. That is what PROVIDE() does:

/* linker.ld -- MMIO register definitions */
SECTIONS
{
    /* ... code and data sections ... */
}

/* Hardware register addresses -- no content, just locations */
PROVIDE(_DAT_B8000000 = 0xB8000000);   /* UART TX */
PROVIDE(_DAT_B8000004 = 0xB8000004);   /* UART RX */
PROVIDE(_DAT_B8000008 = 0xB8000008);   /* UART Status */
PROVIDE(_DAT_B800000C = 0xB800000C);   /* UART Control */
PROVIDE(_DAT_B8010000 = 0xB8010000);   /* GPIO Data */
/* ... 439 more ... */

PROVIDE() tells the linker: “if nobody else defines this symbol, it lives at this address.” The generated code will use LUI/ADDIU to load the address and then read/write the hardware register. No content needed in the binary – MMIO registers exist in the hardware, not in flash.

In the real project, 444 MMIO registers spanned five address blocks (0xB4, 0xB6, 0xB9, 0xBD, 0xBF), covering the switch core, packet processor, PHY interface, DMA engine, and management controller. Each one got a PROVIDE() line.

The Global Pointer Problem

On MIPS, frequently accessed global variables use GP-relative addressing. Instead of a two-instruction LUI/ADDIU pair, the compiler emits a single instruction:

lw  v0, -32000(gp)    # Load global at GP - 32000

This saves one instruction per access – significant in a firmware with thousands of global accesses. But it only works if $gp (the global pointer register) is set to the right value, and all code agrees on what that value is.

The problem: the decompiled code contains hardcoded GP offsets baked in by the original compiler. If the recompiled binary places globals at different addresses – even slightly – every GP-relative access breaks.

/* Ghidra decompiled code contains things like: */
*(int *)(gp + -0x7c50) = 1;    /* original GP offset, hardcoded */

Three ways to fix it. Match GP exactly: set it to the original compiler’s value and reproduce the full original global layout, which requires perfect section placement. Disable GP-relative addressing entirely with -G0 -mno-gpopt and rewrite all accesses as full LUI/ADDIU pairs – the optimization is gone, but at least it links. Or declare all globals properly and let a fresh compiler recalculate offsets from scratch, which requires getting every global’s type and size right first.

Our sample firmware uses option two (-G0 -mno-gpopt in the Makefile) – every global uses the full LUI/ADDIU pair, so there is no GP dependency. Real vendor firmware is rarely that considerate.

The Custom Linker Script

In userspace, the linker uses a default script that puts .text at a dynamic address, .data after it, and .bss at the end. The kernel’s ELF loader handles the rest. In firmware, you need total control:

/* linker.ld -- Firmware memory layout */
MEMORY
{
    ROM (rx)  : ORIGIN = 0x80000000, LENGTH = 256K
    RAM (rw)  : ORIGIN = 0x80040000, LENGTH = 64K
}

SECTIONS
{
    /* Exception vectors MUST be at 0x80000000 */
    .vectors 0x80000000 : {
        *(.vectors)
    } > ROM

    /* Code follows immediately */
    .text : ALIGN(4) {
        *(.text)
        *(.text.*)
    } > ROM

    /* Read-only data (strings, tables) */
    .rodata : ALIGN(16) {
        *(.rodata)
        *(.rodata.*)
    } > ROM

    _etext = .;

    /* Globals in RAM (zeroed at boot by startup code) */
    .bss (NOLOAD) : ALIGN(4) {
        __bss_start = .;
        *(.bss)
        *(COMMON)
        __bss_end = .;
    } > RAM

    /* Stack at top of RAM */
    _stack_top = ORIGIN(RAM) + LENGTH(RAM);
}

It is clean because we built it from scratch. The linker.ld bundled with the sample firmware is a fuller version of this – it adds explicit .rodata.cli_* ordering (so the hidden CLI commands land after the public ones), a real .data section loaded from ROM into RAM, and a _gp assignment – all elided here for clarity. In a real reverse engineering project, you would need to derive the whole thing from the binary’s memory layout – figuring out where each section starts and ends by analyzing address patterns in the code.

The MEMORY block defines two regions: ROM (where the flat binary lives in flash) and RAM (where globals and stack live at runtime). The exception vectors must be at 0x80000000 because that is where the MIPS CPU looks for them at reset. There is no flexibility here – get it wrong by one byte and the device crashes on boot.

Compare this with the 2018 ELF post, where we relied on the kernel’s ELF loader to parse program headers and map segments. In firmware, the linker script is the loader. Everything the kernel did for ELF – segment mapping, BSS zeroing, stack setup – you do yourself in a 20-line startup assembly file and a linker script.

The Compilation Gap

Here is what full recompilation actually requires:

Full Firmware Recompilation Checklist
=====================================

1. Decompile all functions              ✓  (Ghidra automated this)
2. Fix Ghidra output artifacts          ✗  (undefined4, unaff_gp, raw casts)
3. Reconstruct struct definitions       ✗  (Ghidra doesn't recover struct layouts)
4. Reconstruct enum/constant values     ✗  (flag values inferred from usage)
5. Extract strings from binary          ~  (automated but needs validation)
6. Extract const data from binary       ~  (automated, 993 symbols)
7. Map MMIO registers to PROVIDE()      ~  (automated, 444 addresses)
8. Resolve GP-relative globals          ✗  (requires matching GP layout)
9. Stub/extract compiler runtime        ~  (15 functions from vendor compiler)
10. Handle switch tables (LAB_*)        ✗  (91 computed gotos need manual work)
11. Write startup code                  ✗  (exception vectors, BSS clear, GP setup)
12. Write linker script                 ~  (derive from binary memory map)
13. Integration test                    ✗  (need hardware or accurate emulator)

Estimated effort: 100-170 hours for 1.5 MB of firmware

Step 1 is essentially free – Ghidra decompiles thousands of functions in minutes. Steps 5-7 can be automated with scripts. Steps 2-4, 8, and 10-13 require manual analysis and domain knowledge – which is where the time actually goes.

A note on using an LLM for this: tools like Claude Opus 4.8 can compress the annotation work substantially – renaming functions from Ghidra output, reconstructing struct definitions, generating PROVIDE() lists from address patterns. In practice, the 100-170 hour estimate drops by roughly a factor of five, to something in the 20-30 hour range for comparable firmware. But it does not eliminate the need for supervision. The model will confidently reconstruct a struct layout that is subtly wrong, or generate a section offset that is plausible but off by four bytes. Every output still needs a human to verify against the binary. The hours go down; the attention required does not.

The decompilation-to-compilation gap: decompiled C is a description of the binary’s behavior, not a specification of how to build it. That is like having a photograph of a building – you can see every room, but you do not have the blueprints, the materials list, or the building code compliance.

Structural Verification: Compiles, Doesn’t Link

In the real project, we stopped before linking. The goal was structural verification: every source file compiles, the function signatures are correct, the call graph is preserved – but no linked binary.

# Compile every file independently
for f in src/*.c; do
    mipsel-linux-gnu-gcc-14 -mips32r2 -EL -O1 -ffreestanding \
        -nostdlib -mno-abicalls -fno-pic -G0 -mno-gpopt \
        -c -o "${f%.c}.o" "$f"
done

# Result: 191/191 compile successfully
# But: 2,896 symbols remain unresolved at link time

This is not failure – it is a deliberate stopping point. Every file compiling confirms the decompilation is syntactically correct, function signatures match across the 191-file tree, and no conflicting type definitions exist. It does not prove the binary will boot, but for reading the code and finding bugs, it is more than enough.

For the bug fix we did in Part 3, this was exactly that. We understood the code, found the bug, and patched it in the binary. Full recompilation would have been the “right” way to fix it, but binary patching achieved the same result in an afternoon instead of several months.

When Full Recompilation Makes Sense

Binary patching is a scalpel. Recompilation is a complete rebuild. You choose based on the scope of changes:

The real project chose binary patching for the immediate fix and structural verification as a long-term investment. If the vendor never provides an official fix, having 191 compilable C files is a foundation for future work – even if linking them into a single binary is still a 100+ hour project.

It Linked. It Still Didn’t Boot.

Producing a correctly linked binary is not the finish line.

In a separate project – a 4+ MB firmware on a different embedded device – we reached the point this sample firmware’s analysis stops short of: every symbol resolved, the linker produced a binary, the image packed cleanly with valid CRC checksums. We flashed it. The device entered the primary bootloader, the secondary bootloader ran – and then silence. No UART output. No register activity. The firmware died without a word.

The first problem is the bootloader. As we covered in Part 1, the firmware image has separate partitions. The primary bootloader lives at a fixed flash address, is often write-protected, and cannot be replaced – it is the first code that runs at reset. That bootloader was compiled by the vendor with their exact toolchain, and it has specific expectations about the handoff to main firmware: entry point address, initial stack state, BSS layout, exception vector locations. A recompiled binary that differs even slightly from those expectations will fail at the handoff, silently.

The second problem is the compiler. Vendor firmware is frequently built with a proprietary SDK – a vendor-patched GCC, IAR, Green Hills MULTI, or the MIPS Technologies compiler itself. These make choices that mipsel-linux-gnu-gcc-14 does not: interrupt calling conventions, stack alignment under exception handlers, runtime initialization order, startup register state. None of these surface as linker errors. They show up as a device that boots partway and then goes quiet.

Weeks went into educated guesses – adjusting the entry point, rewriting the startup assembly, matching section alignment, trying to reconstruct what the original runtime initialization looked like. The firmware died after the secondary bootloader every single time. At some point the cost of continued guessing exceeded the value of the exercise.

Binary patching was the right answer all along. It was just less satisfying to admit.

The Full Circle

We started this series with an ELF binary on an x86-64 Linux system – sections, symbols, relocations, a well-documented format, a kernel loader that handles everything. Four posts later, we have been to the other end of the spectrum: a flat binary on a MIPS32 embedded device – no headers, no symbols, no loader, a custom container format with CRC checksums.

The journey so far:

1. Part 1: Understood what is missing in flat firmware vs ELF
2. Part 2: Found strings, traced functions, discovered hidden CLI commands
3. Part 3: Decompiled the route engine, found a priority inversion bug, patched it
4. Part 4: Tried to recompile, hit 2,896 unresolved symbols, got 191 files to compile – then learned that a linked binary still might not boot

Every technique came from a real project. Every pattern is something we found in production firmware. The domain was changed (network gateway instead of the actual device), but the methods are universal. If you can find strings in a binary, trace function calls, decompile with Ghidra, and write a linker script, you can reverse engineer any firmware on any architecture.

The firmware doesn’t want you to read it. Read it anyway.

But there is one more question we have not asked: what happens when the vendor ships an update? In Part 5: “Versions, Callgraphs, and other ways to compare what changed”, we will put two firmware versions side by side, compare how the vendor chose to fix the same bug we patched, and use call graph statistics to automatically name functions in stripped binaries.

References

• GNU ld Linker Scripts – MEMORY, SECTIONS, PROVIDE(), and everything else in linker script language
• Ghidra – Decompiler that produces the “almost-source” we tried to compile
• MIPS GP-Relative Addressing – Why $gp makes firmware linking painful
• ELF’s Linker’s and other magical creatures – Where this all started, eight years ago
• Part 1: Headers, Symbols, and other things you won’t find – Building the firmware and the image packager
• Part 2: Opcodes, Prologues, and other hidden patterns – Finding strings, functions, and the CLI table
• Part 3: Decompilers, Annotations, and other ways to read the unreadable – Decompiling the route engine and patching the binary

WARNING: no routes programmed for stack

Part 2 could show that the string exists and that it belongs to the route subsystem. It could not prove when that path runs, or whether it is the exact runtime symptom. For that we need structure. In this post, we will decompile the firmware, trace the execution path through the route engine, find why two boot routes silently miss the forwarding table, patch the binary, and recalculate the firmware image’s CRC. Four bytes is enough to restore forwarding in this sample.

From Disassembly to C: What Decompilers Give You

Reading MIPS assembly works for short functions, but once you are staring at a 60-instruction function with nested loops and multiple branches, you want C. Decompilers like Ghidra take machine code and produce readable (if imperfect) C.

Here is what Ghidra produces for our iterate_active_routes function. This is the raw, unannotated output – no cleanup, exactly what the tool generates:

void FUN_80000efc(int param_1)
{
    int *piVar1;
    int iVar2;
    int iVar3;
    int iVar4;

    iVar3 = 0;
    iVar4 = 0;
    piVar1 = (int *)0x80040008;
    do {
        iVar2 = *piVar1;
        if (iVar2 != 0) {
            if ((iVar2 & 4) != 0) {
                FUN_80000b74(4, s_route_bridge_flag_set_skipping_80001f70);
                iVar3 = iVar3 + 1;
            }
            else if ((iVar2 & 1) != 0) {
                FUN_80000b74(4, s_route_programming_forwarding_entry_80001f94);
                iVar4 = iVar4 + 1;
            }
        }
        piVar1 = piVar1 + 4;
    } while (piVar1 != (int *)0x80040208);
    if (iVar4 == 0) {
        FUN_80000b74(4, s_WARNING_no_routes_programmed_80001fb8);
    }
    if (0 < iVar3) {
        FUN_80000b74(4, s_bridge_routes_deferred_80001fe0);
    }
    return;
}

Ghidra got the structure right. You can see the loop, the flag checks, the log messages. But everything has auto-generated names: FUN_80000efc, param_1, piVar1, iVar2. The string references are the one lifeline – they tell you what each branch does, even when the variable names are meaningless.

This is what decompilation gives you: structure without semantics. You get the control flow and data flow right, but understanding what it means requires annotation.

Annotation: From piVar1 to entry->flags

Let us clean up Ghidra’s output. Using the strings, the function name we discovered in Part 2, and our knowledge of the route table structure, we can annotate everything:

/* iterate_active_routes() at 0x80000EFC
 * Walks the route table and programs active routes into the
 * hardware forwarding table. Bridge-flagged routes are deferred.
 */
void iterate_active_routes(int stack_idx)
{
    route_entry_t *entry;
    int flags;
    int skipped = 0;       /* bridge-deferred routes */
    int programmed = 0;    /* routes sent to forwarding table */

    entry = &g_route_table_entries[0];   /* Ghidra's raw piVar1 points at entry[0].flags */
    do {
        flags = entry->flags;
        if (flags != 0) {                    /* skip empty slots */
            if ((flags & 0x04) != 0) {       /* ROUTE_FLAG_BRIDGE = 0x04 */
                log_msg(MOD_ROUTE, "route: bridge flag set, skipping");
                skipped++;
            }
            else if ((flags & 0x01) != 0) {  /* ROUTE_FLAG_ACTIVE = 0x01 */
                log_msg(MOD_ROUTE, "route: programming forwarding entry");
                programmed++;
            }
        }
        entry++;
    } while (entry != &g_route_table_entries[32]);

    if (programmed == 0) {
        log_msg(MOD_ROUTE, "WARNING: no routes programmed for stack");
    }
    if (skipped > 0) {
        log_msg(MOD_ROUTE, "bridge routes deferred to bridge handler");
    }
}

Now it reads like source code. And now the bug jumps out.

The Bug: Priority Inversion

Look at the two if-statements in the loop:

if ((flags & 0x04) != 0) {       /* check BRIDGE flag first */
    skipped++;
    continue;
}
else if ((flags & 0x01) != 0) {  /* check ACTIVE flag second */
    programmed++;
}

The bridge flag (0x04) is checked before the active flag (0x01). This creates an if/else where a route that has both flags set hits the bridge check first and gets diverted – the active check never runs.

Let us trace what happens during boot. The firmware adds four routes:

route_add(0x0A01, 0x01, ACTIVE | STATIC);           /* 10.1.x.x → port 0 */
route_add(0x0A02, 0x02, ACTIVE | STATIC);           /* 10.2.x.x → port 1 */
route_add(0xC0A8, 0x04, ACTIVE | BRIDGE);           /* 192.168.x → port 2 */
route_add(0xAC10, 0x08, ACTIVE | STATIC | BRIDGE);  /* 172.16.x → port 3 */

Routes 1 and 2 have flags 0x03 (ACTIVE + STATIC). The BRIDGE check fails (0x03 & 0x04 = 0), so they fall through to the ACTIVE check (0x03 & 0x01 = 1) and get programmed. Good.

Routes 3 and 4 have flags 0x05 and 0x07 respectively – both include BRIDGE (0x04). The bridge check succeeds first: 0x05 & 0x04 = 4, which is non-zero, so the bridge path wins. The route’s ACTIVE bit exists, but the programming path never gets to use it.

The result? Only 2 of 4 routes get programmed into the forwarding table. Traffic to 192.168.x.x and 172.16.x.x silently blackholes. What the final boot scan actually produces:

[MOD:4] route: programming forwarding entry    ← route 1 (10.1.x)
[MOD:4] route: programming forwarding entry    ← route 2 (10.2.x)
[MOD:4] route: bridge flag set, skipping       ← route 3 (192.168.x) SKIPPED!
[MOD:4] route: bridge flag set, skipping       ← route 4 (172.16.x) SKIPPED!
[MOD:4] bridge routes deferred to bridge handler

iterate_active_routes is called by route_add after every insertion, so the scan runs four times during boot – not once at the end. Calls 1 and 2 only see ACTIVE+STATIC entries and program them cleanly. Call 3 hits route 3’s BRIDGE flag for the first time. Call 4 does the same for route 4.

Routes 1 and 2 get re-programmed on every rescan (idempotent), so programmed is never zero – the WARNING: no routes programmed guard never fires for this route set. It is a safety net for a completely empty forwarding table, not for a partially-broken one. That is the correction decompilation gives us over the string-only view from Part 2: the warning string was a clue to inspect this function, while the actual bug is the quieter “bridge flag set, skipping” path.

This is a priority inversion – the bridge flag check has higher priority than the active flag check, but the developer intended them to be independent. A route should be programmed if it is ACTIVE, regardless of whether it is also BRIDGE. The bridge handler should get a copy of bridge-flagged routes, not steal them from the forwarding table entirely.

In the real vendor firmware project, we found exactly this pattern: a flag check at instruction 0xC012936C tested a P2P flag before the active-capability flag, causing an entire subsystem to silently fail whenever a specific hardware configuration was present. The symptom was identical – “no rules programmed” – and it took weeks of trace analysis to find those four bytes.

The Disassembly: Four Instructions

Let us look at the exact machine code. The bug lives in four instructions at offsets 0x0F6C-0x0F78:

80000f64:   8e020000    lw      v0, 0(s0)       # v0 = entry->flags
80000f68:   1040fffb    beqz    v0, next_entry  # if flags == 0, skip

80000f6c:   30430004    andi    v1, v0, 0x4     # v1 = flags & BRIDGE  ← THE BUG
80000f70:   1460fff5    bnez    v1, bridge_skip # if BRIDGE set → branch to bridge_skip
80000f74:   30420001    andi    v0, v0, 0x1     # delay slot: executes even when branch is taken
80000f78:   1040fff7    beqz    v0, next_entry  # ← never reached when branch taken

The instruction at 0x80000F70 is the problem: bnez v1, bridge_skip. When the BRIDGE bit is set this branch fires – and because 0x80000F74 is the branch delay slot, the andi there executes either way. What never executes is 0x80000F78: the branch that tests the ACTIVE result before the programming path. Routes with both BRIDGE and ACTIVE flags are treated as bridge-only, not as active routes that also participate in forwarding.

Binary Patching: The Four-Byte Fix

The simplest forwarding fix: NOP out the branch at 0x0F70. Replace bnez v1, bridge_skip with nop, so the bridge flag check still runs but never diverts execution. All non-empty routes reach the ACTIVE check.

import struct

with open('firmware.bin', 'rb') as f:
    data = bytearray(f.read())

# The bug: bnez instruction at offset 0x0F70
bug_offset = 0x0F70
original = struct.unpack_from('<I', data, bug_offset)[0]
print(f"Original: 0x{original:08X}  (bnez v1, bridge_skip)")

# The fix: NOP (0x00000000)
struct.pack_into('<I', data, bug_offset, 0x00000000)
print(f"Patched:  0x00000000  (nop)")

with open('firmware_patched.bin', 'wb') as f:
    f.write(data)

Four bytes. That is the entire forwarding patch. The bridge flag still gets tested by the andi instruction, but the result is never acted on – all non-empty routes proceed to the ACTIVE check.

After patching:
80000f6c:   30430004    andi    v1, v0, 0x4     # v1 = flags & BRIDGE (still runs)
80000f70:   00000000    nop                      # ← PATCHED (was: bnez v1, bridge_skip)
80000f74:   30420001    andi    v0, v0, 0x1     # former delay slot; already executed before patch
80000f78:   1040fff7    beqz    v0, next_entry  # ← now always reached

Now routes with ACTIVE | BRIDGE (flags 0x05 or 0x07) hit the NOP, the former delay-slot andi runs as before, but execution falls through to 0x80000F78 – the beqz that was previously unreachable. The ACTIVE flag is set, the branch is not taken, and the route gets programmed. The forwarding table gets all four boot routes instead of two.

CRC Recalculation: You Break It, You Fix It

Remember the firmware image from Part 1? Each partition has a CRC-32 checksum, and the image header has a global checksum. Patching even a single byte invalidates both:

Original firmware.bin CRC-32:  0x9F25E0DC
Patched firmware.bin CRC-32:   0x156FE139   ← completely different!

If we packed the patched binary into a firmware image without updating the CRC, the device’s bootloader would reject it. We need to repack:

$ python3 fwpack.py firmware_patched.bin firmware_patched.img

Packed firmware image: firmware_patched.img
  Total size:    9152 bytes
  Image CRC-32:  0xC5E88C64
  Partitions:    3

  [bootloader]
    Offset:  0x0120
    Size:    32 bytes
    CRC-32:  0x4F7A6ACA
  [main_fw]
    Offset:  0x0160
    Size:    8496 bytes
    CRC-32:  0x156FE139    ← updated automatically
  [config]
    Offset:  0x22B0
    Size:    256 bytes
    CRC-32:  0x2683AC5B

The packager recalculates all checksums from the payload data. In practice, reverse engineering the vendor’s CRC algorithm (or identifying it as standard CRC-32) is a necessary step before you can deploy any binary patch. Some vendors use non-standard polynomials, or CRC the payload with a salt, or layer RSA signatures on top. Each additional layer makes patching harder – but the fundamental workflow is the same: patch the code, recalculate the integrity checks, repack.

What We Changed (And What We Didn’t)

Let us be precise about what this patch does:

The ACTIVE + BRIDGE path is fixed. The ACTIVE-only path is unchanged. But the BRIDGE-only path changes: before the patch, a BRIDGE-only route incremented skipped and triggered the “bridge routes deferred” log. After the NOP, the branch that drove that path is gone – the route falls through to the ACTIVE check, which fails (flags & 0x01 == 0), and exits via next_entry with no counter and no log.

In this firmware, that is fine. The “bridge handler” is a log message; there is no real deferred queue behind it. If there were, NOPping the branch would trade one bug for another. In a real project you would replace the branch with proper independent checks – test BRIDGE and ACTIVE separately rather than as a chain. The NOP is the minimal patch that fixes the symptom we can measure; the right fix is a code restructure.

The Bigger Picture

We went from suspicious route strings to a root cause and a binary patch in one post. Let us trace back the full path:

1. Part 1: Built the flat binary – 8,496 bytes, no symbols, no sections
2. Part 2: strings found suspicious route messages; xref scanning pointed us at the route subsystem
3. Part 3: Decompilation revealed the flag priority inversion in iterate_active_routes at 0x80000EFC
4. Binary patch: NOP at offset 0x0F70 (4 bytes)
5. CRC update: Repack firmware image with new checksums

That is the real-world firmware RE workflow. Find a clue in the strings. Trace it to a function. Decompile the function. Understand the logic. Patch the binary. Update the checksums.

In the real project, this same workflow – from a warning message in a UART log to a binary patch at a specific address – took weeks. Most of that time was spent building the tools and understanding the firmware’s architecture. The actual bug was four bytes, just like this one.

But here is the thing: we patched four bytes. What if the fix was more complex – restructuring a function, adding new code, changing data structures? You cannot do that with a hex editor. You would need to recompile from source.

And that raises a question: we have Ghidra’s decompiled C for every function. We have the source filenames from the assert strings. Can’t we just… compile it?

In Part 4: “Symbols, Scripts, and other linking nightmares”, we will try. And we will discover why that simple question has a very complicated answer.

References

• Ghidra – NSA’s Software Reverse Engineering Framework (decompiler used in this post)
• MIPS Instruction Reference – ANDI, BNEZ, NOP instruction encoding
• CRC-32 – The checksum algorithm used in firmware image integrity verification
• Part 1: Headers, Symbols, and other things you won’t find – Building the firmware and the image packager
• Part 2: Opcodes, Prologues, and other hidden patterns – Finding strings, functions, and the CLI table

The temptation is to fire up a disassembler and start reading opcodes. Resist it. The first useful thing you can do with an unknown firmware binary isn’t disassembly – it’s reading the text that the developers left behind.

Code and binary: the firmware.bin analyzed below, the build sources, and every script in this post are bundled in sample_firmware.tar.gz. Grab it and follow along on your own copy:

curl -L -O https://res.cloudinary.com/gotocco/raw/upload/v1777182332/sample_firmware.tar.gz
tar xf sample_firmware.tar.gz
cd sample_firmware

Strings: The First Foothold

Every firmware binary contains embedded strings – log messages, error texts, command prompts, version information. The developers who wrote the firmware needed to print things to the serial console, log diagnostic messages, and display help text. All of that text lives in the .rodata section, which survives the objcopy to flat binary untouched.

$ strings firmware.bin

111 strings come pouring out. Let’s categorize what we find:

Boot sequence messages:

system starting
initializing ports
initializing route table
WARNING: no routes programmed for stack
bridge routes deferred to bridge handler
system ready

These tell us the firmware has a boot sequence: it starts up, initializes a port subsystem, sets up a routing table, and then something goes wrong with route programming before declaring “system ready.” That WARNING is interesting – we’ll come back to it.

CLI interface:

gw>
Available commands:
Unknown command. Type 'help' for list.

There’s a command-line interface with a gw> prompt. This is a gateway device.

Command names:

help                    version                 uptime
status                  portdump                routedump
reset                   loopback                memtest
log

And then, further down in the binary, a second cluster:

dbglvl                  peek                    poke
flashid                 crashme                 regdump

Re-running strings with -t x to see file offsets makes the layout visible:

$ strings -t x firmware.bin | grep -E "^\s+[0-9a-f]+ (help|version|uptime|status|portdump|routedump|reset|loopback|memtest|log|dbglvl|peek|poke|flashid|crashme|regdump)$"
   17bc memtest
   17e0 loopback
   1800 reset
   181c routedump
   1844 portdump
   1864 status
   1880 uptime
   18a0 version
   18c0 help
   1958 regdump
   197c crashme
   199c flashid
   19bc poke
   19d8 peek
   19fc dbglvl

The two clusters sit in different regions of .rodata: the familiar-looking names are packed contiguously between 0x17bc and 0x18c0, the rest live ~0x80 bytes further on between 0x1958 and 0x19fc, and the two groups never interleave. That’s not how a compiler lays out a single array. (log is missing from this grep because strings defaults to a minimum length of 4 characters and log is three; pass -n 3 to see it – it sits at 0x17a8, inside the first cluster.)

Wait. peek? poke? crashme? Those don’t sound like commands you’d put in a user manual.

We don’t yet know which of these are reachable from the user-facing CLI and which aren’t – the layout is suggestive, but suggestion isn’t proof. For now we have 16 command-shaped strings split across two clusters. Keep that in mind.

Error messages and source filenames:

ERROR: invalid port number (0-7)
ERROR: routing table full
ASSERT FAILED
main.c    cli.c    port.c    route.c    diag.c    flash.c

The firmware was built from six source files. The assert macro includes the filename – a gift for reverse engineers, because now we know the firmware’s module structure without reading a single instruction.

Firmware identity:

NetGW-MIPS32
v2.4.1-rc3
Apr 2026
(c) Embedded Network Systems

In under a minute, using nothing but strings, we now know:

• This is a network gateway device (NetGW-MIPS32, prompt gw>)
• It manages ports and routes with a forwarding table
• It has a CLI with at least 16 command-shaped strings, in two physically separated clusters (we don’t yet know which are user-visible)
• It was built from 6 source modules (main, cli, port, route, diag, flash)
• Something might be wrong with route programming during boot
• The second cluster has names like peek, poke, crashme – shapes of developer commands, not user ones

All from one command. No disassembly required. In the real vendor firmware project this series is based on, strings was our very first tool – and it revealed over 4,400 embedded strings that mapped directly to every subsystem in a 1.5 MB binary.

MIPS32 Survival Guide

You don’t need to learn MIPS assembly to follow this series. You need exactly five patterns. If you can spot these five things in a disassembly listing, you can trace function calls, find function boundaries, and follow data references across the entire binary.

1. Function prologue – “I’m starting a function”

addiu   sp, sp, -24       # Allocate 24 bytes of stack space
sw      ra, 20(sp)        # Save return address
sw      s0, 16(sp)        # Save callee-saved register

Every function begins by growing the stack downward. The size tells you how many local variables and saved registers it needs. -24 is a small function. -184 is a big one with lots of locals.

2. Function epilogue – “I’m returning to my caller”

lw      ra, 20(sp)        # Restore return address
lw      s0, 16(sp)        # Restore saved register
jr      ra                # Jump to return address
addiu   sp, sp, 24        # Deallocate stack (delay slot!)

Note the addiu sp, sp, 24 after jr ra. On MIPS, the instruction after a jump always executes (the branch delay slot). The stack cleanup happens “while” the CPU is jumping back.

3. Function call – “Call another function”

jal     80000b74          # Jump And Link — call function at 0x80000b74
                          # Sets ra = address of instruction after delay slot

JAL is the MIPS “call” instruction. It saves the return address in register $ra and jumps to the target. Every jal in the binary is a function call.

4. Load a 32-bit address – “Point at something”

lui     a1, 0x8000        # Load Upper Immediate: a1 = 0x80000000
addiu   a1, a1, 0x20e0    # Add lower 16 bits:    a1 = 0x800020e0

MIPS instructions are 32 bits wide, so you can’t load a 32-bit address in one instruction. The compiler uses a LUI/ADDIU pair: LUI loads the upper 16 bits, ADDIU adds the lower 16. This two-instruction pattern is how every string reference, every global variable access, and every table pointer gets loaded.

5. Register convention – “Who’s who”

That’s it. With these five patterns you can read any MIPS disassembly listing well enough to trace program flow.

From Strings to Code: Finding Where Strings Live

We found 111 strings. Now we need to connect them to the code that uses them. The question: which function prints “system starting”?

One convention before we go further. The flat binary starts at file offset 0 but the firmware was linked to run at 0x80000000 (we set that base in the Part 1 linker script – it’s KSEG0, the canonical bootable region on MIPS32). So runtime address = file offset + 0x80000000, and that’s the only translation we’ll need for the rest of the post. When we say “the string at 0x800020E0,” that’s the same byte as “file offset 0x20E0.”

Here’s the key insight. When the compiler generates code that references a string, it emits a LUI/ADDIU pair to load the string’s address into a register. If we know the string’s address in the binary, we can search for the LUI/ADDIU pair that loads it.

Let’s trace it manually. The string “system starting” is at file offset 0x20E0 in our binary:

$ strings -t x firmware.bin | grep "system starting"
   20e0 system starting

Since our binary is loaded at 0x80000000, the runtime address of this string is 0x800020E0. To load this address, the compiler emits:

lui     a1, 0x8000        # a1 = 0x80000000  (upper 16 bits)
addiu   a1, a1, 0x20e0    # a1 = 0x800020E0  (add lower 16 bits)

We can automate this directly against the instruction encoding – LUI is opcode 0x0F (top six bits) and ADDIU is opcode 0x09, so each instruction is a 32-bit word with a known shape, and pattern-matching the bytes is enough to find every pair:

import struct

BASE = 0x80000000

def find_string_xrefs(data, string_addr):
    """Find all LUI/ADDIU pairs that load a given address."""
    hi16 = (string_addr >> 16) & 0xFFFF
    lo16 = string_addr & 0xFFFF

    # Sign extension: if lo16 >= 0x8000, LUI loads hi16+1
    if lo16 >= 0x8000:
        lui_imm = (hi16 + 1) & 0xFFFF
    else:
        lui_imm = hi16

    xrefs = []
    for reg in range(1, 32):  # search all registers
        lui_word = 0x3C000000 | (reg << 16) | lui_imm
        lui_bytes = struct.pack('<I', lui_word)

        pos = 0
        while True:
            pos = data.find(lui_bytes, pos)
            if pos == -1:
                break
            # Search forward for matching ADDIU within 64 instructions
            addiu_word = 0x24000000 | (reg << 21) | (reg << 16) | (lo16 & 0xFFFF)
            addiu_bytes = struct.pack('<I', addiu_word)
            addiu_pos = data.find(addiu_bytes, pos, pos + 256)
            if addiu_pos != -1:
                xrefs.append((BASE + pos, BASE + addiu_pos))
            pos += 4

    return xrefs

The sign-extension detail matters more than it looks: MIPS ADDIU sign-extends its 16-bit immediate. If the lower half of the address is >= 0x8000, the LUI has to load one more than the upper half to compensate. Get this wrong and you’ll miss half your cross-references. (In the real project, getting this right was the difference between finding 2,000 strings and finding all 4,471.)

The runnable script in the bundle (tools/find_string_xrefs.py) takes the same approach but uses Capstone to decode instructions instead of matching byte patterns – mostly so it can track register state across the LUI/ADDIU window and reject pairs where an intervening write has invalidated the upper half. The companion scripts tools/find_word.py, tools/find_function_prologues.py, and tools/find_cli_tables.py cover the data-pointer scan and the prologue/table walks we’ll use in §5 and §6. (apt install python3-capstone on Debian-family systems, or pip install capstone inside a venv elsewhere.)

Finding Function Boundaries

Now that we can locate where a string is referenced, we need to find which function contains that reference. The answer: scan backward for a function prologue.

def find_function_prologues(data):
    """Find all ADDIU SP, SP, -N instructions (function starts)."""
    prologues = []
    for off in range(0, len(data) - 3, 4):
        word = struct.unpack_from('<I', data, off)[0]
        if (word >> 16) == 0x27BD:           # ADDIU $sp, $sp, imm
            imm = word & 0xFFFF
            if imm >= 0x8000:                # negative = stack allocation
                frame_size = imm - 0x10000
                prologues.append((BASE + off, frame_size))
    return prologues

Running this on our firmware.bin:

Found 35 function prologues:
  0x80000248: addiu sp, sp, -32
  0x800002F0: addiu sp, sp, -32
  0x80000354: addiu sp, sp, -40
  ...
  0x80000EFC: addiu sp, sp, -48    ← iterate_active_routes?
  ...
  0x800015C8: addiu sp, sp, -184   ← big function (CLI main loop?)
  0x800016B0: addiu sp, sp, -24    ← last function (firmware_main?)

35 function prologues. The real count is higher – leaf functions that never call anything else don’t need to save $ra and often skip the standard prologue entirely, so they slip past this scan. But 35 is enough to map the major functions, and that’s what we need to follow the strings.

We can also count JAL calls to build a call graph:

JAL call frequency:
  0x800002E4: called 111 times   ← most-called function (uart_puts?)
  0x80000530: called  15 times   ← probably uart_puthex
  0x80000790: called  15 times   ← probably uart_putdec
  0x80000B74: called  12 times   ← log_msg (used in every subsystem)
  0x800002C4: called   6 times   ← uart_putchar
  0x80000AA8: called   4 times   ← fw_assert
  0x80000FEC: called   4 times   ← route_add (called 4 times in main!)

The function at 0x800002E4, called 111 times, is almost certainly the UART print function – every command handler, every log message, every error path calls it. That matches what we’d expect from looking at the source filenames: a CLI-heavy firmware spends most of its time printing strings.

The CLI Table: Following Pointer Chains

Now for the detective work. We know the firmware has CLI commands because strings found “help”, “version”, “routedump”, and friends. But strings are just bytes – they don’t dispatch themselves. When the user types help, some function has to compare the input against every known command name and call the matching handler.

Here’s where the xref scanner from §4 actually pays off. Pick a command-name string and ask: which LUI/ADDIU pair in the binary loads its address? For "system starting" (0x800020e0) the answer is one xref at 0x800016bc – some function loads it and prints it. But for "help" (0x800018c0)?

$ python3 tools/find_string_xrefs.py firmware.bin 0x800018c0
LUI/ADDIU pairs loading 0x800018c0: 0

Zero. Same for every other command name in both clusters. Yet the strings clearly exist – strings found them. So how is the firmware getting at them?

A second scanner answers the complementary question: scan the binary for any 32-bit word that equals the address we’re looking for, regardless of how it got there.

$ python3 tools/find_word.py firmware.bin 0x800018c0
0x800018c8: 32-bit word == 0x800018c0

The address of "help" shows up as a pointer, sitting in .rodata itself, eight bytes after the string. No code loads it; data points at it. Combine the two scanners and you get a sharp diagnostic for any string in the binary:

"help" lands in the second row. So do all 16 command names. That’s the signal: when a string has no code xref but its address appears as a 32-bit word inside .rodata, it’s referenced through a table, not printed inline.

The natural shape of such a table is an array of structs, one per command:

struct cli_entry {
    const char *name;       /* "help", "version", etc. */
    void (*handler)(int, const char **);   /* function pointer */
    const char *help;       /* "Show this help message" */
};

That’s 12 bytes per entry (three 32-bit pointers). We can scan .rodata for sequences of 12-byte entries where the first and third words are pointers to printable strings, and the second word is a pointer into the .text section.

# Scan .rodata for CLI table pattern: [string_ptr, code_ptr, string_ptr] × N
for off in range(rodata_start, rodata_end, 12):
    name_ptr  = read_u32(off)
    handler   = read_u32(off + 4)
    help_ptr  = read_u32(off + 8)

    if (is_string_pointer(name_ptr) and
        is_code_pointer(handler) and
        is_string_pointer(help_ptr)):
        # Found a table entry!

Running this on our firmware binary, we find two tables:

Table at 0x800018C8 (10 entries):
  "help"         handler=0x80000354  "Show this help message"
  "version"      handler=0x800003FC  "Show firmware version"
  "uptime"       handler=0x80000838  "Show system uptime"
  "status"       handler=0x80000874  "Show system status"
  "portdump"     handler=0x80000E18  "Dump port status [port_num]"
  "routedump"    handler=0x800008FC  "Dump routing table"
  "reset"        handler=0x8000047C  "Reset the system"
  "loopback"     handler=0x8000135C  "Run loopback test <port>"
  "memtest"      handler=0x800012A4  "Run memory test"
  "log"          handler=0x800004A4  "Show recent log entries"

Table at 0x80001A04 (6 entries):
  "dbglvl"       handler=0x800009FC  "Set debug verbosity level"
  "peek"         handler=0x800005E4  "Read memory address"
  "poke"         handler=0x800004F4  "Write memory address"
  "flashid"      handler=0x800010C4  "Read SPI flash JEDEC ID"
  "crashme"      handler=0x80000B40  "Trigger deliberate crash"
  "regdump"      handler=0x800006CC  "Dump hardware registers"

Two tables – one with 10 entries, one with 6. Now the obvious question: if all 16 commands are in the binary, why does help only show 10 of them?

Run the xref scanner against each table’s base address:

$ python3 tools/find_string_xrefs.py firmware.bin 0x800018c8   # public table
LUI/ADDIU pairs loading 0x800018c8: 3
LUI/ADDIU @ 0x80000388 (inside function 0x80000354)
LUI/ADDIU @ 0x80001468 (inside function 0x80001424)
LUI/ADDIU @ 0x800014f4 (inside function 0x80001424)

$ python3 tools/find_string_xrefs.py firmware.bin 0x80001a04   # hidden table
LUI/ADDIU pairs loading 0x80001a04: 1
LUI/ADDIU @ 0x80001498 (inside function 0x80001424)

Two functions reference the public table; only one references the hidden table – and the public-only function is the asymmetry the help command rests on.

The function at 0x80000354 is the public-only one. From the table itself we know it’s cmd_help, the handler for the help command: a small function that walks one array and prints its entries. The function at 0x80001424 references both tables – that’s the dispatcher: read a line of input, walk the public table looking for a match, then walk the hidden table, and call the handler if either hit. (The dispatcher loads the public table address twice – once when the search loop sets up its iterator, then again after the matched handler returns and the table base needs to be re-fetched to read the function pointer. The compiler didn’t bother keeping the address live across the call. Routine scheduling, not a second logical reference.)

Nothing was “removed.” The 6 commands in the second table aren’t dead code, aren’t disabled, aren’t gated behind a flag. They just don’t get listed by cmd_help, because cmd_help only iterates one of the two arrays the dispatcher accepts. Whoever wrote the firmware wanted peek, poke, crashme, and friends reachable from the prompt without advertising them in the help output – and they implemented that with the simplest possible mechanism: two arrays, one printed, both dispatched.

In the real vendor firmware, the same xref-the-table-base trick discovered 148 CLI commands across multiple dispatch tables – including handlers that could read and write arbitrary hardware registers, dump internal routing tables, and trigger diagnostic modes not documented in any user manual. None of them were hidden. They were just iterated by a different function than the one printing the help text. The strings gave every single one away the moment we asked which functions referenced each table.

Putting It Together: The Subsystem Map

It’s worth naming what we’ve actually been doing for the last few sections. A “symbol,” in the linker’s sense, is a triple: a name, an address, and a kind (function, data, etc.). The flat binary on disk has none of them – they were dropped on the way from ELF to objcopy -O binary. What we’ve been doing is rebuilding that triple ourselves: prologue scanning gave us addresses, JAL-frequency and string xrefs let us guess the kind (UART printer, log function, command handler, leaf), and the strings themselves – "help", "system starting", the source filenames – gave us names. None of them are real symbols. All of them are good enough to navigate by. That’s the move that makes static RE work on stripped firmware: when the symbol table isn’t there, you reconstruct it.

Starting from nothing but a flat binary and the strings command, we’ve built a surprisingly complete picture of this firmware:

Firmware Subsystem Map (built from static analysis only)
========================================================

Identity:   NetGW-MIPS32 v2.4.1-rc3, network gateway device
Source:     6 modules (main.c, cli.c, port.c, route.c, diag.c, flash.c)
Interface:  UART serial console, prompt "gw>"
Functions:  35 prologue-detected (plus an unknown number of leaf functions
            that don't save $ra and slip past the standard prologue scan)
Strings:    111 embedded
Commands:   16 total — all 16 dispatched, only 10 listed by cmd_help

Module Map:
  MOD_CORE  (1) — System init, boot sequence
  MOD_CLI   (2) — Command dispatch, input processing
  MOD_PORT  (3) — 8-port forwarding engine
  MOD_ROUTE (4) — Route table management
  MOD_DMA   (5) — DMA engine (referenced but minimal)
  MOD_DIAG  (6) — Diagnostic tests (loopback, memtest)
  MOD_FLASH (7) — SPI flash operations

Suspicious:
  - "WARNING: no routes programmed for stack" during boot
  - "bridge flag set, skipping" — routes being skipped?
  - "bridge routes deferred to bridge handler" — are they ever handled?

All of this without running the firmware, without a disassembler GUI, and without symbols. Just strings, some Python scripting, and pattern recognition.

But those suspicious messages are nagging. The boot sequence says routes are being skipped because of a “bridge flag,” and then warns that no routes were programmed. That sounds like a bug – routes that should be programmed into the forwarding table are getting silently dropped.

In Part 3, we’ll throw this binary at Ghidra to decompile it into readable C, trace the execution path through the route engine function by function, and find exactly where – and why – those routes are disappearing. Then we’ll fix it. With a hex editor.

References

• sample_firmware.tar.gz – firmware.bin, build sources (main.c, startup.S, linker.ld, Makefile), and the four Python scripts used in this post under tools/:
  • find_string_xrefs.py – find LUI/ADDIU pairs that load a given address (uses Capstone)
  • find_word.py – find a 32-bit word stored as data anywhere in the binary
  • find_function_prologues.py – find function starts via the addiu sp, sp, -N pattern (uses Capstone)
  • find_cli_tables.py – scan .rodata for {name_ptr, code_ptr, help_ptr} runs
• Capstone Engine – multi-architecture disassembly framework. apt install python3-capstone (Debian/Ubuntu) or pip install capstone (in a venv elsewhere)
• MIPS32 Architecture For Programmers Vol. II – Instruction set reference (LUI, ADDIU, JAL encoding)
• MIPS Calling Convention – Register usage: $a0-$a3, $v0, $ra, $sp
• strings(1) – GNU binutils string finder; use -t x for hex offsets
• Part 1: Headers, Symbols, and other things you won’t find – Building the sample firmware

Then someone hands you a 1.5 MB file. No extension, no headers, no documentation. file says “data.” readelf says “not an ELF file.” objdump refuses to open it. Welcome to firmware reverse engineering.

This is Part 1 of a five-part series that picks up where the ELF post left off, but for bare-metal firmware on MIPS32 embedded devices. Everything you learned about ELF structure, symbol tables, and linking is still useful – mostly as a reference for what isn’t in front of you anymore.

The techniques come from a real project analyzing production firmware on an embedded network device. I can’t name the vendor (lawyers), but every pattern, every tool, every trick here is something we actually used in anger. To keep it reproducible, we’ll build a sample MIPS32 firmware – a small network gateway with a UART serial console – that recreates the same structures.

Bonus – the sample firmware bundled: if you’d rather skip the build and start poking at bytes, the same sources, the resulting firmware.bin and firmware.elf, the multi-partition image, and the small tools/ directory used in Part 2 are all bundled in sample_firmware.tar.gz. Grab it and follow along on your own copy:

curl -L -O https://res.cloudinary.com/gotocco/raw/upload/v1777182332/sample_firmware.tar.gz
tar xf sample_firmware.tar.gz
cd sample_firmware

What’s Missing: ELF vs Flat Binary

The quickest way to understand flat firmware is to compare it with what you already know. Here’s what a standard ELF binary looks like at byte zero:

$ xxd -l 64 firmware.elf
00000000: 7f45 4c46 0101 0100 0500 0000 0000 0000  .ELF............
00000010: 0200 0800 0100 0000 0000 0080 3400 0000  ............4...
00000020: a431 0100 0110 0070 3400 2000 0500 2800  .1.....p4. ...(.
00000030: 0d00 0c00 0300 0070 0821 0100 0821 0080  .......p.!...!..

You can spot the 7F 45 4C 46 magic immediately – .ELF. After that comes the class (32-bit), endianness (little), the machine type (MIPS), the entry point, program header offset, section header offset. Everything the kernel’s loader needs to map this binary into memory.

Now here’s what a firmware binary looks like at byte zero:

$ xxd -l 64 firmware.bin
00000000: 0580 1d3c 0000 bd37 0480 1c3c 0000 9c37  ...<...7...<...7
00000010: 8000 0008 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................

No magic bytes. No headers. The very first byte of this file is a machine instruction. If you happen to read MIPS, you can decode it on sight: lui sp, 0x8005 – loading the upper half of the stack pointer, the canonical first move of a freshly-booted CPU. Once the boot ROM (or first-stage bootloader) copies the payload to RAM and jumps to its entry address, this is where execution begins. (Register conventions come in Part 2. For now, sp, ra, a0..a3, and s0..s7 are the names you’ll keep seeing.)

Here’s what each format gives you:

Remember readelf -S showing section headers? nm listing every function by name? objdump -tT dumping the symbol table? None of that works here. The flat binary has no metadata. The hex dump is the documentation.

Why Firmware Is Flat

This isn’t an accident or a limitation – it’s a deliberate engineering choice. Here’s why embedded firmware ships as raw binary blobs:

Bootloader simplicity. The first code that runs when a MIPS chip powers on is a tiny bootrom, often a few hundred bytes burned into mask ROM. It does not have an ELF parser. It loads a fixed payload from flash into RAM and jumps to a known entry address – in our sample, 0x80000000. Any complexity in the binary format becomes complexity in silicon, and silicon has to be right on the first wafer.

Speed. A router that takes 30 seconds to come up after a power cycle is a router nobody deploys. Flat binaries boot in one memcpy – no parsing, no relocations, no dynamic linking. Copy the bytes, jump to the entry point, you’re running.

Deterministic layout. The developer controls exactly where every byte lands in memory. The linker script says “put the exception vectors at 0x80000000, put .text right after, put .bss at 0x80040000,” and that is exactly what happens. No ASLR, no PIE, no loader surprises. (You miss this guarantee the first time you stop missing it.)

The objcopy pipeline. Here’s how a flat binary is produced from source:

# Step 1: Compile to object files
mipsel-linux-gnu-gcc-14 -mips32r2 -EL -c -o startup.o startup.S
mipsel-linux-gnu-gcc-14 -mips32r2 -EL -O1 -ffreestanding -nostdlib -c -o main.o main.c

# Step 2: Link into ELF (intermediate)
mipsel-linux-gnu-ld -T linker.ld -o firmware.elf startup.o main.o

# Step 3: Strip to flat binary -- THIS is the key step
mipsel-linux-gnu-objcopy -O binary firmware.elf firmware.bin

Step 3 is where everything disappears. objcopy -O binary takes the ELF, extracts only the loadable segments (the raw bytes that need to be in memory at runtime), and writes them sequentially to a file. Section headers? Gone. Symbol table? Gone. String table? Gone. Every piece of metadata that made the ELF navigable goes in the bin.

The developer still has the ELF – they need it to debug – but what ships to the customer is the flat binary. And when you are the one doing the reverse engineering, the flat binary is usually all you get.

Memory Layout: Where Everything Lives

Every flat firmware binary has a base address – the memory address where the bootloader copies it. For MIPS32 devices, this is typically in KSEG0 (0x80000000 - 0x9FFFFFFF), which is cached, unmapped kernel space. Our sample firmware uses 0x80000000:

Virtual Memory Map
==================

0x80000000 +---------------------------+
           | Exception vectors (640 B) |  Startup + exception handlers
0x80000280 +---------------------------+
           | .text (5,392 bytes)       |  All firmware functions
0x80001790 +---------------------------+
           | .rodata (2,396 bytes)     |  Strings, CLI tables, constants
0x800020EC +---------------------------+
           | (end of flash content)    |
           ...
0x80040000 +---------------------------+
           | .bss (1,024 bytes)        |  Globals (zeroed at boot)
0x80040400 +---------------------------+
           | (free RAM)               |
           ...
0x80050000 +---------------------------+
           | Stack top                 |  Stack grows downward
           +---------------------------+

MMIO Registers (not in binary):
  0xB8000000  UART (TX, RX, status, control)
  0xB8010000  GPIO (data, direction)
  0xB8020000  Timer (count, compare, control)
  0xB8030000  System (reset, clock, version)

For this sample, the address translation is trivial: once you’ve extracted the raw payload, any byte at file offset N lives at virtual address 0x80000000 + N. No page tables, no ASLR, no segment mapping – one addition. If you find an interesting string at file offset 0x18CC, you know it sits at 0x800018CC in the running firmware.

Compare with ELF, where the loader reads program headers to figure out which file offset maps to which virtual address, with each segment at its own alignment. Here, you can do the math in your head.

Build Your Own: The Sample Firmware

Before reverse engineering bytes, it helps to watch where the bytes come from. The build is short, and once you’ve seen a flat binary fall out the end of objcopy, the rest of the series stops feeling like archaeology and starts feeling like accounting.

Our sample firmware simulates a small MIPS32 network gateway – think of an embedded router or bridge device managed over a UART serial console. This is a common pattern: millions of MIPS-based routers and switches run firmware exactly like this (Broadcom, MediaTek, and Qualcomm Atheros SoCs all use MIPS32 cores).

The firmware has:

• UART serial console with a command-line interface
• 8-port forwarding engine with route table management
• Public commands: help, version, uptime, status, portdump, routedump, reset, loopback, memtest, log
• Hidden commands: peek, poke, flashid, crashme, regdump, dbglvl (more on these later)
• MMIO register access for UART, GPIO, Timer, and System registers
• Assert/logging framework with source filename references

You’ll need the MIPS cross-compilation toolchain:

# Debian/Ubuntu
sudo apt install gcc-mipsel-linux-gnu binutils-mipsel-linux-gnu

Build and inspect:

$ make clean && make
rm -f *.o *.elf firmware.bin
mipsel-linux-gnu-gcc-14 -mips32r2 -EL -fno-pic -mno-abicalls -c -o startup.o startup.S
mipsel-linux-gnu-gcc-14 -mips32r2 -EL -O1 -std=gnu89 -fno-builtin -ffreestanding \
    -nostdlib -mno-abicalls -fno-pic -G0 -mno-gpopt -Wall -c -o main.o main.c
mipsel-linux-gnu-ld -T linker.ld --no-warn-rwx-segments -o firmware.elf startup.o main.o
mipsel-linux-gnu-objcopy -O binary firmware.elf firmware.bin

========================================
  firmware.bin: 8496 bytes
  Format: Raw MIPS32 LE flat binary
  Base:   0x80000000
========================================

8,496 bytes of raw MIPS32 instructions and data. No headers, no symbols, no sections. Just bytes.

Try running file on it:

$ file firmware.bin
firmware.bin: data

$ file firmware.elf
firmware.elf: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV),
              statically linked, not stripped

file does not even know what firmware.bin is. It just says “data.” That is what you are up against in firmware RE. Meanwhile, the intermediate ELF is recognized on sight – and that ELF is the one thing that never ships to customers.

The firmware has a UART interface that prints boot messages. With a serial cable, you would watch it initialize ports, set up routing tables, and present a gw> prompt. For this series, we don’t need the hardware – the binary is enough.

If you do want to see it run, there are two options. You can emulate it under QEMU with the Malta MIPS board (qemu-system-mipsel -M malta -kernel firmware.elf -serial stdio -nographic) – the build includes a QEMU-compatible UART variant, and using the ELF here is just a convenience for emulation and debugging. The shipped firmware artifact is still the flat firmware.bin payload inside the image. Or if you have real hardware, a Microchip PIC32MX470 Curiosity board (DM320103) has a MIPS32 M4K core with UART-over-USB and can run similar bare-metal firmware. Neither is required to follow along – everything in this series works purely through static analysis of the binary.

In Part 2, we’ll learn how to extract information from those bytes without ever running the firmware.

Firmware Is More Than One File

When you download a firmware update from a vendor, you don’t get a bare .bin. You get a firmware image – a container wrapping one or more payloads with headers, checksums, and a partition table. The flat binary is the payload we care about, but vendors rarely distribute it naked.

Our sample includes a packager that produces exactly this kind of image:

$ python3 fwpack.py firmware.bin firmware.img

Packed firmware image: firmware.img
  Total size:    9152 bytes
  Image CRC-32:  0x0FF01071
  Partitions:    3

  [bootloader]
    Offset:  0x0120
    Size:    32 bytes
    CRC-32:  0x4F7A6ACA
  [main_fw]
    Offset:  0x0160
    Size:    8496 bytes
    CRC-32:  0x9F25E0DC
  [config]
    Offset:  0x22B0
    Size:    256 bytes
    CRC-32:  0x2683AC5B

Notice the two magic strings (FWPK for the image header, FWND at the trailer), the per-partition CRCs, and the global image CRC – those are the three things every custom container format reinvents in some form.

The structure looks like this:

Firmware Image Layout
=====================

0x0000 +---------------------------+
       | Image Header (256 bytes)  |  Magic "FWPK", version, CRC-32,
       |                           |  firmware name, build date, board ID,
       |                           |  partition count, entry point
0x0100 +---------------------------+
       | Partition 0: Bootloader   |  32B header + 32B payload
       |  (tiny MIPS stub)         |  Sets SP, jumps to main FW
0x0140 +---------------------------+
       | Partition 1: Main FW      |  32B header + 8,496B payload
       |  (our firmware.bin)       |  The actual firmware code
0x2290 +---------------------------+
       | Partition 2: Config       |  32B header + 256B payload
       |  (default settings)       |  Hostname, IP, routes, serial config
0x23B0 +---------------------------+
       | Trailer (16 bytes)        |  End magic "FWND", size, CRC-32
0x23C0 +---------------------------+

Each partition has its own CRC-32, and the image has a global CRC in the header. Two reasons this matters: the device’s bootloader verifies them before flashing (a corrupted update bricks the device), and if you ever binary-patch the firmware, you have to recompute every CRC the bootloader checks – or the device rejects your modified image and refuses to boot.

Every vendor invents their own container format. Some build on well-known structures (U-Boot’s uImage). Others ship custom headers with proprietary magic numbers, encryption layers, or RSA signature verification. Your first RE task is always defeating the container before you can read the code inside it.

We can verify our image and extract the partitions back out:

$ python3 fwpack.py --info firmware.img

Firmware Image: firmware.img
  Magic:       FWPK
  Version:     1.0.0.0
  Total size:  9152 bytes (0x23c0)
  Partitions:  3
  Entry point: 0x80000000
  Image CRC:   0x0FF01071
  FW name:     NetGW-MIPS32 v2.4.1-rc3
  Build date:  Apr 2026
  Board ID:    MIPS32-GW-DEV

  CRC verified OK

Partition Table:
  #  Type          Offset     Size       CRC-32      Load Addr
  --------------------------------------------------------------------
  0  bootloader    0x0120         32     0x4F7A6ACA  0x80000000  [OK]
  1  main_fw       0x0160       8496     0x9F25E0DC  0x80000000  [OK]
  2  config        0x22B0        256     0x2683AC5B  0x80000000  [OK]

Remember: ELF is a standard. Documented, parsed by every tool you own. Here? Custom format, custom magic, custom checksums. file won’t help. readelf won’t help. You and a hex editor, on a Friday night.

Debug Symbols: The Cheat Code

Before we dive deeper into the raw binary (that’s Part 2), one more comparison to make the stakes concrete. Same function – firmware_main, the entry point of our gateway – once without symbols and once with.

Without symbols (the flat binary, as shipped to customers):

$ mipsel-linux-gnu-objdump -D -b binary -m mips firmware.bin
...
    16b0:   e8ffbd27    addiu   sp,sp,-24
    16b4:   1400bfaf    sw      ra,20(sp)
    16b8:   1000b0af    sw      s0,16(sp)
    16bc:   0080053c    lui     a1,0x8000
    16c0:   e01ea524    addiu   a1,a1,7904
    16c4:   dd02000c    jal     0xb74
    16c8:   01000424    li      a0,1
    ...
    16f0:   f502000c    jal     0xbd4
    16f4:   00000000    nop
    16f8:   ad03000c    jal     0xeb4
    16fc:   00000000    nop
    1700:   03000624    li      a2,3
    1704:   01000524    li      a1,1
    1708:   fb03000c    jal     0xfec
    170c:   010a0424    li      a0,2561
    ...
    1728:   fb03000c    jal     0xfec
    172c:   a8c00434    li      a0,0xc0a8
    ...
    1784:   7205000c    jal     0x15c8

What can you read here? “Call 0xb74. Call 0xbd4. Call 0xeb4. Call 0xfec four times with different arguments. Call 0x15c8.” That is the entirety of the information. Just addresses. You have no idea what any of those functions do, or whether two of them do the same thing.

With symbols (the ELF, kept for debugging):

$ mipsel-linux-gnu-objdump -d firmware.elf
...
800016b0 <firmware_main>:
800016b0:   27bdffe8    addiu   sp,sp,-24
800016b4:   afbf0014    sw      ra,20(sp)
800016b8:   afb00010    sw      s0,16(sp)
800016bc:   3c058000    lui     a1,0x8000
800016c0:   24a51ee0    addiu   a1,a1,7904
800016c4:   0c0002dd    jal     80000b74 <log_msg>
800016c8:   24040001    li      a0,1
...
800016f0:   0c0002f5    jal     80000bd4 <port_init>
800016f4:   00000000    nop
800016f8:   0c0003ad    jal     80000eb4 <route_init>
800016fc:   00000000    nop
80001700:   24060003    li      a2,3
80001704:   24050001    li      a1,1
80001708:   0c0003fb    jal     80000fec <route_add>
8000170c:   24040a01    li      a0,2561
...
80001728:   0c0003fb    jal     80000fec <route_add>
8000172c:   3404c0a8    li      a0,0xc0a8
...
80001784:   0c000572    jal     800015c8 <cli_main_loop>

Now you can read the whole boot sequence: log “system starting”, initialize ports, initialize the route table, add four routes (10.1.x.x, 10.2.x.x, 192.168.x.x, 172.16.x.x), log “system ready”, print banner, enter the CLI main loop.

Same machine code, same addresses, same behavior. The only difference is that the ELF carries a symbol table mapping addresses to names:

$ mipsel-linux-gnu-nm firmware.elf | head -20
80000000 T _start
80000248 T _exception_handler
800002e4 T uart_puts
80000b74 T log_msg
80000bd4 T port_init
80000eb4 T route_init
80000efc T iterate_active_routes
80000fec T route_add
800013d0 T cli_process_command
800015c8 T cli_main_loop
800016b0 T firmware_main
...

93 symbols in our toy firmware. The vendor firmware we worked on had over 4,400 functions across 1.5 MB of code. Picture 4,400 nameless functions – just addresses and raw MIPS instructions, with no obvious starting point and no obvious end. That is why recovering function names and code structure takes weeks of manual work.

Symbols turn weeks into hours. Vendor firmware almost never ships with them. The rest of this series is about getting them back.

What We’re Up Against

Let me summarize where we stand. We have:

• A flat binary – raw MIPS32 instructions with no headers, no sections, no symbols
• A firmware image container – custom format with magic numbers, CRC checksums, and multiple partitions
• A trivial address mapping – file offset + base address = virtual address
• 109 embedded strings – the main human-readable foothold in this sample binary

And we know what we’re missing:

• No function boundaries (where does one function end and another begin?)
• No function names (what does the code at 0xbd4 do?)
• No variable names (what’s stored at 0x80040340?)
• No type information (is that 32-bit value an integer, a pointer, or a bitfield?)
• No cross-references (who calls this function? Where is this string used?)

The tools from our ELF toolkit – readelf, nm, objdump -tT – are useless against a flat binary. We need a different approach.

Here’s what we’re going to do across the rest of the series:

Part 2: “Opcodes, Prologues, and other hidden patterns” – We’ll learn to read MIPS32 assembly just enough to find function boundaries, trace calls, and discover a hidden CLI with commands that don’t appear in any help output. Our entry point? Those 109 strings.

Part 3: “Decompilers, Annotations, and other ways to read the unreadable” – We’ll throw the binary at Ghidra to decompile all 8,496 bytes into readable C, then follow the execution path through the route engine to discover a priority inversion bug that silently drops traffic. Then we’ll binary-patch the fix – and recalculate the CRC.

Part 4: “Symbols, Scripts, and other linking nightmares” – We’ll try to go from decompiled C back to a working binary. We’ll discover why that’s far harder than it sounds, build custom linker scripts, and learn why firmware linking is fundamentally different from anything in userspace.

Part 5: “Versions, Callgraphs, and other ways to compare what changed” – We’ll compare two firmware versions side by side, decompile the same functions from each, and use diffs and callgraph patterns to see how a vendor fix evolves between releases.

Every technique here comes from a real project. Every pattern is something we hit in production firmware. The sample recreates those patterns so you can follow along hands-on, without anyone’s lawyer getting involved.

So far we’ve established what isn’t in the binary. In Part 2, we start with the only thing that is readable – the strings – and from there pull function boundaries, a call graph, and a hidden CLI out of raw bytes.

References

• ELF’s Linker’s and other magical creatures – The 2018 predecessor to this series
• MIPS32 Architecture For Programmers – MIPS instruction set reference
• GNU binutils documentation – objcopy, objdump, readelf, nm
• Ghidra – NSA’s reverse engineering framework
• Capstone Engine – Lightweight disassembly framework
• MIPS Calling Conventions – Register usage and stack frame layout

Note: This article was written in 2020 as a companion to Part One (AMD64) but was never published. The technical content reflects the state of NetBSD and the RK3399 platform at that time. Published here now without major changes.

Introduction

In Part One we covered legacy initialization of an AMD64-based system. This time we look at ARM processors.

ARM architecture is well known from smaller devices — embedded systems, IoT, smartphones — but ARMv8 already powers modern servers and laptops. The architecture has many revisions with backward compatibility considerations. To keep things focused, we will concentrate on ARMv8 (64-bit), which can also execute the 32-bit instruction set in compatibility mode. We will run 64-bit NetBSD throughout.

There are two main ways manufacturers obtain ARM processors: via RTL design (ARM sells the core design directly) or as an IP license (the company buys the specification). This business model makes the architecture highly flexible and implementation-specific, which also makes learning it harder — there are many chip-specific details that the architecture itself leaves undefined.

For this article we look at generic, high-level ARMv8 details first, then go deeper using a specific hardware platform. I originally considered Raspberry Pi 3 or 4, but the Broadcom GPU plays an essential and poorly-documented role in its boot process — after reset it is the GPU that starts first and later releases the ARM cores. The lack of public documentation makes it a poor teaching example.

Instead I use the RK3399Pro from Rockchip, the chip behind PINE64 devices such as the ROCKPro64 single-board computer and the Pinebook Pro laptop. It has fully open-sourced documentation and real-world relevance.

This article intentionally avoids the Secure Boot process and UEFI internals — those will get their own dedicated part.

The Bigger Picture

Exception Levels

ARMv8 introduces four privilege levels called Exception Levels (EL), numbered 0 to 3. Every implementation must support EL0 and EL1; EL2 and EL3 are optional.

The name “exception level” can be confusing — “privilege level” is the clearer term, and was used in earlier ARM specifications. The best way to understand them is by what runs at each level:

• EL0 — unprivileged user-space applications. No direct access to system registers, page tables, or hardware devices.
• EL1 — the OS kernel. Responsible for hardware configuration, page table setup, and managing EL0 processes.
• EL2 — hypervisor. UEFI also runs here during boot — not because UEFI itself needs hypervisor privileges, but because running at EL2 preserves EL2 availability for the OS. If UEFI ran at EL1, the OS would have nowhere to put a hypervisor later.
• EL3 — Secure Monitor. The most privileged level, used by ARM Trusted Firmware.

EL0 and EL1 are the minimum required to run a modern OS such as NetBSD or Linux.

ARMv8 only allows a decrease in execution state going toward less privileged levels: a 64-bit EL2 can host a 64-bit or 32-bit EL1, a 64-bit EL1 can run 64-bit or 32-bit EL0 applications, but a 32-bit OS cannot run 64-bit applications. The execution state of each level is controlled by bits in the higher level’s configuration registers: SCR_EL3.RW determines whether EL2 runs AArch64 or AArch32, and HCR_EL2.RW determines the same for EL1. The reset execution state of EL3 itself is controlled by RMR_EL3.AA64 — but that register affects the state after a warm reset, not the running state of lower ELs.

Out of Reset

Unlike AMD64 where the reset vector is a fixed physical address (0xFFFFFFF0), ARMv8 processors do not have a fixed reset vector. After reset, the processor reads RVBAR_ELx (where x is the highest implemented exception level, typically EL3) to obtain the implementation-defined reset vector address, then fetches instructions from that address.

For the rest of this article, “EL3” refers to EL3 if implemented, or the highest available exception level otherwise. Most ARMv8 chips do implement EL3, but it is not architecturally mandatory.

After reset, the processor state is architecturally defined as follows:

• All interrupt masks are set (DAIF = 0b1111 — all asynchronous exceptions masked)
• MMU is off; instruction and data caches are disabled (SCTLR_ELx.M, .I, .C = 0)
• The CPU executes physical addresses directly
• TLB and cache contents are implementation-defined (boot ROM typically invalidates both)
• Memory is unconfigured; the interrupt controller state is unknown

General Boot Sequence

Many architectures — including ARMv7 — start code execution from the first entry of an exception table (the reset vector). In ARMv8 the reset vector is no longer part of the exception table. Instead, after reset the processor fetches from the implementation-defined address in RVBAR_ELx (where x is the highest implemented level: 3–1).

The first instructions executed are typically initialization code in a small on-chip ROM (or OTP — One-Time Programmable memory). Most peripherals — memory controllers, bus controllers — are disabled right after reset, so the processor needs some immediately accessible memory. That memory must be inside the chip itself, which makes it expensive and therefore small (order of kilobytes).

Boot ROM prepares the processor and handles chip-specific details: memory and cache initialization, branch predictor configuration. Since TLB and cache state after reset is implementation-defined, boot ROM typically performs explicit invalidations before enabling them.

Boot Stages

The ARMv8 boot process typically has three stages:

1. First stage — Boot ROM: read-only code on-chip, executes immediately after reset
2. Second stage — Boot Loader: loaded by Boot ROM from external storage
3. Third stage — Advanced Boot Loader: loads the operating system (where UEFI lives)

BootROM vs. Bootloader

BootROM is the small program in read-only memory (ROM or OTP), typically provided by the chip manufacturer since it handles low-level, platform-specific initialization. Its main job is to prepare the CPU to execute external code from storage (flash, SD card, NVMe). It may also provide cryptographic features for a chain of trust and handle partition layout parsing. ARM Trusted Firmware defines requirements for this stage, but for non-secure boot there is no mandated BootROM interface standard.

Bootloader is a program (or chain of programs) whose primary task is to load the OS. It typically lives on external storage, is modifiable, and can be upgraded. Some systems have a BootROM advanced enough to load a kernel or UEFI image directly; others have a minimal BootROM that can only fetch and execute from a fixed memory location, requiring the bootloader to be split into multiple stages due to size constraints.

The second-stage bootloader, loaded by BootROM from external storage, is platform-specific and sometimes split further into sub-programs due to hardware constraints. Its goal is to initialize system RAM so the third-stage bootloader has full platform access.

The third stage prepares the entire platform to boot the OS. UEFI operates here at EL2. DRAM is operational at this point. The bootloader reads the OS from storage, verifies the image if secure boot is active, passes boot parameters, and jumps to the kernel entry point.

Before the OS: Device Tree and Platform Differences

ARM hardware covers a wide range — embedded devices through enterprise servers — and the boot environment differs accordingly.

For embedded devices there is usually no universal hardware description standard. What is common is a device tree: a structured description of the peripherals connected to the chip (I2C, SPI, UART, CAN buses and their devices) that the OS cannot discover at runtime. The kernel reads this binary at boot and configures peripherals accordingly. The Embedded Base Boot Requirements (EBBR) specification defines the format.

For server-class systems, the Server Base Boot Requirements (SBBR) apply instead. SBBR-compliant systems must not expose a device tree to the OS — hardware discovery follows ACPI and UEFI conventions.

ROCKPro64 and RK3399

The Rockchip RK3399 is a System-on-Chip with two ARM processor clusters:

• Dual-core Cortex-A72 (big cores)
• Quad-core Cortex-A53 (little cores)

The clusters are connected via ARM Coherent Interconnect (CCI). Both have L1 and L2 caches and support all exception levels (EL0–EL3). The SoC contains a main interconnect and a peripheral interconnect (with low- and high-performance domains), connected through a 128-bit/64-bit/32-bit multi-layer AXI/AHB/APB bus architecture.

Embedded SRAM comes in two units: 8 KB (accessible via the PMU for power management) and 192 KB (accessible via the peripheral bus, protected by TZMA for security). The larger 192 KB block is remapped and accessible at 0xFFFF_0000–0xFFFF_FFFF and is where the bootloader stages execute.

RockPro64 Boot Sequence

RK3399 supports normal and secure boot. This article covers normal boot only.

After power-on reset, the RK3399 Boot ROM starts at 0xFFFF0000. It searches for a valid ID block header at the following locations, in order:

Offset 0          in SPI flash
Offset 0x8000     on eMMC   (sector 64, where a sector is 512 bytes)
Offset 0x8000     on SD card (sector 64)

These are fixed offsets — not partition-table entries. The Boot ROM looks for an ID header at each location and executes from the first one that matches.

Full boot chain using U-Boot:

1. Boot ROM loads U-Boot TPL into SRAM. TPL initializes main system RAM.
2. Control returns from TPL to Boot ROM (return-to-bootrom).
3. Boot ROM loads U-Boot SPL.
4. SPL loads ARM Trusted Firmware (ATF) and U-Boot into main memory.
5. ATF runs U-Boot.
6. U-Boot loads bootaa64.efi from the FAT boot partition (\EFI\BOOT\).
7. The UEFI loader finds and executes the NetBSD kernel.

Glossary for this chain:

• TPL (Tertiary Program Loader): A minimal subset of SPL. Some boards have tight size constraints on early-stage code; TPL handles DDR initialization only, then hands off to SPL.
• SPL (Secondary Program Loader): A small binary generated from U-Boot source that fits in SRAM and loads the main U-Boot into system RAM.
• MLO (Memory Loader): A broader term for any second-stage program that loads the next bootloader into memory. SPL is the U-Boot-specific variant; some boards use their own MLO.
• ATF (ARM Trusted Firmware): The reference implementation for secure boot on ARM. Divides the boot process into stages BL1 (Boot ROM), BL2 (verified by BL1), and BL3 (loads U-Boot/GRUB or similar). Establishes the chain of trust.

A note on return-to-bootrom: this is a Rockchip-specific mechanism. The Boot ROM leaves a function pointer in memory before transferring control to TPL. TPL can call this pointer — roughly equivalent to a longjmp — to return control to the Boot ROM, which then continues to the next stage. If the second-stage loader fails and returns to Boot ROM with the appropriate error code, Boot ROM enters an upgrade mode accessible over USB. In the normal boot path, TPL uses this mechanism to hand off to SPL cleanly.

Disk Layout

U-Boot is flexible and can boot from many sources. For NetBSD on RockPro64 we format the boot medium with an MBR partition table containing two partitions: a FAT32 system partition holding the EFI loader, and a NetBSD partition (ID 0xa9) holding the kernel and root filesystem.

file ./NetBSD-9-aarch64-202101091800Z-pinebook-pro.img:
DOS/MBR boot sector;
partition 1 : ID=0xc, active, start-CHS (0x2,10,9),  end-CHS (0xc,60,48),   startsector 32768, 163840 sectors;
partition 2 : ID=0xa9,        start-CHS (0xc,60,49), end-CHS (0x92,123,41), startsector 196608, 2156672 sectors

The layout of sectors on the medium before the partitions:

Start (sectors) | Size (sectors) | Name                  | Description
----------------|----------------|-----------------------|---------------------------
64              | 16320          | IDBLoader             | SoC init code (TPL + SPL)
16384           | 8192           | OS loader             | Main U-Boot
24576           | 8192           | TrustedFirmware-A     | ARM Trusted Firmware
32768           | 163840         | FAT32 system partition | EFI loader
196608          | 2156672        | NetBSD partition       | FFS filesystem with kernel

The EFI loader (provided in NetBSD sources) finds the kernel file, loads it into RAM, and calls aarch64_exec_kernel. Before transferring control, it invalidates instruction and data caches to ensure the kernel image is coherently placed in RAM, and disables the MMU — the kernel starts with physical addresses and sets up its own address space from scratch.

start.S, locore.S and Machine-Dependent Code

From UEFI to the Kernel

When a multiprocessor SoC powers on, all cores become active after the power-on reset. Early-stage firmware designates one as the boot core and holds the others with a wfi (wait for interrupt) instruction. NetBSD’s early initialization assumes exactly this: one active core, the rest idle.

Execution begins in start.S inside /sys/arch/aarch64. This file is a thin proxy to aarch64_start in locore.S; its main job is extracting boot arguments.

The processor arrives here at EL1 — the EFI loader has already dropped from EL2 to EL1 before jumping to the kernel. This drop is done via the ERET instruction: the EFI loader sets SPSR_EL2 to describe the target PSTATE (EL1h, all interrupts masked), sets ELR_EL2 to the kernel entry address, and executes ERET. The processor atomically switches to EL1 and begins fetching from the kernel entry point. The MMU is off at this point — the EFI loader disabled it before the jump, so the kernel executes physical addresses until it builds its own page tables.

The steps ahead:

1. Initialize system registers
2. Build MMU translation tables
3. Enable the MMU
4. Jump into virtual address space

The PState Register

PState (Processing Element State) holds the processor’s current execution state, including the active exception level. In AArch64, PState is not a single accessible register — it is a collection of fields readable through dedicated system registers (CurrentEL, SPSR_ELx, DAIF, etc.). Each exception level has its own copies of the relevant registers.

The overall sequence in locore.S before main is called:

        /* Disable MMU (ensure clean state) */
        bl      mmu_disable

        /* Initialize system registers and build MMU tables */
        bl      init_sysregs
        bl      init_mmutable
        bl      save_ttbrs

        /* Enable MMU */
        bl      mmu_enable

        /* Load virtual address of vstart into x20 */
        ldr     x20, =vstart

        /* Jump to virtual address space */
        br      x20

/*
 * vstart executes in kernel virtual address space
 */
vstart:

mmu_disable is called first even though the EFI loader already disabled the MMU — it is a defensive measure in case the kernel is entered through a different path (e.g. a kexec-like mechanism or a bootloader that does not follow the EFI handoff convention).

save_ttbrs stores the translation table base registers TTBR0_EL1 and TTBR1_EL1 after they are configured. ARMv8 uses two TTBRs: TTBR0 covers the lower virtual address range (user space, configured per-process), and TTBR1 covers the upper range (kernel space, shared). Saving them at this point lets secondary CPUs reuse the same page tables when they are brought online later.

ARMv8 branch instructions quick reference:

• B label — unconditional branch to a PC-relative 26-bit signed offset
• BL label — branch and link; saves return address in x30
• BR xN — branch to address in register xN (not a subroutine return; use BLR for that)

Initializing System Registers

init_sysregs configures several EL1 system registers before the MMU is enabled:

init_sysregs:
        stp     x0, lr, [sp, #-16]!

        /* Configure debug event register */
        ldr     x0, mdscr_setting
        msr     mdscr_el1, x0

        /* Unlock OS lock (allows external debugger access) */
        msr     oslar_el1, xzr

        /* Clear context ID register */
        msr     contextidr_el1, xzr

        /* Trap FP/SIMD at both EL0 and EL1 until FP context is ready */
        msr     cpacr_el1, xzr

        /* Allow EL0 to read the virtual counter and frequency */
        mrs     x0, cntkctl_el1
        orr     x0, x0, #CNTKCTL_EL0VCTEN
        msr     cntkctl_el1, x0

        /* Unmask all exceptions */
        msr     daif, xzr

        ldp     x0, lr, [sp], #16
        ret

Key registers touched here:

A note on CPACR_EL1: setting it to zero means FPEN = 0b00, which causes any FP or SIMD instruction at either EL0 or EL1 to trap. This is intentional — until the kernel has set up FP register save/restore in its exception handlers, allowing FP instructions would silently corrupt the FP register file across context switches. The kernel re-enables FP access later once the infrastructure is ready.

A note on CONTEXTIDR_EL1: this register holds a process identifier for debug and trace hardware (e.g. ETM). It does not control ASID-based TLB tagging — ASIDs live in bits [63:48] of TTBR0_EL1 and TTBR1_EL1.

Enabling the MMU

Out of reset (and after the EFI loader’s handoff), the MMU is off. The CPU executes physical addresses directly with no access permission enforcement and no memory attribute differentiation. Both instruction and data caches are architecturally disabled after reset (SCTLR_EL1.I = 0, SCTLR_EL1.C = 0). The instruction cache can be enabled independently of the MMU, but enabling the data cache without the MMU requires careful setup — without address translation, the cache cannot enforce memory ordering or access permissions correctly, which makes it unsafe in any general-purpose OS context.

mmu_enable performs the following steps:

mmu_enable:
        dsb     sy

        /* Invalidate all EL1 TLB entries */
        dsb     ishst
        tlbi    vmalle1
        dsb     ish
        isb

        /* Set memory attributes */
        ldr     x0, mair_setting
        msr     mair_el1, x0

        /* Configure translation control; set IPS from hardware capability */
        ldr     x0, tcr_setting
        mrs     x1, id_aa64mmfr0_el1
        bfi     x0, x1, #32, #3
        msr     tcr_el1, x0

        /* Configure and enable via SCTLR_EL1 */
        mrs     x0, sctlr_el1
        ldr     x1, sctlr_clear
        bic     x0, x0, x1
        ldr     x1, sctlr_pac       /* disable PAC */
        bic     x0, x0, x1
        ldr     x1, sctlr_set
        orr     x0, x0, x1

        ldr     x1, sctlr_ee
#ifdef __AARCH64EB__
        orr     x0, x0, x1          /* big-endian */
#else
        bic     x0, x0, x1          /* little-endian */
#endif
        msr     sctlr_el1, x0       /* M bit set — MMU now enabled */
        isb

        ret

Breaking down the key parts:

MAIR_EL1 (Memory Attribute Indirection Register): defines up to 8 memory attribute slots that page table entries reference by index (AttrIndx field). NetBSD sets up five:

mair_setting:
        .quad (                                          \
            __SHIFTIN(MAIR_NORMAL_WB,     MAIR_ATTR0) | \
            __SHIFTIN(MAIR_NORMAL_NC,     MAIR_ATTR1) | \
            __SHIFTIN(MAIR_NORMAL_WT,     MAIR_ATTR2) | \
            __SHIFTIN(MAIR_DEVICE_MEM,    MAIR_ATTR3) | \
            __SHIFTIN(MAIR_DEVICE_MEM_SO, MAIR_ATTR4))

Normal Write-Back, Normal Non-Cacheable, Normal Write-Through, Device memory (Device-nGRE), and MAIR_DEVICE_MEM_SO — the most restrictive device type, corresponding to AArch64 Device-nGnRnE (non-Gathering, non-Reordering, non-Early-write-acknowledgement). This is the AArch64 equivalent of what ARMv7 called “Strongly Ordered.” Page table descriptors point to these slots via their AttrIndx[2:0] field.

TCR_EL1 (Translation Control Register): controls granule size, address space size, and the intermediate physical address size (IPS) at bits [34:32]. The instruction bfi x0, x1, #32, #3 inserts bits [2:0] of ID_AA64MMFR0_EL1.PARange into bits [34:32] of TCR_EL1 — matching the IPS configuration to the physical address range the hardware actually supports. Getting this wrong would cause translation faults on systems with more than 32-bit physical address space.

SCTLR_EL1 (System Control Register): the M bit (bit 0) enables the MMU. The code also clears PAC (Pointer Authentication Code) bits and configures endianness before writing the final value. The isb instruction barrier after the write ensures the pipeline is flushed and the MMU is fully active before any subsequent instruction fetch proceeds.

After mmu_enable returns, the CPU is translating virtual addresses. The br x20 instruction then jumps to vstart, which is a virtual address — this is the moment the kernel fully enters its own virtual address space.

Inside Virtual Memory

Once the MMU is enabled and execution is in virtual address space, vstart completes the remaining initialization before calling main:

• Exception Vector — installs the kernel exception vector table by writing to VBAR_EL1
• Process 0 stack — sets up the initial kernel stack for the idle/init process
• PAC setup — enables Pointer Authentication if ID_AA64ISAR1_EL1 indicates support
• CPU topology — arm_cpu_topology_set reads MPIDR_EL1 to determine cluster and core IDs
• Cache info — aarch64_getcacheinfo reads CLIDR_EL1 and CCSIDR_EL1 to discover cache geometry (levels, associativity, line size)
• initarm — machine-dependent initialization: GIC interrupt controller, clocks, device tree parsing, early console
• main — the C entry point for the rest of kernel initialization

Summary

This part covered the ARMv8 boot process from reset to the kernel’s C entry point, using NetBSD on the RK3399-based ROCKPro64 as a concrete example.

The key structural difference from AMD64 is that ARMv8 provides a formal privilege hierarchy (exception levels) from the start, and the boot chain reflects this: BootROM runs at EL3, ATF establishes secure world services, UEFI runs at EL2, and the OS kernel settles into EL1. The transition between each level is explicit — done via ERET with carefully prepared SPSR and ELR registers. On AMD64, privilege rings exist but the boot chain does not engage them until the OS sets them up.

The other major difference is the flexibility — and the resulting complexity. Rockchip’s return-to-bootrom mechanism, the TPL/SPL split, the fixed-offset sector layout: none of this is mandated by the ARM architecture. Each vendor solves these problems differently, which is both ARM’s strength and the main reason ARM boot sequences are notoriously hard to document generically.

In Part Three we will look at ARM Secure Boot and ATF in more detail, and after that compare both architectures in the context of UEFI.

Resources

1. Embedded Base Boot Requirements (EBBR) Specification
2. Server Base Boot Requirements (SBBR) v1.2
3. longjmp man page
4. Rockchip RK3399 Technical Reference Manual
5. ARM Architecture Reference Manual — ARMv8
6. ARM Trusted Firmware documentation
7. NetBSD source code

Then Gen 6 started showing up everywhere — conferences, hardware announcements, specs dropping into the mailbox. Flit mode is not a minor revision; it is a different framing model entirely, and the TLP format that goes inside it was reworked to match. I had to spend a real amount of time working through the spec before I understood what had actually changed and what had stayed the same. The PCIe 6.0 spec calls the relevant section “TLP Format Revisited.” That name carries more weight than it looks.

The other thing that slows you down when you return to old code is the old code itself. I remembered writing some of it under time pressure and taking shortcuts that made sense at the time. Seeing them again — as someone who had to then build on top of them — was educational in the way that finding your own bugs always is. So the API rename pass in this release is partly Gen 6 groundwork and partly old debts getting paid.

Both tracks took longer than expected. The release is the result.

If you have been using either library for Gen 1–5 parsing, nothing changes — the existing paths are untouched. What follows is what is new.

Flit Mode: What Actually Changed

The previous post on parsing TLPs covers the traditional format: DW0 encodes FMT[2:0] + TYPE[4:0], the rest of the header follows a layout determined by that combination, and parsing is a matter of reading the right bits from the right dwords.

PCIe 6.0 changes the framing entirely. To support PAM4 signaling at 64 GT/s, the spec introduced flits — fixed 256-byte containers with their own forward error correction and CRC. TLPs are no longer delimited by STP/END sequences at the data link layer; they are packed into flits. The TLP format itself was revised to match: DW0 in flit mode carries a different set of fields, there are thirteen type codes rather than the FMT+TYPE matrix of earlier generations, and Optional Header Components (OHC) replace TLP prefixes as the extension mechanism.

None of this is obvious from a quick read. The spec’s framing of it as “revisited” undersells how much the mental model needs to shift.

Library Changes

The new types in rust_tlplib:

• FlitDW0 — flit-mode DW0 with the Gen 6 field layout
• FlitTlpType — the thirteen flit type codes (NOPs, memory, atomics, messages)
• FlitOhcA — Optional Header Component type A, the primary OHC variant
• FlitStreamWalker — walks a byte stream extracting TLPs packed inside a flit

Mode dispatch is now on TlpPacket directly:

match packet.mode() {
    TlpMode::Standard(tlp) => { /* Gen 1–5 handling */ }
    TlpMode::Flit(flit)    => { /* Gen 6 handling */   }
}

The release also includes the API rename pass mentioned above. Existing get_* accessors are deprecated in favor of canonical names — the library ships a migration table in the release notes, and the old names still compile with deprecation warnings for now. 212 tests pass on Rust 1.85.

rtlp-tool

The command-line tool gains a --flit flag:

$ rtlp-tool --flit -i "03 00 00 01 01 00 0A FF AB CD 12 34"

All output modes — table, JSON, CSV — support the flag. JSON output now includes "flit_mode": true to make it unambiguous which parser was used.

This release also cleans up several bugs that had been sitting in the non-flit path: a DW0 byte-slice extraction reading from the wrong offset, a crash on odd-nibble hex input, missing mutual exclusion between --aer and --lspci, and JSON output that was generating malformed escapes for certain field values.

Installing

cargo install rtlp_tool

Binaries for Linux (x86_64, aarch64), macOS (Intel and Apple Silicon), Windows, and FreeBSD are on the release page. Debian/Ubuntu .deb and RPM packages are also available if you would rather not go through Cargo.

The book stuck with me, though. It is a collection of firsthand accounts from the people who built the systems that now run most of the software industry, and the recurring theme is the same across a dozen voices: the transformer architecture arrived, and then everything else followed from scale.

That is what pulled me back to the challenge once I had a hotel and a power outlet. Not because I do this for a living — machine learning is not my area — but because the underlying question is squarely in territory I care about: what does this look like in hardware, and why does it work the way it does?

A previous post covered the GPT-2 decoder block at a conceptual level: what the LetGPU challenge is asking for, how data flows through the block, what each parameter group does, and why the structure is what it is. This post assumes that context — either you have read it, or you already know GPT-2 well enough that you do not need it. Here we go deeper: the actual CUDA implementation, kernel by kernel, with the concrete hardware trade-offs that make each design decision interesting. The code implements a single transformer block at GPT-2 small scale. No PyTorch, no cuBLAS — just kernels.

The Block, Concretely

The implementation is GPT-2 small: hidden dimension D=768, twelve attention heads (H=12), head dimension Dh=64, feed-forward dimension FF=3072. One forward pass through a block takes a token sequence of shape (seq_len, 768) and produces the same shape.

This is a Pre-LN architecture — layer norm applied before each sublayer, not after. The original paper used Post-LN (norm after the residual add). The difference matters: Pre-LN feeds normalized inputs into the attention and FFN, which keeps gradient flow cleaner in deeper models. GPT-2 and most subsequent work moved to Pre-LN; the original formulation is now mostly a historical artifact.

The ten steps of the forward pass:

x  ──►  LN1  ──►  QKV proj  ──►  MHA  ──►  O proj  ──►  add(x, ·)  ──►  x1
x1 ──►  LN2  ──►  FC proj   ──►  GELU  ──►  out proj  ──►  add(x1, ·)  ──►  output

In kernel terms:

1. layernorm768_kernel(x) → ln1
2. matmul_bias_tiled(ln1, W_qkv) → qkv — shape (seq_len, 2304), packing Q, K, V
3. mha_nocausal_kernel(qkv) → attn — shape (seq_len, 768)
4. matmul_bias_tiled(attn, W_attn) → proj
5. add_kernel(x, proj) → x1 — first residual
6. layernorm768_kernel(x1) → ln2
7. matmul_bias_tiled(ln2, W_fc) → ff1 — shape (seq_len, 3072)
8. gelu_kernel(ff1) — in-place
9. matmul_bias_tiled(ff1, W_proj) → ff2 — shape (seq_len, 768)
10. add_kernel(x1, ff2) → output — second residual

The rest of this post walks each step in turn.

Block Reduction

Almost every kernel needs a single value derived from all threads in a block — a sum for layer norm, a maximum for softmax numerical stability. The standard tool is a tree reduction over shared memory:

__device__ __forceinline__ float block_reduce_sum(float v) {
    __shared__ float smem[1024];
    int tid = threadIdx.x;
    smem[tid] = v;
    __syncthreads();
    for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
        if (tid < stride) smem[tid] += smem[tid + stride];
        __syncthreads();
    }
    return smem[0];
}

Each thread writes its local value to shared memory, then the loop halves the active thread count on each iteration: threads 0–511 add from threads 512–1023, then 0–255 add from 256–511, and so on. After log₂(blockDim.x) iterations, smem[0] holds the total. Every thread in the block reads that result.

The cost is the __syncthreads() calls — a block-wide barrier that stalls until every thread in the block reaches it. Here there are log₂(256) = eight barriers per reduction, and layer norm alone runs two reductions. Not free, but unavoidable: the alternative is letting threads read values their neighbors have not written yet.

block_reduce_max is identical with fmaxf in place of addition.

Layer Normalization

One block per token. All threads in the block cooperate on a single 768-element normalization:

// Pass 1: mean
float sum = 0.0f;
for (int i = threadIdx.x; i < 768; i += blockDim.x) sum += x[i];
sum = block_reduce_sum(sum);
float mean = sum / 768.0f;

// Pass 2: variance, then normalize
float vsum = 0.0f;
for (int i = threadIdx.x; i < 768; i += blockDim.x) {
    float d = x[i] - mean; vsum += d * d;
}
vsum = block_reduce_sum(vsum);
float inv_std = rsqrtf(vsum / 768.0f + 1e-5f);

for (int i = threadIdx.x; i < 768; i += blockDim.x)
    y[i] = (x[i] - mean) * inv_std * gamma[i] + beta[i];

rsqrtf is the reciprocal square root — a single hardware instruction on any modern GPU, cheaper than computing sqrtf and then dividing. The epsilon 1e-5f keeps the denominator away from zero. gamma and beta are the learned affine parameters applied after normalization.

Tiled Matrix Multiply

The projection steps — QKV, output, FFN up and down — all go through one templated kernel. The core idea is tiling: loading a 16×16 submatrix of A and a 16×16 submatrix of B into shared memory before computing their product, so the 256 threads in the block see each value once from slow global memory and reuse it sixteen times from fast shared memory.

template<int TILE>
__global__ void matmul_bias_tiled(const float* __restrict__ A,
                                 const float* __restrict__ B,
                                 const float* __restrict__ bias,
                                 float* __restrict__ C,
                                 int M, int K, int N) {
    __shared__ float As[TILE][TILE];
    __shared__ float Bs[TILE][TILE];
    float acc = 0.0f;

    for (int k0 = 0; k0 < K; k0 += TILE) {
        As[threadIdx.y][threadIdx.x] = /* A[row, k0+threadIdx.x] */;
        Bs[threadIdx.y][threadIdx.x] = /* B[k0+threadIdx.y, col] */;
        __syncthreads();

        #pragma unroll
        for (int k = 0; k < TILE; ++k)
            acc += As[threadIdx.y][k] * Bs[k][threadIdx.x];
        __syncthreads();
    }
    C[row * N + col] = acc + bias[col];
}

The thread at (threadIdx.y, threadIdx.x) is responsible for one output element at (row, col). It accumulates the dot product of a row of A with a column of B, advancing through K in strides of TILE=16. The two __syncthreads() per iteration are load-fence and store-fence: the first ensures all threads have finished loading into shared memory before any thread starts reading from it; the second ensures the compute is done before the next iteration overwrites the tiles.

#pragma unroll on the inner loop tells the compiler to unroll the 16 iterations at compile time, generating 16 independent FMAs rather than a loop with a branch and counter update on each pass.

__restrict__ on the pointer arguments is a promise to the compiler that A, B, and C do not alias — the compiler can assume no write to C affects what is read from A or B, enabling more aggressive load scheduling.

Multi-Head Attention

This is the interesting one. One CUDA block per (token, head) pair — the launch grid is (seq_len, 12). Each block computes one row of the attention output for one head.

The attention score for query token t against key token s in head h is:

score(t, s) = dot(Q_t_h, K_s_h) / sqrt(64)

Then softmax across all s, then weighted sum of the value vectors V_s_h.

Softmax has a numerical problem: if scores are large, exp(score) overflows. The standard fix is to subtract the maximum before exponentiating. That requires knowing the maximum, which requires a pass over all scores. So the kernel runs three passes:

// Pass 1: compute raw scores, find max
float local_max = -1e30f;
for (int s = threadIdx.x; s < seq_len; s += blockDim.x) {
    // compute dot(Q[t,h], K[s,h]) and scale
    scores[s] = dot * inv_sqrt_dh;
    local_max = fmaxf(local_max, scores[s]);
}
float max_sc = block_reduce_max(local_max);

// Pass 2: exp(score - max), accumulate sum
float local_sum = 0.0f;
for (int s = threadIdx.x; s < seq_len; s += blockDim.x) {
    float e = expf(scores[s] - max_sc);
    scores[s] = e;         // reuse the shared buffer
    local_sum += e;
}
float inv_sum = 1.0f / block_reduce_sum(local_sum);

// Pass 3: weighted sum of V (first 64 threads only)
if (threadIdx.x < Dh) {
    float acc = 0.0f;
    for (int s = 0; s < seq_len; ++s)
        acc += scores[s] * inv_sum * V[s][h][threadIdx.x];
    attn_out[t * D + h * Dh + threadIdx.x] = acc;
}

The scores buffer lives in dynamic shared memory (extern __shared__), allocated at launch time as seq_len * sizeof(float). The size is not known at compile time; the kernel receives it as the third argument to the <<<>>> launch syntax: mha_nocausal_kernel<<<grid, block, seq_len * sizeof(float)>>>. Note that shared memory is limited per block (48 KB on most devices, up to 96 KB with explicit opt-in). For large seq_len this allocation will exceed the limit — the implementation as written is suitable for the GPT-2 Small context lengths (up to 1024 tokens → 4 KB), not for arbitrarily long sequences.

The QKV layout is interleaved by position: each token’s Q, K, and V are contiguous. The head dimension is packed within each section. Pointer arithmetic for Q at token t, head h is: qkv + t * 2304 + 0 * 768 + h * 64, and similarly for K (offset 1*768) and V (offset 2*768).

Pass 3 only uses the first 64 threads (one per head dimension). The remaining threads in the block are idle for that phase. This is a deliberate simplicity trade-off — for a challenge implementation it is fine; production kernels would use the idle threads to pipeline across heads or fuse operations.

GELU

The feed-forward sublayer uses GELU as its activation function. The true definition involves the error function erf, which is expensive. The standard approximation:

static __device__ __forceinline__ float gelu_tanh(float x) {
    const float k = 0.7978845608028654f; // sqrt(2/pi)
    float x3 = x * x * x;
    return 0.5f * x * (1.0f + tanhf(k * (x + 0.044715f * x3)));
}

This is a Padé approximant to erf(x/sqrt(2)). The maximum absolute error relative to the true GELU is small enough to be irrelevant for inference (on the order of 10⁻⁴ in the active range — the exact bound depends on the measurement interval, but PyTorch uses this same approximation in production). tanhf is hardware-accelerated on every CUDA-capable device; the full expression is five floating-point operations. The kernel applies it elementwise across the (seq_len, 3072) intermediate tensor.

Putting It Together

The solve function allocates eight intermediate tensors on the device, launches the kernels in sequence, and frees them. The weight layout is a flat device buffer. Offsets are computed from the architecture constants:

About 7.1 million parameters per block. GPT-2 small stacks twelve of them on top of embeddings — 85 million parameters total. The numbers in The Scaling Era that stuck with me were the orders of magnitude larger: GPT-3 at 175 billion, models that followed at trillions. The architecture is the same. The arithmetic just runs longer.

The implementation here is not production code. It is not fused, not tuned for tensor cores, and does three passes through memory where a flash attention implementation would do one. But the patterns — shared memory tiling, tree reductions, online softmax — are the same patterns production kernels use, without the extra complexity that comes from optimizing for throughput at scale.

Sometimes it helps to see the skeleton before studying the muscle.

Full Code

#include <cuda_runtime.h>
#include <math_constants.h>
#include <cmath>

#define CHECK_CUDA(x) do { cudaError_t err = (x); if (err != cudaSuccess) return; } while(0)

static __device__ __forceinline__ float gelu_tanh(float x) {
    // GELU(x) = 0.5*x*(1 + tanh(sqrt(2/pi)*(x + 0.044715*x^3)))
    const float k = 0.7978845608028654f; // sqrt(2/pi)
    float x3 = x * x * x;
    return 0.5f * x * (1.0f + tanhf(k * (x + 0.044715f * x3)));
}

// ---------------------------
// Reductions (block-wide)
// ---------------------------
__device__ __forceinline__ float block_reduce_sum(float v) {
    // assumes blockDim.x <= 1024
    __shared__ float smem[1024];
    int tid = threadIdx.x;
    smem[tid] = v;
    __syncthreads();
    for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
        if (tid < stride) smem[tid] += smem[tid + stride];
        __syncthreads();
    }
    return smem[0];
}

__device__ __forceinline__ float block_reduce_max(float v) {
    __shared__ float smem[1024];
    int tid = threadIdx.x;
    smem[tid] = v;
    __syncthreads();
    for (int stride = blockDim.x / 2; stride > 0; stride >>= 1) {
        if (tid < stride) smem[tid] = fmaxf(smem[tid], smem[tid + stride]);
        __syncthreads();
    }
    return smem[0];
}

// ---------------------------
// LayerNorm over 768 features
// One block per token
// ---------------------------
__global__ void layernorm768_kernel(const float* __restrict__ in,
                                   float* __restrict__ out,
                                   const float* __restrict__ gamma,
                                   const float* __restrict__ beta,
                                   int seq_len) {
    int t = blockIdx.x;
    if (t >= seq_len) return;

    const float* x = in + t * 768;
    float* y = out + t * 768;

    // sum
    float sum = 0.0f;
    for (int i = threadIdx.x; i < 768; i += blockDim.x) sum += x[i];
    sum = block_reduce_sum(sum);
    float mean = sum / 768.0f;

    // var
    float vsum = 0.0f;
    for (int i = threadIdx.x; i < 768; i += blockDim.x) {
        float d = x[i] - mean;
        vsum += d * d;
    }
    vsum = block_reduce_sum(vsum);
    float var = vsum / 768.0f;
    float inv_std = rsqrtf(var + 1e-5f);

    for (int i = threadIdx.x; i < 768; i += blockDim.x) {
        float n = (x[i] - mean) * inv_std;
        y[i] = n * gamma[i] + beta[i];
    }
}

// ---------------------------
// Simple tiled GEMM (row-major)
// C = A(M,K) * B(K,N) + bias(N)
// TILE=16; assumes N,K multiples of 16 for best perf (true for your shapes)
// ---------------------------
template<int TILE>
__global__ void matmul_bias_tiled(const float* __restrict__ A,
                                 const float* __restrict__ B,
                                 const float* __restrict__ bias,
                                 float* __restrict__ C,
                                 int M, int K, int N) {
    int row = blockIdx.y * TILE + threadIdx.y;
    int col = blockIdx.x * TILE + threadIdx.x;

    __shared__ float As[TILE][TILE];
    __shared__ float Bs[TILE][TILE];

    float acc = 0.0f;

    for (int k0 = 0; k0 < K; k0 += TILE) {
        float a = 0.0f;
        float b = 0.0f;

        if (row < M && (k0 + threadIdx.x) < K)
            a = A[row * K + (k0 + threadIdx.x)];
        if (col < N && (k0 + threadIdx.y) < K)
            b = B[(k0 + threadIdx.y) * N + col];

        As[threadIdx.y][threadIdx.x] = a;
        Bs[threadIdx.y][threadIdx.x] = b;
        __syncthreads();

        #pragma unroll
        for (int k = 0; k < TILE; ++k)
            acc += As[threadIdx.y][k] * Bs[k][threadIdx.x];

        __syncthreads();
    }

    if (row < M && col < N) {
        acc += (bias ? bias[col] : 0.0f);
        C[row * N + col] = acc;
    }
}

// ---------------------------
// Add residual: y = a + b
// ---------------------------
__global__ void add_kernel(const float* __restrict__ a,
                           const float* __restrict__ b,
                           float* __restrict__ y,
                           int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) y[i] = a[i] + b[i];
}

// ---------------------------
// GELU elementwise
// ---------------------------
__global__ void gelu_kernel(float* __restrict__ x, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) x[i] = gelu_tanh(x[i]);
}

// ---------------------------
// Attention kernel:
// One block per (token t, head h)
// Input: qkv [seq, 2304] = [Q|K|V] each 768
// Output: attn_out [seq, 768] (concatenated heads)
// No causal mask
//
// Shared memory: scores[seq_len] floats
// blockDim.x recommended 256
// ---------------------------
__global__ void mha_nocausal_kernel(const float* __restrict__ qkv,
                                   float* __restrict__ attn_out,
                                   int seq_len) {
    int t = blockIdx.x;      // query token
    int h = blockIdx.y;      // head 0..11
    if (t >= seq_len || h >= 12) return;

    extern __shared__ float shmem[]; // size >= seq_len floats
    float* scores = shmem;

    const int D = 768;
    const int Dh = 64;
    const float inv_sqrt_dh = 1.0f / sqrtf((float)Dh);

    // Q pointer for this (t,h)
    const float* Q = qkv + t * (3 * D) + 0 * D + h * Dh;

    // Pass 1: compute scores[s] = dot(Q, K_s) / sqrt(Dh)
    float local_max = -1e30f;
    for (int s = threadIdx.x; s < seq_len; s += blockDim.x) {
        const float* K = qkv + s * (3 * D) + 1 * D + h * Dh;
        float dot = 0.0f;
        #pragma unroll
        for (int i = 0; i < Dh; ++i) dot += Q[i] * K[i];
        float sc = dot * inv_sqrt_dh;
        scores[s] = sc;
        local_max = fmaxf(local_max, sc);
    }
    float max_sc = block_reduce_max(local_max);
    __syncthreads();

    // Pass 2: exp(scores - max), sum
    float local_sum = 0.0f;
    for (int s = threadIdx.x; s < seq_len; s += blockDim.x) {
        float e = expf(scores[s] - max_sc);
        scores[s] = e;
        local_sum += e;
    }
    float sum_sc = block_reduce_sum(local_sum);
    float inv_sum = 1.0f / sum_sc;
    __syncthreads();

    // Pass 3: weighted sum of V
    if (threadIdx.x < Dh) {
        int i = threadIdx.x;
        float acc = 0.0f;
        for (int s = 0; s < seq_len; ++s) {
            float p = scores[s] * inv_sum;
            const float* V = qkv + s * (3 * D) + 2 * D + h * Dh;
            acc += p * V[i];
        }
        attn_out[t * D + h * Dh + i] = acc;
    }
}

// ---------------------------
// Entry point (device pointers)
// x:      (seq_len, 768)
// output: (seq_len, 768)
// weights packed per your offsets
// ---------------------------
extern "C" void solve(const float* x, float* output, const float* weights, int seq_len) {
    const int D = 768;
    const int FF = 3072;
    const int H = 12;

    const float* gamma1 = weights + 0;
    const float* beta1  = weights + 768;

    const float* W_qkv  = weights + 1536;
    const float* b_qkv  = weights + 1771008;

    const float* W_attn = weights + 1773312;
    const float* b_attn = weights + 2363136;

    const float* gamma2 = weights + 2363904;
    const float* beta2  = weights + 2364672;

    const float* W_fc   = weights + 2365440;
    const float* b_fc   = weights + 4724736;

    const float* W_proj = weights + 4727808;
    const float* b_proj = weights + 7087104;

    float *ln1 = nullptr, *qkv = nullptr, *attn = nullptr, *proj = nullptr, *x1 = nullptr;
    float *ln2 = nullptr, *ff1 = nullptr, *ff2 = nullptr;

    size_t bytes_ln  = (size_t)seq_len * D  * sizeof(float);
    size_t bytes_qkv = (size_t)seq_len * (3 * D) * sizeof(float);
    size_t bytes_ff1 = (size_t)seq_len * FF * sizeof(float);

    CHECK_CUDA(cudaMalloc(&ln1,  bytes_ln));
    CHECK_CUDA(cudaMalloc(&qkv,  bytes_qkv));
    CHECK_CUDA(cudaMalloc(&attn, bytes_ln));
    CHECK_CUDA(cudaMalloc(&proj, bytes_ln));
    CHECK_CUDA(cudaMalloc(&x1,   bytes_ln));
    CHECK_CUDA(cudaMalloc(&ln2,  bytes_ln));
    CHECK_CUDA(cudaMalloc(&ff1,  bytes_ff1));
    CHECK_CUDA(cudaMalloc(&ff2,  bytes_ln));

    // 1) LN1
    layernorm768_kernel<<<seq_len, 256>>>(x, ln1, gamma1, beta1, seq_len);

    // 2) QKV = ln1 * W_qkv + b_qkv   (M=seq_len, K=768, N=2304)
    {
        const int TILE = 16;
        dim3 block(TILE, TILE);
        dim3 grid((2304 + TILE-1)/TILE, (seq_len + TILE-1)/TILE);
        matmul_bias_tiled<TILE><<<grid, block>>>(ln1, W_qkv, b_qkv, qkv, seq_len, 768, 2304);
    }

    // 3) MHA: attn = (seq_len, 768)
    {
        dim3 grid(seq_len, H);
        size_t shmem = (size_t)seq_len * sizeof(float);
        mha_nocausal_kernel<<<grid, 256, shmem>>>(qkv, attn, seq_len);
    }

    // 4) proj = attn * W_attn + b_attn
    {
        const int TILE = 16;
        dim3 block(TILE, TILE);
        dim3 grid((768 + TILE-1)/TILE, (seq_len + TILE-1)/TILE);
        matmul_bias_tiled<TILE><<<grid, block>>>(attn, W_attn, b_attn, proj, seq_len, 768, 768);
    }

    // 5) x1 = x + proj
    {
        int n = seq_len * D;
        add_kernel<<<(n+255)/256, 256>>>(x, proj, x1, n);
    }

    // 6) LN2
    layernorm768_kernel<<<seq_len, 256>>>(x1, ln2, gamma2, beta2, seq_len);

    // 7) ff1 = ln2 * W_fc + b_fc   (M=seq_len, K=768, N=3072)
    {
        const int TILE = 16;
        dim3 block(TILE, TILE);
        dim3 grid((3072 + TILE-1)/TILE, (seq_len + TILE-1)/TILE);
        matmul_bias_tiled<TILE><<<grid, block>>>(ln2, W_fc, b_fc, ff1, seq_len, 768, 3072);
    }

    // 8) GELU(ff1)
    {
        int n = seq_len * FF;
        gelu_kernel<<<(n+255)/256, 256>>>(ff1, n);
    }

    // 9) ff2 = ff1 * W_proj + b_proj  (M=seq_len, K=3072, N=768)
    {
        const int TILE = 16;
        dim3 block(TILE, TILE);
        dim3 grid((768 + TILE-1)/TILE, (seq_len + TILE-1)/TILE);
        matmul_bias_tiled<TILE><<<grid, block>>>(ff1, W_proj, b_proj, ff2, seq_len, 3072, 768);
    }

    // 10) output = x1 + ff2
    {
        int n = seq_len * D;
        add_kernel<<<(n+255)/256, 256>>>(x1, ff2, output, n);
    }

    cudaFree(ln1); cudaFree(qkv);  cudaFree(attn); cudaFree(proj);
    cudaFree(x1);  cudaFree(ln2);  cudaFree(ff1);  cudaFree(ff2);
}

Then around 2017 and 2018 transformers arrived, and the field reorganized itself around them. Not a replacement for neural networks — a different way to structure them, with a mechanism for sequence modeling that turned out to scale in a way nothing before it had.

When LetGPU published a challenge to implement a GPT-2 Small decoder block, the old interest surfaced immediately. Before writing a single line of code I wanted to understand what the block actually is, how data flows through it, and what the challenge is really asking for. This post is that understanding. The implementation — in CUDA, on the December trip to Tokyo — comes separately.

What a Transformer Block Does

At a high level, a neural network is a system that repeatedly transforms a representation into a better one. In language models the representation is a vector per token. Each block takes those vectors, processes them, and passes a refined version forward.

The key difference from older sequence models is attention. RNNs process tokens one at a time and carry state forward; long-range dependencies get compressed into whatever the hidden state can hold. Transformers let every token look at every other token directly and decide which ones matter for its current representation. That is not a minor optimization — it changes what the model can learn and how fast it can train.

Figure 1. A simplified transformer block. Attention lets tokens exchange information; the feed-forward network refines each representation independently. Residual connections add the original signal back after each sublayer; LayerNorm keeps the values numerically stable.

The block does two jobs in sequence. First, attention: tokens share context with each other. Second, a feed-forward network: each token’s representation is deepened independently. Residual connections run around both, so the block learns updates rather than replacements. The output shape matches the input — same number of tokens, same hidden width.

The LetGPU Challenge

The task is to implement one GPT-2 Small decoder block. You are given an input tensor x of shape (seq_len, 768) and a packed flat buffer containing all parameters for the block. Compute the output — same shape as the input.

That sounds compact. It is not. One block contains:

• two LayerNorms (each with learned scale and bias)
• a combined QKV projection (768 → 2304)
• 12-head self-attention
• an output projection (768 → 768)
• two residual connections
• a feed-forward network: 768 → 3072 → GELU → 768

About 7.1 million parameters in one block. GPT-2 Small stacks twelve of them.

Figure 2. The GPT-2 Small decoder block from the LetGPU challenge. Pre-norm: LayerNorm runs before attention and before the feed-forward network. Hidden size 768, twelve heads of 64 dimensions each, feed-forward expansion to 3072.

Pre-Norm and the Flow

GPT-2 uses pre-norm: LayerNorm is applied before each sublayer, not after. The original Attention Is All You Need paper used post-norm (normalize after the residual add). The difference matters for training stability in deep models and is worth knowing before reading the weight layout.

The full sequence through the block:

x → LN1 → QKV projection → multi-head attention → output projection → add(x, ·) → x1
x1 → LN2 → FC projection → GELU → output projection → add(x1, ·) → output

Ten operations, two residual paths, one block.

Attention: Q, K, V

Each token is projected into three vectors — Query, Key, Value — via a single combined weight matrix that is then split. The intuition that actually holds up: the Query describes what this token is looking for, the Key describes what it offers, and the Value is what gets passed along if the match is strong.

Attention weights come from comparing queries against keys (scaled by 1/√64 to keep the dot-product variance at 1 regardless of dimension — without this, the softmax inputs grow with d_k and push gradients into near-zero saturation regions), then those weights determine how values are mixed. Each token ends up with a weighted sum of values from the entire sequence — its representation updated by whatever context the model has learned to attend to.

Multi-head attention runs this in parallel across twelve subspaces of 64 dimensions each. 12 × 64 = 768. Each head can specialize in a different kind of dependency; the results are concatenated and projected back to 768.

Feed-Forward: Refine Independently

After attention, the feed-forward network processes each token on its own — no cross-token interaction here. It expands from 768 to 3072, applies GELU, and projects back to 768. The expansion gives the model room to build richer feature combinations before compressing back down. GELU acts as a smooth gate: it suppresses near-zero activations continuously rather than with a hard threshold, which is why it works better than ReLU for this use case.

The Packed Weight Buffer

The parameters are not handed over as named tensors. They arrive as one flat device buffer and you are expected to know the layout:

Working out the byte offsets is part of the exercise. It forces you to understand the model structure at the level of actual memory, not just diagram boxes.

That is what I found interesting about the challenge. The transformer block is not complicated once you stop treating it as a black box. Attention shares context; the feed-forward network deepens each representation; residuals and LayerNorm keep the whole thing trainable. The packed buffer just makes you prove you understood the structure before the hardware will let you run it.

The CUDA implementation is in the follow-up post.