Bootloaders & QMK

Here are some important things I was unfortunate enough to have to learn about bootloaders during my stay in firmware hell. Had I learned these earlier, I could have averted many days of weeping forlornly into a rat’s nest of jumper cables and toolchain manuals.

I initially made the fatal mistake of assuming that the .hex files I was compiling with QMK should all be written directly with avrdude for my purposes. This is not the case.

Bootloader basics

• Bootloaders are not structurally separate from other firmware. A bootloader is simply a piece of firmware that knows how to write firmware.
• Bootloaders and “other firmware” are compiled into a single binary, and all written to the same memory; there is no special section of memory reserved for bootloaders.
• Firmware does not always contain a bootloader, since not all chips need to be rewritten. This is one of those hilarious things that’s only obvious to a firmware newb in retrospect.

ISP basics

• Unlike high-level toolchains such as QMK and Arduino that have semantic safety checks built in, ISP programming is extremely low-level and does whatever you tell it to, even if that means bricking your chip.
• If you ISP reflash a chip with a .hex file that does not contain a bootloader, then it will happily overwrite all of your existing firmware, including your existing bootloader.
• Remember, the ISP programmer is not opinionated as to whether or not your chip should have a bootloader, nor does it distinguish between “bootloader” and “not bootloader”; it simply wipes all your firmware and replaces it with the new .hex file.

Flashing with QMK toolchain

• On the other hand, QMK writes firmware through the bootloader in such a way that the bootloader (theoretically) doesn’t delete itself. This is extremely convenient, but also obscures a bit about how firmware works.
• QMK can compile you a “production-ready” .hex file, which also contains a bootloader. Production files may be written directly through an ISP programmer.
• If you qmk flash normally, without the :production target, it will compile a .hex file that does not contain a bootloader.
• However, since it writes this file through the existing bootloader, it should work fine and not overwrite an existing working bootloader, usually.

In summary, just because a .hex file can be flashed through QMK and the firmware works doesn’t mean that .hex file is safe to flash directly with an ISP programmer.

Excuse me, “theoretically” doesn’t delete itself??

According to QMK contributors, there are a lot of ways for the Caterina bootloader to accidentally brick itself. One of the more common problems is that if you try to send it a .hex file that is larger than the available space, it will just overwrite itself.

Despite these massive flaws, Caterina is frequently encountered and shipped with Pro Micros because it does not require special drivers on Windows, while DFU does.

The DFU bootloader has better safeguards against this kind of accidental overwrite.