Does it run DOOM?
It’s becoming the defacto question for any piece of technology with a display on it.. fridges, printers, all sorts of things can run DOOM.
So why not this shiny new ESP32S3 based electronic badge I have just received from ElectroMagnetic Field 2022? It has a decent dual core 32bit Xtensa MCU, 8MB of RAM, 8MB of flash storage (partitioned up - see below) and an ST7789 colour pixel display.
Oh, and a joystick :)
Similar work & self-imposed constraints
There is a port of DOOM for an ESP32, proving that it can be done, however I have added additional constraints on myself as follows:
- Co-exist with the Micropython environment that comes pre-installed on the badge.
- Create an app that can be distributed from the TiDAL Hatchery
So what does the micropython runtime environment look like?
- It’s based on the Espressif IoT Development Framework (IDF), which defines a storage partitioning scheme that I cannot replace (but I can use to my advantage..)
- ESP-IDF, and thus Micropython are not designed as an OS with runtime application loading/unloading in mind..
- Micropython is extensible at runtime, via pre-built
.mpyformat modules, which can be compiled from
..and the Hatchery format?
- A folder named after the app (eg:
/apps/Doom), that must be a python module (ie: contain an
__init__.pyfile), plus any other files required to run.
__init__.pymust define an instance of an
Appclass (or subclass) that the badge shell can locate / load.
Cunning plan #1, and failure.
How hard can it be to simply compile up DOOM as an
.mpy extension, patch in the I/O (display, files, user-input)
back to Micropython and load it up?
It turns out the Micropython extension format is very homebrew (for perfectly good reasons), and comes with
significant limitations compared to an off-the-shelf application format such as
COFF, in particular:
- No declared data, there is no support for loading a
.datasegment as part of a module.
- No static global variables (you know, the bedrock of most C programs!) as it cannot re-locate references when loading into memory, only visible globals can be used.
I also had to choose a starting version of DOOM, with an emphasis on portability. This turned out to be the easy bit, enter DOOM generic, which happily compiled and ran first time on my Linux desktop ;)
Oh, and of course I had to write a new scaler for the video to reduce 320x200 => 240x135.
So, first remove all declared data from the DOOM sources, by moving initialisation into suitable
if they are available, or adding them if not. ~2000 lines of code (LoC) changed, a number of ugly corner cases simply
removed (like being able to change key bindings) and it runs on the desktop again - phew!
Now remove that
static keyword from all the global variables, and prefix them all with a local module abbreviation
to avoid collisions. Another ~2000 LoC changed, and it still runs on the desktop - double phew since this took several days!
Chuck all those changes into a large
git patch file, and save it away. Done.
Compile everything according to the Micropython supplied tooling. Discover it cannot link certain object files - grrr :(
Chuck all the Micropython bug fixes into a
git patch file and save that away. Done now?
Errr nope. It turns out Micropython makes more assumptions about how code is compiled, in particular that all things are relocatable in memory via a single Global Offset Table (GOT). DOOM makes different assumptions, and repeatedly crashes in unpredictable locations due to memory corruption.
After a few days trying to debug this fiasco (with
printf of course since I cannot use a debugger due to proprietary
module formatting and a special loader), I give up.
Cunning plan #2, and success (eventually!)
Back to basics
Having dismally failed to run anything so far, I decide to start at the other end, building up from a Hello Mum!
program that relies only on the built-in ESP-ROM (and is compiled without the Micropython “special tooling” that
is causing me problems), designed to run as the first user-provided code on the device, in ESP-IDF parlance,
as the “secondary boot loader”. Thus is born
phlashboot, the shortest program I can write that says ‘Hi’ without
After some trial & error with watchdog timers, this works - relief!
A new plan forms
Remember I mentioned a
Partition table that ESP-IDF uses? This puts application code in a specific location within flash^
(one of two ‘over-the-air’ or OTA update partitions), along with a header that defines where in memory it runs, including
if it’s memory mapped. The normal boot flow of an ESP-IDF application is that the secondary loader applies the defined
memory mapping or copies code/data into RAM (hello
.data!), then jumps to a defined entry point. Neat.
This is how Micropython itself get’s going, can I re-use the same flow to load/run my application at known memory
locations outside Micropython, without having to use the broken
.mpy module format? Baby steps ensue..
^ for those of you wondering why not memory map from the filesystem part of flash? The MMU is not granular enough for this, hence having to map from a specific ‘OTA’ partition offset.
Gently does it..
phlashboot so it doesn’t expect to be the bootloader, instead it looks like an OTA application, which the ESP-IDF
secondary loader is happy to memory map (mmap) and run, next step..
After a great deal of reading through technical manuals
and rummaging in the code, creating the
romread test application and generally
poking at (and crashing) my device, I understand how the memory manager works, what I can and cannot touch and in theory
how I can memory map and execute another app within micropython. There is also a fair amount of investigation into the
Micropython memory layout, see
memstuff folder! Thus is born
doomloader.mpy a short and separate
Micropython module that pokes at the MMU to load an OTA app into Micropython-safe address space. It works :D, next step..
DOOM is back baby!
I can now use standard tooling, to build a standard application and run it on device “underneath” Micropython, so back to DOOM. There is some fiddling with compiler options and load addresses to get a sensible OTA application out, which fits easily into a flash (~460k, OTA partitions are 2MiB), the existing plan for connecting in I/O is still intact(!) so in theory it should work… and it doesn’t.
It turns out that DOOM expects a little more POSIX compliance from it’s C library than Micropython provides, in particular
printf implementation is lacking many features. The good part is that because I’m building an entirely separate
binary application, I can use someone elses printf, and finally, after
a lot of pain, it runs!
The colours are awful. I can’t actually control anything yet. I’m deliriously happy :D
The colour being mangled turns out to be an endian difference between the MPU and the display device, easily determined by looking at the code for the display driver, which byte-swaps everything when drawing into the device, unless it’s blitting a buffer, when it assumes it’s already swapped. Fixed.
Providing user input is a case of finding the right DOOM keys to fake when pressing joystick buttons or others on the badge.
Show me the code!
To horribly mis-quote another of my favourite games come get some baby!
Grab the built objects from the hatchery