Universal Debugger Project

This is a reference and general documentation for a project I (Parasyte) have been interested in pursuing for several years. This article will discuss the project's history, current developments, and future.

Project history

I began my research into debuggers sometime in 1999. Things were different back then; computers were just barely able to emulate SNES, even with emulators of the time taking numerous shortcuts to help speed up emulation. At the time, I only know of one SNES emulator pre-compiled with any kind of debugging functionality, and that was a build of SNES9x (then a chromeless application which was horribly difficult to use) by LordTech. It allowed the user to enable and disable assembly tracing while the game ran. This was a good start; you could see what the game was doing, and gain a lot of information from it. But it was still clunky.

To find the assembly which read or wrote to a specific address, you had to search through (often many megabytes) of text for those addresses. It gave you a good piece of the puzzle, but it did not give you the whole picture. Often many instructions were branched over, and were entirely missing from the log.

A few years later, I had some C programming experience of my own under my belt, and I had also found a new NES emulator called NESten. It wasn't a great NES emulator, but it was decent; it ran most of the NES games I was interested in, and it also had a fairly nice debugger built in! It was during this time that I got my first real taste of debugging. (Previously I experimented with UltraHLE's debugging features, but I don't recall them being much use; it used another "look but don't touch" passive take on debugging.)

NESten's debugger (and the emulator itself) did have some annoying flaws. And after getting sick of it, I decided (like many pioneering hackers before me) to take a decent open source emulator and extend it with some decent debugging capabilities. I chose FCEU (FamiCom Emulator Ultra) by Xodnizel as my base. (Note: FCEU was previously based on FCE, by Bero.) FCEU was a good candidate because it had decent emulation capabilities, and also because it was quite fast; you really need a fast base if you're just going to slow it down with powerful features. And I'll get to that point later.

FCEUd

And with that, FCEUd (FCEU-debug) was born. It did not take long to get a stable debugger integrated into the CPU core and hooked up with a little chrome. Seems like it was only 3 or 4 months before it was usable, and a total of 6 months or so development time before I was happy with it. This debugger took all of the major ideas from NESten (including its disassembler syntax for displaying current work addresses) and added some crucial elements that NESten lacked (write breakpoints triggered by INC/DEC instructions, which increment and decrement bytes in memory). I also used ideas from PSX debuggers, such as "step out" which mutually comples the "step in" and "step over" process. And I'll get to that point, again, later.

Since then, FCEUd has been further extended by other members of the emulation/ROM hacking scenes. To this day, FCE has been the most widely forked emulator, in part, because of the work we did extending it from an NES player to an NES development tool. But as you will see, this will bring me to another point later: The work done on FCE-derived forks is very widespread without much communication between parties. Ultimately, it's a mish-mash mix of talent tackling the same ideas from different vantage points without direct collaboration. And I think it's a shame.

GCNrd

The main development cycle on FCEUd was during 2002. I learned quite a bit from this project, and it got me interested in debugging other architectures. The next architecture I took on was Nintendo GameCube, officially abbreviated "GCN".

Sometime in 2003, hackers discovered a flaw in the GCN game Phantasy Star Online, which was compatible with the GCN BroadBand Adapter (BBA). This quickly became known as the PSO exploit, and allowed running "homebrew" code on a consumer GCN.

The PSO exploit worked something like this: Hacker Joe sets up his own DNS server, and tells PSO to use that server IP address as its default gateway. This way, PSO asks Hacker Joe's DNS server where it can find Sega's PSO server. Hacker Joe's DNS server tells PSO that Sega's server is located on Hacker Joe's PSO server. PSO happily connects to Hacker Joe's PSO server, which then tells PSO that there is an update available. PSO happily downloads the "PSO loader stub" which is posing as an official update to PSO. PSO happily launches the PSO loader stub which takes over the GCN and begins downloading a larger executable (of Hacker Joe's choice) from Hacker Joe's PSO server. The PSO loader stub then launches Hacker Joe's executable, and Hacker Joe now has total control over his GCN. In short, PSO is tricked by Hacker Joe into running code that it never should have in the first place.

This was a huge breakthrough for GCN hacking. I was in a very poorly state at the time, with no real job and no real income. I was doing work for a certain company selling GBA flash carts. It got me by, but it was a far cry from decent living. I mention this because it means I couldn't afford PSO or a BBA; the two things I needed to begin work on GCN debugging. I had the skill to accomplish it certainly, so I took up donations from members in the community. Only a few responded, but that's all I needed. About $100 USD was enough to pick up the two items I needed, and I was on my way to fame...

History of GCNrd

06-22-03: First GCN AR codes decrypted.
06-23-03: GCNcrypt v1.0 released.
06-25-03: First unofficial GCN AR codes released: Sonic Adventure DX (U) codes,
          by Sappharad.
06-27-03: GCNcrypt v1.1 released.
07-03-03: Planning for GCNrd begins.
07-07-03: GCNrd development begins.
08-27-03: Development on GCN network library begins.
08-29-03: First data received from GCN over network, using homebrew network
          library.
09-01-03: First data received by GCN over network, using homebrew network
          library.
09-13-03: Harddrive crash causes problems. Interest in GCNrd is lost.
12-31-03: After 3 months without working on GCNrd, aside from the occasional
          bugfix in the libs, GCNrd development finally continues.
12-31-03: Milestone #1. First successful RAM dump grabbed from a GCN game using
          GCNrd. It takes 35.735 seconds to complete.
12-31-03: BBA is started in 100mbit full duplex mode. Full RAM dump completes
          in 14.687 seconds.
01-03-04: Work on screenshot support begins. First successful screenshots are
          taken from Rayman 3 and Rogue Leader.
01-12-04: Mario Kart: Double Dash!! (U) is supported by GCNrd. This was the
          first successful "BBA Reset" patch for a LAN\network-enabled game.
01-15-04: Milestone #2. GCNrd rightly claims better compatability than Datel's
          FreeLoader v1.06b.
01-17-04: All tested games boot! 64 of 64.
01-19-04: Kenobi begins work on a GUI for GCNrd, making the program a bit
          easier to use, and adding code search features.
01-23-04: Milestone #3. First GCN AR code created using only GCNrd and Kenobi's
          GCNrd GUI. Wave Race: Blue Storm (U), 99 Points, by Knux0rz.
01-25-04: Started adding breakpoint support. Breaks work for Read, Write, and
          Read+Write. Execution breakpoints will be next. BPR\W gave me a bit
          of trouble. The biggest problem I had was getting around the 8-byte
          boundary which PowerPC uses for data breakpoints. This was completely
          unacceptable. As an example, if you set a breakpoint on address
          805E4FF5, a break would be reported every time an address between
          805E4FF0 - 805E4FF8 was accessed. Even though only one address within
          that range had a breakpoint set. My current solution to this problem
          is pretty nasty, but it works. In fact, it only reports a hit if the
          break addresses match perfectly. Maybe a little inconvenient, but
          hey!
03-11-04: GCN CodeType Helper v1.0 released.
03-11-04: GCNcrypt v1.2 released.
03-16-04: Finished up breakpoint support today, which should now be 100% fixed.
          The debugger now waits until just before returning to the game
          before enabling any r\w breakpoint. This will help bypass any
          misleading break hits that are caused by the debugger itself. Such as
          when accessing the stack and etc.
          New plan in the works to work-around memory constraints. May not work
          on all games, but should solve all problems with PacMan World 2.
          Which currently suffers from a nasty crash when accessing the memory
          card. The crash appears to be caused by how much memory I have
          allocated for the debugger.
03-17-04: Work begins on patching AR's code engine. With the patch in place,
          the code engine will run GCNrd's memory-resident debugger, allowing
          users to hack games with multiple executables, such as demo discs and
          the James Bond 007 games.
03-18-04: Kenobi adds real-time AR code list handling to the GUI. Supported AR
          versions are v1.06 and v1.08. Support for additional versions will be
          made available as soon as we receive AR RAM dumps from the other
          versions.
03-19-04: Win9x compatibility issues are worked on. GCNrd console now accepts
          commands passed on the command line. With commands entered on the
          command line, the console program will exit immediately after
          completing the requested command. Support for Win9x compatibility in
          the GUI is limited, but progressing rapidly.
03-20-04: Support for AR v1.11 completed.
03-31-04: First public release! GCNrd v1.00b is made available to hackers
          everywhere.

GCNrd GUI

The GUI for GCNrd was written entirely separately from my own GCNrd development. It was called simply GCNrdGUI and was written by a French hacker, Kenobi, in Delphi. This split up the workload considerably, and also added a new level of modularity to the project overall. The overall GCNrd project consisted of four components:

• GCNrd Loader: The UI that the user is presented with on the GCN itself. The loader is responsible for GCN-side configuration, initializing the network hardware, launching DVDs, and setting up the debugger/hooking the exeacutable.
• GCNrd debugger: This is a piece that the user never sees, but directly interacts with. It's a very small program (32KB total program code and memory usage) that runs "behind" the game, listening for instructions coming over the network, and acting upon them.
• GCNrd client: A command-line application run on the user's PC which can be used to send and receive commands and data to/from the debugger over the network.
• GCNrdGUI: Kenobi's GUI for interacting with the GCNrd client.

It was not necessary for the GUI to interact with the debugger through my CLI client, but that was the option chosen for GCNrdGUI. Later on another hacker, Sappharad, wrote his own GUI in Java which communicated with the debugger directly, bypassing the need for GCNrd client; the modularity of the project comes full circle.

GCNrd v1.10b and beyond

GCNrd v1.10b was released July 28th, 2005. It contained a number of bug fixes and new features. Some new features were only shiney on the surface (background images and UI colors in the loader, date/time display...), but some were actually quite important. The MMU handling, for example, allowed hacking games like Star Wars: Rogue Leader, Star Wars: Rebel Strike, and Metal Gear Solid: The Twin Snakes, which used the MMU extensively. It was a slightly rushed released, even though it did show some promise.

This was the last official release I made, along with a private release (only available to close friends) which included some special features like DVD dumping and loading/debugging DVD images over the BBA. This "DVD simulation" was quite advanced for its time, and allowed me (and a few others) to hack downloaded/dumped games. There are often times that the only way to get a copy of a game is downloading it. Either because it was never officially released in your region, or because it hasn't been officially released at all. This private "v1.10b+" version was never made public due to piracy concerns.

Those piracy concerns were actually forfeited after other hackers disabled the DVD booting code in v1.10b to allow debugging DVD-Rs booted with a "Cobra-like" DVD BIOS.

I had plans at one time to rewrite GCNrd so that it could be relocatable within GCN memory. This would solve problems with memory allocation on games that don't play nice. Resident Evil 4 is a good example of such games; it cannot be booted at all by any version of GCNrd. The rewrite would also make the source code legible enough that I could release it under the GPL. Granted, legibility is not necessary for source code release, but it was important for reasons of maintainability.

GCNrd source code used a wild mix of C and assembly in its main debug core. Escaping the low levels of assembly was impossible, but it could be structured nicely anyway. This wasn't the only problem, of course. Both the loader and debugger had to have their own separate build of the network library. It was too costly (memory-wise) to put the full library into the debugger, even though this would allow the loader to use the debugger for all of its networking tasks.

Instead, the debugger had a very minimal network library; I cut out all of the initialization sequences, all of the CRC code for packet data integrity checking, and whole lot of other not-completely-necessary packet handling code. It was important to keep the debugger as small as possible so it would not interfere with the game's use of memory.

That gave me two entirely different network stacks to maintain; one was already quite a bit of work, as it was. There was also the problem of compiling the debugger separately from the loader, and then compiling the debugger binary directly into the loader itself.

There are a number of ways these issues could have been addressed, and I'm not going to cover them all. But I will get back to the early point about modularity. As you will see, I am hoping to solve a great deal of these problems using that frame of mind. Needless to say, I never did complete the rewrite of GCNrd, which would have become "GCNrd v2". But the problems displayed by it were the cornerstones of what was to become the Universal Debugger Project.