Universal Debugger Project
This is a reference and general documentation for a project I (Parasyte) have been interested in pursuing for several years. This article will discuss the project's history, current developments, and future.
Project history
I began my research into debuggers sometime in 1999. Things were different back then; computers were just barely able to emulate SNES, even with emulators of the time taking numerous shortcuts to help speed up emulation. At the time, I only know of one SNES emulator pre-compiled with any kind of debugging functionality, and that was a build of SNES9x (then a chromeless application which was horribly difficult to use) by LordTech. It allowed the user to enable and disable assembly tracing while the game ran. This was a good start; you could see what the game was doing, and gain a lot of information from it. But it was still clunky.
To find the assembly which read or wrote to a specific address, you had to search through (often many megabytes) of text for those addresses. It gave you a good piece of the puzzle, but it did not give you the whole picture. Often many instructions were branched over, and were entirely missing from the log.
A few years later, I had some C programming experience of my own under my belt, and I had also found a new NES emulator called NESten. It wasn't a great NES emulator, but it was decent; it ran most of the NES games I was interested in, and it also had a fairly nice debugger built in! It was during this time that I got my first real taste of debugging. (Previously I experimented with UltraHLE's debugging features, but I don't recall them being much use; it used another "look but don't touch" passive take on debugging.)
NESten's debugger (and the emulator itself) did have some annoying flaws. And after getting sick of it, I decided (like many pioneering hackers before me) to take a decent open source emulator and extend it with some decent debugging capabilities. I chose FCEU (FamiCom Emulator Ultra) by Xodnizel as my base. (Note: FCEU was previously based on FCE, by Bero.) FCEU was a good candidate because it had decent emulation capabilities, and also because it was quite fast; you really need a fast base if you're just going to slow it down with powerful features. And I'll get to that point later.
FCEUd
And with that, FCEUd (FCEU-debug) was born. It did not take long to get a stable debugger integrated into the CPU core and hooked up with a little chrome. Seems like it was only 3 or 4 months before it was usable, and a total of 6 months or so development time before I was happy with it. This debugger took all of the major ideas from NESten (including its disassembler syntax for displaying current work addresses) and added some crucial elements that NESten lacked (write breakpoints triggered by INC/DEC instructions, which increment and decrement bytes in memory). I also used ideas from PSX debuggers, such as "step out" which mutually comples the "step in" and "step over" process. And I'll get to that point, again, later.
Since then, FCEUd has been further extended by other members of the emulation/ROM hacking scenes. To this day, FCE has been the most widely forked emulator, in part, because of the work we did extending it from an NES player to an NES development tool. But as you will see, this will bring me to another point later: The work done on FCE-derived forks is very widespread without much communication between parties. Ultimately, it's a mish-mash mix of talent tackling the same ideas from different vantage points without direct collaboration. And I think it's a shame.
GCNrd
The main development cycle on FCEUd was during 2002. I learned quite a bit from this project, and it got me interested in debugging other architectures. The next architecture I took on was Nintendo GameCube, officially abbreviated "GCN".
Sometime in 2003, hackers discovered a flaw in the GCN game Phantasy Star Online, which was compatible with the GCN BroadBand Adapter (BBA). This quickly became known as the PSO exploit, and allowed running "homebrew" code on a consumer GCN.
The PSO exploit worked something like this: Hacker Joe sets up his own DNS server, and tells PSO to use that server IP address as its default gateway. This way, PSO asks Hacker Joe's DNS server where it can find Sega's PSO server. Hacker Joe's DNS server tells PSO that Sega's server is located on Hacker Joe's PSO server. PSO happily connects to Hacker Joe's PSO server, which then tells PSO that there is an update available. PSO happily downloads the "PSO loader stub" which is posing as an official update to PSO. PSO happily launches the PSO loader stub which takes over the GCN and begins downloading a larger executable (of Hacker Joe's choice) from Hacker Joe's PSO server. The PSO loader stub then launches Hacker Joe's executable, and Hacker Joe now has total control over his GCN. In short, PSO is tricked by Hacker Joe into running code that it never should have in the first place.
This was a huge breakthrough for GCN hacking. I was in a very poorly state at the time, with no real job and no real income. I was doing work for a certain company selling GBA flash carts. It got me by, but it was a far cry from decent living. I mention this because it means I couldn't afford PSO or a BBA; the two things I needed to begin work on GCN debugging. I had the skill to accomplish it certainly, so I took up donations from members in the community. Only a few responded, but that's all I needed. About $100 USD was enough to pick up the two items I needed, and I was on my way to fame...
History of GCNrd
06-22-03: First GCN AR codes decrypted.
06-23-03: GCNcrypt v1.0 released.
06-25-03: First unofficial GCN AR codes released: Sonic Adventure DX (U) codes,
by Sappharad.
06-27-03: GCNcrypt v1.1 released.
07-03-03: Planning for GCNrd begins.
07-07-03: GCNrd development begins.
08-27-03: Development on GCN network library begins.
08-29-03: First data received from GCN over network, using homebrew network
library.
09-01-03: First data received by GCN over network, using homebrew network
library.
09-13-03: Harddrive crash causes problems. Interest in GCNrd is lost.
12-31-03: After 3 months without working on GCNrd, aside from the occasional
bugfix in the libs, GCNrd development finally continues.
12-31-03: Milestone #1. First successful RAM dump grabbed from a GCN game using
GCNrd. It takes 35.735 seconds to complete.
12-31-03: BBA is started in 100mbit full duplex mode. Full RAM dump completes
in 14.687 seconds.
01-03-04: Work on screenshot support begins. First successful screenshots are
taken from Rayman 3 and Rogue Leader.
01-12-04: Mario Kart: Double Dash!! (U) is supported by GCNrd. This was the
first successful "BBA Reset" patch for a LAN\network-enabled game.
01-15-04: Milestone #2. GCNrd rightly claims better compatability than Datel's
FreeLoader v1.06b.
01-17-04: All tested games boot! 64 of 64.
01-19-04: Kenobi begins work on a GUI for GCNrd, making the program a bit
easier to use, and adding code search features.
01-23-04: Milestone #3. First GCN AR code created using only GCNrd and Kenobi's
GCNrd GUI. Wave Race: Blue Storm (U), 99 Points, by Knux0rz.
01-25-04: Started adding breakpoint support. Breaks work for Read, Write, and
Read+Write. Execution breakpoints will be next. BPR\W gave me a bit
of trouble. The biggest problem I had was getting around the 8-byte
boundary which PowerPC uses for data breakpoints. This was completely
unacceptable. As an example, if you set a breakpoint on address
805E4FF5, a break would be reported every time an address between
805E4FF0 - 805E4FF8 was accessed. Even though only one address within
that range had a breakpoint set. My current solution to this problem
is pretty nasty, but it works. In fact, it only reports a hit if the
break addresses match perfectly. Maybe a little inconvenient, but
hey!
03-11-04: GCN CodeType Helper v1.0 released.
03-11-04: GCNcrypt v1.2 released.
03-16-04: Finished up breakpoint support today, which should now be 100% fixed.
The debugger now waits until just before returning to the game
before enabling any r\w breakpoint. This will help bypass any
misleading break hits that are caused by the debugger itself. Such as
when accessing the stack and etc.
New plan in the works to work-around memory constraints. May not work
on all games, but should solve all problems with PacMan World 2.
Which currently suffers from a nasty crash when accessing the memory
card. The crash appears to be caused by how much memory I have
allocated for the debugger.
03-17-04: Work begins on patching AR's code engine. With the patch in place,
the code engine will run GCNrd's memory-resident debugger, allowing
users to hack games with multiple executables, such as demo discs and
the James Bond 007 games.
03-18-04: Kenobi adds real-time AR code list handling to the GUI. Supported AR
versions are v1.06 and v1.08. Support for additional versions will be
made available as soon as we receive AR RAM dumps from the other
versions.
03-19-04: Win9x compatibility issues are worked on. GCNrd console now accepts
commands passed on the command line. With commands entered on the
command line, the console program will exit immediately after
completing the requested command. Support for Win9x compatibility in
the GUI is limited, but progressing rapidly.
03-20-04: Support for AR v1.11 completed.
03-31-04: First public release! GCNrd v1.00b is made available to hackers
everywhere.
GCNrd GUI
The GUI for GCNrd was written entirely separately from my own GCNrd development. It was called simply GCNrdGUI and was written by a French hacker, Kenobi, in Delphi. This split up the workload considerably, and also added a new level of modularity to the project overall. The overall GCNrd project consisted of four components:
- GCNrd Loader: The UI that the user is presented with on the GCN itself. The loader is responsible for GCN-side configuration, initializing the network hardware, launching DVDs, and setting up the debugger/hooking the exeacutable.
- GCNrd debugger: This is a piece that the user never sees, but directly interacts with. It's a very small program (32KB total program code and memory usage) that runs "behind" the game, listening for instructions coming over the network, and acting upon them.
- GCNrd client: A command-line application run on the user's PC which can be used to send and receive commands and data to/from the debugger over the network.
- GCNrdGUI: Kenobi's GUI for interacting with the GCNrd client.
It was not necessary for the GUI to interact with the debugger through my CLI client, but that was the option chosen for GCNrdGUI. Later on another hacker, Sappharad, wrote his own GUI in Java which communicated with the debugger directly, bypassing the need for GCNrd client; the modularity of the project comes full circle.
GCNrd v1.10b and beyond
GCNrd v1.10b was released July 28th, 2005. It contained a number of bug fixes and new features. Some new features were only shiney on the surface (background images and UI colors in the loader, date/time display...), but some were actually quite important. The MMU handling, for example, allowed hacking games like Star Wars: Rogue Leader, Star Wars: Rebel Strike, and Metal Gear Solid: The Twin Snakes, which used the MMU extensively. It was a slightly rushed released, even though it did show some promise.
This was the last official release I made, along with a private release (only available to close friends) which included some special features like DVD dumping and loading/debugging DVD images over the BBA. This "DVD simulation" was quite advanced for its time, and allowed me (and a few others) to hack downloaded/dumped games. There are often times that the only way to get a copy of a game is downloading it. Either because it was never officially released in your region, or because it hasn't been officially released at all. This private "v1.10b+" version was never made public due to piracy concerns.
Those piracy concerns were actually forfeited after other hackers disabled the DVD booting code in v1.10b to allow debugging DVD-Rs booted with a "Cobra-like" DVD BIOS.
I had plans at one time to rewrite GCNrd so that it could be relocatable within GCN memory. This would solve problems with memory allocation on games that don't play nice. Resident Evil 4 is a good example of such games; it cannot be booted at all by any version of GCNrd. The rewrite would also make the source code legible enough that I could release it under the GPL. Granted, legibility is not necessary for source code release, but it was important for reasons of maintainability.
GCNrd source code used a wild mix of C and assembly in its main debug core. Escaping the low levels of assembly was impossible, but it could be structured nicely anyway. This wasn't the only problem, of course. Both the loader and debugger had to have their own separate build of the network library. It was too costly (memory-wise) to put the full library into the debugger, even though this would allow the loader to use the debugger for all of its networking tasks.
Instead, the debugger had a very minimal network library; I cut out all of the initialization sequences, all of the CRC code for packet data integrity checking, and whole lot of other not-completely-necessary packet handling code. It was important to keep the debugger as small as possible so it would not interfere with the game's use of memory.
That gave me two entirely different network stacks to maintain; one was already quite a bit of work, as it was. There was also the problem of compiling the debugger separately from the loader, and then compiling the debugger binary directly into the loader itself.
There are a number of ways these issues could have been addressed, and I'm not going to cover them all. But I will get back to the early point about modularity. As you will see, I am hoping to solve a great deal of these problems using that frame of mind. Needless to say, I never did complete the rewrite of GCNrd, which would have become "GCNrd v2". But the problems displayed by it were the cornerstones of what was to become the Universal Debugger Project.
PCS and Mupen 64
The next year, in 2006, I went back to hacking N64. This was a time when some very interesting N64 hacks were made [1]. By this time, we were all very familiar with Nemu and its debugging/cheating capabilities. The debugger was not bad, but its emulation left quite a bit to be desired. Once again, unhappy with the tools available, I set out to tackle the problem and create my own tools.
The first was Parasytic Cheat Search, a video plugin for PJ64 (and other emulators compatible with its plugin spec) that allowed cheat searching through the emulator's memory. It contained many bugs and architectural flaws (being a video plugin which passed all video-related messages to a real video plugin) but it provided a start for me to get into N64 emulator development.
My next N64 project was tackling the debugger problem. Unlike a simple passive cheat searching program, a debugger has to be hooked directly to the CPU to work consistently. Like the FCEUd project years earlier, I wanted to start with a decent open source emulator and extend its capabilities to include decent debugging features.
I chose Mupen 64 as my base, this time. It had relatively high compatibility, stability, and speed. It also contained a debugger for its GTK+ build, which I knew I could use to my advantage. I was still a Win32 programmer at the time; I did not have much experience outside of Windows. Naturally, I focused on Mupen's Windows code, which is where my debugger would live.
After patching Mupen's CPU core (all three of them!) I started implementing the debugger window and all of its fancy widgets. This is when a very bad case of Deja Vu hit me. I had done all of this work before; I was reinventing the wheel. Not good. So I left Mupen 64 with a bare-bones cheat feature (including search) and debugger. This is when I began formulating the idea that debugger interfaces should be modular, just like GCNrd. Then came an avalanche of questions and issues. More on that later.
NTRrd and Kwurdi
In 2007, we had Nintendo DS (officially abbreviated NTR, unofficial NDS), and we had a means to hack it thanks to the efforts of the PassMe crew. I did not jump into DS dev quite as enthusiastically as I had GCN, but I eventually got into it and started playing around with ideas to debug the little machine. There were several problems, initially: The first was a communications port. I chose to use the GBA X-port, which was a rather expensive device (especially for video game hackers, who are typically in their teens, and without jobs), but it was attractive to me because of its FPGA.
The X-port comes by default with a simple LPT serial port that can be used as a communications link. But it is generally slow and unreliable for large data transfers. On the other hand, the FPGA (and a LOT of IO) can be leveraged to implement a much faster and more stable communications link. And that's exactly what I did, writing custom FPGA logic for a simple 8-bit ECP-mode LPT port. The result was a much faster (~4MB/s ... still much slower than ethernet, WiFi, or USB would have been) and much more reliable (~0.0001% error rate) communications link that I could use to dump NDS memory to perform cheat searching.
The initial result of this work was "NTRrd" which never saw any kind of release. That's when the ideas from past projects began to meld into an idea well ahead of its time. The idea of a universal debugger. Or at least, a universal debugger interface. So I came up with a silly acronym: KWURDI, for KodeWerx Universal Remote Debugger Interface. And I wanted NTRrd to be its first "client".
The idea was that NTRrd would be a three-piece project: NTRrd loader, NTRrd debugger, NTRrd client. And Kwurdi would be its GUI. The loader would load the debugger, the debugger would talk to the client, and the client would talk to the GUI. That's a lot of levels of abstraction. As it turns out, though, all of those levels are quite important. In fact, it's this level of abstraction and modularity which make this idea of a universal debugger important.
The universal debugger
The original article for the Universal Debugger Project can be found in the Debugging Modern Computer Architectures article. While it covers a good general scope of the idea itself, it does not lay down any foundation other than a very generic description of the roles that each piece of the project might play. To highlight the main points of the project:
- The universal debugger interface itself is a user interface to allow interaction with a low-level debugger.
- The low-level debugger may or may not live on the same machine as the universal debugger interface. The universal debugger interface communicates with the low-level debugger through a standard communications link and protocol.
- The communications link should be an existing standard. For example, UNIX domain sockets or DBUS for a local low-level debugger, and TCP/IP for a remote low-level debugger.
- The protocol should also be an existing standard, but to this date, there does not appear to be anything which fits the goals of a universal debugger interface. One possibility is RFC-909, Loader Debugger Protocol.
Project goals
The long term goals of the Universal Debugger Project include:
- Modularity
- Standardization
These items are to be taken as mutually inclusive. For modularity to work, there must be a set of standards in place. For a set of standards to be successful, the modularity granted by the standards must be exploited.
For our purposes, I wish to implement a set of standards for a "universal debugger" with the help of the community, and implement a debugger interface on top of those standards as an example of modularity. The implementation has the following goals:
- It will be Free Software, released under a free software license compatible with the GNU General Public License.
- It will be a community-oriented project; accepting ideas, patches, code, images, and documentation, among other items, from the community which supports it.
- It will be designed as a cross-platform application, able to be used on multiple operating systems.
- It will be built with internationalization and localization in mind; capable of being adapted to different languages.
- It will communicate with and support of any low-level debugger which supports the standards defined by the Universal Debugger Project.
- It will be configurable, extensible, and modular by nature, to support any conceivable current or future architecture.
- It will support highly advanced debugging techniques and bleeding-edge technologies to get the most out of the user's debugging experience.
- It will support and encourage community activity through collaborative debugging.
I will cover these points one at a time in more detail:
Free software
Free Software released under the GNU General Public License or a compatible license is meant to be community-driven. This means, anyone who wants to participate in the development of the project is free to participate. You don't become a community member by being one of a few elite deemed "worthy"; you become a member simply by joining an active discussion or contributing in some other way.
The source code will be made available for everyone to look at or extend, which benefits from many eyes keeping the code safe and secure, and also from many hands working in parallel to quickly produce a stable product that is free for anyone to use in any way they see fit. These are two of the basic freedoms granted by a Free Software license: Freedom to look at and extent the product, and freedom to use the product in any way. There are many more freedoms granted by Free Software, and if interested you should consult the GPL documentation or other authoritative sources.
Community-oriented, community-driven
A community-oriented and community-driven software project benefits from its main source of use: its own users. Such projects become self-sustaining, as their users take the secondary roles of developers. I will not cover many of the details relating to community-oriented or community-driven software, but there are plenty of great resources available for further reading. Some suggested reading includes The Cathedral and the Bazaar [2] by Eric S. Raymond, the About Mozilla pages [3] [4] [5] [6], and the online book Open Sources: Voices from the Open Source Revolution.
As a community-oriented, community-driven project, I believe there is no end to the ways in which we, as a community of programmers and hackers, will be able to debug. This community would ideally involve the active roles of everyone from video game and ROM hackers, application and operating system developers, and people in the field of academic research of computer architectures. It doesn't have to be the biggest and the best of the debugging communities (and it probably won't!) but it does have to be an accepting community, willing to allow contributions from anyone and everyone who wishes to contribute.
Cross-platform design
Among the design goals of the project, the universal debugger implementation must be able to run on multiple computer platforms and operating systems. This is a difficult undertaking, especially for a program focusing on a graphical user environment. Several bases for cross-platform development already exist, so dealing with intricacies of operating systems can be cut down to a minimum, while retaining a high level of overall product quality.
I have personally researched several offerings for cross-platform frameworks and libraries, and I believe the best choice is the Mozilla platform, XULRunner. XULRunner is still in its infancy, but is already very capable; it's the powerhouse behind the Firefox web browser, which runs on a large percentage of popular operating systems.
Here is a short list of advantages to using XULRunner over just any cross-platform framework:
- XULRunner is Free Software, and its development is community-oriented.
- Unlike Java and .NET, XULRunner applications are built with native code and have a native look-and-feel ("native" is in reference to the host operating system).
- Also unlike Java and .NET, XULRunner applications are written in C++, a highly portable and widely used programming language.
- XULRunner provides many features that Firefox users are already familiar with: scripting with Javascript, application extensibility through "extensions", networking and popular web protocols support, HTML processing (and a LOT more) with the Gecko rendering engine.
- User interfaces in XULRunner are written in XUL (an XML dialect), styled with CSS, and setup to "do things" with Javascript -- all current web standards. The use of CSS allows the user interface to be "skinned" to a user's personal tastes.
- UI functionality is mostly implemented with Javascript, but backbone and critical code can be written in C++ and accessed from the Javascript environment.
For a very informative read on the Mozilla platform, I recommend the online book Creating Applications with Mozilla.
Internationalized and localized
[Work in progress]
Standards compliant
[Work in progress]
Configurable, extensible, and modular
[Work in progress]
Advanced debugging technologies
[Work in progress]
Encouraging community activity
[Work in progress]
