Universal Debugger Project

From Kodewerx
Jump to: navigation, search

This is a reference and general documentation for a project I (Parasyte) have been interested in pursuing for several years. This article will discuss the project's history, current developments, and future.

I've made an effort to write this document with as little technical jargon as possible. The nature of debuggers and debugging, however, is extremely technical, so it has been difficult to keep it down to a decent reading level. For this reason, I've included an Appendix which should ease the use of language throughout this documentation.

Finally, I hope this information is not too redundant. Some things (examples given, explanations, etc) may be repeated more than once. This document is supposed to be highly detailed, but concise. If you see many repetitious things, please point them out on the discussion page.

Project history

I began my research into debuggers sometime in 1999. Things were different back then; computers were just barely able to emulate SNES, even with emulators of the time taking numerous shortcuts to help speed up emulation. At the time, I only knew of one SNES emulator pre-compiled with any kind of debugging functionality, and that was a build of SNES9x (then a chromeless application which was horribly difficult to use) by LordTech. It allowed the user to enable and disable assembly tracing while the game ran. This was a good start; you could see what the game was doing, and gain a lot of information from it. But it was still clunky.

To find the assembly which read or wrote to a specific address, you had to search through (often many megabytes of) text for those addresses. It gave you a good piece of the puzzle, but it did not give you the whole picture. Often many instructions were branched over, and were entirely missing from the log.

A few years later, I had some C programming experience of my own under my belt, and I had also found a new NES emulator called NESten. It wasn't a great NES emulator, but it was decent; it ran most of the NES games I was interested in, and it also had a fairly nice debugger built in! It was during this time that I got my first real taste of debugging. (Previously I experimented with UltraHLE's debugging features, but I don't recall them being much use; it used another "look but don't touch" passive take on debugging.)

NESten's debugger (and the emulator itself) did have some annoying flaws. And after getting sick of it, I decided (like many pioneering hackers before me) to take a decent open source emulator and extend it with some decent debugging capabilities. I chose FCEU (FamiCom Emulator Ultra) by Xodnizel as my base. (Note: FCEU was previously based on FCE, by Bero.) FCEU was a good candidate because it had decent emulation capabilities, and also because it was quite fast; you really need a fast base if you're just going to slow it down with powerful features. And I'll get to that point later.


And with that, FCEUd (FCEU-debug) was born. It did not take long to get a stable debugger integrated into the CPU core and hooked up with a little chrome. Seems like it was only 3 or 4 months before it was usable, and a total of 6 months or so development time before I was happy with it. This debugger took all of the major ideas from NESten (including its disassembler syntax for displaying current work addresses) and added some crucial elements that NESten lacked (write breakpoints triggered by INC/DEC instructions, which increment and decrement bytes in memory). I also used ideas from PSX debuggers, such as "step out" which mutually completes the "step in" and "step over" process. And I'll get to that point, again, later.

Since then, FCEUd has been further extended by other members of the emulation/ROM hacking scenes. To this day, FCE has been the most widely forked emulator, in part, because of the work we did extending it from an NES player to an NES development tool. But as you will see, this will bring me to another point later: The work done on FCE-derived forks is very widespread without much communication between parties. Ultimately, it's a mish-mash mix of talent tackling the same ideas from different vantage points without direct collaboration. And I think it's a shame.


The main development cycle on FCEUd was during 2002. I learned quite a bit from this project, and it got me interested in debugging other architectures. The next architecture I took on was Nintendo GameCube, officially abbreviated "GCN".

Sometime in 2003, hackers discovered a flaw in the GCN game Phantasy Star Online, which was compatible with the GCN BroadBand Adapter (BBA). This quickly became known as the PSO exploit, and allowed running "homebrew" code on a consumer GCN.

The PSO exploit worked something like this: Hacker Joe sets up his own DNS server, and tells PSO to use that server IP address as its default gateway. This way, PSO asks Hacker Joe's DNS server where it can find Sega's PSO server. Hacker Joe's DNS server tells PSO that Sega's server is located on Hacker Joe's PSO server. PSO happily connects to Hacker Joe's PSO server, which then tells PSO that there is an update available. PSO happily downloads the "PSO loader stub" which is posing as an official update to PSO. PSO happily launches the PSO loader stub which takes over the GCN and begins downloading a larger executable (of Hacker Joe's choice) from Hacker Joe's PSO server. The PSO loader stub then launches Hacker Joe's executable, and Hacker Joe now has total control over his GCN. In short, PSO is tricked by Hacker Joe into running code that it never should have in the first place.

This was a huge breakthrough for GCN hacking. I was in a very poorly state at the time, with no real job and no real income. I was doing work for a certain company selling GBA flash carts. It got me by, but it was a far cry from decent living. I mention this because it means I couldn't afford PSO or a BBA; the two things I needed to begin work on GCN debugging. I had the skill to accomplish it certainly, so I took up donations from members in the community. Only a few responded, but that's all I needed. About $100 USD was enough to pick up the two items I needed, and I was on my way to fame...

History of GCNrd

06-22-03: First GCN AR codes decrypted.
06-23-03: GCNcrypt v1.0 released.
06-25-03: First unofficial GCN AR codes released: Sonic Adventure DX (U) codes,
          by Sappharad.
06-27-03: GCNcrypt v1.1 released.
07-03-03: Planning for GCNrd begins.
07-07-03: GCNrd development begins.
08-27-03: Development on GCN network library begins.
08-29-03: First data received from GCN over network, using homebrew network
09-01-03: First data received by GCN over network, using homebrew network
09-13-03: Harddrive crash causes problems. Interest in GCNrd is lost.
12-31-03: After 3 months without working on GCNrd, aside from the occasional
          bugfix in the libs, GCNrd development finally continues.
12-31-03: Milestone #1. First successful RAM dump grabbed from a GCN game using
          GCNrd. It takes 35.735 seconds to complete.
12-31-03: BBA is started in 100mbit full duplex mode. Full RAM dump completes
          in 14.687 seconds.
01-03-04: Work on screenshot support begins. First successful screenshots are
          taken from Rayman 3 and Rogue Leader.
01-12-04: Mario Kart: Double Dash!! (U) is supported by GCNrd. This was the
          first successful "BBA Reset" patch for a LAN\network-enabled game.
01-15-04: Milestone #2. GCNrd rightly claims better compatability than Datel's
          FreeLoader v1.06b.
01-17-04: All tested games boot! 64 of 64.
01-19-04: Kenobi begins work on a GUI for GCNrd, making the program a bit
          easier to use, and adding code search features.
01-23-04: Milestone #3. First GCN AR code created using only GCNrd and Kenobi's
          GCNrd GUI. Wave Race: Blue Storm (U), 99 Points, by Knux0rz.
01-25-04: Started adding breakpoint support. Breaks work for Read, Write, and
          Read+Write. Execution breakpoints will be next. BPR\W gave me a bit
          of trouble. The biggest problem I had was getting around the 8-byte
          boundary which PowerPC uses for data breakpoints. This was completely
          unacceptable. As an example, if you set a breakpoint on address
          805E4FF5, a break would be reported every time an address between
          805E4FF0 - 805E4FF8 was accessed. Even though only one address within
          that range had a breakpoint set. My current solution to this problem
          is pretty nasty, but it works. In fact, it only reports a hit if the
          break addresses match perfectly. Maybe a little inconvenient, but
03-11-04: GCN CodeType Helper v1.0 released.
03-11-04: GCNcrypt v1.2 released.
03-16-04: Finished up breakpoint support today, which should now be 100% fixed.
          The debugger now waits until just before returning to the game
          before enabling any r\w breakpoint. This will help bypass any
          misleading break hits that are caused by the debugger itself. Such as
          when accessing the stack and etc.
          New plan in the works to work-around memory constraints. May not work
          on all games, but should solve all problems with PacMan World 2.
          Which currently suffers from a nasty crash when accessing the memory
          card. The crash appears to be caused by how much memory I have
          allocated for the debugger.
03-17-04: Work begins on patching AR's code engine. With the patch in place,
          the code engine will run GCNrd's memory-resident debugger, allowing
          users to hack games with multiple executables, such as demo discs and
          the James Bond 007 games.
03-18-04: Kenobi adds real-time AR code list handling to the GUI. Supported AR
          versions are v1.06 and v1.08. Support for additional versions will be
          made available as soon as we receive AR RAM dumps from the other
03-19-04: Win9x compatibility issues are worked on. GCNrd console now accepts
          commands passed on the command line. With commands entered on the
          command line, the console program will exit immediately after
          completing the requested command. Support for Win9x compatibility in
          the GUI is limited, but progressing rapidly.
03-20-04: Support for AR v1.11 completed.
03-31-04: First public release! GCNrd v1.00b is made available to hackers


The GUI for GCNrd (Nintendo Gamecube Remote Debugger) was written entirely separately from my own GCNrd development. It was called simply GCNrdGUI and was written by a French hacker, Kenobi, in Delphi. This split up the workload considerably, and also added a new level of modularity to the project overall. The overall GCNrd project consisted of four components:

  • GCNrd Loader: The UI that the user is presented with on the GCN itself. The loader is responsible for GCN-side configuration, initializing the network hardware, launching DVDs, and setting up the debugger/hooking the executable.
  • GCNrd debugger: This is a piece that the user never sees, but directly interacts with. It's a very small program (32KB total program code and memory usage) that runs "behind" the game, listening for instructions coming over the network, and acting upon them.
  • GCNrd client: A command-line application run on the user's PC which can be used to send and receive commands and data to/from the debugger over the network.
  • GCNrdGUI: Kenobi's GUI for interacting with the GCNrd client.

It was not necessary for the GUI to interact with the debugger through my CLI client, but that was the option chosen for GCNrdGUI. Later on another hacker, Sappharad, wrote his own GUI in Java which communicated with the debugger directly, bypassing the need for GCNrd client; the modularity of the project comes full circle.

GCNrd v1.10b and beyond

GCNrd v1.10b was released July 28th, 2005. It contained a number of bug fixes and new features. Some new features were only shiney on the surface (background images and UI colors in the loader, date/time display...), but some were actually quite important. The MMU handling, for example, allowed hacking games like Star Wars: Rogue Leader, Star Wars: Rebel Strike, and Metal Gear Solid: The Twin Snakes, which used the MMU extensively. It was a slightly rushed released, even though it did show some promise.

This was the last official release I made, along with a private release (only available to close friends) which included some special features like DVD dumping and loading/debugging DVD images over the BBA. This "DVD simulation" was quite advanced for its time, and allowed me (and a few others) to hack downloaded/dumped games. There are often times that the only way to get a copy of a game is downloading it. Either because it was never officially released in your region, or because it hasn't been officially released at all. This private "v1.10b+" version was never made public due to piracy concerns.

Those piracy concerns were actually forfeited after other hackers disabled the DVD booting code in v1.10b to allow debugging DVD-Rs booted with a "Cobra-like" DVD BIOS.

I had plans at one time to rewrite GCNrd so that it could be relocatable within GCN memory. This would solve problems with memory allocation on games that don't play nice. Resident Evil 4 is a good example of such games; it cannot be booted at all by any version of GCNrd. The rewrite would also make the source code legible enough that I could release it under the GPL. Granted, legibility is not necessary for source code release, but it was important for reasons of maintainability.

GCNrd source code used a wild mix of C and assembly in its main debug core. Escaping the low levels of assembly was impossible, but it could be structured nicely anyway. This wasn't the only problem, of course. Both the loader and debugger had to have their own separate build of the network library. It was too costly (memory-wise) to put the full library into the debugger, even though this would allow the loader to use the debugger for all of its networking tasks.

Instead, the debugger had a very minimal network library; I cut out all of the initialization sequences, all of the CRC code for packet data integrity checking, and a whole lot of other not-completely-necessary packet handling code. It was important to keep the debugger as small as possible so it would not interfere with the game's use of memory.

That gave me two entirely different network stacks to maintain; one was already quite a bit of work, as it was. There was also the problem of compiling the debugger separately from the loader, and then compiling the debugger binary directly into the loader itself.

There are a number of ways these issues could have been addressed, and I'm not going to cover them all. But I will get back to the early point about modularity. As you will see, I am hoping to solve a great deal of these problems using that frame of mind. Needless to say, I never did complete the rewrite of GCNrd, which would have become "GCNrd v2". But the problems displayed by it were the cornerstones of what was to become the Universal Debugger Project.

PCS and Mupen 64

The next year, in 2006, I went back to hacking N64. This was a time when some very interesting N64 hacks were made [1]. By this time, we were all very familiar with Nemu and its debugging/cheating capabilities. The debugger was not bad, but its emulation left quite a bit to be desired. Once again, unhappy with the tools available, I set out to tackle the problem and create my own tools.

The first was Parasytic Cheat Search, a video plugin for PJ64 (and other emulators compatible with its plugin spec) that allowed cheat searching through the emulator's memory. It contained many bugs and architectural flaws (being a video plugin which passed all video-related messages to a real video plugin) but it provided a start for me to get into N64 emulator development.

My next N64 project was tackling the debugger problem. Unlike a simple passive cheat searching program, a debugger has to be hooked directly to the CPU to work consistently. Like the FCEUd project years earlier, I wanted to start with a decent open source emulator and extend its capabilities to include decent debugging features.

I chose Mupen 64 as my base, this time. It had relatively high compatibility, stability, and speed. It also contained a debugger for its GTK+ build, which I knew I could use to my advantage. I was still a Win32 programmer at the time; I did not have much experience outside of Windows. Naturally, I focused on Mupen's Windows code, which is where my debugger would live.

After patching Mupen's CPU core (all three of them!) I started implementing the debugger window and all of its fancy widgets. This is when a very bad case of Deja Vu hit me. I had done all of this work before; I was reinventing the wheel. Not good. So I left Mupen 64 with a bare-bones cheat feature (including search) and debugger. This is when I began formulating the idea that debugger interfaces should be modular, just like GCNrd. Then came an avalanche of questions and issues. More on that later.

NTRrd and Kwurdi

In 2007, we had Nintendo DS (officially abbreviated NTR, unofficial NDS), and we had a means to hack it thanks to the efforts of the PassMe crew. I did not jump into DS dev quite as enthusiastically as I had GCN, but I eventually got into it and started playing around with ideas to debug the little machine. There were several problems, initially: The first was a communications port. I chose to use the GBA X-port, which was a rather expensive device (especially for video game hackers, who are typically in their teens, and without jobs; in hindsight, it was probably a very bad idea), but it was attractive to me because of its FPGA.

The X-port comes by default with a simple LPT serial port that can be used as a communications link. But it is generally slow and unreliable for large data transfers. On the other hand, the FPGA (and a LOT of IO) can be leveraged to implement a much faster and more stable communications link. And that's exactly what I did, writing custom FPGA logic for a simple 8-bit ECP-mode LPT port. The result was a much faster (~4MB/s ... still much slower than ethernet, WiFi, or USB would have been) and much more reliable (~0.0001% error rate) communications link that I could use to dump NDS memory to perform cheat searching.

The initial result of this work was "NTRrd" (Nintendo DS Remote Debugger) which never saw any kind of release. That's when the ideas from past projects began to meld into an idea well ahead of its time. The idea of a universal debugger. Or at least, a universal debugger interface. So I came up with a silly acronym: KWURDI, for KodeWerx Universal Remote Debugger Interface. And I wanted NTRrd to be its first "client".

The idea was that NTRrd would be a three-piece project: NTRrd loader, NTRrd debugger, NTRrd client. And Kwurdi would be its GUI. The loader would load the debugger, the debugger would talk to the client, and the client would talk to the GUI. That's a lot of levels of abstraction. As it turns out, though, all of those levels are quite important. In fact, it's this level of abstraction and modularity which make this idea of a universal debugger important.

The universal debugger

The original article for the Universal Debugger Project can be found in the Debugging Modern Computer Architectures article. While it covers a good general scope of the idea itself, it does not lay down any foundation other than a very generic description of the roles that each piece of the project might play. To highlight the main points of the project:

  • The universal debugger interface itself is a user interface to allow interaction with a low-level debugger.
  • The low-level debugger may or may not live on the same machine as the universal debugger interface. The universal debugger interface communicates with the low-level debugger through a standard communications link and protocol.
  • The communications link should be an existing standard. For example, UNIX domain sockets or DBUS for a local low-level debugger, and TCP/IP for a remote low-level debugger.
  • The protocol should also be an existing standard, but to this date, there does not appear to be anything which fits the goals of a universal debugger interface. One possibility is RFC-909, Loader Debugger Protocol.

Project goals

The long term goals of the Universal Debugger Project include:

  • Modularity
  • Standardization

These items are to be taken as mutually inclusive. For modularity to work, there must be a set of standards in place. For a set of standards to be successful, the modularity granted by the standards must be exploited.

For our purposes, I wish to implement a set of standards for a "universal debugger" with the help of the community, and implement a debugger interface on top of those standards as an example of modularity. The implementation has the following goals:

  1. It will be Free Software, released under a free software license comparable to the GNU General Public License.
  2. It will be a community-oriented project; accepting ideas, patches, code, images, and documentation, among other items, from the community which supports it.
  3. It will be designed as a cross-platform application, able to be used on multiple operating systems.
  4. It will be built with internationalization and localization in mind; capable of being adapted to different languages.
  5. It will be designed to work well with modern accessibility software.
  6. It will communicate with and support any low-level debugger which supports the standards defined by the Universal Debugger Project.
  7. It will be configurable, extensible, and modular by nature, to support any conceivable current or future architecture.
  8. It will support highly advanced debugging techniques and bleeding-edge technologies to get the most out of the user's debugging experience.
  9. It will support and encourage community activity through collaborative debugging.

I will cover these points one at a time in more detail:

Free software

Free Software released under the GNU General Public License or a comparable license is meant to be community-driven. This means, anyone who wants to participate in the development of the project is free to participate. You don't become a community member by being one of a few elite deemed "worthy"; you become a member simply by joining an active discussion or contributing in some other way.

The source code will be made available for everyone to look at or extend, which benefits from many eyes keeping the code safe and secure, and also from many hands working in parallel to quickly produce a stable product that is free for anyone to use in any way they see fit. These are two of the basic freedoms granted by a Free Software license: Freedom to look at and extent the product, and freedom to use the product in any way. There are many more freedoms granted by Free Software, and if interested you should consult the GPL documentation or other authoritative sources.

Community-oriented, community-driven

A community-oriented and community-driven software project benefits from its main source of use: its own users. Such projects become self-sustaining, as their users take the secondary roles of developers. I will not cover many of the details relating to community-oriented or community-driven software, but there are plenty of great resources available for further reading. Some suggested reading includes The Cathedral and the Bazaar [2] by Eric S. Raymond, the About Mozilla pages [3] [4] [5] [6], and the online book Open Sources: Voices from the Open Source Revolution [7], written by many influential people from many open source communities.

As a community-oriented, community-driven project, I believe there is no end to the ways in which we, as a community of programmers and hackers, will be able to debug. This community would ideally involve the active roles of everyone from video game and ROM hackers, application and operating system developers, and people in the field of academic research of computer architectures. It doesn't have to be the biggest and the best of the debugging communities (and it probably won't!) but it does have to be an accepting community, willing to allow contributions from anyone and everyone who wishes to contribute.

Cross-platform design

Among the design goals of the project, the universal debugger implementation must be able to run on multiple computer platforms and operating systems. This is a difficult undertaking, especially for a program focusing on a graphical user environment. Several bases for cross-platform development already exist, so dealing with intricacies of operating systems can be cut down to a minimum, while retaining a high level of overall product quality.

I have personally researched several offerings for cross-platform frameworks and libraries, and I believe the best choice is the Mozilla platform, XULRunner. XULRunner is still in its infancy, but is already very capable; it's the powerhouse behind the Firefox web browser, which runs on a large percentage of popular operating systems.

Here is a short list of advantages to using XULRunner over just any cross-platform framework:

  • XULRunner is Free Software, and its development is community-oriented.
  • Unlike Java and .NET, XULRunner applications are built with native code and have a native look-and-feel ("native" is in reference to the host operating system).
  • Also unlike Java and .NET, XULRunner applications are written in C++, a highly portable and widely used programming language.
  • XULRunner provides many features that Firefox users are already familiar with: scripting with Javascript, application extensibility through "extensions", networking and popular web protocols support, HTML processing (and a LOT more) with the Gecko layout engine.
  • User interfaces in XULRunner are written in XUL (an XML dialect), styled with CSS, and setup to "do things" with Javascript -- all current web standards. The use of CSS allows the user interface to be "skinned" to a user's personal tastes.
  • UI functionality is mostly implemented with Javascript, but backbone and critical code can be written in C++ and accessed from the Javascript environment.

For a very informative read on the Mozilla platform, I recommend the online book Creating Applications with Mozilla.

Internationalized and localized

Because community strengths are so important to a project of this size and magnitude, the global community becomes an invaluable asset. And as a diverse global community, not everyone will speak or understand a single common language. In this case, I am directly referring to the English language, which is personally my primary language, and the only language I am entirely fluent. Programmers and hackers in a similar situation, though with a different primary language, will benefit from a universal debugger. We should not neglect any user in the community. Making the universal debugger accessible and usable for everybody must be a top priority of the project.

Choosing the Mozilla platform as the application's foundation will play a crucial role in the internationalization of the product. Because the user interface elements are written in an XML dialect, all text can be referenced with DTD entities, just like you might write > in XHTML to print the greater-than sign (>). Likewise, you might write

<caption label="&hello;">

in XUL which would display a label with "Hello" or "こんにちは" depending on the locale set on the user's machine when the XUL application is started.

DTD entities are not magic translators; the text must be translated by a human who is fluent in both (source and destination) languages. However, using DTD entities is a great way to write user interfaces (and other things, like configuration files, for example) without locking the program to a specific language. It also makes the translator's work much easier, since all text strings which require translation will be neatly stored in a single location.

Having the capability to bring a universal debugger to anyone, no matter their ethnic background, will help to expand the debugger's user base and ultimately its supporting community of developers. Designed from the beginning with breaking language barriers in mind, there should be few stumbling points along the road to the success of reaching this goal.

Accessible for everyone

On a personal note, I had the chance to tutor someone in hacking a few years ago who told me that he was blind in one eye. This made me realize that we're not all perfectly capable of using most debuggers available in the wild. Most of us hackers can get along fine with them, while a few are left to try their best at making use of such applications.

Mozilla once again comes to the rescue with a modern, mature, stable API for application accessibility. Some examples include integration and coherence with screen readers and voice recognition software. There may not be a great deal of hackers which could make use of these technologies, but having the option available is important to the individuals who would have never thought to bother with debuggers, otherwise.

And let's face grim reality here; I could one day end up physically mangled in a horrible accident, unable to see or use a keyboard or mouse. That won't change my love and desire to continue hacking. I could almost consider debugger accessibility a personal insurance for myself, let alone many hundreds of potential users today.

Standards compliant

Standards make the technological world work. And standards compliance makes compatibility issues entirely negligible. This is perhaps the most important point I will make within this documentation: defining a series of open standards to encompass all foreseeable debugging functionality is a necessity.

The major reason for adopting a standard or series of standards is for compatibility purposes. Compatibility is critical for modular designs. And modularity is an important part of the overall user-configuration and extensibility of an application or sets of applications. Modularity also has the benefit of splitting workloads, as we did previously with GCNrd. Splitting workloads decreases development times, improves communication, and in some cases even improves product quality.

In simpler terms, starting with splitting developer workloads, we all know that it's easier to cover more ground by splitting up. So long as you have a central location to meet up later, this tactic works wonderfully for rapid application development. In the case of developing a debugger, you might typically have one team focusing on the low-level handling of the CPU and hardware, another team tackling the user interface to the low-level debugger, and perhaps a third team covering the protocol between the two.

I realize that the "third team" in this example is most likely set aside as another task for the first team in the real world. But if your project does require some sort of communication layer and you pull that out of the first team's hands, you can effectively pass that responsibility to a sort of "standards body" who will oversee the development and standardization of that communication layer. The job of such a body would be to ensure the standard is usable by anyone and for any purpose, generic enough to support all current and foreseeable architectures, and extensible enough to support all unforeseeable future architectures.

The extensibility of the standard would be the most difficult to maintain. While a protocol or communication layer of this sort should be extensible for the reasons listed, it should only be extensible to the extent that any new features added would not break previous features or introduce new compatibility issues with previous designs and implementations of the protocol or communication layer. This means, generally, that the extensibility must remain open and optional; no extended feature should ever be mandatory for any user to implement.

There will undoubtedly be important new technologies which require mandatory extensions to the protocol during the life cycle and evolution of debugging technology. Under these circumstances, the standards body would be responsible for designing and publishing a new major version of the protocol with the new mandatory technologies. Implementers of the protocol, should they choose to support newer major versions, would then be required to implement all mandatory features in order to claim standards compliance of their implementation.

What we need is a standards body; a group of individuals, mostly volunteers or "elected" members, to define and publish a series of standards for debugging modern computer architectures. They will be responsible for taking everyone's concerns from the community into account, and act on those concerns within their design and definition. They will be responsible for knowing that if they fail to do a good job, the community will stop listening to them and they will easily be replaced by anyone who steps up with a better idea.

The goals of the standards body will be defining:

  • The wire; how we get data from point A (low-level debugger) to point B (user interface). Ethernet? RS232? ...
  • The language; how our data is interpreted as points A and B speak. XML? JSON? Pure binary packets? ...
  • The expansion; how we can introduce new features without creating vendor locks or introducing incompatibilities. Header/Info/About/Options commands which specify the optional/extended features we support? ...

The goals for us developers will be inheriting some of these responsibilities as we implement these standards, to create a flourishing environment for debugger developers to contribute with innovation and healthy competition.

Configurable, extensible, and modular

Debuggers are all about user experience. I say this with honest integrity because debugging, by its very nature, is a difficult task. Making a debugger very simple to use is a bullet point; you don't want to overwhelm a user (who is already working on a very difficult problem) with bad user experience. To improve the user experience, it is important to allow a level of customization among the user interface. Also important is extensibility and modularity, to make it easy for the user to expand the interface to fit her needs, and easy to replace one component with another that she prefers.

Some of the items of the user interface which should be configurable include the overall look-and-feel of the interface (skin), the organization of windows and widgets (tabs, information panes, etc.), and the styles of text, fonts, and [image-]backgrounds used throughout.

The look of the application, as well as the functionality, should be extensible using styling and scripting languages. The Mozilla platform once again becomes an asset as it allows simple integration of CSS and Javascript for the purposes of extending applications based on the platform. Extensions can be written and managed by the user community, much the same way that Firefox extensions are written and distributed. A central management location for extensions would be a great thing to have, but is not necessary to the success of a thriving extension-based sub-community.

Including certain feature sets in the application's base install as pre-installed extensions means that users may uninstall those extensions at will to remove undesired features, or replace those extensions with other extensions that implement comparable (and often better) features. This modular design will help to increase the user experience as well as the user's personal productivity with the application. This, too, can be supported by a thriving extension-based sub-community, and the community at large.

Advanced debugging technologies

When built on top of a well-designed, well-documented (and equally well-supported) standard, a debugger could implement advanced techniques and technologies to do things we can only dream of now. Some examples of my dreams:

  • The debugger interface records its target's activity using simple standard protocol features such as single stepping, and reading register and memory states. When the user pauses the target, the interface has collected enough information to step backwards in architecture time. Essentially, a runtime undo feature.
  • The communications layer, being very generic, does not need to know the specifics of how the interface uses the data it is transmitting. A very simple example is that the user might be able to use the interface for capturing audio and video samples from a game running on the target architecture, simply by recording the state of the frame and audio buffers (using simple memory reads supported by the protocol) at a specified interval.
  • The debugger interface allows incoming connections from other users to the debugging session in progress. There is a chat pane where connected users can talk about what they are doing, and they can watch each other scroll through the disassembler listing and making changes to instructions and assembler comments in real time. This "collaborative debugging" feature will be expanded upon in the next section.

Creating "high level" debugging capability out of a limited set of "low level" primitive debugging functions is what makes the idea of a standardized communications layer so appealing to me. The first two examples given in the list above are highly specialized and advanced features built on top of a relatively small and "unexciting" set of functions; Reading and writing registers or parts of the memory map, controlling code flow with instruction stepping, etc. These latter functions are the kinds of things you would expect any debugger to do. What you might not expect is how these simple functions can be chained together to create very interesting features.

This is where the foundation of standardization really comes into play with a universal debugger. As an example, if your NDS and PS2 both have debuggers implementing a single, standardized protocol, the universal debugger only needs to differentiate the two by their architectures. That is, the CPU type and configuration, register maps, memory maps, hardware I/O maps, etc. And once the architectures are known to the universal debugger, it can run the same advanced features (stepping backward, rewinding execution, audio/video recording, etc.) on both; just by using slightly different interpretations of the data it receives over the communication layer. No need for anyone to write a whole new codebase or interface; no more reinventing the wheel.

With these kinds of extraordinary possibilities in mind, I can only expect standardization to be the only way forward in the subtle world of debugging modern computer architectures.

Encouraging community activity

Having an entire community to back a project of this type will be incredibly important. Communities are seeded with an idea, and nurtured with equal opportunity to participate. In order to allow greater community participation, I believe it is essential that the project encourage activity as much as the community itself does. One way to accomplish this is by including program features which directly involve multiple users to collaborate with each other.

Two quotations come to mind in the spirit of collaborative debugging:

"Debugging is parallelizable"
"Given enough eyes, all bugs are shallow."

The first is a direct reference to multiple users working on debugging the same item, in parallel. The latter quote is slightly on the opposite; it means that you can have multiple users covering more ground by looking at different pieces of the code being debugged, and viewed from their own personal perspective on the problem. These are both great ideas to employ for large or difficult problems that many projects will face. The only problem is, it's usually handled with an "every man for himself" mentality. Multiple people running their own copies of the target, being debugged with their own debuggers, keeping their own notes, and writing and testing their own patches. All in solemn solitude.

Why not take these two ideas one more step to the extreme? We can have several eyes on the same code, so let's also put several eyes on the same debugging session. Let's parallelize the debugging process with several minds double-checking each others work in real time, as they work; providing ideas and suggestions, and introducing patches directly into the stream for immediate testing.

This would be done with similar ideas to those employed by today's collaborative text editors [8] [9]. Indeed, the universal debugger itself must be a type of editor; users need to make modifications to code and data in real time, while the target is paused, or for those who enjoy living dangerously, while the target runs its program under normal execution.

How would this work? Well, one user, Hacker Sue, would begin a debugging session by connecting to her target over the standard communications layer, and she would then host the session as a server over plain old TCP/IP. She would invite other users to join her session in progress, and she would be notified when others attempt to make a connection. For reasons of practical security, these connections should be authenticated and encrypted with SSL or similar. She may choose to allow a connection from Hacker Joe only once, or she may agree to "remember" his public key/encryption certificate. Perhaps even more importantly, she may choose to reject all connection requests from Hacker Joe.

Now that Hacker Joe has successfully connected and authenticated with Hacker Sue's debugging session, they set to work. The debugger interface contains a chat pane during hosted sessions, so they are able to talk about what they are seeing and doing. The interface also has "user markers" displayed within its content windows, like the disassembler and the memory editor. These user markers would provide a visual indication of what each user is currently looking at in the target. They could also provide a quick method to "jump" the view to the other users' view ports, and could even be "pinned" or made static; a way to automatically follow someone's view port as they move around in the code.

Hacker Sue and Hacker Joe can now both control the target, stepping through code and watching how it works. They can also change data in memory and instructions within the CPU's code flow path. Each change made will be highlighted in colors specific to each user. They can't undo each others' work through standard-undo commands, but they can patch it just like any other data.

After a long debugging session, both users are making good progress. But Hacker Joe starts doing a tedious and repetitive task of changing every other byte in a 10KB section of memory to the hex value 0x01. He suggests to Hacker Sue that this would be much easier if they both work on a different piece of the editing at the same time. Hacker Sue has a better idea: She knows a script can be written that will do all the work for them, but she's not quite sure how to do it. Hacker Joe writes a simple script in the script editor, and submits it to Hacker Sue to run. After Hacker Sue fixes some typos and potential bugs, she likes the script and saves it as a macro to the session. When Hacker Joe runs the script, and Hacker Sue agrees, the tedious task is completed in a fraction of a second.

This scenario highlights an interesting concept of collaborative scripting within a debugging session. The importance of an authentication and encryption scheme over this TCP/IP connection becomes apparent: With access to scripting capabilities and low-level access to programs running on the target machine (some times this target machine may be the same as the host machine!) security will be of utmost importance. That could be Hacker Sue's computer that Hacker Joe has complete access to. It is important to secure such collaborative debugging as best as possible, but then again the true security of this model will all come down to real trust.

Scripting and macros should be written and run with a kind of "approval" model. A script editor built into the interface would provide similar collaborative editing, where everyone can see what's being written and edited. It would then give the debugger host the final say for what becomes a runnable macro or not, and she also has the final say over who can run the available macros, and when.

There is an incredible potential to include an entire community in not only developing a universal debugger interface, but also including anyone from the community to contribute to the work done with the tool they are developing. This is a fine point which would provide many new advantages to an already self-sustaining open source community.

Use-case scenarios

This section will cover some simple use-case scenarios to serve as examples of what kind of advantages we may find in the application of similar technologies. These scenarios will be written in story format, and will be kept concise.

The custom interface

Hacker Joe downloads a debugger for PS2 from the internet. The debugger supports the universal debugger protocol, and comes with a simple debugger interface that he can use to connect to his PS2 from his Windows PC. After using the interface for a few days, he's annoyed by its lack of features. Even though it is an open source interface written in Java, he doesn't like Java and will not add the features himself. Luckily, Hacker Joe knows and trusts C++ and has documentation on the universal debugger protocol (documentation of the spec is a key feature for its use), so he sets out to write his own simple interface.

With the help of the universal debugger protocol documentation and an open source library for it that other users in the community have written, Hacker Joe is able to build his own interface with his own programming language and the help of the community. No Java required.

Choices, choices

Hacker Sue likes debugging GBA. So do a lot of other hackers. This means there are a lot of debugger interfaces she can choose from. When she finds one she likes, she can use it on GBA hardware with her favorite low-level debugger. She can also use it on her favorite GBA emulator with universal debugger support.

Internally, these two low-level debuggers use different means of connecting to the interface (RS232 vs IPC, for example) but neither side knows about it or cares; the universal debugger protocol handles the details almost transparently.

When a newer, prettier, more promising interface comes out, Hacker Sue can switch to it without problems. This is the power of interchangeability and modularity, and she likes it.

One debugger to rule them all

Hacker Tom wrote a universal debugger interface; it supports 32 architectures and runs on 5 different operating systems. When a new low-level debugger is created for a new architecture, Hacker Tom just has to make some small adjustments to his One Debugger to Rule Them All; maybe add a new disassembler, and he's able to use all of his spiffy debugging features on a 33rd architecture. All in about an hour.

Leveraging existing technology

Hacker Kim is writing a low-level debugger for i686. She heard about the universal debugger protocol, and decides to give it a try. She uses an open source library to handle the universal debugger protocol, because this is the easiest solution. The existing debugger interfaces for i686 are perfect; she doesn't have to write her own interface, so she can spend more time on her low-level debugger. She's also very impressed when she discovers she can connect to her debugger over TCP/IP without making any changes to the source. This allows her debugger to be self-hosting; capable of debugging a second instance of itself on a sandbox machine, without interfering with her development machine.

Automagic rollover

Hacker Ted spends months perfecting a low-level debugger he has been adding to a fast, lightweight NES emulator. Rather than write the debugger interface with an OS-dependent user interface API, he gets his friend, Hacker Liz, to write the interface as a separate application. They decide to use the universal debugger protocol to communicate between the two applications.

After a few years, it becomes apparent that Hacker Ted's choice of emulator wasn't the best foundation for his debugger; the emulator runs fast and generally works well for most NES games. But because of some subtle inaccuracies in emulation, there are a few very challenging problems that cannot be debugged. Hacker Ted needs to optimize his homebrew NES game to perform complex artificial intelligence, special video tricks which rely heavily on timing the assembly code exactly in sync with the PPU, and many other fancy effects. Unfortunately, the emulator he is doing his debugging in is not quite accurate enough to run his homebrew game exactly like it does on actual NES hardware, so debugging it is impossible.

Luckily, by this time there are several new NES emulators available which are considered super-accurate (at the cost of blazing fast emulation speed). Hacker Ted decides it is worth it to take the speed hit, so he sets about porting his low-level debugger to the new emulator. He's surprised to find that Hacker Liz's interface works beautifully with the new low-level debugger and emulator, and he's now able to find his bug thanks to the super-accurate emulation. Even better, he can switch back to the old emulator when emulation speed is more important than accuracy, and he still has the same pretty debugger interface, no matter where he goes.

Project implementation

The implementation of the project will be the final say in how well these ideas work in practice. There is a great deal of work to be done until any of this can be properly realized. And I understand that it will be up to me to do the initial foundational work in implementation; the programming. This section will cover the details and specifications of the project implementation. That's a fancy way of saying I will use this portion of the document to guide you through the stages from preparation to having a useful program.


This very document is an integral part of the preparation for a fairly large-scale project. I intend to use the material here as a general guide for the project as a whole. It may not cover every situation and problem encountered along the way, but it should contain enough information to keep the project on track through the initial development stages and into the stable development stages. I will also include some thoughts on the future of the project; what I expect to see accomplished possibly far in the future after the code base has grown and had time to mature.

One (of many) of the goals of this document is to gather resources and information from community members who may hold a valuable interest in the Universal Debugger Project's goals. One such resource is the collective brainpower of today's debugger developers. The immediate use of this collective brainpower would be for laying down the groundwork for the series of standards that the rest of the project will comply with and further be based upon. I want this collective brainpower to be assembled as a standards body independently and neutrally of the main project (and its implementation) in order to provide a constructive neutral territory where the standards body may collaborate without prior reservations or bias.

In preparation for this project, I believe it is essential that a group of volunteers or "elected" members of a standards body assemble to define the series of standards necessary to make modular debugging and interoperability possible. This should be considered a top priority considering the success of universal debugging depends on it entirely. However, the programming work on a universal debugger interface can be done in parallel to, or entirely without, a standards body working out the finer details.

Apart from the documentation guidelines and definition of acceptable open standards in debugging, preparation for this project also has dependencies on the base platform we will be working with. The Mozilla platform, XULRunner, has been chosen as the base for the project's debugger interface. Several potent references exist for the Mozilla platform, including a complete online book [10]. Developers interested in the project's implementation should familiarize themselves with these and other resources.

Familiarity with the following technologies (as applicable to the Mozilla platform) would be beneficial: Javascript, CSS, XHTML, XML, XUL, XBL, XPCOM, C++.

CSS, XUL, and XBL (the latter two being dialects of XML) are useful for building user interfaces on the Mozilla platform. Javascript handles functionality on the user interface elements, and handles data to/from XPCOM. XPCOM is a means to allow Javascript to communicate with (and directly run) functions compiled in C++. Finally, CSS, XHTML, and Javascript are useful for user interface elements rendered with Gecko.

A particularly good example of using XHTML in a Gecko browser widget is with a disassembler: assembly syntax can be highlighted and styled with CSS, subroutines can be contracted and expanded (code folding) with CSS and Javascript. And the entire document can be parsed as an XML document to re-assemble the code at a later time, or to export it to a txt file. Anchors (links) to subroutines and text strings (cross-references) can be clicked on to instantly jump to the relevant code. The browser can be put into "editing mode" to add comments, change instructions, and define new subroutines. Code flow analysis can be visually shown as Bezier curves with arrows pointing to the relevant instructions, drawn within a canvas element. And as with any [X]HTML document, the CSS can be changed to fit the user's tastes.

This use of the Gecko layout engine provides some unique advantages: First, it makes the disassembler super-portable; no need to rewrite platform-specific custom controls that do difficult text layout and line drawing code, not to mention making it all editable. Second, it provides the perfect means to be styled by the user (CSS) any way she likes, and it will always look the same, no matter which operating system it runs on (with the exception of, perhaps, which fonts are installed and which are used by the CSS). Finally, it provides dynamic control over the content via Javascript and DOM manipulation, once again in a completely platform-independent way.

The final element of preparation is defining the naming conventions. It would be unwise to stick to "Universal Debugger Project" because as Lazy|Bastard of GSHI publicly points out, the acronym UDP is already synonymous with the popular User Datagram Protocol. He goes on to suggest a surrogate for the term "universal" (which is inaccurate at best; think "universal remote") with the more appropriate term "versatile". The naming of the project and protocol should be independent and chosen by their respective core contributors. Scalable Remote Debugger Protocol is the name chosen for the protocol.

The project and protocol will each have a healthy start without settling on a name early on, but I might recommend that, at the very least, temporary code-names are chosen so that they may be referred to during the early development stages. I have no reservation on code-names, so any suggestions would be appreciated. As a final word on project naming, I realize that brands are important. Having an easily recognizable and understandable name will become a priority as the project starts coming into fruition.


Initial development requires a great deal of effort. First, I will need a version control system that can handle a fairly large project with a fairly large number of developers. I've decided to go with Mercurial.

I would then devise some coding style guidelines (probably to be hosted on this wiki). Which would define the manner that code is written for the project. It would cover things like indentation style, tab widths, naming conventions... The general idea is to keep the code readable, manageable, and maintainable for all contributors. This document would possibly look similar to the style guidelines I previously wrote for the EnHacklopedia project [11].

Also of importance would be developer- (and eventually user-) documentation. This could also live on the wiki, and would be easy for others to contribute to.

Next I would provide bug tracking software for users and developers to file bug reports, feature/enhancement requests, and a means for developers to track the status of their own points of focus on the project. I've decided on Trac to track the project.

Finally, it would be a matter of hacking away at the project and making an initial commit to the version control repository. From there, provide a few core developers with commit access and watch the project begin to evolve into a tangible and proven debugger interface.



  • Communications Layer - The second part of a two-tier communication system between low-level and high-level debuggers. This layer defines the communications link used between the two, as well as the protocol in which they speak.
  • Communications Link - A physical wire or wireless process for transmitting data.
  • Debugger - A generic term for any program which can be used to aid in the study of the low-level inner workings of another program.
  • High-Level Debugger - Portions and features of the debugger interface which, upon acting on data from the low-level debugger, can perform high-level tasks outside of the scope of the low-level debugger or communications layer.
  • Host - The computer and operating system hosting the debugger user interface. Can also refer to an emulator or similar software which explicitly hosts a target.
  • Low-level Debugger - Usually a very small program (debugger stub) either A) compiled into a target program, B) hooked into a precompiled target program with simple runtime patches, or C) compiled into a host program, for example an emulator which supports debugging its emulated programs with universal debuggers.
  • Protocol - Describes a specific interpretation for data transmitted over a communications link.
  • Realtime Debugger - A high-level debugger which either itself runs a target, or watches the target run from outside of its own context. The realtime debugger typically has full control over its target, including the data and code of the target itself, as well as when and how the target is run.
  • Recursion - See Recursion.
  • Remote Debugger - A target-specific set of low-level and high-level debuggers with each debugger living on different target and host machines.
  • Standard - A formal technical description, usually written by an independent and neutral group or organization, with the intention of providing a set of mandatory and optional "rules" to be used for the purpose of compatibility. When used by the vast majority, a standard becomes a De-Facto standard.
  • Standards Conformance - A program which claims to be in conformance with a standard has conformed its non-standard implementation to be compatible with the standard; the program does not fully comply with the standard, but is compatible.
  • Standards Compliance - Any program which claims to be compliant with a standard is said to have implemented the standard to the best of their ability; the program complies with the standard as-is.
  • Static Debugger - A program which "debugs" a target without actually running or watching a running target. This is typically a simple disassembler or other kind of high-level view of the data contained within the target.
  • Target - The computer or device "targeted" by the debugger. This machine often runs a special low-level debugger program to perform debugging actions. The machine's hardware may or may not be emulated.
  • Universal Debugger - An all-encompassing term to describe the three main pieces of the project: The user interface/high-level host debugger, the low-level target debugger, and the standardized communications layer/universal protocol which they use to speak to one another. Also describes a target- and host-independent series of low- and high-level debuggers which may be used interchangeably anywhere that the universal protocol is supported.
  • Universal Debugger Project - An initiative to assemble a community willing to pursue the research and development of ideas and standards described within the Universal Debugger Project documentation.
  • Universal Debugger Protocol - An open standard definition language to facilitate the use of interchangeable low-level and high-level debuggers; any debugger (high- or low-level) which speaks this language may interact with any other debugger to provide useful results to its users.
  • Universal Protocol - See Universal Debugger Protocol.

See also