Scalable Remote Debugger Protocol

This page is currently serving as a reference to kick-start development of the universal debugger protocol which will be used by the Universal Debugger Project and hopefully many, many other debuggers and debugger interfaces in the years to come.

References

These references are listed in order of relevance; most relevant first.

1. RFC-909: Loader Debugger Protocol
2. GDB Remote Serial Protocol
3. RFC-643: Network Debugging Protocol
4. IEN-158: XNET Debugging Protocol
5. DBGp: A common debugger protocol for languages and debugger UI communication

The relevancy I've determined for this list is due to interest in these specs, as well as potential generic uses and protocol extension.

RFC-909 is so far the closest thing I have found which resembles the general idea I have for a "Universal Debugger Protocol". It's composed as a simple binary packet, it's extensible, and it's designed to be stacked on top of existing transport protocols such as TCP/IP. I doubt this exact spec will fit all of our needs, but it is certainly a good start.

GDB provides a fairly popular protocol. This one is designed for serial communications, so it will work well with small embedded devices. But it could be complicated to extend while retaining its GDB friendliness.

RFC-643 and IEN-158 are interesting only because they show that some experimentation on the ideas of remote debugging have been employed in the past. Unfortunately, these specs were designed for a specific architecture, and are of little practical use for our purposes.

DBGp shows what a modern remote debugging protocol can look like; including modern XML syntax. The downside to this is that low-level debuggers in small embedded devices are unlikely to parse XML at all.

Ideas

This section represents my (Parasyte) own personal opinions and ideas, and should not be taken as advocacy for standardization.

One of the main goals of developing a "universal" protocol for debugging is that it must be usable everywhere; in small embedded devices, and some of the most powerful machines in the world. This kind of flexibility must be designed around multiple layers of abstraction. See OSI Model and Internet Protocol Suite for examples of abstraction layers used in communications technologies.

At the lowest layer, you find the wire; the physical means of transmitting information over distance. For our purposes, we should not limit ourselves to a single wire. Instead, we should allow the use of multiple wires, user-selectable, but never more than one at a time.

The following layers get more and more generic and abstract, until you reach the highest layer which represents what the application sees and interacts with. This would be the "protocol" itself.

So let's break these components down, hypothetically, and get into some details, ordered lowest layer first:

1. Physical layer: Some examples of wires to support include LAN (Ethernet/WiFi), Wireless (Bluetooth), RS-232 (serial port, USB serial port), Inter-Process Communication (Domain Sockets? DBUS?)
2. Transport layer: Some examples of transport protocols include TCP/IP, UDP/IP (LAN, Domain Sockets), UART (RS-232), IPC-specific (DBUS)
3. Application layer: A library (or similar service, E.G. a daemon) to tie all transport layers into a single API that, to the application, looks like one simple interface to connect and send/receive data. The library/daemon will have to handle the transport-specific details behind-the-scenes.

Thinking about this led to a conundrum; If we support multiple wires, we have to support multiple transport protocols which are compatible with those wires. And if we support multiple transport protocols, we have to know which one our target implements. To make the API as simple as possible, we must not force clients to choose from configurable options (for a bad example) that requires a large degree of changes for each different type of connection made. How do we simplify the API so that a user can just plain connect without doing any pre-setup work?

Answer: The URI scheme. The unfortunate downside to this solution is that it is undesired to use URI schemes without registering them with IANA. However, an argument could be made that these schemes would not be used for general network/internet communication. A few popular examples of a similarly non-networked schemes are the file: and about: URI schemes. (The exception here is that at least one physical layer (LAN) could be used for over-the-internet communication; but this has great benefits in its own right.)

The following table represents some examples of how URI schemes could be used as debugger protocols:

srdp://192.168.1.20/	TCP/IP to remote host 192.168.1.20 on a pre-defined default port
srdp+udp://192.168.1.20:9424/	UDP/IP to remote host 192.168.1.20 on port 9424
srdp+usb://localhost/	USB (SRDP-compatible devices) on localhost
srdp+uart://localhost:3/	UART COM port 3 on localhost
srdp+dbus://localhost/	DBUS IPC on localhost

The 'srdp' prefix on these examples is to specify the 'Scalable Remote Debugger Protocol.' The + and suffix defines an additional layer (or protocol) below SRDP.

The latter three examples look a bit odd with localhost being the destination, but this is necessary, since the localhost is the destination for hosting the UART RS-232 port, USB port, and IPC interface. Using non-loopback interfaces (IP addresses outside of the local machine) with these protocols should be undefined, unless there is evidence that connecting to RS-232/USB/IPC interfaces on other machines across a network is practical and plausible.

These URI schemes give a very simple and elegant solution to the concerns they address. No longer will you be stuck with complicated configuration settings like the example below (upper left group box) ... and this is not an incredibly complex configuration dialog, as it is; instead, connecting to ANY low-level debugger in the world will be as simple as typing a URL.

The protocol is defined as a set of usable "commands" or "operations" requested by the client to the debugger, or vice-versa. Operations should be grouped according to a specific metric. The metric I've chosen is hardware (architecture) relationships. The table below shows an example of such groups (current 6 total) and example operations assigned to each group.

1)	Diagnostics (Init, Shutdown, Ping/Pong, Reset, ...)
2)	CPU handling (Get/set process states, register read/write, arbitrary code execution, general CPU control, ...)
3)	Memory handling (read, write, address conversion, hardware I/O, cache control, ...)
4)	Breakpoint handling (add, delete, edit, get, ...)
5)	Stream handling (stdin/stdout/stderr, debugger-specific messages, ...)
6)	Vendor-specific (custom command sets, should be discouraged unless absolutely necessary)

Proposal

This section defines a proposed specification which may be adopted as the "Scalable Remote Debugger Protocol". It is considered a work in progress and is currently open for peer-review, meaning we are interested in receiving comments, criticisms, and suggestions.

Protocol Goals

Goals of the protocol include:

1. Client/server relationship: Target (debuggee) acts as a server, quietly listening for any SRDP requests; User Interface acts as a client, making explicit requests to a listening server.
2. Asynchronous requests: A client must send requests without expecting an immediate response. A server accepting requests may not respond immediately to those requests.
3. Scalable: The data structure (format) used in the protocol must be adaptable to the future; The structure must be as forgiving and dynamic as possible, avoiding fixed contents (except where absolutely necessary) and allowing for [non-mission-critical] non-standard contents.
4. Easy to implement: Basic features of the protocol should be easy to implement from an API point-of-view, as well as having a small memory footprint; the protocol must be usable on small embedded machines with few resources.
5. Robust: Ambiguity should be kept to a minimum in all aspects of the protocol; every bit transferred should have a useful meaning.
6. Easy to debug: A debugger protocol that cannot itself be debugged (observed and verified to work as expected) is a failure in and of itself. For this reason, the protocol should be human-readable in its most basic form.

Underlying Protocols

There are no reservations on any underlying protocols (protocols used to move data from the client to the server, and back again -- SRDP is not one of these protocols). The only requirement is that they provide the hand-shaking and data integrity checking (as applicable). Some examples of suitable underlying protocols include TCP/UDP/IP, and UART.

Protocol Packet Data Structure

The "goals" section outlines the major features which formed the following data structure. Inspiration comes mainly from JSON, the JavaScript Object Notation. As JSON is a serialization [text format] of JavaScript objects, the SRDP data structure is a serialization of the data being transmitted.

The structure also shares some inspiration from RPC; An example is that your client may want to read work RAM from the target. The SRDP request for "read memory" is technically similar to remotely running a "read memory" function on the server, invoked by the client. For this reason, each SRDP packet contains one or more "arguments" which you could imagine are passed directly to a plain old function.

Each packet is sent as a series of 8-bit bytes. Packets are broken down into a series of "arguments". Each argument has a name (made of one or more characters: alpha-numeric, underscore (_), or hyphen (-). The argument name is followed by a colon (:) and then a single byte representing the data type of the argument, then an equals sign (=) and the argument's value. All arguments are "closed" with a semi-colon (;).

The argument syntax is similar to that of CSS. In pseudo-form, it looks something like this:

{name}:{type}={value};

Valid data types:

{type}	Name	Description
n	Number	Any positive integer, encoded as a VLI
s	Signed Number	Any negative integer, encoded as a one's complement VLI
f	Floating Point Number	Any non-integer number, Infinity, or NaN, encoded as a null-terminated UTF-8 string; To be decoded by sscanf
a	Array	Byte-array (binary blob), preceded by a VLI to indicate the length of the array.
c	Compressed Array	Byte-array (binary blob) with RLE compression. See "Compressed Array" [TODO]
t	Text	Null-terminated UTF-8 string without BOM, or null-terminated UTF-16 or UTF-32 with BOM

Some example arguments.

msg:t=Hello, World!␀;

num:n=□;

pi:f=3.141592␀;

ram_dump:a=□■■■■;

my-compressed-data:c=□□■■■■□■□■■■;

What the symbols mean:

 ␀: Null-terminator
 □: VLI
 ■: Data byte

Object Data Type

The basic structure and data types shown so far are very powerful; your server can tell the client vast amounts of information, such as CPU architecture, memory maps, I/O maps, etc. with just a handful of arguments. However, grouping these arguments in a meaningful way may be difficult. You might be inclined to do some "mock" namespacing, like prefixing each CPU-related argument with "cpu_". This is effective, but also slightly wasteful and error-prone.

The "object" data type is designed to handle such situations. This data type allows an argument to be a container for other arguments. Its format looks like this:

{name}:[...];

Where the set of ellipses denotes one or more "regular" arguments. Here is an example of what a "cpu" object might look like:

cpu:[ arch:t=ARM␀; name:t=ARM946E-S␀; ];

(Note the white-space is for readability only; it is not meant to be transferred as part of the protocol data.)

In this example, the "cpu" object contains two arguments: cpu.arch and cpu.name; both strings.

But there are also times when you will want your server to send that same information for two [or more] CPU architectures on a single target. Some platforms may have multiple CPUs, each with its own individual set of resources (memory maps and the like), as well as shared resources between the CPUs. For this, the packet data structure needs a more advanced method of communicating these kinds of details.

For this case, you can optionally create arrays of objects by including a comma (,) after the closing square bracket, followed by another series of arguments enclosed in their own square brackets, ad infinitum.

cpu:[ arch:t=ARM␀; name:t=ARM946E-S␀; ], [ arch:t=ARM␀; name:t=ARM7TDMI␀; ];

Now our "cpu" object defines two CPUs: an ARM9 and ARM7, ready for Nintendo DS hacking. These arguments can be referenced as cpu[0].arch, cpu[0].name, cpu[1].arch, and cpu[1].name respectively.

Objects can be arbitrarily complex, containing arrays of other objects. Here is an example of a simple memory map, containing Work RAM (read/write/execute) and ROM (read/execute) sections. The Work RAM section is broken into two distinct memory ranges. This is quite easily expressed:

memory:[ name:t=Work RAM␀; range:[ start:n=□; end:n=□; ], [ start:n=□; end:n=□; ]; flags:t=RWX␀; ], [ name:t=ROM␀; range:[ start:n=□; end:n=□; ]; flags:t=RX␀; ];