Continued from Enhancing Retro Games - REX...

There needs to be a way for client applications to interact with the emulator and the enhanced features that REX offers. We need to lay down some design goals and constraints first on how our protocol should work:

1. REX server must be easily integrated into an existing emulator's codebase with minimal impact on the build system(s) of that emulator, for all emulators targeted.

2. REX server must be cross-platform, primarily supporting Windows, MacOS, and Linux.

3. REX server must not be tied to any particular emulator's implementation and should remain emulator-neutral.

4. Outside of emulators, subsets of REX server may also be implemented in the FX Pak Pro firmware.

5. REX server should not dictate client application implementation language choices and should not push required dependencies on them.

6. REX server features must be used via a simple command-response paradigm.

7. REX server must be able to notify client applications asynchronously when events occur, such as data-read and data-write completion events, without breaking the normal flow of command-response traffic.

Given these design goals and constraints, I've opted for a custom RPC protocol via TCP built on top of a custom message framing protocol.

Why custom protocols?

Several popular solutions exist such as gRPC and Cap'n Proto RPC but these are heavyweight dependencies requiring complex build system integration which would violate design goal #1 above, as would almost any other existing RPC framework. Embedding source code in C/C++ codebases is (in my experience) always simpler to deal with than requiring external libraries.

gRPC being built on top of http/2 is very much overkill for embedded use in an emulator. The Cap'n Proto RPC protocol looks much simpler and better suited for this use case but its use might hurt the choice of client applications for REX since it's primarily geared towards natively-compiled languages with low-level control over memory layout.

For the FX Pak Pro firmware case, we also need something very lightweight for embedded development.

Protocol Design

It's easy enough to design a simple RPC protocol where the client is always sending commands and receiving replies immediately after. Many such protocols exist. However, the problem comes with the design goal requiring that the server must notify the client of events asynchronously without breaking existing command-response communications.

Essentially, the problem can be summarized as wanting to turn a byte-stream-oriented protocol (such as TCP) into a message-oriented protocol but with at least two streams/channels of messages.

Message Framing

Let's look at the message framing protocol currently in place for REX:

header byte:
  [7654 3210]
   fcll llll   f = final frame of message
               c = channel (0..1)
               l = length of frame (0..63)

followed by `l` bytes of frame data

There are three main concepts:

1. messages

2. frames

3. channels

Every message is broken up into one or more frames.

A frame can be up to 64 bytes in size including the header byte. A frame cannot contain more than 63 bytes of message data.

A message is ended when the final bit of the header byte of the last frame of the message is set to 1. The next message begins with the next frame after a final frame is delivered.

There are two channels available to send and receive messages on: channel 0 and channel 1.

Channel 0 is used for the main RPC command-response messages and channel 1 is used for asynchronous messages sent from the RPC server back to the client. Technically, clients can send messages to the server on channel 1 but these are ignored.

A 0-byte frame is valid because with its final bit set it can be used to signal the end of a longer message whose previous frame was sent without the final bit set but was later determined to be the final frame of the message.

Implementation

The message framing protocol is implemented here. It's a simple design written in plain C for compatibility purposes.

It's broken into two structs, each with its own set of functions for initializing and operating on it:

1. incoming frames

2. outgoing frames

Incoming frames

typedef bool (*frame_incoming_read_more)(struct frame_incoming *s, uint8_t *buf, long size, long *n);
typedef bool (*frame_incoming_received)(struct frame_incoming *s, uint8_t *buf, uint8_t len, uint8_t chn, bool fin);

struct frame_incoming *frame_incoming_init(
    struct frame_incoming *s,
    void *opaque,
    frame_incoming_read_more read_more,
    frame_incoming_received received
);
bool frame_incoming_read(struct frame_incoming *s);

These functions work on incoming frames for reading data from the opposite party. The frame_incoming_init function must be called once before any subsequent calls (per struct frame_incoming *s). It accepts two function pointers: one for a function that reads more data from the opposite party (e.g. read() from a TCP socket), and one that notifies the library user of a received frame's data, what channel it was on, and whether the frame's final bit was set.

The void *opaque argument passed to the init function is for supplying additional context that sticks with the frame handler and can be accessed by the function pointers.

Outgoing frames

typedef bool (*frame_outgoing_write_data_cb)(struct frame_outgoing *s, uint8_t *buf, size_t size, long *n);

struct frame_outgoing *frame_outgoing_init(
    struct frame_outgoing *s,
    void *opaque,
    frame_outgoing_write_data_cb write_data
);

void frame_outgoing_reset(struct frame_outgoing *s);
bool frame_outgoing_send(struct frame_outgoing *s, uint8_t chn, bool fin);
size_t frame_outgoing_len(struct frame_outgoing *s);
bool frame_outgoing_append(struct frame_outgoing *s, uint8_t x);
bool frame_outgoing_append_bytes(struct frame_outgoing *s, const uint8_t *buf, size_t len);

The outgoing frame handling is slightly more complex than incoming frame handling only because we're adding more convenience functions useful to compose frames and messages with.

As with incoming, the initializer function for outgoing frames must be called once before any subsequent call (per struct frame_outgoing *s). It accepts one function pointer to a function that sends a full frame to the opposite party (e.g. send() or write() via a TCP socket).

The void *opaque argument passed to the init function is for supplying additional context that sticks with the frame handler and can be accessed by the function pointer.

RPC Protocol

For REX, I've put together the simplest possible RPC protocol which is built on top of the above message framing protocol.

Command messages are sent by the client to the server on channel 0 and start with a command-type byte and then continue with command-specific data to the end of the message.

Response messages are sent by the server back to the client on channel 0 and start with the same command-type byte, followed by a success/error condition byte, and any further command-specific response detail beyond that to the end of the message. Response messages must always follow command messages to indicate acknowledgment.

Asynchronous event messages are sent by the server to the client on channel 1 and do not interfere with regular channel 0 communication. An event message starts with the event-type byte and continues with event-specific data to the end of the message.

Examples

9B 00 04 00 01 10 05 10 00 36 00 FF 34 43 31 00 35 06 00 39 9C 00 2C 6C EA FF 3A 08

Here we have an example of a framed message that is sent from the client to the server which uploads an IOVM program.

The first byte is the framing header byte 9B which has the final bit set, uses channel 0, and has a length of 1B bytes.

The next byte is the RPC protocol command byte 00 which is the rex_cmd_iovm_load command.

All subsequent bytes are the IOVM program opcodes and data which are broken down in the previous post.

82 04 F0
81 03
81 01

Here are three framed messages which are simple commands to control IOVM execution.

The header byte for all three messages is similar: all have the final bit set, use channel 0, and specify the length of the frame.

The first message is two bytes long and the RPC command byte is 04 which is rex_cmd_iovm_flags to set execution flags for IOVM that control things like which opcodes and stages to send notifications about and whether to auto-restart the program on error and on success. In this case, both the auto-restart flags are set.

The second message is one byte long and is the RPC command byte 03 which is rex_cmd_iovm_reset to reset the IOVM program to the beginning.

The last message is one byte long and is the RPC command byte 01 which is rex_cmd_iovm_start to start the IOVM program running.