RZX FORMAT

Created by Ramsoft ZX Spectrum demogroup et al.


Format revision: v0.13 (March 2nd 2005)



  1. Introduction
  2. Recording logic
  3. Security
  4. Programming interface (SDK)
  5. RZX file format
  6. License and credits
  7. Revision history


1. Introduction

The RZX format provides a standard method to record input events (such as keypresses and joystick movements) in Spectrum emulators, and store them into files which can be later played back to reproduce exactly the same actions. It is pretty much like recording the script of a movie and then having the actors play it again exactly in the same way each time you watch it. For example, it is possible to record the complete solution of a game or take part in internet game competitions.

The RZX standard has been designed to overcome the limitations of other similar formats like AIR. AIR files became quite popular with RealSpectrum and ASCD, but they had the major disadvantage of non-portability, i.e. the files recorded by an emulator could not be played back by other emulators. This happened because the recording logic was very sensitive to emulation timings, and for several reasons each emulator currently has its own implementantion of the core timings (memory contention and so on). Besides, the closed-source policy of the format actually limited its diffusion further.

The main goals of the RZX project are:
  • Portability.
    Timing-independent recording logic for maximum compatibility with different core implementations amongst different emulators; open-source policy; portable C code using standard library functions (possibly ANSI-compliant) to accomodate different platforms.
  • Easy implementation in emulators.
    Adding RZX support to already existing emulation cores simply requires a few minimal changes. The programmer is not required to change the way the emulator reads the keyboard and joystick by adopting an invasive application interface. The internal state variables of the RZX logic are very limited and it does not cost any performance penalties for the emulator.
  • Reduced file sizes.
    RZX files are compressed in order to be easily distributed and shared on the net. Zlib will probably be used for the purpose because it is highly portable.
  • Robust security.
    Files destined to game tournaments are protected by a robust encoding system to prevent hacking while preserving portability and open source principles.

2. Recording logic

Several theories have been proposed and discussed by many people to obtain input recording files that can be replayed on all emulators. The first way you can think of to achieve that is to count the number of t-states between each keypress; this is obviously the most simple approach, but the resulting files would be replayed correctly only by the emulator used to record them. Why? The ZX Spectrum machines (including the ones produced by Amstrad) are built in a way that access to some memory areas is somewhat "contended" between the CPU and the video circuitry - the latter having precedence as it cannot miss data to build the video signal. This means that when the CPU and the video chip try to access the same memory area at the same time, the CPU is slowed down so that instructions effectively take longer to execute. The mechanism that handles the CPU slowdown is partially documented and since various emulators implement it in different ways, this obviously causes different behaviours and could lead to undesired effects on games where events are generated by pseudo-random counters, as they rely on R register or internal frame counters. For this reason, every technique based on timing events in t-states will fail in terms of portability.

The RZX engine is based on recording the input information at the end of the frame together with the number of CPU instruction fetches that have been performed in that frame. When replaying, the emulator will read both the number of fetches to perform in the frame to be executed and the keyboard/joystick input, which will be valid for that number of instructions, and then will force an interrupt when the limit is reached. In very simple words, the RZX algorithm will say that the input data you read will be valid for a given number of instruction fetches; note that the fetch counter stepping is the same as the R register, i.e. it is incremented by 1 for single-opcode instructions and by 2 for double-opcode ones. The interrupt acknowledge cycle is NOT considered a regular instruction fetch, and thus it does not increment the fetch counter. This method ensures absolute compatibility between emulators, no matter how accurate they are. Besides, no rules are imposed concerning the way the emulator which records the RZX acquires and updates the keyboard and joystick data, since the RZX logic guarantees that the same input sequence will be perceived also by the playback application - and that's the only thing that actually matters.

The following diagrams show what the emulator should do in order to take advantage of the new recording format:



The actual input data recording is achieved by logging the result of all the I/O port reads performed by the CPU in a frame. During input recording, the RZX system records the values returned by the IN instructions executed by the CPU; when the RZX file is replayed, these values are given back to the CPU so that the port reads will return the same results as occurred during recording. The RZX design ensures that each IN instruction gets exactly the right value as expected.
This technique not only achieves input recording functionality, but it also solves many problems arising from the differences between emulators. The most important of these issues is the floating bus, i.e. the values fetched from the bus when a non-existent I/O port is read by the CPU. Other situations covered are tape and disk loading (e.g. with multiload games) and accesses to I/O devices in general. For example, any emulator can correctly replay an RZX file where the Kempston mouse and the lightgun have been used - no matter if it doesn't emulate these devices at all!

Critical rules

  • Good CPU emulation.
    A faithful emulation of the Z80 is a basic requirement for the RZX engine to work properly. As you might imagine, the timings aspect is not important: we said that the RZX format doesn't care about that, neither contention nor exact instruction duration. A "proper" emulation must instead cover all known CPU aspects, both documented and undocumented: a missing flag alteration can result in a different branch of a program on some cases, thus leading to different behaviours when replaying a file on different emulators. A famous example is the rhino's behaviour in Sabre Wulf, as reported even in the Z80 emulator documentation - quoting:

    "The rhino in Sabre Wulf walks backward or keeps running in little circles in a corner, if the (in this case undocumented) behaviour of the sign flag in the BIT instruction isn't right. I quote:

    AD86    DD CB 06 7E        BIT 7,(IX+6)
    AD8A    F2 8F AD           JP P,#AD8F
    
    An amazing piece of code!"

  • INT retriggering
    This is a quite rare event: it usually happens when the handling routine enables interrupts immediately (or a few t-states) after it has been called. The INT retriggering probability varies between the various Spectrum models and clones depending on the INT signal duration and might hardly affect correct playback; a possible solution to avoid ambiguities could be to store the INT signal duration in the recording block header so that the emulator can handle it on its own. Unfortunately this is not enough when working on machines that have a different contention mechanism, for example the ZS Scorpion: in this case memory contention is applied at anytime and not only when building the video output, so delay can occur even around the frame-cross instant altering the retrigger probability. The only way to have a perfect trace of what has happened at record time is to insert a frame record even when a retrigger occurs.

    Example: the Z80 is set on IM2, interrupts are enabled and emulator sets INT low for 48 T; the interrupt handler routine is a simple EI - RET.

                            T-STATES   Instruction  Cycle
                            =============================
                            - 70906:        NOP     (t0)
                            - 70907:         *      (t1)
    INT signal goes LOW ->  -     0:         *      (t2)
                            -     1:         *      (t3) <- INT is sampled here
                            -     2:        INT acknowledgment: IFFs are reset
                            ...             (19 T-states for IM2)
                            -    21:        EI      (t0) <- the INT handler
                            -    22:         *      (t1)    sets IFFs immediately
                            -    23:         *      (t2)
                            -    24:         *      (t3) <- INT is not sampled!
                            -    25:        RET     (t0)
                            -    26:         *      (t1)
                            ...
                            -    34:         *      (t8)
                            -    35:         *      (t9) <- INT is resampled here!
                            -    37:        INT acknowledgment IFFs are reset
                            ...             (again 19 T-states)
    INT signal goes HIGH -> -    48:         *           <- 11th T-state of INTACK
                            ... 
                            -    56:        EI      (t0) <- INT handler restarts
                            -    57:         *      (t1)    
                            -    58:         *      (t2)
                            -    59:         *      (t3) <- INT is not sampled!
                            -    60:        RET     (t0)
                            -    61:         *      (t1)
                            ...
                            -    68:         *      (t8)
                            -    69:         *      (t9) <- INT is resampled here!
                            ...             Program continues normally
    
    In this case the interrupt routine is called 2 times in a frame; it's obvious that if the INT signal is low for 69 T-states or longer, the routine will be triggered 3 times or more.

    The RZX algorithm covers this particular case without any problem, and there's no need to include any information about the INT signal as every INT trigger will cause another record write in the file: for each possible branch in the program caused by an INT the algorithm will provide full information so that the emulator will reproduce the exact instruction sequence. If needed, emulators can detect INT-retriggered frames by checking whether the instruction counter is too small for a regular frame; for the Spectrum computers, we suggest to check for a counter value less or equal to 4.
  • 3. Security [obsolete info, needs updating to DSA]

    One of the main applications for the RZX format is represented by game competitions. In such events, players record their attempts at the assigned games into RZX files, which are then submitted to the competition jury for evaluation and ranking. It is easy to see that security is a key feature in order to prevent any sort of cheating and to ensure fair play in the competition. So, two classes of RZX files exist: protected (to be used whenever security is required, e.g. for game competitions) and unprotected (for general purposes like recording game solutions, etc). This chapter describes the security mechanism used in protected RZX files.

    As far as RZX security is concerned, the main points are:
    • Use of encryption techniques to protect against file hacking.
    • Possible control over the number of legal attempts.
    • Additional checks such as emulation speed violations, autofire detection, etc.
    To ensure data protection despite the encryption algorthm being available in source code form, we use a combination of symmetric and asymmetric cryptography. In the next paragraph we recall a few basic concepts about cryptographic algorithms, in very simple words:
    • In symmetric algorithms, the same key is used both for encryption and decryption. A very simple example is XOR scrambling, where the data to protect is bitwise XORed with a certain sequence of bits (bytes) of a given length. This approach isn't suitable for our purpose alone because the key would necessarily be public and so it could be easily used for hacking. A popular symmetric algorithm is DES.
    • In asymmetric algorithms, a couple of complementary keys is used; the public key is made freely available to the public, while the private key is not shared. Data encrypted using a certain public key can be decrypted only with the corresponding private key, meaning that only the owner of that private key can read the message. The only possible attack is brute force (i.e. trying to guess the secret key by exhaustive enumeration), which takes ages even with the most powerful supercomputers available today.
      The drawback is that these techniques are usually computationally intensive, so they are not suited for a real-time environment such as an emulator. A famous asymmetric algorithm is RSA.
    In a typical RZX scenario, the competition jury generates a couple of asymmetric keys, referred to as the public key (p-key) and the secret key (s-key). The public key is made available for download on the competition website, while the secret key is kept in the vault. Besides, another couple of asymmetric keys is associated to each emulator.

    Now let's see how the RZX security system actually works. To ensure good performances, we use a symmetric algorithm to encrypt the input recording data, e.g. XORing the data with a 128-bit key (x-key) with a few additional tricks to increase confusion. Whenever a new RZX file is created, the emulator generates a new random x-key to be used for that file only (session key); then it uses the p-key of the competition (previosely downloaded by the player) to encrypt the x-key using an asymmetric algorithm, and then the encrypted x-key is finally stored into the RZX file itself.

    Mathematical properties of asymmetric algorithms guarantee that only the owner of the s-key (i.e. the jury itself) will be able to decode the x-key contained into the RZX file, which is necessary to decrypt the input recording data. The encrypted RZX file is unreadable for anyone but the jury, and of course any attempt to edit the file results in irreversible data corruption. If desired, the jury can subsequently unlock the received files so that they can be redistributed and played back by anyone; this is accomplished by means of a library function which simply decrypts the x-key contained into the RZX file, so that the s-key will not be necessary anymore.

    The next step is to guarantee that the encrypted RZX file has been generated by a trusted copy of an emulator. Some sort of authentication is required for the source emulator too, so that the competition jury can be sure that the received RZX file has not been illegally produced by, say, a custom (unofficial) build of an open source emulator, specifically modified to allow cheating. This is accomplished by means of electronic signature. The trusted emulator signs the RZX file by computing a hash function (such as MD5) over the original RZX data; this hash value is then encrypted using the secret key of the emulator and stored into the RZX file. When the recording is examined by the jury, the RZX library decrypts the file, recomputes the hash value for the received data and compares it with the one stored by the emulator. If they are the same, the RZX file is certified to have been produced by the trusted emulator, and its contents is unaltered, otherwise it's a fake.
    The weak point is that the no safe vault exists for the emulator to keep it secret key in. The key must be probably stored inside the program itself, so it is potentially vulnerable to reverse engineering of the binary files. Once the malintentioned user obtains the secret key of the emulator, nothing can prevent him to produce arbitrary RZX files which cannot be distinguished from trusted ones. Unfortunately it is very difficult, if not impossible, to give general directions to emulator authors to prevent reverse engineering. All the best must be done to make it as difficult as possible.

    A detailed proof for the method exposed involves several mathematical theorems concerning cryptography, and so it goes beyond the purpose of this document. Here it will be enough to mention that a similar approach is used in commercial environments and in the PGP package.
    Note that the asymmetric encoding performance is not a problem because the data to encode/decode is very short (the x-key is only a few bytes) and it only happens when the RZX file is opened or created (i.e. outside the emulation runtime).

    In conclusion, the adoption of a mixture of symmetric and asymmetric cryptography ensures good performances and rock-solid security, despite the algorithms used being freely available to everybody. With regards to implementation details, we will provide the full source code of the proposed scheme, including the symmetric/asymmetric algorithms and key generators.

    In some cases the competition may allow a maximum number of attempts for each player. The RZX system uses an unique session ID for each recording.

    Additional security features offered by the library are:
    • Speed violation detection: the emulator ensures that the emulation speed stays within the allowed range, e.g. 50 +/- 5 fps (10% tolerance).
    • Autofire detection: an heuristic algorithm is used to detect the use of autofire.
    Some of these features will be available to juries only, e.g. the autofire detection. Such features will be enabled by the presence of a special s-key on the system released by emulator authors.

    4. Programming interface (SDK)

    The RZX SDK consists in an open-source library written in portable C code providing all the necessary functions to add RZX support to your emulator. It is made available as a fast way to get RZX functionality in your work, but of course you can use your own implementation of the RZX system, if you prefer.
    The latest version of the SDK can be downloaded from the RZX official homepage. It has been tested on GCC (Linux and DJGPP) and Visual C++ 6. The code is endian-independent, so it will work on non-Intel platforms too. Check out the include file for some compile options.

    int rzx_init()
    Initializes the RZX library. It must be called before any other RZX function.

    int rzx_update(rzx_u16 *icount)
    In playback mode, this function is called to start a new frame (the new icount is supplied). In recording mode, it writes the current frame (which is icount instructions long) to the RZX file and it is called when an interrupt occurs (or it would occur, if maskable interrupts are disabled).

    rzx_u8 rzx_get_input(void)
    Supplies the result for IN instructions performed by the CPU. This function must be called in playback mode only.

    void rzx_store_input(rzx_u8 value)
    Writes the result returned by a CPU IN instruction to the RZX file. This function must be called in recording mode only.

    int rzx_close()
    Closes the RZX file. Must be called when the recording is finished. It is called automatically at the end of playback.

    int rzx_add_snapshot(const char *filename, const dword flags)
    Stores a snapshot into the RZX.

    5. RZX file format

    Legenda
    WORD2 bytes
    DWORD4 bytes
    BYTE[N]N bytes
    ASCII[N]N ASCII characters
    ASCIIZ[N]ASCII string with zero-padding to N bytes total
    All multi-byte values are stored in Intel byte order (little-endian).
    All reserved or undefined bits must be set to zero.
    All the headers fields must be filled in; blank values are not allowed.


    RZX Header
    RZX global file header - status: required
    Offset Value Type Description
    0x00 "RZX!" ASCII[4] RZX signature (hex 0x21585A52)
    0x04 0x00 BYTE RZX major revision number
    0x05 0x0D BYTE RZX minor revision number
    0x06 - DWORD Flags
    b0: if set, all data following the main header and up to the Security Signature Block is signed.
    0x0A - - RZX blocks sequence


    Creator information block
    Information about the program which created the RZX - status: required
    Offset Value Type Description
    0x00 0x10 BYTE Block ID
    0x01 29+N DWORD Block length
    0x05 - ASCIIZ[20] Creator's identification string
    (e.g. "RealSpectrum")
    0x19 VMAJ WORD Creator's major version number
    0x1B VMIN WORD Creator's minor version number
    0x1D - BYTE[N] Creator's custom data (may be absent)
    The identification string is freely determined by the emulator's author.


    Security information block
    Security parameters for the following recording blocks - status: optional
    Offset Value Type Description
    0x00 0x20 BYTE Signature information block ID
    0x01 13 DWORD Block length
    0x05 - DWORD Key ID
    The lower 32 bits of the DSA public key value y
    0x09 - DWORD Week code (e.g. for game tournaments)
    As a general rule, data are encrypted after compression.


    Security signature block
    Signature block for the following recording blocks - status: optional
    Offset Value Type Description
    0x00 0x21 BYTE Signature block ID
    0x01 5+LR+LS DWORD Block length
    0x05 - BYTE[LR] Parameter r of the DSA signature
    The value is stored in OpenPGP format, see RFC 2440 section 3.2 (multi-precision integers)
    0x05+LR - BYTE[LS] Parameter s of the DSA signature
    The value is stored in OpenPGP format, see RFC 2440 section 3.2 (multi-precision integers)


    Snapshot block
    Snapshot block - status: optional
    Offset Value Type Description
    0x00 0x30 BYTE Snapshot block ID
    0x01 17+SL DWORD Block length
    0x05 - DWORD Flags
    b0: External Data (this block contains a snapshot descriptor structure instead of the snapshot image).
    b1: Compressed data.
    0x09 - ASCIIZ[4] Snapshot filename extension ("SNA", "Z80", etc).
    0x0D USL DWORD Uncompressed snapshot length (same as SL is the snapshot is not compressed).
    0x11 - BYTE[SL] Snapshot data or descriptor.


    Snapshot descriptor
    Payload of the Snapshot Block when the External Data flag is set.
    Offset Value Type Description
    0x00 - DWORD Checksum (0 = ignore)
    0x04 - ASCIIZ[N] Snapshot filename


    Input recording block
    Actual input recording data - status: required
    Offset Value Type Description
    0x00 0x80 BYTE Recording block ID
    0x01 18+? DWORD Block length (with compressed data)
    0x05 NF DWORD Number of frames in the block.
    0x09 0x00 BYTE Reserved.
    0x0A - DWORD T-STATES counter at the beginning
    0x0E - DWORD Flags
    b0: Protected (frames are encrypted with x-key).
    b1: Compressed data.
    0x12 - FRAME Sequence of input frames (compressed)


    I/O Recording frame
    Layout of the input log for this frame - 4 + ?
    Offset Value Type Description
    0x00 IC WORD Fetch counter till next interrupt (i.e. number of R increments, INTA excluded)
    0x02 IN WORD IN counter. Number of I/O port reads performed by the CPU in this frame (their return values follow). If equal to 65535, this was a repeated frame, i.e. the port reads were exactly the same of the last frame.
    0x04 - BYTE[IN] Return values for the CPU I/O port reads.


    To be completed (Session Information Block, Comment Block).

    Blocks are processed as commands, e.g. it is possible to include multiple recording blocks separated by snapshot blocks to deal with multiload games, or change the security parameters, etc. Everything is handled transparently by the RZX library functions.

    6. License and credits

    The RZX format is currently maintained by Ramsoft.
    The latest version of this document and the RZX SDK can be found at: License

    Work in progress.

    Useful links and references for emulator authors

    Thanks

    Work in progress.

    7. Revision history

    Revision 0.13 (March 2nd 2005)
    • Added the security blocks, as already implemented in several emulators.
    • A few corrections and precisations along the document.
    Revision 0.12 DRAFT (Jul 30th 2002)
    • Changed the idle frames marker from INcount=0 to INcount=65535.
    • Added a note about the R increment during interrupt ackowledge, which must not be counted.
    Revision 0.11 DRAFT (Apr 14th 2002)
    • Experimental compression routines using ZLIB. Compression is used for recording data and embedded snapshots. See the source code for comments and various defines. No error checking yet!
    Revision 0.10 DRAFT (Apr 6th 2002)
    • Changed the Frame Recording data format completely. Now it logs the I/O port read values instead of storing the keyboard and joystick array; this is a more general approach which covers a lot of nasty situations like the floating bus, tape/disk loading and possibly supports other I/O devices.
    • Preliminary API description. Added new comments and explanations throughout all the document.
    Revision 0.02 DRAFT (Feb 14th 2002)
    • Changed the Input Recording Block and Creator Information Block headers slightly.
    • Added support for "idle frames": when bit 15 of the instruction counter is set, the input data for that frame is omitted because it is the same as in the previous one. So, an idle frame takes only 2 bytes in the RZX stream instead of 11.
    • Added electronic signature to the security system.
    • Added a note about floating bus in Chapter 2.
    • Revised Snapshot Block and new Snapshot Descriptor structure.
    Revision 0.01 DRAFT (Feb 2nd 2002)
    • First public release of this document. Largely incomplete and buggy...