How to Give Cartridge-Based Video Game Data an Extra Life (Part 2)

The following is a guest post by Christopher Fox, a term employee at the National Institute of Standards and Technology (NIST). For over six years, Christopher has provided NIST's National Software Reference Library project with data entry services and the application development demands that support those services. He just completed his senior year at Hood College in Frederick, Maryland, where he earned a Bachelor's in Computer Science. He started working on the video game software part of the Cabrinety Project when the group hit a dead end trying to collect data from gaming cartridges. Since Chris enjoys gaming and found a way to collect save data from older games for his own purposes, he offered to explore and share methods of extracting data from these obsolete formats.

*Disclaimer: Trade names and company products are mentioned in the text or identified. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the products are necessarily the best available for the purpose.

This is a follow up piece to my previous post on how the National Software Reference Library at NIST is collecting data from several different types of gaming cartridges.

Since my last post I extended my research into Nintendo Entertainment System (NES) and Game Boy cartridge data collection. The earlier problem was difficulty finding a device capable of collecting the ROM from these platforms for a wide variety of games.

NES:

CopyNES

I researched the CopyNES previously, but vendors for them were hard to find. Luckily, they recently came back into production at RetroUSB. RetroUSB is a website that specializes in development tools for older console generations such as the NES, SNES (Super Nintendo Entertainment System), and the Nintendo 64.

The CopyNES is a hardware NES tool that allows the user to collect or even write data to a NES cartridge. We were only interested in collecting data, so the ability to write data to cartridges was disabled to avoid any data being written to the cartridges in the collection. In order to install the CopyNES, the CPU of the NES gets taken out and then plugged into the CopyNES. The CopyNES is then placed directly on top of the NES motherboard so that it can interact directly with all the data that the NES reads while still being able to play cartridges. The CopyNES also has a USB socket that allows the NES to be connected to a computer so that the ROM from the cartridges can be collected directly.

CopyNES in system

Whenever you want to collect a ROM from a cartridge you plug in the game as if you were going to play it normally, turn on the NES with the USB attached, and run the program that comes with the CopyNES hardware to download the ROM in a .nes file. This procedure is great for collecting ROMs from individual cartridges but comes with a few problems since we have such a large amount of cartridges in the Cabrinety collection. To explain these problems I will first have to elaborate on exactly how the ROM collection process works with NES cartridges.

NES put back together

Collecting ROM from Gauntlet

The first thing to understand is that every NES cartridge has what is referred to as a Memory Management Controller (MMC) or what is commonly called a mapper. This is a chip that is soldered directly onto the cartridge that performs all the address decoding and bank switching - in other words, supplies information to the NES on how the data should be read off the cartridge. Without knowing which mapper is used for the cartridge, the NES cannot read and play the game and thus the ROM cannot be collected. There are dozens of different mappers for different cartridges made by different companies. Luckily a list of almost all NES cartridges and their respective mappers has been made and released to the public. Once you look up the mapper for a specific cartridge - let’s say, “Gauntlet” - you will see that the mapper used for that game is MMC3. Now you can run the software that came with the CopyNES and enter in the associated mapper to allow the NES to read and collect data from the cartridge.

The problem with this method was the need to look up the mapper for every cartridge before dumping the ROM, which takes extra time and makes the whole process very tedious. Also since the software that came with the CopyNES is a Graphical User Interface (GUI) it is not easy to use the command line to make an automated script of the ROM dumping process. However, since there is a surprisingly large community of people interested in collecting NES ROMs, someone has developed a Linux/Mac OS X based command line program for the CopyNES called Copynesl. The developer goes by the name Crade and the forum in which he writes about his software is located here. Copynesl also requires a library called LibCopyNES which was developed by Wookie who also writes on the same forum. This software allows the user to specify the mapper while running the software, which makes the ROM collecting process fully automated.

However, another problem still exists. Unlike the SNES and Sega Genesis, which have built in anti-piracy checks, the NES has none. The SNES and Sega Genesis calculate the checksum of the ROM using a console specific algorithm and compare that checksum to a value stored in the ROM itself to confirm that all the data is untampered. The NES did not use this method and therefore causes issues when we want confirm if the ROM we have collected is a valid read. Thanks to the large community of fans collecting NES ROMs, someone has already solved this problem. Bootgod has a website that hosts a large database of NES cartridge information that is uploaded and then confirmed to be accurate by others on the site. On this site, you can look up specific cartridges and a lot of detailed information about each cartridge. The information I focused on using was under “ROM Details.”

Cartridge details

In the “ROMs Combined” row, the value under CRC32 is the output of a cyclic redundancy check (CRC). In laymen’s terms this is an algorithm that will output a string 32 bits in length based on every single bit in the file. If a single bit is changed in the data then the CRC32 will change completely. This value can be used in a very similar way to the checksums I mentioned above. Now that we have a confirmation of what the CRC32 value of the ROM should be, we can calculate the CRC32 of the ROM we have collected and compare it against the database to prove whether or not the ROM is valid. However, the value that is in the database is not the CRC32 of the entire ROM file, but just the ROM data itself. The ROM file that is created with the CopyNES includes the ROM data and header data from the cartridge. The header data contains information that is used when loading and playing the ROM, similar to the NES mapper and is 16 bytes in length. Therefore I needed to create a temporary copy of the file but with the header removed and perform the CRC32 on that file and use that value for comparison.

ROM details

New file for CRC32 check

Once I was able to run the cartridge data collection software in a way that could be scripted, and had a way to confirm that the data collected was accurate, I was able to write an automated script for the entire data collection process.

I’ve adapted the same script I used for collecting data for Sega Genesis and SNES to include all the functionality for NES cartridges.

The CopyNES has a feature to put the NES into a state called “Play Testing.” This allows the user to use the NES as a functional gaming system to see if the cartridge is connected properly, and if it will even run in the first place.

Play testing

Gauntlet playing on NES

Once the cartridge is confirmed to be working properly, the user hits “Enter” and the ROM is collected.

I mentioned before that every NES cartridge needs to have the mapper specified ahead of time. However, having the user look up the mapper for every cartridge and enter it in by hand becomes tedious and leaves room for user error. To fix this issue I found a file that comes with the Copynesl software called “Mappers.dat” which has a list of every mapper that the CopyNES currently supports. I then parsed this file to store every individual mapper so they can be added as an argument to the command line tool. The command line tool is run as follows:

sudo ./copynesl –d $plugin –m $mapperNum –o $filename

The argument after “-d” specifies the plugin/mapper. The argument after “-m” specifies the mapper number which is a value that is stored in the header data of the ROM file (that specifies what the mapper type is for that cartridge.) The mapper number is also listed in the mappers.dat file next to its respective mapper. The argument after “-o” specifies what the ROM file will be named after the dump.

Now that I had these values stored, I tried to collect the ROM using the first plugin and mapper number on the list. After that ROM was collected I compared it against the database for a CRC32 match. If it didn’t match I moved on to the next plugin and mapper number on the list. This continued until I got a match. Once I found a match I renamed the file to the correct game title and the program exited.

ROM dump finished

Now that the ROM has been collected successfully it can be opened in an emulator and played just to make absolutely sure that the data is correct.

ROM opened in emulator

Game Boy ROMS:

I talked about my current progress on collecting Game Boy in my last post and some of the problems I was facing at the time. The first device I tried to use to collect the Game Boy cartridge ROM was a Retrode plugin. However, this device could only collect ROMs from a select few games, and I wasn’t able to collect a single ROM from the cartridges that were currently being processed. The Smart Boy was something I started to look into while working on my last post but upon further research discovered that it produced the same problems as the Windows version of the CopyNES software. It is GUI-based so scripting the whole process wouldn’t be possible. Also, the Smart Boy has several options for overwriting the data in the cartridge, and data could be accidentally deleted.

The newest method that I discovered involves using an Arduino and a Game Boy cartridge adapter. An Arduino is a popular microcontroller that uses several pins to send and receive data. I used the Game Boy Cart Shield which attaches to several pins on the Arduino at once and sits on top. This allows all the pins of the Game Boy cartridge to be accessed by the Arduino. The instructions for this are written out in detail on Inside Gadgets’ website in an article by Alex, one of the moderators on the site. I have had great success with using this method to collect both the ROM and the RAM of every Game Boy cartridge I have come across.

Game Boy Cart Shield on Arduino

Game Boy Cart Shield on Arduino

The ROM contains all of the data on the cartridge that cannot be edited and was uploaded to the cartridge by the manufacturer. The RAM contains all the data written to the cartridge by the user. This is the “save” data. The ROM is saved with a “.gb” extension while the RAM is saved with a “.sav” extension. The Arduino collects the ROM from the connected Game boy cartridge and sends the data via USB to the host computer. A program then waits for this data and writes it to a file. The process is repeated for the RAM on the cartridge. However, some cartridges do not contain any save data so dumping the RAM can be skipped.

Pokemon Red plugged into Arduino

Game Boy cartridges can also be tested for accuracy by using an anti-piracy system similar to the ones used for the Sega Genesis and SNES games in my previous post. However, Game Boy cartridges have multiple checks and other means of verification as opposed to just one like the other cartridges. The first thing that gets verified is the “Header checksum.” The algorithm for calculating the header checksum in PERL is as follows:

my x=0; # x is the calculated checksum
for (my $i=0134h;$i<=014Ch;$i+=0001h){ # 0xxxh is a hexadecimal address
x = x - $bytes[$i] - 1; #$bytes is an array of all the bytes in the dumped ROM’s
}

This algorithm goes through the entire header of the ROM and generates a value in decimal. If you convert this value to hexadecimal and then take the lower 8 bits they should be the same as the value stored at 0x014D in the ROM. The next checksum is referred to as the “Global checksum.” The algorithm for this one is relatively simple:

my x=0; #x is the calculated checksum
for (my $i=0;$i<=$file_length;$i++){
next if ($i == 014Eh || $i == 014Fh);
x = x + $bytes[$i];
}

You take every byte from the beginning to the end of the ROM and add it together. However you skip the bytes at addresses 0x014E and 0x014F where the global checksum value is stored. Once you have the sum you compare the answer to the global checksum.
 

Nintendo Logo bitmap

The last form of verification is different from the rest. The Game Boy checks that the ROM of every cartridge contains the “Nintendo Bitmap.” The bitmap is a series of bytes that make up the Nintendo logo that shows up when you turn on your Game Boy with the cartridge plugged in. Once this series of bytes is confirmed accurate the game will continue to be loaded. The Game Boy will cease running if any of these bytes are out of place.

Once I had all of the checks in place I implemented the entire Game Boy ROM/RAM collection process into my existing PERL script.

Pokemon Red ROM read

The first thing the PERL script does is call a C program that tells the Arduino to send some data located in the header of the ROM through the USB to be used by the C program to determine how much data should be received and therefore how much data to write to the ROM/RAM files. The metadata for the cartridge is also printed out for the user. With this information, it then begins the actual process of saving to POKEMON RED.gb/POKEMON.sav. In this case, the metadata states that the ROM size is 1MB so it will dump until 1024K. It also states that there is RAM in this cartridge and it is 32K in length so that will begin next. Once the ROM/RAM has been saved to their respective files, all of the verification methods are used.

Pokemon Red checksum

After the process is complete the ROM can be loaded into an emulator. The emulator will also automatically load the POKEMON RED.sav file as long as it is in the same directory as the ROM file. The save data below is actually from my own cartridge that I played as a kid.

Pokemon Red title screen Pokemon Red save game

Works in progress:

Commodore 64 ROMs:

I haven’t performed extensive research on how Commodore 64 ROMs work yet but I have found a method of collecting the ROM from Commodore 64 that may potentially be of use. It involves using an actual Commodore 64 similar to how we used a NES console for collecting NES ROMs. I would need to take apart a Commodore 64 and re-wire a few connections on motherboard in order to allow the ROM data to be collected through the disk drive using specific programs for each type of Commodore 64 cartridge. Using an Arduino is always an option as well and would involve a similar setup to the Game Boy. I will continue to research possible solutions.