Practical Advice for Solving Assembly Puzzles

Setting up your debugging environment

There's two main ways to view the assembly for a compiled program.

The first is to use objdump to extract the assembly. For example, if your compiled program is called nanobomb1, you can dump the assembly to the terminal with the command objdump -d nanobomb1. This spills everything onto the screen which can be kind of unwieldy, so you can try doing objdump -d nanobomb1 > nanobomb1.s to create a file called nanobomb1.s that contains all the assembly. This also gives you the benefit of syntax highlighting.

However, I recommend using the second approach which is to view assembly within gdb as you're debugging. To do so, I would use the following commands:

1. gdb ./nanobomb1 (Invokes gdb onto our program we want to debug.)
2. layout asm (Change the UI so that we can see the assembly as we step.)
3. layout reg (Optional: change the UI to also show the values of registers.)
4. b try_defuse (Put a breakpoint on the function of interest. The function name will be different on assign5.)
5. r (Run the program. On assign5, use r input.txt so that previous passwords are passed along.)
6. Use <CTRL+L> as necessary to repaint the screen to clear any visual artifacts that occur.

Once you've done this, you'll be able to step through assembly code and see what line you're on and the values of registers as you go along. Here are the gdb commands that might be helpful here:

• ni (abbreviation for nexti: like next but for assembly)
• si (abbreviation for stepi: like step but for assembly)
• b *0x12345 (abbreviation for break *0x12345: put a breakpoint on a particular line of assembly with address 0x12345)
• c (abbreviation for continue: continues running until next breakpoint)
• p $rax (abbreviation for print $rax: print the contents of register %rax)

General Format of an Assembly Puzzle

CS 107A nanobombs and assign5 generally follow this structure:

• During the execution of the program, input is read in at the command line where you have to enter some text (a "password") to proceed
• Assembly code tests the input to check if it satisfies certain secret requirements
  • For example, perhaps the requirement is that the input is a 5 digit number that is a multiple of 1049.
  • As you discover these requirements, you should write them down! You will forget.
  • assign5 contains a README section that asks you about some of these. Fill it in as you do the assignment.
• If the password fails a requirement, an explode function is called
• If the password passes all requirements, a defuse function is called

General Strategy for Approaching Assembly Puzzles

• First, find all the key components of the assembly code, including any calls to explode or defuse and any jumps
• Trace through all the jumps, perhaps dividing the code into blocks, in order to get a high-level idea of the control flow of the program
• Determine what jumps need to occur in order to call the defuse function
• For each necessary jump, determine what requirement it imposes on the input
• Each time you adjust your theory about what requirement a piece of code imposes, test and confirm it by stepping through the program on an input
• When you pass all requirements, you pass the level

General Advice for Working with Assembly in gdb

• If you want to skip ahead to a future piece of assembly code that you know will run, you can put a breakpoint on it and then use c or continue
• Assembly doesn't have types. You have to guess whether values stored in memory and registers correspond to ints, strings, etc.
  • 1 byte values are probably characters
  • 2 byte values are rare but are probably shorts
  • 4 byte values are probably regular ints
  • 8 byte values are probably either longs or other pointers (maybe even pointers to pointers)
• Try casting unknown values to different types and printing. Sometimes you get something you recognize.
• Some examples of casting in assembly:
  • p (char)$al (Prints the contents of %al as a character)
  • p (char*)$rax (Prints the string pointed to when interpreting $rax as a pointer)
  • p *(int*)$rax (Prints the integer pointed to when interpreting $rax as an int pointer)
  • p *(char**)$rax@5 (Prints 5 elements of a string array pointed to when interpreting $rax as a pointer to an array of char*.