The NandTier SeriesExtra credit

Testing Everything: Testbenches and Simulation

How the project checks every module before it reaches the board: unit testbenches, random programs against a reference simulator, and the whole SoC in simulation.

5 min readTestingSimulationVerification
In this article
  1. What are we trying to do?
  2. The tools
  3. Level 1: one module at a time
  4. Level 2: the core against a reference
  5. Level 3: the whole SoC
  6. Games on the PC
  7. Conclusion

What are we trying to do?

Debugging on the board is slow and blind: when a game does not start, the cause could be anywhere from the ALU to the SD card. So before anything goes onto the FPGA, NandTier tests it in simulation, where every signal of every cycle can be seen. The goal of this article is to show the three levels of tests the project has, and how to run them.

whole SoCsim/: boot, launcher, a game from the SD modelcore vs referencerandom programs vs rv32i_iss.pyone moduletb_alu, tb_uart, tb_tmds …slow, proves it allfast, run always
The three levels: many fast module tests at the bottom, one slow whole-system test at the top.

The tools

Simulation uses Icarus Verilog (iverilog compiles, vvp runs), which comes in the OSS CAD Suite from part 1. Waveforms can be opened in GTKWave. The test Makefile lists every target:

test/Makefilelines 6–20
6#   make            run every test below7#   make unit       one testbench per CPU core module (hw/core/)8#   make periph     UART, GPIO, SPI flash, SD SPI, ILA/JTAG CPU debug and bus decoder9#   make video      TMDS encoder, video timing, text terminal, PPU, ECP5 HDMI output10#   make core       ISA, C and trap programs on nandtier_core, several bus latencies11#   make random     random programs on nandtier_core, checked against the12#                   instruction set simulator core/rv32i_iss.py13#                   (SEEDS="1 2 3", N=<instructions per program>)14#   make sw         sw/nandtier-programmer.py against a simulated boot ROM15#   make game       the platformer's rules, on a PC build with a PPU model16#   make games      every game cartridge on the PC build: plays them with17#                   scripted controls, checks their rules, saves screenshots18#   make carts      the cartridge loader's CRC and the SD cartridge pack format19#   make fw         build both firmware images20#   make clean
InteractiveWhy testbenches care about = and <=: one clock edge in the simulator's event queue.

Level 1: one module at a time

A unit testbench drives one module's inputs and checks its outputs against something known to be right. For the ALU, "known to be right" is Verilog's own operators:

test/unit/tb_alu.vlines 25–39
25function [31:0] expected(input [3:0] op, input [31:0] a, input [31:0] b);26    case (op)27        4'b0000: expected = a + b;28        4'b1000: expected = a - b;29        4'b0001: expected = a << b[4:0];30        4'b0010: expected = $signed(a) < $signed(b) ? 1 : 0;31        4'b0011: expected = a < b ? 1 : 0;32        4'b0100: expected = a ^ b;33        4'b0101: expected = a >> b[4:0];34        4'b1101: expected = $signed(a) >>> b[4:0];35        4'b0110: expected = a | b;36        4'b0111: expected = a & b;37        default: expected = 32'hxxxx_xxxx;38    endcase39endfunction

The interesting values are the edges: 0, 1, all ones, the most negative number, a shift by 31 and by 32. So every operation is tried on every pair of corner cases first:

test/unit/tb_alu.vlines 43–48
43initial begin44    corner[0] = 32'h0000_0000; corner[1] = 32'h0000_0001; corner[2] = 32'hffff_ffff;45    corner[3] = 32'h8000_0000; corner[4] = 32'h7fff_ffff; corner[5] = 32'h0000_001f;46    corner[6] = 32'h0000_0020; corner[7] = 32'h8000_0001; corner[8] = 32'hffff_ffe0;47    corner[9] = 32'h5555_aaaa;48end

then on thousands of random operands. The check task counts errors and prints the first few:

test/unit/tb_alu.vlines 53–63
53task check;54    begin55        #1;56        checks = checks + 1;57        if (y !== expected(op, a, b)) begin58            errors = errors + 1;59            if (errors <= 10)60                $display("  op=%b a=%08x b=%08x: got %08x, expected %08x", op, a, b, y, expected(op, a, b));61        end62    end63endtask

Every module in hw/core/ and every peripheral has a testbench like this. Run them all:

Terminal
$ make -C test unit$ make -C test periph$ make -C test video
InteractiveA tiny waveform viewer on a real simulation dump. Zoom in, move the cursor, read the values.
InteractiveThe same kind of test in Python: 20,000 random ALU operations with cocotb, against a model.

Level 2: the core against a reference

Unit tests check modules; they do not check that the modules together execute RISC-V correctly. For that, test/core runs programs on the whole core. Some are written by hand (test_isa.S goes through every instruction), and some are random: a Python script generates thousands of random instructions, a Python instruction-set simulator (rv32i_iss.py) runs them, and the final registers and memory must match the hardware's.

Terminal
$ make -C test core$ make -C test random SEEDS="1 2 3" N=5000

Random tests find the cases nobody thought of: a branch right after a load into the same register, a shift by a register holding 32, a store to the byte the next instruction is fetched from.

InteractiveBeyond tests: SymbiYosys proofs on the bus decoder and the reset logic, with a counterexample when one fails.

Level 3: the whole SoC

Finally sim/ runs the real SoC top level with the real boot ROM and launcher, a model of the SPI flash and a model of the SD card:

sim/Makefilelines 4–14
4# Whole-SoC simulation (tb_soc.v): the real boot ROM and user firmware on5# hw/nandtier.v, with sim/spi_flash_model.v as the flash.6#7#   make            boot into the firmware, use its menu (several minutes)8#   make offset     the firmware test with the firmware at flash address9#                   0x200000 (FLASH_OFFSET, as on the Tang Nano 20K / ULX3S)10#   make isp        boot ROM ISP: ping, erase, program, restart11#   make carts      the game menu on the RTL: Pong from the SD card model into12#                   the flash cartridge slot and running (tens of minutes);13#                   its first frame must match the PC build's (test/host)14#   make clean

make carts is the full story of this series in one simulation: boot, launcher, game menu, Pong copied from the SD card model into the flash slot, and its first frame compared, pixel for pixel, with a picture made by the PC build of the same game. It takes tens of minutes, because it simulates every one of several hundred million clock cycles.

Games on the PC

The games themselves are tested without any hardware at all. game_hal.h, which we met in part 10, has a second half for GAME_HOST builds: the same game code compiled for the PC, with a software model of the PPU. The tests play each game with scripted key presses, check its rules (an apple makes the snake longer, a wall ends the game) and save screenshots.

Terminal
$ make -C test games

Conclusion

Three levels: modules against simple references, the core against an instruction-set simulator, and the whole system against a PC build of the same games. With them, a change anywhere in the design is checked in minutes, long before it reaches the board, and the board becomes the place where things work rather than the place where they are debugged.