Testing Everything: Testbenches and Simulation
How the project checks every module before it reaches the board: unit testbenches, random programs against a reference simulator, and the whole SoC in simulation.
In this article
What are we trying to do?
Debugging on the board is slow and blind: when a game does not start, the cause could be anywhere from the ALU to the SD card. So before anything goes onto the FPGA, NandTier tests it in simulation, where every signal of every cycle can be seen. The goal of this article is to show the three levels of tests the project has, and how to run them.
The tools
Simulation uses Icarus Verilog (iverilog compiles, vvp runs), which comes in the OSS CAD Suite from part 1. Waveforms can be opened in GTKWave. The test Makefile lists every target:
6# make run every test below7# make unit one testbench per CPU core module (hw/core/)8# make periph UART, GPIO, SPI flash, SD SPI, ILA/JTAG CPU debug and bus decoder9# make video TMDS encoder, video timing, text terminal, PPU, ECP5 HDMI output10# make core ISA, C and trap programs on nandtier_core, several bus latencies11# make random random programs on nandtier_core, checked against the12# instruction set simulator core/rv32i_iss.py13# (SEEDS="1 2 3", N=<instructions per program>)14# make sw sw/nandtier-programmer.py against a simulated boot ROM15# make game the platformer's rules, on a PC build with a PPU model16# make games every game cartridge on the PC build: plays them with17# scripted controls, checks their rules, saves screenshots18# make carts the cartridge loader's CRC and the SD cartridge pack format19# make fw build both firmware images20# make clean= and <=: one clock edge in the simulator's event queue.Level 1: one module at a time
A unit testbench drives one module's inputs and checks its outputs against something known to be right. For the ALU, "known to be right" is Verilog's own operators:
25function [31:0] expected(input [3:0] op, input [31:0] a, input [31:0] b);26 case (op)27 4'b0000: expected = a + b;28 4'b1000: expected = a - b;29 4'b0001: expected = a << b[4:0];30 4'b0010: expected = $signed(a) < $signed(b) ? 1 : 0;31 4'b0011: expected = a < b ? 1 : 0;32 4'b0100: expected = a ^ b;33 4'b0101: expected = a >> b[4:0];34 4'b1101: expected = $signed(a) >>> b[4:0];35 4'b0110: expected = a | b;36 4'b0111: expected = a & b;37 default: expected = 32'hxxxx_xxxx;38 endcase39endfunctionThe interesting values are the edges: 0, 1, all ones, the most negative number, a shift by 31 and by 32. So every operation is tried on every pair of corner cases first:
43initial begin44 corner[0] = 32'h0000_0000; corner[1] = 32'h0000_0001; corner[2] = 32'hffff_ffff;45 corner[3] = 32'h8000_0000; corner[4] = 32'h7fff_ffff; corner[5] = 32'h0000_001f;46 corner[6] = 32'h0000_0020; corner[7] = 32'h8000_0001; corner[8] = 32'hffff_ffe0;47 corner[9] = 32'h5555_aaaa;48endthen on thousands of random operands. The check task counts errors and prints the first few:
53task check;54 begin55 #1;56 checks = checks + 1;57 if (y !== expected(op, a, b)) begin58 errors = errors + 1;59 if (errors <= 10)60 $display(" op=%b a=%08x b=%08x: got %08x, expected %08x", op, a, b, y, expected(op, a, b));61 end62 end63endtaskEvery module in hw/core/ and every peripheral has a testbench like this. Run them all:
$ make -C test unit$ make -C test periph$ make -C test videoLevel 2: the core against a reference
Unit tests check modules; they do not check that the modules together execute RISC-V correctly. For that, test/core runs programs on the whole core. Some are written by hand (test_isa.S goes through every instruction), and some are random: a Python script generates thousands of random instructions, a Python instruction-set simulator (rv32i_iss.py) runs them, and the final registers and memory must match the hardware's.
$ make -C test core$ make -C test random SEEDS="1 2 3" N=5000Random tests find the cases nobody thought of: a branch right after a load into the same register, a shift by a register holding 32, a store to the byte the next instruction is fetched from.
Level 3: the whole SoC
Finally sim/ runs the real SoC top level with the real boot ROM and launcher, a model of the SPI flash and a model of the SD card:
4# Whole-SoC simulation (tb_soc.v): the real boot ROM and user firmware on5# hw/nandtier.v, with sim/spi_flash_model.v as the flash.6#7# make boot into the firmware, use its menu (several minutes)8# make offset the firmware test with the firmware at flash address9# 0x200000 (FLASH_OFFSET, as on the Tang Nano 20K / ULX3S)10# make isp boot ROM ISP: ping, erase, program, restart11# make carts the game menu on the RTL: Pong from the SD card model into12# the flash cartridge slot and running (tens of minutes);13# its first frame must match the PC build's (test/host)14# make cleanmake carts is the full story of this series in one simulation: boot, launcher, game menu, Pong copied from the SD card model into the flash slot, and its first frame compared, pixel for pixel, with a picture made by the PC build of the same game. It takes tens of minutes, because it simulates every one of several hundred million clock cycles.
Games on the PC
The games themselves are tested without any hardware at all. game_hal.h, which we met in part 10, has a second half for GAME_HOST builds: the same game code compiled for the PC, with a software model of the PPU. The tests play each game with scripted key presses, check its rules (an apple makes the snake longer, a wall ends the game) and save screenshots.
$ make -C test gamesConclusion
Three levels: modules against simple references, the core against an instruction-set simulator, and the whole system against a PC build of the same games. With them, a change anywhere in the design is checked in minutes, long before it reaches the board, and the board becomes the place where things work rather than the place where they are debugged.