Reading from the External Flash
Making a 4 MiB serial flash chip look like memory the CPU can run code from, with a small cache in front of it, and writing it from software.
In this article
What are we trying to do?
The FPGA has 16 KiB of RAM for us. The launcher alone is 18 KiB, and a game is 10 to 26 KiB. But next to the FPGA sits a 4 MiB SPI NOR flash chip, with room for all of it. We want the CPU to execute code straight from that flash, as if it were memory at address 0x0000_0000. That is called execute in place, or XIP.
The problem is that the flash is not memory. It is a serial device: to read a word you send a command byte, three address bytes, and then clock the data out bit by bit. We need three pieces:
- a controller that turns a bus read into a SPI read
- a cache, so the same code is not read over and over through a slow serial line
- a way for software to write the flash, to install the launcher and games
A SPI read
The flash talks over four wires: chip select (csb, low while talking), a clock, data in (mosi, IO0) and data out (miso, IO1). A read looks like this:
| Bytes on IO0 | Bytes on IO1 |
|---|---|
03 (read), address 23:16, 15:8, 7:0 | then data, as long as the clock runs |
That is 32 clocks of command and address before the first data bit. The flash keeps sending the following bytes as long as the clock runs, so reading sequential code is cheap once a read has started.
03, three address bytes, then the data streams out.nt_spi_flash reading one word, bit by bit, in single, dual and continuous-read modes. Compare the clock counts.The controller's header describes the three read modes it supports:
// single mode 03h read: cmd, addr, data on IO0 / IO1// dual mode BBh dual I/O read: cmd on IO0, then address, mode// byte and data on IO1:IO0// dual + CRM continuous read mode: mode byte A5h, so the next read// starts directly with the addressIn dual I/O mode the address and the data use both lines at once, two bits per clock. In continuous read mode the flash is told (with the mode byte A5) that the next transfer is another read, so the command byte can be skipped. Every trick shortens the time before the first useful bit.
A cache in front
Even fast, a miss costs a couple of hundred cycles. Code mostly runs in loops, so the same instructions are fetched again and again: a perfect case for a cache. NandTier's is 4 KiB, direct mapped:
4// NandTier flash read cache, between the bus and nt_spi_flash: 4 KiB,5// direct mapped, 256 lines of 16 bytes (4 words).6//7// hit: ready one cycle after valid (like the SRAM)8// miss: the 4 words of the line are read from the flash in order (the9// flash controller streams them), then the access completes10// write: passed through to the flash port (nt_spi_flash ignores them)11//12// invalidate empties the cache (256 cycles, accesses wait); connect it to13// the flash controller's restart, so code is re-read after the flash was14// programmed. It also runs after reset.15//16// Address: bits 23:12 tag, 11:4 line, 3:2 word.An address splits into three parts: bits 3:2 choose a word within a 16-byte line, bits 11:4 choose one of 256 lines, and bits 23:12 are the tag, stored with the line to remember which part of the flash it holds.
The storage is two block RAMs:
49reg [31:0] data [0:1023];50reg [12:0] tags [0:255];51reg [31:0] data_q;52reg [12:0] tag_q;53 54wire data_we = state == C_FILL && f_ready;55wire [9:0] data_wa = {line, fill_word};56wire tag_we = state == C_CLEAR || (data_we && fill_word == 2'd3);57wire [7:0] tag_wa = state == C_CLEAR ? clear_line : line;58wire [12:0] tag_wd = state == C_CLEAR ? 13'd0 : {1'b1, tag};1024 words of data, and 256 tags with a valid bit. Both are read every cycle at the address on the bus:
60always @(posedge clk) begin61 if (data_we)62 data[data_wa] <= f_rdata;63 if (tag_we)64 tags[tag_wa] <= tag_wd;65 data_q <= data[addr[11:2]];66 tag_q <= tags[line];67end68 69wire hit = state == C_CHECK && tag_q == {1'b1, tag};Block RAM answers one cycle after it is given an address, so the tag arrives in the cycle after the request. That is the C_CHECK state: a hit if the stored tag (with its valid bit) equals the address's tag.
The state machine
71always @(posedge clk) begin72 if (!resetn || invalidate) begin73 state <= C_CLEAR;74 clear_line <= 8'd0;75 end else case (state)76 C_IDLE:77 if (valid && !is_write)78 state <= C_CHECK; // tag and data arrive next cycle79 80 C_CHECK:81 if (!hit) begin82 state <= C_FILL;83 fill_word <= 2'd0;84 end else85 state <= C_IDLE;86 87 C_FILL:88 if (f_ready) begin89 fill_word <= fill_word + 2'd1;90 if (fill_word == 2'd3)91 state <= C_IDLE; // look again: now a hit92 end93 94 C_CLEAR: begin95 clear_line <= clear_line + 8'd1;96 if (clear_line == 8'd255)97 state <= C_IDLE;98 end99 endcase100end- Idle: a read comes in, go to Check.
- Check: hit, done (the data is already on
rdata). Miss, go to Fill. - Fill: read the line's four words from the flash controller in order, writing each into the data RAM; with the fourth, write the tag. Then back to Idle, where the same access comes in again and is now a hit.
- Clear: after reset, write 256 empty tags, one per cycle.
The bus side is two lines:
102assign ready = valid && (is_write ? f_ready : hit);103assign rdata = data_q;104 105assign f_valid = state == C_FILL || (valid && is_write);106assign f_addr = state == C_FILL ? {tag, line, fill_word, 2'b00} : addr;107assign f_wstrb = state == C_FILL ? 4'd0 : wstrb;A hit answers in two cycles, as fast as the SRAM. A miss costs a line fill: about 345 cycles in single mode, 185 with dual I/O and continuous reads. The launcher switches to the fastest mode as soon as it starts.
Writing the flash
Reading is all XIP needs, but to install a game the launcher must erase and program the flash. Those are more SPI commands, and rather than build each into hardware, the controller has a bit-bang mode: turn XIP off, and the CPU drives the four pins directly through a register.
That has one big consequence. While XIP is off, the CPU cannot fetch from the flash. So the code that bit-bangs it must run from somewhere else: the boot RAM. That is the flash_io() function whose jump we pinned at 0x8000_0054 in part 6.
Sending one byte, bit by bit:
51static uint8_t spi_byte(uint8_t out)52{53 uint8_t in = 0;54 for (int bit = 7; bit >= 0; bit--) {55 uint8_t mosi = (out >> bit) & 1 ? FLASH_PIN_IO0 : 0;56 FLASHCFG_PINS = mosi; // CS low, SCK low, data out57 FLASHCFG_PINS = mosi | FLASH_PIN_CLK; // rising edge58 in = (in << 1) | ((FLASHCFG_PINS & FLASH_PIN_IO1) ? 1 : 0);59 }60 FLASHCFG_PINS = 0;61 return in;62}For each bit, MSB first: set the data pin with the clock low, then raise the clock (the flash samples on the rising edge), then read the flash's output pin. Three bus accesses per bit. It is slow, about a hundred cycles a byte, but it only runs while installing.
The whole transaction:
69void flash_io(uint8_t *buf, int len, int wren)70{71 FLASHCFG_EN = 0; // XIP off; returns once the pins are ours72 FLASHCFG_PINS = FLASH_PIN_CSB;73 FLASHCFG_OE = FLASH_OE_IO0;74 75 if (wren) {76 spi_byte(FLASH_WREN);77 spi_end();78 }79 80 for (int i = 0; i < len; i++)81 buf[i] = spi_byte(buf[i]);82 spi_end();83 84 if (wren) {85 uint8_t status;86 do {87 spi_byte(FLASH_RDSR);88 status = spi_byte(0);89 spi_end();90 } while (status & 0x01); // write in progress91 }92 93 FLASHCFG_OE = 0;94 FLASHCFG_EN = FLASH_EN_XIP; // XIP on: the flash is woken up again95}FLASHCFG_EN = 0 turns XIP off; the controller makes the store wait until it has cleanly ended its current read. If the command writes (wren), it is preceded by Write Enable (06), and followed by polling the status register (05) until the "write in progress" bit clears: erasing a 4 KiB sector takes tens of milliseconds. Finally XIP goes back on, the controller restarts its read engine, and the cache empties.
Calling it from the launcher
From the launcher's side, flash_io is a function at a fixed address, declared in nandtier.h:
#define NT_BOOT_FLASH_IO (NT_BOOT_BASE + 0x54)typedef void (*nt_flash_io_fn)(uint8_t *buf, int len, int wren);#define nt_flash_io ((nt_flash_io_fn)NT_BOOT_FLASH_IO)And erasing a sector is then four bytes and one call:
162static void flash_erase_sector(uint32_t addr)163{164 uint8_t cmd[4] = {FLASH_SE, addr >> 16, addr >> 8, addr};165 nt_flash_io(cmd, 4, 1);166}Conclusion
A slow serial chip now looks like a megabyte of memory: a controller that speaks SPI, a 4 KiB cache that makes loops run at full speed, and a bit-bang escape hatch for erasing and programming, run from the boot RAM because the flash is busy.
The flash holds the launcher. The games come from somewhere bigger, the SD card, and in the next article we talk to it.