The NandTier SeriesPart 7 of 12

Reading from the External Flash

Making a 4 MiB serial flash chip look like memory the CPU can run code from, with a small cache in front of it, and writing it from software.

9 min readFlashSPICache
In this article
  1. What are we trying to do?
  2. A SPI read
  3. A cache in front
  4. Writing the flash
  5. Calling it from the launcher
  6. Conclusion

What are we trying to do?

The FPGA has 16 KiB of RAM for us. The launcher alone is 18 KiB, and a game is 10 to 26 KiB. But next to the FPGA sits a 4 MiB SPI NOR flash chip, with room for all of it. We want the CPU to execute code straight from that flash, as if it were memory at address 0x0000_0000. That is called execute in place, or XIP.

The problem is that the flash is not memory. It is a serial device: to read a word you send a command byte, three address bytes, and then clock the data out bit by bit. We need three pieces:

  1. a controller that turns a bus read into a SPI read
  2. a cache, so the same code is not read over and over through a slow serial line
  3. a way for software to write the flash, to install the launcher and games

A SPI read

The flash talks over four wires: chip select (csb, low while talking), a clock, data in (mosi, IO0) and data out (miso, IO1). A read looks like this:

Bytes on IO0Bytes on IO1
03 (read), address 23:16, 15:8, 7:0then data, as long as the clock runs

That is 32 clocks of command and address before the first data bit. The flash keeps sending the following bytes as long as the clock runs, so reading sequential code is cheap once a read has started.

csbsckmosi03A23:16A15:8A7:0misoD0D1command and address: 32 clocks of nothing useful
One read: chip select low, the command 03, three address bytes, then the data streams out.
InteractiveThe real nt_spi_flash reading one word, bit by bit, in single, dual and continuous-read modes. Compare the clock counts.

The controller's header describes the three read modes it supports:

hw/periph/nt_spi_flash.v (header)
//   single mode  03h read:              cmd, addr, data on IO0 / IO1//   dual mode    BBh dual I/O read:     cmd on IO0, then address, mode//                                       byte and data on IO1:IO0//   dual + CRM   continuous read mode:  mode byte A5h, so the next read//                                       starts directly with the address

In dual I/O mode the address and the data use both lines at once, two bits per clock. In continuous read mode the flash is told (with the mode byte A5) that the next transfer is another read, so the command byte can be skipped. Every trick shortens the time before the first useful bit.

A cache in front

Even fast, a miss costs a couple of hundred cycles. Code mostly runs in loops, so the same instructions are fetched again and again: a perfect case for a cache. NandTier's is 4 KiB, direct mapped:

hw/periph/nt_flash_cache.vlines 4–16
4// NandTier flash read cache, between the bus and nt_spi_flash: 4 KiB,5// direct mapped, 256 lines of 16 bytes (4 words).6//7//   hit:   ready one cycle after valid (like the SRAM)8//   miss:  the 4 words of the line are read from the flash in order (the9//          flash controller streams them), then the access completes10//   write: passed through to the flash port (nt_spi_flash ignores them)11//12// invalidate empties the cache (256 cycles, accesses wait); connect it to13// the flash controller's restart, so code is re-read after the flash was14// programmed. It also runs after reset.15//16// Address: bits 23:12 tag, 11:4 line, 3:2 word.

An address splits into three parts: bits 3:2 choose a word within a 16-byte line, bits 11:4 choose one of 256 lines, and bits 23:12 are the tag, stored with the line to remember which part of the flash it holds.

tagwhich 4 KiB of the flashlinewhich of 256 lineswordword002312114320hit = the line's stored tag equals these 12 bits (and the line is valid)
The cache splits each flash address in three.

The storage is two block RAMs:

hw/periph/nt_flash_cache.vlines 49–58
49reg  [31:0] data [0:1023];50reg  [12:0] tags [0:255];51reg  [31:0] data_q;52reg  [12:0] tag_q;53 54wire        data_we = state == C_FILL && f_ready;55wire [9:0]  data_wa = {line, fill_word};56wire        tag_we  = state == C_CLEAR || (data_we && fill_word == 2'd3);57wire [7:0]  tag_wa  = state == C_CLEAR ? clear_line : line;58wire [12:0] tag_wd  = state == C_CLEAR ? 13'd0 : {1'b1, tag};

1024 words of data, and 256 tags with a valid bit. Both are read every cycle at the address on the bus:

hw/periph/nt_flash_cache.vlines 60–69
60always @(posedge clk) begin61    if (data_we)62        data[data_wa] <= f_rdata;63    if (tag_we)64        tags[tag_wa] <= tag_wd;65    data_q <= data[addr[11:2]];66    tag_q  <= tags[line];67end68 69wire hit = state == C_CHECK && tag_q == {1'b1, tag};

Block RAM answers one cycle after it is given an address, so the tag arrives in the cycle after the request. That is the C_CHECK state: a hit if the stored tag (with its valid bit) equals the address's tag.

The state machine

hw/periph/nt_flash_cache.vlines 71–100
71always @(posedge clk) begin72    if (!resetn || invalidate) begin73        state      <= C_CLEAR;74        clear_line <= 8'd0;75    end else case (state)76        C_IDLE:77            if (valid && !is_write)78                state <= C_CHECK;           // tag and data arrive next cycle79 80        C_CHECK:81            if (!hit) begin82                state     <= C_FILL;83                fill_word <= 2'd0;84            end else85                state <= C_IDLE;86 87        C_FILL:88            if (f_ready) begin89                fill_word <= fill_word + 2'd1;90                if (fill_word == 2'd3)91                    state <= C_IDLE;        // look again: now a hit92            end93 94        C_CLEAR: begin95            clear_line <= clear_line + 8'd1;96            if (clear_line == 8'd255)97                state <= C_IDLE;98        end99    endcase100end

The bus side is two lines:

hw/periph/nt_flash_cache.vlines 102–107
102assign ready   = valid && (is_write ? f_ready : hit);103assign rdata   = data_q;104 105assign f_valid = state == C_FILL || (valid && is_write);106assign f_addr  = state == C_FILL ? {tag, line, fill_word, 2'b00} : addr;107assign f_wstrb = state == C_FILL ? 4'd0 : wstrb;

A hit answers in two cycles, as fast as the SRAM. A miss costs a line fill: about 345 cycles in single mode, 185 with dual I/O and continuous reads. The launcher switches to the fastest mode as soon as it starts.

InteractiveRun access patterns through the real cache: a loop, a long jump, a table walk. Watch hits, misses and line fills.

Writing the flash

Reading is all XIP needs, but to install a game the launcher must erase and program the flash. Those are more SPI commands, and rather than build each into hardware, the controller has a bit-bang mode: turn XIP off, and the CPU drives the four pins directly through a register.

That has one big consequence. While XIP is off, the CPU cannot fetch from the flash. So the code that bit-bangs it must run from somewhere else: the boot RAM. That is the flash_io() function whose jump we pinned at 0x8000_0054 in part 6.

Sending one byte, bit by bit:

fw/fw-brom/isp.clines 51–62
51static uint8_t spi_byte(uint8_t out)52{53    uint8_t in = 0;54    for (int bit = 7; bit >= 0; bit--) {55        uint8_t mosi = (out >> bit) & 1 ? FLASH_PIN_IO0 : 0;56        FLASHCFG_PINS = mosi;                       // CS low, SCK low, data out57        FLASHCFG_PINS = mosi | FLASH_PIN_CLK;       // rising edge58        in = (in << 1) | ((FLASHCFG_PINS & FLASH_PIN_IO1) ? 1 : 0);59    }60    FLASHCFG_PINS = 0;61    return in;62}

For each bit, MSB first: set the data pin with the clock low, then raise the clock (the flash samples on the rising edge), then read the flash's output pin. Three bus accesses per bit. It is slow, about a hundred cycles a byte, but it only runs while installing.

The whole transaction:

fw/fw-brom/isp.clines 69–95
69void flash_io(uint8_t *buf, int len, int wren)70{71    FLASHCFG_EN   = 0;                  // XIP off; returns once the pins are ours72    FLASHCFG_PINS = FLASH_PIN_CSB;73    FLASHCFG_OE   = FLASH_OE_IO0;74 75    if (wren) {76        spi_byte(FLASH_WREN);77        spi_end();78    }79 80    for (int i = 0; i < len; i++)81        buf[i] = spi_byte(buf[i]);82    spi_end();83 84    if (wren) {85        uint8_t status;86        do {87            spi_byte(FLASH_RDSR);88            status = spi_byte(0);89            spi_end();90        } while (status & 0x01);    // write in progress91    }92 93    FLASHCFG_OE = 0;94    FLASHCFG_EN = FLASH_EN_XIP;     // XIP on: the flash is woken up again95}

FLASHCFG_EN = 0 turns XIP off; the controller makes the store wait until it has cleanly ended its current read. If the command writes (wren), it is preceded by Write Enable (06), and followed by polling the status register (05) until the "write in progress" bit clears: erasing a 4 KiB sector takes tens of milliseconds. Finally XIP goes back on, the controller restarts its read engine, and the cache empties.

Calling it from the launcher

From the launcher's side, flash_io is a function at a fixed address, declared in nandtier.h:

fw/include/nandtier.h
#define NT_BOOT_FLASH_IO   (NT_BOOT_BASE + 0x54)typedef void (*nt_flash_io_fn)(uint8_t *buf, int len, int wren);#define nt_flash_io ((nt_flash_io_fn)NT_BOOT_FLASH_IO)

And erasing a sector is then four bytes and one call:

fw/fw-flash/loader.clines 162–166
162static void flash_erase_sector(uint32_t addr)163{164    uint8_t cmd[4] = {FLASH_SE, addr >> 16, addr >> 8, addr};165    nt_flash_io(cmd, 4, 1);166}

Conclusion

A slow serial chip now looks like a megabyte of memory: a controller that speaks SPI, a 4 KiB cache that makes loops run at full speed, and a bit-bang escape hatch for erasing and programming, run from the boot RAM because the flash is busy.

The flash holds the launcher. The games come from somewhere bigger, the SD card, and in the next article we talk to it.