The NandTier SeriesPart 6 of 12

From C to the Chip: Boot ROM and Linker Scripts

What has to happen between power-on and main(): a reset vector, a stack, zeroed memory, and a linker script that says where every byte goes.

11 min readFirmwareCLinker
In this article
  1. What are we trying to do?
  2. The tool flags
  3. The first instructions
  4. The linker script
  5. What the boot ROM does
  6. Getting it into the boot RAM
  7. The launcher: code in flash, data in RAM
  8. Looking at the result
  9. Conclusion

What are we trying to do?

The CPU from parts 4 and 5 starts fetching at 0x8000_0000 the moment reset is released. Something has to be there. We want to write that something in C, compile it with GCC, and have it in the FPGA when it powers on.

That raises three questions this article answers:

power onreset releasedboot ROM0x8000_0000 · in the bitstreamlauncher0x0000_0000 · SPI flasha game0x0010_0000 · flash slotjumpcallQ: back to the menuwaits ½ s for a programmer (0x55)copied from the SD card first
The whole boot, from power-on to a game and back. Each arrow is a jump to a fixed address.
InteractiveStep through the boot chain: clocks and reset, the boot ROM, the flash cache, the SRAM and the UART as they wake up.

The tool flags

All firmware in the project is built with the same rules, in fw/common.mk:

fw/common.mklines 26–36
26# The core is RV32I only: no M (mul/div) and no C (compressed) extension.27# GCC 12 and later want the CSR instructions (rdcycle) named as Zicsr; older28# compilers (riscv-none-embed 8.x) do not know the name, so ask the compiler.29ZICSR  := $(shell $(CC) -march=rv32i_zicsr -mabi=ilp32 -E -x c /dev/null >/dev/null 2>&1 && echo _zicsr)30ARCH    = -march=rv32i$(ZICSR) -mabi=ilp3231OPT    ?= -O232 33CFLAGS  = $(ARCH) $(OPT) -g -std=gnu11 -Wall -Wextra -ffreestanding -fno-builtin \34          -ffunction-sections -fdata-sections -I$(FW)/include $(EXTRA_CFLAGS)35LDFLAGS = $(ARCH) -nostdlib -nostartfiles -Wl,--gc-sections -Wl,-T,$(LDSCRIPT) \36          -Wl,-Map,$(BUILD)/$(NAME).map -Wl,--print-memory-usage $(EXTRA_LDFLAGS)

The flags that matter for bare-metal code:

The first instructions

When reset is released, the CPU executes whatever is at 0x8000_0000. That cannot be C yet, because C code expects a stack pointer, and sp is zero after reset. So the very first code is a few lines of assembly:

fw/fw-brom/start.Slines 13–16
13    .section .text.start, "ax"14    .globl _start15_start:16    j reset

_start is placed in a section called .text.start, which the linker script will put first. Its first instruction jumps over a gap. The gap is there on purpose:

fw/fw-brom/start.Slines 18–21
18    .org 0x5419    .globl flash_io_entry20flash_io_entry:21    j flash_io

Address 0x8000_0054 holds a jump to flash_io(), a function in the boot ROM that the launcher calls later to write the flash (part 7). Because other programs call it by address, it must never move, so it is pinned with .org.

Then the real startup:

fw/fw-brom/start.Slines 23–34
23reset:24    la sp, __stack_top25 26    la t0, __bss_start27    la t1, __bss_end281:    bgeu t0, t1, 2f29    sw zero, 0(t0)30    addi t0, t0, 431    j 1b322:33    call main343:    j 3b

Three jobs. Set the stack pointer to the top of memory (stacks grow down). Clear the .bss section, where uninitialised globals live: C promises they start at zero, and RAM after power-on holds anything. Then call main, and if it ever returns, loop forever.

The linker script

The compiler produces sections: .text for code, .rodata for constants, .data for initialised variables, .bss for zeroed ones. The linker script decides where each one goes. The boot ROM's is short, because everything lives in one 8 KiB memory:

fw/fw-brom/brom.ldlines 13–16
13MEMORY14{15    BOOT (rwx) : ORIGIN = 0x80000000, LENGTH = 8K16}

A MEMORY block names the regions the program may use: here just BOOT, at the reset vector.

fw/fw-brom/brom.ldlines 20–23
20.text : {21    KEEP(*(.text.start))22    *(.text .text.*)23} > BOOT

KEEP(*(.text.start)) puts our _start first, at 0x8000_0000, and stops --gc-sections from throwing it away (nothing calls _start, the hardware just starts there). All other code follows.

fw/fw-brom/brom.ldlines 33–43
33ASSERT(SIZEOF(.data) == 0, "boot ROM must not have initialised writable data")34 35.bss (NOLOAD) : {36    . = ALIGN(4);37    __bss_start = .;38    *(.bss .bss.* .sbss .sbss.* COMMON)39    . = ALIGN(4);40    __bss_end = .;41} > BOOT42 43__stack_top = ORIGIN(BOOT) + LENGTH(BOOT);

The ASSERT makes the build fail if the boot ROM ever has initialised writable data, and the comment at the top of the file says why: the programmer can restart the boot ROM without reloading it, so such variables would keep stale values. The linker defines __bss_start, __bss_end and __stack_top, the symbols start.S uses.

What the boot ROM does

With all that in place, main() can be plain C. The boot ROM's job is short: give a PC programmer half a second to say hello, and otherwise start the real firmware in the flash.

fw/fw-brom/isp.clines 198–212
198int main(void)199{200    UART_DIV = (NT_CLK_HZ + NT_UART_BAUD / 2) / NT_UART_BAUD;201 202    uint32_t start = nt_cycles();203    while (nt_cycles() - start < ISP_WAIT_CYCLES) {204        if (UART_DATA == 0x55) {205            uart_put(0x56);206            serve();207        }208    }209 210    ((void (*)(void))NT_FLASH_BASE)();211    return 0;212}

First the UART divider for 115200 baud (rounded: 217). Then for half a second it watches the UART for the byte 0x55; if it comes, it answers 0x56 and serves programming commands (erase, write the flash, write the SD card). Otherwise, on line 210, it casts the flash address to a function pointer and calls it. That one line is the hand-over from the boot ROM to the launcher.

InteractiveThe real boot ROM image in the emulator, serving a programmer over the UART: ping, fill, erase, program, with the logic analyser on the flash pins.

Getting it into the boot RAM

The boot RAM is block RAM inside the FPGA, and block RAM can have initial contents that are part of the bitstream. So the boot ROM is not loaded by anything: it is in the chip from the moment the FPGA configures. The build turns the ELF file into four hex files, one per byte lane:

Terminal
$ make brom$ ls fw/fw-brom/build/fw-brom_*.hex

The FPGA build copies them next to the netlist (part 1), and the boot RAM reads them at synthesis with $readmemh. Change the boot ROM, rebuild the bitstream, and the new code is in the chip.

The launcher: code in flash, data in RAM

The firmware in the flash is bigger and lives in two places: its code and constants are read straight from the flash (part 7 explains how), but variables must be in RAM, because the flash cannot be written like memory. Its linker script has two regions:

fw/fw-flash/flash.ldlines 14–18
14MEMORY15{16    FLASH (rx)  : ORIGIN = 0x00000000, LENGTH = 1M17    RAM   (rwx) : ORIGIN = 0x40000000, LENGTH = 8K18}
SPI flash (load)SRAM (run).text · code.rodata · constants.data · initial values.data · variables.bss · zeroedfreestack ↓start.S copies← start.S writes zeros← sp starts at the toplinked for SRAM, stored in flash: "> RAM AT > FLASH"
Where each section lives. Code and constants are read straight from the flash; variables are copied to the SRAM before main().
InteractiveSection sizes from the real ELF files: what is in the boot RAM, the flash and the SRAM, for the launcher and for a game.

And one new idea, the AT > on the .data section:

fw/fw-flash/flash.ldlines 32–40
32.data : {33    __data_start = .;34    *(.data .data.*)35    __global_pointer$ = . + 0x800;36    *(.sdata .sdata.*)37    . = ALIGN(4);38    __data_end = .;39} > RAM AT > FLASH40__data_load = LOADADDR(.data);

> RAM AT > FLASH means: link the variables as if they were in RAM (that is where the code will use them), but store their initial values in the flash. __data_load is where they are stored. Someone has to copy them over before main(), and that is the second half of the startup code for flash programs:

fw/common/start.Slines 18–26
18    la t0, __data_load19    la t1, __data_start20    la t2, __data_end211:    bgeu t1, t2, 2f22    lw t3, 0(t0)23    sw t3, 0(t1)24    addi t0, t0, 425    addi t1, t1, 426    j 1b

t0 walks the stored values in the flash, t1 walks their place in RAM, one word at a time. The rest of that file is the same .bss clearing as the boot ROM's.

There is also gp, the global pointer:

fw/common/start.Slines 12–16
12la sp, __stack_top13.option push14.option norelax15la gp, __global_pointer$16.option pop

GCC can reach any variable within ±2 KiB of gp with a single instruction. The linker script sets __global_pointer$ to the middle of the small data, and .option norelax stops the assembler from "optimising" this very instruction into a gp-relative one before gp is set.

Looking at the result

Each build writes a disassembly next to the ELF. It is the best way to see what your C became:

InteractiveOne C program through GCC 8.2 at -O0 and -O2: click a line of C to see the machine code it became.
Terminal
$ less fw/fw-flash/build/fw-flash.asm$ riscv-none-elf-objdump -h fw/fw-flash/build/fw-flash.elf

objdump -h lists the sections with their two addresses, VMA (where the code expects them) and LMA (where they are stored). For .data you will see 0x4000_0000 and an address in the flash: exactly the AT > above.

Conclusion

Between reset and main() there are only a few instructions, but each one matters: a reset vector, a stack, zeroed .bss, copied .data, and a linker script that tells every byte where to live. With those, any C program can run on our CPU.

The launcher, though, runs from the flash, and the CPU can only fetch from memory on its bus. In the next article we make a SPI flash chip look like memory.