Difference between revisions of "RK3399 boot sequence"

From PINE64
Jump to navigation Jump to search
m (Fix headings)
(6 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Rockchip RK3399 boot sequence =
This article describes the boot sequence of the RK3399, the SoC on which the [[ROCKPro64]] single board computer, [[Pinebook Pro]] laptop and the [[PinePhone Pro]] are based.
This is for the Pine64 ROCKPro64 single board computer, and the Pinebook Pro laptop boot sequence details. While some of this may also apply to other RK3399 computers, that is up to the reader to make any appropriate changes.


== Boot sequence details ==
== Boot ROM operation ==


There are two types of internal memory inside rk3399:
After a hardware reset, the SoC's boot ROM (BROM) starts running on CPU0, one of the Cortex-A53s on the chip. This code is stored on a block of read-only memory, likely hard-wired into the chip.


* 32KB BootROM - containing a small program which loads initial bootloader.It supports loading it from SPI, eMMC, SD and USB OTG in so called - maskrom mode.
It employs 5 strategies, in order, to load a bootloader from off-chip:
* 200KB SRAM


Actual boot sequence:
# loading from NOR flash on SPI1
# loading from NAND flash on SPI1
# loading from eMMC
# loading from SD on the SDMMC controller
# bringing up the OTG0 USB controller in device mode and accepting control transfers to load programs


1. BootROM loads U-Boot TPL into SRAM. It's job is to initialize main system RAM.<br>
Loading a bootloader from storage is done in 4 steps:
2. Control goes back from U-Boot TPL to BootROM.<br>
3. BootROM loads U-Boot SPL.<br>
4. U-Boot SPL loads ATF and U-Boot itself into main memory.<br>
5. ATF runs U-Boot.<br>
6. U-Boot loads payload (eg. EFI binaries, grub). Direct Linux kernel load is also supported.


U-boot can boot from many sources - eMMC, SD, USB, NVME.
# Probing the boot device
# Finding the ID block
# Loading the first stage of the bootloader to main SRAM and running it
# (if the first stage returned to BROM) Loading the second stage to low DRAM and running it


'''idbloader.img''' contains TPL and SPL.<br>
=== ID block ===
'''u-boot.itb''' contains ATF and U-Boot.


== Description ==
The ID block is a 512-byte block at one of several well-known locations (depending on boot device type) scrambled using RC4 with a well-known key, which begins with the bytes 0x3b 8c dc fc (ciphertext; plaintext: 0x55 aa f0 0f).
 
In addition to the magic number, it contains (all fields little-endian):
 
* a flag to RC4-scramble the bootloader images too (32b at offset 8)
* offset (in 512 B-sectors) of the first bootloader stage (16b at offset 12)
* the size (in 2 KiB blocks) of the first bootloader stage (16b at offset 506)
* the size (in 2 KiB blocks) of the second bootloader stage (16b at offset 508)
 
These fields can be seen in U-Boot '''tools/rkcommon.c'''.
 
=== SPI boot ===
 
The boot ROM tries to probe the device on SPI1 by issuing the SPI-NOR 0x9f (Read Identification) command.
 
If the type and capacity bytes are all-ones or all-zeroes, probe fails.
If the first byte (manufacturer ID) is all-ones, the BROM uses NAND protocol, otherwise NOR protocol.
 
NOR protocol uses command 0x03 (low-frequency read without turnaround cycles) to read blocks, NAND protocol uses command 0x13.
 
The ID block is searched by loading the 32 bytes at the start of flash (TODO other load offsets) and testing for the magic number at any offset within those 32 bytes.
 
The SPI code has a bug that means that the 2 KiB blocks in which the bootloader is loaded have a stride of 4KiB, leaving the 2KiB inbetween as unused padding.
 
=== eMMC/SD boot ===
 
The boot ROM probes for eMMC on the eMMC controller using the normal MMC enumeration sequence (CMD0/1/2/3) at 375 kHz and runs the bus at 24 MHz, in 8-wide mode. The BROM does not support using the standard option for eMMC boot partitions.
 
SD is probed on the SDMMC (not the functionally equivalent SDIO) controller, using the normal SD enumeration sequence at 375 kHz and then runs the card at 12 MHz (TODO confirm) in 4-wide mode.
 
After initialization, eMMC and SD behave much the same. The ID block is searched by loading the blocks at sector 64 + 1024*n for n from 0 to 4 (inclusive) and checking for the magic number at the start of that sector. The BROM handles both byte- and block-addressed SD/MMC cards.
 
=== Hardware root of trust ===
 
The BROM theoretically supports SHA256/RSA2048 authentication with rollback protection.
 
The relevant data is stored in Secure eFuses: an enable flag, a SHA256 hash of the public key (which is stored on the boot medium) and a range of bits that are sequentially set for rollback protection.
 
In practice, the BROM does not verify the full length of the public key against the hash in eFuses, which means that it is most likely broken (expert review would be appreciated).
 
TOOD details
 
=== USB gadget boot ===
 
If no ID block is found on SPI, eMMC or SD, the BROM sets up the OTG0 USB controller to act as a USB high-speed (480 Mb/s) device with Vendor/Product ID of 2207:330c. In this state it supports most standard control requests, as well as 2 vendor control requests:
 
; 0x0471
: The code is loaded to main SRAM. When the transfer is done, it does a function call to the load address.
; 0x0472
: The code is loaded to low DRAM. When the transfer is done, USB is torn down and BROM jumps to the load address.
 
Transfer is performed by doing the requests on 4 KiB blocks. BROM considers the transfer complete as soon as a request supplies a block not 4096B long. (This means images a multiple of 4096 bytes long must be padded to achieve this condition.
 
== U-Boot boot sequence ==
 
Bootloaders based on U-Boot (including Tow-Boot) run in 4 stages on the RK3399:
 
# TPL, loaded by the BROM into main SRAM. Its [https://www.denx.de/wiki/pub/U-Boot/MiniSummitELCE2013/tpl-presentation.pdf job] is to initialize DRAM (main system memory). It returns to BROM. This job can alternatively be performed by proprietary Rockchip DDR blobs.
# SPL, loaded by the BROM into low DRAM. It loads the respective parts of TF-A BL31 into DRAM, main and PMU SRAM, and U-Boot proper into DRAM.
# TF-A BL31. It sets up EL2 to run U-Boot and stays resident until system shutdown.
# U-Boot proper. It can load EFI binaries (Grub, systemd-boot, …) from a variety of block devices (SD, eMMC, NVMe, USB Mass Storage, …).
 
For mainline-based U-Boots, these stages usually come in 2 images:
 
;idbloader.img
: contains TPL and SPL.
;u-boot.itb
: contains TF-A and U-Boot.
 
=== Load offsets ===
 
{{Note|This section applies to BSP U-Boot. Mainline-based U-Boots pack TF-A into '''u-boot.itb'''.}}


There are 3 sections for the boot loader. They are in order, without gap, though their is no need to use all the space in each section.<br>
There are 3 sections for the boot loader. They are in order, without gap, though their is no need to use all the space in each section.<br>
Line 49: Line 119:
|}
|}


== General maintenance ==
=== General maintenance ===
If a new U-Boot is supplied, it is generally installed similar to this:
If a new U-Boot is supplied, it is generally installed similar to this:
<pre># dd if=/boot/idbloader.img conv=notrunc seek=64    of=/dev/mmcblkX
<pre># dd if=/boot/idbloader.img conv=notrunc seek=64    of=/dev/mmcblkX
# dd if=/boot/u-boot.itb    conv=notrunc seek=16384 of=/dev/mmcblkX</pre>
# dd if=/boot/u-boot.itb    conv=notrunc seek=16384 of=/dev/mmcblkX</pre>


== Different devices ==
=== Different devices ===
The RK3399 boots to multiple devices. Boot device selection is done in the following order, and it cannot be changed.<br>
The RK3399 boots to multiple devices. Boot device selection is done in the following order, and it cannot be changed.<br>
If a device is blank / unused, the SoC code moves on to the next device in the list.
If a device is blank / unused, the SoC code moves on to the next device in the list.
Line 66: Line 136:
* WiFi
* WiFi
They can be made bootable by using one of the other devices as an initial bootloader. For example, several people have gotten their NVMe drives to be bootable with "/boot" and "/" on the NMVe. This either entails using the SPI or eMMC as the initial bootloader, with code to support PCIe NVMe devices.
They can be made bootable by using one of the other devices as an initial bootloader. For example, several people have gotten their NVMe drives to be bootable with "/boot" and "/" on the NMVe. This either entails using the SPI or eMMC as the initial bootloader, with code to support PCIe NVMe devices.
== Grub as the target of the bootloader ==
It is possible to use Grub as the target of U-Boot. This would allow;
* Selecting a different boot device
* Choosing a partition on a boot device for booting
* Different kernels
* Changes in kernel command line options
However, at present, Grub does not support the video & keyboard of the Pinebook Pro. So, any selection is done through the serial console.


== Boot loader development ==
== Boot loader development ==
There are several projects that have their own versions of U-Boot, with different features. Here are some of the more common ones at present, 2020/06/14:
There are several projects that have their own versions of U-Boot, with different features. Here are some of the more common ones at present, 2020/06/14:
* Rockchip
* Rockchip
Line 74: Line 153:
* U-Boot mainline
* U-Boot mainline


== Grub as the target of the boot loader ==
Bootloaders not based on U-Boot:
It is possible to use Grub as the target of U-Boot. This would allow;
 
* Selecting a different boot device
* coreboot runs on RK3399-based Chromebooks, it has not been ported to Pine64 boards yet.
* Choosing a partition on a boot device for booting
* levinboot is a bootloader developed by CrystalGamma in the Pine64 community. It runs on RockPro64 and Pinebook Pro. Its development is on hiatus as of April 2022, but a fork porting it to the PinePhone Pro exists.
* Different kernels
 
* Changes in kernel command line options
[[Category:PineBook Pro]] [[Category:ROCKPro64]] [[Category:Rockchip RK3399]] [[Category:PinePhone Pro]]
However, at present, Grub does not support the video & keyboard of the Pinebook Pro. So, any selection is done through the serial console.

Revision as of 16:57, 25 April 2022

This article describes the boot sequence of the RK3399, the SoC on which the ROCKPro64 single board computer, Pinebook Pro laptop and the PinePhone Pro are based.

Boot ROM operation

After a hardware reset, the SoC's boot ROM (BROM) starts running on CPU0, one of the Cortex-A53s on the chip. This code is stored on a block of read-only memory, likely hard-wired into the chip.

It employs 5 strategies, in order, to load a bootloader from off-chip:

  1. loading from NOR flash on SPI1
  2. loading from NAND flash on SPI1
  3. loading from eMMC
  4. loading from SD on the SDMMC controller
  5. bringing up the OTG0 USB controller in device mode and accepting control transfers to load programs

Loading a bootloader from storage is done in 4 steps:

  1. Probing the boot device
  2. Finding the ID block
  3. Loading the first stage of the bootloader to main SRAM and running it
  4. (if the first stage returned to BROM) Loading the second stage to low DRAM and running it

ID block

The ID block is a 512-byte block at one of several well-known locations (depending on boot device type) scrambled using RC4 with a well-known key, which begins with the bytes 0x3b 8c dc fc (ciphertext; plaintext: 0x55 aa f0 0f).

In addition to the magic number, it contains (all fields little-endian):

  • a flag to RC4-scramble the bootloader images too (32b at offset 8)
  • offset (in 512 B-sectors) of the first bootloader stage (16b at offset 12)
  • the size (in 2 KiB blocks) of the first bootloader stage (16b at offset 506)
  • the size (in 2 KiB blocks) of the second bootloader stage (16b at offset 508)

These fields can be seen in U-Boot tools/rkcommon.c.

SPI boot

The boot ROM tries to probe the device on SPI1 by issuing the SPI-NOR 0x9f (Read Identification) command.

If the type and capacity bytes are all-ones or all-zeroes, probe fails. If the first byte (manufacturer ID) is all-ones, the BROM uses NAND protocol, otherwise NOR protocol.

NOR protocol uses command 0x03 (low-frequency read without turnaround cycles) to read blocks, NAND protocol uses command 0x13.

The ID block is searched by loading the 32 bytes at the start of flash (TODO other load offsets) and testing for the magic number at any offset within those 32 bytes.

The SPI code has a bug that means that the 2 KiB blocks in which the bootloader is loaded have a stride of 4KiB, leaving the 2KiB inbetween as unused padding.

eMMC/SD boot

The boot ROM probes for eMMC on the eMMC controller using the normal MMC enumeration sequence (CMD0/1/2/3) at 375 kHz and runs the bus at 24 MHz, in 8-wide mode. The BROM does not support using the standard option for eMMC boot partitions.

SD is probed on the SDMMC (not the functionally equivalent SDIO) controller, using the normal SD enumeration sequence at 375 kHz and then runs the card at 12 MHz (TODO confirm) in 4-wide mode.

After initialization, eMMC and SD behave much the same. The ID block is searched by loading the blocks at sector 64 + 1024*n for n from 0 to 4 (inclusive) and checking for the magic number at the start of that sector. The BROM handles both byte- and block-addressed SD/MMC cards.

Hardware root of trust

The BROM theoretically supports SHA256/RSA2048 authentication with rollback protection.

The relevant data is stored in Secure eFuses: an enable flag, a SHA256 hash of the public key (which is stored on the boot medium) and a range of bits that are sequentially set for rollback protection.

In practice, the BROM does not verify the full length of the public key against the hash in eFuses, which means that it is most likely broken (expert review would be appreciated).

TOOD details

USB gadget boot

If no ID block is found on SPI, eMMC or SD, the BROM sets up the OTG0 USB controller to act as a USB high-speed (480 Mb/s) device with Vendor/Product ID of 2207:330c. In this state it supports most standard control requests, as well as 2 vendor control requests:

0x0471
The code is loaded to main SRAM. When the transfer is done, it does a function call to the load address.
0x0472
The code is loaded to low DRAM. When the transfer is done, USB is torn down and BROM jumps to the load address.

Transfer is performed by doing the requests on 4 KiB blocks. BROM considers the transfer complete as soon as a request supplies a block not 4096B long. (This means images a multiple of 4096 bytes long must be padded to achieve this condition.

U-Boot boot sequence

Bootloaders based on U-Boot (including Tow-Boot) run in 4 stages on the RK3399:

  1. TPL, loaded by the BROM into main SRAM. Its job is to initialize DRAM (main system memory). It returns to BROM. This job can alternatively be performed by proprietary Rockchip DDR blobs.
  2. SPL, loaded by the BROM into low DRAM. It loads the respective parts of TF-A BL31 into DRAM, main and PMU SRAM, and U-Boot proper into DRAM.
  3. TF-A BL31. It sets up EL2 to run U-Boot and stays resident until system shutdown.
  4. U-Boot proper. It can load EFI binaries (Grub, systemd-boot, …) from a variety of block devices (SD, eMMC, NVMe, USB Mass Storage, …).

For mainline-based U-Boots, these stages usually come in 2 images:

idbloader.img
contains TPL and SPL.
u-boot.itb
contains TF-A and U-Boot.

Load offsets

This section applies to BSP U-Boot. Mainline-based U-Boots pack TF-A into u-boot.itb.

There are 3 sections for the boot loader. They are in order, without gap, though their is no need to use all the space in each section.
Here are the details:

Start in
sectors
Size in
sectors
Name Description
64 16320 IDBLoader SoC initialization code
16384 8192 OS loader Generally U-Boot
24576 8192 TrustedFirmware-A  

General maintenance

If a new U-Boot is supplied, it is generally installed similar to this:

# dd if=/boot/idbloader.img conv=notrunc seek=64    of=/dev/mmcblkX
# dd if=/boot/u-boot.itb    conv=notrunc seek=16384 of=/dev/mmcblkX

Different devices

The RK3399 boots to multiple devices. Boot device selection is done in the following order, and it cannot be changed.
If a device is blank / unused, the SoC code moves on to the next device in the list.

  • SPI
  • eMMC
  • SD card

However, whence the user boot code runs, it can then give priority to other devices, if available. The following devices are not directly bootable:

  • NVMe
  • USB 3
  • WiFi

They can be made bootable by using one of the other devices as an initial bootloader. For example, several people have gotten their NVMe drives to be bootable with "/boot" and "/" on the NMVe. This either entails using the SPI or eMMC as the initial bootloader, with code to support PCIe NVMe devices.

Grub as the target of the bootloader

It is possible to use Grub as the target of U-Boot. This would allow;

  • Selecting a different boot device
  • Choosing a partition on a boot device for booting
  • Different kernels
  • Changes in kernel command line options

However, at present, Grub does not support the video & keyboard of the Pinebook Pro. So, any selection is done through the serial console.

Boot loader development

There are several projects that have their own versions of U-Boot, with different features. Here are some of the more common ones at present, 2020/06/14:

  • Rockchip
  • The original default Debian
  • Manjaro
  • U-Boot mainline

Bootloaders not based on U-Boot:

  • coreboot runs on RK3399-based Chromebooks, it has not been ported to Pine64 boards yet.
  • levinboot is a bootloader developed by CrystalGamma in the Pine64 community. It runs on RockPro64 and Pinebook Pro. Its development is on hiatus as of April 2022, but a fork porting it to the PinePhone Pro exists.