QEMU SPARC64 machine description (MD) binaries: Solaris boot matrix

Tracks which guest OS test images have been verified to boot cleanly against which Machine Description (MD) and hypervisor (HV) binaries (1up-md.bin, 1up-hv.bin) build in bin. These two files are loaded by the Niagara machine's qemu-sparc64 emulation. The detail how to use it in section below.

Boot status: MD/hv-config build (row) × OS image (column)
OS image/MD build disk.s10hw20 snv_134 ai ISO Solaris 11 Express 11/2010 ai Solaris 11 11/11 text Solaris 11.111 Solaris 10u8 GA DVD Solaris 10u10 GA2 DVD Solaris 10u117
Reference
OpenSPARC_T1_rebuild (binary-identical)1 PASS FAIL/trap
Fast Data Access MMU Miss
3
FAIL/trap
Level 14 Interrupt
FAIL/trap
Level 14 Interrupt
untested FAIL/panic
lgrp_traverse
FAIL/panic
boot temp overflow
5
FAIL/trap
Level 14 Interrupt
1GiB-snv_77
(1GiB-experimental)2
PASS PASS9 PASS9 FAIL/trap
Level 14 Interrupt
untested FAIL/panic
kmem64-prom_alloc
FAIL/panic
kmem64-prom_alloc
FAIL/panic
kmem64-prom_alloc
test builds
t1_boot_fix/256
(mem/cpu/cache/mmu in MD)
PASS FAIL/panic
bop_alloc_chunk
4,5
FAIL/panic
kmem64-prom_alloc
FAIL/panic
boot temp overflow
5
untested FAIL/panic
bop_alloc_chunk
5
FAIL/panic
boot temp overflow
5
FAIL/trap
Level 14 Interrupt
t1_boot_fix/1024
(+1024MB in MD)
PASS PASS6,9 FAIL/panic
sfmmu_cpu_init
FAIL/panic
sfmmu_cpu_init
untested PASS FAIL/panic
sfmmu_cpu_init
FAIL/panic
sfmmu_cpu_init
t1_boot_fix/2048
(+2048MB in MD)
PASS FAIL/hang
MMU_MAP_ADDR
FAIL/hang FAIL/panic
sfmmu_cpu_init
untested FAIL/trap
Level 14 Interrupt
FAIL/hang FAIL/hang
tlb_fix/1024
(t1_boot_fix/1024 + TLB in MD)
PASS PASS9 PASS9 PASS (kernel), installer stall9 untested PASS PASS PASS
tlb_fix/2048
(t1_boot_fix/2048 + TLB in MD)
PASS FAIL/hang
alloc_page_freelists
8
FAIL/hang
alloc_page_freelists
FAIL/hang
after kernel init
untested FAIL/hang FAIL/hang
alloc_page_freelists
FAIL/trap
Level 14 Interrupt
pagesize_256M/409610 PASS PASS9 PASS9 PASS9 PASS9,11 PASS PASS PASS
pagesize_256M/819210 PASS PASS9 PASS9 untested PASS9,11 PASS PASS PASS
PASS — verified clean boot FAIL — hang or trap FAIL — kernel panic FAIL — known limitation (not enough memory for this OS image) PASS (kernel) — no panic/hang, but boot never completes (installer/service stall, not a kernel failure) — untested

Usage

qemu-system-sparc64 -M niagara \
    -L <config_dir> [-L <firmware_dir>] \
    -nographic -m <memsize> [-trace load_file] \
    -drive if=pflash,readonly=on,file=<image>

-M niagara loads six files by name — 1up-md.bin, 1up-hv.bin, nvram1, openboot.bin, q.bin, reset.bin — and finds them across the -L paths, searched in the order given, first match winning.

The second -L is needed when the first directory holds only part of them. The config directories in bin hold just 1up-md.bin and 1up-hv.bin, so the firmware has to come from a second path — fw_blobs (from this project), or the S10image dir from the OpenSPARC T1 archive, whose blobs are byte-identical. One -L suffices only if that single directory holds all six — make symlinks in a config's source directory links the firmware in to make it so.

Order matters especially with S10image: it also ships its own 1up-md.bin and 1up-hv.bin, so listing it first silently overrides the config you meant to test. Keep the config directory first.

-m is required and must match the config — the memory size is the last path component, <config>/<memsize>.

-trace load_file is optional and prints where each of the six files was resolved from.

Example:

Solaris 10 update 11 boot example (two -L paths of blobs):

qemu-system-sparc64 -M niagara \
    -L /tmp/md/bin/latest/4096 -L /tmp/md/fw_blobs \
    -nographic -m 4096 \
    -drive if=pflash,readonly=on,file=/zroot/iso/s10u11/V36434-01.iso

Boot the OpenSPARC T1 Legion image, referenced in the QEMU sparc64 documentation:

qemu-system-sparc64 -M niagara \
    -L /tmp/md/bin/latest/4096 -L /path-to/S10image \
    -nographic -m 4096 \
    -drive if=pflash,readonly=on,file=/path-to/S10image/disk.s10hw2

If the md config files were built with make symlinks, a single -L path is enough:

qemu-system-sparc64 -M niagara \
    -L /tmp/md/bin/latest/4096 \
    -nographic -m 4096 \
    -drive if=pflash,readonly=on,file=/zroot/iso/s10u11/V36434-01.iso

Confirm which directory each file was taken from with -trace load_file:

qemu-system-sparc64 -M niagara \
    -L /tmp/md/bin/latest/4096 -L /path-to/S10image \
    -nographic -m 4096 -trace load_file \
    -drive if=pflash,readonly=on,file=/zroot/iso/s10u11/V36434-01.iso

load_file name nvram1 location /path-to/S10image/nvram1
load_file name 1up-md.bin location /tmp/md/bin/latest/4096/1up-md.bin
load_file name 1up-hv.bin location /tmp/md/bin/latest/4096/1up-hv.bin
load_file name reset.bin location /path-to/S10image/reset.bin
load_file name q.bin location /path-to/S10image/q.bin
load_file name openboot.bin location /path-to/S10image/openboot.bin
load_file name /zroot/iso/s10u11/V36434-01.iso location /zroot/iso/s10u11/V36434-01.iso
cpu Probing I/O buses

The MD pair resolves out of the config directory and the four firmware files out of S10image — which is what the -L order asked for. Had S10image been listed first, 1up-md.bin and 1up-hv.bin would show its paths here instead, and the config under test would never have been loaded.

Build Procedure

The build procedure is described here: https://github.com/unix0cc/md/. Requires a toolkit (mdgen/mdlint) used for compiling the .bin files (1up-md.bin, 1up-hv.bin).

Notes

0 disk.s10hw2 — Solaris 5.10: Solaris 10 3/05 HW2 (s10s_hw2wos_05) SPARC, derived from Legion code as boot reference for QEMU sun4v (wiki.qemu.org/Documentation/Platforms/SPARC).

1 Derived from the OpenSparc T1 Legion image — Sun's original 2006 emulator reference with shipped hypervisor and MD binaries, which was rebuilt using mdgen tool from mdbuild toolchain.

2 1GiB-snv_77 — Artyom Tarasenko's MD build with 1024MB (github.com/artyom-tarasenko/qemu-sun4v-md).

3 Starting point of investigation: part #1 and part #2, ended by t1_boot_fix/256.

4 Bumping the memory fix 256 -> 1024MB — resolved by t1_boot_fix/1024, see link#3.

5 bop_alloc_chunk and boot temp overflow are both out-of-memory panics for Solaris kernels — 256MB is too little for these images.

6 Reference point: enable TLB property (part #4).

7 Available via edelivery.oracle.com, V36434-01.iso.

8 Requires 256M paging, which currently is not available via qemu-devel. Suppressing mmu_exported_pagesize_mask from 0x2b to 0xb live via kmdb helps it boot, but the system is extremely slow (boot to console in ~18 hours) -- see artefact.

9 Automated-install (AI) / text-install images: these reach system maintenance mode (Console login service(s) cannot runEnter user name for system maintenance) rather than a full console login. The installer goes on to mount a second slice of the same hsfs image, which is not RAM-backed here — the ISO's kernel carries no hsimd (Legion virtual-disk) driver, so that slice is unreachable. An image/driver limitation, not an MD or paging failure: the kernel itself boots cleanly. (pagesize_256m/4096 × Solaris 11 11/11 text additionally needed boot -kdvs; every other cell here booted -kdv.)

10 Root cause found and fixed on the QEMU side: target/sparc's sun4v TTE page-size field was only 2 bits wide, so any 32M/256M request silently aliased down to 64K/512K -- the QEMU TTE-page-size patch widens it to the full 3 bits (bit 48), so a 256M mapping is installed correctly instead of truncated. mmu-page-size-list=0x2b (what this build widens from tlb_fix's 0x9) is the MD-side half: it's what tells the guest kernel 256M pages exist at all, which is what makes it request one and exercise the fix in the first place. Footnote 8's workaround went the opposite direction -- suppressing this same value to 0xb at runtime via kmdb, dropping 256M entirely to dodge the bug on unfixed QEMU. With the fix in place, 0x2b no longer needs suppressing -- moreover, some kernels get 0x2b forced back in regardless of what mmu-page-size-list actually declares, whenever domaining-enabled is absent from the MD (init_md_broken()'s broken_md_flag ORs it in; tlb_fix itself only declares 0x9 but ends up with 0x2b at runtime for exactly this reason).

11 sol-11_1-ai-sparc.iso — Solaris 11.1 automated-install image. Boots, but extremely slowly: 09:11 to 17:34, about 8h20m to reach the single-user root shell, on an --enable-debug (-O0) QEMU build. A plausible cause is the cost QEMU pays to evict a live 256M TLB entry — replace_tlb_entry() invalidates the mapping one TARGET_PAGE_SIZE at a time, so 32768 tlb_flush_page() calls per eviction — but this run was not profiled, so that is a hypothesis, not a measurement.