GitHub - danindiana/rtx3080-rebar-vbios: Scripts and guide for flashing RTX 3080 VBIOS to enable Resizable BAR (8GB BAR1) on Linux

Enabling Resizable BAR (8 GB BAR1) on an ASUS TUF-RTX3080-O10G-GAMING under Linux, without Windows, without a GUI flash tool, and without bricking the card.

Table of Contents

Background System What Is Resizable BAR and Why Does It Matter? Prerequisites Scripts How the Flash Works ROM Selection Lessons Learned — What Went Wrong First Flash Results Post-Flash Steps Recovery Plan Diagrams File Inventory Analysis: ReBAR Speedup Model Analysis: BAR1 Mechanics

Background This repo documents a multi-attempt Linux VBIOS flash campaign to enable Resizable BAR on an ASUS TUF RTX 3080 10 GB card in a dual-GPU system that also contains an RTX 5080. The RTX 5080 had ReBAR active out of the box; the 3080 did not because its factory VBIOS lacked the ReBAR capability bit. ASUS ships VBIOS updates as Windows .exe self-installers. This project extracts the ROM from inside the installer and flashes it directly via the Linux build of nvflash, handling all the complications that come with doing this on a live X11 desktop without a spare recovery machine. End result: VBIOS updated from 94.02.42.40.65 (AS03) to 94.02.42.40.66 (AS02), flash exit code 0, reboot pending for BAR1 expansion.

System

Component Detail

Machine worlock

OS Xubuntu 22.04.5 LTS, Linux 6.8.12

Motherboard ASRock X570 Taichi

CPU AMD Ryzen 9 5950X (32 threads)

RAM 128 GB DDR4

GPU 0 RTX 5080 — PCIe 0E:00.0

— 16 GB — ReBAR ENABLED (control)

GPU 1 RTX 3080 — PCIe 0F:00.0 — 10 GB GDDR6X — ReBAR target

3080 Subsystem 10DE:2206 / 1043:87B0 (TUF-RTX3080-O10G-GAMING)

Display Connected to GPU 1 (RTX 3080, Disp.A: On)

Driver NVIDIA 580.142 / CUDA 13.0

Display Manager LightDM (Xfce default)

Inference Ollama — both GPUs, CUDA_VISIBLE_DEVICES=0,1

nvflash 5.867.0 (Linux x64 binary)

Target ROM 94.02.42.40.AS02.rom (V6 package, 2023-10-18)

What Is Resizable BAR and Why Does It Matter? PCIe devices expose their memory to the CPU through fixed-size windows called Base Address Registers (BARs). GPU BAR1 is the window through which the CPU can directly read and write VRAM. Traditionally, BIOS firmware assigned this window at a fixed 256 MiB regardless of how much VRAM the GPU actually has — a legacy of 32-bit address space constraints. Resizable BAR (also marketed as AMD Smart Access Memory / SAM) is a PCIe feature that allows the BIOS to negotiate a BAR1 window as large as the GPU's full VRAM. On a 10 GB RTX 3080, a working ReBAR implementation expands BAR1 from 256 MiB to 8192 MiB. Why 8192 MiB, not 10240 MiB? PCIe BAR sizes must be powers of two. 8 GiB is the largest power-of-two that fits within 10 GB, so that is what the BIOS assigns. The remaining ~2 GB of VRAM is still accessible by the GPU internally; only the CPU-visible window is bounded by the BAR.

Performance implications for multi-GPU inference (Ollama) When Ollama loads a model across two GPUs (RTX 5080 + RTX 3080), the runtime may need to move activations, KV-cache entries, and layer state across the PCIe fabric. ReBAR expands the CPU-addressable window into VRAM from 256 MiB to 8192 MiB, reducing remapping pressure for large transfers. It is not equivalent to CUDA Peer-to-Peer access, which is a separate runtime capability gated on topology, NVLink/PCIe connectivity, and driver negotiation: cudaDeviceCanAccessPeer(&canAccess, deviceA, deviceB);

ReBAR improves the memory-mapping situation. It does not, by itself, guarantee CUDA P2P.

Prerequisites Motherboard BIOS (do this first) These settings must be active before any VBIOS flash will have visible effect:

Above 4G Decoding → Enabled (ASRock X570 Taichi: Advanced → AMD CBS → NBIO Common Options → SMU Common Options) Re-Size BAR Support → Enabled (same submenu; may also appear as "SAM" or "Smart Access Memory") CSM (Compatibility Support Module) → Disabled (Boot → CSM → Disabled — ReBAR is a UEFI feature; CSM silently breaks it)

Verify the 5080 (which has ReBAR natively) is already showing a large BAR1 to confirm these settings are active before proceeding: sudo lspci -vvv -s 0e:00.0 | grep -A5 "Resizable BAR" # Expected: BAR 1: supported: 64MB 128MB ... 16GB Kernel parameter # Check if pci=realloc is active cat /proc/cmdline | grep -o 'pci=realloc'

# Add it if missing (edit /etc/default/grub):

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=realloc" sudo update-grub sudo reboot Tools required sudo apt install p7zip-full expect

expect is required because nvflash opens /dev/console directly for confirmation prompts and does not respond to piped yes — a pseudo-TTY is mandatory.

Scripts

Script Purpose Run as

preflight-check.sh Read-only prerequisite check. Safe to run any time. Confirms nvflash, ROM, UEFI mode, current BAR1 state. user

vbios-preflight.sh Stop services, unload drivers, backup VBIOS. Kills X session. root, from TTY

rebar-enable.sh Full end-to-end flash: stop services → unload → backup → flash → reload → restart X. root

vbios-verify.sh Post-reboot verification. Checks VBIOS version, BAR1 size, PCIe capability register, GPU inventory. user

Quick start (if prerequisites are met) # 1. Sanity check (read-only, safe) bash ~/vbios-work/preflight-check.sh

# 2. Run the flash (kills X — desktop will go black, then come back) sudo ~/vbios-work/rebar-enable.sh

# 3. Reboot sudo reboot

# 4. Verify bash ~/vbios-work/vbios-verify.sh

How the Flash Works rebar-enable.sh orchestrates the entire flash in a single run, designed to be launched from an X terminal even though it will kill X partway through (via systemctl stop lightdm). The script is a systemd service in all but name — once LightDM drops, the script keeps running in the background as a child of the shell that launched it. Step-by-step [0] Stop all GPU consumers ├── ollama.service (systemctl stop) ├── gpu-watchdog.service (stop + pkill -9, has Restart=always) └── nvidia-persistenced (MASK first, then stop — mask blocks auto-restart)

[1] Stop LightDM ← X session dies here; terminal window closes (sleep 4s for Xorg

to fully exit)

[2] Unload NVIDIA kernel modules in dependency order: nvidia_uvm → nvidia_drm → nvidia_modeset → nvidia

[3] nvflash --list (confirm GPU 1 visible without driver)

[4] Backup current VBIOS (skip if 3080-backup-pre-rebar.rom already exists) nvflash -i 1 --save backup.rom

[5] ROM compatibility check (nvflash --check; proceed regardless) [5b] Write-protect off (nvflash --protectoff; expect-driven)

[6] FLASH ← ~30 seconds, EEPROM write nvflash -i 1 94.02.42.40.AS02.rom (expect drives the y/n confirmation via pseudo-TTY)

[7] Read-back verification (nvflash --save /tmp/verify.rom; check non-empty)

[8] Reload NVIDIA modules (nvidia → nvidia_modeset → nvidia_drm → nvidia_uvm) [9] Restore services (unmask + start nvidia-persistenced, gpu-watchdog) [10] Restart LightDM ← desktop comes back

[11] Report SUCCESS / FAILURE to flash-session.log

See diagram 03 for the full annotated flowchart.

ROM Selection The ASUS V6 package (2023-10-18, RTX3080_V6.exe, 15 MB) contains four ROM files. Only two are valid for the TUF-RTX3080-10G-GAMING (subsystem 1043:87B0):

ROM Subsystem Version Board Notes

AS02 1043:87B0 94.02.42.40.66 TUF-RTX3080 Use this. Newer build.

AS03 1043:87B0 94.02.42.40.65 TUF-RTX3080 Valid fallback, older build.

AS08 1043:87EB — TURBO-RTX3080 Wrong card. Do not flash.

AS09 1043:87EB — TURBO-RTX3080 Wrong card. Do not flash.

Important: Neither the displayed decimal suffix nor the ASUS ASxx number is globally monotonic across the ROM set. For this specific 1043:87B0 TUF PCB, AS02 displays as .66 and AS03 displays as .65. AS02 is the preferred V6 target, while AS03 is a valid fallback — but this conclusion comes from the ROM metadata and package comparison table, not from assuming .66 > .65 or AS03 > AS02. Across the full package, AS08/AS09 have higher AS numbers but belong to the Turbo 1043:87EB branch and must not be flashed to the TUF 87B0 card.

ImportantROM selection invariantDo not trust the displayed version suffix. Do not trust the AS-series number order. Do not trust ROM filenames or package names.Trust only: Subsystem ID: must be 1043:87B0 Power target: must be 340 W, not 320 W Boost clock: must be 1785 MHz, not 1710 MHz nvflash --check: must report no subsystem mismatch The safe comparator is not .65 vs .66, and not AS02 vs AS03. The safe comparator is:subsystem + board family + power target + boost clock + nvflash check

See diagram 05 for the full decision tree.

Lessons Learned — What Went Wrong First Three earlier flash attempts failed silently. Here is what happened and why. Root cause: Windows-only nvflash flags on Linux The flash script originally used: nvflash -i 1 --force-subsystem-id --force-board-id rom.rom These flags do not exist in the Linux build of nvflash 5.867. When the binary encounters unknown flags, it prints Command format not recognized and falls into interactive menu mode instead of aborting.

The expect script was written to handle the direct-flash prompts, but it was now navigating an unexpected interactive menu. It exited with code 5 (nvflash's interactive compatibility check failure code). The EEPROM was written — but via the interactive path, which landed on AS03 (.65) instead of the intended AS02 (.66). How this was diagnosed: strings ~/vbios-work/x64/nvflash | grep force # Output: --forcesub, --forceboard — but those also fail # The Linux binary simply has no equivalent of the Windows force flags # for same-subsystem ROMs, because they aren't needed: nvflash accepts # a matching ROM without any force flag Correct invocation on Linux: nvflash -i 1 <rom_file> # That's it. No force flags. The binary prompts y/n if subsystems match. Why nvidia-persistenced had to be masked, not just stopped nvidia-persistenced holds open file descriptors to every NVIDIA device file (/dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, /dev/nvidia-modeset) even when no GPU workload is running. It cannot be simply stopped with systemctl stop if it has Restart=always in its unit file. The solution is to mask the unit first, which prevents systemd from restarting it, then stop it. After the flash, unmask and restart. systemctl mask nvidia-persistenced systemctl stop nvidia-persistenced # ... flash ... systemctl unmask nvidia-persistenced systemctl start nvidia-persistenced Why gpu-watchdog needed pkill -9 gpu-watchdog.service (a thermal monitoring daemon) also has Restart=always. A plain systemctl stop sends SIGTERM and systemd immediately restarts it before the stop completes. The script uses systemctl stop followed by pkill -9 -f gpu_thermal.watchdog to terminate the process before systemd can react, then waits for the unit to reach inactive state. Why expect is required (not yes | nvflash) nvflash opens /dev/console directly for confirmation prompts, bypassing stdin.