Fixing a TrueNAS SCALE Boot Issue with Proxmox After a Cold Start

TL;DR: TrueNAS SCALE was failing to boot after a cold start of my Proxmox server because the HBA being passed through to the VM had ROM-Bar enabled. Disabling ROM-Bar fixed the issue.

Problem

TrueNAS SCALE will fail to boot whenever I restarted or cold booted my proxmox server. The strange thing? Clicking enter to continue past the error, or restarting the VM would make it magically start normally with a healthy boot. So what’s wrong?

In this post we’ll go through my findings, how I fixed it with the incredibly simple fix. But the journey there is the important part. I couldn’t find many well indexed resources on this combination so finding potential solutions took a bit more digging than I’d like… Hopefully this post prevents the next person suffering like I did!

Could AI of done it faster? Yes! But that removes the learning and that’s what my labs about… Independent investigation → Search Engine → AI. It’s great as a tool, and feeding it through after the fact with a bit of prompting got me to a solution in ~10 minutes and 2 reboots.

Identifying the cause… The fun bit

The biggest issue with this and why it took so long was it became a horrible cycle of try a fix, wait til a time I can reboot and then see if it worked (and probably not was the answer for the first few attempts). There’s also the fact that as it’s a bios error;

• Proxmox thinks it’s started fine!
• The OS hasn’t taken over yet!
• No boot issues from proxmox in the journal and other logs.

Investigating timings and other Proxmox behavior didn’t prove particularly useful. There were some weird oddities, but nothing that stood out as relevant, which is why I’ve not provided logs. They simply didn’t help.

This means that no matter where you looked the error was as below it’s all there was to work with.

So now I’ve completed all the diagnosis I know of, it’s off to google. Without much success I eventually got there with some miscellaneous search terms which I can’t find in my history but eventually stumbled on a useful fix.

However, before this came a lot of different attempts, gaining more understanding and trying; GRUB changes, PCI commands and log investigations and finally playing with boot and VM startup orders. All to no success and chosen words made.

I can’t prove or find what caused this as it used to be ok and also can’t identify when it became an issue as restarts are rare. My theory is that I-GPU passthrough implemented several months ago caused this, but its also likely I just thumbed the button at some point.

So finally the fix found, applying and restarting was getting more nerve wracking after each attempted fix and check. Not to mentioned each one resulted in an onslaught of outage notifications and me scratching my head along with maybe some swearing. Thankfully watching it boot it all worked perfectly! A huge sense of relief a nice reward and a nice post to write about and publish to hopefully help others.

The fix

The fix in the end was super simple. It was a case of removing ROM Bar on the PCI device pass through. The issue? Loading the ROM BAR as part of the VM caused issues. Solution Link reference 1 or Solution Link reference 2

Lessons

• I guess the main one is that, on a VM with PCI passthrough, rombar is an important diagnostic setting to try when troubleshooting errors. But there are plenty of other things that can cause passthrough issues too, including drivers, IOMMU configuration, failed device resets and more.
• This added to the lessons of the Physical GPU reset issue I found when doing I-GPU passthrough (maybe a backdated blog post for some time but I’m still recovering from that pain.).
• I need to be more conscious of the hardware and OS level specifics when using emulation going forward.
• The docs just say some devices will have issues without it but never go into more detail. Piecing together some uni knowledge with hindsight and I understand how it’s happening with microcode loading and device takeover but thats out of scope for here.
• Automated provisioning of VM’s is important to continue working on… If I did thumb the checkbox at some point in time then it would have self resolved on a regular run of said system.