PID 1 - Post 1

So firstly I want to point out why this post is here. I wanted to create a file server for my house. I didn't want to use NAS hardware or a pre-rolled NAS software because I enjoy figuring things out on my own and playing around with various things.

I guess I should also state that I had no intention of using a RAIDZ file system for my root file system. I am most likely going to use a ZFS mirror of two smaller drives. These wont even be on the LSI SAS controller, and will be on the motherboards built-in SATA connectors. I was mainly testing out the hardware I already had when I found this issue.

So the hardware I have assembled so far (that is important to this post) is a Super Micro X10SL7-F Motherboard and 6 x Western Digital Red 3TB. The drives are all connected to the LSI SAS controller (on the motherboard).

Before I go any further I want to state very clearly...

I DO NOT SUPPORT OR RECOMMEND FLASHING FIRMWARE!

DO SO AT YOUR OWN RISK!

Since I would be using FreeBSD and ZFS I flashed the firmware on the LSI 2308 controller to IT Mode. I used the firmware from SuperMicro's site and the instructions that came with it. There are also other guides on the Internet to do this. So I wont include any of that here.

Once the LSI controller is in IT Mode it no longer does RAID operations on the drives and each drive shows up separately as an individual device. This is perfect for ZFS.

That part went smoothly and I could see each drive listed. So I booted FreeBSD's installer (from a USB stick, not that it matters). Every drive showed up as an individual device just as I expected and wanted. So I went a head and did a default auto layout install. 2GB swap, RAIDZ2 root with GPT. No settings adjusted except choosing the 6 drives. The installer completed without a single error or oddity. Great right?

Nope. After a reboot I got some weird errors...

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS object directory
ZFS: can't find root filesystem
gptzfsboot: failed to mount default pool zroot

Of course zroot could be any name, but I went with the default install. Then I was left at a boot prompt. I was really shocked to be here. This was all new hardware and a default install. What could be wrong, was it a hardware issue like a controller issue or dead drive already?! Was it an assembly error like a cable not in tight? Or some other setting or issue I wasn't even thinking of?

So here I am at a boot prompt wondering what to do or even what commands to type. I'm still trying to make the move from Linux to various BSD systems. So this was a little confusing.

Now I start to use my searching abilities and look online to see if someone had a similar issue. Of course when it came to this hardware MANY FreeNAS or NAS4Free users had a similar setup, but they were all booting off a small USB drive and not the LSI controller. In fact most of them have the Boot ROM option in the BIOS off for their LSI controller to speed up the boot.

After not really finding anything of help, I started to try various things. This is not in any particular order (in fact I'm sure this is not the exact order I did it in). I made sure the other components had the latest firmware/bios updates... still no luck. I rewired all the SATA cables... still no luck. I re-installed FreeBSD making sure I was modifying as little as possible... still no luck. I went through every BIOS setting, and tried some different combinations... still no luck. I checked the LSI controller configuration (but not as closely as I should have, we will get to that later)... still no luck.

Now I focused a bit more on what to type at the prompt. I found that I could specify a drive, but that didn't help much because no matter what da drives (which is what the LSI controller drives show up as in FreeBSD) I tried I got the same result or a not found error. That was some what useful, as I new all the drives weren't visible or present. Still no clue why though.

Then I found out you could type the word "status" and it would show a zpool status. This was great, because it showed one drive as Online and all the others Offline.

So back to the LSI configuration. I had looked through here before and setup Boot and Alternate Boot devices, and thought I had seen all the screens. This time I found the key setting...

Select Adapter (LSI2308-IT in my case)

Advanced Adapter Properties

Advanced Device Properties

Maximum INT 13 devices for this adapter

I guess my mind glazed over it, as it wasn't obvious to me what that was until I used the Help function. Which in the most basic terms is the number of devices that the LSI controller will present to the system pre-OS. This is why all the installs I did worked, because post-OS was just fine. Which is also why the FreeNAS, NAS4Free and other people are perfectly fine with the board. Mine was set to 1 by default, the exact same number of drives that were showing in my boot prompt ZFS status. Of course RAIDZ wont boot if you are missing more than the number of drives that can be failed. So in my RAIDZ2 setup this would be 2. Basically I could have 4 drives and boot just fine, but any less and it would fail. Not that booting degraded would be that great either.

I figured I have 8 ports and only 6 drives anyways (I think the max setting is 24 or 32), so I set Maximum INT 13 devices for this adapter to 8 and rebooted and it worked!

To summaries up what is required (most of these are defaults):

SuperMicro BIOS (Del) -> Advanced -> PCIe/PCI/PnP Configuration -> Load Onboard SAS Option ROM = Enabled
LSI Corp Config Utility (Ctrl-C) -> Select Adapter (LSI2308-IT in my case) -> Boot Support = [Enabled BIOS & OS]
LSI Corp Config Utility (Ctrl-C) -> Select Adapter (LSI2308-IT in my case) -> Advanced Adapter Properties -> Advanced Device Properties -> Maximum INT 13 devices for this adapter = 8

I hope someone finds this and finds it helpful. It may even be useful for other motherboards and controllers if you look for similar settings. The concept should be the same.