PC-BSD Forums

PC-BSD Forums (http://forums.pcbsd.org/index.php)
-   Installing PC-BSD (http://forums.pcbsd.org/forumdisplay.php?f=11)
-   -   Multiboot PC-BSD 9.1 RC3 on ZFS GPT partition, using Grub2 (http://forums.pcbsd.org/showthread.php?t=18701)

darman 12-02-2012 04:36 PM

Multiboot PC-BSD 9.1 RC3 on ZFS GPT partition, using Grub2
Hi! I am happy to share my success booting PC-BSD 9.1 RC3 on ZFS GPT partition, using Grub2.

Over the last 4/5 years I rapidly became a multiboot freak. From dual boot to triple, quad and multi it was a flash!
As disks grow bigger and cheap, I started using some of my older disks that were being replaced by the terabyte disks, to install and tryout different OSes and distros.

My basic goal is to manage to install and manage internet, audio, display and print.
Of course meanwhile I set things up I learn a few things and manage to get the 'feeling' of the OS or distro. I also learned a few things about partitioning and boot managers, backup/restore software, recover from disaster software, etc.

Freak? yes because presentaly I have around 70 partitions on 1 disk (GPT partitioning) and 30 partitions on another (msdos partitioning) not to mention my windows disks.
So, briefly I have windows 32 bit and 64 bit installed, hackintoshes Mountain Lion and Lion, Open Indiana Solaris, all 'major' Linux distros and many less known ones, covering the different graphic environments available and I also love tiny distros (puppy, slax, austrumi, slitaz, etc) ... you name it, I try them all. I tried PC-BSD (ufs2 partioning) for the first time on version 8.0.

The 2 major problems for any OS or distro on a multiboot environment: being able to install it (not 'cleaning' the others already installed) and being able to make it boot using a boot manager.
I use bcdedit on windows, grub2 and grub legacy on linux , chameleon or chimera on hackintosh. They generally can chainload each other which is convenient.

So, installing and booting an OS or distro and configure it in a separate disk is not a big challenge. Many can install directly to a partition but some like PC-BSD needs to be installed on a separate disk.
I have an old 20 GB ide harddisk stored and I use it for these situations. So, I chose zfs and gpt partitioning and installed PC-BSD. It makes a tiny 64 k boot partition and the rest is a zfs raid partition. It booted fine in this separate disk with it's own loader and configured automatically wired internet, bluetooth and display. Audio not yet.

I backed up the raid partition with DD to an .img file directly to a windows(eheheh) partition (pity fsarchiver is not working for BSD). Created a partition slightly bigger than the one I backed up on my multidistro gpt disk and DD restored the .img file to it.

Searched the net, wiki's and how to's in order to make it boot and even the solutions described in official PC-BSD sites did not work.
Grub2 or bcd edit did not work. Chameleon that is the only one to auto detect and chainload Solaris openIndiana could not see it.
Grub legacy also manages to chainload openIndiana legacy grub but is not working on gpt disks unless is a special patched version...my god I was lost.

My only option was grub2 and I tried everything I could think of and could find on the net (see bellow). I tried to boot my original installation small disk using these boot managers and also not working but with one advantage: I could see the disk was being accessed using grub2 but failed to boot. The errors were: could not find 'freebsd', or 'boot', or 'root'...,etc

Fortunately I came across this page http://wiki.freebsd.org/ZFSOnRoot and adapted it to my situation. With the help of some other pages for zpool and zfs commands (see page links bellow), I managed to create a bootdir in my tank1 pool, copy all the boot files from my tiny 64 k boot partition inside and renamed just for the exercise from bootdir to boot.
So, I created a dir, mounted it, copied files, unmounted and renamed, inside the pool. I also changed the mountpoint from /mnt to /boot. Not so bad for me that knows nothing about zfs.
As I had made a 2nd install on my 20 GB disk, the pool automatically was named tank1 (on 1st install was tank0).

[darman@pcbsd-208] ~% zfs list
tank1 11.1G 7.27G 144K legacy
tank1/ROOT 5.54G 7.27G 144K legacy
tank1/ROOT/default 5.54G 7.27G 5.54G /mnt
tank1/boot 220M 7.27G 220M /boot # here is my created bootdir, renamed to boot and mountpoint changed from /mnt to /boot. I have to work on the size though. It's too big.
tank1/root 1.58M 7.27G 1.58M /root
tank1/swap 2.06G 9.33G 72K -
tank1/tmp 428K 7.27G 428K /tmp
tank1/usr 3.25G 7.27G 144K /mnt/usr
tank1/usr/home 72.0M 7.27G 152K /usr/home
tank1/usr/home/darman 71.9M 7.27G 71.9M /usr/home/darman
tank1/usr/jails 144K 7.27G 144K /usr/jails
tank1/usr/obj 144K 7.27G 144K /usr/obj
tank1/usr/pbi 2.40G 7.27G 2.40G /usr/pbi
tank1/usr/ports 790M 7.27G 790M /usr/ports
tank1/usr/ports/distfiles 144K 7.27G 144K /usr/ports/distfiles
tank1/usr/src 144K 7.27G 144K /usr/src
tank1/var 880K 7.27G 144K /mnt/var
tank1/var/audit 144K 7.27G 144K /var/audit
tank1/var/log 432K 7.27G 432K /var/log
tank1/var/tmp 160K 7.27G 160K /var/tmp

It was still not working sometimes with 'unknown filesystem' error. So I started thinking it could be also a grub2 problem.
I found this page with a version of grub2 with zfs fix: https://launchpad.net/~rlaager/+archive/ppa/ and installed it on one of my distros (Ubuntu 12.1 Quantal Quetzal). It supports the last grub2 2.00 and rlaager version improves and corrects zfs recognition.
After a few trials with my menuentry's I managed a winner config and made it boot (see bellow).

menuentry "PC-BSD 9.1" {
insmod zfs
search --no-floppy --fs-uuid --set=root 4378674875213651791
search --fs-label tank0 --hint hd0,gpt63
kfreebsd /freebsd@/boot/kernel/kernel
kfreebsd_module_elf /freebsd@/boot/kernel/opensolaris.ko
kfreebsd_module_elf /freebsd@/boot/kernel/zfs.ko
kfreebsd_module /freebsd@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank0/freebsd
menuentry "PC-BSD ZFS" {
insmod zfs
search -s -l tank0
kfreebsd /@/boot/kernel/kernel
kfreebsd_module_elf /@/boot/kernel/opensolaris.ko
kfreebsd_module_elf /@/boot/kernel/zfs.ko
kfreebsd_module /@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank0
menuentry 'PC-BSD' {
insmod zfs
search -s -l tank0 --hint hd1,gpt63
kfreebsd /root@/boot/kernel/kernel
kfreebsd_module_elf /root@/boot/kernel/opensolaris.ko
kfreebsd_module_elf /root@/boot/kernel/zfs.ko
kfreebsd_module /root@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank0/root

And the winner is:

menuentry 'PC-BSD' {
insmod zfs
search -s -l tank1 --hint hd1,gpt63
kfreebsd /boot@/loader

Nothing more and it works. It chainloads the PC-BSD loader and it boots without errors.

Thank you to all posters for the wonderful info posted on the net that helped me succeed. Hope my post can help others.


My system: Corsair Carbide 400R box, XFX PSU, Gigabyte Z77-DS3H (onboard sound disconected in Bios), 16 GB ram, 6 or 7 sata disks, NVidia 9400 GT, Audiophile M-Audio 2496 PCI board, bluetooth pen and LG HD led flat screen.

http://www.rodsbooks.com/gdisk/ # special mention for this excellent software that I use for quite a while now


darman 12-03-2012 07:31 AM

Hi again. Sound already working now and openSuse 12.2 grub is also able to boot PC-BSD using my winner config.
Anyone care to help to resize my /boot dir and make it smaller? 64k is enough, I guess!

jaxxed 12-03-2012 07:36 AM

This is very usefull for me. Thanks for the grub settings.

darman 12-31-2012 12:11 AM

I did it again!
Hi again. A bit after the final 9.1 release I decided to upgrade. I had installed 9.1 RC3 so I did a net upgrade. It went well at least on the end result. There was a reboot after the download and it continue the install. Maybe that is the way...but on reboot I thought it had crashed.
Upgraded fine but on the system about screen there was a reference to the 9.1 final release and also to the RC3 so, I thought " I do it from scratch!
Download, burn and install, dd to image and dd restore to gpt disk partition, as I did with RC3. Problem was it did not work!

Install was fine and running on my 20 GB HD but after the dd restore to gpt disk boot was stopping at "...trying to mount zfs:tank0/ROOT/default..." and it hang on a shell to manually set the path to the boot device.
After dd restore trials including new saves and new restores I investigated the restored partition and tried to import it. 'tank0' was allways faulty or damaged or unavailable. Again on all (and I mean a lot) of these trials I included some new installs also and dd restores directly to gpt partition, so, I realized it was the dd backup that was not working on this final 9.1 release. It worked fine on RC3 and so something changed for the final release or I did something I don't remember on the RC3 and was just lucky.
Meanwhile I searched the net and indeed I found references to dd not working on zfs or working only in special conditions.

I found a method using zfs-guru live cd, tried it and did not work. Read various suggestions and explanations it was because of the cache file in zfs and of course tried it but failed. The resulting dd restored partition was always damaged when I try to import and system advised to zpool destroy tank0 and restore from a backup. I ended up corrupting my original install 20 GB HD with all the multiple trials of mounting and importing... and today after 1 new install in my small disk I had a good idea. On the final install screen where it says 'Install finished ...click to reboot' I thought: "why not try dd before the system first run?" So tried to open a terminal with keyboard keys and found it on the right mouse button. Opened and dd directly my ada0p2 partition to my ada2p62 partition.
By the way, I had created my bootdir already on the customize partitions installer screen that gives you the option to create dirs.

After dd I rebooted, tested my 20 GB disk and found it was ok on first boot.
As it automounted my windows partitions I copied my empty bootdir into a ntfs partition and copied everything inside my folder /boot to the bootdir in the ntfs partition. Shutdown and disconnected my 20 GB HD.
Started up with zfs-guru live cd and did zpool import.
It now showed an error free online tank1 pool on my gpt disk. So I imported it and mounted also my ntfs partition. Copied all dirs and files from ntfs/bootdir into the empty tank1/bootdir.
As I know nothing from zfs I made te mistake of exporting tank1 after and later I read I should not do it (bootdir continued empty).

So, I repeated the process and did not export, rebooted and tried to boot from my ada2p62 gpt partition. It worked! I had to change boot@/loader to bootdir@/loader on my grub2 config but it started up fine into a first boot, setting up screen, root password, user...
Up and running again and I learned a few more things about PC-BSD and zfs.

bobwya 01-27-2013 12:14 AM


No you're not a freak... Uhmmm you are perfectly normal - it's them others :)

I have just started dabbling with PC-BSD on my laptop. It is a multi-boot setup with various Linux distros (Gentoo, ARCH, Mint-KDE, Sabayon, etc.) and Windows 7. I wanted to install a BSD-derivative to play about with ZFS.

Anyway I have the boot (SSD) disk setup with Grub-2 and an DOS partition table - gotta love Windows (I know I could probably use GPT with a dummy MBR for Windows). So I tried installing PC-BSD on a primary partition using ZFS and not installing a boot loader (figured that would nuke my Grub-2 boot-loader). Then I set about installing the native ZFS module on my Linux-distro controlling Grub (currently Chakra) - not so easy. I've managed to important the tank0 zpool into a mountpoint on my ARCH partition.

I'm still a complete noob about ZFS. So I'm just experimenting at present. But am I on the right track do you think? If my zpool has a boot directory (I will test later - when I've finished messing about in Gentoo) and I can load the ZFS kernel module in my Linux distro controlling Grub-2?

I've used Berkley UNIX way back at Uni. But I'm getting a bit soft and have gotten used to my Linux-based distros - what with BASH completion, Gawk extensions, systemd, etc. So FreeBSD is a bit out of my comfort zone - everything (commands, file-system structure) is a bit like "Alice Through the Looking Glass"!!

Thanks for the great tips, :cool:

darman 10-30-2013 12:31 AM

Back again with 9.2
Hi. I had some problems to make Grub2 boot PC-BSD 9.2 from a partition on my GPT multisystem / multiboot disk (a mix of OSX and many linux distros).
Grub I use is from Linux distro, not PC-BSD although it's probably possible.
I also did not manage to make the bsd loader to boot the system. I managed to initiate the bootstrap process but it dies soon after cause it can't chainload the system (probably it's pointing the wrong path and it can't find it).

I added my PC-BSD 9.2 installation on a separate disk to partition 63 usin DD (like I did before on previous versions) but that resulted also in a lot of problems with faulted pool and boot problems.

I ended up using a much better method to clone my system to my ada1p63 described here: http://www.aisecure.net/2011/03/26/c...otable-system/

Partition size can be smaller than source partition (unlike DD) and pool guid is different from source partition (I did not know about reguid on 1st trials and I don't know how easy or difficult it is cause I have not tried it).

The boot menuentry is basically the one from PC-BSD grub with some tweaks.
It loads kernel directly, loads modules, sets hints and boots.
Grub on linux distro needs to be well setup so that it is able to read the pool guid.
Test it with command:
grub-probe --target=fs_uuid --device /dev/sdb63 (sdb63 in my case).
To me it returns: 450eb04d8517f599 and needs to be specified on search line.
And menuentry I am using is:

menuentry "PC-BSD 9.2" {
insmod zfs
search --set=root --fs-uuid 450eb04d8517f599 --hint hd1,gpt63
kfreebsd /ROOT/default@/boot/kernel/kernel
kfreebsd_module /ROOT/default@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank2/ROOT/default
kfreebsd_module_elf /ROOT/default@/boot/modules/vboxdrv.ko
kfreebsd_module_elf /ROOT/default@/boot/modules/nvidia.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/sdhci.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/zlib.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/opensolaris.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/zfs.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/tmpfs.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/linux.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/crypto.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/geom_journal.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/geom_mirror.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/geom_eli.ko
kfreebsd_module_elf /ROOT/default@/boot/kernel/aesni.ko
set kFreeBSD.hint.fdc.0.at=isa
set kFreeBSD.hint.fdc.0.port=0x3F0
set kFreeBSD.hint.fdc.0.irq=6
set kFreeBSD.hint.fdc.0.drq=2
set kFreeBSD.hint.fd.0.at=fdc0
set kFreeBSD.hint.fd.0.drive=0
set kFreeBSD.hint.fd.1.at=fdc0
set kFreeBSD.hint.fd.1.drive=1
set kFreeBSD.hint.atkbdc.0.at=isa
set kFreeBSD.hint.atkbdc.0.port=0x060
set kFreeBSD.hint.atkbd.0.at=atkbdc
set kFreeBSD.hint.atkbd.0.irq=1
set kFreeBSD.hint.psm.0.at=atkbdc
set kFreeBSD.hint.psm.0.irq=12
set kFreeBSD.hint.sc.0.at=isa
set kFreeBSD.hint.sc.0.flags=0x100
set kFreeBSD.hint.uart.0.at=isa
set kFreeBSD.hint.uart.0.port=0x3F8
set kFreeBSD.hint.uart.0.flags=0x10
set kFreeBSD.hint.uart.0.irq=4
set kFreeBSD.hint.uart.1.at=isa
set kFreeBSD.hint.uart.1.port=0x2F8
set kFreeBSD.hint.uart.1.irq=3
set kFreeBSD.hint.ppc.0.at=isa
set kFreeBSD.hint.ppc.0.irq=7
set kFreeBSD.hint.atrtc.0.at=isa
set kFreeBSD.hint.atrtc.0.port=0x70
set kFreeBSD.hint.atrtc.0.irq=8
set kFreeBSD.hint.attimer.0.at=isa
set kFreeBSD.hint.attimer.0.port=0x40
set kFreeBSD.hint.attimer.0.irq=0
set kFreeBSD.hint.wbwd.0.at=isa
set kFreeBSD.kern.ipc.shmseg=1024
set kFreeBSD.kern.ipc.shmmni=1024
set kFreeBSD.kern.maxproc=10000
set kFreeBSD.legal.intel_ipw.license_ack=1
set kFreeBSD.legal.intel_iwi.license_ack=1
set kFreeBSD.vfs.zfs.prefetch_disable=1
set kFreeBSD.hint.acpi_throttle.0.disabled=1
set kFreeBSD.machdep.disable_mtrrs=1
set kFreeBSD.kern.geom.eli.visible_passphrase=2
set kFreeBSD.kern.cam.scsi_delay=500
set kFreeBSD.kern.hz=100

I am still tying to simplify it and / or make the bsd bootstrap chainload the system and boot as on normal PC-BSD install.
If somebody knows how, appreciate the tips.


P.S. If pool name is changed like I did from original tank to clone tank2, loader.conf on clone needs to be changed accordingly

MarkBenavides 11-09-2013 12:05 PM

grub-probe error
Good morning! I had no problems using Mint Olivia / LMDE grub2 to boot PC-BSD 9.1 but simply cannot do the same for chainloading the new PC-BSD 9.2 ZFS filesystem. I get an 'invalid signature' that others have posted. In fact I do 'chainload' FreeBSD 9.2 and this works just fine.


# grub-probe --target=fs_uuid --device /dev/sdb1
grub-probe: error: unknown filesystem.

However, using blkid give me 13 lines of output (only posting the 2 pertinent lines)

# blkid
/dev/sdb1: LABEL="tank2" UUID="919430145" UUID_SUB="1717606850"
/dev/sdc4: TYPE="ufs"

Note the lack of a UUID for the UFS filesystem (FreeBSD9.2) that I posted here.

Bottom line - I've not seen a UUID_SUB before, as listed above. So what's the recommended method to tweak a menu entry to get PC-BSD to boot from linux grub2? The expected (hd1,1) fails:

menuentry 'PC-BSD 9.2(64) (on /dev/sdb1) (hd1,1)'{
set root='(hd1,1)'
chainloader +1


All times are GMT. The time now is 09:35 AM.

Powered by vBulletin® Version 3.8.5
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.

Copyright 2005-2010, The PC-BSD Project. PC-BSD and the PC-BSD logo are registered trademarks of iXsystems.
All other content is freely available for sharing under the terms of the Creative Commons Attribution License.