Reply
 
Thread Tools Display Modes
  #1  
Old 12-21-2012, 11:24 AM
gregober gregober is offline
Junior Member
 
Join Date: Oct 2012
Location: Paris
Posts: 23
Thanks: 0
Thanked 0 Times in 0 Posts
Default Impossible to start jail
Hi,

I am trying to create a jail on a freshly installed server.
The only thing that i have done was moving /usr/jails on a different mountpoint located on a different pool.

But this is a long term requirement. I have my PCBSD (TrueOS) installed on a specific mirror disks and a quite large pool installed on other disks.

Quote:
root@back:/root # zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank0 3.39G 88.2G 144K legacy
tank0/ROOT 911M 88.2G 144K legacy
tank0/ROOT/default 911M 88.2G 911M /mnt
tank0/root 200K 88.2G 200K /root
tank0/swap 2.06G 90.2G 72K -
tank0/tmp 196K 88.2G 196K /tmp
tank0/usr 443M 88.2G 144K /mnt/usr
tank0/usr/jails 442M 88.2G 152K /mnt/usr/jails
tank0/usr/jails/.warden-chroot-amd64 442M 88.2G 442M /usr/jails/.warden-chroot-amd64
tank0/usr/jails/10.20.50.200 8K 88.2G 442M /mnt/usr/jails/10.20.50.200
tank0/usr/obj 144K 88.2G 144K /usr/obj
tank0/usr/pbi 256K 88.2G 256K /usr/pbi
tank0/usr/ports 296K 88.2G 152K /usr/ports
tank0/usr/ports/distfiles 144K 88.2G 144K /usr/ports/distfiles
tank0/usr/src 144K 88.2G 144K /usr/src
tank0/var 692K 88.2G 144K /mnt/var
tank0/var/audit 144K 88.2G 144K /var/audit
tank0/var/log 252K 88.2G 252K /var/log
tank0/var/tmp 152K 88.2G 152K /var/tmp
tank1 303M 8.01T 43.4K /tank1
tank1/home 82.3K 8.01T 82.3K /usr/home
tank1/jails 302M 8.01T 302M /usr/jails
tank1/snapshots 43.4K 8.01T 43.4K /usr/snapshots
And the pool :

Code:
root@back:/root # zpool status
  pool: tank0
 state: ONLINE
  scan: none requested
config:

	NAME                                            STATE     READ WRITE CKSUM
	tank0                                           ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/24d2fec6-4873-11e2-a6d3-001e67549fcd  ONLINE       0     0     0
	    gptid/239ae99f-4873-11e2-a6d3-001e67549fcd  ONLINE       0     0     0

errors: No known data errors

  pool: tank1
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	tank1       ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0

errors: No known data errors

Here is how the creation process went…*

Code:
root@back:/root # warden create 10.20.50.200 jail1-back.local --ports --startauto --portjail
Getting regional mirror...
Using mirror: http://ftp.free.org/pub/PC-BSD
Fetching jail environment. This may take a while...
Downloading http://ftp.free.org/pub/PC-BSD/9.1/amd64/netinstall/fbsd-release.txz ...
fbsd-release.txz                              100% of   91 MB 1388 kBps 00m00s

fbsd-release.txz.md5                          100% of   33  B  202 kBps

Creating ZFS /usr/jails/.warden-chroot-amd64 dataset...
Building new Jail... Please wait...
Fetching ports...
Done
touch: /usr/jails/10.20.50.200/etc/fstab: No such file or directory
/usr/local/share/warden/scripts/backend/createjail.sh: cannot create /usr/jails/10.20.50.200/etc/rc.conf: No such file or directory
/usr/local/share/warden/scripts/backend/createjail.sh: cannot create /usr/jails/10.20.50.200/etc/hosts: No such file or directory
cp: /usr/jails/10.20.50.200/etc/resolv.conf: No such file or directory
cp: /usr/jails/10.20.50.200/etc/resolv.conf: No such file or directory
cp: /usr/jails/10.20.50.200/etc/passwd: No such file or directory
cp: /usr/jails/10.20.50.200/etc/master.passwd: No such file or directory
cp: /usr/jails/10.20.50.200/etc/spwd.db: No such file or directory
cp: /usr/jails/10.20.50.200/etc/pwd.db: No such file or directory
cp: /usr/jails/10.20.50.200/etc/group: No such file or directory
cp: /etc/localtime: No such file or directory
chroot: ln: No such file or directory
mount: /usr/jails/10.20.50.200/dev: No such file or directory
mount: /usr/jails/10.20.50.200/proc: No such file or directory
Mounting /usr/jails/10.20.50.200/tmp
Mounting /usr/jails/10.20.50.200/media
Mounting /usr/jails/10.20.50.200/usr/home
Enabling linprocfs support.
jail -c path=/usr/jails/10.20.50.200 host.hostname=jail1-back.local  ip4.addr=10.20.50.200  persist
Starting jail with: /etc/rc
jexec: execvp(): /bin/sh: No such file or directory
Success!
Jail created at /usr/jails/10.20.50.200
But from there, I can't really do anything…*?
Do you have any idea ?


Sincerely yours.
Reply With Quote
  #2  
Old 12-21-2012, 12:14 PM
gregober gregober is offline
Junior Member
 
Join Date: Oct 2012
Location: Paris
Posts: 23
Thanks: 0
Thanked 0 Times in 0 Posts
Default [Bug report] zfs pool not handled correcly by warden
The warden definitely does not seem to like to have another pool specified as it's target… even though I have specified this in the warden config file*?

I have re-configured my zfs pool in order to have a the same dataset as it was when the system was fisrt brought up. then I have created a new mountpoint called /usr/myjails

But no luck…*It still wants to install all it's stuff everywhere and not on the requested pool target !

Code:
root@back:/root # zfs list
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
tank0                                   3.39G  88.2G   144K  legacy
tank0/ROOT                               911M  88.2G   144K  legacy
tank0/ROOT/default                       911M  88.2G   911M  /mnt
tank0/root                               200K  88.2G   200K  /root
tank0/swap                              2.06G  90.2G    72K  -
tank0/tmp                                220K  88.2G   220K  /tmp
tank0/usr                                444M  88.2G   144K  /mnt/usr
tank0/usr/jails                          144K  88.2G   144K  /mnt/usr/jails
tank0/usr/myjails                        443M  88.2G   152K  /mnt/usr/myjails
tank0/usr/myjails/.warden-chroot-amd64   443M  88.2G  31.7M  /usr/myjails/.warden-chroot-amd64
tank0/usr/myjails/10.20.50.200             8K  88.2G   442M  /mnt/usr/myjails/10.20.50.200
tank0/usr/obj                            144K  88.2G   144K  /usr/obj
tank0/usr/pbi                            256K  88.2G   256K  /usr/pbi
tank0/usr/ports                          296K  88.2G   152K  /usr/ports
tank0/usr/ports/distfiles                144K  88.2G   144K  /usr/ports/distfiles
tank0/usr/src                            144K  88.2G   144K  /usr/src
tank0/var                                692K  88.2G   144K  /mnt/var
tank0/var/audit                          144K  88.2G   144K  /var/audit
tank0/var/log                            252K  88.2G   252K  /var/log
tank0/var/tmp                            152K  88.2G   152K  /var/tmp
tank1                                    448K  8.01T  43.4K  /tank1
tank1/home                              82.3K  8.01T  82.3K  /usr/home
tank1/myjails                           43.4K  8.01T  43.4K  /usr/myjails
tank1/snapshots                         43.4K  8.01T  43.4K  /usr/snapshots
The init script still tries to install a lot of things on /mnt/usr/myjails
Which It mounts on the main pool (the pool where the original system is installed). It shouldn't install this in here… !

It is not what I want : my system is installed on high end SSD with only 100Go of data available - and I have a pool with 8Tb - this is where It belongs.


Can someone please let me know if there is a clean way of installing this with my configuration :

•*Main system on a pool
•*Jails system on another pool


Thanks.
Reply With Quote
  #3  
Old 12-22-2012, 10:44 PM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Default
I can confirm this - I have exactly the same problem when trying to use another pool then the root pool for warden/jails. All config is hard-coded on tank0 it seems.
Reply With Quote
  #4  
Old 12-23-2012, 11:28 PM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Wink Tried a hack
Hi,

I messed around with this a bit, just wanted to post what I found so far. Hopefully someone can benefit from it.
Unfortunatly, I just have a laptop with one disk and one pool here (on xmas vacation, my lab is faaar away), but I think this hack will work with a 2nd pool. This is far away from best practice, proceed with caution, and back up your data and the conf files before testing this! I started with deleting the original zfs dataset for jails so that warden wouldn't try to use it:

Code:
zfs destroy -r tank0/usr/jails

Then I created a new dataset with a new mountpoint:

Code:
zfs create -o mountpoint=/prison tank0/prison

Then I edited /usr/local/share/warden/scripts/backend/functions.sh since it seems the line with jail location is misinterpreted. I know you aren't supposed to touch this file, but without this hardcoding of the location, nothing worked:

Comment out the line:
Code:
# JDIR="$(grep ^JDIR: /usr/local/etc/warden.conf | cut -d' ' -f2)"
Add below it:
Code:
JDIR="/prison"
Do the same for the Tmp directory WTMP line:
Code:
# WTMP="$(grep ^WTMP: /usr/local/etc/warden.conf | cut -d' ' -f2)"
And add just below it:
Code:
WTMP="/prison"
Now this location is hardcoded, and the settings in warden.conf is ignored (obviously).

Create a new jail using the terminal:
Code:
warden create 192.168.1.15 test1

This now works fine, for me at least. Jail is created and it seems to work. But, if you start the warden gui from terminal, you will get a live log of everything warden is doing. When you start the jail and then try to update it via the update button, you will see this:
Code:
Locale: "en" 
"Invalid chroot dir: /usr/jails/192.168.1.15" 
cp: /usr/jails/192.168.1.15/tmp/.fbupdatechk: No such file or directory
chmod: /usr/jails/192.168.1.15/tmp/.fbupdatechk: No such file or directory
rm: /usr/jails/192.168.1.15/tmp/.fbupdatechk: No such file or directory
()
I haven't found any reference to /usr/jails left in any conf, so I guess it must be hardcoded in the gui bin.

The automatic updater however works, and the terminal one (warden checkup 192.168.1.15) works.

The gui user administrator also fails, it can't find /etc/shells and /etc/groups, most likely same error. Meta packages isn't visible etc.

I can actually live with this hack, although it would be great to get the meta packages...

Warden/jails with beadm and zfs has so much potential, looking forward to see how everything evolves!

Merry xmas!
Reply With Quote
  #5  
Old 12-24-2012, 10:13 AM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Default Correction
My bad - there seems to be no need to comment out and hard code the links in functions.sh. I tried to use the default line for JDIR and WTMP in functions.sh and it works. I don't know why it didn't work for me yesterday.

I noticed now that warden gui metapackages tab points at "Metapkgs for /usr/jails/192.168.1.15" wich is another indicator for warden having hard coded dirs somewhere.

Going back to old location for now, will try to hook up zfs datasets located in 2nd pool to the jails so that the bulk of data can be stored there. Good thing is that it's easy to export the jails and move them when this issue is solved.
Reply With Quote
  #6  
Old 12-24-2012, 10:58 AM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Default
Hm, where did my other post go? Anyway, I had some luck with deleting the zfs dataset /usr/jails so that warden can't look there at all, then create a new dataset and setting the warden.conf to point to that instead. The Warden gui still doesn't work properly since it has the default location hard coded in some way. Warden in terminal works almost flawless.
Reply With Quote
  #7  
Old 12-24-2012, 10:57 PM
gregober gregober is offline
Junior Member
 
Join Date: Oct 2012
Location: Paris
Posts: 23
Thanks: 0
Thanked 0 Times in 0 Posts
Default Impossible to start jail
I'll try to have one person from my team working on this (= updating the script).

If Kris Moore could give us some clue on how his script has been structured, this might save a a little time…*


I'll keep you updated once we've found smthg working.
Reply With Quote
  #8  
Old 12-25-2012, 02:04 AM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Default
Ah, nice!

It's possible that Warden and Warden gui needs to be recompiled with the hard settings for /usr/jails (and possibly poolname tank0) removed, since some commands doesn't seem to check the JDIR variable in functions.sh.

But this can also be an exclusive ZFS issue in the scripts since the poolname is added before the given path to the JDIR, when creating a new jail for example. This is obviously done because ZFS needs the poolname when creating a new dataset in a pool.

One fairly easy solution would be to change the setup so that ZFS users have to enter/change the poolname in warden.conf, then add a function in functions.sh (similar to the JDIR one) that captures and turns the poolname into a variable that is used in the scriptblocks that handles ZFS. I will try to fiddle with it, but need to put down a few hours on this xmas thingie first... ;-)
Reply With Quote
  #9  
Old 12-28-2012, 03:59 PM
kmoore134's Avatar
kmoore134 kmoore134 is offline
Administrator
 
Join Date: May 2005
Location: Knoxville, TN
Posts: 2,568
Thanks: 0
Thanked 163 Times in 127 Posts
Default
Its probably a bug in the function that gets the ZFS pool name.. Can you post the warden.conf file you are trying to use, along with the output of "mount"
__________________
----
Kris Moore
PC-BSD Founder
Reply With Quote
  #10  
Old 12-29-2012, 01:03 AM
Klumpo Klumpo is offline
Member
 
Join Date: Dec 2012
Posts: 42
Thanks: 3
Thanked 5 Times in 5 Posts
Default My settings
Hello,
first, the settings in warden.conf:
Code:
# Configuration options for the Warden
######################################################################

# Network Interface for the jails to use
NIC: em0.51

# Directory to use for compressing / decompressing files 
WTMP: /usr/prisons

# Location of the jails
JDIR: /usr/prisons
And the output from mount, the 2nd pool is named "lagring", and the dataset is mounted at usr/prisons:

Code:
tank0/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
procfs on /proc (procfs, local)
linprocfs on /compat/linux/proc (linprocfs, local)
tank0/root on /root (zfs, local, noatime, nfsv4acls)
tank0/tmp on /tmp (zfs, local, noatime, nfsv4acls)
tank0/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
tank0/usr/home/rickard on /usr/home/rickard (zfs, local, noatime, nfsv4acls)
tank0/usr/jails on /usr/jails (zfs, local, noatime, nfsv4acls)
tank0/usr/obj on /usr/obj (zfs, local, noatime, nfsv4acls)
tank0/usr/pbi on /usr/pbi (zfs, local, noatime, nfsv4acls)
tank0/usr/ports on /usr/ports (zfs, local, noatime, nfsv4acls)
tank0/usr/ports/distfiles on /usr/ports/distfiles (zfs, local, noatime, nfsv4acls)
tank0/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
tank0/var/audit on /var/audit (zfs, local, noatime, nfsv4acls)
tank0/var/log on /var/log (zfs, local, noatime, nfsv4acls)
tank0/var/tmp on /var/tmp (zfs, local, noatime, nfsv4acls)
lagring on /lagring (zfs, local, nfsv4acls)
lagring/prisons on /usr/prisons (zfs, local, nfsv4acls)
tank0/usr/prisons on /mnt/usr/prisons (zfs, local, noatime, nfsv4acls)
tank0/usr/prisons/.warden-chroot-amd64 on /usr/prisons/.warden-chroot-amd64 (zfs, local, noatime, nfsv4acls)
tank0/usr/prisons/192.168.51.3 on /mnt/usr/prisons/192.168.51.3 (zfs, local, noatime, nfsv4acls)
The output from the warden gui when trying to create a jail:

Code:
Getting regional mirror...
Using mirror: http://mirror.7he.at/pub/PCBSD
Fetching jail environment. This may take a while...
Downloading http://mirror.7he.at/pub/PCBSD/9.1/amd64/netinstall/fbsd-release.txz ...
fbsd-release.txz                                        91 MB 2632 kBps

fbsd-release.txz.md5                                    33  B   46 kBps

Creating ZFS /usr/prisons/.warden-chroot-amd64 dataset...
Building new Jail... Please wait...
touch: /usr/prisons/192.168.51.3/etc/fstab: No such file or directory
/usr/local/share/warden/scripts/backend/createjail.sh: cannot create /usr/prisons/192.168.51.3/etc/rc.conf: No such file or directory
/usr/local/share/warden/scripts/backend/createjail.sh: cannot create /usr/prisons/192.168.51.3/etc/hosts: No such file or directory
cp: /usr/prisons/192.168.51.3/etc/resolv.conf: No such file or directory
Success!
Jail created at /usr/prisons/192.168.51.3
Changing root password on: 192.168.51.3 chroot: /.chpass.sh: No such file or directory
FAILED!
Reply With Quote
Reply

Tags
pcbsd, problem creating jail, warden, zfs

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 09:48 AM.


Powered by vBulletin® Version 3.8.5
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.

Copyright 2005-2010, The PC-BSD Project. PC-BSD and the PC-BSD logo are registered trademarks of iXsystems.
All other content is freely available for sharing under the terms of the Creative Commons Attribution License.