Experiences in deploying MyPLC PlanetLab Central v4.2
This page is dedicated to documenting the procedure we used to deploy a MyPLC PlanetLab Central v4.2 distribution on the DRAGON network.
Install CentOS 5.2
On a fresh PC, install CentOS 5.2 (Linux RedHat variant).
Begin by downloading the netinstall ISO iamge at
ftp://linux.mirrors.es.net/centos/5.2/isos/i386 (e.g.
CentOS-5.2-i386-netinstall.iso)
The CentOS netinstall procedure requires the user to supply some information about the mirror. See
http://www.chrisgountanis.com/technical/45-centos-netinstall.html for detailed instructions. We selected the 'HTTP' method for installation, using
mirror.centos.org for the 'Website Name', and
centos/5.2/os/i386 for the 'CentOS Directory'.
- NOTE: Don't select 'Xen' during the installation. If you do, you will get exceptions in
dmesg later on because you will be booting a vserver-enabled kernel, which does not include Xen support. If you accidentally select 'Xen', you will want to turn off libvirtd using chkconfig.
- In our installation, we named this host
planetlab1.domain.org
DNS configuration
- We configured our DNS server as follows:
-
planetlab1.domain.org is an A record pointing to the IP address 192.168.100.133.
-
myplc.domain.org is an A record pointing to the IP address 192.168.100.131.
Install vserver-enabled kernel
- After installing CentOS 5.2, create a new file
/etc/yum.repos.d/dhozac-vserver.repo and populate it with the following:
[dhozac-vserver]
name=Linux-VServer related packages for CentOS $releasever - $basearch
baseurl=http://rpm.hozac.com/dhozac/centos/$releasever/vserver/$basearch
gpgkey=http://rpm.hozac.com/conf/keys/RPM-DHOZAC-GPG-KEY
- Next, run
yum -y update yum and yum -y install kernel
- Edit
/boot/grub/menu.lst (is actually a symlink to grub.conf in CentOS)
- The last
yum command should have added something like this (it should be the first kernel entry):
title CentOS (2.6.22.19-vs2.3.0.34.1)
root (hd0,0)
kernel /vmlinuz-2.6.22.19-vs2.3.0.34.1 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.22.19-vs2.3.0.34.1.img
-
- Make sure you have
default=0 in order to boot this kernel by default
Tweak SSH daemon
- Edit
/etc/ssh/sshd_config, change ListenAddress to the IP address configured eth0 (would block any guest's attempt to run sshd if you do not do this because sshd binds to '*:22' by default)
Turn SElinux off
- edit
/etc/sysconfig/selinux, set SELINUX=disabled
Reboot
- Reboot the PC, when it comes back up, ensure that you can SSH to it and that the new vserver-enabled kernel is running:
[root@planetlab1 ~]# uname -a
Linux planetlab1 2.6.22.19-vs2.3.0.34.1 #1 SMP Mon Mar 17 05:32:04 EDT 2008 i686 i686 i386 GNU/Linux
Install util-vserver packages
[root@planetlab1 ~]# yum install util-vserver-core
[root@planetlab1 ~]# yum install util-vserver-lib
[root@planetlab1 ~]# yum install util-vserver-sysv
[root@planetlab1 ~]# yum install util-vserver-build
- NOTE: We could have probably just done
yum -y install util-vserver instead of the four commands above.
- Additional steps required:
[root@planetlab1 ~]# /etc/init.d/vprocunhide start
Fixing /proc entries visibility... [ OK ]
[root@planetlab1 ~]# chkconfig vservers-default on
[root@planetlab1 ~]# chkconfig --list vservers-default
vservers-default 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Build Guest Systems
- yum install all of these packages in "root context" (e.g. not inside the vserver):
- subversion
- httpd (on machines that mirror fedora repositories)
- createrepo is needed in the root context on build boxes - this is for when just built rpms get published to the test framework
- qemu is required on boxed that may host qemu-based test boxes
- bridge-utils is mandatory on your qemu boxes as well
- check out planetlab SVN build
- this is really just a bunch of scripts to setup an environment so that
myplc-native will run inside a vserver
[root@planetlab1 ~]# cd /root
[root@planetlab1 ~]# svn co http://svn.planet-lab.org/svn/build/trunk svn-build
Fedora and CentOS mirroring
- We mirror the entire Fedora and CentOS distribution so that vserver instantiation is particularly fast.
- Run one of the
vbuild-fedora-mirror.sh commands below, we found the first one to give the best performance from MAX:
[root@planetlab1 ~]# cd /root/svn-build
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -u rsync://fedora.mirror.iweb.ca/fedora
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -u rsync://rsync.gtlib.gatech.edu/fedora-enchilada/linux
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -s
- NOTE: This process takes a long time! The Fedora 8 mirror consumes approximately 19GB on our
planetlab1.domain.org server.
Start apache in root context
- Apache will run in the root context of
planetlab1.domain.org and serve up Fedora/CentOS RPMs so that vservers can quickly be instantiated.
[root@planetlab1 ~]# /etc/init.d/httpd start
- Make sure Apache is configured to automatically start whenever this machine is booted:
[root@planetlab1 ~]# chkconfig --level 2345 httpd on
[root@planetlab1 ~]# chkconfig --list httpd
httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
Apache configuration
[root@planetlab1 ~]# cd /var/www/html
[root@planetlab1 html]# ln -s /mirror mirror
- Modify Apache's
/etc/httpd/conf/httpd.conf to only listen on 127.0.0.1:80 and root context's public IP, for example:
Listen 127.0.0.1:80
Listen 192.168.100.133:80
ServerName planetlab1.domain.org
- We also set ServerName above, just to make sure it gets set to the proper value..
Update iptables
Since a complete mirror of Fedora and CentOS will be available from this Apache server (without any authentication), we would recommend adding a couple of
iptables rules to the
RH-Firewall-1-INPUT chain in
/etc/sysconfig/iptables. For example, we limit incoming requests on TCP port 80 and 443 to our local network (206.196.176.0/21):
-A RH-Firewall-1-INPUT -s 206.196.176.0/21 -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -s 206.196.176.0/21 -p tcp --dport 443 -j ACCEPT
- Run
/etc/init.d/iptables restart to apply the changes
Tweaking yum config for vservers
- The following commands will update
/etc/vservers/.distributions/[f8|centos5]/yum.repos.d/building.repo so that baseurl points to http://planetlab1.domain.org/mirror/ instead of http://localhost/mirror/
[root@planetlab1 ~]# cd /root/svn-build/mirroring/
[root@planetlab1 mirroring]# ./mirror.sh -f f8 -u http://planetlab1.domain.org/mirror/ init
[root@planetlab1 mirroring]# ./mirror.sh -f centos5 -u http://planetlab1.domain.org/mirror/ init
- NOTE: You don't want http://localhost/mirror/ because vserver creation will fail after installing the base operating system and starting the vserver and running yum!
- After running the commands above, make sure the output of
mirror.sh -a looks like the following (check that baseurl is correct):
[root@planetlab1 mirroring]# ./mirror.sh -a
==================== Contents of /etc/vservers/.distributions/f8/yum.repos.d/
==> /etc/vservers/.distributions/f8/yum.repos.d//building.repo <==
[fedora]
name=Fedora 8 - $basearch
baseurl=http://planetlab1.domain.org/mirror/fedora/releases/8/Everything/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=http://redhat.download.fedoraproject.org/pub/fedora/linux/releases/8/Everything/$basearch/os/RPM-GPG-KEY-fedora http://redhat.download.fedoraproject.org/pub/fedora/linux/releases/8/Everything/$basearch/os/RPM-GPG-KEY
[updates]
name=Fedora 8 - $basearch - Updates
baseurl=http://planetlab1.domain.org/mirror/fedora/updates/8/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
[updates-newkey]
name=Fedora 8 - $basearch - Updates Newkey
baseurl=http://planetlab1.domain.org/mirror/fedora/updates/8/$basearch.newkey/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-and-9
==================== /etc/vservers/.distributions/f9/yum.repos.d/ does not exist
==================== Contents of /etc/vservers/.distributions/centos5/yum.repos.d/
==> /etc/vservers/.distributions/centos5/yum.repos.d//building.repo <==
[base]
name=CentOS 5.2 - $basearch - Base
baseurl=http://planetlab1.domain.org/mirror/centos/5.2/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
[updates]
name=CentOS 5.2 - $basearch - Released Updates
baseurl=http://planetlab1.domain.org/mirror/centos/5.2/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
[epel]
name=Extra Packages for Enterprise Linux 5 - $basearch
baseurl=http://planetlab1.domain.org/mirror/epel/5/$basearch/
gpgcheck=1
gpgkey=http://download.fedora.redhat.com/pub/epel/RPM-GPG-KEY-EPEL
Create a vserver to run myplc-native
myplc-native is a software package that includes everything you need to manage and slice PlanetLab nodes. It will run inside a vserver on
planetlab1.domain.org. We will call that vserver
myplc.domain.org.
- Please note that the root context (
planetlab1.domain.org) has an IP address of 192.168.100.133. We will use a different IP address and hostname for the vserver running myplc-native, in this case, it will be myplc.domain.org with an IP address of 192.168.100.131.
- Build a new vserver using the
vtest-init-vserver.sh script:
[root@planetlab1 ~]# cd /root/svn-build
[root@planetlab1 svn-build]# ./vtest-init-vserver.sh -p linux32 myplc http://build.planet-lab.org/planetlab/f8/planetlab-f8-linux32-4.2-rc24/RPMS/ -- \
--interface eth0:192.168.100.131 --hostname myplc.domain.org &> myplc.log &
NOTE: Visit
http://build.planet-lab.org/planetlab/f8/?C=M;O=D to find the latest release and change the URL which you pass to the
vtest-init-vserver.sh script. If you are interested in running the
OneLab build of the PlanetLab software, begin by visiting
http://build.onelab.eu/ to find a suitable URL to pass to
vtest-init-vserver.sh.
- Once that finishes, enter the vserver and install the
myplc-native package:
[root@planetlab1 svn-build]# vserver myplc enter
[myplc] / # yum -y install myplc-native
- Try to start/stop the
plc service:
[myplc] / # service plc start
[myplc] / # service plc stop
- Use
chkconfig to make sure it comes up when this vserver is started:
[myplc] / # chkconfig plc on
[myplc] / # chkconfig --list plc
plc 0:off 1:off 2:on 3:on 4:on 5:on 6:off
- Configure myplc-native from the console using
plc-config-tty
[myplc] ~ # plc-config-tty
Enter command (u for usual changes, w to save, ? for help) u
== PLC_NAME : [My Domain PlanetLab Central]
== PLC_SLICE_PREFIX : [myplc]
== PLC_ROOT_USER : [pl-admin@domain.org]
== PLC_ROOT_PASSWORD : [root] password
== PLC_MAIL_ENABLED : [true]
== PLC_MAIL_SUPPORT_ADDRESS : [pl-admin@domain.org]
== PLC_DB_HOST : [localhost.localdomain] myplc.domain.org
== PLC_API_HOST : [localhost.localdomain] myplc.domain.org
== PLC_WWW_HOST : [localhost.localdomain] myplc.domain.org
== PLC_BOOT_HOST : [localhost.localdomain] myplc.domain.org
== PLC_NET_DNS1 : [192.168.1.1]
== PLC_NET_DNS2 : [192.168.2.1]
Enter command (u for usual changes, w to save, ? for help) w
Wrote /etc/planetlab/configs/site.xml
Merged
/etc/planetlab/default_config.xml
and /etc/planetlab/configs/site.xml
into /etc/planetlab/plc_config.xml
+ service plc reload
You might want to type 'r' (restart plc) or 'q' (quit)
Enter command (u for usual changes, w to save, ? for help) r
==================== Stopping plc
...
==================== Starting plc
...
Fix procfs warnings in dmesg
- If you run dmesg on
planetlab1.domain.org (the root context), you will see messages like this:
> vxW: [<BB>crond<AB>,12389:#40056|40056|40056] messing with the procfs.
- The current workaround requires running this command inside the guest vserver:
-
sed -i -e 's/.*pam_loginuid.*/#\0/' /etc/pam.d/*
- NOTE: This is fixed in newer vserver kernels...(According to D.Hozac)...but these kernels are not ready for prime-time yet [as of December 2008].
Allow new users to be created
- The
is_public variable in the sites table of the planetlab4 postgresql database must be set to 't' (true) to allow new users to be created.
- For some reason, there is no other way to set this besides mucking around manually with SQL.. The issue has been raised on the planetlab-devel mailing list, so the developers are aware of this issue.
[root@planetlab1 ~]# vserver myplc enter
[myplc] ~ # su -l postgres
-bash-3.2$ psql planetlab4
planetlab4=# update sites set is_public='t';
UPDATE 1
planetlab4=#
Skip hardware requirement checks
PlanetLab enforces sites to run relatively modern hardware, as described at
http://www.planet-lab.org/hardware. There is actually a check upon bootup to make sure that the system meets the minimum hardware requirements. Since a lot of our PCs are a little older (or have small hard drives), we want to skip this requirement in our environment.
- Modify bootmanager configuration to skip minimum hardware requirements:
- Inside the myplc vserver, edit the file
/usr/share/bootmanager/source/configuration
- set
SKIP_HARDWARE_REQUIREMENT_CHECK=1
- Now, run
service plc stop and service plc start within the myplc vserver
- This should re-create the boot CD and ensure that any ISOs downloaded from the MyPLC web interface will not do the minimum hardware requirements check!
Login to MyPLC web-based user interface
The MyPLC web interface is a place where you can add new PlanetLab nodes, create and manage slices, etc. All of this can be done through an API as well, but the web interface is a good starting point for seeing how everything works.
- Visit the URL: https://myplc.domain.org/
- Username:
admin@domain.org (see above - set by running plc-config-tty)
- Password:
password (see above - set by running plc-config-tty)
Create user accounts
To create a new user account, visit
https://myplc.domain.org and click "Create an account".
If your site does not appear under the 'Site' drop-down menu, see the section above entitled "Allow new users to be created". You probably need to tweak a variable in your planetlab4 SQL database.
Once your account is created, you should login to your account and upload an SSH key. To do this, login to
https://myplc.domain.org, click 'My Acccount' under 'Users' (and
not 'my account' under 'drupal'). Click on 'Manage Keys' and use the form to upload your public SSH key. If don't already have a public key, generate one by running a command like
ssh-keygen -t dsa. Then upload the file
~/.ssh/id_dsa.pub.
You will need to use this SSH key to login to "slices" that are created for you in later steps.
Add some PlanetLab nodes
Find some unused PCs to be used as PlanetLab nodes. These nodes can boot off of a CD-ROM or USB flash device. As of this writing, we have only experimented with the Boot CD option. PXE netboot is also being worked on, and would be much more convenient if you have nodes scattered throughout many POPs. Once these nodes are active, they may be "sliced" meaning that a Linux vserver will be automatically added/removed and handed off to an end user. They will be able to login via their SSH private key.
Login to the MyPLC web interface, click "Add Nodes" and fill out the form. We currently have two nodes --
planetlab2.domain.org and
planetlab3.domain.org.
Once the node is added, you must download the boot materials for that node. Click on "My Nodes" on the left bar, then click on the hostname of the node you just added. On this page, you will be able to download an ISO image (for burning to CD), a USB image, the planetlab config file, or a generic ISO/USB image.
Burn the ISO image to a CD-ROM. Boot the node off of that CD.
You can control the 'boot state' of each node from the MyPLC web interface. Click on "My Nodes" on the left bar, click a hostname, and then examine the first option 'Boot State'. The link to the right of it ("What's this?") explains what each boot state option means.
If all goes well, after you boot up the CD the machine should register itself with MyPLC. It will ask you for confirmation before wiping the entire machine. The boot state in MyPLC should eventually come up to 'boot' if all goes well. The node should be "sliceable" at this point.
Create Slices
The whole point of setting up this environment is to be able to create "slices" for end users. This is essentially a vserver, which gets some percentage of CPU time and some chunk of disk space on a PlanetLab node. To create slices, do the following:
- Login to MyPLC web interface
- Click on "Create Slice"
- Fill out the form with meaningful values for name/URL/description (we used
testing for the name field in our examples below)
- For the "Instantiation" option, choose "plc-instantiated"
- Click "Add"
Now you need to associate nodes and users with this slice. This will control which users' SSH keys will be copied to the vserver, and which nodes they will be able to access.
- Under "Slice Details" there is a "Choose Action" option
- On this drop-down menu, select "Manage Nodes"
- Put checkboxes in the nodes that you wish to be associated with this "slice", e.g. check
planetlab2 and planetlab3, then click "Add Nodes"
- Click "Back to Slice"
- Now select "Manage Users" under the "Choose Action" drop-down list
- Select which users you want to be able to login to the nodes (their SSH keys will be copied to that node), then click "Add People"
- Click "Back to Slice"
It appears as though each planetlab node runs a "NodeManager" process in its root context. This will occasionally "check in" with MyPLC to see if it needs to do anything -- I believe this happens every 10 minutes. Within 10 minutes, you should see new vservers appear on the nodes which you selected (look under /etc/vservers in the root context), and SSH keys should be copied over to that node so the users can access their slices.
Logging into your "slice"
In order to login to your planetlab slice, you must add an SSH key to your user profile in MyPLC. Assuming you have done this, after you setup a slice and associate your user with that slice, the SSH keys will be copied to the node.
How does this actually work?
Assume you have just created a new slice called
myplc_testing and have associated node
planetlab2.domain.org and user
admin@domain.org with that slice.
What actually happens when you use "create slice" above is this:
- new user named
myplc_testing is created on planetlab2.domain.org (NOTE: in the root context! not the vserver!)
- SSH keys for
admin@domain.org are copied to planetlab2.domain.org:/home/myplc_testing/.ssh/authorized_keys (NOTE: in the root context! not the vserver!)
- new vserver named
myplc_testing is created (see /etc/vservers/myplc_testing/...)
The new vserver is not actually running an SSH daemon, nor does it have a unique IP address assigned to it. PlanetLab nodes run a special process called
pl_sshd on which somehow logs users into their vservers automatically. Take a look at the
sshd processes running on a PlanetLab node:
[root@planetlab2 ssh]# ps auxwww | grep sshd | grep sbin
root 1263 0.0 0.2 6880 1060 ? Ss Nov06 0:52 /usr/sbin/sshd
root 1289 0.0 0.2 6880 1056 ? Ss Nov06 0:00 /usr/local/sbin/pl_sshd -o PidFile /var/run/pl_sshd.pid -p 806
[root@planetlab2 ssh]# lsof -i -U -P | grep ^sshd | grep LISTEN
sshd 1263 root 3u IPv6 4467 TCP *:22 (LISTEN)
sshd 1289 root 3u IPv6 4512 TCP *:806 (LISTEN)
Note that 'sshd' listens on port 22, but 'pl_sshd' listens on port 806.
Whenever I login as user
myplc_testing using the SSH key for
admin@domain.org, I am transferred into the myplc_testing vserver:
somehost% ssh myplc_testing@planetlab2.domain.org
[myplc_testing@planetlab2 ~]$ sudo -s
[root@planetlab2 ~]# ps auxwww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
508 26605 0.5 0.2 2632 1416 ? Ss 19:32 0:00 /bin/bash -l
root 26617 2.0 0.2 2632 1424 ? S 19:32 0:00 /bin/bash
root 26621 0.0 0.1 2456 824 ? R+ 19:32 0:00 ps auxwww
Note that I cannot "see" my SSH
.authorized_keys file -- because that actually lives in the root context, not the vserver!
[myplc_testing@planetlab2 ~]$ ls -l ~/.ssh
ls: cannot access /home/myplc_testing/.ssh: No such file or directory
How to SSH to PlanetLab node's "root context"
Normally, users would upload their public key to the MyPLC web interface, and when new slices are created, vservers are setup on the PlanetLab nodes and the SSH keys for that user are automatically installed.
In order to SSH to the "root context" of a PlanetLab node, that has booted off of the Boot CD, you must use the SSH private key. Login via password is disabled. Use the private key inside the MyPLC vserver in
/etc/planetlab/root_ssh_key.rsa.
NOTE: If the node has come up in debug mode (either intentionally or otherwise) you must use the
/etc/planetlab/debug_ssh_key.rsa private key. For example:
[root@planetlab1 ~]# ssh root@planetlab1.domain.org
[root@planetlab1 ~]# vserver myplc enter
[myplc] / # ssh -i /etc/planetlab/root_ssh_key.rsa root@planetlab2.domain.org
[root@planetlab2 ~]# uname -a
Linux planetlab2.domain.org 2.6.22.19-vs2.3.0.34.29.planetlab #1 SMP Fri Oct 17 12:33:05 EDT 2008 i686 i686 i386 GNU/Linux
Adding tagged 802.1Q VLAN interface to vserver
We have found two ways to add a tagged 802.1Q VLAN interface to a vserver. Normally this would be done using something like
vconfig add eth1 3000 and configured with an IP address by running
ifconfig eth1.3000 inet A.B.C.D. However, this method will not work for vservers.
Option 1: during vserver instantiation
With the latest releases (sometime around 12/15/2008), you can pass
--interface ethX.YYYY directly to the
vserver build command. For example, this could be passed into the
vtest-init-vserver.sh script:
./vtest-init-vserver.sh [...] -- --interface eth0:10.1.1.1/24 --interface eth1.3000:10.2.2.1/24 --hostname [...]
Option 2: modifying an existing vserver
- Do something like this within the root context:
[root@planetlab1 ~]# vconfig add eth1 1234
Added VLAN with VID == 1234 to IF -:eth1:-
[root@planetlab1 ~]# ifconfig eth1.1234 inet 10.1.1.1 netmask 255.255.255.0
- NOTE: By default, eth1.1234 will now show up in all PlanetLab-instantiated vservers!
- Now the vserver has a tagged VLAN interface:
[root@planetlab1 ~]# vserver test-vlan enter
bash-3.2# ifconfig eth1.1234
eth1.1234 Link encap:Ethernet HWaddr 00:30:48:9A:D3:89
inet addr:10.30.30.1 Bcast:10.30.30.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:8286 (8.0 KiB)
Using the API
- To get started with the PlanetLab API, use the PlanetLab shell (plcsh) as a starting point
- On planetlab1, check out
plcsh using subversion:
[root@planetlab1 ~]# svn co http://svn.planet-lab.org/svn/plcmdline/trunk plcsh
A plcsh/plcapilib.py
A plcsh/setup.py
A plcsh/plcapilib.spec
A plcsh/plcsh
A plcsh/Makefile
A plcsh/plcmdline.py
Checked out revision 12695.
- Run
plcsh as follows, using -h to point it to your PLCAPI and -u to specify the user that you created above:
[root@planetlab1 ~]# cd plcsh
[root@planetlab1 plcsh]# ./plcsh -h https://myplc.domain.org/PLCAPI/ -r user -u admin@domain.org
Password: [enter password here]
PlanetLab Central API server at https://myplc.domain.org/PLCAPI/
admin@domain.org connected as user using password authentication
Type "system.listMethods()" or "help(method)" for more information.
[chris@maxgigapop.net user]>>>
Tweaking Node Manager's Polling Interval
- Each "sliceable" planetlab node has a process called 'nm.py'. This is a python application that acts as a Node Manager, which will start/stop vservers as needed by checking in with the central MyPLC node.
- By default, Node Manager has a polling interval of 600 seconds, plus a 301 second range for "additional random polling interval":
[root@planetlab2 ~]# grep interval /usr/share/NodeManager/nm.py
parser.add_option('-p', '--period', action='store', dest='period', default=600, help='Polling interval (sec)')
parser.add_option('-r', '--random', action='store', dest='random', default=301, help='Range for additional random polling interval (sec)')
- If you have hundreds/thousands of nodes, these are probably reasonable values. However, if you have a small number of nodes and would like to see vservers created/deleted very quickly, you can edit
/etc/init.d/nm and add these options to the command line.
- Example:
- edit
/etc/init.d/nm and tweak this line: options=${OPTIONS-"-d -s -p 30 -r 1"}
- stop nm service:
/etc/init.d/nm stop
- start nm service:
/etc/init.d/nm start
- check output of
ps auxwww|grep nm to make sure you see the new options -p 30 -r 1 listed on the command line
Miscellaneous Notes
Deploying geniwrapper
- start with svn checkout:
- read geniwrapper/INSTALL file
- in myplc vserver:
yum install pyOpenSSL (necessary to run python code)
- had to tweak /etc/yum.repos.d/myplc.repo before yum would start working
- baseurl=URL was invalid (site must have changed?)
- changed to: baseurl=http://build.planet-lab.org/planetlab/f8/planetlab-f8-linux32-4.2-rc24/RPMS/
- in myplc vserver:
yum install m2crypto (also necessry to run python code)
- in myplc vserver:
yum install make (geniwrapper now requires make package)
[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py
Base Directory: /usr/local/share/geniwrapper/
Import: creating top level authorities
Hierarchy: creating authority: plc
Segmentation fault
- This segfault appears to be due to a bug in pyOpenSSL v0.6. Upgrade to a newer version of pyOpenSSL to fix this problem.
- Getting
gcc installed in the myplc vserver was not as easy as it should be. yum install gcc kept complaining with "Error: Missing Dependency: kernel-headers is needed by package glibc-headers-2.7-2.i386"
- However, it was not possible to simply run
yum install kernel-headers for some reason. I did the following to install the kernel-headers package and finally install gcc so that I could build pyOpenSSL v0.8:
% vserver myplc enter
[myplc] ~ # wget http://planetlab1.domain.org/mirror/fedora/releases/8/Everything/i386/os/Packages/kernel-headers-2.6.23.1-42.fc8.i386.rpm
[myplc] ~ # rpm -i kernel-headers-2.6.23.1-42.fc8.i386.rpm
[myplc] ~ # yum install gcc
- Finally I had a working gcc and could proceed with building pyOpenSSL v0.8. First, I had to install the openssl header files by using
yum install openssl-devel. Now I could build pyOpenSSL using python setup.py build. I removed pyOpenSSL v0.6 with yum erase pyOpenSSL and the installed v0.8 with python setup.py install.
- Now
python gimport.py succeeds without any segfaults, but it fails to import a user:
[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py
Base Directory: /usr/local/share/geniwrapper
Import: creating top level authorities
maxplc :
testing :
Import: adding maxplc to trusted list
Import_Site: importing site testing.maxpl
Import: importing person testing.maxpl.chris
XXX: DSA key encountered, ignoring
Failed to import: {'bio': '', 'first_name': 'Default', 'last_name': 'Administrator', 'last_updated': 1239288497, 'roles': ['admin', 'pi', 'user', 'tech'], 'title': 'Engineer', 'url': '', 'key_ids': [1], 'enabled': True, 'slice_ids': [], 'phone': '240-687-1498', 'peer_person_id': None, 'role_ids': [10, 20, 30, 40], 'person_id': 2, 'date_created': 1225830970, 'site_ids': [1], 'peer_id': None, 'email': 'chris@maxgigapop.net'}
...
- Notice above that it says "XXX: DSA key encountered, ignoring", this script does not seem to like DSA keys. Create an RSA key, upload this via the MyPLC? web interface, and then re-run
gimport.py:
[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py
Base Directory: /usr/local/share/geniwrapper
Import: creating top level authorities
Hierarchy: creating authority: maxpl
maxpl :
Import: creating table for authority maxpl
NOTICE: table "geni$maxpl" does not exist, skipping
inserting authority record for maxpl
maxpl :
Import: adding maxpl to trusted list
Import_Site: importing site maxpl.maxpl
Hierarchy: creating authority: maxpl.maxpl
inserting authority record for maxpl.maxpl
Import: importing person maxpl.maxpl.chris
Import: creating table for authority maxpl.maxpl
NOTICE: table "geni$maxpl$maxpl" does not exist, skipping
/usr/share/keyconvert/keyconvert /tmp/tmp9PeVdQ /tmp/tmp1Q8B7f
keytype = ssh-rsa
completed
inserting user record for maxpl.maxpl.chris
/usr/share/keyconvert/keyconvert /tmp/tmpXt8DJE /tmp/tmpAgJ2Xp
keytype = ssh-rsa
completed
updating user record for maxpl.maxpl.chris
Import: importing person maxpl.maxpl.poneil
person maxpl.maxpl.poneil does not have a PL public key
inserting user record for maxpl.maxpl.poneil
Import: importing person maxpl.maxpl.tlehman
person maxpl.maxpl.tlehman does not have a PL public key
inserting user record for maxpl.maxpl.tlehman
Import: importing person maxpl.maxpl.jflidr
person maxpl.maxpl.jflidr does not have a PL public key
inserting user record for maxpl.maxpl.jflidr
Import: importing slice maxpl.maxpl.netflow
inserting slice record for maxpl.maxpl.netflow
Import: importing slice maxpl.maxpl.sirius
inserting slice record for maxpl.maxpl.sirius
Import: importing node maxpl.maxpl.planetlab2
inserting node record for maxpl.maxpl.planetlab2
Import: importing node maxpl.maxpl.planetlab3
inserting node record for maxpl.maxpl.planetlab3
Import: importing node maxpl.maxpl.planetlab5
inserting node record for maxpl.maxpl.planetlab5
Import: importing node maxpl.maxpl.planetlab4
inserting node record for maxpl.maxpl.planetlab4
- NOTE: If you want to clean up whatever
gimport.py did, try rm -rf trusted_roots/ authorities/ then su -l postgres, psql planetlab4, type \dt, drop any tables starting with geni$ using drop table [tablename]
- set DESTDIR environment variable, run 'make' and 'make install':
[max-myplc] /usr/local/share/geniwrapper # export DESTDIR=/
[max-myplc] /usr/local/share/geniwrapper # make
...
[max-myplc] /usr/local/share/geniwrapper # make install
...
- check that these files have been deployed:
[max-myplc] /usr/local/share/geniwrapper # ls -l /etc/geni/geni_config
-rw-r--r-- 1 root root 1809 2009-05-27 11:36 /etc/geni/geni_config
[max-myplc] /usr/local/share/geniwrapper # ls -l /etc/init.d/geniwrapper
-rw-r--r-- 1 root root 1351 2009-05-27 11:17 /etc/init.d/geniwrapper
[max-myplc] /usr/local/share/geniwrapper # ls -l /usr/bin/plc.py
-rwxr-xr-x 1 root root 4602 2009-05-27 11:36 /usr/bin/plc.py
[max-myplc] /usr/local/share/geniwrapper # ls -l /usr/bin/sfi.py
-rwxr-xr-x 1 root root 20776 2009-05-27 11:36 /usr/bin/sfi.py
- if
/etc/geni/geni_config has not been configured, run geni-config-tty
- geniwrapper installer does not set +x on
/etc/init.d/geniwrapper:
[max-myplc] /usr/local/share/geniwrapper # chmod +x /etc/init.d/geniwrapper
[max-myplc] /usr/local/share/geniwrapper # /etc/init.d/geniwrapper start
Starting GENIWrapper: Registry
Aggregate [ OK ]
SliceMgr [ OK ]
- verify that
plc.py is running and listening on TCP ports 12345, 12346, 12347:
[max-myplc] /usr/local/share/geniwrapper # lsof -P | grep plc.py | grep LISTEN
plc.py 10210 root 5u IPv4 8126880 TCP *:12345 (LISTEN)
plc.py 10223 root 5u IPv4 8126891 TCP *:12346 (LISTEN)
plc.py 10236 root 5u IPv4 8126903 TCP *:12347 (LISTEN)
[max-myplc] /usr/local/share/geniwrapper/cmdline # cat configSfi.sh
export SFI_AUTH=maxpl.maxpl
export SFI_USER=$SFI_AUTH.chris
export SFI_REGISTRY=http://max-myplc.dragon.maxgigapop.net:12345/
export SFI_SM=http://max-myplc.dragon.maxgigapop.net:12346/
[max-myplc] /usr/local/share/geniwrapper/cmdline # source configSfi.sh
[max-myplc] /usr/local/share/geniwrapper # mkdir ~/.sfi
[max-myplc] /usr/local/share/geniwrapper # cd ~/.sfi
[max-myplc] ~/.sfi # cat > chris.pkey
-----BEGIN RSA PRIVATE KEY-----
[ NOTE!!! this must be an RSA key! ]
-----END RSA PRIVATE KEY-----
- don't forget this step, or else you will receive an error like
CertMissingParent: u'testing' in /var/log/geni.daemon
[max-myplc] /usr/local/share/geniwrapper/geni # cp authorities/maxpl/maxpl.pkey server.key
- don't forget to do this either, or else you will receive an error like 'no shared cipher' in
/var/log/geni.daemon (or 'sslv3 alert handshake failure' in the sfi.py traceback)
[max-myplc] /usr/local/share/geniwrapper/geni # cp authorities/maxpl/maxpl.gid server.cert
- view the contents of .gid/.cert/.key/.pkey files:
- for certificates: =openssl x509 -in [cert file] -noout -text
- for private keys: =openssl rsa -in [key file] -noout -text
- you should be able to run
sfi.py now, for example:
[max-myplc] /usr/local/share/geniwrapper/cmdline # ./sfi.py slices
maxpl.maxpl.netflow
maxpl.maxpl.sirius
[max-myplc] /usr/local/share/geniwrapper/cmdline # ./sfi.py show maxpl.maxpl.chris
RECORD maxpl.maxpl.chris
hrn: maxpl.maxpl.chris
type: user
gid:
hrn: maxpl.maxpl.chris
uuid: 97506516643947256837792453982171939435
pointer: 2
geni_info:
email : admin@domain.org
pl_info:
bio :
first_name : Default
last_name : Administrator
last_updated : 1243443965
roles : ['admin', 'pi', 'user', 'tech']
title : Engineer
url :
slices : []
enabled : True
sites : ['maxpl.maxpl']
phone : 123-456-7890
keys : ['...']
date_created : 1225830970
email : admin@domain.org
MyPLC v4.3
--
ChrisTracy - 13 Jan 2009