Skip to topic | Skip to bottom
Home
GENI


Start of topic | Skip to actions

Experiences in deploying MyPLC PlanetLab Central v4.2

This page is dedicated to documenting the procedure we used to deploy a MyPLC PlanetLab Central v4.2 distribution on the DRAGON network.

Install CentOS 5.2

On a fresh PC, install CentOS 5.2 (Linux RedHat variant).

Begin by downloading the netinstall ISO iamge at ftp://linux.mirrors.es.net/centos/5.2/isos/i386 (e.g. CentOS-5.2-i386-netinstall.iso)

The CentOS netinstall procedure requires the user to supply some information about the mirror. See http://www.chrisgountanis.com/technical/45-centos-netinstall.html for detailed instructions. We selected the 'HTTP' method for installation, using mirror.centos.org for the 'Website Name', and centos/5.2/os/i386 for the 'CentOS Directory'.

  • NOTE: Don't select 'Xen' during the installation. If you do, you will get exceptions in dmesg later on because you will be booting a vserver-enabled kernel, which does not include Xen support. If you accidentally select 'Xen', you will want to turn off libvirtd using chkconfig.

  • In our installation, we named this host planetlab1.domain.org

DNS configuration

  • We configured our DNS server as follows:
    • planetlab1.domain.org is an A record pointing to the IP address 192.168.100.133.
    • myplc.domain.org is an A record pointing to the IP address 192.168.100.131.

Install vserver-enabled kernel

  • After installing CentOS 5.2, create a new file /etc/yum.repos.d/dhozac-vserver.repo and populate it with the following:
[dhozac-vserver]
name=Linux-VServer related packages for CentOS $releasever - $basearch
baseurl=http://rpm.hozac.com/dhozac/centos/$releasever/vserver/$basearch
gpgkey=http://rpm.hozac.com/conf/keys/RPM-DHOZAC-GPG-KEY

  • Next, run yum -y update yum and yum -y install kernel

  • Edit /boot/grub/menu.lst (is actually a symlink to grub.conf in CentOS)
    • The last yum command should have added something like this (it should be the first kernel entry):
title CentOS (2.6.22.19-vs2.3.0.34.1)
        root (hd0,0)
        kernel /vmlinuz-2.6.22.19-vs2.3.0.34.1 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.22.19-vs2.3.0.34.1.img
    • Make sure you have default=0 in order to boot this kernel by default

Tweak SSH daemon

  • Edit /etc/ssh/sshd_config, change ListenAddress to the IP address configured eth0 (would block any guest's attempt to run sshd if you do not do this because sshd binds to '*:22' by default)

Turn SElinux off

  • edit /etc/sysconfig/selinux, set SELINUX=disabled

Reboot

  • Reboot the PC, when it comes back up, ensure that you can SSH to it and that the new vserver-enabled kernel is running:
[root@planetlab1 ~]# uname -a
Linux planetlab1 2.6.22.19-vs2.3.0.34.1 #1 SMP Mon Mar 17 05:32:04 EDT 2008 i686 i686 i386 GNU/Linux

Install util-vserver packages

[root@planetlab1 ~]# yum install util-vserver-core
[root@planetlab1 ~]# yum install util-vserver-lib
[root@planetlab1 ~]# yum install util-vserver-sysv
[root@planetlab1 ~]# yum install util-vserver-build

  • NOTE: We could have probably just done yum -y install util-vserver instead of the four commands above.

  • Additional steps required:
[root@planetlab1 ~]# /etc/init.d/vprocunhide start
Fixing /proc entries visibility...                         [  OK  ]
[root@planetlab1 ~]# chkconfig vservers-default on
[root@planetlab1 ~]# chkconfig --list vservers-default
vservers-default        0:off   1:off   2:on    3:on    4:on    5:on    6:off

Build Guest Systems

  • yum install all of these packages in "root context" (e.g. not inside the vserver):
    • subversion
    • httpd (on machines that mirror fedora repositories)
    • createrepo is needed in the root context on build boxes - this is for when just built rpms get published to the test framework
    • qemu is required on boxed that may host qemu-based test boxes
    • bridge-utils is mandatory on your qemu boxes as well

  • check out planetlab SVN build
    • this is really just a bunch of scripts to setup an environment so that myplc-native will run inside a vserver
[root@planetlab1 ~]# cd /root
[root@planetlab1 ~]# svn co http://svn.planet-lab.org/svn/build/trunk svn-build

Fedora and CentOS mirroring

  • We mirror the entire Fedora and CentOS distribution so that vserver instantiation is particularly fast.

  • Run one of the vbuild-fedora-mirror.sh commands below, we found the first one to give the best performance from MAX:
[root@planetlab1 ~]# cd /root/svn-build
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -u rsync://fedora.mirror.iweb.ca/fedora
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -u rsync://rsync.gtlib.gatech.edu/fedora-enchilada/linux
[root@planetlab1 svn-build]# ./vbuild-fedora-mirror.sh -v -l -f f8 -s

  • NOTE: This process takes a long time! The Fedora 8 mirror consumes approximately 19GB on our planetlab1.domain.org server.

Start apache in root context

  • Apache will run in the root context of planetlab1.domain.org and serve up Fedora/CentOS RPMs so that vservers can quickly be instantiated.
[root@planetlab1 ~]# /etc/init.d/httpd start

  • Make sure Apache is configured to automatically start whenever this machine is booted:
[root@planetlab1 ~]# chkconfig --level 2345 httpd on
[root@planetlab1 ~]# chkconfig --list httpd
httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

Apache configuration

  • make symlink to /mirror:
[root@planetlab1 ~]# cd /var/www/html
[root@planetlab1 html]# ln -s /mirror mirror

  • Modify Apache's /etc/httpd/conf/httpd.conf to only listen on 127.0.0.1:80 and root context's public IP, for example:
Listen 127.0.0.1:80
Listen 192.168.100.133:80
ServerName planetlab1.domain.org

  • We also set ServerName above, just to make sure it gets set to the proper value..

Update iptables

Since a complete mirror of Fedora and CentOS will be available from this Apache server (without any authentication), we would recommend adding a couple of iptables rules to the RH-Firewall-1-INPUT chain in /etc/sysconfig/iptables. For example, we limit incoming requests on TCP port 80 and 443 to our local network (206.196.176.0/21):

-A RH-Firewall-1-INPUT -s 206.196.176.0/21 -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -s 206.196.176.0/21 -p tcp --dport 443 -j ACCEPT

  • Run /etc/init.d/iptables restart to apply the changes

Tweaking yum config for vservers

  • The following commands will update /etc/vservers/.distributions/[f8|centos5]/yum.repos.d/building.repo so that baseurl points to http://planetlab1.domain.org/mirror/ instead of http://localhost/mirror/
[root@planetlab1 ~]# cd /root/svn-build/mirroring/
[root@planetlab1 mirroring]# ./mirror.sh -f f8 -u http://planetlab1.domain.org/mirror/ init
[root@planetlab1 mirroring]# ./mirror.sh -f centos5 -u http://planetlab1.domain.org/mirror/ init

  • NOTE: You don't want http://localhost/mirror/ because vserver creation will fail after installing the base operating system and starting the vserver and running yum!

  • After running the commands above, make sure the output of mirror.sh -a looks like the following (check that baseurl is correct):
[root@planetlab1 mirroring]# ./mirror.sh -a
==================== Contents of /etc/vservers/.distributions/f8/yum.repos.d/
==> /etc/vservers/.distributions/f8/yum.repos.d//building.repo <==
[fedora]
name=Fedora 8 - $basearch
baseurl=http://planetlab1.domain.org/mirror/fedora/releases/8/Everything/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=http://redhat.download.fedoraproject.org/pub/fedora/linux/releases/8/Everything/$basearch/os/RPM-GPG-KEY-fedora http://redhat.download.fedoraproject.org/pub/fedora/linux/releases/8/Everything/$basearch/os/RPM-GPG-KEY

[updates]
name=Fedora 8 - $basearch - Updates
baseurl=http://planetlab1.domain.org/mirror/fedora/updates/8/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

[updates-newkey]
name=Fedora 8 - $basearch - Updates Newkey
baseurl=http://planetlab1.domain.org/mirror/fedora/updates/8/$basearch.newkey/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-and-9
==================== /etc/vservers/.distributions/f9/yum.repos.d/ does not exist
==================== Contents of /etc/vservers/.distributions/centos5/yum.repos.d/
==> /etc/vservers/.distributions/centos5/yum.repos.d//building.repo <==
[base]
name=CentOS 5.2 - $basearch - Base
baseurl=http://planetlab1.domain.org/mirror/centos/5.2/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[updates]
name=CentOS 5.2 - $basearch - Released Updates
baseurl=http://planetlab1.domain.org/mirror/centos/5.2/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[epel]
name=Extra Packages for Enterprise Linux 5 - $basearch
baseurl=http://planetlab1.domain.org/mirror/epel/5/$basearch/
gpgcheck=1
gpgkey=http://download.fedora.redhat.com/pub/epel/RPM-GPG-KEY-EPEL

Create a vserver to run myplc-native

myplc-native is a software package that includes everything you need to manage and slice PlanetLab nodes. It will run inside a vserver on planetlab1.domain.org. We will call that vserver myplc.domain.org.

  • Please note that the root context (planetlab1.domain.org) has an IP address of 192.168.100.133. We will use a different IP address and hostname for the vserver running myplc-native, in this case, it will be myplc.domain.org with an IP address of 192.168.100.131.

  • Build a new vserver using the vtest-init-vserver.sh script:
[root@planetlab1 ~]# cd /root/svn-build
[root@planetlab1 svn-build]# ./vtest-init-vserver.sh -p linux32 myplc http://build.planet-lab.org/planetlab/f8/planetlab-f8-linux32-4.2-rc24/RPMS/ -- \
  --interface eth0:192.168.100.131 --hostname myplc.domain.org &> myplc.log &

NOTE: Visit http://build.planet-lab.org/planetlab/f8/?C=M;O=D to find the latest release and change the URL which you pass to the vtest-init-vserver.sh script. If you are interested in running the OneLab build of the PlanetLab software, begin by visiting http://build.onelab.eu/ to find a suitable URL to pass to vtest-init-vserver.sh.

  • Once that finishes, enter the vserver and install the myplc-native package:
[root@planetlab1 svn-build]# vserver myplc enter
[myplc] / # yum -y install myplc-native

  • Try to start/stop the plc service:
[myplc] / # service plc start
[myplc] / # service plc stop

  • Use chkconfig to make sure it comes up when this vserver is started:
[myplc] / # chkconfig plc on
[myplc] / # chkconfig --list plc
plc             0:off   1:off   2:on    3:on    4:on    5:on    6:off

  • Configure myplc-native from the console using plc-config-tty
[myplc] ~ # plc-config-tty
Enter command (u for usual changes, w to save, ? for help) u
== PLC_NAME : [My Domain PlanetLab Central]
== PLC_SLICE_PREFIX : [myplc]
== PLC_ROOT_USER : [pl-admin@domain.org]
== PLC_ROOT_PASSWORD : [root] password
== PLC_MAIL_ENABLED : [true]
== PLC_MAIL_SUPPORT_ADDRESS : [pl-admin@domain.org]
== PLC_DB_HOST : [localhost.localdomain] myplc.domain.org
== PLC_API_HOST : [localhost.localdomain] myplc.domain.org
== PLC_WWW_HOST : [localhost.localdomain] myplc.domain.org
== PLC_BOOT_HOST : [localhost.localdomain] myplc.domain.org
== PLC_NET_DNS1 : [192.168.1.1]
== PLC_NET_DNS2 : [192.168.2.1]
Enter command (u for usual changes, w to save, ? for help) w
Wrote /etc/planetlab/configs/site.xml
Merged
        /etc/planetlab/default_config.xml
and     /etc/planetlab/configs/site.xml
into    /etc/planetlab/plc_config.xml
+ service plc reload
You might want to type 'r' (restart plc) or 'q' (quit)
Enter command (u for usual changes, w to save, ? for help) r
==================== Stopping plc
...
==================== Starting plc
...

Fix procfs warnings in dmesg

  • If you run dmesg on planetlab1.domain.org (the root context), you will see messages like this:
> vxW: [<BB>crond<AB>,12389:#40056|40056|40056] messing with the procfs.

  • The current workaround requires running this command inside the guest vserver:
    • sed -i -e 's/.*pam_loginuid.*/#\0/' /etc/pam.d/*

  • NOTE: This is fixed in newer vserver kernels...(According to D.Hozac)...but these kernels are not ready for prime-time yet [as of December 2008].

Allow new users to be created

  • The is_public variable in the sites table of the planetlab4 postgresql database must be set to 't' (true) to allow new users to be created.

  • For some reason, there is no other way to set this besides mucking around manually with SQL.. The issue has been raised on the planetlab-devel mailing list, so the developers are aware of this issue.

  • Workaround:
[root@planetlab1 ~]# vserver myplc enter
[myplc] ~ # su -l postgres
-bash-3.2$ psql planetlab4
planetlab4=# update sites set is_public='t';
UPDATE 1
planetlab4=#

Skip hardware requirement checks

PlanetLab enforces sites to run relatively modern hardware, as described at http://www.planet-lab.org/hardware. There is actually a check upon bootup to make sure that the system meets the minimum hardware requirements. Since a lot of our PCs are a little older (or have small hard drives), we want to skip this requirement in our environment.

  • Modify bootmanager configuration to skip minimum hardware requirements:
    • Inside the myplc vserver, edit the file /usr/share/bootmanager/source/configuration
    • set SKIP_HARDWARE_REQUIREMENT_CHECK=1

  • Now, run service plc stop and service plc start within the myplc vserver

  • This should re-create the boot CD and ensure that any ISOs downloaded from the MyPLC web interface will not do the minimum hardware requirements check!

Login to MyPLC web-based user interface

The MyPLC web interface is a place where you can add new PlanetLab nodes, create and manage slices, etc. All of this can be done through an API as well, but the web interface is a good starting point for seeing how everything works.

  • Visit the URL: https://myplc.domain.org/
  • Username: admin@domain.org (see above - set by running plc-config-tty)
  • Password: password (see above - set by running plc-config-tty)

Create user accounts

To create a new user account, visit https://myplc.domain.org and click "Create an account".

If your site does not appear under the 'Site' drop-down menu, see the section above entitled "Allow new users to be created". You probably need to tweak a variable in your planetlab4 SQL database.

Once your account is created, you should login to your account and upload an SSH key. To do this, login to https://myplc.domain.org, click 'My Acccount' under 'Users' (and not 'my account' under 'drupal'). Click on 'Manage Keys' and use the form to upload your public SSH key. If don't already have a public key, generate one by running a command like ssh-keygen -t dsa. Then upload the file ~/.ssh/id_dsa.pub.

You will need to use this SSH key to login to "slices" that are created for you in later steps.

Add some PlanetLab nodes

Find some unused PCs to be used as PlanetLab nodes. These nodes can boot off of a CD-ROM or USB flash device. As of this writing, we have only experimented with the Boot CD option. PXE netboot is also being worked on, and would be much more convenient if you have nodes scattered throughout many POPs. Once these nodes are active, they may be "sliced" meaning that a Linux vserver will be automatically added/removed and handed off to an end user. They will be able to login via their SSH private key.

Login to the MyPLC web interface, click "Add Nodes" and fill out the form. We currently have two nodes -- planetlab2.domain.org and planetlab3.domain.org.

Once the node is added, you must download the boot materials for that node. Click on "My Nodes" on the left bar, then click on the hostname of the node you just added. On this page, you will be able to download an ISO image (for burning to CD), a USB image, the planetlab config file, or a generic ISO/USB image.

Burn the ISO image to a CD-ROM. Boot the node off of that CD.

You can control the 'boot state' of each node from the MyPLC web interface. Click on "My Nodes" on the left bar, click a hostname, and then examine the first option 'Boot State'. The link to the right of it ("What's this?") explains what each boot state option means.

If all goes well, after you boot up the CD the machine should register itself with MyPLC. It will ask you for confirmation before wiping the entire machine. The boot state in MyPLC should eventually come up to 'boot' if all goes well. The node should be "sliceable" at this point.

Create Slices

The whole point of setting up this environment is to be able to create "slices" for end users. This is essentially a vserver, which gets some percentage of CPU time and some chunk of disk space on a PlanetLab node. To create slices, do the following:

  • Login to MyPLC web interface
  • Click on "Create Slice"
  • Fill out the form with meaningful values for name/URL/description (we used testing for the name field in our examples below)
  • For the "Instantiation" option, choose "plc-instantiated"
  • Click "Add"

Now you need to associate nodes and users with this slice. This will control which users' SSH keys will be copied to the vserver, and which nodes they will be able to access.

  • Under "Slice Details" there is a "Choose Action" option
  • On this drop-down menu, select "Manage Nodes"
  • Put checkboxes in the nodes that you wish to be associated with this "slice", e.g. check planetlab2 and planetlab3, then click "Add Nodes"
  • Click "Back to Slice"
  • Now select "Manage Users" under the "Choose Action" drop-down list
  • Select which users you want to be able to login to the nodes (their SSH keys will be copied to that node), then click "Add People"
  • Click "Back to Slice"

It appears as though each planetlab node runs a "NodeManager" process in its root context. This will occasionally "check in" with MyPLC to see if it needs to do anything -- I believe this happens every 10 minutes. Within 10 minutes, you should see new vservers appear on the nodes which you selected (look under /etc/vservers in the root context), and SSH keys should be copied over to that node so the users can access their slices.

Logging into your "slice"

In order to login to your planetlab slice, you must add an SSH key to your user profile in MyPLC. Assuming you have done this, after you setup a slice and associate your user with that slice, the SSH keys will be copied to the node.

How does this actually work?

Assume you have just created a new slice called myplc_testing and have associated node planetlab2.domain.org and user admin@domain.org with that slice.

What actually happens when you use "create slice" above is this:

  • new user named myplc_testing is created on planetlab2.domain.org (NOTE: in the root context! not the vserver!)
  • SSH keys for admin@domain.org are copied to planetlab2.domain.org:/home/myplc_testing/.ssh/authorized_keys (NOTE: in the root context! not the vserver!)
  • new vserver named myplc_testing is created (see /etc/vservers/myplc_testing/...)

The new vserver is not actually running an SSH daemon, nor does it have a unique IP address assigned to it. PlanetLab nodes run a special process called pl_sshd on which somehow logs users into their vservers automatically. Take a look at the sshd processes running on a PlanetLab node:

[root@planetlab2 ssh]# ps auxwww | grep sshd | grep sbin
root      1263  0.0  0.2   6880  1060 ?        Ss   Nov06   0:52 /usr/sbin/sshd
root      1289  0.0  0.2   6880  1056 ?        Ss   Nov06   0:00 /usr/local/sbin/pl_sshd -o PidFile /var/run/pl_sshd.pid -p 806

[root@planetlab2 ssh]# lsof -i -U -P | grep ^sshd | grep LISTEN
sshd       1263             root    3u  IPv6       4467           TCP *:22 (LISTEN)
sshd       1289             root    3u  IPv6       4512           TCP *:806 (LISTEN)

Note that 'sshd' listens on port 22, but 'pl_sshd' listens on port 806.

Whenever I login as user myplc_testing using the SSH key for admin@domain.org, I am transferred into the myplc_testing vserver:

somehost% ssh myplc_testing@planetlab2.domain.org
[myplc_testing@planetlab2 ~]$ sudo -s
[root@planetlab2 ~]# ps auxwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
508      26605  0.5  0.2   2632  1416 ?        Ss   19:32   0:00 /bin/bash -l
root     26617  2.0  0.2   2632  1424 ?        S    19:32   0:00 /bin/bash
root     26621  0.0  0.1   2456   824 ?        R+   19:32   0:00 ps auxwww

Note that I cannot "see" my SSH .authorized_keys file -- because that actually lives in the root context, not the vserver!

[myplc_testing@planetlab2 ~]$ ls -l ~/.ssh
ls: cannot access /home/myplc_testing/.ssh: No such file or directory

How to SSH to PlanetLab node's "root context"

Normally, users would upload their public key to the MyPLC web interface, and when new slices are created, vservers are setup on the PlanetLab nodes and the SSH keys for that user are automatically installed.

In order to SSH to the "root context" of a PlanetLab node, that has booted off of the Boot CD, you must use the SSH private key. Login via password is disabled. Use the private key inside the MyPLC vserver in /etc/planetlab/root_ssh_key.rsa. NOTE: If the node has come up in debug mode (either intentionally or otherwise) you must use the /etc/planetlab/debug_ssh_key.rsa private key. For example:

[root@planetlab1 ~]# ssh root@planetlab1.domain.org
[root@planetlab1 ~]# vserver myplc enter
[myplc] / # ssh -i /etc/planetlab/root_ssh_key.rsa root@planetlab2.domain.org
[root@planetlab2 ~]# uname -a
Linux planetlab2.domain.org 2.6.22.19-vs2.3.0.34.29.planetlab #1 SMP Fri Oct 17 12:33:05 EDT 2008 i686 i686 i386 GNU/Linux

Adding tagged 802.1Q VLAN interface to vserver

We have found two ways to add a tagged 802.1Q VLAN interface to a vserver. Normally this would be done using something like vconfig add eth1 3000 and configured with an IP address by running ifconfig eth1.3000 inet A.B.C.D. However, this method will not work for vservers.

Option 1: during vserver instantiation

With the latest releases (sometime around 12/15/2008), you can pass --interface ethX.YYYY directly to the vserver build command. For example, this could be passed into the vtest-init-vserver.sh script:

./vtest-init-vserver.sh [...] -- --interface eth0:10.1.1.1/24 --interface eth1.3000:10.2.2.1/24 --hostname [...]

Option 2: modifying an existing vserver

  • Do something like this within the root context:
[root@planetlab1 ~]# vconfig add eth1 1234
Added VLAN with VID == 1234 to IF -:eth1:-
[root@planetlab1 ~]# ifconfig eth1.1234 inet 10.1.1.1 netmask 255.255.255.0

  • NOTE: By default, eth1.1234 will now show up in all PlanetLab-instantiated vservers!

  • Now the vserver has a tagged VLAN interface:
[root@planetlab1 ~]# vserver test-vlan enter
bash-3.2# ifconfig eth1.1234
eth1.1234 Link encap:Ethernet  HWaddr 00:30:48:9A:D3:89
          inet addr:10.30.30.1  Bcast:10.30.30.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:8286 (8.0 KiB)

Using the API

  • On planetlab1, check out plcsh using subversion:
[root@planetlab1 ~]# svn co http://svn.planet-lab.org/svn/plcmdline/trunk plcsh
A    plcsh/plcapilib.py
A    plcsh/setup.py
A    plcsh/plcapilib.spec
A    plcsh/plcsh
A    plcsh/Makefile
A    plcsh/plcmdline.py
Checked out revision 12695.

  • Run plcsh as follows, using -h to point it to your PLCAPI and -u to specify the user that you created above:
[root@planetlab1 ~]# cd plcsh  
[root@planetlab1 plcsh]# ./plcsh -h https://myplc.domain.org/PLCAPI/ -r user -u admin@domain.org
Password: [enter password here]
PlanetLab Central API server at https://myplc.domain.org/PLCAPI/
admin@domain.org connected as user using password authentication
Type "system.listMethods()" or "help(method)" for more information.
[chris@maxgigapop.net user]>>> 

Tweaking Node Manager's Polling Interval

  • Each "sliceable" planetlab node has a process called 'nm.py'. This is a python application that acts as a Node Manager, which will start/stop vservers as needed by checking in with the central MyPLC node.

  • By default, Node Manager has a polling interval of 600 seconds, plus a 301 second range for "additional random polling interval":
[root@planetlab2 ~]# grep interval /usr/share/NodeManager/nm.py
parser.add_option('-p', '--period', action='store', dest='period', default=600, help='Polling interval (sec)')
parser.add_option('-r', '--random', action='store', dest='random', default=301, help='Range for additional random polling interval (sec)')

  • If you have hundreds/thousands of nodes, these are probably reasonable values. However, if you have a small number of nodes and would like to see vservers created/deleted very quickly, you can edit /etc/init.d/nm and add these options to the command line.

  • Example:
    • edit /etc/init.d/nm and tweak this line: options=${OPTIONS-"-d -s -p 30 -r 1"}
    • stop nm service: /etc/init.d/nm stop
    • start nm service: /etc/init.d/nm start
    • check output of ps auxwww|grep nm to make sure you see the new options -p 30 -r 1 listed on the command line

Miscellaneous Notes

Deploying geniwrapper

  • start with svn checkout:
  • read geniwrapper/INSTALL file
  • in myplc vserver: yum install pyOpenSSL (necessary to run python code)
    • had to tweak /etc/yum.repos.d/myplc.repo before yum would start working
    • baseurl=URL was invalid (site must have changed?)
    • changed to: baseurl=http://build.planet-lab.org/planetlab/f8/planetlab-f8-linux32-4.2-rc24/RPMS/
  • in myplc vserver: yum install m2crypto (also necessry to run python code)
  • in myplc vserver: yum install make (geniwrapper now requires make package)

[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py 
Base Directory:  /usr/local/share/geniwrapper/
Import: creating top level authorities
Hierarchy: creating authority: plc
Segmentation fault

  • This segfault appears to be due to a bug in pyOpenSSL v0.6. Upgrade to a newer version of pyOpenSSL to fix this problem.

  • Getting gcc installed in the myplc vserver was not as easy as it should be. yum install gcc kept complaining with "Error: Missing Dependency: kernel-headers is needed by package glibc-headers-2.7-2.i386"

  • However, it was not possible to simply run yum install kernel-headers for some reason. I did the following to install the kernel-headers package and finally install gcc so that I could build pyOpenSSL v0.8:
% vserver myplc enter
[myplc] ~ # wget http://planetlab1.domain.org/mirror/fedora/releases/8/Everything/i386/os/Packages/kernel-headers-2.6.23.1-42.fc8.i386.rpm
[myplc] ~ # rpm -i kernel-headers-2.6.23.1-42.fc8.i386.rpm
[myplc] ~ # yum install gcc

  • Finally I had a working gcc and could proceed with building pyOpenSSL v0.8. First, I had to install the openssl header files by using yum install openssl-devel. Now I could build pyOpenSSL using python setup.py build. I removed pyOpenSSL v0.6 with yum erase pyOpenSSL and the installed v0.8 with python setup.py install.

  • Now python gimport.py succeeds without any segfaults, but it fails to import a user:

[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py
Base Directory:  /usr/local/share/geniwrapper
Import: creating top level authorities
maxplc : 
testing : 
Import: adding maxplc to trusted list
Import_Site: importing site testing.maxpl
Import: importing person testing.maxpl.chris
XXX: DSA key encountered, ignoring
Failed to import: {'bio': '', 'first_name': 'Default', 'last_name': 'Administrator', 'last_updated': 1239288497, 'roles': ['admin', 'pi', 'user', 'tech'], 'title': 'Engineer', 'url': '', 'key_ids': [1], 'enabled': True, 'slice_ids': [], 'phone': '240-687-1498', 'peer_person_id': None, 'role_ids': [10, 20, 30, 40], 'person_id': 2, 'date_created': 1225830970, 'site_ids': [1], 'peer_id': None, 'email': 'chris@maxgigapop.net'}
...

  • Notice above that it says "XXX: DSA key encountered, ignoring", this script does not seem to like DSA keys. Create an RSA key, upload this via the MyPLC? web interface, and then re-run gimport.py:
[max-myplc] /usr/local/share/geniwrapper/geni # python gimport.py
Base Directory:  /usr/local/share/geniwrapper
Import: creating top level authorities
Hierarchy: creating authority: maxpl
maxpl :
Import: creating table for authority maxpl
NOTICE:  table "geni$maxpl" does not exist, skipping
  inserting authority record for maxpl
maxpl :
Import: adding maxpl to trusted list
Import_Site: importing site maxpl.maxpl
Hierarchy: creating authority: maxpl.maxpl
  inserting authority record for maxpl.maxpl
Import: importing person maxpl.maxpl.chris
Import: creating table for authority maxpl.maxpl
NOTICE:  table "geni$maxpl$maxpl" does not exist, skipping
/usr/share/keyconvert/keyconvert /tmp/tmp9PeVdQ /tmp/tmp1Q8B7f
keytype = ssh-rsa
completed
  inserting user record for maxpl.maxpl.chris
/usr/share/keyconvert/keyconvert /tmp/tmpXt8DJE /tmp/tmpAgJ2Xp
keytype = ssh-rsa
completed
  updating user record for maxpl.maxpl.chris
Import: importing person maxpl.maxpl.poneil
   person maxpl.maxpl.poneil does not have a PL public key
  inserting user record for maxpl.maxpl.poneil
Import: importing person maxpl.maxpl.tlehman
   person maxpl.maxpl.tlehman does not have a PL public key
  inserting user record for maxpl.maxpl.tlehman
Import: importing person maxpl.maxpl.jflidr
   person maxpl.maxpl.jflidr does not have a PL public key
  inserting user record for maxpl.maxpl.jflidr
Import: importing slice maxpl.maxpl.netflow
  inserting slice record for maxpl.maxpl.netflow
Import: importing slice maxpl.maxpl.sirius
  inserting slice record for maxpl.maxpl.sirius
Import: importing node maxpl.maxpl.planetlab2
  inserting node record for maxpl.maxpl.planetlab2
Import: importing node maxpl.maxpl.planetlab3
  inserting node record for maxpl.maxpl.planetlab3
Import: importing node maxpl.maxpl.planetlab5
  inserting node record for maxpl.maxpl.planetlab5
Import: importing node maxpl.maxpl.planetlab4
  inserting node record for maxpl.maxpl.planetlab4

  • NOTE: If you want to clean up whatever gimport.py did, try rm -rf trusted_roots/ authorities/ then su -l postgres, psql planetlab4, type \dt, drop any tables starting with geni$ using drop table [tablename]

  • set DESTDIR environment variable, run 'make' and 'make install':

[max-myplc] /usr/local/share/geniwrapper # export DESTDIR=/
[max-myplc] /usr/local/share/geniwrapper # make
...
[max-myplc] /usr/local/share/geniwrapper # make install
...

  • check that these files have been deployed:

[max-myplc] /usr/local/share/geniwrapper # ls -l /etc/geni/geni_config 
-rw-r--r-- 1 root root 1809 2009-05-27 11:36 /etc/geni/geni_config
[max-myplc] /usr/local/share/geniwrapper # ls -l /etc/init.d/geniwrapper 
-rw-r--r-- 1 root root 1351 2009-05-27 11:17 /etc/init.d/geniwrapper
[max-myplc] /usr/local/share/geniwrapper # ls -l /usr/bin/plc.py
-rwxr-xr-x 1 root root 4602 2009-05-27 11:36 /usr/bin/plc.py
[max-myplc] /usr/local/share/geniwrapper # ls -l /usr/bin/sfi.py 
-rwxr-xr-x 1 root root 20776 2009-05-27 11:36 /usr/bin/sfi.py

  • if /etc/geni/geni_config has not been configured, run geni-config-tty

  • geniwrapper installer does not set +x on /etc/init.d/geniwrapper:
[max-myplc] /usr/local/share/geniwrapper # chmod +x /etc/init.d/geniwrapper 

  • start GENIWrapper:
[max-myplc] /usr/local/share/geniwrapper # /etc/init.d/geniwrapper start
Starting GENIWrapper:  Registry
Aggregate                                                  [  OK  ]
SliceMgr                                                   [  OK  ]

  • verify that plc.py is running and listening on TCP ports 12345, 12346, 12347:
[max-myplc] /usr/local/share/geniwrapper # lsof -P | grep plc.py | grep LISTEN
plc.py    10210     root    5u     IPv4    8126880               TCP *:12345 (LISTEN)
plc.py    10223     root    5u     IPv4    8126891               TCP *:12346 (LISTEN)
plc.py    10236     root    5u     IPv4    8126903               TCP *:12347 (LISTEN)

  • configure sfi:
[max-myplc] /usr/local/share/geniwrapper/cmdline # cat configSfi.sh
export SFI_AUTH=maxpl.maxpl
export SFI_USER=$SFI_AUTH.chris
export SFI_REGISTRY=http://max-myplc.dragon.maxgigapop.net:12345/
export SFI_SM=http://max-myplc.dragon.maxgigapop.net:12346/
[max-myplc] /usr/local/share/geniwrapper/cmdline # source configSfi.sh

  • setup keys:
[max-myplc] /usr/local/share/geniwrapper # mkdir ~/.sfi
[max-myplc] /usr/local/share/geniwrapper # cd  ~/.sfi
[max-myplc] ~/.sfi # cat > chris.pkey
-----BEGIN RSA PRIVATE KEY-----
[ NOTE!!! this must be an RSA key! ]
-----END RSA PRIVATE KEY-----

  • don't forget this step, or else you will receive an error like CertMissingParent: u'testing' in /var/log/geni.daemon
[max-myplc] /usr/local/share/geniwrapper/geni # cp authorities/maxpl/maxpl.pkey server.key 

  • don't forget to do this either, or else you will receive an error like 'no shared cipher' in /var/log/geni.daemon (or 'sslv3 alert handshake failure' in the sfi.py traceback)
[max-myplc] /usr/local/share/geniwrapper/geni # cp authorities/maxpl/maxpl.gid server.cert 

  • view the contents of .gid/.cert/.key/.pkey files:
    • for certificates: =openssl x509 -in [cert file] -noout -text
    • for private keys: =openssl rsa -in [key file] -noout -text

  • you should be able to run sfi.py now, for example:
[max-myplc] /usr/local/share/geniwrapper/cmdline # ./sfi.py slices
maxpl.maxpl.netflow
maxpl.maxpl.sirius
[max-myplc] /usr/local/share/geniwrapper/cmdline # ./sfi.py show maxpl.maxpl.chris
RECORD maxpl.maxpl.chris
        hrn: maxpl.maxpl.chris
       type: user
        gid:
          hrn: maxpl.maxpl.chris
         uuid: 97506516643947256837792453982171939435
    pointer: 2
  geni_info:
        email : admin@domain.org
    pl_info:
        bio : 
        first_name : Default
        last_name : Administrator
        last_updated : 1243443965
        roles : ['admin', 'pi', 'user', 'tech']
        title : Engineer
        url : 
        slices : []
        enabled : True
        sites : ['maxpl.maxpl']
        phone : 123-456-7890
        keys : ['...']
        date_created : 1225830970
        email : admin@domain.org

MyPLC v4.3

-- ChrisTracy - 13 Jan 2009


You are here: GENI > Deploying-MyPLC-v42

to top

Copyright © 1999-2012.
The information contained in these pages is the property of the Mid-Atlantic Crossroads (MAX).
If you have questions or comments, please contact MAX Administration