[edit]
Current workaronds:
apt-get install libjetty-java ln -sf ../java/asm3.jar /usr/share/eucalyptus/ apt-get install python-boto=1.8d-1 (not Debian's 1.9x)[edit]
Followed this guide:
NB: When registering you cannot use localhost, or often even the hostname - you'll just get "error" spat back at you via the GUI or CLI. You _must_ use a non 127.x IP address. /var/log/eucalyptus/cloud-error.log will say: Components cannot be registered using local, link-local, or multicast addresses.You can either use GUI:
visit https://localhost:8443log in using admin/admin (changing password etc as prompted) select "Configuration" tab select "Register Walrus" Walrus host: euca.example.com (everything else is default) Buckets path: /var/lib/eucalyptus/bukkits Maximum buckets per user: 5 MB maximum bucket size: 5120 MB of disk are reserved for the image cache: 30720 (NB: on old versions I had to increase this to 8000 (from 0?)) GB of disk are reserved for snapshots: 50 select "Save Walrus configuration" select "Register cluster" Name: examplecluster Host: euca.example.com (rest default) Port: 8774 Dynamic public IP address assignment: enabled Reserve for assignment: 10 public IP addresses Maxiumum of: 5 public IP addresses per user Use VLAN tags: 10 through 4095 storage controller: Host: euca.example.com Interface: lo (change from eth0) Volumes path: /var/lib/eucalyptus/volumes Max volume size: 10 GB Space reserved for volumes: 50 GB Zero fill volumes: disabled select "Save cluster configuration" NB: this _silently_ fails on my servers as you cannot rsync as root scp /var/lib/eucalyptus/keys/cloud-cert.pem /var/lib/eucalyptus/keys/examplecluster/* clustercontroller:/var/lib/eucalyptusOr use the CLI:
euca_conf --register-walrus euca_conf --register-cluster NB: this fails on my servers as you cannot rsync as root scp /var/lib/eucalyptus/keys/cloud-cert.pem /var/lib/eucalyptus/keys/examplecluster/* clustercontroller:/var/lib/eucalyptus{node-cert.pem,cluster-cert.pem,node-pk.pem,cloud-cert.pem} node:/var/lib/eucalyptus/keys (obviously that's an illegal command, but you get the idea)
euca_conf --register-scNow register a node:
euca_conf --register-nodes " ... " this failed with: ERROR: you need to be on the CC host and the CC needs to be running. "/etc/init.d/eucalyptus-cc restart" fixed this even so, it still fails since it tries to rsync (as current user) /var/lib/eucalyptus/keys to node0 which is bad since it's the current box... fortunately it tells you what commands to type in - something like this: ERROR: could not synchronize keys with 192.168.101.10! The configuration will not have this node. Hint: to setup passwordless login to the nodes as user eucalyptus, you can run the following commands on node 192.168.101.10: sudo -u eucalyptus mkdir -p ~eucalyptus/.ssh sudo -u eucalyptus tee ~eucalyptus/.ssh/authorized_keys > /dev/null < it also complained that: Warning: cannot file file node-cert.pem in //var/lib/eucalyptus/keys/ Warning: cannot file file cluster-cert.pem in //var/lib/eucalyptus/keys/ Warning: cannot file file node-pk.pem in //var/lib/eucalyptus/keys/ this seems to be bug 439087 and fixed by the extra copies we do above now unless the node you add is elsewhere in which case you need to: scp clustercontroller:/var/lib/eucalyptus/keys/{node-cert.pem,cluster-cert.pem,node-pk.pem,cloud-cert.pem} node:/var/lib/eucalyptus/keys (obviously that's an illegal command, but you get the idea)You must also change libvirt to run as "eucalyptus" (rather than the standard libvirt-qemu):
edit /etc/libvirt/qemu.conf: user="eucalyptus" /etc/init.d/libvirt-bin restart I had old copies lying around, so ensure it's restarted and only one copy is running edit /etc/eucalyptus/eucalyptus.conf and check these values (NB: you probably want eth0, not wlan1): HYPERVISOR="kvm" VNET_BRIDGE="br0" VNET_PUBINTERFACE="wlan1" VNET_PRIVINTERFACE="wlan1" ensure eucalyptus is in the libvirt group (id eucalyptus), and if it is missing: adduser libvirt eucalyptus bounce everything: /etc/init.d/eucalyptus-cloud restart /etc/init.d/eucalyptus-cc restart /etc/init.d/eucalyptus-nc restart[edit]
See Network config.
[edit]
Ensure you have a dhcp server listening on the bridged interface, otherwise the instances will never get an IP address.
On each node (you'll have to do this on boot of course):
. /etc/eucalyptus/eucalyptus.conf ip addr add 169.254.169.254/32 scope link dev $VNET_PRIVINTERFACE iptables -t nat -A PREROUTING -d 169.254.169.254 -p tcp --dport 80 -j DNAT --to-destination 169.254.169.254:8773 if your cloud controller isn't local, use that as the --to-destination IP address[edit]
[edit]
Kernel image uploading (following Image Management):
. ~/.euca/eucarc euca-bundle-image -i karmic-server-uec-i386-vmlinuz-virtual [-r (i386|x86_64)] --kernel true NB: -r x86_64 is the default creates files in /tmp: i386 Checking image Tarring image Encrypting image Splitting image... Part: karmic-server-uec-i386-vmlinuz-virtual.part.0 Generating manifest /tmp/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml euca-upload-bundle -b kernel -m /tmp/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml Checking bucket: kernel Uploading manifest file Uploading part: karmic-server-uec-i386-vmlinuz-virtual.part.0 Uploaded image as kernel/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml euca-register kernel/karmic-server-uec-i386-vmlinuz-virtual.manifest.xml Originally needed this patch (but not now) IMAGE eki-B66F18B4Kernel initrd (if required - not needed on lucid ATM):
euca-bundle-image -i karmic-server-uec-i386-initrd-virtual [-r (i386|x86_64)] --ramdisk true i386 Checking image Tarring image Encrypting image Splitting image... Part: karmic-server-uec-i386-initrd-virtual.part.0 Generating manifest /tmp/karmic-server-uec-i386-initrd-virtual.manifest.xml euca-upload-bundle -b initrd -m /tmp/karmic-server-uec-i386-initrd-virtual.manifest.xml Checking bucket: initrd Creating bucket: initrd Uploading manifest file Uploading part: karmic-server-uec-i386-initrd-virtual.part.0 Uploaded image as initrd/karmic-server-uec-i386-initrd-virtual.manifest.xml euca-register initrd/karmic-server-uec-i386-initrd-virtual.manifest.xml IMAGE eri-984C1830Disk image:
NB: you can alternatively supply ramdisk/kernels at runtime euca-bundle-image -i karmic-server-uec-i386.img [-r (i386|x86_64)] [--kernel eki-B66F18B4] [--ramdisk eri-984C1830] takes a while i386 Checking image Tarring image Encrypting image Splitting image... Part: karmic-server-uec-i386.img.part.0 Part: karmic-server-uec-i386.img.part.1 Part: karmic-server-uec-i386.img.part.2 Part: karmic-server-uec-i386.img.part.3 Part: karmic-server-uec-i386.img.part.4 Part: karmic-server-uec-i386.img.part.5 Part: karmic-server-uec-i386.img.part.6 Part: karmic-server-uec-i386.img.part.7 Part: karmic-server-uec-i386.img.part.8 Part: karmic-server-uec-i386.img.part.9 Part: karmic-server-uec-i386.img.part.10 Part: karmic-server-uec-i386.img.part.11 Part: karmic-server-uec-i386.img.part.12 Part: karmic-server-uec-i386.img.part.13 Part: karmic-server-uec-i386.img.part.14 Part: karmic-server-uec-i386.img.part.15 Part: karmic-server-uec-i386.img.part.16 Part: karmic-server-uec-i386.img.part.17 Part: karmic-server-uec-i386.img.part.18 Generating manifest euca-upload-bundle -b image -m /tmp/karmic-server-uec-i386.img.manifest.xml Checking bucket: image Uploading manifest file Uploading part: karmic-server-uec-i386.img.part.0 Uploading part: karmic-server-uec-i386.img.part.1 Uploading part: karmic-server-uec-i386.img.part.2 Uploading part: karmic-server-uec-i386.img.part.3 Uploading part: karmic-server-uec-i386.img.part.4 Uploading part: karmic-server-uec-i386.img.part.5 Uploading part: karmic-server-uec-i386.img.part.6 Uploading part: karmic-server-uec-i386.img.part.7 Uploading part: karmic-server-uec-i386.img.part.8 Uploading part: karmic-server-uec-i386.img.part.9 Uploading part: karmic-server-uec-i386.img.part.10 Uploading part: karmic-server-uec-i386.img.part.11 Uploading part: karmic-server-uec-i386.img.part.12 Uploading part: karmic-server-uec-i386.img.part.13 Uploading part: karmic-server-uec-i386.img.part.14 Uploading part: karmic-server-uec-i386.img.part.15 Uploading part: karmic-server-uec-i386.img.part.16 Uploading part: karmic-server-uec-i386.img.part.17 Uploading part: karmic-server-uec-i386.img.part.18 Uploaded image as image/karmic-server-uec-i386.img.manifest.xml euca-register image/karmic-server-uec-i386.img.manifest.xml IMAGE emi-656D1322[edit]
[edit]
Can't talk to your instance?
check "euca-describe-instances" RESERVATION r-3C1F071C admin default INSTANCE i-482D07A3 emi-656D1322 0.0.0.0 0.0.0.0 terminated 0 m1.small 2010-03-10T20:36:53.345Z ashcluster eki-B66F18B4 eri-984C1830 oh dear it's died. look in /var/log/eucalyptus/nc.log: [EUCAFATAL ] error: insufficient disk capacity remaining (2042MB) in VM Type of instance i-i-482D07A3 for component disk aha we mention this in #Errors, so we increase the m1.small disk allowance to 3GB, only to get this: [EUCAINFO ] currently running/booting: i-i-49BA093C [EUCAERROR ] libvirt: monitor socket did not show up.: Connection refused (code=38) [EUCAFATAL ] hypervisor failed to start domain we could look in /var/log/libvirt/qemu/i-49BA093C to see what went wrong, but for the sake of completeness, let's dig deeper now we set "MANUAL_INSTANCES_CLEANUP=1 in /etc/eucalyptus/eucalyptus.conf, restart the NC and try again [EUCAINFO ] vrun(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/add_key.pl //usr/lib/eucalyptus/euca_mountwrap 32256 /var/lib/eucalyptus/instances/admin/i-482D07A3/disk ] [EUCADEBUG ] system_output(): [//usr/lib/eucalyptus/euca_rootwrap //usr/share/eucalyptus/gen_kvm_libvirt_xml --ramdisk --ephemeral] [EUCAINFO ] currently running/booting: i-482D07A3 [EUCAERROR ] libvirt: monitor socket did not show up.: Connection refused (code=38) [EUCAFATAL ] hypervisor failed to start domain same problem of course, but now we can debug it by looking at it cd /var/lib/eucalyptus/instances/admin/i-482D07A3 edit libvirt.xml appropriately, some suggestions: add " " in the "devices" section now when the instance is running, if you run "ps axf" you'll see part of the command is "-vnc 0.0.0.0:2" or similar, run "xvncviewer :2" to attach VNC to it virsh define libvirt.xml Domain i-482D07A3 defined from libvirt.xml virsh start i-482D07A3 error: Failed to start domain i-482D07A3 error: monitor socket did not show up.: Connection refused looking in /var/log/libvirt/qemu/i-482D07A3.log shows: LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin HOME=/ /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 128 -smp 1,sockets=1,cores=1,threads=1 -name i-482D07A3 -uuid e0bdb05b-6cda-4a74-248f-3c77f47eac95 -nographic -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-482D07A3.monitor,server,nowait -mon chardev=monitor,mode=readline -boot c -kernel /var/lib/eucalyptus/instances/admin/i-482D07A3/kernel -initrd /var/lib/eucalyptus/instances/admin/i-482D07A3/ramdisk -append root=/dev/sda1 console=ttyS0 -device lsi,id=scsi0,bus=pci.0,addr=0x5 -drive file=/var/lib/eucalyptus/instances/admin/i-482D07A3/disk,if=none,id=drive-scsi0-0-0,boot=on -device scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0 -device e1000,vlan=0,id=net0,mac=d0:0d:3a:11:06:71,bus=pci.0,addr=0x4 -net tap,fd=39,vlan=0,name=hostnet0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances/admin/i-482D07A3/console.log -device isa-serial,chardev=serial0 -usb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 chardev: opening backend "file" failed it's trying to create console.log but doesn't have permissions as I'd missed the step which tells libvirt to run as the "eucalyptus" user once finished with the VM: virsh destroy i-482D07A3 virsh undefine i-482D07A3 if you want to adjust the libvirt.xml file, you'll have to undefine it (see above) before redefining it[edit]
Interesting things to check:
503180 - delays on networkudev issuesgrant SSH access (step 2) - relates to #452556[edit]
When an instance is started, a copy of the EMI is created as part of the instance's storage. In fact the instance normally gets three pieces of storage on Eucalyptus (Eucalyptus storage locations):
/dev/sda1: EMI image as a root filesystem (mounted on /) /dev/sda2: ephemeral (short term) storage (mounted on /mnt) of whatever space is left in the VM type /dev/sda3: swap (according to what is set in /etc/eucalyptus/eucalyptus.conf, however the maths seems inaccurate - I had 640MB swap, not 512MB)However there are inconsistencies between different types on EC2. Eucalyptus is self consistent so this can cause differences. (Amazon storage locations):
/dev/sda1: EMI image as a root filesystem (mounted on /) /dev/sda2: ephemeral (short term) storage (mounted on /mnt) for m1.small and c1.medium VM types /dev/sda3: swap for m1.small and c1.medium types /dev/sdb: ephemeral (short term) storage (mounted on /mnt) for all other types /dev/sd{cde}: exist on certain types http://open.eucalyptus.com/wiki/EucalyptusWalrusInteracting_v1.6http://open.eucalyptus.com/wiki/EucalyptusBlockStoreInteracting_v1.6To create some permanent storage, you need to create a volume and attach it. This storage will be in S3 (for EC2) or Walrus (for Eucalyptus):
euca-create-volume -S size -z zone VOLUME vol-5AF5064E 1 creating 2010-03-15T08:58:50.961Z euca-describe-volumesVOLUME vol-5AF5064E 1 ashcluster available 2010-03-15T08:58:50.961Z
euca-attach-volume -i i-4E6E0A01 -d /dev/sdc vol-5AF5064E Volume: Storage services are not available for the requested availability zone. Check the Storage Controller details in the web interface volume fails to attach nc.log contains "libvirt: cannot open file '/dev/etherd/e0.1': No such file or directory (code=38)" sadly this seems to be a known issue - you can't currently use storage where the NC (node controller - where the VMs run) is on the same box as the SC (storage controller) brief attempt to work around this, installed iscsitarget (and ran it), added 'DISABLE_ISCSI="N"' to /etc/eucalyptus/eucalyptus.conf and restarted[edit]
euca-run-instances then failed:
walrus_request(): failed to open /var/lib/eucalyptus/instances/admin/i-39ED07BE/kernel-digest for writing mkdir /var/lib/eucalyptus/instances/admin; chown to eucalyptus error: touch(): failed to create/open file /var/lib/eucalyptus/instances/eucalyptus/cache/eki-B64E18B0/kernel-staging (No such file or directory) mkdir /var/lib/eucalyptus/instances/eucalyptus; chown to eucalyptus and now it's currently not running despite no errors in nc.log. time to look at KVM logs... grep -v DEBUG nc.log |grep -v "no domain with mat" | grep -v INFO libvirt: internal error no supported architecture for os type 'hvm' (code=1) yep - we are on a VM ourselves here (since I'm testing) aha, edit /etc/eucalyptus/eucalyptus.conf VNET_BRIDGE="virbr0"[edit]
[edit]
(e.g. m1.small)
FIX: login to cloudcontroller:8443, increase disk size for the appropriate VMtype in "Configuration" KVM fails with (from /var/log/libvirt/qemu) 'chardev: opening backend "file" failed' libvirt cannot create the console.log file (permissions problem) FIX: run libvirt as "eucalyptus" (see earlier in these notes) also see #536966 Hanging on "Waiting for EC2 meta-data service" (see 424065 although this _is_ Eucalyptus - maybe I need to setup more networking?) FIX: see Network config - this is because we are running in SYSTEM mode and not MANAGED so we need to set this up manually on each node (see #Networking) Or for a bad workaround, add 'ec2init=0' to the command line (by editing gen_kvm_libvirt_xml")[edit]
SCSI disks weren't working with my setup at some point, so here is a patch to use virtio :-)
--- /usr/share/eucalyptus/gen_kvm_libvirt_xml.orig 2009-11-13 13:57:52.000000000 +0000 +++ /usr/share/eucalyptus/gen_kvm_libvirt_xml 2009-11-13 13:58:55.000000000 +0000 @@ -92,7 +92,7 @@ } print < root=/dev/sda1 console=ttyS0 + root=/dev/vda1 console=ttyS0 @@ -103,7 +103,7 @@ /usr/bin/kvm - +[edit]
Fixing image (1.6 tools now):
euca-describe-images euca-download-bundle -b image emi-39F60D7D euca-unbundle -m lenny.img.manifest.xml (destination)Image creation (ubuntu - failed) EC2VMBuilder docs or see creating an UEC image:
sudo vmbuilder kvm ubuntu --suite hardy --flavour virtual --arch i386 -t $PWD --rootsize 2048 --swapsize 256 on debian this fails: 2009-08-06 13:36:28,607 INFO : Cleaning up Traceback (most recent call last): File "/usr/bin/vmbuilder", line 29, in VMBuilder.run() File "/usr/lib/python2.5/site-packages/VMBuilder/__init__.py", line 65, in run frontend.run() File "/usr/lib/python2.5/site-packages/VMBuilder/plugins/cli/__init__.py", line 68, in run vm.create() File "/usr/lib/python2.5/site-packages/VMBuilder/vm.py", line 483, in create self.install() File "/usr/lib/python2.5/site-packages/VMBuilder/vm.py", line 416, in install self.distro.install_bootloader() File "/usr/lib/python2.5/site-packages/VMBuilder/plugins/ubuntu/distro.py", line 164, in install_bootloader EOT''') File "/usr/lib/python2.5/site-packages/VMBuilder/util.py", line 105, in run_cmd proc = subprocess.Popen(args, stdin=stdin_arg, stderr=subprocess.PIPE, stdout=subprocess.PIPE, env=proc_env) File "/usr/lib/python2.5/subprocess.py", line 594, in __init__ errread, errwrite) File "/usr/lib/python2.5/subprocess.py", line 1153, in _execute_child raise child_exception OSError: [Errno 2] No such file or directoryImage creation:
dd if=/dev/zero of=lenny.img bs=1M count=2048 mkfs.ext3 lenny.img sudo mount -o loop lenny.img lenny sudo debootstrap --arch =i386 --variant=minbase lenny foo sudo chroot lenny apt-get install netbase puppet sudo umount lenny[edit]
from: http://www.smop.co.uk/mediawiki/index.php/Eucalyptus