Sunday, January 27, 2019

Exporting an Amazon EC2 Instance and running it as a VM in QEMU

Some of you may know that exporting a live instance off EC2 is not easy. From all the limitations the most important is that the only way to export an instance is if it was imported from another virtualization environment in the first place.

In my opinion these area artificial limitations designed to make it harder for you to leave the Amazon eco-system. Legitimate, but not nice. It gets even worse due to the fact that the formats accepted are all connected with paid virtualization solutions.

As I do not care about paid virtualization solutions and just wanted to dump the instance as a way to keep a perfect copy of the instance for backup and make tests I decided to research on how to dump an instance the "monkey" way.

Dumping and running the instance locally

For my goals I consider an instance as a virtual private server with a main storage device. The storage device should contain all that is required to boot and start "as if" it was still running in the cloud.

There is a quick and dirty way, and there is a proper way

The quick and dirty involves making the disk dump while it is being used.  This will almost certainly lead to a slightly corrupted image, but it is more than recoverable if you call sync and are not doing any particularly intensive or important IO operation.

The proper way is just a few extra steps before the dirty way.
  1. Start a random instance to which you have ssh credentials and root access.
  2. Stop the instance which you want to backup.
  3. Detach the EBS volume from your main instance and attach it the newly created temporary instance.
  4. Start the temporary instance but do not mount the EBS storage.
  5. Get the device name of the EBS volume.
    lsblk
  6. Start of the quick and dirty way

  7.  On your local machine download the device contents.
    ssh -T $INSTANCE_ADDRESS 'dd bs=16M if=$EBS_DEVICE | bzip2 -c' | bzcat > $IMAGE_DESTINATION
    
  8. (Optional) Convert image to qcow2
  9. qemu-img convert -f raw -O qcow2 $IMAGE_DESTINATION $IMAGE_DESTINATION.qcow2
    
  10. Boot image on qemu.
    qemu-system-x86_64 -drive format=raw,file=$IMAGE_DESTINATION -enable-kvm -serial mon:stdio -vga virtio -device rtl8139,netdev=net0 -netdev user,id=net0,hostfwd=tcp::10022-:22,hostfwd=tcp::10443-:443,hostfwd=tcp::1080-:80 -m 2G
    

Notes and small explanations

Different compression algorithms

Instead of bzip it is possible to use other compression algorithms as well as different levels of compression. An example would be a gzip -> zcat pipe pair, or xz  -> xzcat pair.

 

 Convert image

What you download should be a raw disk image. You may want the disk image to be a more clever underlying storage like for example qcow2. You can have other formats though, including vdl, to use on Virtualbox application. See more information about qemu-img convert here.

 

System interaction

Contrary to the Amazon EC2, you can actually have live view of the booting process and kernel printing. This is quite useful to see if something went wrong with the dump. For example if you went with the quick and dirty way your file systems may need to be checked or some systems fail due to on-line file system dumping. This is in most cases survivable but you will be glad to have this information.

The command line part that redirects the kernel console to the console of the qemu application is
 -serial mon:stdio

Connectivity


QEMU connectivity and command line is huge and highly configurable. It is assumed that anybody doing work with an instance will want to connect to the machine through SSH or expose server services running there. There are mutiple ways to do it with different degrees of difficulty. In the qemu command presented above you can find that the ports of the Guest(Instance) will be exposed on the localhost.

  • Port 22 (SSH) of the Guest  mapped to localhost:10022
  • Port 443 (SSL) mapped to 10443
  • Port 80 (HTTP) mapped to 1080
Note that mapping ports to different values may lead to, for example broken web services. This is due to the fact that there may be automatic redirections on the browser side that discard the port numbers and will point to the Guest VM ports, which will then not match. The reason I did not match the ports of the guest directly to the localhost(host) is because some lower ports like 443 and 80 require root privileges, so qemu would need to be root.

Another trick to really fool all the redirections in you host to point to your virtual machine is to add an entry to the /etc/hosts file mapping for example a domain name of your real instance to your loopback address. This has worked really well for me but be aware that you need to disable after you are done otherwise you will be very confused :D

Wednesday, June 13, 2018

qemu serial device

I had massive trouble getting additional serial devices recognized by my kernel because for some very stupid reason (including me possibly being even stupider): the serial devices are attached as ISA devices by default. 

Yes, when you add the "-serial stdio" or something similar the serial device is on an ISA bus instead of the ultra-modern PCI. I have not verified this for myself but read here. Also I have verified that if I add the serial character device as a PCI device it magically is detected by the kernel.

As I am basing my kernel in multi_v7_defconfig, I cannot even find CONFIG_ISA in the menuconfig kernel configuration tool. I did not venture into manually adding it to the .config, as when the menuconfig does not show me love, i learned to be a gentleman. Also do not forget to add the serial driver, usable by QEMU, if you are building your own kernel .config. The relevant fragment is:

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
# CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_EXAR=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_OF_PLATFORM=y


To run 'serial for tcp' but with a pci-serial I found this explanation.
 -chardev socket,host=127.0.0.1,port=59715,id=hostserial      -device pci-serial,chardev=hostserial

Sunday, March 04, 2018

ENV{DEVTYPE} does not seem to match

For some reason even though udevadm reports DEVTYPE, no udev rule matches when I try to use ENV{DEVTYPE}. Used ID_TYPE instead. Example udevadm report


UDEV [16712.707352] add /devices/platform/soc/soc:usb3-0/12000000.dwc3/xhci-hcd.3.auto/usb3/3-1/3-1.1/3-1.1:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block) ACTION=add DEVLINKS=/dev/disk/by-id/usb-Lexar_USB_Flash_Drive_0FBC0FD4BECE8A47B3FE-0:0-part1 /dev/disk/by-partuuid/8ec91e13-01 /dev/disk/by-path/platform-xhci-hcd.3.auto-usb-0:1.1:1.0-scsi-0:0:0:0-part1 /dev/disk/by-uuid/662e6fe9-a29c-4a12-bc51-4a13c58b57c4 DEVNAME=/dev/sda1 DEVPATH=/devices/platform/soc/soc:usb3-0/12000000.dwc3/xhci-hcd.3.auto/usb3/3-1/3-1.1/3-1.1:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1 DEVTYPE=partition ID_BUS=usb ID_FS_TYPE=ext4 ID_FS_USAGE=filesystem ID_FS_UUID=662e6fe9-a29c-4a12-bc51-4a13c58b57c4 ID_FS_UUID_ENC=662e6fe9-a29c-4a12-bc51-4a13c58b57c4 ID_FS_VERSION=1.0 ID_INSTANCE=0:0 ID_MODEL=USB_Flash_Drive ID_MODEL_ENC=USB\x20Flash\x20Drive\x20 ID_MODEL_ID=a209 ID_PART_ENTRY_DISK=8:0fgf ID_PART_ENTRY_FLAGS=0x80 ID_PART_ENTRY_NUMBER=1 ID_PART_ENTRY_OFFSET=2048 ID_PART_ENTRY_SCHEME=dos ID_PART_ENTRY_SIZE=62535680 ID_PART_ENTRY_TYPE=0x83 ID_PART_ENTRY_UUID=8ec91e13-01 ID_PART_TABLE_TYPE=dos ID_PART_TABLE_UUID=8ec91e13 ID_PATH=platform-xhci-hcd.3.auto-usb-0:1.1:1.0-scsi-0:0:0:0 ID_PATH_TAG=platform-xhci-hcd_3_auto-usb-0_1_1_1_0-scsi-0_0_0_0 ID_REVISION=8.07 ID_SERIAL=Lexar_USB_Flash_Drive_0FBC0FD4BECE8A47B3FE-0:0 ID_SERIAL_SHORT=0FBC0FD4BECE8A47B3FE ID_TYPE=disk ID_USB_DRIVER=usb-storage ID_USB_INTERFACES=:080650: ID_USB_INTERFACE_NUM=00 ID_VENDOR=Lexar ID_VENDOR_ENC=Lexar\x20\x20\x20 ID_VENDOR_ID=05dc MAJOR=8 MINOR=1 PARTN=1 SEQNUM=44772 SUBSYSTEM=block USEC_INITIALIZED=712706913

Sunday, February 25, 2018

wic in yocto

As of the time of writing the manual of wic in the the Yocto Mega manual has a massive omission. It only explains how to use wic individually and does not talk about the IMAGE_FSTYPES = "wic". Because of this if you ran wic create you will get strange errors like.

Please make sure wic-tools have %s-native in its DEPENDS, bake it with 'bitbake wic-tools'

This message will not appear if you set IMAGE_FSTYPES = "wic", because the IMAGE_DEPENDS_wic already contains the above dependencies.

Also with the IMAGE_DEPENDS_wic_append_${MACHINE} you can deploy boot packages which wic could required to create the images.

PS: I hate blogger interface.

Edit: While grepping my poky I found that the included documentation refers to exactly what I wrote above. Well may be some googler will find it useful.

Wednesday, February 21, 2018

Nice trick to diagnose gcc search paths

Recently I needed to use a compiler as part of an sdk and needed to point it to the correct sysroot. For some reason the --sysroot flag did not work correctly. To inspect what was going on in gcc 'mind' a colleague told me about a "trick":

gcc -o  ./g.c -E  -P  -v -dD

Wednesday, October 04, 2017

External Source kernel modules install does not fail as it should

Another small post to serve for future memory that when you are building Linux Kernel modules from external tree source, the kernel build system behaves differently. Yesterday it happened that I lost a huge amount of time trying to understand why the installation of a module was not exiting with error, when I clearly saw errors in the logs. The reason is that this is on purpose. In the build scripts for kernel installation there is this hidden gem: https://github.com/torvalds/linux/commit/f6a79af8f3701b5a0df431a76adee212616154dc Don't stop modules_install if we can't sign external modules. I have been warned.

Monday, August 28, 2017

ParseError not a BitBake file

This error occurred to me when I made a require or include with the path to the file inside " quotation marks. The error message of course just said that  

ERROR: ParseError in "conf/machine/include/tune-cortexa53.inc": not a BitBake file

Solution: Remove the quotation marks around the file path in any require or include directives.