RUNNING AIX v7.2 VM ON QEMU HYPERVISOR ON AN UBUNTU HOST

This procedure documents setting up the latest available QEMU on Ubuntu in order to run an AIX v7.2 VM.
Most of the steps are from http://aix4admins.blogspot.com/2020/04/qemu-aix-on-x86-qemu-quick-emulator-is.html?m=1

The host in this case is an AWS t3.xlarge compute instance running Ubuntu 22.04.1 LTS (Jammy Jellyfish)
I also attached a secondary EBS volume (55G) to the instance which I mounted on /wip and where I stored all the relevant files.


- Login to the Ubuntu host and install QEMU:

root@ip-172-31-23-252:~# apt update -y
root@ip-172-31-23-252:~# apt install -y gcc make ninja-build
root@ip-172-31-23-252:~# wget https://download.qemu.org/qemu-7.2.0.tar.xz
root@ip-172-31-23-252:~# tar xvf qemu-7.2.0.tar.xz
root@ip-172-31-23-252:~# cd qemu-7.2.0/
root@ip-172-31-23-252:~/qemu-7.2.0# apt install libglib2.0-dev
root@ip-172-31-23-252:~/qemu-7.2.0# apt-get install -y libpixman-1-dev
root@ip-172-31-23-252:~/qemu-7.2.0# apt install ncurses-dev
root@ip-172-31-23-252:~/qemu-7.2.0# ./configure
// ALTERNATIVELY - build only PPC64 support: # ./configure --target-list=ppc64-softmmu --enable-curses --disable-gtk && make
root@ip-172-31-23-252:~# make
root@ip-172-31-23-252:~# make install


- Partition the secondary volume and format the file system:

root@ip-172-31-23-252:~# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 24.4M  1 loop /snap/amazon-ssm-agent/6312
loop1          7:1    0 55.6M  1 loop /snap/core18/2632
loop2          7:2    0 63.2M  1 loop /snap/core20/1695
loop3          7:3    0  103M  1 loop /snap/lxd/23541
loop4          7:4    0 49.6M  1 loop /snap/snapd/17883
nvme0n1      259:0    0    8G  0 disk
├─nvme0n1p1  259:1    0  7.9G  0 part /
├─nvme0n1p14 259:2    0    4M  0 part
└─nvme0n1p15 259:3    0  106M  0 part /boot/efi
nvme1n1      259:4    0   55G  0 disk
root@ip-172-31-23-252:~#
root@ip-172-31-23-252:~# fdisk /dev/nvme1n1
root@ip-172-31-23-252:~# partprobe
root@ip-172-31-23-252:~# mkfs -t ext4  /dev/nvme1n1p1
root@ip-172-31-23-252:~# blkid
/dev/nvme0n1p1: LABEL="cloudimg-rootfs" UUID="687fab62-1ba5-4282-890e-9266064f7d27" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="895d8984-5441-4c70-b87c-a6b6ebb8c95e"
/dev/nvme0n1p15: LABEL_FATBOOT="UEFI" LABEL="UEFI" UUID="B2B4-82AC" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="0cf1c52c-98f5-48ae-8a07-fff782190e30"
/dev/loop0: TYPE="squashfs"
/dev/nvme1n1p1: UUID="a5051753-344e-43da-ba1f-cc785cab98b0" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="d816173f-01"
root@ip-172-31-23-252:~# vi /etc/fstab
root@ip-172-31-23-252:~# grep wip /etc/fstab
UUID="a5051753-344e-43da-ba1f-cc785cab98b0"  /wip  ext4  defaults 0 0
root@ip-172-31-23-252:~#
root@ip-172-31-23-252:~# mkdir /wip
root@ip-172-31-23-252:~# mount /wip


- Copy the AIX v7.2 ISO files to the Ubuntu instance. Please ensure you get it from a legal source.

root@ip-172-31-23-252:~# cd /wip
root@ip-172-31-23-252:/wip# mkdir AIX72ISOs
root@ip-172-31-23-252:/wip# cd AIX72ISOs/
root@ip-172-31-23-252:/wip/AIX72ISOs# scp -i ~/.ssh/wipalinux ubuntu@172.31.18.141:/wip/AIX72ISOs/aix_7200-04-02-2027_1of2_072020.iso .


- Create a disk for the AIX VM:
root@ip-172-31-23-252:~# cd /wip/
root@ip-172-31-23-252:/wip# qemu-img create -f qcow2 hdisk0.qcow2 20G


- Install AIX (you can change install settings e.g., to include SSH client and server). The installation tool approx 110 mins 
root@ip-172-31-23-252:/wip# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom /wip/AIX72ISOs/aix_7200-04-02-2027_1of2_072020.iso -prom-env "boot-command=boot cdrom:"

   - NOTE: the VM will get stuck in a reboot loop at the end of installation. Use CTRL+C to terminate the VM
   
   
- Fix/solve the fsck64 issue to fix the reboot loop by booting the VM into maintenance mode:
root@ip-172-31-23-252:/wip# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -cdrom /wip/AIX72ISOs/aix_7200-04-02-2027_1of2_072020.iso -prom-env "boot-command=boot cdrom:"
   - menu options to select:  1 to "define the System Console" > 1 for English > 3 for Maintenance mode > 1 to access root VG > 0 to continue > 1 to select VG/disk > 1 to "Access this Volume Group and start a shell"
  - NOTE: no keyboard BACKSPACE key, and don't use CTRL+C as that terminates the VM.

  # cd /sbin/helpers/jfs2
  # cp fsck64 fsck64.org

  - truncate the fsck64 exeutable binary file and replace content with shell script
  # > fsck64
  # cat > fsck64 << EOF
  #!/bin/ksh
  exit 0
  EOF
  #
  # cat fsck64
  #!/bin/ksh
  exit 0
  #

  - Alternative to the cat sequence above is to edit the fsck64 file after truncating it and add the 2 lines to the file:
  # > fsck64
  # export TERM=vt100
  # vi fsck64
  # cat fsck64
  #!/bin/ksh
  exit 0
  #


  - Shutdown the VM:
  #
  # sync; sync
  # halt


- Create a snapshot of the AIX O/S disk for backup purposes:
root@ip-172-31-23-252:/wip# qemu-img create -f qcow2 -b hdisk0.qcow2 -F qcow2 hdisk0.snap.qcow2 10G


- Boot the VM to AIX O/S 7.2 using the O/S disk, and accept license (I excluded cdrom since I no longer need it):
root@ip-172-31-23-252:/wip# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:"
   - choose vt100 (type it and press ENTER) when prompted for terminal type
   - choose to accept the license (default is no, press TAB key to change it to yes) then ENTER to accept
   - Esc+0 (hold down ESC then press 0) to go back
   - accept the software maintenance terms/conditions
   - Esc+0 (hold down ESC then press 0) to go back
   - Set any of the additional settings as required (date/time; root password; etc)
   - Option "Tasks completed - Exit to Login"
   
   - Login as root on the console (prompt)
-----------------------------------------------
   
- Fix the RPM DB error:  https://bobcares.com/blog/rpm-db_runrecovery-errors/
# cd /opt/freeware
# tar -chvf `date +"%d%m%Y"`.rpm.packages.tar packages
# rm -f /opt/freeware/packages/__*
# /usr/bin/rpm --rebuilddb
# /usr/bin/rpm -qa

-----------------------------------------------
   
   
- Setup networking: https://kwakousys.wordpress.com/2020/09/06/run-aix-7-2-on-x86-with-qemu/
    - in this example, we assign IP address 10.0.2.16 to AIX and 10.0.2.20 to the bridge we defined on the Ubuntu host.

- Setup a bridge (br0) on the Ubuntu host:
    root@ip-172-31-23-252:/wip# apt-get install bridge-utils
    root@ip-172-31-23-252:/wip# mkdir -p /usr/local/etc/qemu
    root@ip-172-31-23-252:/wip# echo "allow br0" > /usr/local/etc/qemu/bridge.conf
	
    NOTE: you can put the following network-related commands a single script that you can just run as a single command
	
    root@ip-172-31-23-252:/wip# ip link add name br0 type bridge
    root@ip-172-31-23-252:/wip# ip link set dev br0 up
    root@ip-172-31-23-252:/wip# ip addr add 10.0.2.20/24 dev br0


- Setup the tap NIC for the AIX VM:
    root@ip-172-31-23-252:/wip# ip tuntap add tap0 mode tap
    root@ip-172-31-23-252:/wip# ip link set dev tap0 up
    root@ip-172-31-23-252:/wip# ip link set dev tap0 master br0

    NOTE: tap0 interface comes up when the VM is started


- Setup the Ubuntu host for routing (including Internet access from the AIX VM):
    root@ip-172-31-23-252:/wip# echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp
    root@ip-172-31-23-252:/wip# ip route add 10.0.2.16 dev tap0
    root@ip-172-31-23-252:/wip# arp -Ds 10.0.2.16 eth0 pub
    root@ip-172-31-23-252:/wip# echo 1 > /proc/sys/net/ipv4/ip_forward
    root@ip-172-31-23-252:/wip# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    root@ip-172-31-23-252:/wip# iptables -I FORWARD 1 -i tap0 -j ACCEPT
    root@ip-172-31-23-252:/wip# iptables -I FORWARD 1 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
        

- Start the AIX VM normally (assign a randomly selected MAC address to the VM's NIC):
root@ip-172-31-23-252:/wip# qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:" -net nic,macaddr=be:16:43:37:16:ec -net tap,script=no,ifname=tap0,downscript=no


- Assign the IP address 10.0.2.16 to the en0 NIC in AIX  (use SMIT to make it permanent)
# chdev -l en0 -a netaddr=10.0.2.16 -a netmask=255.255.255.0 -a state=up

- Make the IP assignment permanent with SMIT (assign any IP on the same network as the gateway e.g., 10.0.2.254):
# smit tcpip > Min Config & Startup > en0 > (setup hostname/netmask/IP/nameserver & domain name & gateway e.g., aix7vm/10.0.2.16/255.255.255.0/8.8.8.8 & acme.com/10.0.2.254) > "START Now" = yes (TAB key to change it) then ENTER key to execute the change
   NOTE: the name server (e.g., Google's 8.8.8.8 DNS server) and a domain name MUST be provided if you decide to set the name server.


- Install BASH shell in AIX VM (bash is easier to use than the default Korn shell):

- increase /opt as the bash instal requires about 40MB space:
# chfs -a size=+60M /opt

# wget http://www.oss4aix.org/download/latest/aix71/libiconv-1.16-1.aix5.1.ppc.rpm
# wget http://www.oss4aix.org/download/latest/aix71/bash-5.0-8.aix5.1.ppc.rpm
# wget http://www.oss4aix.org/download/latest/aix71/gettext-0.19.8.1-1.aix5.1.ppc.rpm
# wget http://www.oss4aix.org/download/RPMS/gcc/libgcc-6.3.0-1.aix7.2.ppc.rpm
# rpm -ivh bash_5_0_8_aix5_1_ppc.rpm gettext_0_19_8_1_1_aix5_1_ppc.rpm libiconv_1_16_1_aix5_1_ppc.rpm libgcc_6_3_0_1_aix7_2_ppc.rpm

- In AIX, after installing bash, "authorize" AIX to allow bash shell to run:
# export TERM=vt100
   - Edit file /etc/security/login.cfg, append "/usr/bin/bash" (without the double quotes)  to the line containing "shells ="
   - Edit file /etc/shells, append this on a new line "/usr/bin/bash" (without the double quotes)
   

--------------- END OF PROCEDURE ---------------


- Extra step in order to access the AIX VM using SSH from outside the Ubuntu host (particularly useful if you are using the "--daemonize" headless option when starting the AIX VM):
Summary is to use iptables to redirect incoming attempts to connect to the Ubuntu instance on some alternate port (e.g., 2222/tcp) to port 22 on the AIX VM. Note that you also need to allow incoming traffic on this alternate port in your AWS/OCI/GCP VPC/subnet using the relevant security group rule.

root@ip-172-31-23-252:/wip# iptables -A INPUT -p tcp --dport 2222 -j ACCEPT
root@ip-172-31-23-252:/wip# iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.2.16:22


You can then connect to the AIX with putty (Ubuntu IP address and port 2222) or using SSH with a command such as: ssh root@<ubuntu-ip> -p 2222


   
References:
http://aix4admins.blogspot.com/2020/04/qemu-aix-on-x86-qemu-quick-emulator-is.html?m=1
Run AIX 7.2 on x86 with QEMU
https://worthdoingbadly.com/aixqemu/
http://www.visidon.com/blog/2015/02/bash-on-aix-7-1/
RPM DB_RUNRECOVERY errors: How to resolve
http://www.oss4aix.org/download/latest/aix71/ - download RPMs for various packages http://www.oss4aix.org/download/RPMS/gcc/ https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/ - Some useful commands: # qemu-system-ppc64 ... -daemonize (to run the VM in "headless" mode so access it using SSH). If you use this option, delete the "-serial stdio" from the command. QEMU starts the VM and gives a message such as "VNC server running on 127.0.0.1:5900" so you can use VNC to manage the VM as well. root@ip-172-31-23-252:/wip# mount -o loop /wip/bash51-aix71.iso /iso root@ip-172-31-23-252:/wip# ip link set dev br0 down root@ip-172-31-23-252:/wip# ip link delete dev br0 root@ip-172-31-23-252:/wip# ip a # mount -vcdrfs -oro /dev/cd0 /mnt # entstat -d en0 | grep -i hard Hardware Address: be:16:43:37:16:ec #aix# chsh <username> /bin/bash logout from session, after AIX shutdown is possible using "~~.", same as in HMC console root@ip-172-31-23-252:/wip# apt install -y genisoimage root@ip-172-31-23-252:/wip# mkisofs -max-iso9660-filenames -o bash50.iso ./bash50 - boot the AIX with the ISO image containing the bash rpm package; qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -serial stdio -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:" -net nic,macaddr=be:16:43:37:16:ec -net tap,script=no,ifname=tap0 -cdrom /wip/bash50.iso - Some notes: "make" of QEMU took about 85 mins on t3.xlarge when compiling all supported platforms, but under 10 mins when compiling for ppc64 support only) ensure you have plenty of space for the compiler. qemu-7.2.0.tar.xz is 117M, extracted folder qemu-7.2.0 is 799M, compiled, the extracted folder goes to almost 6GB! if you didn't include the ssh client/server during the installation, you will need to start the VM with the ISO image inserted in the CDROM so you can install them. - Optionally disable some un-needed services to speed up the boot process: - edit file /etc/rc.tcpip and comment out # some services if not required e.g., sendmail, snmpd, hostmibd, snmpmibd, aixmibd (look for lines similar to: start /usr/sbin/aixmibd "$src_running") - to disable the NFSD server, edit file /etc/rc.nfs and comment out the line: start biod /usr/sbin/biod - you may use the "stopsrc -s <service-name>" command to shut them down in the current session as well. - Optionally disable additional services defined in the /etc/inittab file to make subsequent boot ups faster (using the following commands): # rmitab rcnfs # rmitab cron # rmitab piobe # rmitab qdaemon # rmitab writesrv # rmitab naudio2 # rmitab naudio # rmitab aso # rmitab clcomd # chrctcp -S -d tftpd
- The networking setup, and AIX VM launch command scripts (execute the network script before the AIX VM launch script so that the VM will have network access):

root@ip-172-31-23-252:/wip# cat setup_networking_for_aix.sh
#!/usr/bin/bash

#- Setup the tap NIC for the AIX VM:
ip tuntap add tap0 mode tap
ip link set dev tap0 up

#NOTE: tap0 interface comes up when the VM is started:

#- Setup the host for routing (including Internet access from the AIX VM):
echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp
ip route add 10.0.2.16 dev tap0
arp -Ds 10.0.2.16 eth0 pub
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -I FORWARD 1 -i tap0 -j ACCEPT
iptables -I FORWARD 1 -o tap0 -m state --state RELATED,ESTABLISHED -j ACCEPT

#- Setup port forwarding so that the AIX VM is accessible remotely:
iptables -A INPUT -p tcp --dport 2222 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.2.16:22



root@ip-172-31-23-252:/wip# cat launch_aix72_vm.sh
#!/usr/bin/bash

/usr/local/bin/qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:" -net nic,macaddr=be:16:43:37:16:ec -net tap,script=no,ifname=tap0,downscript=no --daemonize
#/usr/local/bin/qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -drive file=hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:" -net nic,macaddr=be:16:43:37:16:ec -net tap,script=no,ifname=tap0,downscript=no

- VNC connection to the AIX VM
When qemu is executed with the --daemonize option, it also creates a VNC session that you can connect to. By default the VNC session is started on only the loopback (127.0.0.1) interface. In the sample command below, it is started on the primary interface of the Ubuntu host with the IP 172.31.23.252. I can then use any VNC viewer such as tightvnc to connect to the VM's console using the Ubuntu hosts' public IP:
root@ip-172-31-23-252:/wip# /usr/local/bin/qemu-system-ppc64 -cpu POWER8 -machine pseries -m 4096 -drive file=/wip/hdisk0.qcow2,if=none,id=drive-virtio-disk0 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=drive-virtio-disk0 -prom-env "boot-command=boot disk:" -net nic,macaddr=be:16:43:37:16:ec -net tap,script=no,ifname=tap0,downscript=no --daemonize -vnc 172.31.23.252:0