Tag Archives: KVM

Forwarding traffic from external hosts to guest VMs on KVM host

Posted on by
Reply

This is so simple but took me forever to find a concise source on the Internet even with the AI results provided by most search engines now.

Background: I set up KVM (qemu/libvirtd) on a Linux host, create a VM on it. I want to be able to SSH (goes for other in-bound traffic such as HTTP as well) from an external host e.g., a separate windows laptop to the guest VM running on the KVM Linux host.

<Windows Laptop> — SSH —> <KVM host> —> <Guest-VM>

The default network created for the guest-VM on KVM is NAT (and goes over the virbr0 interface created on the KVM host during the KVM installation)

To illustrate, I created two guest-VMs (both Linux) and I want to be able to SSH to them from another Windows system in my home network. This means I need to connect to a designated unused port (e.g., 2222 or 2223) on the KVM host via SSH, and have the KVM host forward that connection to the SSHD service running on the guest VM

1/ You MUST add a rule to the LIBVIRT_FWI that allows that traffic otherwise you get something like “Connection refused” for SSH for example. By default, only traffic that is part of on-going session is forwarded, which in effect means you can’t initiate traffic to the VM from outside the KVM host (inserts as rule #1 on top of the chain).
For some reason, the firewall-cmd command works until you restart the system or restart firewalld service (which fails that there is no table/chain with that name), so using the iptables command instead. The “-t filter” can also be omitted since it is the default table (as opposed to others such as nat, mangle, etc)
sudo firewall-cmd –permanent –direct –add-rule ipv4 filter LIBVIRT_FWI 0 -p tcp –dport=22 -m state –state NEW,ESTABLISHED -j ACCEPT
sudo iptables -t filter -I LIBVIRT_FWI -p tcp –dport 22 -m state –state NEW,ESTABLISHED -j ACCEPT

So to make it persistent using systemd methodology, I created a service for the command:
a) Create the file /etc/systemd/system/libvirt_fwi.service with the following content:
[Unit]
Description=Enable libvirtd SSH forwarding to VMs
After=graphical.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -t filter -I LIBVIRT_FWI -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

[Install]
WantedBy=graphical.target


b) Run the commands to create and start the service:
sudo systemctl restart firewalld.service
sudo systemctl restart libvirtd
sudo systemctl daemon-reload
sudo systemctl enable libvirt_fwi.service
sudo systemctl start libvirt_fwi.service

2/ Add the rules to forward the traffic from the KVM host to the target VM. In this example, any incoming traffic to the KVM host on port 2222/tcp is forwarded to a specific VM (e.g., 192.168.124.217) on port 22, and incoming traffic on 2223/tcp is forwarded to a second guest VM with IP 192.168.122.100.
sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 -p tcp –dport 2222 -j DNAT –to-destination 192.168.124.217:22
sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 -p tcp –dport 2222 -j DNAT –to-destination 192.168.124.100:22
sudo firewall-cmd –reload

3/ You can SSH from the KVM host to the VMs using their IP addresses.
These commands allows one to SSH to the VMs from the KVM host itself (the IPs belong to the VMs):
sudo iptables -t nat -A PREROUTING -p tcp –dport 2222 -j DNAT –to-destination 192.168.122.217:22
sudo iptables -t nat -A PREROUTING -p tcp –dport 2223 -j DNAT –to-destination 192.168.122.100:22

4/ SKIP THIS STEP – because KVM already added masquerade rules for the guest network to the NAT table – I am just including it here for reference in case it is needed for other purposes e.g., a hypervisor that requires you to do the NAT set up yourself: Enable NAT of out-going traffic using the KVM host interface the traffic will traverse out. In this example we only masquerade the subnet range used by KVM for the VMs, but you can also masquerade all outgoing traffic on that host interface wlp0s20f3 by leaving out the “-s network” parameter
sudo iptables -t nat -A POSTROUTING -s 192.168.124.0/24 -o wlp0s20f3 -j MASQUERADE

5/ Depending on your Linux distro, there are many ways to make the iptables changes permanent (across reboots). Below is one method for RHEL and its variants (CentOS, Fedora, Rocky, etc):
sudo yum install iptables-services
sudo systemctl enable iptables
sudo systemctl enable ip6tables
sudo systemctl status iptables
sudo systemctl status firewalld
sudo iptables-save > /etc/sysconfig/iptables

5/ To check the IP of your VM (outside from logging into it), you can use “sudo virsh domifaddr <guest-vm>” and the “sudo virsh net-dumpxml default” shows the IP range KVM allocates IP from to the guest VMs.

  • NOTE: You can use this guide to set up KVM on the host, but I didn’t need step 4 because I am not using the ethernet interface/NIC on my KVM host to create a bridge to my home WIFI (and it is very difficult to create a bridge using the WiFi interface – if this is something you need, better to use a hypervisor like VMWare or VirtualBox or Promox, etc.)
How to Install KVM on Rocky Linux 9 / AlmaLinux 9
  • Making iptables rules permanent: https://www.cyberciti.biz/faq/how-to-save-iptables-firewall-rules-permanently-on-linux/
  • IF you have a multi-NIC KVM host, and your in-coming traffic that you want to forward to the VM is coming on a specific NIC, then you use a slightly different rule (where the SOURCE_IP_ADDRESS is the IP on that NIC):
    sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 \
    -s <SOURCE_IP_ADDRESS> -p <PROTOCOL> –dport <PUBLIC_PORT> \
    -j DNAT –to-destination <INTERNAL_IP_ADDRESS>:<INTERNAL_PORT>
  • Use “sudo firewall-cmd –direct –get-all-rules” to display the direct rules

Minishift notes

Posted on by
Reply

Background: attempting to allow external hosts on the same network as the KVM host on which minishift VM is running to be able to access the minishift web UI and optionally via SSH. In this example, 192.168.42.62 is the IP assigned to minishift after starting it, and 192.168.10.0/24 is the IP range of the network on which the KVM host is running.

On my KVM Linux host, installing libvirtd had created the virbr0 bridge. Subsequently setting up minishift setup a second bridge named virbr1 to which the vNIC assigned to the minishift VM is slaved. Interestingly enough, the minishift VM is then given 2x interfaces with one attached to each bridge (virbr0 and virbr1). In the minishift VM, the default route is assigned to the interface (e.g., eth0) attached to the first bridge (virbr0) with IP 192.168.122.30, even though when minishift starts, it displays a URL with the IP address (192.168.42.62) assigned to the interface connected to the second bridge as the way to access the minishift web UI. This works fine as long as you are attempting to access the URL from the KVM host, but won’t work without some extra steps if you want to access the minishift UI from an external host.
wlp5s0 is the “public” interface on my KVM host.

Setup and start minishift (equivalent to RedHat CDK if you have the right subscription):
itababa@itamint:~$ sudo apt update -y
itababa@itamint:~$ sudo apt install qemu-kvm qemu-system qemu-utils python3 python3-pip libvirt-clients libvirt-daemon-system bridge-utils virtinst libvirt-daemon virt-manager cpu-checker -y
itababa@itamint:~$ usermod -aG libvirt $(whoami)
itababa@itamint:~$ newgrp libvirt
itababa@itamint:~$ sudo systemctl enable libvirtd
itababa@itamint:~$ sudo systemctl start libvirtd
itababa@itamint:~$ sudo virsh net-start default
itababa@itamint:~$ sudo virsh net-autostart default
itababa@itamint:~$ sudo curl -L https://github.com/dhiltgen/docker-machine-kvm/releases/download/v0.10.0/docker-machine-driver-kvm-ubuntu16.04 -o /usr/local/bin/docker-machine-driver-kvm
itababa@itamint:~$ sudo chmod +x /usr/local/bin/docker-machine-driver-kvm
itababa@itamint:~$ curl -L https://github.com/minishift/minishift/releases/download/v1.34.3/minishift-1.34.3-linux-amd64.tgz -o minishift-1.34.3-linux-amd64.tgz
itababa@itamint:~$ tar xf minishift-1.34.3-linux-amd64.tgz
itababa@itamint:~$ sudo cp minishift-1.34.3-linux-amd64/minishift /usr/local/bin/
itababa@itamint:~$ minishift start

NOTE: the “minishift start” command will display the minishift web UI at the end of its run.
Reference: https://docs.okd.io/3.11/minishift/getting-started/setting-up-virtualization-environment.html

Settings needed on the KVM host:
- configure the KVM host to redirect connections to it on 8443/tcp to the minishift host (192.168.42.62). Optionally redirect 2222/tcp to the minishift host as well:
iptables -F
iptables -F -t nat
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/virbr0/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/virbr1/proxy_arp    (required so that the host can answer when the minishift VM tries to find the ARP of external hosts, otherwise the minshift will not respond to any connection attempts)
iptables -t nat -A POSTROUTING -o wlp5s0 -j MASQUERADE
iptables -t nat -A PREROUTING -i wlp5s0 -p tcp --dport 8443 -j DNAT --to-destination 192.168.42.62:8443
iptables -t nat -A PREROUTING -i wlp5s0 -p tcp --dport 2222 -j DNAT --to-destination 192.168.42.62:22
Settings needed on the minishift VM after it is running (execute "minishift ssh" to SSH into the minishift VM):
- NOTE: this whole section can be skipped if you choose to "DNAT" port 8443/tcp to the IP address on the virbr0 vNIC on the KVM host:
itababa@itamint:~$ minishift ssh
[docker@minishift ~]$
 sudo su -
[root@minishift ~]# ip a     (find the interface with the 192.168.42.x IP e.g., eth1)
[root@minishift ~]# ip route add 192.168.10.0/0 via dev eth1 (this is because the default route uses the NIC with 192.168.122.x IP and used by minishift to access the Internet)
[root@minishift ~]# ip route show
default via 192.168.122.1 dev eth0 proto dhcp metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.10.0/24 dev eth1 scope link
192.168.42.0/24 dev eth1 proto kernel scope link src 192.168.42.62 metric 101
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.30 metric 100
[root@minishift ~]#
NOTE: if you restart the minishift VM, you have to SSH into it again and re-add the route to the 192.168.10.0/24 network.

Settings needed on the external host:
In this example, I want to access the minishift web UI from a Windows system on my home network. You need to add a route to the minishift IP address using the KVM host IP as the gateway (using the command line):
C:\Windows\system32>route add 192.168.42.206 MASK 255.255.255.255 192.168.10.4

Launch a browser on the external host and go directly to https://192.168.42.206:8443/console/catalog

NOTE: if you attempt to login via https://192.168.42.206:8443/console , you will encounter this error/bug after entering your credentials (e.g., admin/admin)
Error: “Error. Invalid request. Client state could not be verified. Return to the console.”
Bug: “https://bugzilla.redhat.com/show_bug.cgi?id=1589072”
Found the solution (use the /console/catalog URL): https://github.com/openshift/openshift-azure/issues/236

after login with admin/admin credential

Other commands:
minishift delete –clear-cache (solution for error similar to: “Cannot start – error starting the VM: error getting the state for host: machine dies not exist-docker” when attempting to start minishift)
sudo virsh stop minishift; sudo undefine minishift; (might be needed if the “delete” command does not fix the issue)
sudo arp -a (on the KVM host to see the IPs of the minishift VM or SSH into the minishift VM)

– To restart the minishift VM (from the KVM host):
itababa@itamint:~$ minishift stop
itababa@itamint:~$ minishift start


INSTALL OSX VENTURA (13.x) ON QEMU/KVM (Linux)

Posted on by
Reply

NOTE: for educational purposes only

GUIDE: Used OSX-KVM with modifications: https://github.com/kholia/OSX-KVM

  • I installed the latest QEMU (compiled from source):
    root@itamint:~/# apt install -y gcc make ninja-build libglib2.0-dev libpixman-1-dev ncurses-dev libusb-dev libusb-1.0-0-dev libusbredirparser1 libusbredirhost1 usbutils
    root@itamint:~# git clone https://github.com/qemu/qemu.git
    root@itamint:~#cd qemu
    root@itamint:~/qemu# ./configure –enable-vde –enable-libusb –prefix=/usr/local –target-list=”i386-softmmu x86_64-softmmu”
    root@itamint:~/qemu# make
    root@itamint:~/qemu# make install
    root@itamint:~/qemu# qemu-system-x86_64 –version
    QEMU emulator version 7.2.50 (v7.2.0-2313-g9832009d9d)

  • Install libvirtd/QEMU
    root@itamint:~# apt update -y
    root@itamint:~# apt install qemu-kvm qemu-system qemu-utils python3 python3-pip libvirt-clients libvirt-daemon-system bridge-utils virtinst libvirt-daemon virt-manager cpu-checker -y
    root@itamint:~# usermod -aG root
    root@itamint:~# newgrp kvm
    root@itamint:~# systemctl enable libvirtd
    root@itamint:~# systemctl start libvirtd
    root@itamint:~# virsh net-start default
    root@itamint:~# virsh net-autostart default

    NOTE: this is to make networking config easy. We won’t use the qemu version 6.x installed above. We will use the much newer version we compiled earlier which we installed in /usr/local/bin/ whereas the one installed by the apt command puts qemu in /usr/bin/. To confirm you are using the right qemu, run “which qemu-system-x86_64”

  • Setup networking so that the VM has network access:
    I have put all the networking and firewall rules in a script which can then be executed with a command such as “sudo /home/itababa/setup_firewall.sh”

    itababa@itamint:~/OSX-KVM$ cat /home/itababa/setup_firewall.sh
    #!/usr/bin/bash

    # create a VDE switch and add a subnet range to it. also add an IP address to the switch as it will be the gateway for connected VMs:
    vde_switch -tap vde0 -daemon
    ip link set dev vde0 up
    ip addr add 10.0.2.1/24 dev vde0
    ip route add 10.0.2.0/24 dev vde0
    echo 1 > /proc/sys/net/ipv4/conf/vde0/proxy_arp

    # NOTE: it is possible to setup DHCP on the switch but I will be using static IPs in this guide

    # Internet access to the VMs (execute on the QEMU hypervisor host) where wlp5s0 is the WAN NIC on the Ubuntu QEMU host:
    echo 1 > /proc/sys/net/ipv4/ip_forward
    iptables -t nat -A POSTROUTING -o wlp5s0 -j MASQUERADE
    iptables -I FORWARD 1 -i vde0 -j ACCEPT
    iptables -I FORWARD 1 -o vde0 -m state –state RELATED,ESTABLISHED -j ACCEPT

    # On the QEMU hypervisor (Linux) host, configure the rules for the ports to be forwarded to the OSX VM:
    iptables -A INPUT -p tcp –dport 2222 -j ACCEPT
    iptables -t nat -A PREROUTING -p tcp –dport 2222 -j DNAT –to-destination 10.0.2.100:22
    iptables -A INPUT -p tcp –dport 5905 -j ACCEPT
    iptables -t nat -A PREROUTING -p tcp –dport 5905 -j DNAT –to-destination 10.0.2.100:5900
    # end of networking script

  • “Install” OSX in the VM using OSX-KVM:

itababa@itamint:~/$ sudo echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs
itababa@itamint:~/$ sudo cp kvm.conf /etc/modprobe.d/kvm.conf

itababa@itamint:~/$ sudo groupadd kvm
itababa@itamint:~/$ sudo groupadd libvirt
itababa@itamint:~/$ sudo groupadd input

itababa@itamint:~/$ sudo usermod -aG kvm $(whoami)
itababa@itamint:~/$ sudo usermod -aG libvirt $(whoami)
itababa@itamint:~/$ sudo usermod -aG input $(whoami)

itababa@itamint:~/$ cd ~
itababa@itamint:~/$ git clone –depth 1 –recursive https://github.com/kholia/OSX-KVM.git
itababa@itamint:~/$ cd OSX-KVM
itababa@itamint:~/OSX-KVM$ pwd
/home/itababa/OSX-KVM

  • Fetch the Ventura Installer (option 6):
    itababa@itamint:~/OSX-KVM$ ./fetch-macOS-v2.py

  • Convert the downloaded BaseSystem.dmg file into the BaseSystem.img file.
    itababa@itamint:~/OSX-KVM$ dmg2img -i BaseSystem.dmg BaseSystem.img

  • Create a virtual HDD image where macOS will be installed.
    itababa@itamint:~/OSX-KVM$ qemu-img create -f qcow2 mac_hdd_ng.img 128G

  • Edit the OpenCore-Boot.sh script and make the following changes:
    – increase the RAM from 4096MB to 8192MB (ALLOCATED_RAM variable)
    – add avx2 to the CPU flags list (MY_OPTIONS variable)
    – change the CPU from Penryn to Cascadelake-Server-noTSX (only one that didn’t cause random freezing after installation and reboot loop during installation from the several I tested. Unfortunately the VM still freezes/hangs consistently between 10 and 12 minutes after startup.)
    – since we are VDE, comment out the default NIC entry starting with “-netdev user” and replace with our VDE line (see below).
    – Below is how the relevant lines looked in my modified OpenCore-Boot.sh file:
    root@itamint:/home/itababa/OSX-KVM# diff OpenCore-Boot.sh OpenCore-Boot.sh.org
    MY_OPTIONS=”+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+xsave,+xsaveopt,check”
    ALLOCATED_RAM=”8192″ # MiB
    -enable-kvm -m “$ALLOCATED_RAM” -cpu Cascadelake-Server-noTSX,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,”$MY_OPTIONS”
    -device virtio-net-pci,netdev=net0,mac=52:54:00:e6:5d:16 -netdev vde,id=net0
  • Run the OpenCore-Boot.sh script (with admin rights as root or using sudo) to create the VM and start the installation (this starts a VNC session listening on localhost:5901 for example. Use Mobaxterm/VNC or putty tunnel+VNC client to access the VNC session)
    itababa@itamint:~/OSX-KVM$ sudo ./OpenCore-Boot.sh
    VNC server running on 127.0.0.1:5901 (output of executing the OpenCore-Boot.sh script)
    NOTE: without “sudo” or “root” user, you will get an error similar to “Could not open vde: No such file or directory”

  • Connnect to the VNC session (using a VNC client):

  • Go through the installation steps (use Disk Utility to erase the target disk, then install Ventura)

  • Optional: Login into the OSX and enabled “remote management” (VNC) and “remote login” (SSH) (from “System Settings …” – it is no longer called “System Preferences”)

  • Optional: if you enabled “remote management” (VNC) and/or “remote login” (SSH), you can connect directly to the OSX VM via a VNC client (using the IP address of the Linux host and port 5901) and/or via SSH (using the IP address of the Linux host and port 2222). The connections work because we included the relevant forwarding rules in the firewall script (above).

  • Naviate to “System Settings” > Network > Ethernet > “Details…”. Assign a static IP address such as 10.0.2.100 (see the setup_firewall.sh script above) to the OSX with subnet mask 255.255.255.0 and Router 10.0.2.1 (IP address of the VDE switch). Also assign the VDE switch IP address as the DNS Server.

  • Optional: Update OpenCore-Boot.sh and comment out the 2 lines that attaches the installer disk to the VM i.e.,
    # -device ide-hd,bus=sata.3,drive=InstallMedia
    # -drive id=InstallMedia,if=none,file=”$REPO_PATH/BaseSystem.img”,format=raw
  • Optional: shutdown OSX. Modify OpenCore-Boot.sh so that it starts the VM “headless” (i.e., running as a background process):
    Comment out the line “-monitor stdio” and add a line below it with “-daemonize” without the double-quotes.

  • Start the VM in the (OSX-KVM directory) with the command: sudo ./OpenCore-Boot.sh

  • Configure the VM to auto-boots from the OSX disk, run the command at the UEFI shell:
    bcfg boot add 0 FS0:\EFI\boot\BOOT_X64.efi “my_boot”

  • Boot to OSX from the UEFI shell with the command sequence (ENTER key after each one): FS0: > cd EFI > cd BOOT > BOOTx64.efi

  • You can now connect to the OSX VM with VNC (Linux host IP and port 5905 in this example) or SSH (Linux host IP and port 2222 in this example).

NOTE: Several methods to attach a USB stick/device (e.g., one containing TimeMachine backups) to a QEMU VM

Option 1: In Linux Mint (Debian/Ubuntu), attached USB storage gets the group “disk”. So add your normal user to that group.
$ sudo usermod -aG disk itababa
$ newgrp disk

– Then add the device nodename of the USB device to the “args” list for QEMU in the OpenBoot-core.sh file (get the device file from the output of “blkid” or “lsblk” (e.g., /dev/sdb)
-hdc /dev/sdb

– you can also use the following 2x lines (equivalent to the single “-hdc /dev/sdb” line above)
-drive id=USBstick,if=none,file=”/dev/sdb”
-device ide-hd,bus=sata.5,drive=USBstick

Note: in OSX, since the USB is “direct-attached”, it is treated as a HDD. To make the volume appear on the desktop, go to “Finder” (menu) > “Settings”. Check/select/enable the “Hard disks” option.

Option 2: Add the USB device’s “location” to the args list for QEMU in the OpenBoot-core.sh file (get the hostbus and hostaddr from the output of the “lsusb” commmand). You must also change the permission on the device files (e.g., “sudo chmod o+rw /dev/bus/usb/002/*”). This operation does not survive a reboot of the physical host or re-insertion of the USB, so if you want it permanent consider UDEV rules):
-device qemu-xhci,id=usbxhci
-usb -device usb-host,hostbus=2,hostaddr=6

For example, in the “lsusb” output below, “Bus” is “hostbus” and “Device” is “hostaddr”
root@itamint:/home/itababa/OSX-KVM# lsusb
…
Bus 002 Device 006: ID 05dc:a838 Lexar Media, Inc. JumpDrive Tough

IMPORTANT NOTE: to prevent the OSX on QEMU from freezing/hanging (after about 10 mins uptime):
– Go to “System Settings …” > Energy Saver > Conserve battery > “Put hard disks to sleep when possible” disabled.
– Go to “System Settings …” > “Lock Screen” > “Turn display off when inactive” to “Never”
– Also, you MUST login to an account, if you leave the system on the login screen, it will freeze/hang.

* If you restore a time machine back along with “System & Networking”, check afterwards that the two settings above have not reverted to their defaults.

  • NOTE: Take snapshots often (e.g., before using “Migration Assistant” to restore TimeMachine backups from another OSX)
    – take an internal snapshot: qemu-img snapshot -c mac_hdd_ng.ss.030823.img mac_hdd_ng.img
    – list internal snapshots: qemu-img snapshot -l mac_hdd_ng.img
    – revert/restore to a snapshot: qemu-img snapshot -a mac_hdd_ng.ss.030823.img mac_hdd_ng.img
  • NOTE: IF “managed device attestation” is configured on your source OSX from which you restored a Time Machine backup, you may notice ACME (or a similar policy enforcement) app is installed and running.
    IF your source OSX is part of a corporate network, unless you decide to connect to the corporate VPN (unlikely for the purpose of this procedure), ACME won’t be able to validate your OSX VM config and may isolate your restored OSX VM (no networking access). To fix this, temporarily stop the ACME app, or permanently remove it:
    – Use the Utilities > “Activity Monitor” > to stop the ACME process
    – Go to Applications folder and delete (“move to trash”) the ACME app shortcut
    – Go to folder ~/Library/ and delete (“move to trash”) the ACME folder

References:
https://wiki.qemu.org/Documentation/Networking
https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-qemu-running.html
https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
https://kb.nmsu.edu/page.php?id=99123 (enable remote management i.e., VNC)
https://setapp.com/how-to/how-to-access-your-mac-remotely
https://www.qemu.org/2017/11/22/haxm-usage-windows/
https://qemu-project.gitlab.io/qemu/system/devices/usb.html
https://askubuntu.com/questions/15570/configure-udev-to-change-permissions-on-usb-hid-device/15643
https://unix.stackexchange.com/questions/141255/give-a-specific-user-permissions-to-a-device-without-giving-access-to-other-user
https://blog.programster.org/qemu-img-cheatsheet
https://www.linux-kvm.org/images/6/65/02x08B-Max_Reitz-Backups_with_QEMU.pdf
https://unix.stackexchange.com/questions/530674/qemu-doesnt-respect-the-boot-order-when-booting-with-uefi-ovmf
https://github.com/sickcodes/Docker-OSX
https://www.cyberciti.biz/faq/kvm-forward-ports-to-guests-vm-with-ufw-on-linux/
https://support.apple.com/guide/deployment/managed-device-attestation-dep28afbde6a/web