Forwarding traffic from external hosts to guest VMs on KVM host

This is so simple but took me forever to find a concise source on the Internet even with the AI results provided by most search engines now.

Background: I set up KVM (qemu/libvirtd) on a Linux host, create a VM on it. I want to be able to SSH (goes for other in-bound traffic such as HTTP as well) from an external host e.g., a separate windows laptop to the guest VM running on the KVM Linux host.

<Windows Laptop> — SSH —> <KVM host> —> <Guest-VM>

The default network created for the guest-VM on KVM is NAT (and goes over the virbr0 interface created on the KVM host during the KVM installation)

To illustrate, I created two guest-VMs (both Linux) and I want to be able to SSH to them from another Windows system in my home network. This means I need to connect to a designated unused port (e.g., 2222 or 2223) on the KVM host via SSH, and have the KVM host forward that connection to the SSHD service running on the guest VM

1/ You MUST add a rule to the LIBVIRT_FWI that allows that traffic otherwise you get something like “Connection refused” for SSH for example. By default, only traffic that is part of on-going session is forwarded, which in effect means you can’t initiate traffic to the VM from outside the KVM host (inserts as rule #1 on top of the chain).
For some reason, the firewall-cmd command works until you restart the system or restart firewalld service (which fails that there is no table/chain with that name), so using the iptables command instead. The “-t filter” can also be omitted since it is the default table (as opposed to others such as nat, mangle, etc)
sudo firewall-cmd –permanent –direct –add-rule ipv4 filter LIBVIRT_FWI 0 -p tcp –dport=22 -m state –state NEW,ESTABLISHED -j ACCEPT
sudo iptables -t filter -I LIBVIRT_FWI -p tcp –dport 22 -m state –state NEW,ESTABLISHED -j ACCEPT

So to make it persistent using systemd methodology, I created a service for the command:
a) Create the file /etc/systemd/system/libvirt_fwi.service with the following content:
[Unit]
Description=Enable libvirtd SSH forwarding to VMs
After=graphical.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -t filter -I LIBVIRT_FWI -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

[Install]
WantedBy=graphical.target


b) Run the commands to create and start the service:
sudo systemctl restart firewalld.service
sudo systemctl restart libvirtd
sudo systemctl daemon-reload
sudo systemctl enable libvirt_fwi.service
sudo systemctl start libvirt_fwi.service

2/ Add the rules to forward the traffic from the KVM host to the target VM. In this example, any incoming traffic to the KVM host on port 2222/tcp is forwarded to a specific VM (e.g., 192.168.124.217) on port 22, and incoming traffic on 2223/tcp is forwarded to a second guest VM with IP 192.168.122.100.
sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 -p tcp –dport 2222 -j DNAT –to-destination 192.168.124.217:22
sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 -p tcp –dport 2222 -j DNAT –to-destination 192.168.124.100:22
sudo firewall-cmd –reload

3/ You can SSH from the KVM host to the VMs using their IP addresses.
These commands allows one to SSH to the VMs from the KVM host itself (the IPs belong to the VMs):
sudo iptables -t nat -A PREROUTING -p tcp –dport 2222 -j DNAT –to-destination 192.168.122.217:22
sudo iptables -t nat -A PREROUTING -p tcp –dport 2223 -j DNAT –to-destination 192.168.122.100:22

4/ SKIP THIS STEP – because KVM already added masquerade rules for the guest network to the NAT table – I am just including it here for reference in case it is needed for other purposes e.g., a hypervisor that requires you to do the NAT set up yourself: Enable NAT of out-going traffic using the KVM host interface the traffic will traverse out. In this example we only masquerade the subnet range used by KVM for the VMs, but you can also masquerade all outgoing traffic on that host interface wlp0s20f3 by leaving out the “-s network” parameter
sudo iptables -t nat -A POSTROUTING -s 192.168.124.0/24 -o wlp0s20f3 -j MASQUERADE

5/ Depending on your Linux distro, there are many ways to make the iptables changes permanent (across reboots). Below is one method for RHEL and its variants (CentOS, Fedora, Rocky, etc):
sudo yum install iptables-services
sudo systemctl enable iptables
sudo systemctl enable ip6tables
sudo systemctl status iptables
sudo systemctl status firewalld
sudo iptables-save > /etc/sysconfig/iptables

5/ To check the IP of your VM (outside from logging into it), you can use “sudo virsh domifaddr <guest-vm>” and the “sudo virsh net-dumpxml default” shows the IP range KVM allocates IP from to the guest VMs.

  • NOTE: You can use this guide to set up KVM on the host, but I didn’t need step 4 because I am not using the ethernet interface/NIC on my KVM host to create a bridge to my home WIFI (and it is very difficult to create a bridge using the WiFi interface – if this is something you need, better to use a hypervisor like VMWare or VirtualBox or Promox, etc.)
  • Making iptables rules permanent: https://www.cyberciti.biz/faq/how-to-save-iptables-firewall-rules-permanently-on-linux/
  • IF you have a multi-NIC KVM host, and your in-coming traffic that you want to forward to the VM is coming on a specific NIC, then you use a slightly different rule (where the SOURCE_IP_ADDRESS is the IP on that NIC):
    sudo firewall-cmd –permanent –direct –add-rule ipv4 nat PREROUTING 0 \
    -s <SOURCE_IP_ADDRESS> -p <PROTOCOL> –dport <PUBLIC_PORT> \
    -j DNAT –to-destination <INTERNAL_IP_ADDRESS>:<INTERNAL_PORT>
  • Use “sudo firewall-cmd –direct –get-all-rules” to display the direct rules

Solaris 10 x86 VM running on Qemu hypervisor on an Ubuntu host

Solaris 10 VM on QEMU running on a Ubuntu host
==============================================

- Login to the Ubuntu host and install QEMU:

root@ip-172-31-23-252:~# apt update -y
root@ip-172-31-23-252:~# apt install -y gcc make ninja-build
root@ip-172-31-23-252:~# wget https://download.qemu.org/qemu-7.2.0.tar.xz
root@ip-172-31-23-252:~# tar xvf qemu-7.2.0.tar.xz
root@ip-172-31-23-252:~# cd qemu-7.2.0/
root@ip-172-31-23-252:~/qemu-7.2.0# apt install libglib2.0-dev
root@ip-172-31-23-252:~/qemu-7.2.0# apt-get install -y libpixman-1-dev
root@ip-172-31-23-252:~/qemu-7.2.0# apt install ncurses-dev
root@ip-172-31-23-252:~/qemu-7.2.0# ./configure
root@ip-172-31-23-252:~# make
root@ip-172-31-23-252:~# make install


- Setup Networking (tap2 will be the NIC of the Solaris VM):
root@ip-172-31-23-252:/wip# ip tuntap add tap2 mode tap  &&  ip link set dev tap2 up


- Setup the host for routing (including Internet access from the Solaris VM). IP address 10.0.2.50 will be assigned to the Solaris VM.
NOTE: it might be better to put this in a start-up script so it "permanent".
echo 1 > /proc/sys/net/ipv4/conf/tap2/proxy_arp
ip route add 10.0.2.50 dev tap2
arp -Ds 10.0.2.50 eth0 pub
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -I FORWARD 1 -i tap2 -j ACCEPT
iptables -I FORWARD 1 -o tap2 -m state --state RELATED,ESTABLISHED -j ACCEPT


- Create VM disk:
root@ip-172-31-23-252:/wip# mkdir -p /wip/sol10x86
root@ip-172-31-23-252:/wip# cd /wip/sol10x86
root@ip-172-31-23-252:/wip/sol10x86# qemu-img create -f qcow2 solaris-disk-amd64.img 10G

root@ip-172-31-23-252:/wip/sol10x86# wget https://ftp.deu.edu.tr/pub/Solaris/iso/Solaris-10,8_11/sol-10-u10-ga2-x86-dvd.iso

- Create and Start the VM booting from the DVD (QEMU launches a VNC server running on 127.0.0.1:5900 for example):
root@ip-172-31-23-252:/wip/sol10x86# qemu-system-x86_64 -name s10qemu -m 4G -cdrom sol-10-u10-ga2-x86-dvd.iso -boot d -hda solaris-disk-amd64.img -net nic,model=e1000,macaddr=7a:30:2a:cd:d0:1c -net tap,script=no,ifname=tap2,downscript=no

- for Solaris 11, boot from CDROM ISO file sol-11_4-text-x86.iso instead of sol-10-u10-ga2-x86-dvd.iso
- use the MAC address of the tap2 NIC ("ip a") in the qemu-system-x86_64 command when starting the VM
- use putty to connect to the Ubuntu host while setting up tunnelling on 5900 (5900 on the Windows client tunneled to 127.0.0.1:5900 on the remote Ubuntu host)
- use real VNC (or tigerVNC) to connect to the VM's console and install solaris 10
- if setting up network, you MUST assign a gateway e.g., 10.0.2.254, otherwise networking doesn't work in the VM
- it will reboot to the install DVD. shutdown the VM ( e.g., CTRL+Z to kill the qemu-system-x86_64 process on the Ubuntu host). For Solaris 11, choose to halt the system at the end of the installation (instead of reboot) then shutdown the VM ( e.g., CTRL+Z to kill the qemu-system-x86_64 process on the Ubuntu host).

- boot VM from O/S disk with networking:
//root@ip-172-31-23-252:/wip# ip tuntap add tap2 mode tap  &&  ip link set dev tap2 up
root@ip-172-31-23-252:/wip# qemu-system-x86_64 -name s10qemu -m 4G -hda solaris-disk-amd64.img -net nic,model=e1000,macaddr=7a:30:2a:cd:d0:1c -net tap,script=no,ifname=tap2,downscript=no

- Connect via VNC client, login to the GUI desktop (choose Java Desktop or CDE) with root and password supplied during install, 

- shutdown the system e.g., from the terminal (Applications > Utilities) run "shutdown -y -g0 -i0"

- take a snapshot of the O/S disk for backup purposes:
# qemu-img create -f qcow2 -b solaris-disk-amd64.img -F qcow2 solaris-disk-amd64.snap.img 5G

- boot VM from O/S disk with networking:
root@ip-172-31-23-252:/wip# qemu-system-x86_64 -name s10qemu -m 4G -hda solaris-disk-amd64.img -net nic,model=e1000,macaddr=7a:30:2a:cd:d0:1c -net tap,script=no,ifname=tap2,downscript=no
NOTE: add the --daemonize to detach the running VM from the host bash session. This is a good way to start the VM once you can access it via SSH.


- Disable the GUI/desktop (graphic environment) since I chose to manage the system remotely using SSH. Note that Solaris 11 boots to run-level 3 so no need to disable the desktop.
# svcadm disable cde-login   
(older Solaris version may use the command "/usr/dt/bin/dtconfig -d")

- Add a user for remote SSH connection (itababa/James007!)
# useradd itababa
# password itababa

- SSH connection from host to Solaris VM (can't use root by default unless you reconfigure SSHD on the Solaris to "PermitRootLogin")
root@ip-172-31-23-252:/wip/sol10x86# ssh -oKexAlgorithms=+diffie-hellman-group-exchange-sha1  -o HostKeyAlgorithms=ssh-rsa itababa@10.0.2.50

- you can make the above connection options permanent by editing the SSH client configuration file on the host/client and adding a line to it such as the following. (Once done, you can connect to the Solaris VM with "ssh itababa@10.0.2.50" for example):
root@ip-172-31-23-252:/wip/sol10x86# vi $HOME/.ssh/config
Host 10.0.2.50
    KexAlgorithms +diffie-hellman-group1-sha1
	HostKeyAlgorithms ssh-rsa



---------- IF YOU SETUP NETWORKING DURING INSTALL SKIP THIS SECTION ---------

- Configure the network interface e1000g0 (in Solaris 11, the interface is called net0 instead of e1000g0)

1. Edit /etc/hostname.e1000g0 and add the IP address and Netmask:
# echo "10.0.2.50 netmask 255.255.255.0" > /etc/hostname.e1000g0

2. Add the IP-address/hostname entry to file /etc/inet/hosts :
# echo "10.0.2.50 `hostname`" >> /etc/inet/hosts

3. Add the default router/gateway:
# echo "10.0.2.254" > /etc/defaultrouter

4. Bring up the NIC in the current running session:
# ifconfig e1000g0 plumb
# ifconfig e1000g0 10.0.2.50 netmask 255.255.255.0 up
# route add default 10.0.2.254 -ifp e1000g0

6. Configure DNS
# echo "nameserver 8.8.8.8" >> /etc/resolv.conf
# vi /etc/nsswitch.conf  (change the line "hosts:     files" to "hosts:     files dns")

------ END OF NETWORKING SETUP IF YOU DIDN'T SETUP IT UP DURING INSTALL ------
========================================================================


Unable to successfully installing Solaris 10/11 SPARC:
- Solaris 10 boots but has no keyboard input so unable to actually install. Solaris 11 doesn't boot at all.

NOTE: For Solaris 11 x86, be sure to add a user account/password aside the root user during the installation as you can't login directly using the root account from the console. 

-------------------------------------------------------------------



-------------------------------------------------------------------

Some commands:

- to disable sound and set the language e.g., launch with:
LC_ALL=C QEMU_AUDIO_DRV=none qemu-system-sparc -m 256 -cdrom ...

- to use terminal install, add the "-nographic" option which prevents QEMU from creating a VNC listener

- Create a snapshot of the AIX O/S disk for backup purposes:
qemu-img create -f qcow2 -b sol10hdisk.qcow2 -F qcow2 sol10hdisk.snap.qcow2 5G

- How to disable the graphic environment, this can be done with that command: "/usr/dt/bin/dtconfig -d" or better with latest Solaris 10: "svcadm disable cde-login".

- Halt the VM:  # shutdown -y -g0 -i0

Keyboard not working as expected in QEMU? When starting QEMU just pass these flags to give yourself a USB keyboard and mouse:
-usb -device usb-mouse -device usb-kbd  -device usb-tablet
You may also try the flag for some ppc/sparc (https://github.com/qemu/SLOF/issues/1 ):  -prom-env 'input-device=hvterm'


// qemu-system-arm -M overo -m 256 -sd ./test.img -clock unix -serial stdio -device usb-mouse -device usb-kbd
// -vga [std|cirrus|vmware|qxl|xenfb|tcx|cg3|virtio|none]


- remove a tap device from a master
root@ip-172-31-23-252:/wip/sol10x86# ip link set dev tap2 nomaster



Links:
https://archive.org/download/sunsolaris10operatingsystem1106x86sparc/SOL_10_1106_SPARC.mdf
https://www.oracle.com/solaris/solaris10/downloads/solaris10-get-jsp-downloads.html
Download Solaris 11:  http://ftp.escuelaing.edu.co/pub/Unix/Solaris/11/
Download Solaris 11:  http://ftp.escuelaing.edu.co/pub/Unix/Solaris/10/
http://ftp.escuelaing.edu.co/pub/Unix/Solaris/10/sol-10-u11-ga-x86-dvd.iso
http://ftp.escuelaing.edu.co/pub/Unix/Solaris/10/sol-10-u11-ga-sparc-dvd.iso
https://ftp.deu.edu.tr/pub/Solaris/iso/Solaris-10,8_11/
https://wiki.qemu.org/Documentation/Platforms/SPARC
https://helpmanual.io/help/qemu-system-sparc/
https://chrispinnock.com/stuff/emulation/running-solaris-in-vms/
https://learn.adafruit.com/build-your-own-sparc-with-qemu-and-solaris
Solaris 10 Network Config
https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator https://download.oracle.com/technetwork/systems/opensparc/OpenSPARCT1_Arch.1.5.tar.bz2