Setting up networking so I can pxeboot from another machine

English Virtualization
1

Hi all,

I have a virtual machine configured through virt-manager that I want to be able to use UEFI firmware (I have this working), but want to boot it off my DHCP and PXE servers which are on different machines on the network.

I have setup a bridge using the following commands :

nmcli con add ifname br0 type bridge con-name br0
nmcli con add type bridge-slave ifname enp2s0 master br0

When I show the connections I get

traken{dgnemu}# nmcli connection show
NAME                 UUID                                  TYPE      DEVICE 
Wired connection 1   2f23ef92-c7da-3fcd-9e70-2f97016fd249  ethernet  enp2s0 
br0                  9e02493d-7ec2-4697-b3a8-4eae3d5b4ba5  bridge    br0    
lo                   9a2bff71-0bd9-47f3-b181-0c260d52f631  loopback  lo     
bridge-slave-enp2s0  3fdf356c-cc90-44d8-87c8-9b046175a256  ethernet  --

In my virtual machine’s virtual network devices, I have set the Network source to ‘bridge device’, device name to ‘br0’ for device mode I have tried both virtio and e1000, neither seemed to work :frowning:

if I start the virtual machine and list the connections I now get :

traken{dgnemu}# nmcli connection show
NAME                 UUID                                  TYPE      DEVICE 
Wired connection 1   2f23ef92-c7da-3fcd-9e70-2f97016fd249  ethernet  enp2s0 
br0                  9e02493d-7ec2-4697-b3a8-4eae3d5b4ba5  bridge    br0    
lo                   9a2bff71-0bd9-47f3-b181-0c260d52f631  loopback  lo     
vnet11               bd10600f-7f82-4bbb-ac79-8fd9a41f7e51  tun       vnet11 
bridge-slave-enp2s0  3fdf356c-cc90-44d8-87c8-9b046175a256  ethernet  --

But a few seconds later the vmnet connection disappears. Looking at the log files on my DHCP server, it never receives any packets from my virtual machine, which suggests that there is something not right about the bridge.

ifconfig displays

br0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 6e:02:8e:41:f9:15  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 2  overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.12  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::1ac8:e307:6ad7:ac43  prefixlen 64  scopeid 0x20<link>
        ether 58:47:ca:7e:69:1a  txqueuelen 1000  (Ethernet)
        RX packets 1599271  bytes 1794050800 (1.6 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 633135  bytes 1223014304 (1.1 GiB)
        TX errors 0  dropped 0  overruns 0  carrier 0  collisions 0

vnet11: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:fe3f:d134  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:3f:d1:34  txqueuelen 1000  (Ethernet)
        RX packets 19  bytes 5722 (5.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 111  bytes 30270 (29.5 KiB)
        TX errors 0  dropped 0  overruns 0  carrier 0  collisions 0

(lo, and unused wireless connection removed for brevity).

Any idea what the problem is?

Cheers.

Phill.

2

At a quick glance, the bridge doesn’t appear to be fully configured. enp2s0 is still using the “Wired connection 1” profile, while bridge-slave-enp2s0 isn’t attached to any device. That suggests the physical NIC isn’t actually a member of br0.

In a working bridge configuration, the IP address should normally be assigned to the bridge, not the ethernet interface.

Show the output from

nmcli connection show --active
ip addr
ip link
3

@phillhs further to @deano_ferrari’s query, also ensure ipv4/ipv6 addressing is disabled on the interface as well as ip forwarding enabled;

nmcli connection show --active
NAME           UUID                                  TYPE      DEVICE  
wired  cbe823f6-2793-4013-9420-0dfb0c1fd8b7  ethernet  ens4f0  
iSCSI          87482179-54a7-4e77-942f-3f9cb02f1f35  ethernet  ens4f1  
br0 port 1     0eb64d85-b621-4f0c-a383-f72d8796e415  ethernet  enp12s0 
Bridge0        69e43a4b-781a-4a2c-a247-3d6b10dafe40  bridge    br0     
lo             f286c3ab-1d4d-4036-9cd2-125de10c0b9f  loopback  lo    

7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:e6:df:3d:3a:ab brd ff:ff:ff:ff:ff:ff

cat /etc/sysctl.d/10-frowarding.conf
 
net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.disable_ipv6 = 1

I don’t use IPv6…

I have no ip address assigned to a bridge device, that is controlled by the virtual machine… (I have multiple vm’s and multiple bridges in use at times)

cat br0\ port\ 1.nmconnection 

[connection]
id=br0 port 1
uuid=0eb64d85-b621-4f0c-a383-f72d8796e415
type=ethernet
controller=br0
interface-name=enp12s0
port-type=bridge

[ethernet]
mac-address=00:0A:CD:24:D2:D2

[bridge-port]

cat Bridge\ connection\ 1.nmconnection 

[connection]
id=Bridge0
uuid=69e43a4b-781a-4a2c-a247-3d6b10dafe40
type=bridge
interface-name=br0

[ethernet]

[bridge]

[ipv4]
method=disabled

[ipv6]
addr-gen-mode=stable-privacy
method=disabled

[proxy]
4

I have been doing some iSCSI rootfs for ARM SBC’s and PXE boot (RPi flavor) and as a quick and dirty method I did boot the same thing with a generic distro kernel by using ‘Direct kernel boot’ under Boot options in Virt-Manager.
A more clean method would be also to PXE load that distro kernel by using the virtual Tianocore EDK2 UEFI aarch64 firmware. It should be possible and seems the goal in this topic.

But if the VM should get its IP config via DHCP, make sure a persistent MAC address is known in the router so known IP address can be allocated. That MAC address can be copied from the .xml file to the router .

For this to work, I have the host machine using br0 by default with its main ethernet port enslaved. In the br0 connection profile, I put a
cloned-mac-address=
under section [ethernet] with the HW MAC address of the host main ethernet port. This way, the host also gets allocated the same IP address in case no bridge is used but just the main ethernet port, even on first boot after install of a OS or boot from live CD-ROM or so.

At least IP forwarding is not needed; Bridging is OSI layer 2 and IP forwarding is OSI layer 3.

I find it easiest to use text tool nmtui to create a br0 and enslaved port(s), also easy to set them to autoconnect and disable autoconnect on that 'Wired connection 1 ’ or even just delete it.
So as minimum, the host should have a 2 devices, which I prefer to name ‘br0’ and ‘enp1s0-br0-slave’. The latter one defacto has no IP address if all is setup with nmtui. The cloned-mac-address I usually enter with text editer in the .nmconnecion file and then do nmcli con reload or just reboot.

For every Virtual Machine started, a device vnet? is then added, easy to see as overview by doing modern ‘ip a’ (ifconfig is legacy).