IB + ethernet both operating at the same time?

The only way I know how to get infiniband (mellanox card) running under YaST is to select the Wicked service and configure the indicated IB port. This works well. But I can’t seem to get any of the server’s ethernet ports (em1 - em4) to work, either with a static IP address or DHCP when using Wicked. (the ethernet is connected to a standard router, and IB to a separate IB switch - they are separate subnets) On the other hand, it’s easy to get em1 to work fine, I just choose Network Manager - but NM can’t seem to set up the IB port correctly.

Now I’ve run into a situation where I need both IB (say, p1p1 and em1) running at the same time. Can someone help me with what I’m missing here?

Thank You!!!

PS: This was the same for 15.0, 15.1, and 15.2

Hi
What does the ip output show (munge ip addresses!);


ip addr
ip route

Thanks for the reply, Malcom!


Network Manager
===============
~> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
<snip!>
4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/24 brd xxx.xxx.xxx.xxx scope global noprefixroute dynamic em3
       valid_lft 604082sec preferred_lft 604082sec
    inet6 xxxx::xxxx:xxxx:xxxx:xxxx/xx scope link noprefixroute
       valid_lft forever preferred_lft forever
<snip!>
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
~>
~> ip route
default via xxx.xxx.xxx.xxx dev em3 proto dhcp metric 101
xxx.xxx.xxx.xxx/xx dev em3 proto kernel scope link src xxx.xxx.xxx.xxx metric 101
~>

YaST - Switched to Wicked
=========================
~> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
<snip!>
 4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f0:4d:a2:02:d3:6d brd ff:ff:ff:ff:ff:ff
    inet 198.162.1.186/24 brd 198.162.1.255 scope global noprefixroute dynamic em3
       valid_lft 603817sec preferred_lft 603817sec
<snip!>
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/xx brd xxx.xxx.xxx.xxx scope global p1p1
       valid_lft forever preferred_lft forever
~>
~> ip route
default via xxx.xxx.xxx.xxx dev em3 proto dhcp metric 101
xxx.xxx.xxx.xxx/xx dev p1p1 proto kernel scope link src xxx.xxx.xxx.xxx
xxx.162.xxx.xxx/xx dev em3 proto kernel scope link src xxx.xxx.xxx.xxx metric 101
~>

Power-cycled
============
~> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
<snip!>
4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
 <snip!>
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/xxx brd xxx.xxx.xxx.xxx scope global p1p1
       valid_lft forever preferred_lft forever
~>
~> ip route
xxx.xxx.xxx.xxx/xxx dev p1p1 proto kernel scope link src xxx.xxx.xxx.xxx
~>

Hi pattiM. With NM both interfaces appear to be active, but you have no ip address assigned for p1p1. Did you try to configure that interface explicitly via the connection editor?

YaST - Switched to Wicked
=========================
~> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
<snip!>
 4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f0:4d:a2:02:d3:6d brd ff:ff:ff:ff:ff:ff
    inet 198.162.1.186/24 brd 198.162.1.255 scope global noprefixroute dynamic em3
       valid_lft 603817sec preferred_lft 603817sec
<snip!>
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/xx brd xxx.xxx.xxx.xxx scope global p1p1
       valid_lft forever preferred_lft forever
~>
~> ip route
default via xxx.xxx.xxx.xxx dev em3 proto dhcp metric 101
xxx.xxx.xxx.xxx/xx dev p1p1 proto kernel scope link src xxx.xxx.xxx.xxx
xxx.162.xxx.xxx/xx dev em3 proto kernel scope link src xxx.xxx.xxx.xxx metric 101
~>

Here you appear to have both interfaces configured and working as desired?

Power-cycled
============
~> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
<snip!>
4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
 <snip!>
6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/xxx brd xxx.xxx.xxx.xxx scope global p1p1
       valid_lft forever preferred_lft forever
~>
~> ip route
xxx.xxx.xxx.xxx/xxx dev p1p1 proto kernel scope link src xxx.xxx.xxx.xxx
~>

Only p1p1 active? Can you bring the em3 interface up manually with ip command?

ip link set em3 up

Check addresses and routing again.

Hi Deano - Thanks for the reply!

I’ll try your command to bring em up manually when using Wicked.
My examples were to show:

  • Wicked: em doesn’t work but p1p1 does work
  • NM: the other way around (em* works, but NM can’t can’t find what it needs to complete the set-up of p1p1)

In my perception, Wicked is ‘old-school,’ sort of - I enter IP/SNM statically and it works fine for p1p1 (IB). The IB LAN is just a Mikrotik switch (no router) between some compute nodes, but the switch has a pre-setup IP range, so I assign appropriate IP/SNM to network nodes manually (with Wicked).
(Here’s the switch: https://mikrotik.com/product/crs309_1g_8s_in )

I have no idea what Network Manager is doing behind-the-scenes. There’s only one input I recognize in NM, and that’s DNS, which I have never needed to enter.
(but there’s no DNS on the IB switch LAN)

I tried setting up em* with Wicked using either DHCP or assigned IP address (I got the IP address by letting NM set up em*, then copying the IP address from ifconfig)
I also tried the options for network-start at boot time or on-cable-connection. it seems like that should have worked.

Of course - this morning the computer is hanging at the splash-screen (right after Wicked setup) :beat-up:

But ALL my computers have shown this duality between Wicked and NM - so this non-boot is just a typical computer morning-PITA

EDIT: OK, it’s booted…

There’s nothing in the NM interface to put IP/SNM/hostname? maybe I missed something… I’ll click more. Oh, maybe there it is…

NetworkManager has a connection editor used to configure network interfaces as desired.

The openSUSE guide is Gnome-orientated, but the KDE GUI is essentially the same…
https://doc.opensuse.org/documentation/leap/reference/html/book-opensuse-reference/cha-nm.html#sec-nm-configure

Alternatively, there is ‘nmcli’ if you’re comfortable using the CLI approach.

man nmcli

One of many guides…

*- Wicked: em doesn’t work but p1p1 does work

  • NM: the other way around (em* works, but NM can’t can’t find what it needs to complete the set-up of p1p1)*

OK - after the booting thing this AM, I noticed both em* and p1p1 were up and working using Wicked. :open_mouth:

I had set Wicked to DHCP and IP4 (…did I never do that combination before? It seems so straightforward)
Still need to test, but “apparently” a reboot was all that was needed? (even with the option to activate-upon-cable-connect).

side question: Is IP6 needed for reliable Internet data-connections these days? (e.g., downloading binary data from NOAA weather-data servers)

Thanks for the feedback, I guess I get stuck on stuff sometimes if I can’t get feedback, so thank you very much.
I can’t believe I wasted your guyz time with something that should have been so simple and straightforward. :embarrassed:rotfl!

I’m testing repeatedly right now - but it looks solved with Wicked (simple/robust is good).

Geeze, EVERYTHING has been hacked, hasn’t it? Sakari, Accelion, Spectre, ESS Voting Machines, MS Exchange, Signed PDFs, SolarWinds, BrowserExtensions…
…makes me wonder how long the *nix kernels will be safe!

They’re not safe… :stuck_out_tongue: