multipathd does not start

Hello

Multipathd does not start on HP ProLiant DL380 G4 Packaged Cluster with openSUSE 12.1 installed. Each server is equipped with two Emulex Corporation LP952 Fibre Channel Host Adapters:

clam2:/home/amadrits # lspci
...
06:01.0 Fibre Channel: Emulex Corporation LP952 Fibre Channel Host Adapter (rev 01)
06:02.0 Fibre Channel: Emulex Corporation LP952 Fibre Channel Host Adapter (rev 01)

There is a storage with two MSA1000 RAID controllers and five arrays defined connected to the Fibre Channel Host Adapters:

clam2:/home/amadrits # cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: COMPAQ   Model: MSA1000          Rev: 4.24
  Type:   RAID                             ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 00 Lun: 01
  Vendor: COMPAQ   Model: MSA1000 VOLUME   Rev: 4.24
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 00 Lun: 02
  Vendor: COMPAQ   Model: MSA1000 VOLUME   Rev: 4.24
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 00 Lun: 03
  Vendor: COMPAQ   Model: MSA1000 VOLUME   Rev: 4.24
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 00 Lun: 04
  Vendor: COMPAQ   Model: MSA1000 VOLUME   Rev: 4.24
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi0 Channel: 00 Id: 00 Lun: 05
  Vendor: COMPAQ   Model: MSA1000 VOLUME   Rev: 4.24
  Type:   Direct-Access                    ANSI  SCSI revision: 04
...

This is my multipath.conf:

defaults {
	udev_dir		/dev
	polling_interval	5
	selector		"round-robin 0"
	path_grouping_policy	failover
	getuid_callout		"/lib/udev/scsi_id --whitelisted --device=/dev/%n"
	prio			const
	prio_args		""
	path_checker		directio
	rr_min_io		1000
	rr_weight		uniform
	failback		manual
	no_path_retry		fail
	user_friendly_names	no
}

blacklist {
	devnode	"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode	"^hd[a-z]"
	devnode	"^dcssblk[0-9]*"
	devnode "^cciss!c[0-9]d[0-9]*"
}

multipaths {
	multipath {
		wwid			3600805f300033090a3f91046b5bf0008
		alias			ArrayA
	}
	multipath {
		wwid			3600805f300033090a5197018b4d80006
		alias			ArrayB
	}
	multipath {
		wwid			3600805f300033090a6b94fc304840004
		alias			ArrayC
	}
	multipath {
		wwid			3600805f300033090ac29602fad360007
		alias			ArrayD
	}
	multipath {
		wwid			3600805f300033090ae495ff1514d0005
		alias			ArrayE
	}
}

devices {
	device {
		vendor			"COMPAQ  "
		product			"MSA1000         "
		path_grouping_policy	failover
		path_checker		tur
		rr_weight		priorities
	}
}

boot.multipath starts fine, but multipathd does not:

clam2:/home/amadrits # /etc/init.d/boot.multipath start
redirecting to systemctl
clam2:/home/amadrits # /etc/init.d/boot.multipath status
redirecting to systemctl
multipath.service - LSB: Create multipath device targets
          Loaded: loaded (/etc/init.d/boot.multipath)
          Active: active (exited) since Thu, 17 May 2012 18:20:01 +0200; 4s ago
         Process: 3711 ExecStart=/etc/init.d/boot.multipath start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/multipath.service
clam2:/home/amadrits # /etc/init.d/multipathd start
redirecting to systemctl
Job failed. See system logs and 'systemctl status' for details.
clam2:/home/amadrits # /etc/init.d/multipathd status
redirecting to systemctl
multipathd.service - LSB: Starts multipath daemon
          Loaded: loaded (/etc/init.d/multipathd)
          Active: failed since Thu, 17 May 2012 18:20:27 +0200; 10s ago
         Process: 3968 ExecStart=/etc/init.d/multipathd start (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/multipathd.service

After starting boot.multipath the desired device nodes are created:

clam2:/home/amadrits # ls /dev/mapper/
ArrayA  ArrayA_part1  ArrayB  ArrayB_part1  ArrayC  ArrayC_part1  ArrayD  ArrayD_part1  ArrayE  ArrayE_part1  control

After all the following kernel modules are loaded:

clam2:/home/amadrits # lsmod
Module                  Size  Used by
dm_round_robin         12643  5 
scsi_dh_hp_sw          12724  5 
dm_multipath           26806  6 dm_round_robin
scsi_dh                14344  2 scsi_dh_hp_sw,dm_multipath
dm_mod                 84713  21 dm_multipath
ip6t_LOG               12847  5 
xt_tcpudp              12788  2 
xt_pkttype             12456  3 
ipt_LOG                12880  5 
xt_limit               12541  10 
af_packet              26605  0 
ip6t_REJECT            12809  3 
nf_conntrack_ipv6      13936  3 
nf_defrag_ipv6         18017  1 nf_conntrack_ipv6
ip6table_raw           12603  1 
xt_NOTRACK             12456  4 
ipt_REJECT             12512  3 
iptable_raw            12598  1 
iptable_filter         12706  1 
ip6table_mangle        12652  0 
nf_conntrack_netbios_ns    12585  0 
nf_conntrack_broadcast    12541  1 nf_conntrack_netbios_ns
nf_conntrack_ipv4      14319  3 
nf_defrag_ipv4         12649  1 nf_conntrack_ipv4
ip_tables              18302  2 iptable_raw,iptable_filter
xt_conntrack           12664  6 
nf_conntrack           79552  6 nf_conntrack_ipv6,xt_NOTRACK,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_conntrack_ipv4,xt_conntrack
ip6table_filter        12711  1 
ip6_tables             18432  4 ip6t_LOG,ip6table_raw,ip6table_mangle,ip6table_filter
x_tables               26199  16 ip6t_LOG,xt_tcpudp,xt_pkttype,ipt_LOG,xt_limit,ip6t_REJECT,ip6table_raw,xt_NOTRACK,ipt_REJECT,iptable_raw,iptable_filter,ip6table_mangle,ip_tables,xt_conntrack,ip6table_filter,ip6_tables                                                                                                                                                                                                         
cpufreq_conservative    13525  0
cpufreq_userspace      12973  0
cpufreq_powersave      12554  0
acpi_cpufreq           14206  0
mperf                  12603  1 acpi_cpufreq
microcode              22828  0
tg3                   149581  0
hpilo                  17204  0
sg                     35770  0
sr_mod                 22048  0
shpchp                 32325  0
pci_hotplug            34483  1 shpchp
iTCO_wdt               17575  0
iTCO_vendor_support    13502  1 iTCO_wdt
floppy                 64406  0
e752x_edac             18180  0
cdrom                  46732  1 sr_mod
pcspkr                 12614  0
edac_core              46858  1 e752x_edac
container              12735  0
button                 13661  0
autofs4                37842  2
uhci_hcd               36055  0
rtc_cmos               18167  0
ehci_hcd               60362  0
usbcore               190567  3 uhci_hcd,ehci_hcd
thermal                18377  0
processor              43803  1 acpi_cpufreq
thermal_sys            20200  2 thermal,processor
hwmon                  12936  1 thermal_sys
ata_generic            12809  0
ata_piix               30541  0
libata                205032  2 ata_generic,ata_piix
lpfc                  577477  5
scsi_transport_fc      52158  1 lpfc
scsi_tgt               19434  1 scsi_transport_fc
cciss                 114444  6

The following messages are reported in the syslog:

clam2:/home/amadrits # less /var/log/messages
... 
May 17 18:20:17 clam2 multipathd[3968]: Starting multipathdWARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
May 17 18:20:17 clam2 kernel:   868.970351] multipathd (3978): /proc/3978/oom_adj is deprecated, please use /proc/3978/oom_score_adj instead.
May 17 18:20:27 clam2 multipathd[3968]: ..failed
May 17 18:20:27 clam2 systemd[1]: multipathd.service: control process exited, code=exited status=1
May 17 18:20:27 clam2 systemd[1]: Unit multipathd.service entered failed state.
...

This is the output of dmesg:

clam2:/home/amadrits # dmesg
...
  852.160310] device-mapper: uevent: version 1.0.3
  852.160470] device-mapper: ioctl: 4.21.0-ioctl (2011-07-06) initialised: dm-devel@redhat.com
  852.222154] device-mapper: multipath: version 1.3.0 loaded
  852.289981] sd 0:0:0:1: hp_sw: attached to active path
  852.290126] sd 0:0:0:2: hp_sw: attached to active path
  852.290269] sd 0:0:0:3: hp_sw: attached to active path
  852.290411] sd 0:0:0:4: hp_sw: attached to active path
  852.290546] sd 0:0:0:5: hp_sw: attached to active path
  852.290553] hp_sw: device handler registered
  852.304584] device-mapper: multipath round-robin: version 1.0.0 loaded
  868.970351] multipathd (3978): /proc/3978/oom_adj is deprecated, please use /proc/3978/oom_score_adj instead.

Is there anyone who can say me, what is going wrong and how I can fix it.

Let me know, if you use more informations.

Thanks in advance.

In advance I have to mention, that I had to change to Tumbleweed for some other reasons, before I solved this problem with the multipath-tools.

I eventually solved the problem by removing the multipath-tools breaking dependencies and downloading the source code from multipath-tools:Home.

After making and installing the multipath-tools I had to copy and rename the following files from the directory containing the source code manually:

kpartx/kpartx.rules → /etc/udev/rules.d/70-kpartx.rules
multipath/multipath.rules → /etc/udev/rules.d/71-multipath.rules
multipathd/multipathd.init.suse → /etc/init.d/multipathd

finally I had to adjust the permissions of the file multipathd.

even more I have to mention, that you should not update multipath-tools with Yast, other ways you have to reinstall multipath-tools from source.

lpfc module that you have?. I really does not work LP592:

MODINFO LPFC
filename: /lib/modules/3.0.76-0.11-pae/kernel/drivers/scsi/lpfc/lpfc.ko
version: 0:8.3.7.10.6p
author: Emulex Corporation - tech.support@emulex.com
description: Emulex LightPulse Fibre Channel SCSI driver 8.3.7.10.6p
license: GPL

dmesg
10.447949] scsi2 : Emulex LP952 PCI Fibre Channel Adapter on PCI bus 06 device 10 irq 26
16.888087] lpfc 0000:06:02.0: 0:0442 Adapter failed to init, mbxCmd x88 CONFIG_PORT, mbxStatus x12 Data: x0
19.060084] lpfc 0000:06:02.0: 0:0442 Adapter failed to init, mbxCmd x88 CONFIG_PORT, mbxStatus x12 Data: x0
19.060197] lpfc 0000:06:02.0: 0:0445 Firmware initialization failed
19.060294] lpfc 0000:06:02.0: 0:1477 Failed to set up hba
19.161712] lpfc 0000:06:02.0: PCI INT A disabled

have settings for the module lpfc? lpfc.conf