Fusion-IO SSD driver install

English Hardware
#1

I recently bought a Fusion-IO SSD for cheap and have trouble getting it to work on OpenSUSE Leap 15.1, kernel 4.12.14-lp151.28.13-default
The card is an SX300-6400.
Since there are no factory drivers available, I used and followed the instructions here: https://github.com/snuf/iomemory-vsl
Compiling the 3 driver packages using

rpmbuild -ba fio-driver.spec

went without a hitch. I then installed them along with the additional fio-utils from https://link.westerndigital.com/enterprisesupport/software-download.html (seems to be down at the moment)

I then continued with the guide using

sudo cp -r iomemory-vsl/root/usr/src/iomemory-vsl-3.2.15 /usr/src/
sudo mkdir -p /var/lib/dkms/iomemory-vsl/3.2.15/build
sudo ln -s /usr/src/iomemory-vsl-3.2.15 /var/lib/dkms/iomemory-vsl/3.2.15/source
sudo dkms build -m iomemory-vsl -v 3.2.15
sudo dkms install -m iomemory-vsl -v 3.2.15
sudo modprobe iomemory-vsl

Each command seemed to work, at least I did not get any obvious error messages.
But when I try probing the drive using fio-status, I still get

No ioMemory devices found for installed driver. Check if there are ioMemory devices
and the correct ioMemory driver package for those devices has been installed.

I know the drive can be recognized in this workstation, because I tried it on a Windows install.
Is there something in the procedure that needs to be changed for OpenSUSE? Or anything I need to do in addition to that? I’m a bit lost, since I never had to install drivers this way.

#2

Is there anything wrong with the way I asked the question?
Or does really nobody know anything about this?

#3

I do not think you did it wrong. Apart may be from the fact that you do not have the wording “SSD” in your title and “Fusion-IO” may not be well known enough to trigger people.

I will add “SSD” to the title. maybe it helps.

#4

Hi
Can you check the card PCI ID’s via;


/sbin/lspci -nnk | grep A3

Then check if the module you built supports the PCI ID via;


/sbin/modinfo iomemory-vsl
#5

Thanks for chiming in. In the meantime, I decided that it is probably not worth the hassle.
After the last reboot, I got an infinite login loop. None of the workarounds I found seemed to work for me.
Since this is probably related to the Nvidia drivers too (like 90% of the problems I have with Linux PCs), I will switch to an AMD card while I’m at it.
Sorry, I know this might come off like I was trying to waste your time. I’m just fed up with dealing with hardware issues.

#6

Hi
Start a new thread on the Nvidia issue, is it a modern card?

#7

It’s a GTX 960, no idea if that still counts as modern.
For now, I am using the Nouveau driver, but it definitely has its limits.

I probably won’t open a separate thread for this. I rarely end up solving the the problem at hand, and more often break other stuff in the process :shame:

#8

All right, gave it another go, seems like I was just not in the mood last time I posted. Now with the RX570 which works flawlessly so far.

I get no output from /sbin/lspci -nnk | grep A3
I can’t post the full output of /sbin/lspci due to post length restrictions. What am I searching for?

output of /sbin/modinfo iomemory-vsl


/sbin/modinfo iomemory-vsl
filename:       /lib/modules/4.12.14-lp151.28.16-default/updates/iomemory-vsl.ko
license:        GPL
suserelease:    openSUSE Leap 15.1
alias:          pci:v00001AEDd00002001sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001008sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001007sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001006sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001005sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001004sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001003sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001001sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001000sv*sd*bc*sc*i*
depends:        
retpoline:      Y
vermagic:       4.12.14-lp151.28.16-default SMP mod_unload modversions 
parm:           strict_sync:Force strict write flushing on early non-powercut safe cards. (1=enable, 0=disable, -1=auto) Do not change for newer cards. (int)
parm:           disable_msix:N/A (int)
parm:           bypass_ecc:N/A (int)
parm:           groomer_low_water_delta_hpcnt:The proportion of logical space free over the runway that represents 'the wall' (int)
parm:           disable_scanner:For use only under the direction of Customer Support. (int)
parm:           use_command_timeouts:Use the command timeout registers (int)
parm:           use_large_pcie_rx_buffer:If true, use 1024 byte PCIe rx buffer. This improves performance but causes NMIs on some specific hardware. (int)
parm:           rsort_memory_limit_MiB:Memory limit in MiBytes for rsort rescan. (int)
parm:           capacity_warning_threshold:If the reserve space is below this threshold (in hundredths of percent), warnings will be issued. (int)
parm:           enable_unmap:Enable UNMAP support. (int)
parm:           fio_qos_enable:For use only under the direction of Customer Support. (int)
parm:           fio_dev_wait_timeout_secs:Number of seconds to wait for device file creation before continuing. (int)
parm:           iodrive_load_eb_map:For use only under the direction of Customer Support. (int)
parm:           use_new_io_sched:N/A (int)
parm:           groomer_high_water_delta_hpcnt:The proportion of logical space over the low watermark where grooming starts (in ten-thousandths) (int)
parm:           preallocate_mb:The megabyte limit for FIO_PREALLOCATE_MEMORY. This will prevent the driver from potentially using all of the system's non-paged memory. (int)
parm:           exclude_devices:List of cards to exclude from driver initialization (comma separated list of <domain>:<bus>:<slot>.<func>) (string)
parm:           parallel_attach:For use only under the direction of Customer Support. (int)
parm:           enable_ecc:N/A (int)
parm:           rmap_memory_limit_MiB:Memory limit in MiBytes for rmap rescan. (int)
parm:           expected_io_size:Timeout for data log compaction while shutting down. (int)
parm:           tintr_hw_wait:N/A (int)
parm:           iodrive_load_midprom:Load the midprom (int)
parm:           auto_attach_cache:Controls directCache behavior after an unclean shutdown: 0 = disable (cache is discarded and manual rebinding is necessary), 1 = enable (default). (int)
parm:           retire_on_psub:For use only under the direction of Customer Support. (int)
parm:           use_modules:Number of NAND modules to use (int)
parm:           groom_harder_threshold:For use only under the direction of Customer Support. (int)
parm:           make_assert_nonfatal:For use only under the direction of Customer Support. (int)
parm:           enable_discard:For use only under the direction of Customer Support. (int)
parm:           max_md_blocks_per_device:For use only under the direction of Customer Support. (int)
parm:           force_soft_ecc:Forces software ECC in all cases (int)
parm:           max_requests:How many requests pending in iodrive (int)
parm:           global_slot_power_limit_mw:Global PCIe slot power limit in milliwatts. Performance will be throttled to not exceed this limit in any PCIe slot. (int)
parm:           read_pipe_depth:Max number of read requests outstanding in hardware. (int)
parm:           force_minimal_mode:N/A (int)
parm:           auto_attach:Automatically attach drive during driver initialization: 0 = disable attach, 1 = enable attach (default). Note for Windows only: The driver will only attach if there was a clean shutdown, otherwise the fiochkdrv utility will perform the full scan attach, 2 = Windows only: Forces the driver to do a full rescan (if needed). (int)
parm:           external_power_override:Override external power requirement on boards that normally require it. (comma-separated list of adapter serial numbers) (string)
parm:           compaction_timeout_ms:Timeout in ms for data log compaction while shutting down. (int)
parm:           disable_groomer:For use only under the direction of Customer Support. (int)
parm:           include_devices:Whitelist of cards to include in driver initialization (comma separated list of <domain>:<bus>:<slot>.<func>) (string)
parm:           preallocate_memory:Causes the driver to pre-allocate the RAM it needs (string)
parm:           fio_dev_optimal_blk_size:Optimal block size hint for the linux block layer. (int)
parm:           disable_msi:N/A (int)
parm:           scsi_queue_depth:The queue depth that is advertised to the OS SCSI interface. (int)
parm:           numa_node_forced_local:Only schedule fio-wq completion threads for use on NUMA node local to fct-worker (int)
parm:           numa_node_override:Override device to NUMA node binding (array of charp)
parm:           use_workqueue:int

However, I can probe some details of the drive with the fio-utils. E.g.


# fio-firmware 
PCI Address, FW Version, FW Revision
61:00.0, 8.9.5, 20160412



# fio-pci-check 


Root Bridge PCIe 282000 MB/sec needed max


      Bridge 0000:60:03.01 (61-61) (1022:1453)
          Needed 4000 MB/sec Avail 4000 MB/sec


            ioDrive3 0000:61:00.0 (3001) Firmware 118177
#9

Hi
OK, what about the output from;


/sbin/lspci -vvs 61:00.0

Hopefully can glean some more info from above lspci output.

#10 

/sbin/lspci -vvs 61:00.0
61:00.0 Mass storage controller: SanDisk ioMemory FHHL
        Subsystem: SanDisk ioMemory FHHL
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 5
        NUMA node: 6
        Region 2: Memory at ccb00000 (32-bit, non-prefetchable) 
        Region 4: Memory at ccba0000 (32-bit, non-prefetchable) [size=8]
        Region 5: Memory at ccb80000 (32-bit, non-prefetchable) [size=128]
        Expansion ROM at cca00000 [disabled] [size=1]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [c0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000

Another thing I am not quite sure about: I downloaded the additional fio utilities for SLES 12, because to me it seems like the closest match to openSUSE.
And they only have release versions down to 4.2.1, nothing older. I picked the latest, 4.3.5.
The driver from github on the other hand seems to be a version 3.2.16.

The utils only go down to 3.2.16 if I choose the utils for “IoDrive”. But with them, I get no output like I did from fio-firmware.
But the github description states[size=2] “The master branch is based on the 3.2.15 iomemory driver version with several enhancements from the 4.3.4 driver for SX/PX Fusion cards.” So it should be ok for my SX300 card :[/size][/size][/size][/size]

#11

Hi
So normally the last few lines of the output would show the kernel driver in use. So have the module loaded on the system?[/size][/size][/size][/size]

#12

I am not quite sure about that. I thought that was the purpose of sudo modprobe iomemory-vsl. I get no output from this command.
How can I check if it is running? Or what do I actually have to do in order to load the module?

#13

Hi
Check the output from the lsmod command to see it’s present…

If not would need to add a conf file in /etc/modules-load.d/ called iomemory-vsl.conf

In the file just have the oneliner;


iomemory-vsl
#14 
lsmod
Module                  Size  Used by
iomemory_vsl         1298432  0

And from the services manager in Yast

* iomemory-vsl4.service - iomemory-vsl4
   Loaded: loaded (/usr/lib/systemd/system/iomemory-vsl4.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2019-09-25 19:36:45 CEST; 5h 5min ago
 Main PID: 2525 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 9830)
   CGroup: /system.slice/iomemory-vsl4.service


Sep 25 19:36:45 linux-aobn systemd[1]: Starting iomemory-vsl4...
Sep 25 19:36:45 linux-aobn iomemory-vsl4[2525]: iomemory-vsl4 is not enabled in the config '/etc/sysconfig/iomemory-vsl4'
Sep 25 19:36:45 linux-aobn systemd[1]: Started iomemory-vsl4.

The service is listed as start: on boot; status: Active(Exited)

Creating the .conf file did not change the current behavior after a reboot.

#15

Adding to my last post:

/etc/sysconfig/iomemory-vsl4 looks like this

# -*- sh -*-


# If ENABLED is not set (non-zero) then iomemory-vsl4 init script will not be
# used.
#ENABLED=1


# If using systemd to start, set this to 1
#SYSTEMD_ENABLED=1


# Timeout in seconds when waiting for an operation
TIMEOUT=15


# Set VERBOSE=1 for more information, VERBOSE=0 for quiet mode.
VERBOSE=1


# Set KILL_PROCS_ON_UMOUNT=1 to kill any active processes
# that might prevent unmounting of the filesystems.
KILL_PROCS_ON_UMOUNT=1


# Set SWAP=1 if you are using an ioMemory device as a swap space.
# Configure what to use as swap in /etc/fstab.
#NOTE: Using ioMemory devices as swap space results in a net performance loss
#compared to bypassing the page cache and allowing applications to access the
#ioMemory devices directly. If you do use an ioMemory device as swap, you must
#preallocate memory using the preallocate_memory VSL module parameter.
#See the VSL User Guide for more information.
SWAP=0


# Any special module parameters for iomemory-vsl4: "modinfo iomemory-vsl4"
# for a listing of driver parameters.
IOMEMORY_MOD_OPTS=""


# An IFS separated list of md arrays to start once the driver is
# loaded.  Arrays should be configured in the mdadm.conf file.
# Example: MD_ARRAYS="/dev/md0 /dev/md1"
MD_ARRAYS=""


# An IFS separated list of LVM volume groups to start once the driver is
# loaded.  Volumes should be configured in lvm.conf.
# Example: LVM_VGS="/dev/vg0 /dev/vg1"
LVM_VGS=""


# Autodetect LVM volume groups to stop at driver unload.
LVM_DETECT=0


# An IFS separated list of VXVM diskgroups to import once the driver is
# loaded.
# Example: VXVM_DGS="oradg1 oradg2"
VXVM_DGS=""


# An IFS separated list of mount points to mount once the driver is
# loaded.  These mount points should be listed in /etc/fstab with
# "noauto" as one of the mount options.
# Example /etc/fstab:
#/dev/fioa      /mnt/fioa       ext3    defaults,noauto 0 0
#/dev/fiob      /mnt/xyz        ext3    defaults,noauto 0 0
# Example: MOUNTS="/mnt/fioa /mnt/xyz"
MOUNTS=""


# Set FSTRIM_ON_MOUNT=1 to run fstrim after successful mount of
# volumes listed in MOUNTS above
FSTRIM_ON_MOUNT=0
#16

Hi
To me it looks like all is working, however the card you have doesn’t appear to have it’s PCI ID (what ever it is) added to the module aliases…

Can you post the output from;


/sbin/lspci -vns 61:00.0
#17 
# /sbin/lspci -vns 61:00.0
61:00.0 0180: 1aed:3001
        Subsystem: 1aed:3001
        Flags: bus master, fast devsel, latency 0, IRQ 5, NUMA node 6
        Memory at ccb00000 (32-bit, non-prefetchable) [size=512]
        Memory at ccba0000 (32-bit, non-prefetchable) [size=8]
        Memory at ccb80000 (32-bit, non-prefetchable) [size=128]
        Expansion ROM at cca00000 [disabled] [size=1]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

[/size][/size][/size][/size]

#18

Hi
OK, so the iomemory-vsl module doesn’t have 3xxx series in the the alias list.


alias:          pci:v00001AEDd00002001sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001008sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001007sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001006sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001005sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001004sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001003sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001001sv*sd*bc*sc*i*
alias:          pci:v00001AEDd00001000sv*sd*bc*sc*i*

(Your output)
Subsystem: 1aed:3001

I the output from the modinfo command, was the anything in the ‘parm’ entries about pci ids?

#19

Hi
I see it in the pci.h code, so looks more like a bug with the driver…


include/fio/port/pci.h:#define GEN3_DEVICEID                       0x3000
#20

Sorry for the late reply, work took more time than it should have.
So in your opinion, there is nothing more I (aka you) could try? Other than submitting an issue on github?