Sata card erratic behaviour & failure - Marvell 88SE9128 (9123?) Chipset

English Hardware
#1

Dear all,

maybe someone here has a better idea what is happening than me…
In August 2014 I set up a Home Server (Dell T20) which has 4 internal SATA ports (Intel) where I wanted 1 OS drive (500GB, 2.5") an 4 3TB drives with RAID6 for data - so I needed an extra SATA port, supplied via a PCIe SATA card.
A SiI based card is not detected by the sever (card issue, server issue?) - while a Marvell based card work in that it is recognised by the server, so far so good.

Over the time, I noticed that one disk would drop from the array - the disk could sometimes be re-added, sometimes it failed immediately again - erratically, finally having enough I’m trying to figure out where the issue lies…
This issue has occurred on OpenSuSE 13.2 and persists in Leap 42.1.
(I have also contacted TechSupport, but I sort of hope that there may be some people with personal experience here.)

  1. The card has a chip that is labelled as 88SE9128 - a DOS based BIOS update utility as well as Linux however identify the chip as a 88SE9123 - the card is also sold as having an 88SE9128.

Label on the Chip:
88SE9128-NAA2
PPH7290.04
1322 B1P
TW
(Startech card with 2 Sata 3 Ports, PEXSAT32)

  1. Using dd to zero fill the drive (outside of mdadm), the process initially starts with around 50-54MB/s and then drops, generally to 10-20MB/s, I have seen it as low as 500KB/s.
    The process WILL fail - however it has written 1GB, 2.5GB, 20GB before it fails with no discernible pattern.
    This happens on a 3TB drive, but also on a 2TB drive that I tested.
  2. After failing, the drive will not show up in the partitioner in Yast - when querying the SMART data, it will now report a size of 600PB until the server is rebootet - after which the drive shows again and the SMART data comes back with no problems.
  3. The SMART data for the drive is fine - the drive itself is most definitely fine too (recertified return) - it also passes a SeaTools test. In case anybody wants to know, it is a 7200rpm, 3TB Seagate Barracuda

Tech Support Sugggestion:

Inquiring with StarTech, it was suggested that I should try the card in another computer - the only suitable machine for this is my Windows PC (Windows 7, 64bit) - where I had to install the dedicated Marvell Driver.

  • With it, the card seemingly works fine, at least it works on both the 2TB drive doing an 8GB CrystalDisk benchmark and on the 3TB drive doing a 16GB CrystalDisk benchmark.
  • This time the drives got around 170-192MB/s sequential read and a good 150MB/s sequential write.
    (This is in line with the reported speed when looking at an array rebuild on the internal Intel SATA ports in the server.)

Searching around, it seems that people consider the Marvell chips evil - well, unfortunately, I’m not in a position to purchase a several hundred pounds RAID card.

  • One person apparently had issues and resolved them by limiting the rebuild speed - I don’t think this applies as dd slows down rapidly before reporting an I/O error.
    (Link: http://tobias.kleemann.net/2013/08/solved-linux-software-raid-1-fallover-problem/ )
  • Searching around some more, it is suggested that the Marvell chip corrupts drives larger than 3TB - however this has only been reported by a Russian user (a big thanks to online translators) and is echoed in one forum once and a comment on SATA/SAS card candidates. It is suggested that the problem is an overflow problem in the controller - however I cannot evaluate the accuracy of that statement - it was also reported with a firmware slightly older than the latest version which is 1.0.0.38.
    (Link: http://blog.zorinaq.com/?e=10 - see post by Artem Ryabov and here is a more detailed analysis in Russian http://ru-root.livejournal.com/2659575.html )

So, any ideas, suggestions?
IF there is a driver problem in the controller, this could account for the 3TB drives - BUT would not explain why the 2TB drive also fails with dd. The slow write speed in Linux is also peculiar - as Windows is a little over 3 times as quick and normally it is Windows that is slow…
It is no the SATA cable as the same cable was used in Windows - and I also tried swapping it to no avail.

#2

I guess nobody has any ideas? :frowning:

I also posted the query to Uni Stackexchange to see if a wider audience could help - http://unix.stackexchange.com/questions/244419/marvell-88se9128-9123-sata-card-weird-behaviour-opensuse/245087

After some time I received an initial suggestion that the Native Command Queuing queue depth could be to blame - the default is 31.
However setting it to 1 seems to bring no improvement, zero filling the drive with dd still fails. Just in case filling the drive itself is the problem, I tried filling a partition that spans the entire drive instead - it still fails.

Again, the failure point is pretty erratic - 639MB is, 3.9GB in, 6.9GB in…

So while it looked like a promising lead, it turned out not to lead to a solution :(.

#3

OK, some more suggestions from Stackexchange - but no resolution so far:

Basically the respondent on Stackexchange suggested that Native Command Queueing is to blame - this can in theory be set, either manually or using a Kernel parameter. I attempted to use manual control of NCQ but it seems the parameter does not stick.
Log files are shared on Pastebin for those that wish to peruse them (with some initial comments at the start of the file):

Updates following the suggestions from chanik: (25th Nov 2015)
I used the recommended command,

echo 1 > /sys/block/sde/device/queue_depth

to set the queue depth to one, using

cat /sys/block/sde/device/queue_depth

I verified that it was set to 1 (whether the setting was respected is another question).

In either case, using dd to zero the drive or actually a partition on the drive fails. Following some further comments I reran the test - just in case something funny happens with the controller with dd, I created a fresh GPT table on the drive with a brand new ext4 partition spanning the entire drive and then copied a large directory to the drive. (Failed in both cases, but weirdly enough lived for 48GB with NCQ and 180GB without NCQ this time…)

For troubleshooting I copied/collected the output in /var/log/messages after the error occured and for NCQ set to 1, I also dumped the dmesg output to a logfile after the error occured.
(Text hosted on Pastebin)

-> if I read the dmesg log correctly, this may suggest that NCQ=1 was not respected

dmesg after reboot and after manually setting ncq to 1, looks like it is truly not respected…

#4

I have an ASMedia ASM1062 SATA Controller card (2 ports, sata III 6GB) connected to two DVD writers. The 6 motherboard SATA ports have one SSD and 5 HDDs hooked up, on an oS 13.2 KDE 4 desktop system, with nvidia proprietary driver.

Occasionally during DVD burning the system would freeze, only a reboot worked. This would happen (but seldom) if there was desktop/SDD/HDD activity during burn, leaving the system alone, the burn would complete. This problem happened only during burn, intense read access (as intense as a DVD can be, that is) had no problems. Freeze is instant, no apparent write degrading before

Some card and cable reinsertion did minimize the issue, now it happens very rarely, like in one or two percent of the burns, seldom enough that I don’t worry about it any more.

So probably not related to your problem, but I do have a few suggestions/workarounds:

  1. if you can test both in w7 and linux in the same machine, i.e., a dual-boot or the windows one with a few different distros liveCDs, perhaps, you can narrow the problem down to the driver, even to a specific version. In this case, a bug report to the driver developer would probably the best option.

  2. Use only one of the card ports, as apparently you don’t need both. If already so, try using the other port.

  3. Keep the four 3TB drives on the mobo ports, leave the card port to the oS drive, as it won’t be writing nearly as much - check for stability issues, of course.

  4. Could it be a temperature issue? If you didn’t already, test with the case open. Perhaps the other computer with windows has a better thermal solution.

There it is, some ideas, suggestions. Good luck :slight_smile:

#5
  1. Test the 3TB drive but with a smaller partition, say, less than 2TB, or even 1TB.
#6

  1. That would be difficult - I potentially could take the card out of my server and set up a Linux install on my desktop (where it tested OK with Windows) on a new harddrive. (prior home server experience means I do have too many harddrives…)
    Though it means carrying the server around again…

  2. I am using only one port on the card - 1 mainboard to OS drive (port 1), then 3 ports to the storage drives - and then the 4th is on the card. (it is why I am not in panic mode - 3 out of 4 drives with RAID6 means there is still a safety margin.)

  3. see two, using only one port

  4. definitely not a case temperature issue - unless the chip on its own overheats (had it with the case open, closed)
    I did do some copy tests again where it ran for longer before it failed - and it is colder downstairs. On the other hand, upstairs it failed on a fraction of the transferred files compared to Windows on lower speed - so unless there is some bug in the code that results in excessive heat, it should have overheated when testing under Windows where it was writing with 150MB/s and not Linux where it only achieves 54MB/s maximum…

  5. I tested a physical 2TB drive - shouldn’t make a difference to a 2TB partition or do you think it would?

The StackExchange Poster found some curious entries in the dmesg… (comments)
http://unix.stackexchange.com/questions/244419/marvell-88se9128-9123-sata-card-weird-behaviour-opensuse

#7

OK…

I got a response from Tech Support (someone with more Linxu experience) - though it is still troubleshooting, including the comments from Unix Stackexhange, so…
dmesg output was collected immediately after a failure.

1) I got one of my older spare drives, unplugged all drives in my OpenSUSE server except the offending drive and tried to install Windows 7 - only 32 bit while OpenSUSE is 64bit.
By default Windows wants to install to the 3TB drive, weird… disconnect, try again.
Installed the Marvell driver (not actually 100% sure if that was necessary, but I believe it is). Nice surprise, CrystalDiskMark 32 Bit can run a benchmark with a 32GB file - OK, lets run that with sequential read and write:
170MB/s read and 143MB/s write - so slightly slower read than expected (20MB/s less) and slightly slower write than expected (10MB/s). So lets try another port - AND a smaller 8GB file - voila, 185MB/s read and 183 MB/ write. (8GB file on the other slower port was near identical).

Conclusion - as tested previously with my desktop, the card itself is physically fine (it did 2 benchmarks in quick succession followed by another, oh and CrystalDiskMark did 5 passes - so if heat or stability were an issue it MUST have failed but it didn’t).

2) On Stackexchange it was suggested to try an older Kernel - well, I tried 4.1 and then used the spare drive to install a brand new fresh OpenSUSE 13.2 with Kernel 3.x - tested the card, updated OpenSUSE, tested the card, disabled native command queueing and tested the card again.
Well, it still failed.
Again, every time it would start at around 50 odd MB/s and then drop - except possibly on the fresh install that only wrote at KB/s…

3) Back to my actual OpenSUSE install, just out of curiosity, I still have the Windows formatted drive - instead of 1 Partition, Windows reserves 128MB and creates a second data partition. Well, copying a large file to the NTFS partition and using a stopwatch as an unscientific measurement, it would have been around 120MB/s write speed - and of course, failed again…

#8

Hi
Might I suggest using the fio program, should be able to give more
information. I use it with bcache setups to check throughput. There is
also a script I use called fio.bash, there is a link to it on my blog
post;
https://forums.opensuse.org/entry.php/159-Setting-up-bcache-on-openSUSE-13-2

But fio has many other options.

–
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 | GNOME 3.10.1 | 3.12.48-52.27-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

#9

Ok, installed fine, but I’m not 100% sure where I should go with it. If I mount the drive in /drivetest and then run fio.bash on it, this happens:

Baseline reads with hdparm

/dev/sde1:
 Timing cached reads:   13854 MB in  2.00 seconds = 6931.04 MB/sec
 Timing buffered disk reads: 492 MB in  3.01 seconds = 163.73 MB/sec
Sequential read
  read : io=1105.7MB, bw=111032KB/s, iops=27758, runt= 10197msec
Sequential write
  write: io=559284KB, bw=55271KB/s, iops=13817, runt= 10119msec
Random read
  read : io=12884KB, bw=1222.2KB/s, iops=305, runt= 10542msec
Random write
  write: io=13664KB, bw=1308.4KB/s, iops=327, runt= 10444msec
Mixed 70/30 random read and write with 8K block size




fio: pid=0, err=5/file:filesetup.c:186, func=fsync, error=Input/output error
fio: pid=0, err=30/file:filesetup.c:173, func=write, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
fio: pid=0, err=30/file:filesetup.c:63, func=unlink, error=Read-only file system
rm: cannot remove ‘/drivetest/8k7030test.0.0’: Read-only file system
rm: cannot remove ‘/drivetest/8k7030test.15.0’: Read-only file system

Basically it starts and then dies with the drive dropping offline.
Or is there something specific you would like me to run in fio? - If I use --help I see a nice number of options and I’m just seeing if I can figure out what might be useful…

#10

Hi
Question is why is it going read only…? Drives failing…? Have you run smartctl on the devices?

#11

The drive is definitely fine - it passes a SMARTCTL test and is actually a new recertified drive from Seagate.
(The last one failed a SeaTools test eventually.)
The same drive is fine with Windows and the problem is irrespective of the drive used - i.e. also occurs with a different 2TB drive.

The same cable works fine under Windows too.

I am pretty much certain it is not hardware as in cable/drive/the card itself (the latter works fine under Windows) nor the Dell T20 itself.
It could be bug in the kernel or the firmware - but I wouldn’t know how to troubleshoot either without explicit help.

#12

Hi
Does it go like that every time you try it the script, or just random?

You don’t have a thermal probe to point at the card chips when it’s running the test?

I use one of these;
http://www.duratrax.com/accessories/nitro/dtxp3100.html

But also have a multimeter with a temperature probe.

Did you check the drive before/after the test with smartctl and compare the results;


smartctl -a /dev/sdX
#13

The drive drops any time Linux tries to write to it via the SATA card - be it as part of the RAID array, copying files, writing zeroes or doing a fio test.
The only thing that seems to work is partitioning it…

When the drive fails, smartctl will report it as a 600PB drive… after a reboot all is back to normal with the drive reported correctly.
The time it fails is also arbitrary - generally within about 10GB where anything above 5GB is a longer run but it sustained nearly 100GB on one occasion - I don’t know why though.

I don’t have any kind of temperature probe - and getting to the chip while installed would be inconvenient as it points down.
(Having said that, I don’t think it is temperature as 5 passes with a 32GB test in CrystalDiskMark worked successfully under Windows, unless a bug on the Linux side causes excess heat…)

And those wondering about Kernel bugs - juts tried Ubuntu with Kernel 3.13 from a USB drive, still an error…
(dmesg: http://pastebin.com/pSeeAbNm , lshw: http://pastebin.com/tP6c874P )

#14

Hi
I’m thinking a trip to the Linux kernel Mailing list may be the best bet, or try the openSUSE kernel Mailing List first;
https://en.opensuse.org/Communicate

Or if you have a spare US$20 get a IO Crest 2 Port SATA III PCI-Express x1 Card (SY-PEX40039 - Asmedia 1061 SATA Host Controller) from Amazon, I have had one of these since last August on a HP system, works a treat and doesn’t miss a beat…

#15

I have just emailed the kernel mailinglist.
Thanks for the suggestion (I possibly would have refrained from it myself) - maybe something comes of that.

I just had a look at amazon.uk - well, ouch… the card you suggest is 100 pounds in the UK, whyever…
(There are other cheap cards though.)

I have a SiI based card that does not work in the Dell (or my Fujitsu Desktop) - this could be a dead card or an incompatibility, I have seen people commenting on SiI issues with Dells… (And of course Dell doesn’t have a list of compatible cards, I asked, despite originally selling the T20 with the StarTech card that I have as an expansion card…)
I did find a different card with an Asmedia chip “CSL” for 10.85 pounds… and wonder if it is worth trying… - I just don’t want to have more hardware lying around… (I have a nice stack of disks that work but aren’t in good shape from my old Windows Home Server for example… I should use them as “cold offsite storage” but never seem to get round to it…)
(I was briefly considering a 30 pounds card with mini SAS + a 15 pound cable but it would be Marvell based too and might just have the same issue…)

Well, if no ideas transpire from the mailing list a new card might be tried…

#16

Hi
Yes, saw the ML post :wink:

WOW, it’s less than US$20 here, when I brought mine I think it was around US$16!!

Can you confirm the driver in use via lspci -nnk and then check the module in use parameters for any tweakable options.

#17

Pricing can be mad at times. The other issue I have is that everything in the UK is generally more expensive - 10% extra is pretty standard from my experience…

Anyway - back to the issue at hand, using your command, the reported driver is “ahci” (as should be for Marvell chips from what I have read).
(Output in detail: http://pastebin.com/KtWAkZAs )

Where would I look for tweaks? Modinfo?

#18

Hi
OK, so run;


/sbin/modinfo ahci

There should be a marvell_enable parm?

#19

If I run

/sbin/modinfo ahci

as root I get

modinfo: ERROR: Module ahci not found.
#20

Hi
OK, so it must be aliased to one of the modules? Or the changes in the 4.X kernels, I see the same on my leap system.

If you look at the output from lsmod, The ahci module should have a count? In the output from lsmod do you spot any sata module?

The list is from ls /lib/modules/4.1.12-1-default/kernel/drivers/ata/ so it must be using one of these.