Bad sectors ... what to do ?

[keyb_user]

Hi,

I have had my share of misfortune in what concerns hardware.
But in all this years (many many years ... ) one thing I was very very fortunate about, bad hdd's.

So far I only had two problems and even those where on servers that run 24/7 during many years on data-centers ...

But last week I using my 4 hdd 4TB Raid 10 Sofware and Encrypted Raid when I decided to enable smartd ... errors where reported ...

Yup!

I made a normal install of Linux on a Flash USB, then took an older MB + Box that I had lying around and installed there 4 1TB Hitashi hdd's.

As soons as they arived they where big problems.
One hdd did not even got recognized by the bios, I returned it and then they gave me a new one.

Last week the problem was smart diagnotics.

smartctl -a /dev/sdb gave 12 errors, all relating to bad sectors.

The problem is that I think the disk did not report any errors prior to that usage ...

Now it only indicates de error messages but nothing else really occurred ...

Also this is on hdd in a RAID 10 setup.

I created the raid,.that is:

1- Formated all disk to sustain a RAID partition.

2- Created the raid lvel 10 with the 4 devices.

3- Encrypted the raid partition ... a long process ... one and a half week of writing random data on it ...

And now I have a Raid 10 2TB encrypted partition that I mount after booting ...


The actuall system is a Ubuntu 9.04 that boots from flash USB pen.

My problem is what should I do with the hdd ?

I do not have any tools from the vendor that I can use.

Any advice ?

Output of smartctl is this:

sudo smartctl -a /dev/sdb

[sudo] password for samp:
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     Hitachi HDT721010SLA360
Serial Number:    STF605MH2DWPMK
Firmware Version: ST6OA31B
User Capacity:    1,000,204,886,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Mon Nov  2 21:32:32 2009 WET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 (14090) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 235) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   098   098   016    Pre-fail  Always       -       262145
  2 Throughput_Performance  0x0005   130   130   054    Pre-fail  Offline      -       122
  3 Spin_Up_Time            0x0007   121   121   024    Pre-fail  Always       -       465 (Average 484)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       51
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       8
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   123   123   020    Pre-fail  Offline      -       34
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       246
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       51
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       54
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       54
194 Temperature_Celsius     0x0002   150   150   000    Old_age   Always       -       40 (Lifetime Min/Max 25/51)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       12
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 12 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 12 occurred at disk power-on lifetime: 126 hours (5 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 0f 30 bd 1e e9  Error: UNC 15 sectors at LBA = 0x091ebd30 = 153009456

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 3f b9 1e e0 08      00:39:47.900  READ DMA EXT
  27 00 00 00 00 00 e0 08      00:39:47.900  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 0a      00:39:47.800  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      00:39:47.800  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 08      00:39:47.800  READ NATIVE MAX ADDRESS EXT

Error 11 occurred at disk power-on lifetime: 126 hours (5 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 0f 30 bd 1e e9  Error: UNC 15 sectors at LBA = 0x091ebd30 = 153009456

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 3f b9 1e e0 08      00:39:43.400  READ DMA EXT
  27 00 00 00 00 00 e0 08      00:39:43.400  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 0a      00:39:43.400  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      00:39:43.400  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 08      00:39:43.400  READ NATIVE MAX ADDRESS EXT

Error 10 occurred at disk power-on lifetime: 126 hours (5 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 0f 30 bd 1e e9  Error: UNC 15 sectors at LBA = 0x091ebd30 = 153009456

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 3f b9 1e e0 08      00:39:39.000  READ DMA EXT
  27 00 00 00 00 00 e0 08      00:39:39.000  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 0a      00:39:39.000  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      00:39:39.000  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 08      00:39:39.000  READ NATIVE MAX ADDRESS EXT

Error 9 occurred at disk power-on lifetime: 126 hours (5 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 0f 30 bd 1e e9  Error: UNC 15 sectors at LBA = 0x091ebd30 = 153009456

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 3f b9 1e e0 08      00:39:34.600  READ DMA EXT
  27 00 00 00 00 00 e0 08      00:39:34.600  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 0a      00:39:34.600  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      00:39:34.600  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 08      00:39:34.600  READ NATIVE MAX ADDRESS EXT

Error 8 occurred at disk power-on lifetime: 126 hours (5 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 0f 30 bd 1e e9  Error: UNC 15 sectors at LBA = 0x091ebd30 = 153009456

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 3f b9 1e e0 08      00:39:30.100  READ DMA EXT
  27 00 00 00 00 00 e0 08      00:39:30.100  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 a0 0a      00:39:30.100  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 0a      00:39:30.100  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 e0 08      00:39:30.100  READ NATIVE MAX ADDRESS EXT

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Regards,
Pedro

[microchip8]

It's always good to get a second opinion, so instead of just relying on SMART info, you should also run badblocks on the disks/partitions. badblocks has an option to save the possibly present bad sectors info to a file which then you can feed to to mkfs when creating a new partition. The file system creation program will mark those into its internal "db" list for bad sectors and will never use them. When running badblocks, it's important to specify the correct block size of the partition(s) or else you'll get incorrect sector numbers, thus flagging the incorrect and good sectors but leaving the bad ones alone. Most file systems use a block size of 4KB so this must be provided with the -b option to badblocks, eg -b 4096 (if you use a different block size, then of course provide the correct one). Beware that on really big partitions/disks, it can take a long time to scan the whole, so you may need to increase how many blocks are scanned at a time (at expense of a bit more memory usage) with the -c option (I often use 2048)

Also note that as time progresses and you keep using these disks, if they really have bad sectors, they'll spread over to others. There are two different types of bad sectors, those caused by a disk shock and those caused by magnetic weakening. If yours are the latter, like I said, they have a tendency to spread to other healthy sectors thus resulting in more problems and corrupting/making data inaccessible. Also there is another thing that can bring the illusion as if there are bad sectors present and that is failed parity checks. If those are corrupt for some reason, software will report bad sectors even though there may be no bad sectors at all but the parity corruption is "confusing" things. Also, often bad sectors are due to read/write failure of the disk heads

On 11/02/2009 02:06 PM, keyb user wrote:
>

> My problem is what should I do with the hdd ?
>
> I do not have any tools from the vendor that I can use.
>

Most hd manufacturers provide stand alone (bootable from floppy, CD) low
level diagnostic programs...could check with the manufacturers web site
for availability of such a program. IMHO, utilizing such a diagnostic
program is the first step if there is any chance of harddrive failure.
YMMV. :-)

[buckesfeld]

* keyb user wrote, On 11/02/2009 11:06 PM:
> The actuall system is a Ubuntu 9.04 that boots from flash USB pen.

Then what are you doing here in the openSUSE forums? ;-)

Uwe

good info, thanks for posting!

--
palladium
Have a lot of fun..

[keyb_user]

Hi,


I was also thinking about using badblocks.

Originally Posted by microchip8

It's always good to get a second opinion, so instead of just relying on SMART info, you should also run badblocks on the disks/partitions. badblocks has an option to save the possibly present bad sectors info to a file which then you can feed to to mkfs when creating a new partition. The file system creation program will mark those into its internal "db" list for bad sectors and will never use them. When running badblocks, it's important to specify the correct block size of the partition(s) or else you'll get incorrect sector numbers, thus flagging the incorrect and good sectors but leaving the bad ones alone. Most file systems use a block size of 4KB so this must be provided with the -b option to badblocks, eg -b 4096 (if you use a different block size, then of course provide the correct one). Beware that on really big partitions/disks, it can take a long time to scan the whole, so you may need to increase how many blocks are scanned at a time (at expense of a bit more memory usage) with the -c option (I often use 2048)

I was also thinking about badbloks.
But my fear is that the disk will degrade .. so a replacement would be inevitable.
I will try to run first a utility disk Hitachi provides. But I suspect that will not be sufficient.
I just really want to know what happened.

Also note that as time progresses and you keep using these disks, if they really have bad sectors, they'll spread over to others. There are two different types of bad sectors, those caused by a disk shock and those caused by magnetic weakening. If yours are the latter, like I said, they have a tendency to spread to other healthy sectors thus resulting in more problems and corrupting/making data inaccessible. Also there is another thing that can bring the illusion as if there are bad sectors present and that is failed parity checks. If those are corrupt for some reason, software will report bad sectors even though there may be no bad sectors at all but the parity corruption is "confusing" things. Also, often bad sectors are due to read/write failure of the disk heads

I could not have written that better.
Smart errors can also be caused by, for example, bad connections on the sata cables ... there are a all lot of factors to consider.
I do not have a lot of experience with hhd failures (fortunately ), but your comments are very correct. Things can degrade over time.
There is really not a very reliable way to know the reason of what happened.

And I suspect some one time even could be the culprit. The errors where all detected at boot in the same circumstances at the same time. They were never again reported by smart ...

I will try to use the Hitachi tool know more ... but i fear a new hdd is required ... i do not trust that stuff in there anymore

Regards,
Pedro

[keyb_user]

Hi,

The main reasons are in descending order of relevance:

1- Annoying the OpenSuSE forum admins

2- Make the world know that we, the OpenSuSE community, also have Bad Sectors

3- Take some hits from the Ubuntu forums ... they are already overcrowded ...

4- By Clearly Mentioning it was not an OpenSuSE distro running the system this can led some naive users to think that stuff like that does not happen when one Install OpenSuSE! Imagine! I have 15 PC's and that stuff Just happened on the Ubuntu Box !!! (implied thought conditioning)

5- Heck! There is always the possibility to start a Distro War, which will Annoy even more the Forum Admins

Originally Posted by buckesfeld

* keyb user wrote, On 11/02/2009 11:06 PM:
> The actuall system is a Ubuntu 9.04 that boots from flash USB pen.

Then what are you doing here in the openSUSE forums? ;-)

Uwe

Regards,
Pedro