hdd.. think i'm fsck'd

English Hardware
1

I am having trouble with hdds. I’m running a virtualization machine that actually has a few drives. I had 2 of the drives drop from the normal rw to ro. No significant event took place with the system. I really know enough bash and linux to be dangerous. I really only worked with 1 of the 2 drives so far, which is /dev/sdc1 and this is what I have done.

removed mount points that I see as not important and/or security sensitive but not the /dev/sdc and changed hostnames

root@vit18:~# mount -lsysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=3070599,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=2458444k,mode=755)
/dev/mapper/pve-root on / type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4916880k)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sda2 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sdb1 on /ssd1 type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
/dev/sdd1 on /storage/hdd0 type ext3 (ro,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)

root@vit18:~# fdisk -l /dev/sd*

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.




Disk /dev/sda: 240.1 GB, 240057409536 bytes
255 heads, 63 sectors/track, 29185 cylinders, total 468862128 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   468862127   234431063+  ee  GPT


Disk /dev/sda1: 1 MB, 1048576 bytes
255 heads, 63 sectors/track, 0 cylinders, total 2048 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda1 doesn't contain a valid partition table


Disk /dev/sda2: 534 MB, 534773760 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1044480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda2 doesn't contain a valid partition table


Disk /dev/sda3: 239.5 GB, 239519924224 bytes
255 heads, 63 sectors/track, 29119 cylinders, total 467812352 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda3 doesn't contain a valid partition table


Disk /dev/sdb: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   500118191   250059064+  83  Linux
Partition 1 does not start on physical sector boundary.


Disk /dev/sdb1: 256.1 GB, 256060482048 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118129 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Alignment offset: 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdb1 doesn't contain a valid partition table


Disk /dev/sdd1: 1000.2 GB, 1000203837440 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953523120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdd1 doesn't contain a valid partition table

The scary part *for me at least *starts here

root@vit18:~# fsck /dev/sdc1fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/sdc1
Could this be a zero-length partition?

root@vit18:~# dd if=/dev/sdc1 of=/storage/smb/sdc1dd: reading `/dev/sdc1': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00107448 s, 0.0 kB/s

Is there a hope of saving the data?

Thanks!!

2

On Mon, 26 Oct 2015 16:36:01 +0000, ar3inc wrote:

> Is there a hope of saving the data?

I would start with smartctl - see if the drive’s own diagnostic can give
you an idea what the problem is.

If the drive is experiencing a physical problem, fsck is not likely to
help. dd_rescue might help create an image that you can then extract the
data from.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

3

Yep know just enough to be dangerous. smartctl is not on this machine. apt-get and aptitude can’t find it. but honestly I would lean towards this being a filesystem issue over hardware being that several drives are show not having a valid partition table.

Not trying to be difficult just ignorant.

4

On Mon, 26 Oct 2015 17:16:01 +0000, ar3inc wrote:

> hendersj;2733656 Wrote:
>>
>> I would start with smartctl - see if the drive’s own diagnostic can
>> give you an idea what the problem is.
>
> Yep know just enough to be dangerous. smartctl is not on this machine.
> apt-get and aptitude can’t find it.

Those aren’t the openSUSE application management tools. You’ll find it
with zypper or YaST.

> but honestly I would lean towards
> this being a filesystem issue over hardware being that several drives
> are show not having a valid partition table.
>
> Not trying to be difficult just ignorant.

I/O errors like the one you’re receiving tend to be hardware related.
The only way to know for sure is to query the controller, and that’s what
smartctl can do.

Having recently had a hard drive crash myself, I can tell you that you
will see all kinds of weird things, including invalid partition tables.

Trust the experience of those with experience. Myself, I’ve been working
with computers for over 30 years - and what you have posted so far leads
me to think hardware first.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

5

Okay I adapted a bit since I’m running a debian based system. Sorry this forum seemed to have best responses. wasn’t really paying attention to to flavor of linux.

root@vit18:~# smartctl --health /dev/sdcsmartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.32-32-pve] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net


Log Sense failed, IE page [scsi response fails sanity test]
6

On Mon, 26 Oct 2015 18:06:01 +0000, ar3inc wrote:

> hendersj;2733659 Wrote:
>>
>>
>> Trust the experience of those with experience. Myself, I’ve been
>> working with computers for over 30 years - and what you have posted so
>> far leads me to think hardware first.
>>
>>
>>
> Okay I adapted a bit since I’m running a debian based system. Sorry
> this forum seemed to have best responses. wasn’t really paying attention
> to to flavor of linux.
>
>
> Code:
> --------------------
> root@vit18:~# smartctl --health /dev/sdcsmartctl 5.41 2011-06-09
> r3365 [x86_64-linux-2.6.32-32-pve] (local build)
> Copyright (C) 2002-11 by Bruce Allen,
> http://smartmontools.sourceforge.net
>
>
> Log Sense failed, IE page [scsi response fails sanity test]
> --------------------

That almost sounds more like a controller failure than a driver failure.
You might try transplanting the drives in another system to see if it’s
the controller on the drive or the controller on the motherboard.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

7

You run smartctrl in the HOST not the guest

8

Okay that will probably take a bit of time to shut down the machine. Since the virtual containers are still running. fortunately :\ the effected drives are only used for install images and backups. Once I get it shutdown I’ll pull the drives.

meaning only use it for the local system not remote drives?

Thanks

9

On Mon, 26 Oct 2015 21:06:01 +0000, ar3inc wrote:

> hendersj;2733672 Wrote:
>>
>>
>> That almost sounds more like a controller failure than a driver
>> failure.
>> You might try transplanting the drives in another system to see if it’s
>> the controller on the drive or the controller on the motherboard.
>>
>>
>>
> Okay that will probably take a bit of time to shut down the machine.
> Since the virtual containers are still running. fortunately :\ the
> effected drives are only used for install images and backups. Once I
> get it shutdown I’ll pull the drives.
>
> gogalthorp;2733682 Wrote:
>> You run smartctrl in the HOST not the guest
>
> meaning only use it for the local system not remote drives?

Yes, that should definitely be done on the host - sorry that I wasn’t
clear about that. gogalthorp is correct - it needs to talk to the
hardware directly, and in a VM it can’t.

Start with that, then go from there. I thought (assumed) that you had
run the command on the host and not the guest.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

10

yes that was a correct assumption.

11

MMMM local/remote drive??? maybe you need to explain the set up more.

Smartctl is used on the machine that has the hard drive that holds the files for the virtual drives to test the real hardware.

Don’t confuse real hardware with virtual hardware. The concern is that the real hardware may have a problem

Also the problem may be the one that a recent update (about a month ago) caused to mount the partitions RO.

This thread may help or not :stuck_out_tongue:

https://forums.opensuse.org/showthread.php/510027-Read-Only-Filesystem

12

On Mon, 26 Oct 2015 22:36:02 +0000, ar3inc wrote:

> hendersj;2733701 Wrote:
>>
>>
>> Yes, that should definitely be done on the host - sorry that I wasn’t
>> clear about that. gogalthorp is correct - it needs to talk to the
>> hardware directly, and in a VM it can’t.
>>
>> Start with that, then go from there. I thought (assumed) that you had
>> run the command on the host and not the guest.
>>
>>
>>
> yes that was a correct assumption.

And also assuming the drives are physically connected to the system and
not remote volumes.

If they’re remote volumes, then you need to run the command where the
drive is physically located.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

13

No problem hdds are locally attached via sata cable. other drives that are there but not giving problems - 2 ssds also attached via sata and 1 one remote drive but those are a problem.

14

On Tue, 27 Oct 2015 12:56:01 +0000, ar3inc wrote:

> No problem hdds are locally attached via sata cable. other drives that
> are there but not giving problems - 2 ssds also attached via sata and 1
> one remote drive but those are a problem.

OK - yeah, then it sounds like a controller issue, and isolating whether
it’s the drive or the motherboard would be the next step.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

15

Thank you for the insight to this one. however I think it doesn’t apply since I’m on Grub1.99-27+deb7u2. the smart errors appear to be pointing towards hdds, mother board, or power.

16

You should fully describe the system then since it is getting obvious that you don’t have what could be considered a standard install

:open_mouth:

17

yeah… it’s a different setup. OS is proxmox, a virtualization software debian base proxmox, intel LGA1155 proc, Intel Z77 MB, 24 GB Ram DDR3 (can’t remember the speed (maybe 1600)), 2x SSDs SATA, 2x HDD SATA (also can’t remember speed (maybe 7k)). Did i miss anything?

18

did some digging Ram speed 1333. can’t get failed hdds speed since they’re failed.

19

Okay, thank you for y’alls help!! after shutdowning the vm containers, and host machine, I tested the hdds in question with an external usb adapter. everything passed.:
hooked the drives back to original system and sure enough hdds loaded right up. Not really sure if i trust the hdds much anymore but they’ll hold for the meantime.

20

On Wed, 28 Oct 2015 16:06:01 +0000, ar3inc wrote:

> Okay, thank you for y’alls help!! after shutdowning the vm containers,
> and host machine, I tested the hdds in question with an external usb
> adapter. everything passed.:
> hooked the drives back to original system and sure enough hdds loaded
> right up. Not really sure if i trust the hdds much anymore but they’ll
> hold for the meantime.

Very strange, but glad you got access to the data again. Might be time
to start making backups. :slight_smile:

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C