How long will 'dd' take on a 1TB drive?

[amarildojr]

Hello. I was using openSUSE 12.3 on a 320GB S-ATAII Seagate (7200RPM, 16MB buffer), but it was failing (+4000 reallocated bsectors) so I bought a new drive, the model is:

Western Digital WD10EARS 00Y5B1](http://www.ebay.com/itm/WESTERN-DIGITAL-WD10EARS-00Y5B1-HARD-DRIVE-1TB-SATA-3-5-R-N-701640-/180752453761)
The reviews are no good but this seems to be a “good” one of this model, no performance issues, not big blocks etc.

So, I all do now when I want to format any HD is:

• Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda count=1

• Create random size partitions with random file systems

• Delete all partitions

• Erase the partition table again and then create the partitions I needed.

With the other drive I used to to a Wipe with Seagate’s disk utility, filling it with zeroes used to take 1 hour. How long will it take with this drive?

[hendersj]

On Fri, 26 Apr 2013 02:56:09 +0000, amarildojr wrote:

> So, I all do now when I want to format any HD is:
>
> * Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda
> count=1
>
> * Create random size partitions with random file systems
>
> * Delete all partitions
>
> * Erase the partition table again and then create the partitions I
> needed.
>
> With the other drive I used to to a Wipe with Seagate’s disk utility,
> filling it with zeroes used to take 1 hour. How long will it take with
> this drive?

That seems like overkill to me. What is your goal?

Overwriting the partition table as you do in your first step is
sufficient to frag the partition table. If you’re installing an OS on
the drive afterwards, overwriting the entire drive with zeros is
pointless.

But to answer your question, we’d need to know the effective throughput
of the I/O channel. Then it’s simple math.

If your 1 TB drive is USB 1.1, it’s going to be very slow. If it’s USB
3, it’ll be faster. If it’s SATA, it’ll be faster still. But without
specific performance characteristics, it’d be difficult to estimate.

You can speed it up by using a large bs= parameter - to a point. But
once the I/O channel is saturated, it’s a question of how fast the
mechanism can write.

You can do this a little quicker, probably, by installing the OS and then
using dd to copy from /dev/zero to a file in the filesystem, and then
delete the file.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

[robin_listas]

On 2013-04-26 04:56, amarildojr wrote:

> So, I all do now when I want to format any HD is:

The new disk, eh?

> * Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda
> count=1

Why urandom?

> * Create random size partitions with random file systems

Why?

> * Delete all partitions

Why?

> * Erase the partition table again and then create the partitions I
> needed.

Ok.

> With the other drive I used to to a Wipe with Seagate’s disk utility,
> filling it with zeroes used to take 1 hour. How long will it take with
> this drive?

Why do you want to zero a new disk? I would understand you’d want to
zero the old disk, not the new. But then, I would not understand why
partition the old disk.

And in any case, new or old, why create random partitions on it, and
delete them… :-o

Well, if it is SATA and you can achieve 80MB/s continuously, and it is 1
TB (mind, 1 TB, not 1TiB), then the math is:

10¹²/(80·10⁶) = 12500 seconds = 3.47 hours:

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

[tsu2]

zeroing can be a security measure or preparing a virtual disk for shrinking.
Shouldn’t be needed if you’re simply re-using a disk which has had data written to it, when you create new partitions and format, the “pointers” to data are all changed so the previous data is normally inaccessible (but recoverable with proper tools until overwritten, and maybe even over-written numbers of times).

But, answering your main Q,

The dd command takes “forever” if you copy only one byte at a time. You can speed things up by specifying larger blocks

eg I keep the following in my toolbag which describes two ways of using dd to zero space. Note how the dd command specifies a block size of 1024 bytes so in theory should work approx 1000 times faster than not specifying a large block size… Of course, can be modified to copy anything

HTH,
TSU

# wiping free space - create a file filled with zeroes, then delete
cat /dev/zero > Filename.txt
rm Filename.txt

# Much faster
dd if=/dev/zero of=zero.small.file bs=1024 count=102400
cat /dev/zero > zero.file 
rm zero.small.file
rm zero.file

[robin_listas]

On 2013-04-26 17:36, tsu2 wrote:
> Note how the dd command specifies a block size of 1024
> bytes so in theory should work approx 1000 times faster than not
> specifying a large block size… Of course, can be modified to copy
> anything

Some say that the optimal size is that of the internal disk cache.
I use “bs=100M” to image devices, but have been told to use
“bs=4K conv=noerror, sync” (size arbitrary) instead.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

[tsu2]

@robin_listas

On 2013-04-26 17:36, tsu2 wrote:
> Note how the dd command specifies a block size of 1024
> bytes so in theory should work approx 1000 times faster than not
> specifying a large block size… Of course, can be modified to copy
> anything

Some say that the optimal size is that of the internal disk cache.
I use “bs=100M” to image devices, but have been told to use
“bs=4K conv=noerror, sync” (size arbitrary) instead.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

You can use larger sizes (especially if you’re just copying zeroes), but when copying data although I haven’t read it anywhere I would suggest 256 or sometimes 512 bytes because they are the most common disk block sizes. I haven’t read that there is actually a problem anywhere but I’d guess that any larger and potential partition/block alignment issues might occur. If someone can say that my guess is without meritI’d gladly use very large block sizes.

TSU

[robin_listas]

On 2013-04-26 22:46, tsu2 wrote:

> You can use larger sizes (especially if you’re just copying zeroes),
> but when copying data although I haven’t read it anywhere I would
> suggest 256 or sometimes 512 bytes because they are the most common disk
> block sizes. I haven’t read that there is actually a problem anywhere
> but I’d guess that any larger and potential partition/block alignment
> issues might occur. If someone can say that my guess is without meritI’d
> gladly use very large block sizes.

I routinely image or clone disks or partitions using block sizes of
100M, it does not affect alignment or anything. You can use block sizes
of 2 bytes or 2 megabytes or whatever, the end result is always the
same. What varies is the speed.

Too small a block, means many operations, so at least use the same size
as a disk block write operation. Too large a block, as dd reserves that
block in ram, could cause inefficiencies, even swapping (as an
experiment try bs=2G and look at top).

The kernel will take care of issues such as alignment.

This is for operations that end at the device boundary; ie, when you do
not have to specify a block count.

All that is assuming there are no read errors; if there are, add
“conv=noerror, sync”, I learned that the other day. Or use dd_rescue
instead.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

[amarildojr]

@hendersj

On Fri, 26 Apr 2013 02:56:09 +0000, amarildojr wrote:

> So, I all do now when I want to format any HD is:
>
> * Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda
> count=1
>
> * Create random size partitions with random file systems
>
> * Delete all partitions
>
> * Erase the partition table again and then create the partitions I
> needed.
>
> With the other drive I used to to a Wipe with Seagate’s disk utility,
> filling it with zeroes used to take 1 hour. How long will it take with
> this drive?

That seems like overkill to me. What is your goal?

Overwriting the partition table as you do in your first step is
sufficient to frag the partition table. If you’re installing an OS on
the drive afterwards, overwriting the entire drive with zeros is
pointless.

But to answer your question, we’d need to know the effective throughput
of the I/O channel. Then it’s simple math.

If your 1 TB drive is USB 1.1, it’s going to be very slow. If it’s USB
3, it’ll be faster. If it’s SATA, it’ll be faster still. But without
specific performance characteristics, it’d be difficult to estimate.

You can speed it up by using a large bs= parameter - to a point. But
once the I/O channel is saturated, it’s a question of how fast the
mechanism can write.

You can do this a little quicker, probably, by installing the OS and then
using dd to copy from /dev/zero to a file in the filesystem, and then
delete the file.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

My goal is to avoid conflicting files. I’ll explain that in the following posts. I did dd/zero to it, left it over night. It said “1TB copied” so I’m assuming it went well, even if the output also was “No disk space” heheh

[amarildojr]

@robin_listas

The new disk, eh?

Yes

@robin_listas

> * Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda
> count=1

Why urandom?

It works as well as zero. I like changes sometimes =)

@robin_listas

> * Create random size partitions with random file systems

Why?

Once I messed up an Ubuntu install, and even after zeroing the partition table the same archives were there after fresh system install, the same bugs were found. So just deleting the PT didn’t work, so I create random partition sizes to make sure that won’t happen again, and it works =P

@robin_listas

> With the other drive I used to to a Wipe with Seagate’s disk utility,
> filling it with zeroes used to take 1 hour. How long will it take with
> this drive?

Why do you want to zero a new disk? I would understand you’d want to
zero the old disk, not the new. But then, I would not understand why
partition the old disk.

And in any case, new or old, why create random partitions on it, and
delete them… :-o

Well, if it is SATA and you can achieve 80MB/s continuously, and it is 1
TB (mind, 1 TB, not 1TiB), then the math is:

10¹²/(80·10⁶) = 12500 seconds = 3.47 hours:

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

Windows updates failed, making my system to crash every boot, I tried System Recovery but it didn’t work. I then formatted and again the system crashed after a specific update. Zeroing the drive would ensure that no leftovers would again be read by the system somehow.

It’s SATA, but chances are it’s a 5400 RPM one. With dd I assume it took less then 6 hours, considering bs=16M. With dban it took about 5 and half hour.

On my old (and failed) 320GB drive, zeroing it took 1 hour. The math would be OK, but it took 6 hours with dban (1 pass with zeros). I plan to do dd again in the future and watch it’s progress, but I’m comfident it took about the amount of time you said.

[amarildojr]

@tsu2

zeroing can be a security measure or preparing a virtual disk for shrinking.
Shouldn’t be needed if you’re simply re-using a disk which has had data written to it, when you create new partitions and format, the “pointers” to data are all changed so the previous data is normally inaccessible (but recoverable with proper tools until overwritten, and maybe even over-written numbers of times).

But, answering your main Q,

The dd command takes “forever” if you copy only one byte at a time. You can speed things up by specifying larger blocks

eg I keep the following in my toolbag which describes two ways of using dd to zero space. Note how the dd command specifies a block size of 1024 bytes so in theory should work approx 1000 times faster than not specifying a large block size… Of course, can be modified to copy anything

HTH,
TSU

# wiping free space - create a file filled with zeroes, then delete
cat /dev/zero > Filename.txt
rm Filename.txt

# Much faster
dd if=/dev/zero of=zero.small.file bs=1024 count=102400
cat /dev/zero > zero.file 
rm zero.small.file
rm zero.file

I was aware of that. I was gonna use bs=512 (default) but it would take for ever and a bit more.
Somehow, for me, after deleting the partition table all things are again accessible for the OS. That means if I had a .cfg that was screwed it would be use again even after PT deletion and OS reinstall. Idk why that happens, but it does happen to me.

[robin_listas]

On 2013-05-01 04:06, amarildojr wrote:
>
> robin_listas;2551285 Wrote:
>> The new disk, eh?
> Yes
>
> robin_listas;2551285 Wrote:
>>> * Erase the partition table with $sudo dd if=/dev/urandom of=/dev/sda
>>> count=1
>>
>> Why urandom?
> It works as well as zero. I like changes sometimes =)

Nope, it doesn’t. You risk that the partitioner you run later thinks
that there is a partition table and tries to do things on it guessing it
wrong.

Please, just zero the partition table

If you want to, say, give the disk for recycling, you might want to
randomize it first, then zero it on a second step.

>
> robin_listas;2551285 Wrote:
>>> * Create random size partitions with random file systems
>>
>> Why? Once I messed up an Ubuntu install, and even after zeroing the
> partition table the same archives were there, the same bugs were found.
> So just deleting the PT didn’t work when I did it, so I create random
> partition sizes to make sure that won’t happen again.

Well, zeroing the partition table does not erase any files. If you
create the same partition on top, then the filesystems and files are
intact and accessible.

You forgot to reformat that one time ago.

You simply need to zero the partition table, create a new partition
table, then “format”, ie create the target filesystems you want. YaST
does this just fine, has a fine tool for doing it.

>
> Windows updates failed making my system to crash every boot, I tried
> System Recovery but it didn’t work. I then formatted and again the
> system crashed after a specific update. Zeroing the drive would ensure
> that no leftovers would again be read by the system somehow.

There are no “somehows” in computers. These are logical machines with
defined behaviours, not magical things >:-)

If a system crashes on two repeated installs when applying an specific
update, well, chances are that the update is incompatible with your system.

Another possibility is that your disk has errors. Writing to a bad
sector triggers a remapping of that sector. Installing twice would
probably write updates to the same disk region.

> On my old (and failed) 320GB drive, zeroing it took 1 hour. The math
> would be OK, but it took 6 hours with dban (1 pass with zeros). I plan
> to do dd again in the future and watch it’s progress, but I’m comfident
> it took about the amount of time you said.

It is just a question of knowing how many bytes per second your hardware
can manage

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

[amarildojr]

Hi Robin.

Nope, it doesn’t. You risk that the partitioner you run later thinks
that there is a partition table and tries to do things on it guessing it
wrong.

Please, just zero the partition table

If you want to, say, give the disk for recycling, you might want to
randomize it first, then zero it on a second step.

It never gave me any problems, but if you say so =)

Well, zeroing the partition table does not erase any files. If you
create the same partition on top, then the filesystems and files are
intact and accessible.

You forgot to reformat that one time ago.

You simply need to zero the partition table, create a new partition
table, then “format”, ie create the target filesystems you want. YaST
does this just fine, has a fine tool for doing it.

I did. I zeroed the partition table, created a new one, created the file systems and formatted. This didn’t prevented the system to use the old files as they were new. This is very strange for me, I really don’t know why that happened, 4 or 5 times.

PS: I know that zeroing the PT doesn’t erase any files besides the PT itself I’m a noob but not at such level.

There are no “somehows” in computers. These are logical machines with
defined behaviours, not magical things >:-)

If a system crashes on two repeated installs when applying an specific
update, well, chances are that the update is incompatible with your system.

Another possibility is that your disk has errors. Writing to a bad
sector triggers a remapping of that sector. Installing twice would
probably write updates to the same disk region.

Well, the same error did happened, I can’t remember which one but it was the exact same error. That drive did have more than 4000 bad sectors (whoooooaaah!!), it could’ve been it but it’s just strange for me that it wasn’t a system install error, that error happened after I messed up my /home cfg files and persisted after a partition table erase + new PT creation + new file systems (exact the same size) + Formatt. Just note that I always use the same username.

It is just a question of knowing how many bytes per second your hardware
can manage

Another strange thing, on DBAN 1 pass of 0’s was going through 50MB/s and took about 6 hours if I recall (or 5 and a half). On dd I shredded a file with bs=16M and it gave me 500MB/s, and that I suppose took the same amount of time.

[hendersj]

On Wed, 01 May 2013 02:06:02 +0000, amarildojr wrote:

> My goal is to avoid conflicting files.

You won’t have “conflicting files” whether you do this or not - if you
delete a file, create new partitions, and install over, the new filesystem

• not the old filesystem - is used.

Unless you mean something else by “conflicting files” - that phrase is
unclear to me.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

[amarildojr]

@hendersj

On Wed, 01 May 2013 02:06:02 +0000, amarildojr wrote:

> My goal is to avoid conflicting files.

You won’t have “conflicting files” whether you do this or not - if you
delete a file, create new partitions, and install over, the new filesystem

• not the old filesystem - is used.

Unless you mean something else by “conflicting files” - that phrase is
unclear to me.

Jim

–
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

That would be nice for me… if it were true

I was using Ubuntu so far and it happened on it and on Windows. Again, it HAPPENED, Idk why but I can guarantee that it happened about 4 or 5 times.

[gogalthorp]

Formatting a partition will erase all thing You are missing some step I suppose. Tell the installer to format the partition and all the old is gone.

[amarildojr]

OMG hehehehehe

I guess my english is not as good as I thought.

I DID format. The steps were:

sudo dd if=/dev/zero of=/dev/sda count=1 (erased the partition table)
create partitions on the same size as before
formatted them
installed Linux and some of my config files were configured the same way as they were before.

[robin_listas]

On 2013-05-01 21:56, amarildojr wrote:
>
> OMG hehehehehe
>
> I guess my english is not as good as I thought.
>
> I DID format. The steps were:
>
> sudo dd if=/dev/zero of=/dev/sda count=1 (erased the partition table)
> create partitions on the same size as before
> formatted them
> installed Linux and some of my config files were configured the same
> way as they were before.

Well, YaST installs every time in the same manner, the config files will
be the same, of course.

However, you said that your disk had “more than 4000 bad sectors”, so
perhaps things were not written, the disk failed to write.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

[amarildojr]

@robin_listas

Well, YaST installs every time in the same manner, the config files will
be the same, of course.

However, you said that your disk had “more than 4000 bad sectors”, so
perhaps things were not written, the disk failed to write.

–
Cheers / Saludos,

Carlos E. R.
(from 12.1 x86_64 “Asparagus” at Telcontar)

It makes sence except for the part where it was the same error that happened after I was messing with my cfgs. A zero-fill fixed the problem =)

YEAH, weird things happen to me.

[hendersj]

On Wed, 01 May 2013 17:26:01 +0000, amarildojr wrote:

> That would be nice for me… if it were true
>
> I was using Ubuntu so far and it happened on it and on Windows. Again,
> it HAPPENED, Idk why but I can guarantee that it happened about 4 or 5
> times.

Then there’s something odd about the way you’re doing it.

Are you restarting the machine after wiping out the partition table? If
you don’t restart the machine, there is some information maintained in
memory.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

[amarildojr]

@hendersj

On Wed, 01 May 2013 17:26:01 +0000, amarildojr wrote:

> That would be nice for me… if it were true
>
> I was using Ubuntu so far and it happened on it and on Windows. Again,
> it HAPPENED, Idk why but I can guarantee that it happened about 4 or 5
> times.

Then there’s something odd about the way you’re doing it.

Are you restarting the machine after wiping out the partition table? If
you don’t restart the machine, there is some information maintained in
memory.

Jim

Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

Yes, I did restart.