Rsync causing write problems on HDD?

Hello, this has been tested on 3 different fresh HDDs so it’s not a failing drive.

I am rsyncing from NTFS HDD in my desktop to my NTFS HDD in my laptop, and every time after writing ~1TB (in 2TB HDD) rsync starts complaining not enough space error. The only way to fix it is to use Windows 10 HDD repair software but same problem occurs almost immediately after if I continue the rsync.

I am also noticing from W10, that the HDD is heavily fragmented.

It seems rsyncing is causing some sort of problem on my HDD and like I said same thing happens multiple other HDDs. I cannot tell why it’s doing this, any suggestions? I am thinking maybe the source HDD is failing?

Sorry, the source HDD is ext4 not NTFS. Apparently I changed that.

Try using the “–inplace” option.

This option changes how rsync transfer file(s). Instead of the default method of creating new temporary copy(copies) of the file(s) and moving into place when it is complete, rsync instead writes the updated data directly to the destination file(s).

Be aware though, this is a slightly “riskier” method of using rsync.

Try to run rsync under Windows. I presume a native process will yield better writes.

NTFS is not fully supported in Linux. Maybe Paragon NTFS driver will help with that.

Well, now the NTFS drive does something super weird.

I write/access the NTFS drive from W10, W10 goes to sleep, literally then the drive becomes inaccessible. Right now, I am moving the data over to an external HDD, going to reformat the NTFS drive and put the data back on.

Beware of hardware problems. Even an USB cable may fail. Rsync issues resulted from disk damage:

I did not read all of this thread because it is out of my knowledge area (Windows involved of which I know almost nothing and usage of NTFS tgat goes beyond just a simple exchange of files between Linux and non-Linux).
But are you aware of the fact that you can only use an NTFS file system on a Linux system after it is used on a Windows system when that Windows system was properly shutdown? No sleep, no “fast boot” set!

Henk, I am aware, I also used to “have fun” when bitlocker automatically locked shared partitions and I spent days trying to figure out what’s going on. I don’t allow my computer to hibernate/sleep and etc, always full shutdown and disabled fast-boot in power options because I’ve had exactly that issues in the past. Maybe it was you who told me in the first place years ago haha.

Karl, I don’t think it’s Hardware damage, these are internal drives, and almost 6 (2 Seagate Firecuda, 2 Seagate Barracuda, 2 WD Blue) of them show same signs and the rsync is done through ethernet. Sometimes 2m away through a router, sometimes over internet from office to home.

For now, I moved everything over to a spare HDD through USB, did a WD dashboard wipe on the funky HDD, re-initialize, re-partition, re-format and putting the data back on from the Windows side.

So far, the write/read speeds are encouraging (before the reinitializing, sometimes the speeds were down below 1MB/s, now I see steady above 5MB/s [lots of small files, so I don’t expect see close to 100MB/s]).

Still, I really do not know if/how rsync can mess up a HDD. When I checked after data transfer using rsync, the HDD was HEAVILY fragmented, that was for sure. Would Paragon NTFS driver replace the regular one just by installing it by the way? I will also try the inpalce option in the next batch. For now… I just want the HDD to function.

I’ve had that kind of error when I had a failing HDD, which failed SMART test but this time, I am getting the “not enough space” error when there is more than 800GB of free space.

The hard drive shown below does not exhibit fragmentation despite being used solely for daily backups with rsync:

**erlangen:~ #** fdisk -l /dev/sdb 
**Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors**
Disk model: WDC WD40EZRX-22S 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 4096 bytes 
I/O size (minimum/optimal): 4096 bytes / 4096 bytes 
Disklabel type: gpt 
Disk identifier: 27C8C52A-8091-403C-ADF1-E9C791667D40 

**Device****Start****       End****   Sectors**** Size****Type**
/dev/sdb1   4096 7814035455 7814031360  3.6T Linux filesystem 
**erlangen:~ #** df -h /HDD/ 
Filesystem      Size  Used Avail Use% Mounted on 
/dev/sdb1       3.6T  1.7T  2.0T  46% /HDD 
**erlangen:~ #** e4defrag -c /HDD/ 
e4defrag 1.46.4 (18-Aug-2021) 
<Fragmented files>                             now/best       size/ext 
1. /HDD/backup/home/karl/.mozilla/firefox/... 
                                                 5/1              7 KB 
2. /HDD/backup/home/karl/.mozilla/firefox/...
                                                 4/1              7 KB 
3. /HDD/backup/home/karl/.mozilla/firefox/...
                                                 4/1              7 KB 
4. /HDD/backup/home/karl/.mozilla/firefox/... 
                                                 4/1              7 KB 
5. /HDD/backup/home/karl/.mozilla/firefox/... 
                                                 4/1              7 KB 

 Total/best extents                             1285795/1243473 
 Average size per extent                        1362 KB 
 Fragmentation score                            0 
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag] 
 This directory (/HDD/) does not need defragmentation. 
**erlangen:~ #**

Usage is 47% only. NTFS may behave differently.

Check for messages of scsi driver by running: journalctl _KERNEL_SUBSYSTEM=scsi. Sane links are absolutely silent during operation.

SMR HDDs in use?

Source is CMR and target is SMR I think… or maybe both. Should that be an issue?

SMR HDD needs much longer time to write. So some network packets can be dropped.
This is an assumption - test it to be sure.

Well there was a disruption once during the rsync, then from that point on write was extremely slow, and often ended up in HDD fail followed up by Windows trying to fix the drive.

At the end, I took the data out, reformatted and so far so good.

Will be so with a clean SMR drive.
But if drive starts to rewrite files…
Right now just wait.