file transfer problem (nighmare?)

Hi Opensuse,

Just did a fresh install of Leap 42.2 onto new partitions, so far so good.

Now I want to transfer data from an Opensuse V13.2 disk to a Leap 42.2 disk.

My observations lead me to believe that it is IMPOSSIBLE to copy files from one disk to another
that is 100.000% certain, without extraordinary planning, execution, and verification.

Problem 1)
A team member has the habit of leaving very useful markers in directories, by using file names
like Project_AAA_2014_Jan, but with nothing in the file. (Thank you for fixing the find function in Dolphin.)

A Dolphin copy and paste will not copy an empty file. By now, there are hundreds of these files
and fixing it by hand is not humanly possible.

Problem 2)
Firefox will save a web page which can later have problematic characters in the name. Later on, a copy and paste will cause
a pop-up window to appear asking how to handle the bad directory name. For the pop-up window, the OS displays
the directory name replacing the bad characters with ??, asking if the file should be skipped. Treating every instance by hand isn’t humanly possible.

Problem 3)
A complicating factor is that a somewhat recent backup snapshot was made using a script with rsync, but in light of this discovery, it’s integrity is being questioned.

Question 1)
Why does the OS allow me to make an empty file but refuse to copy it to another disk at a later time?

Question 2)
Why does the OS allow me to copy and save a web-page but refuse to copy it to another disk at a later time?

Question 3)
How do I do an automated and 100.000% certain file transfer between disks?

Question 4)
How do I compare the contents of 2 directories to prove data transfer integrity?

All the disk file systems are linux, encrypted, but not identical, btrfs (?), ext4(?), and possibly other settings.

The data consumes about 1Tb, takes hours to copy, and my trial and error approach has run out of time.

HELP!!!

Thank you.

While Dolphin (or any other file manager) might have quirks that prevent you from doing something special, AFAIK the basic command “cp” with proper options is not afraid of zero-content-files and copies Firefox-saved pages without problems.
If you want to copy the whole content of a directory, something like the following might be what you need:


cp -R <original_dir>/* <copy_dir>/

The command can copy across different file systems as long as both the source and the destination are supported on the system.
Please check “man cp” for additional options; for instance “–archive” or “–preserve” might be useful in some cases.

ramble ahead: well on my system (TW) dolphin can create and copy 0kb files. weird file names could be result of unicode? (possible that the character encoding has changed between new and old versions?) though things should still copy i would assume. the most important thing you mention is encryption. e.g. encfs and ecryptfs have limitations of filename length (which can change dependent on encoding). perhaps they also have trouble with 0kb files? test copying to raw disk would at least eliminate 1 possibility. i doubt this has anything to do with the fs type.

With Leap 42.2, Dolphin Version 16.08.2, KDE Frameworks 5.26.0, Qt 5.6.1 (compiled against 5.6.1) and the xcb Window system, I can transfer a test directory containing a 0 bytes file named ‘Project_AAA_2014_Jan’ from the local XFS file system via NFS to a remote XFS file system on a 13.2 NFS Server.

The only issue is, the KDE Plasma 5 Dolphin issue mentioned in the Leap 42.2 Release Notes: the Leap 42.2 directory had the “T-bit” set and no world access – the 13.2 directory has the “T-bit” cleared and, has world access.

Removing the remote file, removing the world access on the local file, and then copying it again does not raise any issues.

Removing the remote directory and it’s contents, and then copying (with ‘cp --archive’) the test directory by means of the bash CLI, everything copied as expected with no issues:


 > cp --archive --verbose ./Test_Leap-42.2/ /mnt/xxx/Users/yyy/
'./Test_Leap-42.2/' -> '/mnt/xxx/Users/yyy/Test_Leap-42.2'
'./Test_Leap-42.2/Project_AAA_2014_Jan' -> '/mnt/xxx/Users/yyy/Test_Leap-42.2/Project_AAA_2014_Jan'
'./Test_Leap-42.2/Dolphin_Version' -> '/mnt/xxx/Users/yyy/Test_Leap-42.2/Dolphin_Version'
'./Test_Leap-42.2/.directory' -> '/mnt/xxx/Users/yyy/Test_Leap-42.2/.directory'
 > 

Yes, there are a couple of KDE Plasma 5 Dolphin issues – a KDE Bug Report was raised some time ago – the KDE folks are aware of the issues – and, they’re working on a repair to resolve the issues.

I saved the openSUSE web site (German language variant) to the Leap 42.2 local test directory and then copied it with Dolphin via NFS to the remote 13.2 NFS server.
No issues seen, apart from Firefox not saving the “up” button icon correctly.
On the other hand, the German version of the openSUSE web site doesn’t have any of “ä ö ü ß Ä Ö Ü ^” in the file names.

I created a zero bytes file named “ä ö ü Ä Ö Ü ß ^” on the Leap 42.2 system. Dolphin copied it to the 13.2 NFS Server without any complaints.

I was about to suggest that “rsync” be used to synchronise the directory contents.

  • Can you please post the “rsync” options the script uses?

The option list needed to make “rsync” behave as required for the file set in question, can be quite long.

Please see my answer above – there is a known issue with KDE Plasma 5 Dolphin.

Please see my answer above.
Can you please tell us which character set and human language are displaying this issue with web sites being saved by Firefox?

I was going to recommend “rsync” but, please see my answer above.

By comparing the checksums.

My examples above all copy directories and files between XFS files systems – the behaviour may be different for Btrfs and/or ext4 file systems.

The terabyte issue is a bandwidth issue: even with (10) Gigabit Ethernet the time needed to transfer a terabyte or two will be constrained by the read and write speeds of the disk drives concerned plus, the bandwidth of the hardware channels transferring the data.

You can do a very fast disk copy by cloning… Use something like Clonezilla (there are many other apps that can clone partitions or disks). Cloning (bit level copying) is very fast because it avoids the overhead of individual files. Note that cloning is bit-level so although fairly reliable, it’s not perfect. You’ll need to do a checksum to verify integrity(some cloning apps will also do a checksum compare). So, for example I’d expect that 1TB disks cloned over a machines internal bus might be copied within an hour or two, if there are many small files then copying individual files might take 7 hours or more. Copying over a network, particularly if it’s not gigabit can be much longer, so should be avoided if possible.

Check up on what you need to decrypt files on the target disk, it’ll depend on your method of encryption.

When copying extremely large file trees, I generally break down the copying operations into something manageable, like about 50-200GB at a time (Again, this can depend on numbers of, and sizes of files), and of course this also depends on any other disk activity that might be happening at the same time.

HTH,
TSU

Thank you for a thorough reply.

I can’t explain exactly what I did, it was probably a dolphin copy and paste, followed by an rsync.

The zero size file copy and paste problems might have happened when I booted opensuse V13.2 and then dolphin copy and pasted to the leap42.2 disk.

rsync had the bad character in the file name issue. the problematic website name contains the character “|” (broken vertical line).

The rsync script, as requested.

  

rsync --archive --checksum --verbose --delete --prune-empty-dirs --progress  --stats   --include='*.*'    --log-file='/home/david/Eight_Copy_Log.txt'   /home/david/___David   /run/media/david/fc5c263e-a647-4797-81e9-face656a625f/david

 

With my fresh installed leap42.2, dolphin copies empty files correctly.

Copy via the network or copy to the Leap 42.2 disk connected hardware-wise (S-ATA, IDE, or USB) to the 13.2 system?

Just for fun, I created the following file names (with spaces, üÜöÖäÄߧ, and “|”) with “touch” (size == 0 bytes) in a Leap 42.2 test directory without world access and with the “T” bit set, and then successfully copied the test directory with Dolphin via NFS to my 13.2 system:


 > ls -1
üäöß | ^°123§$< | hdfkhg> | }´`#334
üöäß_ÜÖÄ | ^°§123&
 &gt; 

Archiving these example files with ‘rsync’ to NAS on the local LAN via NFS was also successful (the T-bit on the directory was also correctly archived):


 > rsync -C8rlptoDhW --progress --stats /home/Users/xxx/Test_Leap-42.1 /mnt/NAS/homes/xxx/yyy/home
sending incremental file list
./
.directory
             58 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=2/4)
üäöß | ^°123§$< | hdfkhg> | }´`#334
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=1/4)
üöäß_ÜÖÄ | ^°§123&
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=0/4)

Number of files: 4 (reg: 3, dir: 1)
Number of created files: 3 (reg: 3)
Number of deleted files: 0
Number of regular files transferred: 3
Total file size: 58 bytes
Total transferred file size: 58 bytes
Literal data: 58 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 366
Total bytes received: 76

sent 366 bytes  received 76 bytes  884.00 bytes/sec
total size is 58  speedup is 0.13
 &gt; 

My ‘rsync’ shell scripts normally include the following ‘rsync’ options: “-C8rlptoDhW”:

  • auto-ignore files in the same way CVS does
  • leave high-bit chars unescaped in output
  • recurse into directories
  • copy symlinks as symlinks
  • preserve permissions
  • preserve modification times
  • preserve owner (super-user only)
  • same as --devices --specials
  • output numbers in a human-readable format
  • copy files whole (w/o delta-xfer algorithm)

Therefore, issue resolved.