Dolphin losing files when copying many files or large folders

English Applications
1

I’ve discovered that Dolphin seems to lose random files when copying many large folders.

I first noticed this a few months ago when I tried to copy my music library from one folder to another on the same HDD. It consisted of around 600 folders and 6500 files. During the copy there were no errors but after the copy I found that some of the newly copied folders were missing files. I put it down to human error or a glitch.

Yesterday I tried to copy 13 folders containing rips of some of my DVDs. Each folder basically had one film of either 700MB or 1.4GB. Again no errors showed up during the copy but I found 3 of the newly copied folders were empty.

This is very worrying. It’s not so critical with music or films but I can’t afford to lose work data like this.

Has anyone experienced or seen a similar problem with Dolphin? I’m going to have to do some more extensive testing but this is not good.

The first time I noticed the problem I was running KDE4.3.4 (I think) and now the latest was with KDE4.4.0.

2

This is bad news. Have you filed any bug?

Personally, I never use GUI for copying large number of files. Still can’t get mentally adjusted with those little GUI windows that show the status of the copy process.

3

Did a test, copied entire music collection to USB disk (731 folders, 15781 files) with Dolphin. No trouble at all.

But it is scary. Can you reproduce this? Or could it be, that a HDD is degrading?

4

Starting dolphin via command line and copying some large folders could give us some interesting output.

5

I’ll start doing some tests during the week. I haven’t filed any bug yet as I want to try reproduce it reliably if possible. I’m sure I saw a bug for something similar when I first came to openSUSE last year but I cannot find anything in bugzilla. It may just have been a rant by someone who didn’t like KDE4 I can’t really remember any more.

Good suggestion gropiuskalle. Hopefully I’ll see something useful in the output.

Can someone suggest an app / script which can do a compare of the original and copied directories to verify the copied folders and files are all present and correct?

6

This is both good and bad. Good that I finally have validation for problems I had encountered with dolphin which some in the community had called me troll/FUD for stating and bad that another is experiencing the problem. Here is what I have found with my system and my workarounds:

  1. Dolphin does not always show the files which were moved, copied or changed in real time.
    Thusly, if you copy files or folders as groups, not all will appear at the destination right away. Close Dolphin and reopen and sometimes they will now show. other times it takes a reboot before they show. Other times you have to re-copy the files
    When you change a file a back-up is normally created but does not show up immediately all the time. If you immediately try to open another file for editing (and the back-up hasn’t shown) you will at times suddenly get a box asking how the backup should be opened instead of the file you requested. You simply cancel the open and re-select the file you want.
    When you move files Dolphin will on occasion delete the file from the source location and fail to add it in the destination resulting in a lost file(s). Resolve, use CLI or midnight commander or another file manager at all times for moves!!

IMHO someone has dropped the ball here. Dolphin did not exhibit this behavior before when using 11.1 32bit/KDE 4.3.1 but does when using 11.1 64bit/KDE 4.3.1, 11.2 32/64bit/KDE >=4.3.1.

7

Do they show up if you hit the reload?

8

I can’t say that I ever experienced what you describe. In my case the files were definitely missing. The problem is that it seems to be random when the files get missed. I just tried copying a the same 13 folders containing the dvd rips and they all copied properly this time.

I wonder if it has anything to do with what other activities are being performed on the PC or file system at the time. I’ll set up a few test during the week and see if I can reproduce the problem.

9

I thought I’d better sign up and chime in on this one as I believe the problem goes deeper than Dolphin and openSUSE (I’m a Linux Mint user.) I lost some files months ago when I copied and pasted folders to a backup drive and failed to notice that all the files hadn’t made it across. I figured I’d screwed up. But I’ve verified the copies since and I just saw it happen again. Not a matter of the file count not updating immediately – the files and folders were definitely not written. So I tried the same operation in Krusader (in root mode, too.) It happened again! The number of files and folders that were missing changed, so I don’t see a pattern. So this isn’t limited to Dolphin. But I should also mention that in all cases I was writing to an external USB Seagate drive. I haven’t tried duplicating folders on the same hard drive. Oh, I’m using EXT4. I wonder if it could be some weird journaling goof or something.

10

I have also experienced the same problem suse_tpx60s et al have described, when using either Dolphin or Konqueror on suse 11.2. I had the impression that neither could handle complex directory trees but I have not done any hard analysis.

It is a little sad, since Konqueror (under KDE 3.5.9) on suse 11.0 did not appear to have this problem and was a godsend. When I now copy large complex directories, I always ask Dolphin to calculate the total directory size (down to the last byte) and the number of files and directories for both input and output before relaxing.

Strangely, a number of well know third party apps on Windows seem to exhibit the same behaviour such as e.g. Ipswich (?) FTP LE.

I use ReiserFS for most of my Linux file systems so it is nothing to do with Ext4.

11

I copy large folders with many files as a daily routine. Never seen loss of files whilst moving or copying.
File/folder count means nothing to me, if you don’t explain how you count.
If refering to external disks: seen this happening when people disconnect the drive too soon.

12

No they do not show up for me upon reload. It takes a complete shutdown of dolphin or a reboot. Problem still exists 11.2 32/64 bit

13

I noticed the problem first when I was editing 45000 files one right after the other. dolphin didn’t show the back-ups all the time but when I would go to open the next file I would get the back-up copy instead and have to close out.

Then later when I went to copy files to new folder files would go missing. Copy to external usb harddisk was even more problematic forcing CLI ‘cp’ and lots of verify files were copied. Since 11.1 I now never use ‘mv’ with multiple files even in dolphin.

14

I haven’t done any large copies of folders recently but must add that I experienced the problem with external USB drives. These drives are permanently connected to the PC (as back-up and storage drives) so it’s not a case of removing or disconnecting them before they were ready.

It’s very un-nerving not being sure if everything copied across properly and having to re-verify all large and complex copies and moves.

15

Editing 45000 files, one after another? Mmmm, not scriptable ?

If it’s about the same folders/files, I prefer ‘rsync’ or some GUI that uses it. It’s flexible, secure, stable and gives lots of options to verify the copy actions.

16

Unfortunately not, these were xhtml documents with built-in style sheet references which were different on each page. I would have needed to open the page, find out which styles, which scripts, which <div…> … </div> were used so as to choose which script to use to fix, so faster to manually do the cut copy paste moves to get the pages working in a consistant way.:cry: but at least I got them done eventually.

17

I also experienced such a problem using Kubuntu 10.04.

Not all files are copied if I copy 370 files from a samba server using konqueror. If I try again, some other files are missing.

Copiing from the same share using the cp command, everything works fine.

This is a very serious bug as data is lost.

Ernst

18

I meant Dolphin instead of Konqueror. I did not try using konqueror.

19

Don’t mean to be rude, but this may be important, so… BUMP! :wink:

I was wondering if anyone knows of any progress on this front. I am not a SuSE user right now (although I was from version 9.1 to 10.2), but I can confirm this problem in Mint 9 and Mint 10 with KDE installed. It is really disturbing. I was going to switch to SuSE because I was hoping it didn’t have this problem, but now I might as well stick with Mint.

In my latest run-in with this problem happened on Mint 10 with KDE installed, I was trying to /move/ a folder with source code, o-files, and Doxygen documentation ( inheritance maps, highlighted source pages, the whole shebang). This amounted to about 3 GB and 40,000 files. I was moving from an NTFS partition to my home directory which is Ext4.

While the files were moving, Xorg was taking 100% of the cpu (with a niceness of 0, :’( ) and it bogged down my computer to where it was unusable. After the job “finished” there were several hundred files left over in the original directory that were not moved.

I dragged the leftovers to the destination again, hit “write into” when it complained about folders already existing, and it finished the job on the second pass.

I don’t think any files were actually lost, it simply “did not get to them”.

I also ran into very similar situations while copying this type of folder onto an external HDD (also NTFS) (this was using Mint 9 KDE), except then it was really annoying because I had no idea the folder didn’t copy all the way through and I came to work missing a bunch of files.

I think I also had similar problems copying into a FAT32 flashdrive.

If anyone wants me to run any tests / give logs, please ask, I really would like to help get this fixed, but I’m not enough of a Linux guru to know what to do here. I honestly don’t even know where to submit a bug report about something like this since it’s unclear whether its a KDE issue or goes deeper than that.

Here are my system specs:
AM2 Athlon X2 5400+
2G RAM
GeForce 8600, proprietary NVIDIA driver
1TB HDD, partitioned into pieces around 200-250G
Win7 and Mint 10 (ext4) dual-boot, GRUB
KDE 4.5.1
I use dolphin for file management.

20

With my first comment in this thread I asked for some output by starting Dolphin via command line.

Without any output, error-messages or logfiles all this is just a quizshow.