Dolphin - How much files are too much?

[suse_rasputin]

Hi!

I access a samba share on a raspi 2b (USB-Sata adapter with WD red 1TB, latest raspbian buster light) with Dolphin from Leap 15.1 (both devices on same gigabit switch). When I try to access a directory with a VERY high number of small files (.jpg from a cam with motion detection), Dolphin times out and I get an error message.

On the raspi I can access the directory via console and if I move a large number of the .jpgs to subdirectories Dolphin can access the directory, so no problem with file system or so.

My question: How much files are “too much” for the Dophin? Any idea?

[deano_ferrari]

I assume you’re just viewing the remote share “casually” via Dolphin smb:// KIO, (as opposed to viewing a mounted samba share?)

In any case, it may well to be down to the Raspi Pi 2b performance impacting here.

[suse_rasputin]

Yepp, no mount, just smb://

What times out with many files? The “fetch” of kio? I have no real idea what Dolphin does when loading a smb:// location…

[deano_ferrari]

I have seen similar reports for other file managers relying on libsmbclient and large directory listings, but don’t have a good handle on it. I don’t have larger directories of files to contend with, so haven’t really noticed any issues.

Does smbclient performance suffer the same way?

smbclient //SERVER/path/to/shared/folder -c "ls *"

FWIW, there was a kde bug discussing inefficient samba comms to get directory listings, now fixed in samba 4.12+ apparently, (and also requires Dolphin 20.04.0+). It might be interesting to use the KDE version using KDE_repositories and samba from network:samba:STABLE to see if this improves things for you. I’ll leave you with that.

[suse_rasputin]

Many thanks for reply!

I tried to access the same share from Dolphin via smb://… on a TW install (samba and Dolphin have there the 4.12+ 20.4+ versions recommended in your final post).

But I also get

Timeout on server
192.168.111.108

with smbclient I get a cryptic error

tree connect failed: NT_STATUS_BAD_NETWORK_NAME

for any subdirectory in the share, can list only the highest directory… oO

[deano_ferrari]

I wonder if forcing SMB3 (server end) might help here? (It tends to be less chatty than the older protocols.) In the [global] section of smb.conf add…

client max protocol = SMB3

and restart samba service. For Debian Buster, do

sudo systemctl restart smbd

See if that helps.

If not, you’d need to investigate the comms with wireshark perhaps.

Side note: Do you really need samba connectivity in your environment? Assuming only Linux hosts, can you use sftp:// instead?

[deano_ferrari]

@suse_rasputin

with smbclient I get a cryptic error

tree connect failed: NT_STATUS_BAD_NETWORK_NAME

for any subdirectory in the share, can list only the highest directory… oO

Yes, I get the same error, but if I specify the sub-directory with the -D option eg “test”

smbclient //SERVER/Downloads/ -D test  -c "ls *"

it succeeds without error, and you can go further down like this…

~> smbclient //server01.local/Downloads/ -D "test/devices" -c "ls *"
Enter WORKGROUP\dean's password: 
  .                                   D        0  Wed Apr 29 12:58:29 2020
  ..                                  D        0  Wed Apr 29 12:58:29 2020
  sample.list                      A      498  Fri Apr 24 18:58:03 2020

                116861054 blocks of size 4096. 72673863 blocks available

[suse_rasputin]

That -D thing in principle works, but on the directory with the large number of files I still get

NT_STATUS_IO_TIMEOUT listing \cam1\*

in the cam1 directory

**ls -1 | wc -l**

gives me a count of 38446 fiels. cough

[deano_ferrari]

@suse_rasputin

That -D thing in principle works, but on the directory with the large number of files I still get

NT_STATUS_IO_TIMEOUT listing \cam1\*

in the cam1 directory

Yes, I only mentioned that as a way to get around the error you reported. It won’t solve the time-out issue of course. It would be interesting to see if using SMB3 helps at all.

**ls -1 | wc -l**

gives me a count of 38446 fiels. cough

That is a very large number of files!

[deano_ferrari]

BTW, did you read my comments/suggestions in post #6?

[deano_ferrari]

Just in case the following is relevant to your situation…
https://wiki.samba.org/index.php/Performance_Tuning#Directories_with_a_Large_Number_of_Files

[suse_rasputin]

@deano_ferrari

BTW, did you read my comments/suggestions in post #6?

Yepp, I though that was already in my config file, double-checked, wasn’t there, added it:

Timeout on server
192.168.111.108

from with Dolphin from Leap 15.1 and TW (updated Saturday…)

[suse_rasputin]

@deano_ferrari

Just in case the following is relevant to your situation…
https://wiki.samba.org/index.php/Performance_Tuning#Directories_with_a_Large_Number_of_Files

Are there any drawbacks of this approach? Don’t want to change anything if not necessary, as currently very busy…

[deano_ferrari]

@suse_rasputin

Are there any drawbacks of this approach? Don’t want to change anything if not necessary, as currently very busy…

It’s not about being necessary, but an optimisation that may help with large directories for the reason outlined below…

Using these settings, all new files on the share are created using lowercase. Samba no longer has to scan the directory for upper- and lowercase. This improves the performance. For further details about the parameters, see the descriptions in the smb.conf(5) man page.