huge masses of cache files in KDE 4 slow down system - 18544 files in /home/(user)/.thumbnails

hi all,

I’m just doing some backup of the data in my user directory /home/(user)/ prior to some installation of a more recent version of openSUSE.

When trying to just copy all (hidden and non-hidden) data from there (only 6GB) it tells me that that would take more than 19 hours.

By that way I discovered that there are more than 18500 thumbnails of partly quite old private photos in (invisible) directory .thumbnails in /home/(user)/ ,
which seemed to have been stored during previews of these photos using KDE 4, but never removed.

Because of the great number of files, entering that directory through KDE (which results in the listing of the files) takes much time, which on my PC (Pentium III) will be more obvious than on faster systems.

Are there more features of that kind in KDE 4 ?

E.g. during copying the system anyway hangs from time to time (sometimes even the clock stops counting the seconds).

I’m running KDE 4.3.5 / openSUSE 11.2.

Is this resolved in later versions of KDE 4 / openSUSE ?

Or are there any tools like ‘springcleaning’ to remove obsolete files under KDE / openSUSE ?

Best wishes
Mike

So thumbnails in KDE 4 are kept in the hidden folder (where ~ is your home folder name)

~/.thumbnails

I find two folders within called large and normal. WIth no program running that creates them, you can remove any of the png files held within. If needed, they will just be recreated for you.

In Gwenview, as it can be used to create these, you can set it to: Configure -> Advanced -> Cache -> delete on exit, which will reduce the left over png files done by default. I found one person’s description on when/how these are created for you:

From the source code it seems as though the preview handler does the following checks to see if a cached thumbnail can be loaded:

  • is there a .thumbnail folder
  • is there a png-file in that folder with the correct name (an md5 hash of the path to the movie)
  • does the png contain a chunk (metadata) containing the correct timestamp
  • does the png contain a chunk with the full path to the movie

Thank You,

On 2012-05-13 17:46, ratzi wrote:
> Because of the great number of files, entering that directory through
> KDE (which results in the listing of the files) takes much time, which
> on my PC (Pentium III) will be more obvious than on faster systems.

I don’t know how to limit the number of thumbnails, but you can browse it
with another browser: use “mc”, midnight commander. Text mode. You can
erase all those thumbnails.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On Sun, 13 May 2012 21:36:02 +0530, jdmcdaniel3
<jdmcdaniel3@no-mx.forums.opensuse.org> wrote:

>
> ratzi;2462398 Wrote:
>> hi all,
>>
>> I’m just doing some backup of the data in my user directory
>> /home/(user)/ prior to some installation of a more recent version of
>> openSUSE.
>>
>> When trying to just copy all (hidden and non-hidden) data from there
>> (only 6GB) it tells me that that would take more than 19 hours.
>>
>> By that way I discovered that there are more than 18500 thumbnails of
>> partly quite old private photos in (invisible) directory .thumbnails in
>> /home/(user)/ ,
>> which seemed to have been stored during previews of these photos using
>> KDE 4, but never removed.
>>
>> Because of the great number of files, entering that directory through
>> KDE (which results in the listing of the files) takes much time, which
>> on my PC (Pentium III) will be more obvious than on faster systems.
>>
>> Are there more features of that kind in KDE 4 ?
>>
>> E.g. during copying the system anyway hangs from time to time
>> (sometimes even the clock stops counting the seconds).
>>
>> I’m running KDE 4.3.5 / openSUSE 11.2.
>>
>> Is this resolved in later versions of KDE 4 / openSUSE ?
>>
>> Or are there any tools like ‘springcleaning’ to remove obsolete files
>> under KDE / openSUSE ?
>>
>> Best wishes
>> Mike
>
> So thumbnails in KDE 4 are kept in the hidden folder (where ~ is your
> home folder name)
>
> ~/.thumbnails
>
> I find two folders within called large and normal. WIth no program
> running that creates them, you can remove any of the png files held
> within. If needed, they will just be recreated for you.
>
> In Gwenview, as it can be used to create these, you can set it to:
> Configure -> Advanced -> Cache -> delete on exit, which will reduce the
> left over png files done by default. I found one person’s description
> on when/how these are created for you:
>
>> From the source code it seems as though the preview handler does the
>> following checks to see if a cached thumbnail can be loaded:
>>
>> - is there a .thumbnail folder
>> - is there a png-file in that folder with the correct name (an md5 hash
>> of the path to the movie)
>> - does the png contain a chunk (metadata) containing the correct
>> timestamp
>> - does the png contain a chunk with the full path to the movie
>

i think it would be better to leave those cache folders (and links; under
~/.kde4/ there’s links to various KDE folders under /tmp) out of your
backup.

if you do your backup manually, you only need a list of folders that don’t
have to be backed up. if you use rsync, it’s pretty much the same –
either from the command line, or via exclude file(s). i haven’t used any
dedicated backup applications, but believe they all allow you to exclude
directories or file/type patterns.

depending how often you do these backups, if you empty all caches every
time, KDE will have to re-create those and take it’s time to do so.


phani.

On 2012-05-13 18:31, phanisvara das wrote:
> depending how often you do these backups, if you empty all caches every
> time, KDE will have to re-create those and take it’s time to do so.

Yes, but as the cache does not timeout, it can keep thumbnails of files
that are no longer in the system. I don’t know what it does with files that
were moved elsewhere: are the thumbnails retained, or cleared somehow?


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On Sun, 13 May 2012 22:18:06 +0530, Carlos E. R.
<robin_listas@no-mx.forums.opensuse.org> wrote:

> On 2012-05-13 18:31, phanisvara das wrote:
>> depending how often you do these backups, if you empty all caches every
>> time, KDE will have to re-create those and take it’s time to do so.
>
> Yes, but as the cache does not timeout, it can keep thumbnails of files
> that are no longer in the system. I don’t know what it does with files
> that
> were moved elsewhere: are the thumbnails retained, or cleared somehow?
>

some of the stuff actually resides under /tmp, and that’s cleared out
whenever you tell it to.

re. non-/tmp cache files under ~/.kde4, i don’t know. but i never noticed
the space increase into unmanageable proportions.

i don’t really care; these files don’t need to be backed up, so i leave
them out. they don’t disturb me where they are, but if you want to make
sure not to waste a few MB, you can clean them on the second friday of
every month, or whatever. (backups should happen much more frequent than
that, of course.)


phani.

I never back up .cache or .local because all the things in them can easily be recreated; as they are normally recreated one at a time as you re-use them, the time taken is not significant unless you happen to go over something you did in the past which created a lot of them all at once.

I have never suffered any adverse effects from this policy; others may have had different experiences.

Hi you,

thank you for your advice!

Leaving just the (now) empty folders, I just deleted all those 18500+ thumbnails, which set free 1.2 GB of disk space.

jdmcdaniel3:

thank you for the hint with respect to Gwenview !
I use it for about 1-2 years now.

Many of the (quite old) thumbnails must however been created by previews made using KDE.

phanisvara, john_hudson:

I now left aside all of the hidden folders for my backup, except for one: ~/.thunderbird .
I always had doubts that it would make sense to save mails in a hidden directory,
but that clearly would be a question to mozilla.

The ~/.kde4 folder after a few years of using the system still isn’t that large:
517 files in 87 directories using only 22 MB of disk space.
That’s quite OK!

And Carlos: Yes, truely:
>> Yes, but as the cache does not timeout, it can keep thumbnails of files
>> that are no longer in the system.

I probably may even have some of those thumbnails left from an older version of openSUSE, since - during the installation of newer versions of openSUSE - I used to only delete the system partition, leaving the user partition as is (in order to avoid restoring my data).

Best wishes
Mike

Sounds like you found what you were looking for Mike. We do have a good bunch of guys (and Gals) here. We just need to all row in the same direction. Actually, it is often more like a herd of cats, but who is complaining. I was happy to help and happy to see you get what you needed.

Thank You,

On 2012-05-14 03:36, ratzi wrote:

> I now left aside all of the hidden folders for my backup, except for
> one: ~/.thunderbird .
> I always had doubts that it would make sense to save mails in a hidden
> directory, but that clearly would be a question to mozilla.

You can change where thunderbird really stores its mail. Most hidden
directories are important for something or other, should not be deleted.

> I probably may even have some of those thumbnails left from an older
> version of openSUSE, since - during the installation of newer versions
> of openSUSE - I used to only delete the system partition, leaving the
> user partition as is (in order to avoid restoring my data).

My home directory is the same since 1998. If I migrate to another computer,
I copy my entire home folder, with all the hidden directories. No problem :slight_smile:


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

Hi all again,

thank you all for your helpful advice - and for keeping things going !

But urgh, after deleting the files in ~/.thumbnails, the system was frozen for about 5 min and continiously worked on the HD after passing with the mouse over an entry of a .pdf file.

5 min don’t seem to be that much, but waiting behind the PC they almost feel like ‘never’.

Now after that, I can open .pdf files again quickly, and everything seems to run smoothly.

Carlos:

> My home directory is the same since 1998. If I migrate to another computer,
> I copy my entire home folder, with all the hidden directories. No problem :slight_smile:

Do you have a backup ? :slight_smile:
And are you sure you really need those hidden files ?

Except for ~/.thunderbird, I rather used to take the approach of john_hudson (see above):

> I never back up .cache or .local because all the things in them can easily be recreated; as they are normally recreated one at a time as you re-use them,
> the time taken is not significant unless you happen to go over something you did in the past which created a lot of them all at once.
>
> I have never suffered any adverse effects from this policy; others may have had different experiences.

OK, in the end it will depend on which applications one uses.

All of you, wish you all the best
Mike

On Tue, 15 May 2012 06:06:02 +0530, ratzi
<ratzi@no-mx.forums.opensuse.org> wrote:

> And are you sure you really need those hidden files ?

not really, unless you want to keep your email addresses, messages,
calendar entries, and application & system configurations.

you can delete all that (all hidden files, “dot” files & directories),
and your programs will still run – but they’ll start from scratch.

in some cases that’s desired, when things got so messed up that nothing
works anymore, but usually it’s better then to create a new user, and when
whatever problem one was having doesn’t happen with that new user, see
what’s different between the new and old users’ configurations.


phani.

On 2012-05-15 02:36, ratzi wrote:

> Now after that, I can open .pdf files again quickly, and everything
> seems to run smoothly.

There should be a setting somewhere to select the maximum size beyond which
a thumbnail is not created, and instead kde displays an icon. I think you
should lower it.

> Carlos:
>
>> My home directory is the same since 1998. If I migrate to another computer,
>> I copy my entire home folder, with all the hidden directories. No problem :slight_smile:
>
> Do you have a backup ? :slight_smile:

Ugh. Not as often as I should. O:-)

> And are you sure you really need those hidden files ?

Well, they belong to something I configured, so if I delete the file I have
to configure it again. Some configurations are trivial, some are complex.
Days of work from scratch.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

Dear phani,

you’ll probably agree that starting from scratch after every upgrade of an OS may be the usual experience (older versions of applications don’t work anymore, settings are stored in a different place now, etc. etc.), but that just that is very unpleasant on the other hand.

Now, given the vast number of possible applications that may be used under e.g. openSUSE, a ‘whitelist’ of hidden folders to keep may not be the best idea.

Could you give a ‘blacklist’, e.g. a list of hidden folders (and possibly files), that under openSUSE could be deleted without any loss of necessary information - or even with the advantage of increased speed and space ?

Best wishes
Mike

Yes Carlos,

> Days of work from scratch.

That’s the usual experience …

But it would be good if that could be avoided.

Take care
Mike

OK, a specific ‘blacklist’ will probably more depend on the desktop used, like KDE. Gnome, etc., but not on openSUSE itself.

On 2012-05-15 03:46, ratzi wrote:

> Could you give a ‘blacklist’, e.g. a list of hidden folders (and
> possibly files), that under openSUSE could be deleted without any loss
> of necessary information - or even with the advantage of increased speed
> and space ?

I don’t have a list, but usually a folder named “cache” can be deleted. I
usually sort the folders by size, and then I try to estimate if a big one
can be deleted or not. Small directories, I don’t bother, I save them.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On 2012-05-15 03:56, ratzi wrote:
>
> Yes Carlos,
>
>> Days of work from scratch.
>
> That’s the usual experience …
>
> But it would be good if that could be avoided.

Well, that’s why I keep them :slight_smile:

The idea of having a directory or file hidden, is so you do not meddle with
it, leave it alone :wink:


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

Dear Carlos,

yes, that appears to be a reasonable approach.

On Tue, 15 May 2012 07:16:02 +0530, ratzi
<ratzi@no-mx.forums.opensuse.org> wrote:

> you’ll probably agree that starting from scratch after every upgrade of
> an OS may be the usual experience (older versions of applications don’t
> work anymore, settings are stored in a different place now, etc. etc.),
> but that just that is very unpleasant on the other hand.
> Now, given the vast number of possible applications that may be used
> under e.g. openSUSE, a ‘whitelist’ of hidden folders to keep may not be
> the best idea.
> Could you give a ‘blacklist’, e.g. a list of hidden folders (and
> possibly files), that under openSUSE could be deleted without any loss
> of necessary information - or even with the advantage of increased speed
> and space ?

not really; what may not be important for me may be for you, or the other
way around.

in general, upgrading to a new major version of openSUSE (or any other
distro, for that matter) i start over with a fresh HOME directory since,
as you wrote, programs change how they keep things. this way i’ll get a
fresh setup, as it was intended by the developers. i’ve tried it the other
way, too, keeping the whole HOME directory in place, and sometimes it
works; at other times, you get strange inconsistencies or behavior
because of that.

what i find most useful (and that may be different for you) is to keep the
old HOME safe, but create a new one for the new OS. then i copy over my
data files (documents, music, etc.), and selected configuration files,
after renaming those new ‘dot’ directories to “.<whatever>.new”. when
things don’t work well with the old config. files, i can switch back to
the new version.


phani.