KMail acts strange with moving mails to other folders

You can check physical storage for messages:

karl@erlangen:~> du -csh /home/karl/.local/share/local-mail/inbox/*
344M    /home/karl/.local/share/local-mail/inbox/cur
28K     /home/karl/.local/share/local-mail/inbox/new
4,0K    /home/karl/.local/share/local-mail/inbox/tmp
344M    insgesamt
karl@erlangen:~> 

tmp should be empty, new contains new messages. They should move to cur upon being read. Moving messages between folders should physically move them physically between directories. That is what my instance of kmail does. No reappearing messages whatsoever.

Thank you Karl, I can indeed see all the mail items there. Unfortunately they all have crypted names so I don’t know which one is which one. Sure, I can open them one by one but the weekend is too short for that. :slight_smile:

This what I see for my inbox:
cur 0 items
new 78 items
tmp 0 items
It means messages, although I have read them all, are not moved to the cur directory in the inbox? Kmail shows them as read, just checked. What can cause the mails not to be transferred to the correct folder?

Hello Karl,

I managed to move the mails in my inbox from the new to the cur folder. Lost all my mails while doing so, but happily I made a backup first so restoring them was piece of cake.
Let’s see if the filters work any better now with the mails in the correct folder. I can imagine when they stay in the new folder, the filters keep on working on them giving me a lot of duplicates.
Question is: why were they in the new folder, why not in cur?
I have to write this: when I started with openSUSE this week I copied all my mail from Kmail backup files into Kmail. Could this have caused it?

Hard to figure out what is going on. Symptoms point to a permission problem. Double check permissions and ownership of source and destination directory as well of mail file:

karl@erlangen:~> ls -dl /home/karl/.local/share/local-mail/inbox/new/1538844339.V10303I283c30M773432.erlangen /home/karl/.local/share/local-mail/inbox/new/ /home/karl/.local/share/local-mail/inbox/cur/
drwxr-xr-x 2 karl users 253952  6. Okt 18:16 /home/karl/.local/share/local-mail/inbox/cur/
drwxr-xr-x 2 karl users  28672  6. Okt 18:45 /home/karl/.local/share/local-mail/inbox/new/
-rw------- 1 karl users    494  6. Okt 18:45 /home/karl/.local/share/local-mail/inbox/new/1538844339.V10303I283c30M773432.erlangen
karl@erlangen:~> 

Reading the new message moves it to /home/karl/.local/share/local-mail/inbox/cur/1538844339.V10303I283c30M773432.erlangen:2,S

**[User: jan] @ [Server: linux-yyc4] - [Directory: ~/.local/share/local-mail/inbox]**
**$** ls
total 36
drwxr-xr-x 2 jan users 12288 Oct  6 16:44 **cur**
drwxr-xr-x 2 jan users 20480 Oct  6 16:44 **new**
drwxr-xr-x 2 jan users  4096 Oct  6 16:39 **tmp**

**[User: jan] @ [Server: linux-yyc4] - [Directory: ~/.local/share/local-mail/inbox]**
**$** ls cur
total 6900
-rw-r--r-- 1 jan users    8621 Oct  6 16:39 1538836771043.R637.linux-yyc4:2,S
-rw-r--r-- 1 jan users  118892 Oct  6 16:39 1538836771128.R41.linux-yyc4:2,S
-rw-r--r-- 1 jan users   48032 Oct  6 16:39 1538836771132.R683.linux-yyc4:2,S

Permissions look okay to me.
Well, at the moment, since all the mail messages ar in the cur folder I don’t expect any problem. Just tested it and a message arrived, I could see something flashing in the inbox but the mail was delivered straight to the correct folder and is not in the inbox anymore. When I look in Dolphin I see the folder inbox/new being empty, so it looks like the problem is solved.
Thank you very much for your help. I’m really happy with this cause it caused me a lot of work to get the messages in the correct folder, and only 1 copy of them.

I have this problem too; it’s the issue I was attempting to describe here:

and dcurtisfra noted:

While that almost certainly explains why moving mail between folders results in a new-mail notification, the issue with duplicate emails feels to me like a timing issue, as though the indexer hasn’t caught up. I find that giving Kmail time and then deleting the spurious duplicates seems to clean things up.

But it’s a bug which needs to be fixed with some priority in my opinion because moving around a a swag of new emails can become very confusing, with the liklihood of mistakenly deleting one in the process.

David L.

I agree with you that it is some kind of bug. As Karl wrote the messages don’t belong in the ‘new’ but in the ‘cur’ folder, once they are read. But because they remain in the ‘new’ folder, the program thinks it has to do something with those “new” messages, resulting in duplicated all over the place. I managed to move the messages from ‘new’ to ‘cur’ and now things have stabilized. But I only did this for the inbox, in all other boxes messages are still in ‘new’ but since no filters are working on those messages, nothing goes wrong.

Kmail is currently improving. Many of the well known annoyances are now gone. I did not experience duplicate messages or messages suddenly appearing as unread since a long time. On rare occasions Kmail shows a duplicate entry in the inbox, which can’t be read. But this entry has no physical counterpart in the file system and can only be deleted using akonadiconsole > Browser > Folder > Clear Akonadi Cache.

The following behavior can be problematic, but currently does not cause problems in my local folders (some 30,000 messages in 60 subfolders):

  • When moving messages they go to the destination/new directory instead of destination/cur

  • some messages are displayed as read by Kmail, but they don’t have the :2,S suffix. This may be fixed by clearing the akonadi cache as already mentioned above. Being invoked again, Kmail will eventually display them as unread, but after some time elapsed they will be marked as read an given the proper suffix automatically.

Just received an e-mail for which I had no filter setup, so the mail remained in the Inbox.
When I clicked it to read it, the colour changed from blue (unread) to black (read).
The file on disk however remained in the folder Inbox/new, was not moved over to Inbox/cur and did not get the extension :2,S
I then created a filter (thought I had that one already but no), ran the filter which made the mail go to the desired folder, but it also staid in /Inbox/new. I can understand that, since it had to go from /Inbox/cur to the desired folder, but since it is in Inbox/new it just stays there.
I think the error lies in the red line above. But what is causing this?

Apart from the fact that I do move messages to folders ( Mailing lists from my @o.o account do get filtered to their own folders as an exception ), I can confirm the other things you’re writing re. Kmail / akonadi. I also only use IMAP accounts these days. The reason why I don’t use subfolders anymore, is that tagging them does the job ( at least ) as good.
Another thing that I have seen causing issues with moving messages / folders aroung in kmail, is the generation of huge ammounts of no-RID messages, In my case the folder ‘file_lost+found’ in ~/.local/share/akonadi was missing (?) , the number of no-RID messages increased. Creating the folder and running ‘akonaditcl fsck’ then took a minute, but it worked. The first thing I noticed was that some duplicates ( one filtered, i.e. moved, one in the original , gone after akonadictl restart ) had disappeared and that no new duplicates occurred.
FWIW, I’m quite pleased with the state Kmail / akonadi is in at the moment. A reliable workhorse for me.

Well, the contents of my inbox/new is:

karl@erlangen:~> ll .local/share/local-mail/inbox/new/
insgesamt 8
-rw-r--r-- 1 karl users 6152 20. Sep 2016  1474390356.R55.erlangen:2,S
karl@erlangen:~> 

Kmail seems to append the extension and move the messages since some 2 years without failure. However I refrain from Kmail fetching the messages from the server. This is done by fetchmail and postfix/qmgr. I sent some 75 local mails/second (some 800 mails alltogether) with Kmail processing each of them flawlessly. Tests with remote mail also did not exhibit any problems.

I was just looking in my e-mail backup files, files I create from within Kmail on a daily basis. In there all mail is located in the cur folders, every new folder is empty.
I am thinking of starting new with Kontact/Kmail/Akonadi, setup the accounts and import the messages from the backup to see if they will be placed in the correct folders.
But I will do that tomorrow.

Thanks again for your help.

Okay, this is the situation right now:
Inbox contains 115 messages. Looking at my disk, they are all in /inbox/cur which is good since they are all read.
I received a new one raising the total to 116. When I opened the inbox it was automatically read because it is the first mail in the list and therefore selected. I looked at the disk and yes, 116 messages in cur.
Since I did not have a rule for this one I created one, moving the message to one of my archive boxes. I started the rule and indeed the message disappeared from the inbox and appeared in the archive box, BUT on disk there were still 116 messages in the inbox/cur folder.
I selected the outbox and then the inbox again and the moved mail re-appeared.
I then shift-deleted it and now the folder tells me there are 115 messages.

So, the message was read, it was in the correct /inbox/cur folder, it did transfer to an archive box and still re-appears in the inbox as well.
What am I doing wrong, or what is wrong with my system?

Hard to tell. What is the output of:


karl@erlangen:~> akonadictl fsck 2>&1|grep Found
Found 21 external files.
Found 16 external parts.
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/62/86562_r0
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/64/173864_r0
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/62/88762_r0
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/50/107850_r0
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/84/124284_r0
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Found 3 collections without RID.
Found 0 items without RID.
Found 0 dirty items.
karl@erlangen:~>karl@erlangen:~> find /home/karl/.local/share/akonadi/file_db_data/ -type f|xargs ls -lrt
-rw-r--r-- 1 karl users  7847  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/96/41696_r4
-rw-r--r-- 1 karl users  5243  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/75/48175_r4
-rw-r--r-- 1 karl users  5087  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/77/49277_r4
-rw-r--r-- 1 karl users  6095  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/97/50297_r4
-rw-r--r-- 1 karl users  5087  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/15/50715_r1
-rw-r--r-- 1 karl users  9517  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/05/50705_r4
-rw-r--r-- 1 karl users  5243  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/77/50677_r1
-rw-r--r-- 1 karl users  5295  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/77/52377_r4
-rw-r--r-- 1 karl users  9527  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/51/52951_r4
-rw-r--r-- 1 karl users  5253  8. Okt 13:06 /home/karl/.local/share/akonadi/file_db_data/41/52941_r4
-rw-r--r-- 1 karl users  5669  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/04/173904_r1
-rw-r--r-- 1 karl users  5457  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/06/173906_r1
-rw-r--r-- 1 karl users  5501  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/08/173908_r1
-rw-r--r-- 1 karl users  5747  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/10/173910_r1
-rw-r--r-- 1 karl users  9521  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/14/173914_r1
-rw-r--r-- 1 karl users  5677  8. Okt 13:21 /home/karl/.local/share/akonadi/file_db_data/18/173918_r1
-rw-r--r-- 1 karl users 36226  8. Okt 14:08 /home/karl/.local/share/akonadi/file_db_data/64/173864_r0
-rw-r--r-- 1 karl users 22864  8. Okt 14:09 /home/karl/.local/share/akonadi/file_db_data/62/86562_r0
-rw-r--r-- 1 karl users 24765  8. Okt 14:10 /home/karl/.local/share/akonadi/file_db_data/84/124284_r0
-rw-r--r-- 1 karl users  6222  8. Okt 14:10 /home/karl/.local/share/akonadi/file_db_data/50/107850_r0
-rw-r--r-- 1 karl users 64797  8. Okt 14:11 /home/karl/.local/share/akonadi/file_db_data/62/88762_r0
karl@erlangen:~> 
 

Post only the relevant part of the latter command.

akonadictl fsck 2>&1|grep Found
**Found** 10791 external files.
**Found** 10789 external parts.
**Found** unreferenced external file: /home/jan/.local/share/akonadi/file_db_data/05/66105_r0
**Found** unreferenced external file: /home/jan/.local/share/akonadi/file_db_data/07/66107_r0
**Found** 0 parts to be moved to external files
**Found** 0 parts to be moved to database
**Found** 67 collections without RID.
**Found** 11693 items without RID.
**Found** 0 dirty items.

find /home/jan/.local/share/akonadi/file_db_data/ -type f|xargs ls -lrt                                                                                               
-rw-r--r-- 1 jan users     4883 Oct  4 16:38 /home/jan/.local/share/akonadi/file_db_data/30/27830_r0
-rw-r--r-- 1 jan users   291005 Oct  4 16:38 /home/jan/.local/share/akonadi/file_db_data/48/27848_r0
-rw-r--r-- 1 jan users   217237 Oct  4 16:38 /home/jan/.local/share/akonadi/file_db_data/51/27851_r0
-rw-r--r-- 1 jan users   229138 Oct  4 16:38 /home/jan/.local/share/akonadi/file_db_data/63/27863_r0

I stopped after the 4th line cause each line has the same permissions.

Most puzzling is Found 67 collections without RID. I guess the folders restored from backup are not welcome by akonadi.

  • Make a sanity check: Set up a test user with pristine storage. Don’t reuse an existing home directory. Try creating some folders, sending mail and moving messages between folders. Make sure you understand what is going on and what is going on is what you want to to have.
  • Only then add a new local folder account by specifying the path to the backup folder and try to view messages directly in local folders of backup. If that works you may try moving messages from backup to local folders in the home directory.

But the fact remains that these are all bugs, and they’re not diminished in importance by the possibility of obscure work-arounds or explanations, or infrequent occurrence.

I’d like to see a stable, bug-free version of Kmail, not one with lots of new “features” and new bugs. Maybe it’s time to review the whole PIM architecture.

David L.

But the fact remains that these are all bugs, and they’re not diminished in importance by the possibility of obscure work-arounds or explanations, or infrequent occurrence.

I’d like to see a stable, bug-free version of Kmail, not one with lots of new “features” and new bugs. Maybe it’s time to review the whole PIM architecture.

David L.

I think you’re not doing anything wrong and there’s nothing wrong with your system. This is exactly what I see. It’s probably a bug in Kmail, though possibly something more than a simple coding error. An outline of Kmail’s program design would probably shed light on what exactly is happening, but I suspect it’s a timing problem associated with indexing and update of the user interface.

David L.

I just sent an e-mail from my work address to my personal address, a message which is filtered and moved into an archive folder.
It was filtered correctly, also it did disappear completely from the inbox.
I checked the filter logs (Tool -> Filter Log viewer) and it did what it needed to do.

Will be continued.