Kmail filters and duplicated emails

[DavidL]

I’m trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked “unread” and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.

A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?

One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes “format=flowed”. This is just a single sed command followed by exit, and the “stop if filter matches” option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?

DL

[dcurtisfra]

@dlochrin

I’m trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked “unread” and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.

Same sort of thing happens to me on Leap 15.0 and especially recently, like today, following some patches which arrived over the last week or so …
I haven’t tried your solution – I tend to use the solution mentioned below.

@dlochrin

A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?

It’s a mixture – there’s a background database which mostly handles indexing and other things plus, a directory structure located by default under ‘~/.local/share/’.
And yes, sometimes “moved” e-Mails appear OK in the KMail window but are not to be found in the directory structure …
My solution to get the KMail GUI to behave properly is:

1. Close the KMail GUI window;
2. From a CLI Konsole terminal window run “akonadictl fsck” and then “akonadictl vacuum”.

What I’ve noticed with “moving” e-Mails is, that selecting more than 1 e-Mail to be moved, causes the symptoms you’re describing – moving the e-Mails 1-by-1 seems to work just fine, except that the moved e-Mail is not present in the ‘.local/share/’ directory – and therefore, presumably, “somewhere in the Akonadi database” …

@dlochrin

One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes “format=flowed”. This is just a single sed command followed by exit, and the “stop if filter matches” option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?

I suspect that, the KDE documentation is the authoritative answer – yes, the program (script) is executed synchronously: <https://docs.kde.org/stable5/en/pim/kmail2/filters.html>.

Pipe Through
This will feed the message to a program: if the program returns output, the entire message (including the headers) will be replaced with this output; if the program does not return output or exits with a return code other than 0 (indicating an error occurred), the message will not change. Specify the full path to the program. The same substitutions (%n, %{foo} as with execute command are performed on the command line.

**Warning**

Be cautious with this action, as it will easily mess up your messages if the filter program returns garbage or extra lines.

[DavidL]

@dcurtisfra

Same sort of thing happens to me on Leap 15.0 and especially recently, like today, following some patches which arrived over the last week or so …
…]
What I’ve noticed with “moving” e-Mails is, that selecting more than 1 e-Mail to be moved, causes the symptoms you’re describing – moving the e-Mails 1-by-1 seems to work just fine, except that the moved e-Mail is not present in the ‘.local/share/’ directory – and therefore, presumably, “somewhere in the Akonadi database” …

Thanks for your response! I was beginning to wonder whether I was chasing a phantom.

The issue I described above is part of another problem concerning filters: I’m now quite sure that my filters don’t always work properly if Kmail is started during the boot process. Typically, a filter which should have been actioned on a particular email has not resulted in the appropriate action, and manually applying the filter subsequently has no effect. But if I stop Kmail, log out and back in again, then restart it when the system is fully up, the filter(s) works properly.

Your comment about moving more than one email at a time seems familiar. And I’ve seen instances where the GUI displays the bodypart text from another email.

I notice there’s a variable in Kmail’s ‘advanced’ account setup which enables download pipelining. This can be set manually and it’s also set if the server announces it supports pipelining when the autoDetect button is clicked, but I’m wondering how far pipelining extends into message handling. For example, is filtering pipelined? Could these problems result from a code module which isn’t reentrant? I might disable pipelining to see if that improves matters.

What do you think?

DL

[DavidL]

With reference to my point about having to start Kmail manually after logging out and back in again in order for filtering to work correctly, the following sounds as though it could be very relevant:

@nrickert

I still have Leap 42.3 on one of my computers, though I rarely use it.

When I last used it (to update in early November), Plasma was malfunctioning. I logged out and logged in again, and all was fine.

My experience is that if I delay login after boot for long enough, then this doesn’t happen. If I login too quickly after boot, it does. I think there’s a timing issue when the system is still busy from booting up.

I have not had this happen on 15.0. But maybe something like that is happening for you on 15.0. If so, it is probably a Plasma bug to have anything where the timing is that critical.

DL

[karlmistelberger]

@dlochrin

I’m trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked “unread” and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.

A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?

One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes “format=flowed”. This is just a single sed command followed by exit, and the “stop if filter matches” option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?

Filtering with works fine with Kmail. I use formail for processing messages. In case of trouble you may try one or each of the following:

• Restart kmail
• Restart akonadi
• Repair data base

Multiple moves are prone to generating items without RID:

karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
Found 2 external files.
Found 2 external parts.
Found no unreferenced external files.
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Found 4 collections without RID.
Found 11 items without RID.
Found 0 dirty items.
karl@erlangen:~> ps ux|grep mysql
karl     26668  4.5  0.4 3397832 151976 ?      Sl   05:59   0:02 /usr/sbin/mysqld --defaults-file=/home/karl/.local/share/akonadi/mysql.conf --datadir=/home/karl/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-karl.QAPLBp/mysql.socket --pid-file=/tmp/akonadi-karl.QAPLBp/mysql.pid
karl     26969  0.0  0.0   6680   804 pts/1    S+   06:00   0:00 grep --color=auto mysql
karl@erlangen:~> mysql --socket=/tmp/akonadi-karl.QAPLBp/mysql.socket
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 39
Server version: 10.2.18-MariaDB openSUSE package

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB (none)]> use akonadi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
Query OK, 11 rows affected (0.01 sec)

MariaDB [akonadi]> Bye
karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
Found 2 external files.
Found 0 external parts.
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/93/434193_r0
Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/94/434194_r0
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Found 4 collections without RID.
Found 0 items without RID.
Found 0 dirty items.
karl@erlangen:~> find .local/share/akonadi/file_db_data/ -type f|xargs rm
karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
Found 0 external files.
Found 0 external parts.
Found no unreferenced external files.
Found 0 parts to be moved to external files
Found 0 parts to be moved to database
Found 4 collections without RID.
Found 0 items without RID.
Found 0 dirty items.
karl@erlangen:~> 

The above renders moving, copying and deleting folders unnecessary and clears file_db_data.

[dcurtisfra]

@dlochrin

I notice there’s a variable in Kmail’s ‘advanced’ account setup which enables download pipelining. This can be set manually and it’s also set if the server announces it supports pipelining when the autoDetect button is clicked, but I’m wondering how far pipelining extends into message handling. For example, is filtering pipelined? Could these problems result from a code module which isn’t reentrant? I might disable pipelining to see if that improves matters.

AFAICS, KMail2 pipelining for received e-Mails isn’t well documented – I suspect that some code reading is required to understand what it does …Personally, I haven’t enabled the feature, mainly due to the uncertainty and because I’ve absolutely no idea if my ISP supports it – they (the ISP) have absolutely no documentation stating whether or not they support receive pipelining …

Only one thing is fairly certain, it ain’t the RFC 2920 SMTP Service Extension for Command Pipelining …

[DavidL]

I think the problems discussed here are very likely to be another instance of “Bug 283682 - KMail duplicates filtered messages” on the KDE bug-tracking system - see https://bugs.kde.org/show_bug.cgi?id=283682 I’ll add the appropriate reference to my own bug report there.

@karlmistelberger

MariaDB (none)]> use akonadi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
Query OK, 11 rows affected (0.01 sec)

MariaDB [akonadi]> Bye

DO NOT TRY THIS AT HOME!!!
Most of my mail archive disappeared with the null RIDs. Fortunately I had archived my email folders only 20 days ago and the damage didn’t seem to affect emails received since then, so I’ve now recovered.

DL

[karlmistelberger]

@dlochrin

DO NOT TRY THIS AT HOME!!!
Most of my mail archive disappeared with the null RIDs. Fortunately I had archived my email folders only 20 days ago and the damage didn’t seem to affect emails received since then, so I’ve now recovered.

Never experienced data loss when repairing the database as described above. However I always backup the data as it could happen in the future.

[dcurtisfra]

@dlochrin

I think the problems discussed here are very likely to be another instance of “Bug 283682 - KMail duplicates filtered messages” on the KDE bug-tracking system - see https://bugs.kde.org/show_bug.cgi?id=283682 I’ll add the appropriate reference to my own bug report there.

AFAICS, the KMail and Kontact maintainers are currently doing a much better job of coming to grips with old issues – I’m not sure but, I suspect that they’re beginning to use tools such as “Valgrind” to come to grips with issues such as this one. We need to patient and, support them with information such as that being raised here …

[DavidL]

@dcurtisfra

We need to patient and, support them with information such as that being raised here …

I’ve done so, see Bug 400770 - https://bugs.kde.org/show_bug.cgi?id=400770

DL

[davidcbryant1951]

@karlmistelberger

  [big snip ...]
MariaDB (none)]> use akonadi;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
Query OK, 11 rows affected (0.01 sec)

MariaDB [akonadi]> Bye
[snip some more] 

The above renders moving, copying and deleting folders unnecessary and clears file_db_data.

Not being at all familiar with “MariaDB”, I was a little confused by one thing. The conversation (for me) actually read

...
MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
Query OK, 2 rows affected (0.01 sec)

**MariaDB [akonadi]> \q
Bye**

(BASH command prompt re-appeared)
...

Other than that, the procedure worked well. For a little while I had a “phantom” unread e-mail message in the affected folder (KMail reported one unread message, which was not really there), but that problem went away the next time I moved a real message into the affected folder. Thanks, Karl!