Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Kmail filters and duplicated emails

  1. #1
    Join Date
    Dec 2008
    Location
    NSW, Australia
    Posts
    93

    Default Kmail filters and duplicated emails

    I'm trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked "unread" and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.

    A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?

    One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes "format=flowed". This is just a single sed command followed by exit, and the "stop if filter matches" option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?

    DL

  2. #2
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,846

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dlochrin View Post
    I'm trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked "unread" and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.
    Same sort of thing happens to me on Leap 15.0 and especially recently, like today, following some patches which arrived over the last week or so …
    I haven't tried your solution -- I tend to use the solution mentioned below.

    Quote Originally Posted by dlochrin View Post
    A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?
    It's a mixture -- there's a background database which mostly handles indexing and other things plus, a directory structure located by default under ‘~/.local/share/’.
    And yes, sometimes “moved” e-Mails appear OK in the KMail window but are not to be found in the directory structure …
    My solution to get the KMail GUI to behave properly is:
    1. Close the KMail GUI window;
    2. From a CLI Konsole terminal window run “akonadictl fsck” and then “akonadictl vacuum”.

    What I've noticed with “moving” e-Mails is, that selecting more than 1 e-Mail to be moved, causes the symptoms you're describing -- moving the e-Mails 1-by-1 seems to work just fine, except that the moved e-Mail is not present in the ‘.local/share/’ directory -- and therefore, presumably, “somewhere in the Akonadi database” …

    Quote Originally Posted by dlochrin View Post
    One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes "format=flowed". This is just a single sed command followed by exit, and the "stop if filter matches" option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?
    I suspect that, the KDE documentation is the authoritative answer -- yes, the program (script) is executed synchronously: <https://docs.kde.org/stable5/en/pim/kmail2/filters.html>.
    Pipe Through
    This will feed the message to a program: if the program returns output, the entire message (including the headers) will be replaced with this output; if the program does not return output or exits with a return code other than 0 (indicating an error occurred), the message will not change. Specify the full path to the program. The same substitutions (%n, %{foo} as with execute command are performed on the command line.

    Warning
    Be cautious with this action, as it will easily mess up your messages if the filter program returns garbage or extra lines.

  3. #3
    Join Date
    Dec 2008
    Location
    NSW, Australia
    Posts
    93

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dcurtisfra View Post
    Same sort of thing happens to me on Leap 15.0 and especially recently, like today, following some patches which arrived over the last week or so …
    [...]
    What I've noticed with “moving” e-Mails is, that selecting more than 1 e-Mail to be moved, causes the symptoms you're describing -- moving the e-Mails 1-by-1 seems to work just fine, except that the moved e-Mail is not present in the ‘.local/share/’ directory -- and therefore, presumably, “somewhere in the Akonadi database” …
    Thanks for your response! I was beginning to wonder whether I was chasing a phantom.

    The issue I described above is part of another problem concerning filters: I'm now quite sure that my filters don't always work properly if Kmail is started during the boot process. Typically, a filter which should have been actioned on a particular email has not resulted in the appropriate action, and manually applying the filter subsequently has no effect. But if I stop Kmail, log out and back in again, then restart it when the system is fully up, the filter(s) works properly.

    Your comment about moving more than one email at a time seems familiar. And I've seen instances where the GUI displays the bodypart text from another email.

    I notice there's a variable in Kmail's 'advanced' account setup which enables download pipelining. This can be set manually and it's also set if the server announces it supports pipelining when the autoDetect button is clicked, but I'm wondering how far pipelining extends into message handling. For example, is filtering pipelined? Could these problems result from a code module which isn't reentrant? I might disable pipelining to see if that improves matters.

    What do you think?

    DL

  4. #4
    Join Date
    Dec 2008
    Location
    NSW, Australia
    Posts
    93

    Default Re: Kmail filters and duplicated emails

    With reference to my point about having to start Kmail manually after logging out and back in again in order for filtering to work correctly, the following sounds as though it could be very relevant:

    Quote Originally Posted by nrickert View Post
    I still have Leap 42.3 on one of my computers, though I rarely use it.

    When I last used it (to update in early November), Plasma was malfunctioning. I logged out and logged in again, and all was fine.

    My experience is that if I delay login after boot for long enough, then this doesn't happen. If I login too quickly after boot, it does. I think there's a timing issue when the system is still busy from booting up.

    I have not had this happen on 15.0. But maybe something like that is happening for you on 15.0. If so, it is probably a Plasma bug to have anything where the timing is that critical.
    DL

  5. #5
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,284

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dlochrin View Post
    I'm trying to track down an obscure problem which (when it happens) frequently results in two copies of an email, one of which may be permanently marked "unread" and impossible to delete. The solution is to rename the folder, create a new version, copy everything else to the new folder and delete the original.

    A question, please: When an email is copied to another folder is it physically copied as received? Or is the existence of the copy managed by creating a new entry in some sort of database (or modifying an entry if the email is moved rather than copied)?

    One of my filters pipes the incoming email to a BASH script to remove flowed-format line wraps if Content-Type includes "format=flowed". This is just a single sed command followed by exit, and the "stop if filter matches" option is not checked. Am I correct in believing the script is run synchronously and control returns to the filter processor on exit?
    Filtering with works fine with Kmail. I use formail for processing messages. In case of trouble you may try one or each of the following:

    • Restart kmail
    • Restart akonadi
    • Repair data base


    Multiple moves are prone to generating items without RID:

    Code:
    karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
    Found 2 external files.
    Found 2 external parts.
    Found no unreferenced external files.
    Found 0 parts to be moved to external files
    Found 0 parts to be moved to database
    Found 4 collections without RID.
    Found 11 items without RID.
    Found 0 dirty items.
    karl@erlangen:~> ps ux|grep mysql
    karl     26668  4.5  0.4 3397832 151976 ?      Sl   05:59   0:02 /usr/sbin/mysqld --defaults-file=/home/karl/.local/share/akonadi/mysql.conf --datadir=/home/karl/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-karl.QAPLBp/mysql.socket --pid-file=/tmp/akonadi-karl.QAPLBp/mysql.pid
    karl     26969  0.0  0.0   6680   804 pts/1    S+   06:00   0:00 grep --color=auto mysql
    karl@erlangen:~> mysql --socket=/tmp/akonadi-karl.QAPLBp/mysql.socket
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MariaDB connection id is 39
    Server version: 10.2.18-MariaDB openSUSE package
    
    Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    MariaDB [(none)]> use akonadi;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A
    
    Database changed
    MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
    Query OK, 11 rows affected (0.01 sec)
    
    MariaDB [akonadi]> Bye
    karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
    Found 2 external files.
    Found 0 external parts.
    Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/93/434193_r0
    Found unreferenced external file: /home/karl/.local/share/akonadi/file_db_data/94/434194_r0
    Found 0 parts to be moved to external files
    Found 0 parts to be moved to database
    Found 4 collections without RID.
    Found 0 items without RID.
    Found 0 dirty items.
    karl@erlangen:~> find .local/share/akonadi/file_db_data/ -type f|xargs rm
    karl@erlangen:~> akonadictl fsck 2>&1|grep ^Found
    Found 0 external files.
    Found 0 external parts.
    Found no unreferenced external files.
    Found 0 parts to be moved to external files
    Found 0 parts to be moved to database
    Found 4 collections without RID.
    Found 0 items without RID.
    Found 0 dirty items.
    karl@erlangen:~>
    The above renders moving, copying and deleting folders unnecessary and clears file_db_data.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

  6. #6
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,846

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dlochrin View Post
    I notice there's a variable in Kmail's 'advanced' account setup which enables download pipelining. This can be set manually and it's also set if the server announces it supports pipelining when the autoDetect button is clicked, but I'm wondering how far pipelining extends into message handling. For example, is filtering pipelined? Could these problems result from a code module which isn't reentrant? I might disable pipelining to see if that improves matters.
    AFAICS, KMail2 pipelining for received e-Mails isn't well documented -- I suspect that some code reading is required to understand what it does …
    Personally, I haven't enabled the feature, mainly due to the uncertainty and because I've absolutely no idea if my ISP supports it -- they (the ISP) have absolutely no documentation stating whether or not they support receive pipelining …

    Only one thing is fairly certain, it ain't the RFC 2920 SMTP Service Extension for Command Pipelining …

  7. #7
    Join Date
    Dec 2008
    Location
    NSW, Australia
    Posts
    93

    Default Re: Kmail filters and duplicated emails

    I think the problems discussed here are very likely to be another instance of "Bug 283682 - KMail duplicates filtered messages" on the KDE bug-tracking system - see https://bugs.kde.org/show_bug.cgi?id=283682 I'll add the appropriate reference to my own bug report there.

    Quote Originally Posted by karlmistelberger View Post
    MariaDB [(none)]> use akonadi;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A

    Database changed
    MariaDB [akonadi]> delete from pimitemtable where remoteid is NULL;
    Query OK, 11 rows affected (0.01 sec)

    MariaDB [akonadi]> Bye
    DO NOT TRY THIS AT HOME!!!
    Most of my mail archive disappeared with the null RIDs. Fortunately I had archived my email folders only 20 days ago and the damage didn't seem to affect emails received since then, so I've now recovered.

    DL

  8. #8
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,284

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dlochrin View Post
    DO NOT TRY THIS AT HOME!!!
    Most of my mail archive disappeared with the null RIDs. Fortunately I had archived my email folders only 20 days ago and the damage didn't seem to affect emails received since then, so I've now recovered.
    Never experienced data loss when repairing the database as described above. However I always backup the data as it could happen in the future.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

  9. #9
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,846

    Talking Re: Kmail filters and duplicated emails

    Quote Originally Posted by dlochrin View Post
    I think the problems discussed here are very likely to be another instance of "Bug 283682 - KMail duplicates filtered messages" on the KDE bug-tracking system - see https://bugs.kde.org/show_bug.cgi?id=283682 I'll add the appropriate reference to my own bug report there.
    AFAICS, the KMail and Kontact maintainers are currently doing a much better job of coming to grips with old issues -- I'm not sure but, I suspect that they're beginning to use tools such as “Valgrind” to come to grips with issues such as this one. We need to patient and, support them with information such as that being raised here …

  10. #10
    Join Date
    Dec 2008
    Location
    NSW, Australia
    Posts
    93

    Default Re: Kmail filters and duplicated emails

    Quote Originally Posted by dcurtisfra View Post
    We need to patient and, support them with information such as that being raised here …
    I've done so, see Bug 400770 - https://bugs.kde.org/show_bug.cgi?id=400770

    DL

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •