heap of deferred mails : manually instruct postfix to forward to user's maildir

English Network/Internet
#1

Hi all

I am working on setting our own mail server . At the time being, we use opensuse 12.3_64. Postfix has been set up by yast mail server module, no outgoing mails, local delivery to maildir (at the moment at least).

I know, I should have been more cautous about this and finally better have used a tool like imapoffline to get the mails from our ISP. However while messimg around, in an uncontrolled moment (phone call, then coffee break :-)), I let postfix running and it fetched all mails from my ISP. The inbound mails (sent from external source to our users) could be delivered to the users maildirs. We can live with that at the moment. What is disturbing, is that all outbound mails (mails our users sent to external destinations) are held in the deferred queue. Using all sorts of grepping we can filter message id’s as to find what a relevant message and through postcat -q ID we can see it’s content. But on the long run that cannot be the solution.

Is there a way that we can bring postfix to reque the deferred formerly outbound messages to the users maildir?

To get it more clear:

Assume I got a mail from soccer@fiva.org (inbound message) to me@mydomian. Then I replied to it (outbound message). During my coffe break postfix fetched the inbound message, gave it an ID of F19BA60116E and was able to dump it in my maildir. Also during my coffe break postfix fetched my answer, from me@mydomian to soccer@fiva.org gave it an ID of A1C19601164. Now A1C19601164 dumps araound in deferred queue or will be resent to soccer@fiva.org (how embarassing, given some mails are several years old). So I would love to be able to get postfix to dump A1C19601164 in my maildir. How can I do that?

greez

chris

#2

On 2014-02-27 14:06, seuchato wrote:

> Is there a way that we can bring postfix to reque the deferred formerly
> outbound messages to the users maildir?

“sendmail -q” does it, I think. Also have a look at “postsuper -r
queue_id”.

But the first issue is to find out why those posts were deferred. Issue
“mailq” and it should tell you the list of all messages in transit
somewhere, and why. Unless the root cause is corrected, the above
commands can cause more damage.

> To get it more clear:
>
> Assume I got a mail from soccer@fiva.org (inbound message) to
> me@mydomian. Then I replied to it (outbound message). During my coffe
> break postfix fetched the inbound message, gave it an ID of F19BA60116E
> and was able to dump it in my maildir. Also during my coffe break
> postfix fetched my answer, from me@mydomian to soccer@fiva.org gave it
> an ID of A1C19601164. Now A1C19601164 dumps araound in deferred queue or
> will be resent to soccer@fiva.org (how embarassing, given some mails are
> several years old). So I would love to be able to get postfix to dump
> A1C19601164 in my maildir. How can I do that?

Hum.
I’ll have to reread this later. I just had lunch and I’m feeling too
sleepy. O:-)

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#3

Hi, have nice siesta please :slight_smile:

For the time being, I have copied /var/spool/postfix/deferred to a safe place and deleted most of the files in /var/spool/postfix/deferred.

mailq
...
000C26012B0     8081 Tue Feb 25 17:28:41  MAILER-DAEMON
                                                  (mail transport unavailable)
                                         x@y

0071B6012B1     5401 Tue Feb 25 17:28:42  MAILER-DAEMON
                                                  (mail transport unavailable)
                                        a@b


grep 0071B6012B1 /var/log/mail.info

2014-02-27T15:45:32.521092+01:00 brutus postfix/qmgr[10765]: 0071B6012B1: from=<>, size=5401, nrcpt=1 (queue active)
2014-02-27T15:45:32.681530+01:00 brutus postfix/error[10994]: 0071B6012B1: to=<a@b>, relay=none, delay=166611, delays=166611/0.05/0/0.11, dsn=4.3.0, status=deferred (mail transport unavailable)

tail /var/log/mail.warn

2014-02-27T15:45:32.519141+01:00 brutus postfix/qmgr[10765]: warning: connect to transport private/smtp: Connection refused

/var/log/mail.err is empty

 postcat 0071B6012B1
*** ENVELOPE RECORDS 0071B6012B1 ***
message_size:            5401             217               1               0            5401
message_arrival_time: Tue Feb 25 17:28:42 2014
create_time: Tue Feb 25 17:28:41 2014
named_attribute: log_message_origin=local
named_attribute: trace_flags=0
sender:
original_recipient: a@b
recipient: a@b
*** MESSAGE CONTENTS 0071B6012B1 ***
Received: by brutus.mylan (Postfix)
        id 0071B6012B1; Tue, 25 Feb 2014 17:28:42 +0100 (CET)
Date: Tue, 25 Feb 2014 17:28:42 +0100 (CET)
From: MAILER-DAEMON@brutus.mylan (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: a@b
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="C2E926024D4.1393345721/brutus.mylan"
Content-Transfer-Encoding: 8bit
Message-Id: <20140225162842.0071B6012B1@brutus.mylan>

This is a MIME-encapsulated message.

--C2E926024D4.1393345721/brutus.mylan
Content-Description: Notification



postconf -n
alias_maps = hash:/etc/aliases, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ldap:/etc/postfix/ldapalias_maps.cf, ldap:/etc/postfix/ldapgalias_maps_both.cf, ldap:/etc/postfix/ldapgalias_maps_member.cf, ldap:/etc/postfix/ldapgalias_maps_folder.cf, ldap:/etc/postfix/ldapgalias_maps_forward.cf, ldap:/etc/postfix/ldapualias_maps_folder.cf, ldap:/etc/postfix/ldapualias_maps_forward.cf
biff = no
canonical_maps = hash:/etc/postfix/canonical
command_directory = /usr/sbin
config_directory = /etc/postfix
content_filter =
daemon_directory = /usr/lib/postfix
data_directory = /var/lib/postfix
debug_peer_level = 2
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd $daemon_directory/$process_name $process_id & sleep 5
defer_transports =
delay_warning_time = 1h
disable_dns_lookups = no
disable_mime_output_conversion = no
disable_vrfy_command = yes
home_mailbox = Mail/
html_directory = /usr/share/doc/packages/postfix-doc/html
inet_interfaces = all
inet_protocols = ipv4
mail_owner = postfix
mail_spool_directory =
mailbox_command =
mailbox_size_limit = 0
mailbox_transport =
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
masquerade_classes = envelope_sender, header_sender, header_recipient
masquerade_domains = , ldap:/etc/postfix/ldapmasquerade_domains.cf
masquerade_exceptions = root
message_size_limit = 10240000
message_strip_characters = \0
mydestination = $myhostname,localhost.$mydomain,$mydomain, ldap:/etc/postfix/ldapmydestination.cf
myhostname = brutus.mylan
mynetworks = 127.0.0.0/8, 192.168.23.0/24
mynetworks_style = subnet
newaliases_path = /usr/bin/newaliases
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/packages/postfix-doc/README_FILES
relay_clientcerts =
relayhost =
relocated_maps = hash:/etc/postfix/relocated
sample_directory = /usr/share/doc/packages/postfix-doc/samples
sender_canonical_maps = hash:/etc/postfix/sender_canonical
sendmail_path = /usr/sbin/sendmail
setgid_group = maildrop
smtp_enforce_tls = no
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile =
smtp_tls_CApath = /etc/postfix/ssl/cacerts
smtp_tls_cert_file =
smtp_tls_enforce_peername = yes
smtp_tls_key_file =
smtp_tls_per_site = ldap:/etc/postfix/ldapsmtp_tls_per_site.cf
smtp_tls_session_cache_database = btree:/var/lib/postfix/smtpd_tls_session_cache
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP $mail_name
smtpd_client_restrictions = ldap:/etc/postfix/ldapaccess.cf
smtpd_delay_reject = yes
smtpd_helo_required = no
smtpd_helo_restrictions =
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_auth_destination, permit_mynetworks, reject_unauth_destination, reject
smtpd_sasl_auth_enable = yes
smtpd_sender_restrictions = ldap:/etc/postfix/ldapaccess.cf
smtpd_tls_CAfile =
smtpd_tls_CApath = /etc/ssl/certs
smtpd_tls_ask_ccert = no
smtpd_tls_auth_only = no
smtpd_tls_cert_file = /etc/ssl/servercerts/servercert.pem
smtpd_tls_key_file = /etc/ssl/servercerts/serverkey.pem
smtpd_tls_received_header = no
smtpd_use_tls = yes
soft_bounce = yes
strict_8bitmime = no
strict_rfc821_envelopes = no
transport_maps = ldap:/etc/postfix/ldaptransport_maps.cf
unknown_local_recipient_reject_code = 550
virtual_alias_domains = ldap:/etc/postfix/ldapvirtual_alias_domains.cf
virtual_alias_maps = ldap:/etc/postfix/ldapuser_recipient_maps.cf, ldap:/etc/postfix/ldapgroup_recipient_maps.cf
postconf: warning: /etc/postfix/master.cf: unused parameter: =
postconf: warning: /etc/postfix/master.cf: unused parameter: =
postconf: warning: /etc/postfix/master.cf: unused parameter: =
postconf: warning: /etc/postfix/master.cf: unused parameter: =

The heap of “,” “in alias_maps = hash:/etc/aliases, , …” worries me, as well as those postcon warnings.

Can’t really see what is wrong here.

greez

chris

#4

On 2014-02-27 16:16, seuchato wrote:
>
> robin_listas;2627437 Wrote:

>> Hum.
>> I’ll have to reread this later. I just had lunch and I’m feeling too
>> sleepy. O:-)
>>
>
> Hi, have nice siesta please :slight_smile:

:slight_smile:

And then I had to go out. Ok, let’s read this again.

>> Assume I got a mail from soccer@fiva.org (inbound message) to
>> me@mydomian. Then I replied to it (outbound message). During my coffe
>> break postfix fetched the inbound message, gave it an ID of F19BA60116E
>> and was able to dump it in my maildir. Also during my coffe break
>> postfix fetched my answer, from me@mydomian to soccer@fiva.org gave it
>> an ID of A1C19601164. Now A1C19601164 dumps araound in deferred queue or
>> will be resent to soccer@fiva.org (how embarassing, given some mails are
>> several years old). So I would love to be able to get postfix to dump
>> A1C19601164 in my maildir. How can I do that?

Ok, I think I understand. What is happening is that emails that were
already sent with your previous setup, even years are go, are being
retrieved, and postfix wants to send them again. Right…

Ok, you are doing it wrong, I think. These emails should not pass via
postfix at all.

The way to do it, I think, is to setup a local storage, like a dovecot
imap server, and then use one of many tools out there to sync imap
servers (I use “imapsync”, but my needs are small). It reads from an
imap server, and writes all the folders and posts, unmodified, on the
target imap server. Postfix is not involved, so there is no possible
attempt to resend them again.

If you do not want to use a local imap server, then you have to do with
other means. Perhaps fetching with “fetchmail”, but not forwarding with
local smtp server, but directly to folders. You would probably have to
script it, if you have several folders and users.

> For the time being, I have copied /var/spool/postfix/deferred to a safe
> place and deleted most of the files in /var/spool/postfix/deferred.

Well, the proper command is “postsuper -d queue_id”, but of course, you
have to get the IDs of those emails first, and I guess there are a lot
of them.

> Code:
> --------------------
> mailq
> …
> 000C26012B0 8081 Tue Feb 25 17:28:41 MAILER-DAEMON
> (mail transport unavailable)
> x@y
>
> 0071B6012B1 5401 Tue Feb 25 17:28:42 MAILER-DAEMON
> (mail transport unavailable)
> a@b
>
> --------------------

This looks like a bounce message for an email that can not be delivered,
itself being impossible to send.

Which in your case was fortunate, or those year old sent emails would
have been sent again :wink:

Well, I see two issues. One, is getting those many emails on the ISP (if
I understood correctly) to local storage. For this, as I said, I would
setup a local storage with an imap server (dovecot, courier…), and
then copy them over. This is independent of postfix.

If your emails are still on the ISP server, I would do this part again.

Another task is setting up postfix, and whatever more is needed, so that
you can receive and send email properly. And of course, you have to test
it, to see that it works, before attempting a massive fetch :wink:

> Code:
> --------------------
> postconf -n
> alias_maps = hash:/etc/aliases, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,_ , ldap:/etc/postfix/ldapalias_maps.cf, ldap:/etc/postfix/ldapgalias_maps_both.cf, ldap:/etc/postfix/ldapgalias_maps_member.cf, ldap:/etc/postfix/ldapgalias_maps_folder.cf, ldap:/etc/postfix/ldapgalias_maps_forward.cf, ldap:/etc/postfix/ldapualias_maps_folder.cf, ldap:/etc/postfix/ldapualias_maps_forward.cf

That is indeed strange. Well, you will probably have to edit that manually.

> virtual_alias_domains = ldap:/etc/postfix/ldapvirtual_alias_domains.cf
> virtual_alias_maps = ldap:/etc/postfix/ldapuser_recipient_maps.cf, ldap:/etc/postfix/ldapgroup_recipient_maps.cf
> postconf: warning: /etc/postfix/master.cf: unused parameter: =
> postconf: warning: /etc/postfix/master.cf: unused parameter: =
> postconf: warning: /etc/postfix/master.cf: unused parameter: =
> postconf: warning: /etc/postfix/master.cf: unused parameter: =

Have a look at the “/etc/postfix/master.cf” file with an editor…

> --------------------
>
>
> The heap of “,” “in alias_maps = hash:/etc/aliases, , …” worries me,
> as well as those postcon warnings.

Well, they are wrong, yes. Really important or not, I don’t know yet.

If you are using ldap, I have to tell you that I’m not familiar with
that beast, though…

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#5

Reading this,
I think that Carlos touched on a fundamental setup issue…

Postfix (and sendmail) are fundamentally simple relays. It can be faced in any direction (in or out) and it can be configured to forward in any direction.
That is why some kind of SMTP server (Postfix, Sendmail, etc) is required for sending outbound email.

But, it is only purely optional to be used otherwise… typically it can be used as a caching mail server (inbound or outbound), filtering, or even some fancy routing.

Since this (SMTP server) is not required for inbound mail, you need to ask yourself how this came to be, if it was intentional or not. For inbound mail, if you don’t need to execute any special functionality mailservers receive mail directly. At most, an SMTP sink might be configured (a tightly configured add-on to the mail server) but that is different than a standalone instance of Postfix or Sendmail.

So, I recommend before you go further you should analyze and understand the topology for the particular mailserver you’re running and be sure you have it configured correctly.

IMO,
TSU

#6

On 2014-02-28 00:56, tsu2 wrote:

…

> Since this (SMTP server) is not required for inbound mail, you need to
> ask yourself how this came to be, if it was intentional or not. For
> inbound mail, if you don’t need to execute any special functionality
> mailservers receive mail directly. At most, an SMTP sink might be
> configured (a tightly configured add-on to the mail server) but that is
> different than a standalone instance of Postfix or Sendmail.

Right.

I made some guesses on his setup, but I could be wrong. It could be some
thing similar to what I do: I get emails from my ISP using fetchmail.
Mail is passed to postfix, which filters using amavis, then passes to
the local delivery agent, which is procmail, which distributes it on
folders.

I think he was getting emails from his ISP, to store them locally. But
the thing also fetched emails from the “Sent” folders, which postfix
thought it had to send. Outside. Even if they were already sent years
ago :slight_smile:

If this is the situation, email has to be copied locally directly to
folders, not via postfix. Or, do not fetch “Sent” email.

> So, I recommend before you go further you should analyze and understand
> the topology for the particular mailserver you’re running and be sure
> you have it configured correctly.

Yep, absolutely.

We do not know the full setup, we have to guess :slight_smile:

But I also think this is a new setup, not fully designed yet.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#7

Hi Carlos and Tsu

This is a new setup and indeed not fully designed yet. As a matter of fact, it is actually a reduced setup. I eliminated cyrus, Amavis, in order to get the mail handling chain as simple as possible for the time being and to build my knowledge to understand what is going on from ground up.

So, thanks a ton for your answers so far. Regarding my setup, I guess what ist still missing is master.cf (main.cf has been included above as posctonf -n printout).

cat master.cf (removed all comments)

# service type  private unpriv  chroot  wakeup  maxproc command + args
#                          (yes)    (yes)   (yes)   (never) (100)
smtp            inet     n       -       n       -        -       smtpd   -o =

pickup         fifo      n       -       n       60       1       pickup
cleanup       unix     n       -       n       -         0       cleanup
  -o =
qmgr          fifo       n       -       n       300     1       qmgr
  -o =

tlsmgr        unix      -      -       n       1000?    1       tlsmgr
rewrite       unix      -      -       n       -        -       trivial-rewrite
bounce       unix      -      -       n       -        0       bounce
defer          unix      -      -       n       -        0       bounce
trace          unix      -      -       n       -        0       bounce
verify         unix      -      -       n       -        1       verify
flush          unix      n      -       n       1000?    0       flush
proxymap   unix      -      -       n       -        -       proxymap
proxywrite unix       -      -       n       -        1       proxymap
relay         unix       -      -       n       -        -       smtp

showq       unix      n      -       n       -        -       showq
error         unix      -      -       n       -        -       error
retry         unix      -      -       n       -        -       error
discard      unix      -      -       n       -        -       discard
local         unix       -      n       n       -        -       local
virtual      unix       -      n       n       -        -       virtual
lmtp         unix       -      -       n       -        -       lmtp
  -o =
anvil        unix       -      -       n       -        1       anvil
scache     unix       -      -       n       -        1       scache

Basic network setup
one server (brutus), OS 12.3, latest updates as of an hour ago

  • ldap for user managment, mail server, dns server, samba
  • 2 nics, one internal 192.168.23.4, 1 external 192.168.0.4, to router
  • dhcp and dns with automatic dns updates dnssec for 192.168.23.0 network, no issues indeed
  • postfix without amavis, local delivery into maildir, no imap, no virusscan (for the time being to eliminate every potential source of troubles), no outgoin mails (for the time being)

I hope above info completes the missing parts of our setup. If not, let me know what part is still needed. I tried to follow all (to my judging) relevant instructions as indicated here: http://www.postfix.org/DEBUG_README.html.

My first intention, is to get it running smoothly, my second (on the long term more important) to understand as much as I can.

greez

chris

#8

Taking my post one step further,

Again, you have to ask yourself if you really want to setup Postfix for any reason if you’re not configuring outbound mail. Whatever your mailserver is, you <can> configure it to accept mail directly and not through an SMTP server like Postfix. If you want to setup Postfix because you intend to <eventually> use it for something (like filtering), then so be it.

But, if you setup Postfix (or any other SMTP smarthost), as I noted you need to configure it correctly, facing in the proper direction.
My guess is that your current config is trying to send mail outbound because that’s the <normal> and most used configuration so that is why it sucked up a bunch of emails from your mailserver.

But, since you want to configure it as an <inbound> SMTP server, you’d have to configure it that way. Take a look at your Postfix config. I don’t know how it looks but it might be as simple as swapping IP addresses. As for your messages in queue, you may need to take a look at their headers to see if they are already modified to go to their remote destinations or if they’re simply waiting in queue. If modified(likely), of course re-importing might be a task that requires re-writing those headers. If you have a backup, I assume that’d be an easier approach and then you can just purge the queue. Unless someone can think of another solution…

TSU

#9

TSU

It was never the intention to have an inbound postfix only. As outlined in previous posts, I simply reconfugured everything to safely prevent resending of all these defferred files. Since I could not see a solution based on postfix and other available unix tools, I meanwhile wrote a programme myself to do all that was required. The mails in the deferred queue have now all been delivered in the users maildirs and this issue is resolved for me.

Thanks to TSU and Roberto for helping.

greez
chris