Postfix logs

[tradziej]

Where I could find Postfix logs? I tried to find the path but I couldn't. I think it is a good idea to update README with this info.

[me1299]

Yeah I really need them as well. Because any SMTP server is a party for scanners, I need to parse the logs through fail2ban in order to block spammers.

Seems like all log entries are sent to STD_OUT and therefor only shown in the docker logs. This is rather unhandy for parsing purposes.

[JeffHiggy]

If all you're interesting in is fail2ban blocking and f2b is running on your docker host you can simply parse the docker logs directly, this is what I'm doing. It's not the most elegant solution, as f2b is now parsing the logs of every container, and it will only work if docker is your orchestrator, but it's simple and works well.

filter.d/saslmail.conf

[Definition]
failregex = ^.*\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed.*$
            ^.*reject: RCPT from \S+\[<HOST>\]:.*Relay access denied.*$
            ^.*lost connection after .*\[<HOST>\].*$

jail.d/saslmail.conf

[saslmail]
enabled  = true
port     = 25
filter   = saslmail
logtimezone = UTC
logpath  = /var/lib/docker/containers/*/*-json.log
banaction = iptables-docker

That said I still wanted to get the data into my elastic stack so I could get some pretty charts of the email I'm getting. I could have done this the same way as above by reading the docker logs directly, but I really only wanted the raw mail logs and not the other random stuff. Fortunately this image is already using syslog-ng which supports many destinations, so I forked the project and edited the syslog-ng.conf file to send the data to my elastic instance and stdout. syslog-ng does have a built in elasticsearch module but unfortunately it requires java, so for simplicity I just added a file that I can pass to my filebeat container to process. I will probably update this in the future to send via syslog so that I don't have this extra file that I have to store and pass around.

syslog-ng.conf

filter f_postfix {
  program(postfix);
};

# Destinations
destination d_file {
  file(
    "/var/log/simple-mail-forwarder/postfix.log"
    create-dirs(yes)
    template("${ISODATE} ${MSGHDR}${MESSAGE}\n")
  );
};

# Logs
log {
  source(s_dgram);
  filter(f_postfix);
  destination(d_file);
};

[cenk1cenk2]

Hello @huan ,

Thank you for your work in this, I have been happily using this without any overhead of hosting the mail service for quite some time now.

Is there any progress or thought on implementing file logs, my problem is even though I use the docker parsing for fail2ban, it wastes 1 cpu core fully every 5 seconds or so since my server hosts a lot of containers outputting a lot of data as log.

It would be great that we can enable the file logging on-demand with an environment variable.

[huan]

Hello @cenk1cenk2 ,

Glad to hear from you and get to know that SMF helped you!

I understand that the file logs are important in some cases, so I'd like to accept PR for that.

[tamaro-skaljic]

Hello @cenk1cenk2 ,

I'm not a user of SMF but think it will help me in the future. I've looked to the official documentation of postfix and just made a pull request for SMF to enable file logging. I would like to ask you to test this one, since I don't know your exact requirements and I've never used/build SMF yet. Feel free to contact me.

Of course, this also applies to you @huan and the others who wish to use file logging.

[cenk1cenk2]

It seems that this function already implemented a recent pull request. Namely this one.

With this adding an environmental variable to the configuration enables the logs.

SMF_POSTFIXMAIN_maillog_file=/var/log/postfix/postfix.log

I have not realized this merge since, it is rather new and when I dived in to the code itself, I realized this is already implemented and even documented.

Over @tamaro-skaljic 's merge request I reverted to the base version, and just added postfix upgrade-configuration after these variables and it just works now. Opened up a merge request as well here.

Thanks to the contribution of @tamaro-skaljic , now I realized that this was even possible.

The container can be tested with cenk1cenk2/simple-mail-forwarder from docker hub until this reaches main stream.

Log rotating can be enabled via blacklabelops/logrotate.

This made a marginal difference comparing to reading logs from /var/docker.
from:

to:

[huan]

I have merged the two related PRs, please let me know if they work as expected or not, thank you very much!

[cenk1cenk2]

For me it does not work both together like it if set the env variable as SMF_POSTFIXLOG=/var/log/postfix/postfix.log it tries to use /var/log/postfix.log and fails the test as well.

It was working with sole #92 and update-configuration afterwards.

[tamaro-skaljic]

The test for the default configuration requires that SMF_POSTFIXLOG is not set. The test will probably fail because the variable was set before the test was executed in your environment.

Edit: I think adding SMF_POSTFIXLOG="" after

docker-simple-mail-forwarder/test/simple-mail-forwarder.bats

Line 206 in 3133faa

# Check if not specified variable will result in default configuration

will result in successful test execution.

Edit2: Making a commit that fixes the test and to reproduce the behaviour SMF_POSTFIXLOG=/var/log/postfix/postfix.log will result in using /var/log/postfix.log.

[tamaro-skaljic]

@cenk1cenk2 The Build hasn't reproduced this behaviour. I think the following happened:

• You've set SMF_POSTFIXLOG=/var/log/postfix/postfix.log
• You've run the tests
• Test 22 expects that SMF_POSTFIXLOG is not set. Because it is, the test failes.
• Test 23 overrides the postfix logging configuration and will not fail except when the algo is broken
• But Test 24 does it. It configures Postfix to log to /var/log/postfix.log and that's what it does.

When you're executing the tests now (#99 ) they should run successful and also they should not affect your local terminal session after execution (tests should save the value of SMF_POSTFIXLOG before test execution and afterwards they should restore your configured variable to have no side affects on ongoing commands executed in the session terminal). Also it should log to /var/log/postfix/postfix.log because of changed log file path in one of these commits, to test whether their is a problem with e. g. the doubled String "postfix" in the variable or not. It looks like it isn't.

[cenk1cenk2]

Yes, that was exactly what was happening. But now it may clash with SMF_POSTFIXMAIN_ variable looking at the main.conf and seeing the log file entry there. I have to build and check it again.

[huan]

Hi guys, thanks for the testing and the feedback!

Please help me to approve or request for changes with a reason to the related PR if you want to speedup the merging, because I'm not very sure about those codes.

[cenk1cenk2]

Just tried it out works as expected. It does not overwrite the log path anymore while doing tests and do not fail any tests anymore. So very grateful to you both @tamaro-skaljic and @huan for making this happen. This was an ongoing problem for me which I kept postponing.

[tamaro-skaljic]

@cenk1cenk2 Could you please test something for me?

• Set SMF_POSTFIXLOG=/var/log/postfix.log in your environment.
• Execute the tests.
• Do anything else that results in logs from postfix.
• Tell me where Postfix has logged after that.

I think it will log to /var/log/postfix/postfix.log instead of your specified /var/log/postfix.log because after test execution there is no restart of postfix.

And could you ensure that multiple test executions in a row not result in test failures?

With current master and also with #101 please.

Should mention that the tests do all sorts of things to your environment, so you should never run this in production. In general, the execution of a test may result in the environment being left in a different state than it was before.

[cenk1cenk2]

Sorry for the late answer @tamaro-skaljic missed this one. Yeah it still logs to /var/log/postfix/postfix.log even I set it to the /var/log/postfix.log, so you were right about this concern.

[tamaro-skaljic]

@cenk1cenk2 No problem. Could you please test it with #101 too?