The Life of a Sysadmin

Carousel is a lie!

Instructions for yak shaving
Mon Oct 5 12:45:51 PDT 2009
  1. Install logwatch on Solaris fileserver.

  2. Notice that logwatch emails are not coming in.

  3. Log in and run logwatch by hand.

  4. Inspect mail log and notice lack of any entries.

  5. Notice that Postfix is in maintenance mode; start it up.

  6. Notice continued lack of emails.

  7. Notice that Postfix is running, which confused svcadm when told to start up Postfix. It fails to do so and fails to log this.

  8. killall postfix, svcadm enable postfix.

  9. man svcadm; svcadm clear postfix; svcadm enable postfix.

  10. Run logwatch by hand; notice emailed report to "root@localhost.localdomain", which gets bounced by Postfix on the mail server because it's a non-existent host.

  11. Resist temptation to go down that rabbit hole just now, and stick to the problem at hand.

  12. Edit /opt/csw/etc/log.d/logwatch.conf and set MailTo to proper address.

  13. Re-run logwatch and note that reports are still going to root@localhost.

  14. After much swearing, notice that actually, logwatch is set to look in /opt/csw/etc/log.d/conf/logwatch.conf for configuration.

  15. Edit that file, re-run logwatch.

  16. Notice errors from Postfix: "postdrop[13848]: [ID 947731 mail.warning] warning: mail_queue_enter: create file maildrop/908447.13848: Permission denied".

  17. Run "postfix set-permissions". Test mail; still failing.

  18. Check permissions on another system and set by hand.

  19. Re-run logwatch. Still no email. Re-run with debug=high and get email.

  20. Wonder idly about futility of self-aware log watching system that can't report on its own heisenbug-induced failure, crappy packaging practices, inability to check end-to-end email connectivity, other career options.

  21. (Update) Realize that the emails show up if "Detail" is set to Medium or High ; Low, the default, makes the report silent.

  22. (Update) Uninstall the package and reinstall, only to find that the symlink to conf/logwatch.conf is set up at installation, and that this is probably a case of $EDITOR breaking the symlink. Apply head to desk.

Tags: monitoring, packagemanagement, solaris, yakshaving.

Comments On This Entry

Legoooolas
Logwatch is horrible anyway :P Perhaps use centralised logging and SEC or something else nicer than logwatch (and SEC is useful for immediate notifications) instead of one mail per host per day from logwatch? :)
Saint Aardvark
I had come across SEC once and promptly forgot about it...thanks for the reminder. I was going with logwatch because it comes with CentOS, which is what (almost) all our other servers run. SEC looks cool, but seems pretty barebones -- I mean, that it looks like it's up to you to make it work, build rules and so on. Is that the case or have I misunderstood?