【IT168 专稿】最近在一个新安装的Freebsd 7系统部署一套nagios监控系统,主要用来监控主机的存活状态、网络服务的状态以及远程主机的资源利用情况。尽管Freebsd安装源码有些麻烦,但最终还是把nagios正确安装上了,并且能正常工作,通过web方式也能看见监控对象的状况。
有故障报警行为,但看不到?
如上图所示,真有一个服务器的443端口对应的服务发生故障了,可是等了半天就是收不到报警邮件。登录nagios所在的系统,检查与邮件发送相关的情况,其基本操作是:检查sendmail是否起来(ps aux | grep sendmail),结果正常;用mail程序手动发一封邮件给我的一个邮箱(mail –s “This is a mail test project” sery@163.com < ip_scan ),能发送和接收到这封邮件。问题在哪里呢?
由上面的排查,基本可以断定sendmail没什么问题,域名解析也是正常的(专门为这个监控服务器解析MX及A记录),现在可能的原因只能在nagios这边了。进入nagios配置文件所在的目录,挨个察看配置文件。我的nagios配置目录的情况如下:
[root@nagios /usr/local/nagios]# pwd
/usr/local/nagios/etc
[root@nagios /usr/local/nagios/etc]# ls *.cfg
cgi.cfg contacts.cfg localhost.cfg services.cfg
commands.cfg hostgroups.cfg nagios.cfg timeperiods.cfg
contactgroups.cfg hosts.cfg resource.cfg没看见有什么异常的情况,改了其中的某些设置,如cgi.cfg文件,重启nagios,还是不能发报警邮件。可是,但我点击web管理界面的时候,确实是有邮件报警行为,如下图所示:
既有主机故障通知,又有服务故障报警通知,而且都应该按我的定义发送邮件的呀!
nagios自己也有日志记录
查去查来,找不到头绪。再查sendmail 的日志/var/log/maillog,只发现我手动发送邮件的记录,而没有其他发送记录---只有下面这么一条记录:
Jul 27 14:27:48 nagios sm-mta[37141]: m6RERkYR037139: to=<sery@163.com>, ctladdr=<nagios@nagios.sery.com> (1003/1003), delay=00:00:02, xdelay=00:00:01, mailer=esmtp, pri=30623, relay=163mx02.mxmail.netease.com. [220.181.12.66], dsn=2.0.0, stat=Sent (Mail OK queued as mx16,QsCowLDbPSxWFYxIb6TzGw==.27600S2 1217140055)看来nagios并没有调用sendmail发送邮件。
差点忘了,nagios自己也有日志记录呢!赶快打开看一眼,发现里面有不少Warning,抽一个出来,其内容如下:
[1217166816] HOST NOTIFICATION: sery;mail-server;DOWN;host-notify-by-email;CRITICAL - Plugin timed out after 10 seconds
[1217166816] Warning: Attempting to execute the command "/usr/bin/printf "%b" "***** Nagios 2.9 *****\n\nNotification Type: PROBLEM\nHost: mail-server\nState: DOWN\nAddress: 211.155.115.66\nInfo: CRITICAL - Plugin timed out after 10 seconds\n\nDate/Time: Sun Jul 27 13:53:36 UTC 2008\n" | /bin/mail -s "Host DOWN alert for mail-server!" sery@163.com" resulted in a return code of 127. Make sure the script or binary you are trying to execute actually exists...
原因:邮件路径不对
其他的行也更这个类似;最有用的信息我用红色标记,其大意是不能执行上面的2进制或可执行文件。在这个条目中,只有2个执行文件—printf及mail。我把它按原样单独拿出来执行,操作过程如下:
(1)/usr/bin/printf “"%b" "***** Nagios 2.9 *****\n” 输出 ***** Nagios 2.9 *****,这是正常的结果。
(2)/bin/mail -s "Host DOWN alert for mail-server!" sery@163.com 输出su: /bin/mail: No such file or directory,没找到路径或目录。前面还手动发了邮件的,明明有mail这个客户端程序呀!可能这个路径不对,是linux的mail路径。查一下freebsd的mail路径,执行find / -name 得到mail在freebsd的路径为/usr/bin/mail 。
到这里,我们知道了为啥不能发邮件的根本原因,接下来,我把nagios的配置文件commands.cfg的host-notify-by-email、service-notify-by-email的”/bin/mail”替换为“/usr/bin/mail”。其完整形式为:
# 'host-notify-by-email' command definition
define command{
command_name host-notify-by-email
command_line /usr/bin/printf "%b" "***** Nagios 2.9 *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "Host $HOSTSTATE$ alert for $HOSTNAME$!" $CONTACTEMAIL$
}
# 'notify-by-email' command definition
define command{
command_name service-notify-by-email
command_line /usr/bin/printf "%b" "***** Nagios 2.9 *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}
修改完配置文件commands.cfg后重启 Nagios,再查看nagios日志,不再有“Make sure the script or binary you are trying to execute actually exists...”报错,并且有发送报警邮件的记录了:
[root@nagios /usr/local/nagios/var]# tail -f nagios.log
[1217170467] SERVICE ALERT: mail-server;check_tcp 995;CRITICAL;SOFT;1;CRITICAL - Socket timeout after 10 seconds
[1217170534] Auto-save of retention data completed successfully.
[1217170577] HOST ALERT: mail-server;DOWN;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1217170587] HOST ALERT: mail-server;DOWN;SOFT;2;CRITICAL - Plugin timed out after 10 seconds
[1217170597] HOST ALERT: mail-server;DOWN;SOFT;3;CRITICAL - Plugin timed out after 10 seconds
[1217170607] HOST ALERT: mail-server;DOWN;SOFT;4;CRITICAL - Plugin timed out after 10 seconds
[1217170607] HOST ALERT: mail-server;UP;SOFT;5;PING OK - Packet loss = 0%, RTA = 111.63 ms
[1217170607] SERVICE ALERT: mail-server;check_tcp 995;CRITICAL;SOFT;2;CRITICAL - Socket timeout after 10 seconds
[1217170687] SERVICE ALERT: mail-server;check_tcp 995;OK;SOFT;3;TCP OK - 3.137 second response time on port 995
[1217171057] SERVICE NOTIFICATION: sery;fav-0;check_tcp 443;CRITICAL;service-notify-by-email;CRITICAL - Socket timeout after 10 seconds
收邮件,迫不及待,哈哈,我的163邮箱收到久违的报警信息了。再回去瞧一眼邮件日志/var/log/malllog,也记录了这个发送情况。
经验总结:通过日志记录,对于我们排查故障确实有着不可估量的好处。在实际的工作中,我们应该随时检查系统日志以及应用程序相关的日志,从记录项中寻找蛛丝马迹,从而得出解决问题的方法。
2008/7/27
海淀福缘门悟真阁