Remote Monitoring
I added “agentaddress tcp:161″ to snmpd.conf which cases snmpd to listen on tcp port 161. Then it was easy to tunnel the tcp communication with ssh and eliminated the need for the buggy socat method.
SSH tunnel
ssh -f -N -L 6003:localhost:161 user@digitaldogma.org
Then I setup the host using tcp:127.0.0.1 and port 6003 in cacti and everything works as expected.
Monit
I also set a monit entry for localhost:6003 to monitor the ssh tunnel. If the connection fails it restarts the tunnel using the ssh command above.
Apache, Exim4, and MySQL stats
To graph apache, exim4 and mysql stats, I created the following cron jobs:
# collect exim stats
2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/local/bin/exim_perf.pl > /etc/snmp/exim.stats 2> /dev/null
# collect apache stats
2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/bin/lynx -dump localhost/server-status?auto | head -9 | sed -e 's/: /\n/' > /etc/snmp/apache.stats
# collect mysql stats
2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/bin/mysqladmin -u root status | sed -e 's/: /\n/g' -e 's/[0-9|.]\+/& \n/g' -e 's/ //g' | sed '$d' > /etc/snmp/mysql.stats
This writes the stats to a file in /etc/snmp every 5 min, these are then available via snmp by adding the following to /etc/snmp/snmpd.conf:
exec .1.3.6.1.4.1.14464.25 exim_perf.pl /bin/cat /etc/snmp/exim_perf.stats
exec .1.3.6.1.4.1.14464.80 apache /bin/cat /etc/snmp/apache.stats
exec .1.3.6.1.4.1.14464.3360 mysql /bin/cat /etc/snmp/mysql.stats
For my own sanity, I used the tcp port of the daemon as the last number of the OID. As I understand it, this is a reserved OID for just such custom things.
After some final tweaking, I’ll keep an exported copy of the cacti templates here as well.
Snort
I also thought it would be neat to graph snort alerts. As cacti runs on the same server that hosts the snot db, this was pretty simple. I’m using the following shell script to get the number of alerts in the past 5 min:
#!/bin/sh
# list the number of snort events in the last 5min
# is protocol number is provided, show only events for that protocol
# TCP = 6
# UDP = 17
# ICMP = 1
# portscan = 255
PROTO=""
if [ $1 ]; then
PROTO="and iphdr.ip_proto = '$1'"
fi
echo "select count(*) from iphdr, event where event.cid = iphdr.cid and DATE_SUB(NOW(),INTERVAL 5 MINUTE) <= event.timestamp $PROTO" | /usr/bin/mysql -s -u user --password=pass -D snort
Links
smoke ping graphs