Cacti
Contents |
Remote Monitoring
I was able to monitor my internal servers via snmp and cacti with no issues. However I also have a hosted server (the one running this wiki in fact) that I wanted to monitor, but didn't want to expose snmp to the world.
After some digging I found these instructions. Here are the exact commands I used:
SSH tunnel
ssh -f -N -L 6004:localhost:6004 user@digitaldogma.org
The socat commands
For the monitoring server:
socat UDP4-LISTEN:16101,fork TCP:localhost:6004
For the remote server:
socat TCP4-LISTEN:6004,fork UDP4:localhost:161
Then I setup the host using 127.0.0.1 and port 16101 in cacti and everything works as expected.
Next Steps
As the socat commands are restarted from cron, they will continue to run after a reboot, however the ssh tunnel will need to be restarted. It will need to be wrapped up in a /etc/init.d script, and added it to the appropriate run levels. I'll also want to have monit baby sit it so it's always up.
Issues
This method works great, however after leaving it to run over night, the next day I discovered over 800 defunct socat processes. It seems that after forking the connection wasn't ever closed which caused the process to hang around forever. I've not yet been able to determine the cause of this, but I was able to work around it by having a cron job kill and restart socat every hour.
For the monitoring server:
03 * * * * kill `pidof socat`; sleep 2; socat UDP4-LISTEN:16101,fork TCP:localhost:6004 &
For the remote server:
02 * * * * kill `pidof socat`; sleep 2; socat TCP4-LISTEN:6004,fork UDP4:localhost:161 &
The sleep is to allow for a bit of time for the existing processes to die. I also had the remote server start first (at 02 past the hour) to ensure it was up, before starting the monitoring server. This has worked well thus far, but it does prevent the use of any further socat tunnels (as they will be killed every hour). Ultimately I need to determine why the children do not exit.
Apache, Exim4, and MySQL stats
To graph apache, exim4 and mysql stats, I created the following cron jobs:
# collect exim stats 2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/local/bin/exim_perf.pl > /etc/snmp/exim.stats 2> /dev/null # collect apache stats 2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/bin/lynx -dump localhost/server-status?auto | head -9 | sed -e 's/: /\n/' > /etc/snmp/apache.stats # collect mysql stats 2,7,12,17,22,27,32,37,42,47,52,57 * * * * root /usr/bin/mysqladmin -u root status | sed -e 's/: /\n/g' -e 's/[0-9|.]\+/& \n/g' -e 's/ //g' | sed '$d' > /etc/snmp/mysql.stats
this writes the stats to a file in /etc/snmp every 5 min, these are then avaiable via snmp by adding the following to /etc/snmp/snmpd.conf:
exec .1.3.6.1.4.1.14464.25 exim_perf.pl /bin/cat /etc/snmp/exim_perf.stats exec .1.3.6.1.4.1.14464.80 apache /bin/cat /etc/snmp/apache.stats exec .1.3.6.1.4.1.14464.3360 mysql /bin/cat /etc/snmp/mysql.stats
For my own sanity, I used the tcp port of the daemon as the last number of the OID. As I understand it, this is a reserved OID for just such custom things.
After some final tweaking, I'll keep an exported copy of the cacti templates here as well.
Snort
I also thought it would be neat to graph snort alerts. As cacti runs on the same server that hosts the snot db, this was pretty simple. I'm using the following shell script to get the number of alerts in the past 5 min:
#!/bin/sh
# list the number of snort events in the last 5min
# is protocol number is provided, show only events for that protocol
# TCP = 6
# UDP = 17
# ICMP = 1
# portscan = 255
PROTO=""
if [ $1 ]; then
PROTO="and iphdr.ip_proto = '$1'"
fi
echo "select count(*) from iphdr, event where event.cid = iphdr.cid and DATE_SUB(NOW(),INTERVAL 5 MINUTE) <= event.timestamp $PROTO" | /usr/bin/mysql -s -u user --password=pass -D snort
Once I get the kinks worked out of the templates, I'll keep an exported copy here.
