I recently got turned onto OMD/check_mk, which is probably the most user friendly tool for Nagios. It has a nearly complete Nagios stack, and more:
It has a very simple install and works with Debian/Ubuntu/Red Hat/SUSE. I recently deleted my Icinga build and went all in with this.
As well as regular servers, vmware and other things, I'm able to graph Google Nest's temperature, humidity and "leaf" status. You can set up alerts if it crosses a certain threshold and graph for as many years and you wish.
This is something that is sorely lacking in Google's current site since the graphing information is only stored for 10 days, and doesn't provide the granular level of detail I want.
While I won't go into much detail on the install of OMD (it's pretty straightforward) and I won't go into much detail about the Nest integration (Great writeup on how to do that here) I will be showing a way to get another product, the Elertus, monitored with check_mk as an example of what you can do with it.
The Elertus, is a wifi temperature, sound, light and water detector that runs on batteries. I wrote up an article about it on this blog before. It's a little trickier to monitor than the Nest due to the fact that there is no API, no way to scrape information off of the device since it only makes outbound connections.
Since my last post, last year, nothing has changed with it, nor has my opinion. It's still sluggish, no graphing, no "all clear" alerts... and a painfully slow (and useless) app to compliment. The plus side is that it's been almost a year and the batteries still work (AA batteries). I have had no connection issues from it as far as I can tell.
Other than an alerts tab, there's not much more to the Elertus website
When I had my Icinga server, I stood up another box that basically sniffed the traffic as it was in delivery to their servers.(which is still sent cleartext, btw). I cut out the bits and pieces I needed to get a basic graph up. Since I started this server, I decided to try a new method.
Using pfSense, I created a static DCHP connection for the device and an internal NAT rule to relay the traffic from the Elertus to my own web server running Apache with mod_dumpio turned on.
While this basically kills any communication with the Elertus servers, they won't be missed. With, my prior setup I was able to enjoy both my internally generated alerts, and their alerts, but I found my own to be a lot more useful.
The Elertus sends out a POST to their servers as a check in.
Apache has a virtualhost set up to receive those incoming PHP POST requests with dumpio enabled and the trace level set to 7.
I also made sure to set custom logs, for just this module, as they will be filling up quick.
The requests come in 2 at a time, sometimes more if there is any movement/light. So, to remedy the lack of a consistent time period to check the logs, I set up a cron job to pull the newest line in the log with POST data. This runs every 5 minutes.
The scripts within it, in turn, pull information out of the latest POST and updates the raw values to separate files. I've kept it modular so I can add more checks as I need them. I'm not too concerned about the water, movement and light alerts just yet.
The temperature is in Kelvin, so I converted it to Fahrenheit.
The battery and humidity, are pretty much the same thing, with different names. It just requires another {print $X} position. The light, movement and water sensor can be added just as easily since they are only values of 0 or 1.
Now, unlike Nagios, getting devices to check_mk is a breeze. Making custom checks, with RRD graphing is just as easy. You add the custom checks to the host itself, not the monitoring server.
This web server is Ubuntu and it has check_mk installed. There is a folder that allows you to put custom scripts to make local checks, above what it already monitors.
The setup is pretty straightforward if you want monitoring with performance graphs.
Any script that is in this folder runs when check_mk runs and grabs the other server global readings.
The output appends itself to the bottom of the generated file, as a local check like this:
It tells check_mk that it's available to be added as a service, with graphing
The name of the service
The variable name
The output from $TEMP1, the WARN min:max, the CRIT min:max and UNKNOWN lower;upper (for graphing reasons and not required). Notice that the colons and semicolons are there for a reason.
When you scan the host in WATO on the main check_mk site, the host should have basic performance graphs and automatically added to the notification ruleset the host was part of.
While it's not an ideal solution, it gets the job done. I wish Elertus would open their API up to developers, the would probably sell a lot more units if they did.
I recommend anyone who's given up with Nagios/Icinga to give OMD a try. It's pretty good. The documentation is a little poor (and mostly in German), but with Google Translate and some coffee, you can get through it if you've ever set up any other monitoring before.
- Nagios
- Monitoring Plugins
- check_nrpe
- mrpe (a check_mk clone of nrpe)
- Icinga
- Shinken
- NagVis
- pnp4nagios
- rrdtool/rrdcached
- Check_MK
- MK Livestatus
- Multisite
- Dokuwiki
- Thruk
- Mod-Gearman
- check_logfiles
- check_oracle_health
- check_mysql_health
- jmx4perl
- check_webinject
- check_multi
It has a very simple install and works with Debian/Ubuntu/Red Hat/SUSE. I recently deleted my Icinga build and went all in with this.
As well as regular servers, vmware and other things, I'm able to graph Google Nest's temperature, humidity and "leaf" status. You can set up alerts if it crosses a certain threshold and graph for as many years and you wish.
Check_MK with Google Nest
This is something that is sorely lacking in Google's current site since the graphing information is only stored for 10 days, and doesn't provide the granular level of detail I want.
Google Nest's graphs.
While I won't go into much detail on the install of OMD (it's pretty straightforward) and I won't go into much detail about the Nest integration (Great writeup on how to do that here) I will be showing a way to get another product, the Elertus, monitored with check_mk as an example of what you can do with it.
The Elertus, is a wifi temperature, sound, light and water detector that runs on batteries. I wrote up an article about it on this blog before. It's a little trickier to monitor than the Nest due to the fact that there is no API, no way to scrape information off of the device since it only makes outbound connections.
Since my last post, last year, nothing has changed with it, nor has my opinion. It's still sluggish, no graphing, no "all clear" alerts... and a painfully slow (and useless) app to compliment. The plus side is that it's been almost a year and the batteries still work (AA batteries). I have had no connection issues from it as far as I can tell.
Other than an alerts tab, there's not much more to the Elertus website
When I had my Icinga server, I stood up another box that basically sniffed the traffic as it was in delivery to their servers.(which is still sent cleartext, btw). I cut out the bits and pieces I needed to get a basic graph up. Since I started this server, I decided to try a new method.
Using pfSense, I created a static DCHP connection for the device and an internal NAT rule to relay the traffic from the Elertus to my own web server running Apache with mod_dumpio turned on.
While this basically kills any communication with the Elertus servers, they won't be missed. With, my prior setup I was able to enjoy both my internally generated alerts, and their alerts, but I found my own to be a lot more useful.
The Elertus sends out a POST to their servers as a check in.
device_type=1&posix_time=1423925134&email_id=myemail@gmail.com&mac_address=000680000000&alert_flags=&light=14&temp=298&humidity=30&battery=70&motion=0&int_contact=1&ext_contact=1&ext_temp=-1&fw_ver=4.0.1_EL_v7&debug=rssi:46, ant:I, af:, pkt:l14_t298_h30_b70_m0_i1_e1_x-1_p1423925134, wdog:1, crtry:4, queue:3, ctime:w2285_d410_n130_s205_t3040, &
Apache has a virtualhost set up to receive those incoming PHP POST requests with dumpio enabled and the trace level set to 7.
<IfModule dumpio_module>
DumpIOInput On
DumpIOOutput On
LogLevel dumpio:trace7
</IfModule
ErrorLog ${APACHE_LOG_DIR}/dumpio_module_error.log
CustomLog ${APACHE_LOG_DIR}/dumpio_module_access.log combined
The requests come in 2 at a time, sometimes more if there is any movement/light. So, to remedy the lack of a consistent time period to check the logs, I set up a cron job to pull the newest line in the log with POST data. This runs every 5 minutes.
#!/bin/bash
tac /var/log/apache2/dumpio_module_error.log | grep -m1 "email" > /tmp/tempout.txt
~/scripts/temp.sh
~/scripts/humidity.sh
~/scripts/battery.sh
The scripts within it, in turn, pull information out of the latest POST and updates the raw values to separate files. I've kept it modular so I can add more checks as I need them. I'm not too concerned about the water, movement and light alerts just yet.
The temperature is in Kelvin, so I converted it to Fahrenheit.
#!/bin/bash
tempk=$(cat /tmp/tempout.txt | awk -F "=" '/light/ {print $8}' | sed 's/&.*//')
tempb=$(awk "BEGIN {print "$tempk" - 273.15}")
temp=$(echo ""$tempb"*1.8+32" | bc)
echo "$temp" > ~/perfdata/temp
(I know, this can be cleaned up a lot)
The battery and humidity, are pretty much the same thing, with different names. It just requires another {print $X} position. The light, movement and water sensor can be added just as easily since they are only values of 0 or 1.
Now, unlike Nagios, getting devices to check_mk is a breeze. Making custom checks, with RRD graphing is just as easy. You add the custom checks to the host itself, not the monitoring server.
This web server is Ubuntu and it has check_mk installed. There is a folder that allows you to put custom scripts to make local checks, above what it already monitors.
/usr/lib/check_mk_agent/local/
The setup is pretty straightforward if you want monitoring with performance graphs.
#!/bin/bash
TEMP1=$(cat ~/perfdata/temp)
echo '<<<local>>>'
echo "P Temperature temp="$TEMP1";35:89;32:91;0;110"
Any script that is in this folder runs when check_mk runs and grabs the other server global readings.
The output appends itself to the bottom of the generated file, as a local check like this:
<<<local>>>
P Temperature temp=73.13;35:89;32:91;0;110
It tells check_mk that it's available to be added as a service, with graphing
- P
The name of the service
- Temperature
The variable name
- temp=
The output from $TEMP1, the WARN min:max, the CRIT min:max and UNKNOWN lower;upper (for graphing reasons and not required). Notice that the colons and semicolons are there for a reason.
When you scan the host in WATO on the main check_mk site, the host should have basic performance graphs and automatically added to the notification ruleset the host was part of.
While it's not an ideal solution, it gets the job done. I wish Elertus would open their API up to developers, the would probably sell a lot more units if they did.
I recommend anyone who's given up with Nagios/Icinga to give OMD a try. It's pretty good. The documentation is a little poor (and mostly in German), but with Google Translate and some coffee, you can get through it if you've ever set up any other monitoring before.
BACKGROUND
ReplyDeleteI have a Safelert (Liberty Safe rebranded Elertus) in my safe. I was also disappointed to learn there is no public API. I asked Elertus several times to add a IFTTT channel, but the response has always been they have no plans to do so. So I used IMAPFILER to look for incoming email notifications to tigger actions. (https://github.com/elvisimprsntr/imapfilter-ifttt) it works for what is does, but is is not real time and it relies on having a live internet connection.
IFTTT SMS
I added an IFTTT SMS number and added a second contact to send SMS text messages for motion and lighting (door open) events. Unfortunately, it appears IFTTT SMS triggers requires the text message to be from the phone number you associate with IFTTT SMS, not to mention, the IFTTT SMS service is thru Level3 (aka. TextPlus) which does not use an email SMS gateway, thus you cannot select an appropriate service provider to send the SMS message. I did notice Elertus supports Google Project Fi, but I have not looked into it. I might be able to use a Google SMS number to relay a SMS message to IFTTT, but that seems a bit over complicated and of course still require a live internet connection.
WIRESHARK/TCPDUMP
I too was able to capture the clear text HTTP POST from the Safelert using Wireshark and a remote SSH TCPDUMP packet capture (DD-WRT router.)
LIGHTING/SECURITY CONTROLLER
My lighting/security system has a REST API which I used in the my IMAPFILTER implementation.
QUESTION
If there a way to use TCPDUMP (or other Linux method) with an appropriate filter to trigger a separate HTTP command to the REST API of my lighting/security controller when the Safelert sends a motion light event (alert_flag=X)? For example pipe TCPDUMP POST form data into BASH script which parses the alarm_flag, or an all inclusive script which runs as a systemctl daemon?