Friday, November 13, 2015

How to install Guacamole 0.9.8 for Ubuntu 14.04 and secure with Nginx ssl

Guacamole is a pretty straightforward RDP/VNC/SFTP utility that requires no plugins on client systems. It utilizes HTML5 to serve up the connections directly over a browser.

This is a pretty standard install to connect to a windows RDP host. We will be securing the server with UFW, fail2ban and SSL using NGINX as a reverse proxy.

First, enable the firewall and allow the following ports:

sed -i 's/ENABLED=no/ENABLED=yes/g' /etc/ufw/ufw.conf
ufw allow 22 && ufw allow 8080

Install the prerequisite software

apt-get update && apt-get install -y fail2ban build-essential htop libcairo2-dev libjpeg62-dev libpng12-dev libossp-uuid-dev tomcat7
apt-get install -y libfreerdp-dev libpango1.0-dev libssh2-1-dev libtelnet-dev libvncserver-dev libpulse-dev libssl-dev libvorbis-dev

Download and extract the guacamole server files

cd ~
tar -xzf guacamole-server-0.9.8.tar.gz && cd guacamole-server-0.9.8/

Compile and make the program (this may take some time depending on your hardware).

./configure --with-init-dir=/etc/init.d && make && make install

Now we want to update the library cache and update the init scripts so it will start on bootup

ldconfig && update-rc.d guacd defaults

Create the main Guacamole configuration folder and file

cd ~ && mkdir /etc/guacamole

Vi (or, you can use Nano/your file editor of choice) the main configuration which provides the location of the user-mapping.xml file.

vi /etc/guacamole/
# Hostname and port of guacamole proxy
guacd-hostname: localhost
guacd-port: 4822

# Location to read extra .jar's from
lib-directory: /var/lib/tomcat7/webapps/guacamole/WEB-INF/classes

# Authentication provider class

# Properties used by BasicFileAuthenticationProvider
basic-user-mapping: /etc/guacamole/user-mapping.xml

For the Guacamole client, we are going to add a simple RDP connection. This is highly configurable, so be sure to read up on their site to see all the variables.

The Guacamole website will have a username of guacadmin and a password of guacpass.

For our purposes we are going to connect to a windows box, with the username: winuser and the password winpassword on the standard port 3389. Change these fields to the Windows box you want to connect to.

vi /etc/guacamole/user-mapping.xml
 <authorize username="guacadmin" password="guacpass">
  <param name="hostname"></param>
  <param name="port">3389</param>
  <param name="username">winuser</param>
  <param name="password">winpass</param>

You may need to adjust your remote desktop settings to allow connections from non-NLA authenticated servers, such as this.

Update tomcat to point to the user authentication files.

mkdir /usr/share/tomcat7/.guacamole
ln -s /etc/guacamole/ /usr/share/tomcat7/.guacamole
cp guacamole-0.9.8.war /var/lib/tomcat7/webapps/guacamole.war

Restart the tomcat and guacamole services

service guacd start && service tomcat7 restart

You should now be able to access your server via port 8080. For instance, my server which is

Log in with the guacadmin/guacpass credentials and it will automatically log you into windows using the credentials you supplied in the user-mapping.xml file.

If you do not want it running in the /guacamole subfolder and want to place it in the server root

service tomcat7 stop
mv /var/lib/tomcat7/webapps/ROOT /var/lib/tomcat7/webapps/ROOT.bkp
mv /var/lib/tomcat7/webapps/guacamole /var/lib/tomcat7/webapps/ROOT
service tomcat7 start

You should now be able to access it via the ip and port:

To further strengthen the server and allow https over a standard port, nginx can be installed side-by-side with tomcat to provide a reverse proxy and allow encryption.

apt-get install -y nginx

Don't forget to update the firewall to allow port 443 and remove port 8080

ufw allow 443 && ufw delete allow 8080

Generate the SSL keys

mkdir /etc/nginx/ssl && openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/guacamole.key -out /etc/nginx/ssl/guacamole.crt

If you are using a DNS name to access the server, or anything other than an IP address, make sure you include the FQDN.

We want to clear out the default nginx configuration and add our own

mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.bkp
rm /etc/nginx/sites-enabled/default

vi /etc/nginx/sites-available/guacamole
server {
        listen 443 ssl;

        access_log   /var/log/nginx/guacamole.access.log ;
        error_log    /var/log/nginx/guacamole.error.log info ;

        ssl_certificate /etc/nginx/ssl/guacamole.crt;
        ssl_certificate_key /etc/nginx/ssl/guacamole.key;

        location / {
        proxy_buffering off;

ln -s /etc/nginx/sites-available/guacamole /etc/nginx/sites-enabled/guacamole

Restart nginx

service nginx restart

You should now be able to access the server via the standard https port

Saturday, September 5, 2015

Export Oracle Virtualbox signed hardware cert and slipstream it into a Windows install

If you deploy a lot of Windows 7 boxes using Virtualbox and resort to slipstreaming prerequisite files using nlite and/or answer files, you know the fully automated Virtualbox Guest Additions stops and prompts for input due to an unsigned driver from Oracle:

Thus, the install is not fully automated. To get around this, you can export the cert from another Windows guest that has Virtualbox installed using Powershell.

Run powershell with elevated privileges and execute these commands:

cd cert:\LocalMachine\TrustedPublisher
$cert = dir | where { $_.Subject -like "*Oracle*" }
$type = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert
$bytes = $cert.Export($type)
[System.IO.File]::WriteAllBytes("C:\oracle.cer", $bytes)

You should have an Oracle hex encoded cert in the root of your C: drive named oracle.cert.

To make this easier to install during the slipstream process, encode the cert in a 7zip SFX executable file. (sounds more complicated than it is if you've never used it. It simply creates an executable zip file),

Now, using NTLite, or whatever app of choice, for your post process scripts, we need to extract the cert to the C: drive and import it into the host using certutil.

oracle.exe -oc:\ -y

(There is no space between the o and c in the script above)

certutil -addstore -f "TrustedPublisher" c:\oracle.cer

You should have something like this:

After you compile the ISO, your fully automated installs shouldn't prompt to trust that cert in the middle of your post processing.

Saturday, August 15, 2015

Build a Yahoo Pipes replacement with Tiny Tiny RSS on Ubuntu 14.04

TT-RSS is an open source Yahoo Pipes (and to a large part, Google Reader) replacement. You can filter feeds, import OPML files and access via APIs and Android. It also has plugin support.

I'm going to list the steps to install Tiny Tiny RSS on a clean install of Ubuntu 14.04 using Apache, PostgreSQL, GIT and PHP5.

For demonstration purposes I'm not including the best security practices (SSL, firewall, fail2ban, user permission best practices, etc). Please refer to this great guide to secure your installation.

From a clean Ubuntu 14.04LTS install the following packages:
apt-get update && apt-get install -y apache2 git ntp postgresql-contrib php5 php5-curl php5-cli php5-pgsql

Make sure your hostname is set and update the Apache config to prevent FQDN error messages.
echo "ServerName $HOSTNAME" >> /etc/apache2/apache2.conf

Also, since timing is important with RSS feeds, we installed the NTP service.
service ntp reload

We are going to recycle the standard html output rather than create a virtualhost for demonstration purposes so we will git clone into the generic Apache directory.
git clone /var/www/tt-rss
mv /var/www/html /var/www/html2 && mv /var/www/tt-rss /var/www/html

Now we are going to create a PostgresSQL user and database. Make sure you enter your own, unique username and password (and document it).
su - postgres
createuser -P -s -e ttpguser

It will prompt you to create a password for the ttpguser (or whichever username you chose).
Now we will create the database
createdb ttrssdb

After you exit, you should be back as root. We need to modify the PostgreSQL client authentication conf file: /etc/postgresql/9.3/main/pg_hba.conf (Use nano if you are more comfortable with it.) and add the user we created in the database.

After this line:
local   all             postgres                                peer
Add this entry (make sure that it's the same username you created earlier):
local   all             ttpguser                                   md5

Update permissions for the cache and various other elements
chmod -R 777 /var/www/html/cache/images
chmod -R 777 /var/www/html/cache/upload
chmod -R 777 /var/www/html/cache/export
chmod -R 777 /var/www/html/cache/js
chmod -R 777 /var/www/html/feed-icons
chmod -R 777 /var/www/html/lock

Restart the services
service postgresql restart && service apache2 restart

Browse to the IP address of the server you created and you will be directed to http://<yourIP>/install if everything went alright.
Enter the following parameters
Username ttpguser
DB name ttrssdb
Hostname (leave blank)
Port 5432

Once that's done, initialize the database. You will get some code you will need to copy and import into your root www directory.
vi /var/www/html/config.php

Paste the code in and save the file. Go back to the root site, http://<yourip> and you should be welcomed with a login screen. The default username and password are admin and password.

To change this, go to Actions -> Preferences -> Users -> Admin -> Change password -> Fill out email (for some reason, cant change password for admin without entering an email address) -> Save -> (Refresh the browser)
Log back in as the admin and the new password. Head back to the user section. Add a new user and create a username you wish to use. Click on it again and enter the password.

You should be able to log in with the non-admin user.

To add a feed, go to Actions -> Subscribe to feed. You will notice the feed does not update. There are multiple ways to update, check the main site for more information on setting cron jobs, etc. For testing, we can simple type
su -c "php /var/www/html/update_daemon2.php" -s /bin/sh www-data&

This will start scrolling text, and will continue to run in the background. If you end the process, the updates will stop. Read the main tt-rss page on updating for more information.

You should see feeds start to appear. You can also import your OPML file now.

Monday, February 23, 2015

Dropbear SSH vulnerabilities in stock Arduino Yun. How to update to OpenSSH remotely.

The Arduino Yún ships with Dropbear version 2011.54-2. There are multiple vulnerabilities with it and is not advised to be used. If your arduino is in a remote location and you want to update to OpenSSH, without losing remote access to the device, follow these steps.

  • Change the Dropbear port to an unused/free one on your box and restart Dropbear
    uci set dropbear.@dropbear[0].Port=2222
    uci commit dropbear
    /etc/init.d/dropbear restart
  • Reconnect to your Yun via SSH on the configured port above
  • Install the openssh-server
    opkg update
    opkg install openssh-server
  • Enable and start OpenSSH server. OpenSSH will listen now on port 22
    /etc/init.d/sshd enable
    /etc/init.d/sshd start
  • Reconnect to your yun via SSH on port 22
  • Now you can disable Dropbear
    /etc/init.d/dropbear disable
    /etc/init.d/dropbear stop
  • Install the openssh-sftp-server package to install support for the SFTP protocol which SSHFS uses
    opkg update
    opkg install openssh-sftp-server
Log into the Yun from the web address. Go to Configure > Advanced Configuration > System > Software and under Installed Packages and remove Dropbear.

High (CVSS: 7.1)
NVT: Dropbear SSH Server Use-after-free Vulnerability (OID:
This host is installed with Dropbear SSH Server and is prone to a use-after-free vulnerability.
Vulnerability Detection Result
Vulnerability was detected according to the Vulnerability Detection Method.
This flaw allows remote authenticated users to execute arbitrary code and bypass command restrictions via multiple crafted command requests, related to channels concurrency.
Updates are available.
Affected Software/OS
Versions of Dropbear SSH Server 0.52 through 2011.54 are vulnerable.
Vulnerability Insight
A use-after-free vulnerability exists in Dropbear SSH Server 0.52 through 2011.54 when command restriction and public key authentication are enabled.
Vulnerability Detection Method
Check the version.
Details: Dropbear SSH Server Use-after-free Vulnerability (OID:
Version used: $Revision: 809 $
CVE: CVE-2012-0920
BID: 52159
Medium (CVSS: 5.0)
NVT: Dropbear SSH Server Multiple Security Vulnerabilities (OID:
This host is installed with Dropbear SSH Server and is prone to multiple vulnerabilities.
Vulnerability Detection Result
Vulnerability was detected according to the Vulnerability Detection Method.
The flaws allows remote attackers to cause a denial of service or to discover valid usernames.
Updates are available.
Affected Software/OS
Versions prior to Dropbear SSH Server 2013.59 are vulnerable.
Vulnerability Insight
Multiple flaws are due to, - The buf_decompress function in packet.c in Dropbear SSH Server before 2013.59 allows remote attackers to cause a denial of service (memory consumption) via a compressed packet that has a large size when it is decompressed. - Dropbear SSH Server before 2013.59 generates error messages for a failed logon attempt with different time delays depending on whether the user account exists.
Vulnerability Detection Method
Check the version.
Details: Dropbear SSH Server Multiple Security Vulnerabilities (OID:
Version used: $Revision: 809 $
CVE: CVE-2013-4421, CVE-2013-4434
BID: 62958, 62993
CERT: DFN-CERT-2013-1865 , DFN-CERT-2013-1772

Saturday, February 14, 2015

Graph Elertus and Google Nest in Nagios/check_mk

I recently got turned onto OMD/check_mk, which is probably the most user friendly tool for Nagios. It has a nearly complete Nagios stack, and more:

  • Nagios
  • Monitoring Plugins
  • check_nrpe
  • mrpe (a check_mk clone of nrpe)
  • Icinga
  • Shinken
  • NagVis
  • pnp4nagios
  • rrdtool/rrdcached
  • Check_MK
  • MK Livestatus
  • Multisite
  • Dokuwiki
  • Thruk
  • Mod-Gearman
  • check_logfiles
  • check_oracle_health
  • check_mysql_health
  • jmx4perl
  • check_webinject
  • check_multi

It has a very simple install and works with Debian/Ubuntu/Red Hat/SUSE. I recently deleted my Icinga build and went all in with this.

As well as regular servers, vmware and other things, I'm able to graph Google Nest's temperature, humidity and "leaf" status. You can set up alerts if it crosses a certain threshold and graph for as many years and you wish.

Check_MK with Google Nest

This is something that is sorely lacking in Google's current site since the graphing information is only stored for 10 days, and doesn't provide the granular level of detail I want.

Google Nest's graphs.

While I won't go into much detail on the install of OMD (it's pretty straightforward) and I won't go into much detail about the Nest integration (Great writeup on how to do that here)  I will be showing a way to get another product, the Elertus, monitored with check_mk as an example of what you can do with it.

The Elertus, is a wifi temperature, sound, light and water detector that runs on batteries. I wrote up an article about it on this blog before. It's a little trickier to monitor than the Nest due to the fact that there is no API, no way to scrape information off of the device since it only makes outbound connections.

Since my last post, last year, nothing has changed with it, nor has my opinion. It's still sluggish, no graphing, no "all clear" alerts... and a painfully slow (and useless) app to compliment. The plus side is that it's been almost a year and the batteries still work (AA batteries). I have had no connection issues from it as far as I can tell.

Other than an alerts tab, there's not much more to the Elertus website

When I had my Icinga server, I stood up another box that basically sniffed the traffic as it was in delivery to their servers.(which is still sent cleartext, btw). I cut out the bits and pieces I needed to get a basic graph up. Since I started this server, I decided to try a new method.

Using pfSense, I created a static DCHP connection for the device and an internal NAT rule to relay the traffic from the Elertus to my own web server running Apache with mod_dumpio turned on.

While this basically kills any communication with the Elertus servers, they won't be missed. With, my prior setup I was able to enjoy both my internally generated alerts, and their alerts, but I found my own to be a lot more useful.

The Elertus sends out a POST to their servers as a check in.

device_type=1&posix_time=1423925134&, ant:I, af:, pkt:l14_t298_h30_b70_m0_i1_e1_x-1_p1423925134, wdog:1, crtry:4, queue:3, ctime:w2285_d410_n130_s205_t3040, &

Apache has a virtualhost set up to receive those incoming PHP POST requests with dumpio enabled and the trace level set to 7.

<IfModule dumpio_module>
 DumpIOInput On
 DumpIOOutput On
 LogLevel dumpio:trace7

I also made sure to set custom logs, for just this module, as they will be filling up quick.

ErrorLog ${APACHE_LOG_DIR}/dumpio_module_error.log
CustomLog ${APACHE_LOG_DIR}/dumpio_module_access.log combined

The requests come in 2 at a time, sometimes more if there is any movement/light. So, to remedy the lack of a consistent time period to check the logs, I set up a cron job to pull the newest line in the log with POST data. This runs every 5 minutes.


tac /var/log/apache2/dumpio_module_error.log | grep -m1 "email" > /tmp/tempout.txt


The scripts within it, in turn, pull information out of the latest POST and updates the raw values to separate files. I've kept it modular so I can add more checks as I need them. I'm not too concerned about the water, movement and light alerts just yet.

The temperature is in Kelvin, so I converted it to Fahrenheit.


tempk=$(cat /tmp/tempout.txt | awk -F "=" '/light/ {print $8}' | sed 's/&.*//')
tempb=$(awk "BEGIN {print "$tempk" - 273.15}")
temp=$(echo ""$tempb"*1.8+32" | bc)

echo "$temp" > ~/perfdata/temp
(I know, this can be cleaned up a lot)

The battery and humidity, are pretty much the same thing, with different names. It just requires another {print $X} position. The light, movement and water sensor can be added just as easily since they are only values of 0 or 1.

Now, unlike Nagios, getting devices to check_mk is a breeze. Making custom checks, with RRD graphing is just as easy. You add the custom checks to the host itself, not the monitoring server.

This web server is Ubuntu and it has check_mk installed. There is a folder that allows you to put custom scripts to make local checks, above what it already monitors.


The setup is pretty straightforward if you want monitoring with performance graphs.


TEMP1=$(cat ~/perfdata/temp)

echo '<<<local>>>'
echo "P Temperature temp="$TEMP1";35:89;32:91;0;110"

Any script that is in this folder runs when check_mk runs and grabs the other server global readings.

The output appends itself to the bottom of the generated file, as a local check like this:

P Temperature temp=73.13;35:89;32:91;0;110

It tells check_mk that it's available to be added as a service, with graphing

  • P

The name of the service

  • Temperature

The variable name

  • temp=

The output from $TEMP1, the WARN min:max, the CRIT min:max and UNKNOWN lower;upper (for graphing reasons and not required). Notice that the colons and semicolons are there for a reason.

When you scan the host in WATO on the main check_mk site, the host should have basic performance graphs and automatically added to the notification ruleset the host was part of.

While it's not an ideal solution, it gets the job done. I wish Elertus would open their API up to developers, the would probably sell a lot more units if they did.

I recommend anyone who's given up with Nagios/Icinga to give OMD a try. It's pretty good. The documentation is a little poor (and mostly in German), but with Google Translate and some coffee, you can get through it if you've ever set up any other monitoring before.

Friday, February 13, 2015

How to enable check_mk for pfSense 2.2

The official check_mk plugin (v0.1.1) for pfSense 2.2 does not work. Here are the steps to configure it manually.

Edit the rc.conf (notice, this location is different from the official FreeBSD)

vi /etc/rc.conf.local

At the bottom of /etc/services, add the check_mk port definition.
vi /etc/services
 check_mk        6556/tcp   #check_mk agent

Add the tcp wrappers for check_mk
vi /etc/inetd.conf
 check_mk        stream  tcp     nowait          root    /usr/local/bin/check_mk_agent check_mk

Add the following and replace x.x.x.x with your ip
vi /etc/hosts.allow
 # Allow nagios server to access us
 check_mk_agent : x.x.x.x : allow
 check_mk_agent : ALL : deny

Adding custom inetd scripts to allow service control
vi /etc/rc.d/inetd
 # $FreeBSD$
 # PROVIDE: inetd
 # KEYWORD: shutdown
 . /etc/rc.subr
 load_rc_config $name
 run_rc_command "$1"

Make it executable
chmod +x /etc/rc.d/inetd

This is the only check_mk agent I could find that would work with pfSense 2.2. Replace all the text, or create a new file (if creating a new file, make sure to chmod +x it)
vi /usr/local/bin/check_mk_agent
# +------------------------------------------------------------------+
# |             ____ _               _        __  __ _  __           |
# |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
# |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
# |           | |___| | | |  __/ (__|   <    | |  | | . \            |
# |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
# |                                                                  |
# | Copyright Mathias Kettner 2014    |
# +------------------------------------------------------------------+
# This file is part of Check_MK.
# The official homepage is at
# check_mk is free software;  you can redistribute it and/or modify it
# under the  terms of the  GNU General Public License  as published by
# the Free Software Foundation in version 2.  check_mk is  distributed
# in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
# out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
# PARTICULAR PURPOSE. See the  GNU General Public License for more de-
# ails.  You should have  received  a copy of the  GNU  General Public
# License along with GNU Make; see the file  COPYING.  If  not,  write
# to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
# Boston, MA 02110-1301 USA.

# Author: Lars Michelsen <>
#         Florian Heigl <>
#           (Added sections: df mount mem netctr ipmitool)

# NOTE: This agent has beed adapted from the Check_MK linux agent.
#       The most sections are commented out at the moment because
#       they have not been ported yet. We will try to adapt most
#       sections to print out the same output as the linux agent so
#       that the current checks can be used.

# This might be a good source as description of sysctl output:

# Remove locale settings to eliminate localized outputs where possible
export LC_ALL=C
unset LANG

export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_TMPDIR="/var/run/check_mk"

# Make sure, locally installed binaries are found

# All executables in PLUGINSDIR will simply be executed and their
# ouput appended to the output of the agent. Plugins define their own
# sections and must output headers with '<<<' and '>>>'

# All executables in LOCALDIR will by executabled and their
# output inserted into the section <<<local>>>. Please refer
# to online documentation for details.

# close standard input (for security reasons) and stderr
#if [ "$1" = -d ]
#    set -xv
#    exec </dev/null 2>/dev/null

# Runs a command asynchronous by use of a cache file

echo '<<<check_mk>>>'
echo Version: 1.2.7i1
echo AgentOS: freebsd

osver="$(uname -r)"
is_jailed="$(sysctl -n security.jail.jailed)"

# Partitionen (-P verhindert Zeilenumbruch bei langen Mountpunkten)
# Achtung: NFS-Mounts werden grundsaetzlich ausgeblendet, um
# Haenger zu vermeiden. Diese sollten ohnehin besser auf dem
# Server, als auf dem Client ueberwacht werden.

echo '<<<df>>>'
# no special zfs handling so far, the ZFS.pools plugin has been tested to
# work on FreeBSD
if df -T > /dev/null ; then
    df -kTP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)'
    df -kP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)' | awk '{ print $1,"ufs",$2,$3,$4,$5,$6 }'

# Check NFS mounts by accessing them with stat -f (System
# call statfs()). If this lasts more then 2 seconds we
# consider it as hanging. We need waitmax.
#if type waitmax >/dev/null
#    STAT_VERSION=$(stat --version | head -1 | cut -d" " -f4)
#    STAT_BROKE="5.3.0"
#    echo '<<<nfsmounts>>>'
#    sed -n '/ nfs /s/[^ ]* \([^ ]*\) .*/\1/p' < /proc/mounts |
#        while read MP
#  do
#   if [ $STAT_VERSION != $STAT_BROKE ]; then
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" || \
#    echo "$MP hanging 0 0 0 0"
#   else
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" && \
#      printf '\n'|| echo "$MP hanging 0 0 0 0"
#   fi
#  done

# Check mount options.
# FreeBSD doesn't do remount-ro on errors, but the users might consider
# security related mount options more important.
echo '<<<mounts>>>'
mount -p -t ufs

# processes including username, without kernel processes
echo '<<<ps>>>'
if [ is_jailed = 0 ]; then
    ps ax -o state,user,vsz,rss,pcpu,command | sed -e 1d  -e '/\([^ ]*J\) */d' -e 's/*\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\2,\3,\4,\5) /'
    ps ax -o user,vsz,rss,pcpu,command | sed -e 1d -e 's/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /'

# Produce compatible load/cpu output to linux agent. Not so easy here.
echo '<<<cpu>>>'
echo `sysctl -n vm.loadavg | tr -d '{}'` `top -b -n 1 | grep -E '^[0-9]+ processes' | awk '{print $3"/"$1}'` `sysctl -n kern.lastpid` `sysctl -n hw.ncpu`

# Calculate the uptime in seconds since epoch compatible to /proc/uptime in linux
echo '<<<uptime>>>'
  up_seconds=$(( `date +%s` - `sysctl -n kern.boottime  | cut -f1 -d\, | awk '{print $4}'`))
idle_seconds=$(ps axw | grep idle | grep -v grep | awk '{print $4}' | cut -f1 -d\: )

# second value can be grabbed from "idle" process cpu time / num_cores
echo "$idle_seconds $up_seconds"

# Platten- und RAID-Status von LSI-Controlleren, falls vorhanden
#if which cfggen > /dev/null ; then
#   echo '<<<lsi>>>'
#   cfggen 0 DISPLAY | egrep '(Target ID|State|Volume ID|Status of volume)[[:space:]]*:' | sed -e 's/ *//g' -e 's/:/ /'

# Multipathing is supported in FreeBSD by now
if kldstat -v | grep g_multipath > /dev/null ; then
    echo '<<<freebsd_multipath>>>'
    gmultipath status | grep -v ^Name

# Soft-RAID
echo '<<<freebsd_geom_mirrors>>>'
gmirror status | grep -v ^Name

# Performancecounter Kernel
echo "<<<kernel>>>"
date +%s
forks=`sysctl -n vm.stats.vm.v_forks`
vforks=`sysctl -n vm.stats.vm.v_vforks`
rforks=`sysctl -n vm.stats.vm.v_rforks`
kthreads=`sysctl -n vm.stats.vm.v_kthreads`
echo "cpu" `sysctl -n kern.cp_time | awk ' { print $1" "$2" "$3" "$5" "$4 } '`
echo "ctxt" `sysctl -n vm.stats.sys.v_swtch`
echo "processes" `expr $forks + $vforks + $rforks + $kthreads`

# Network device statistics (Packets, Collisions, etc)
# only the "Link/Num" interface has all counters.
echo '<<<lnx_if:sep(58)>>>'
date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'

# State of LSI MegaRAID controller via MegaCli.
# To install: pkg install megacli
if which MegaCli >/dev/null ; then
    echo '<<<megaraid_pdisks>>>'
    MegaCli -PDList -aALL -NoLog < /dev/null | egrep 'Enclosure|Raw Size|Slot Number|Device Id|Firmware state|Inquiry'
    echo '<<<megaraid_ldisks>>>'
    MegaCli -LDInfo -Lall -aALL -NoLog < /dev/null | egrep 'Size|State|Number|Adapter|Virtual'
    echo '<<<megaraid_bbu>>>'
    MegaCli -AdpBbuCmd -GetBbuStatus -aALL -NoLog < /dev/null | grep -v Exit

# OpenVPN Clients. 
# Correct log location unknown, sed call might also be broken
if [ -e /var/log/openvpn/openvpn-status.log ] ; then
    echo '<<<openvpn_clients:sep(44)>>>'
    sed -n -e '/CLIENT LIST/,/ROUTING TABLE/p' < /var/log/openvpn/openvpn-status.log  | sed -e 1,3d -e '$d' 

if which ntpq > /dev/null 2>&1 ; then
   echo '<<<ntp>>>'
   # remote heading, make first column space separated
   ntpq -np | sed -e 1,2d -e 's/^\(.\)/\1 /' -e 's/^ /%/'

# Checks for cups monitoring
#if which lpstat > /dev/null 2>&1; then
#  echo '<<<cups_queues>>>'
#  lpstat -p
#  echo '---'
#  for i in $(lpstat -p | grep -E "^(printer|Drucker)" | awk '{print $2}' | grep -v "@"); do
#    lpstat -o "$i"
#  done

# Heartbeat monitoring
#if which cl_status > /dev/null 2>&1; then
#  # Different handling for heartbeat clusters with and without CRM
#  # for the resource state
#  if [ -S /var/run/heartbeat/crm/cib_ro ]; then
#    echo '<<<heartbeat_crm>>>'
#    crm_mon -1 -r | grep -v ^$ | sed 's/^\s/_/g'
#  else
#    echo '<<<heartbeat_rscstatus>>>'
#    cl_status rscstatus
#  fi
#  echo '<<<heartbeat_nodes>>>'
#  for NODE in $(cl_status listnodes); do
#    if [ $NODE != $HOSTNAME ]; then
#      STATUS=$(cl_status nodestatus $NODE)
#      echo -n "$NODE $STATUS"
#      for LINK in $(cl_status listhblinks $NODE 2>/dev/null); do
#        echo -n " $LINK $(cl_status hblinkstatus $NODE $LINK)"
#      done
#      echo
#    fi
#  done

# Number of TCP connections in the various states
echo '<<<tcp_conn_stats>>>'
netstat -na | awk ' /^tcp/ { c[$6]++; } END { for (x in c) { print x, c[x]; } }'

# Postfix mailqueue monitoring
# Only handle mailq when postfix user is present. The mailq command is also
# available when postfix is not installed. But it produces different outputs
# which are not handled by the check at the moment. So try to filter out the
# systems not using postfix by searching for the postfix user.
# Cannot take the whole outout. This could produce several MB of agent output
# on blocking queues.
# Only handle the last 6 lines (includes the summary line at the bottom and
# the last message in the queue. The last message is not used at the moment
# but it could be used to get the timestamp of the last message.
#if which mailq >/dev/null 2>&1 && getent passwd postfix >/dev/null 2>&1; then
#  echo '<<<postfix_mailq>>>'
#  mailq | tail -n 6

#Check status of qmail mailqueue
#if type qmail-qstat >/dev/null
#   echo "<<<qmail_stats>>>"
#   qmail-qstat

# check zpool status
#if [ -x /sbin/zpool ]; then
#   echo "<<<zpool_status>>>"
#   /sbin/zpool status -x | grep -v "errors: No known data errors"

# Memory Usage
# currently we'll need sysutils/muse for this.
if [ -x /usr/local/bin/muse ]
echo '<<<mem>>>'
# yes, i don't know sed well.
muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'
swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'

# Fileinfo-Check: put patterns for files into /etc/check_mk/fileinfo.cfg
if [ -r "$MK_CONFDIR/fileinfo.cfg" ] ; then
    echo '<<<fileinfo:sep(124)>>>'
    date +%s
    stat -f "%N|%z|%m" $(cat "$MK_CONFDIR/fileinfo.cfg")

Restart inetd
service inetd restart

You should be able to telnet into it it from the IP you specified in /etc/hosts.allow
telnet x.x.x.x 6556

If you cannot reach that port, allow the port in the firewall rules.

Go to Firewall > Rules

Click the LAN tab.

Create new rule.

Set the rule to pass traffic, on interface LAN, with the source IP of your check_mk server and to port 6556.

Give it a description, then save.

You should be able to pick up checks from OMD/check_mk.

Thursday, November 27, 2014

Hot to set up pfSense software raid in 2.1.5-RELEASE (amd64)

Here is an example of how to create a software RAID1 in pfSense 2.1.5.

I created a lab in Virtualbox with two 8GB thin provisioned disks and installed pfSense. "pfsense.vdi "and "pfsense2.vdi"

During the install, I chose "1," to boot with the default settings.
The initialization screen defaults to the LiveCD installer. Skip that and press "I" to install directly.
Accept the default settings for the Video and Keymap
Then we want to choose "Setup GEOM Mirror"
Confirm the selection
Now we choose the Primary disk and press enter
Choose the Mirror disk and press enter.
Verify no errors exist. Press Enter.
Choose the Custom Install
Then we choose the mirror/pfSenseMirror we just created.
Format the disk
Use the default disk geometry (just tab to "Use this Geometry")
Format the mirror/pfSenseMirror
Choose Partition Disk
Accept and Create the default settings
Choose "Yes, partition mirror/pfSenseMirror"
Press "OK"
Now, we want to uncheck "Install Bootblock" and make sure "Packet mode" is unchecked as well.
Accept and install
Press "OK"
Choose the default partition slice.
Confirm "OK", then "OK" again
Choose the defaults for the subpartitions (tab to "Accept and Create")
Once the install writes to the mirror, choose "Symmetric multiprocessing kernel", unless you are creating a headless RS232 serial-only interface.
Eject the virtual CD and Reboot.
Once the system reboots, configure pfSense like normal.

We now have a RAID1 mirror of the disks. We can now test booting by removing either of the virtual disks and booting pfSense. In the lab, I've removed the primary disk "pfsense.vdi" and it's booting off the mirror "pfsense2.vdi"

pfSense lacks notification (by default) on a degraded RAID mirror. You can manually check the status of the disk health by going into the console and typing "gmirror status". You can also see the status of the mirror when I shut down the VM (at about the 58 second mark):
GEOM_MIRROR: Device pfSenseMirror destroyed.

To mimic rebuilding a disk, in my lab I created a new volume called "pfsense3.vdi" and made it a blank 8GB, thin provisioned disk to match what I was replacing.

To rebuild the disk, I first checked the status of the disks, "gmirror status"
I destroyed the mirror "gmirror forget pfSenseMirror"
Now, "gmirror status" shows COMPLETE (with just one disk, ad0)
I looked at which disks were present, "atacontrol list" and saw ad1 available and not part of the mirror. This is the new blank disk we want to become part of the mirror.
Inserted it into the mirror with "gmirror insert pfSenseMirror /dev/ad1"
It will start rebuilding. We can check the sync status with "gmirror status" again.
Once complete, you will get the message:
GEOM_MIRROR: Device pfSenseMirror: rebuilding provider ad1 finished.
This will take some time in a normal install. This VM was installed over a SSD on a blank install. Expect some time for it to synchronize.
"gmirror status" should now show us both ado and ad1 as ACTIVE