Have you ever wanted to keep track of every packet going through your firewall? How about getting some stats on the hosts using your network. Stats like most bandwidth used or most popular ports or ip's. Well NetFlow is what your looking for. NetFlow is an open but proprietary network protocol developed by Cisco Systems to run on Cisco IOS-enabled equipment for collecting IP traffic information.
Flow-tools is a suite of programs that takes Cisco NetFlow datagrams and generates reports based on the information it has recorded. The Cisco Netflow datagrams are generated by a program called Pfflowd.
The Pfflowd website describes it's programs functionality the best by saying: Converts OpenBSD PF status messages (sent via the pfsync interface) to Cisco NetFlow datagrams. These datagrams may be sent (via UDP) to a host of one's choice. Utilising the OpenBSD stateful packet filter infrastructure means that flow tracking is very fast and accurate.
Note: The PFflowd only accounts packets that get passed statefully. Since flow reporting is coupled to PF's state tracking, only traffic flows which are passed via a "keep state", "modulate state" or "synproxy state" rule are accounted. Blocked packets are not accounted for.
Below are some simple steps to getting this going. This install was done on OpenBSD 3.8.
1. Install python 2.4 from the version of OpenBSD your using. Follow the install instructions and make it the default Python for the system.
pkg_add -v http://openbsd.mirrors.pair.com/ftp/3.9/packages/i386/python-2.4.2p0.tgz
2. Install the Pfflowd package on any version of OpenBSD before 3.9.pkg_add -v http://openbsd.mirrors.pair.com/ftp/3.8/packages/i386/pfflowd-0.6.tgz
If your using OpenBSD 3.9 then you will need to download the latest version from CVS from the authors website. A fix was committed to CVS on the authors site in June 2006 to fix it. If you need that the site is here. Just download the Makefile, pfflowd.8, pfflowd.c, and pfflowd.h into a directory. Then just issue the "make" command. It should compile no problem in OpenBSD 3.9. Then copy the made binary to /usr/local/sbin/ and the man page to /usr/local/man/man8/. Last step is to create the user "_pfflowd". If you don't then Pfflowd will not start.
3. Install the flow-tools .68 or higher. As of OpenBSD 3.8 & 3.9 you will need to download the source and compile it as both only have version .67. The reason for this is we will need the new flow-rptfmt program that only comes with .68 and later. You can get the source from the flow-tools site. If any later version of OpenBSD comes with .68 or later just install the package and forget the source install. The commands to make and install flow-tools are below. This will install flow-tools to /usr/local/netflow.
cd /tmp
wget ftp://ftp.eng.oar.net/pub/flow-tools/flow-tools-0.68.tar.gz
tar xvzf flow-tools-0.68.tar.gz
cd flow-tools-0.68
./configure
make
make install
4. Let's make the directory for the log files then start the flow-tools collector. This will listen on localhost for Netflow datagrams sent from Pfflowd. It uses the program flow-capture from the flow-tools package to collect datagrams sent from Pfflowd. It will keep binary logs of the collected datagrams in /var/log/netflow. It will begin a new file every 10 mins. It listens on localhost port 12345. Make sure you allow this in your PF rules if need be.
mkdir /var/log/netflow
/usr/local/netflow/bin/flow-capture -w /var/log/netflow -n 143 127.0.0.1/0/12345
5. Let's start Pfflowd so it will start sending data to the flow-capture daemon. This starts the Pfflowd daemon which will send Netflow datagrams to the flow-tools daemon on localhost port 12345. You might need bring up the pfsync0 interface so this will work. So we will do that also. To get it to come up on reboot just make the file /etc/hostname.pfsync0 and put the work "up" on the first line. Then save it.
ifconfig pfsync0 up
/usr/local/sbin/pfflowd -n 127.0.0.1:12345
6. Now that it's collecting data we will need to create a config file that will filter the data before we generate a report. The following lines need to be put in a file to be accessed by reports.conf config file. The name of my file is nfilter.conf. This is an example of a filter that could be used to narrow down the logs to ip ranges and ports if you would like to.
############################ filter-primitive popularports type ip-port permit 80 permit 443 permit 20 permit 21 permit 22 permit 25 permit 53 permit 110 permit 143 default deny filter-primitive intif type ip-address-prefix permit 192.168.0.0/24 default deny filter-primitive intif-ip type ip-address permit 192.168.0.10 default deny filter-primitive 10min-ago type time-date permit lt 600 seconds ago ############################## filter-definition intif-src match ip-source-address intif filter-definition intif-dst match ip-destination-address intif filter-definition intif-src-popular match ip-source-address intif match ip-destination-port popularports filter-definition intif-ip1 match ip-source-address intif-ip
7. Now that we have filters for the data we need to generate the reports from the filtered data. The following lines need to be put in a file and called from the command line. I named mine reports.conf.
#The filter lines below refer to this file include-filter /etc/nfilter.conf #### Start: Source IP -> Dest Port. Filter Name: intif-src #### stat-report srcip-dstprt type ip-source-address/ip-destination-port filter intif-src output sort +octets records 15 stat-definition srcip-dstprt report srcip-dstprt #### End: Source IP -> Dest Port. Filter Name: intif-src #### #### Start: Destination IP -> Dest Port. Filter Name: intif-dst #### stat-report dstip-dstprt type ip-destination-address/ip-destination-port filter intif-dst output sort +octets records 15 stat-definition dstip-dstprt report dstip-dstprt #### End: Destination IP -> Dest Port. Filter Name: intif-dst #### #### Start: Source IP -> Dest Port. Filter Name: intif-src-popular #### stat-report srcip-dstprt-pop type ip-source-address/ip-destination-port filter intif-src-popular output sort +octets records 15 stat-definition srcip-dstprt-pop report srcip-dstprt-pop #### End: Source IP -> Dest Port. Filter Name: intif-src-popular ####
8. Now we need a script to tie all this together so it runs the programs from flow-tools to generate the reports in html format. This is a korn shell script. Change the variables on the top to fit your needs. The output is very simple and can be changed to suit your needs. Save this to a file and execute it (chmod +x filename if need be). This should dump a very simple html file to the path you set in the script. It will show the output from the current day it is run.
#!/bin/sh
# Add to your path statement in /root/.profile: /usr/local/netflow/bin/
# Variables for settings below
REPORTPATH=/var/www/htdocs
CFGFILES=/etc
LOGPATH=/var/log/netflow
HTMLFILENAME=netflow.shtml
YEAR=`date "+%Y"`
MONTH=`date "+%m"`
DAY=`date "+%d"`
createreport () {
flow-cat $LOGPATH/$YEAR/$YEAR-$MONTH/$YEAR-$MONTH-$DAY/ \
|flow-report -s$CFGFILES/reports.conf -S $1 \
|flow-rptfmt -fhtml >>$REPORTPATH/$HTMLFILENAME
echo "<br><br><br>" >>$REPORTPATH/$HTMLFILENAME
}
# Create html file
echo "<html><center>">$REPORTPATH/$HTMLFILENAME
# Run Create file function with report name
createreport dstip-dstprt
createreport srcip-dstprt
createreport srcip-dstprt-pop
# Close html file
echo "</center></html>">>$REPORTPATH/$HTMLFILENAME
9. Once you have verfied the script works then put it in a cron job to run every night. Since the script looks at the current day it is run I like to run mine right before the end of the day. Use the command "crontab -e" and insert the line below. This runs the script at 11:51 PM every night.
51 23 * * * /var/www/cgi-bin/reports.sh >>/dev/null 2>&1
View the file netflow.shtml your favorite web browser to see the stats.
That is a simple example. But I wanted an email with the stats every night. I did not find a way to get it into a nice format so I wrote a perl script to put the output in a nice e-mail friendly format. The script takes the output from flow-report and gives the "octets" column more human readable numbers in Bytes, Megabytes, and Gigabytes. It also lines up all the columns neatly and just spits out acsii. You can then redirect the output of the script to an email that can be sent each day. The script will read all of the current days (the day it's run) stats. It's rough but it gets the job done. Just run the script by itself to see the output on the command line. Don't forget to put in the report file below in /etc/
Below is the report file I use. I use the file name and path /etc/report.cfg. There is no filter being used.
stat-report ip-address type ip-address output sort +octets records 15 stat-definition ip-address report ip-address stat-report ip-source-address type ip-source-address output sort +octets records 15 stat-definition ip-source-address report ip-source-address stat-report ip-destination-address type ip-destination-address output sort +octets records 15 stat-definition ip-destination-address report ip-destination-address stat-report ip-source/destination-address type ip-source/destination-address output sort +octets records 15 stat-definition ip-source/destination-address report ip-source/destination-address stat-report ip-source-port type ip-source-port output sort +octets records 15 stat-definition ip-source-port report ip-source-port stat-report ip-destination-port type ip-destination-port output sort +octets records 15 stat-definition ip-destination-port report ip-destination-port stat-report ip-port type ip-port output sort +octets records 15 stat-definition ip-port report ip-port stat-report ip-source/destination-port type ip-source/destination-port output sort +octets records 15 stat-definition ip-source/destination-port report ip-source/destination-port ######### stat-report ip-source-address/ip-source-port type ip-source-address/ip-source-port output sort +octets records 15 stat-definition ip-source-address/ip-source-port report ip-source-address/ip-source-port stat-report ip-source-address/ip-destination-port type ip-source-address/ip-destination-port output sort +octets records 15 stat-definition ip-source-address/ip-destination-port report ip-source-address/ip-destination-port stat-report ip-destination-address/ip-source-port type ip-destination-address/ip-source-port output sort +octets records 15 stat-definition ip-destination-address/ip-source-port report ip-destination-address/ip-source-port stat-report ip-destination-address/ip-destination-port type ip-destination-address/ip-destination-port output sort +octets records 15 stat-definition ip-destination-address/ip-destination-port report ip-destination-address/ip-destination-port stat-report ip-source-address/ip-source/destination-port type ip-source-address/ip-source/destination-port output sort +octets records 15 stat-definition ip-source-address/ip-source/destination-port report ip-source-address/ip-source/destination-port stat-report ip-destination-address/ip-source/destination-port type ip-destination-address/ip-source/destination-port output sort +octets records 15 stat-definition ip-destination-address/ip-source/destination-port report ip-destination-address/ip-source/destination-port stat-report ip-source/destination-address/ip-source-port type ip-source/destination-address/ip-source-port output sort +octets records 15 stat-definition ip-source/destination-address/ip-source-port report ip-source/destination-address/ip-source-port stat-report ip-source/destination-address/ip-destination-port type ip-source/destination-address/ip-destination-port output sort +octets records 15 stat-definition ip-source/destination-address/ip-destination-port report ip-source/destination-address/ip-destination-port stat-report ip-source/destination-address/ip-source/destination-port type ip-source/destination-address/ip-source/destination-port output sort +octets records 15 stat-definition ip-source/destination-address/ip-source/destination-port report ip-source/destination-address/ip-source/destination-port
Below is the perl script that uses the above report config file. Change the variables at the top to what you want. Currently it's set to look for the report file /etc/report.cfg. It is also set to look in the path /var/log/flow for it's log files.
#!/usr/bin/perl
# Make sure all the flow-tools binaries are in your shells path.
$cfg_file="/etc/report.cfg";
$log_path="/var/log/flow";
# Get and fix up current date.
($Second, $Minute, $Hour, $Day, $Month, $Year, $WeekDay, $DayOfYear, $IsDaylightSavings) = localtime(time);
$Month += 1;
$Year += 1900;
if($Month < 10) { $Month = "0" . $Month; }
if ($Hour < 10) { $Hour = "0" . $Hour; }
if ($Minute < 10) { $Minute = "0" . $Minute; }
if ($Second < 10) { $Second = "0" . $Second; }
if($Day < 10) { $Day = "0" . $Day; }
# Run Create file function with report name
createreport ("ip-source-address");
createreport ("ip-destination-address");
createreport ("ip-address");
createreport ("ip-source/destination-address");
createreport ("ip-source-port");
createreport ("ip-destination-port");
createreport ("ip-port");
createreport ("ip-source/destination-port");
createreport ("ip-source-address/ip-source-port");
createreport ("ip-source-address/ip-destination-port");
createreport ("ip-destination-address/ip-source-port");
createreport ("ip-destination-address/ip-destination-port");
createreport ("ip-source-address/ip-source/destination-port");
createreport ("ip-destination-address/ip-source/destination-port");
createreport ("ip-source/destination-address/ip-source-port");
createreport ("ip-source/destination-address/ip-destination-port");
createreport ("ip-source/destination-address/ip-source/destination-port");
sub createreport {
@reportarray = (); #clear array
open(IN, "flow-cat $log_path/$Year/$Year-$Month/$Year-$Month-$Day/ |flow-report -s$cfg_file -S $_[0] |") or die ("Can't open file. Permissions?");
while( <IN> ) {
if ( /# recn: / || !/^#/ ) {
s/# recn: //;
chomp ();
push @reportarray, [ split "," ]; # Split elements into an array of arrays.
}
}
close(IN);
printout (); #print each report using the print subroutine
}
sub printout {
# loop through each element in first line. count postion's over till we hit octets. exit loop.
my $octethit=0;
for my $titlecount ( @{$reportarray[0]} ) {
if ( $titlecount eq "octets" ) {
last;
} else {
$octethit++;
}
}
# skip first array. then use octets count from above to find octets line. convert that to kb,mb,gb. write back to array.
for my $i ( 1 .. $#reportarray ) {
for my $j ( 0 .. $#{$reportarray[$i]} ) {
if ( $j == $octethit ) {
my $kilo = sprintf ("%.4f", ($reportarray[$i][$j]/1024));
my $kiloint = $reportarray[$i][$j]/1024;
my $mega = sprintf ("%.4f", ($kilo/1024));
my $megaint = $kilo/1024;
my $giga = sprintf ("%.4f", ($mega/1024));
my $gigaint = $mega/1024;
if ( $kiloint < 1 ) {
$reportarray[$i][$j] = "$reportarray[$i][$j] BT";
} elsif ( $megaint < 1 ) {
$reportarray[$i][$j] = "$kilo KB";
} elsif ( $gigaint < 1 ) {
$reportarray[$i][$j] = "$mega MB";
} else {
$reportarray[$i][$j] = "$giga GB";
}
next;
}
}
}
@maxlength = (); # clear array
for my $ii ( 0 .. $#reportarray ) {
for my $jj ( 0 .. $#{$reportarray[$ii]} ) {
$tmplength = length($reportarray[$ii][$jj]);
if ( $tmplength > $maxlength[$jj] ) {
$maxlength[$jj] = $tmplength;
}
}
}
# print to stdout all lines
for my $q ( 0 .. $#reportarray ) {
for my $r ( 0 .. $#{$reportarray[$q]} ) {
my $diff = $maxlength[$r] - length ($reportarray[$q][$r]);
print "$reportarray[$q][$r] ";
print ' ' x $diff;
}
print "\n";
}
print "\n";
}
This is the line from cron that emails the stats: /var/www/cgi-bin/flow-rptfmtprint.pl | /usr/bin/mailx -s "`uname -n` daily flow stats" root
