Back in 2012 I wrote an entry that said to not use FreeBSD 9.0 as a PF firewall. It was mildly controversial and I was just relaying my experience of installing FreeBSD 9.0 on release day. It was a frustrating experience that I really wanted to work out and I was sadly disappointed that it did not. I wanted to try it again in the future in hopes that the FreeBSD team could put together something that work work better for me. That time is now.
Reading about all of the improvements in FreeBSD 10 was very encouraging. Some of the most encouraging things that inspired me to try again was a fork of PF. The FreeBSD devs forked PF because Gleb Smirnoff wrote an amazing patch to make PF SMP-friendly. But this was not compatible with the newest PF code so hence the fork. Making PF SMP aware improves performance immensely. OpenBSD has not done this nor does it look like it is on their radar.
Since I tried it last the binary package system for FreeBSD has been rewritten. The new package system pkgng is a major improvement over the old package system. Now FreeBSD actually has a package manager that can deal with dependencies well and we can live off of binaries instead of ports (Yea!).
I have been really interested in using ZFS as a file system, and our friends over at Calomel.org have convinced me that the time is right to try it. ZFS has added TRIM support in FreeBSD 10. ZFS also has added LZ4 compression, which is an amazing new fast compression algorithm. Plus you can now install ZFS as your root drive right from the installer. I do wish they would add the ability to choose options like compression during your ZFS install, but hopefully they will add this as advanced options in the future.
The install went great. I followed the great instructions from our friends over at Calomel.org and made my usb boot drive. I fetched the script from Calomel.org and changed a few things. I ran the script and a few mins later I rebooted and hit a nice big error screen.
The error was "ahcich1: Timeout on slot 30". This error was referring to the first channel on my SATA port where my hard drive lives. I could not figure out why it was having this issue until I checked my BIOS. My drives were set to IDE instead of AHCI. For some reason FreeBSD installed fine using IDE as the setting, but on first boot it fell flat on its face. AHCI is better and is what I really wanted it set to, but I never bothered checking. After changing to AHCI in the BIOS FreeBSD booted right up.
The install was done on my internal network so I had the ability to use another machine to do the install and download everything I needed to get it done. Then I tried hooking it up to my external firewall. I could not get lease from Verizon's DHCP server to save my life.
In the install script I used you have to change your external interface to DHCP and change (spoof) its MAC address to match the MAC that Verizon has in their system. This could be the MAC of your original router or your Verizon router. That MAC was changed correctly to what I have on the OpenBSD machine the was hooked up, but I could not get a lease from the Verizon DHCP server at all.
Long story short, with Verizon you have to release the DHCP lease before you can get a new one. With FreeBSD there is no way to release a DHCP lease with the version of dhclient that comes with FreeBSD. In the Linux version of dhclient you can. I had to end up hooking up my old Verizon router and push the "Release" button on the GUI interface. After doing that and quickly powering off the router I booted right up and got a lease. Now, I also believe if I waited enough hours I could have waited for the lease to expire, but I wanted to get this done so I found a faster way.
Installing and configuring the system was pretty easy. I worked through many of the issues I had last time. Let me give you some examples.
Before I used to use Postfix with SASL to talk to remote mail servers using TLS. I now use OpenSMTPD to do this as it is built in and they also have built in CA Certs for Gmail. That solved the annoyance of having to build Postfix from ports. As far as I can tell FreeBSD still does not build Postfix with SASL, but they do build Sendmail with SASL. Go figure.
PF still works great in FreeBSD, but they still do not build in ALTQ as a kernel module or into the kernel. This is not as big of an issue for me as it once was since turning it off I did not see any noticeable difference on my home network. For a big corp that has lots of different users this might be different. For me I can live without it for now as turning it on requires a kernel recompile and I'm trying to avoid that. OpenBSD is abandoning its work on ALTQ as they said it was a hack in the fist place and the code is showing its age with limited 32 bit registers and the like. They are working on a new queuing system called prio. I hope FreeBSD can eventually work this into their fork of PF or make one of their own. What I really want back from ALTQ is priority ACKs. I wish someone would hack in priority acks into the tcp stack as a kernel tunable. That would rock.
All PF tools I installed from packages this time and they worked great right out of the box. No fuss at all. This was a big improvement from last time as most of the PF tools were broken on release day.
In the last entry I bid farewell to FreeBSD, but this time I'm waving bye bye to OpenBSD. OpenBSD we need to talk. You are falling behind. FreeBSD is eating your lunch on performance and features.
I'm not saying I won't be back OpenBSD, but I'm going to ride the FreeBSD train for now and see where it takes me. Thanks for all of the stability and security you have given me over the years. Good luck and I hope you can start addressing the issues I mentioned above.
Update2:I have updated my experience with trying to use FreeBSD 10 as a PF firewall. Spoiler alert, it goes much better than this. Please read the new review for an update.
Update1: Let me preface this article by saying that the below install was done on 9.0 release day. I've been told that on release day ports might not be totally up to speed. The packages mentioned below that were broke have been reported to me as fixed. I have not checked this myself. In any event every word below is true and reflects a FreeBSD 9.0 install on release day.
It seems like every 3 or 4 years I try out FreeBSD to see if it can replace my OpenBSD firewall. I was assembling a new firewall and decided to try the just released FreeBSD 9.0. It had so many cool new features and most importantly it had PF as an available packet filter. I would be replacing an older install of PF and my rulsets would have worked perfectly on this box without any modification (Later releases of PF changed the structure of the rules).
The process started out great. Put a pre-made usb image of the installer on a old usb stick. OpenBSD does not offer this so score one for FreeBSD. During install you can turn on Trim support for your filesystems if you have an SSD. OpenBSD does not have this either. Score two for Free. The install was a breeze. This was looking fantastic so far. Logged in for the first time and did an update. That went very well. Unfortanatly, it was a downward spiral from there.
Before doing any of my PF setup I needed to get a few packages installed that I use on my firewall. I use Postfix as a mail relay on my network. Postfix talks to my ISP via SASL and TLS. Any machine on my network can send mail to it and it will relay that mail through the ISP. I install the FreeBSD prebuilt package for Postfix. I setup the config and fire up Postfix. I send a test email that does not go through. Checking the logs it tells me SASL is not built into Postfix. No problem I think. OpenBSD has a seperate package built with SASL for Postfix, surely FreeBSD has done the same right? Wrong! Crap, now we have to use ports.
In FreeBSD ports is a collection of files you will need to compile (build) applications. I thought I could get through a full system setup and not use the ports system like I can on OpenBSD. I was sadly mistaken about this. As I find out later with PF and Postfix and who knows what else, unless you have the most basic of setups your going to need ports with FreeBSD. So I go to install the files for ports since I did not do it during install. The fantastic FreeBSD handbook guides you through installing ports. One little issue. The FreeBSD handbook has not be updated for FreeBSD 9.0. FreeBSD 9.0 does not use sysinstall anymore yet they have not disabled it. So it looks like it might work but then bombs out. It took a while to find this out no thanks to the handbook. Many google searches point to using sysinstall to install ports. I took some other advice from the handbook and just used csup and portsnap to get the source. Not as easy but it finally worked. I got Postfix compiled with SASL and it worked fine after it installed.
I installed a few other basic packages I needed from the precompiled packages and then started on PF. I checked the handbook again on PF just to make sure there were no suprises. Suprise, I find out ALTQ is not built into the FreeBSD kernel, nor is it built as a kernel module for the generic kernel. Really? You can't even build it as a kernel module so it can be loaded if need be. Good grief. Now we have to build a new kernel with ALTQ. Glad we already have ports. ALTQ is built into the generic OpenBSD kernel by default. Now I'm starting to wonder if this was a good idea. I built the new kernel with ALTQ in it and the install went great. I'm not done yet but I can't take much more of this constant building of things that just seem to work on OpenBSD. But I'm a trooper so I continue.
Now that PF w/ALTQ is working we need some tools to help with managing pf. Pftop is a fantastic way to view all of the traffic going through your PF firewall in realtime. It is a must have for anyone using PF as a firewall. I can't say I'm shocked that there is no precompiled package for it. That seems to be the theme. On to ports then. I switch to ports and run my make to start the compile. Low and behold I get this nice message "PFtop port is broke ===> pftop-0.7_1 is marked as broken: does not compile on 9.X". Are you f'ing kidding me! Broken! Thats just great. Well I wonder, how about another PF package I want to install called PFflowd. I switch to that ports dir and run a make. I get "PFFlowd is broke "===> pfflowd-0.7 is marked as broken: does not compile.". That is my breaking point. Both of these can be installed as packages in OpenBSD in about 10 seconds. That is when I knew I was done with FreeBSD.
I wanted this to work out so bad. Your community looks so much friendler than OpenBSD's. You focus on performance and more cutting edge things than OpenBSD, but alas when it comes to being PF firewall you stink. Your PF ports are broken, you have to compile ALTQ into the kernel or a module, and even your Postfix package needs to be recompiled to support SASL. I'm sure your good at many other things like webservers or big filesystems using ZFS, but you don't seem to give to much love to PF or its packages. Hopefully in the future all the packages will be fixed by 9.1, and someone will make the decision that ALTQ is worthy of being compiled into the generic kernel (or as a module). I wish you the best FreeBSD
One of the reasons I fought so hard to stay with FreeBSD was for the TRIM support it's filesystem offered for my SSD. Also, FreeBSD supported the old PF ruleset format I had, so I would not have had to update my rules. Doing more research I found out that my SSD has a built in garbage collection routine so TRIM support was not a must, it would just help expedite cleanup. After reading that I was willing to just update the PF rules so I could get back to a nice simple OpenBSD box. PF is made by the OpenBSD group and its no wonder why they have so much support for it. I learned a lot about FreeBSD in this process but the journey was way to long and invloved. My install of OpenBSD went smoothly, and all of the packages for PF installed fine and worked without issue. Postfix w/SASL installed right from a package and there were no kernel recompiles. Also, there was no need to load the OpenBSD ports collection which saved me a ton of space (did I mention FreeBSD ports was a few Gigs just by itself). The whole OpenBSD install was less than 1 Gig. When you can run your whole distro from pre made packages it can really cut down on disk space and time to install.
I tried to stray but nobody does PF better than the creator. The grass was not greener. The simple and fast install is a pleasure to use. The minimal disk space it takes up is rare these days. The package maintainers make multiple versions of popular packges with different options compiled in so each person can have what they want. OBSD has everthing a person could want when making a firewall using PF. I do wish that in the future they will update the filesystem with some speed improvements and more features. Also, possibly make a bootable install image that can easily be put on a memory stick like FreeBSD does. Time to head over to the OpenBSD store to buy some things to help support the cause.
My OpenBSD firewalls all have default block policies. Any connection attempt not initiated from the inside will be blocked by the firewall and logged in the PF log files. Usually these hits are people/scripts scanning for open ports. I thought it would be fun to see what ports people are scanning for and what ip's they are from.
To do this I needed a script to parse the log files and add up all blocks. So I made a perl script to do this. I am not a fantastic Perl programmer but the script does what I need it to. There is not much error checking but like I said it works. It is not intended for gigantic log files but it does not do to bad with them. Feel free to make modifications and if your so inclined send them to me. I just might throw them up if I like them. The script runs from a cron job and dumps the stats to an html file on my webservers. I can access it from a browser on the firewall every night to see who's scanning for what. Watch out for the settings like thresholds. If you set them to high you might not get any data back. Same goes for turning on port filtering.
Script Output Example
Log file Data from 10-05-2007 21:00:14 to 10-06-2007 13:24:02
Showing data from 10-06-2007 13:20:00 to 10-06-2007 13:24:00
Script run on: 10-06-2007 17:54:08
|
|
|
|
|
| ||||||||||||||||||||||||||||
|
| ||||||||||||||||||||||||||||||
|
|
| ||||||||||||||||||||||||||||||
Below is the code for the perl script that generates the example above. Just configure it and run it. It's pretty well commented on what to do.
#!/usr/bin/perl -s
# This perl script generates some stats (in html format) for OpenBSD's firewall
# PF. It was made with a home firewall in mind. One that has a default block
# in policy. Which means any packets hitting the firewall inbound that were
# not allowed by other rules or not part of another connection don't belong.
# These are the ones we want to see. Most likely these have malicious intent.
# The stats are generated from the binary formated PF logs. OpenBSD's
# implementation of Tcpdump is the only Tcpdump that can read this log. This
# script was tested on an OpenBSD 4.2 machine with the version of PF and
# Tcpdump that comes with 4.2. The script only does TCP and UDP stats
# currently. Also, it only does stats for block in rules and only for
# destination ports and source IP addresses.
# The external programs needed to run this script are: Tcpdump (OpenBSD's
# version), gzcat, and of course Perl. The "host" program is need if you want
# to do hostname lookups. All programs mentioned come with OpenBSD by default.
# You will also need to run this script as root because the permissions on
# the pflog files can only be read by root.
# I have recently tested this script on FreeBSD 6.3 successfully. The
# only thing to watch out for is FreeBSD compresses it's logfiles using
# bzip2 by default. You will need to change the gzcat line in the script
# to bzcat for it to work. FreeBSD's Tcpdump was modified to work with PF.
# I welcome any additions or added features to the script just send them to
# webmaster at pantz dot org. This code is free to distribute as long as credit
# is given to the website (pantz.org) in the modified code.
use Time::Local;
################################
# Start Configuration Settings #
################################
# You can set any variable below on the command line in the form of: -[variable]=[value]
# Example: -interface=em0 would set the $interface variable to 'em0'.
# Arrays (begin with @) can't be set from command line.
# Command line settings overides variables set below.
# Interface to be evaluated (usually the external network interface).
$interface = "fxp0" unless $interface;
# Path to output html file.
$pfhtmlfile = "/var/www/pantz.org/stats/pfhtmlstats.html" unless $pfhtmlfile;
# Analyze one file in the dir or all files.
# Set to "one" for one file.
# Set to "all" for all pf files (even compressed ones)
# in the /var/log dir that start with name "pflog".
$oneorallfiles = "all" unless $oneorallfiles;
# Full path to a single pflog file you want analyzed (compressed or uncompressed).
# Used if var above is set to "one". Example: /home/user/pflog.0.gz
$pflogfile = "/var/log/pflog" unless $pflogfile;
# Set to "exclude" to exclude only certain ports in stats output.
# "include" to include only certian ports in stats output.
# "off" to show all ports in stats output.
$exclude_include_ports = "off" unless $exclude_include_ports;
# Port list (array) for the above setting. For ranges use ".." in between the # range.
# Example of ports 1 thru 10 and 22 and 30 would be: 1..10,22,30
@in_ex_clude_ports = (99..100);
# Threshold values. Will not show or count "# of blocks in" totals from a host
# to a port or ip at or below the value set. Ex. Set to 10. No port or ip hit
# count (per host) that totaled 10 or below is counted.
$src_ip_threshold_value = 0 unless $src_ip_threshold_value;
$src_port_threshold_value = 0 unless $src_port_threshold_value;
# Set to "on" to have hostnames instead of ip's in the output. Set to "off" for just ip addresses.
# If this is turned on the script will take longer to run. Depending on the ammout of unique hosts.
$hostname_lookup = "off" unless $hostname_lookup;
# Filter by date? Settings are: Yes or No.
$date_filter = "no" unless $date_filter;
# Type of filter. Specific date range, current date and time run (now) backwards, or end of the current day backwards.
# Settings are: range, now, or curday.
$filter_type = "now" unless $filter_type;
# If now filter then set Units of time. Settings are: sec, min, hour, or day.
$time_unit = "hour" unless $time_unit;
# Amount of time units. How far back from now or end of current day? Whole number.
$time_amount = "12" unless $time_amount;
# If filtering by date range then specify date range. Form: YearMonthDayHourMinSec.
# Pad all single digits with zero. Ex 5 becomes 05.
# December 6th, 2006 20:05:01 would be: 20061206200501.
$lower_date = 20070205000000 unless $lower_date;
$upper_date = 20070206235959 unless $upper_date;
##############################
# End Configuration Settings #
##############################
#####Begin: Assembling date/time code.#####
$curtimeformat = format_epoch(time);
if ($date_filter eq "yes") {
if ($filter_type eq "range") {
($low_year,$low_month,$low_day,$low_hour,$low_min,$low_sec) = ($lower_date =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/);
($up_year,$up_month,$up_day,$up_hour,$up_min,$up_sec) = ($upper_date =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/);
# timelocal, it expects values with the same range as those which localtime returns.
# Namely, months start at 0, and years need 1900 subtracted from them.
$low_year -= 1900;
$low_month -= 1;
$up_year -= 1900;
$up_month -= 1;
# DMYHMS to epoch seconds corrected for year and month -1900 and -1 respectively.
$low_epochsec = timelocal($low_sec, $low_min, $low_hour, $low_day, $low_month, $low_year);
$up_epochsec = timelocal($up_sec,$up_min,$up_hour,$up_day,$up_month,$up_year);
} elsif ($filter_type eq "now") {
$up_epochsec = timelocal(localtime(time));
calc_low_epoch_sec();
} elsif ($filter_type eq "curday") {
($cur_second, $cur_minute,$cur_hour,$cur_Day,$cur_month,$cur_year) = localtime(time);
$up_epochsec = timelocal(($cur_second, $cur_minute,$cur_hour,$cur_Day,$cur_month,$cur_year) = (59,59,23,$cur_Day,$cur_month,$cur_year));
calc_low_epoch_sec();
} else {
die ("Date filter was requested but no filter type matched. Check your date filter type.");
}
}
#####End: Assembling date/time code.#####
#####Begin: Get one or all pflog filenames.#####
if ($oneorallfiles eq "all") {
@pflogfilenames = </var/log/pflog*>;
if ($date_filter eq "yes") {
foreach $pflogfilesstat (@pflogfilenames) {
($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat $pflogfilesstat;
# Don't open log files with dates not in our date range. Using last modify time on log file as date.
if ( $mtime >= $low_epochsec ) {
push(@newpflogfilenames, "$pflogfilesstat");
}
}
@pflogfilenames = @newpflogfilenames;
}
@pflogfilenames = sort { $b cmp $a } @pflogfilenames;
} else {
push(@pflogfilenames, "$pflogfile");
}
#####End: Get one or all pflog filenames.#####
#####Begin: Reading pflog file(s) and inputing data for sorting and minipulation.#####
foreach $pflogfilename (@pflogfilenames) {
open(IN, "gzcat -f $pflogfilename | tcpdump -nettqr - 2>&1 |") or die ("Can't open file. Permissions?");
$isfirstline = 0;
while( <IN> ) {
($line_date,$line_points,$line_rulenum,$line_action,$line_interface,$line_src_host,$line_src_port,$line_dst_host,$line_dst_port,$line_remainder)
= ($_ =~ /(\d+)\.(\d+) rule (\d+).*\(match\)\:?? (\w+ \w+) \w+ (\w+)\: (\d+\.\d+\.\d+\.\d+)\.(\d+) > (\d+\.\d+\.\d+\.\d+)\.(\d+)\:(.*)/);
# If line had no match with date it is probably no good. Go to next line. When a line finally matches increase by one.
next unless $line_date;
$isfirstline++;
# Get date from first and last line of each file. Put in array.
if ($isfirstline == "1" || eof(IN)) { push(@date_array, "$line_date"); }
next unless (/block in/ && /$interface/);
$exclude_include_port_match = "no";
# If using date filter go to the next log line if date does not fall within set filters.
if ($date_filter eq "yes") {
next unless ($line_date >= $low_epochsec && $line_date <= $up_epochsec)
}
if ($line_remainder =~ /tcp/) {
if ($exclude_include_ports ne "off") {
foreach $exin_ports (@in_ex_clude_ports) {
if ($exin_ports eq $line_dst_port) {
$exclude_include_port_match = "yes";
}
}
if (($exclude_include_port_match eq "yes") && ($exclude_include_ports eq "include")) {
push_tcp();
next;
}
if (($exclude_include_port_match eq "no") && ($exclude_include_ports eq "exclude")) {
push_tcp();
next;
}
} else {
push_tcp();
next;
}
}
if ($line_remainder =~ /udp/) {
if ($exclude_include_ports ne "off") {
foreach $exin_ports (@in_ex_clude_ports) {
if ($exin_ports eq $line_dst_port) {
$exclude_include_port_match = "yes";
}
}
if (($exclude_include_port_match eq "yes") && ($exclude_include_ports eq "include")) {
push_udp();
next;
}
if (($exclude_include_port_match eq "no") && ($exclude_include_ports eq "exclude")) {
push_udp();
next;
}
} else {
push_udp();
next;
}
}
#if ($line_remainder =~ /icmp/) {
# push_icmp();
#}
}
close(IN);
}
#####End: Reading pflog file(s) and inputing data for sorting and minipulation.#####
#####Begin: Output of HTML file.#####
open(PFHTMLSTATS, ">$pfhtmlfile") or die ("Can't create html file");
print PFHTMLSTATS qq{
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head><title>Pantz PFlog Stats</title>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
</head>
<body style="text-align: center;">
<h2>Pantz PFlog Stats</h2>
};
print PFHTMLSTATS "<p>Log file Data from " . format_epoch($date_array[0]) . " to " . format_epoch($date_array[-1]) . "<br></p>\n";
if ($date_filter eq "yes") {
print PFHTMLSTATS "<p>Showing data from " . format_epoch($low_epochsec) . " to " . format_epoch($up_epochsec) . "<br></p>\n";
}
print PFHTMLSTATS qq {
<p>Script run on: $curtimeformat <br></p>
<TABLE BORDER="0">
<tr><td valign="top">
<TABLE BORDER="1">
};
# Create host totals. Hash of hash. With threshold evaluation. Lookup host name if set. Sort most to least hits.
for $tcud_src_hstct_key ( keys %tcud_src_hstct ) {
for $tcud_src_hst_dst_portct_key ( keys %{ $tcud_src_hstct{$tcud_src_hstct_key} } ) {
# Check if our totals meet the threshold set. If so add it to a total for that ip.
if ($tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key} > $src_ip_threshold_value) {
$porthitcount += $tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key};
}
}
# Lookup hostname if var set. Else set to host's ip.
if ($hostname_lookup eq "on") {
hostnamelookup ($tcud_src_hstct_key);
} else {
$key_hostname = "$tcud_src_hstct_key";
}
# Make hash of arrays. IP, hostname, and hit count. Only if we met thresholds.
if ($porthitcount != 0) {
$tcud_src_hst_tot_ct{$tcud_src_hstct_key} = [$key_hostname,$porthitcount];
}
$porthitcount = 0;
}
# Print heading line with hostname if var is set. If not just print the ip heading line.
if ($hostname_lookup eq "on") {
print PFHTMLSTATS "<tr><td><b>Hostname</b></td><td><b>Source IP</b></td>
<td><b># of blocks in</b></td></tr>\n";
} else {
print PFHTMLSTATS "<tr><td><b>Source IP</b>
</td><td><b># of blocks in</b></td></tr>\n";
}
# Print hostname, ip, and count if hostname var is set. Otherwise just print ip and count lines.
for $print_tot_src_ip_key1 ( sort { $tcud_src_hst_tot_ct{$b}[1] <=> $tcud_src_hst_tot_ct{$a}[1] } keys %tcud_src_hst_tot_ct ) {
if ($hostname_lookup eq "on") {
print PFHTMLSTATS "<tr><td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[0]</td>
<td><a href=\"#IP:$print_tot_src_ip_key1\">$print_tot_src_ip_key1</a>
</td><td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[1]</td></tr>\n";
} else {
print PFHTMLSTATS "<tr><td><a href=\"#IP:$print_tot_src_ip_key1\">$print_tot_src_ip_key1</a></td>
<td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[1]</td></tr>\n";
}
}
print PFHTMLSTATS qq{
</table>
</td> <td></td> <td valign="top">
<TABLE BORDER="1">
<tr><td><b>Destination Port</b></td><td><b># of blocks in</b></td></tr>
};
# Create destination port totals. Hash of hash. With threshold evaluation. Sort most to least hits.
for $tcud_dst_portct_key1 ( keys %tcud_dst_portct ) {
# Check if our totals meet the threshold set. If so add it to a total for that port.
for $tcud_dst_portct_key2 ( keys %{ $tcud_dst_portct{$tcud_dst_portct_key1} } ) {
if ($tcud_dst_portct{$tcud_dst_portct_key1}{$tcud_dst_portct_key2} > $src_port_threshold_value) {
$iphitperportcount += $tcud_dst_portct{$tcud_dst_portct_key1}{$tcud_dst_portct_key2};
}
}
# Create hash of ip and port count. Only if we met thresholds.
if ($iphitperportcount != 0) {
$tcud_dst_port_tot_ct{$tcud_dst_portct_key1} = $iphitperportcount;
}
$iphitperportcount = 0;
}
# Print the hash from most to least port hits.
foreach $print_tot_dst_port_key (sort { $tcud_dst_port_tot_ct {$b} <=> $tcud_dst_port_tot_ct {$a} } keys %tcud_dst_port_tot_ct) {
print PFHTMLSTATS "<tr><td><a href=\"#PORT:$print_tot_dst_port_key\">$print_tot_dst_port_key</a></td>
<td>$tcud_dst_port_tot_ct{$print_tot_dst_port_key}</td></tr>\n";
}
print PFHTMLSTATS qq {\n</table>\n</td></tr>\n</table>\n\n<p><br><br></p><hr><p><br><br></p>};
$hostportcount = 4;
$hostporttablerowcount = 4;
$was_anything_printed = 0;
print PFHTMLSTATS "\n\n<TABLE BORDER=\"0\">\n\n";
# Start looping thru hash of hash. Print tables if thresholds were met.
for $tcud_src_hstct_key ( keys %tcud_src_hstct ) {
# Start a new row if our counter is reset.
if ($hostportcount == $hostporttablerowcount) {
print PFHTMLSTATS "<tr valign=\"top\">\n\n";
$hostportcount = 0;
}
for $tcud_src_hst_dst_portct_key (sort { $tcud_src_hstct{$tcud_src_hstct_key}{$b} <=> $tcud_src_hstct{$tcud_src_hstct_key}{$a} } keys %{
$tcud_src_hstct{$tcud_src_hstct_key} } ) {
# Check if threshold is met for total host ip hits to a port.
if ($tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key} > $src_ip_threshold_value) {
# Trip the counter if threshold was met.
$was_anything_printed++;
# If this is our first time through and we met the threshold print the table heading.
if ($was_anything_printed == 1) {
print PFHTMLSTATS "<td><TABLE BORDER=\"1\">\n";
print PFHTMLSTATS "<tr><td colspan=\"2\"><a name=\"IP:$tcud_src_hstct_key\"></a>
<a href=\"http://ws.arin.net/cgi-bin/whois.pl\?queryinput=$tcud_src_hstct_key\">
<b>$tcud_src_hstct_key</b></a></td></tr><tr><td><b>Destination Port</b>
</td><td> <b># of blocks in</b></td></tr>\n";
}
# Print a table row.
print PFHTMLSTATS "<tr><td><a href=\"http://isc.sans.org/port_details.php\?port=$tcud_src_hst_dst_portct_key\">
$tcud_src_hst_dst_portct_key</a></td><td>
$tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key}</td>
</tr>\n";
}
}
# If something was printed at least once increase column counter. Close our table.
if ($was_anything_printed > 0) {
$hostportcount++;
print PFHTMLSTATS "</table></td><td> </td>\n\n";
}
# If host port counter has it its max (set above) close the whole row.
if ($hostportcount == $hostporttablerowcount) {
print PFHTMLSTATS "</tr><tr><td> </td></tr>\n\n";
}
$was_anything_printed = 0;
}
# If hash ended before row finished then close it.
if ($hostportcount < $hostporttablerowcount) {
print PFHTMLSTATS "<td> </td></tr>\n";
}
print PFHTMLSTATS "\n</table>\n\n<p><br><br></p><hr><p><br><br></p>\n";
$porthostcount = 4;
$porthosttablerowcount = 4;
$was_anything_printed = 0;
print PFHTMLSTATS "\n\n<TABLE BORDER=\"0\">\n";
for $tcud_dst_portct_key3 ( keys %tcud_dst_portct ) {
# Start a new row if our counter is reset
if ($porthostcount == $porthosttablerowcount) {
print PFHTMLSTATS "<tr valign=\"top\">\n\n";
$porthostcount = 0;
}
for $tcud_dst_portct_key4 (sort { $tcud_dst_portct{$tcud_dst_portct_key3}{$b} <=> $tcud_dst_portct{$tcud_dst_portct_key3}{$a} } keys
%{$tcud_dst_portct{$tcud_dst_portct_key3} } ) {
# Check if threshold is met for total host port hits to a ip.
if ($tcud_dst_portct{$tcud_dst_portct_key3}{$tcud_dst_portct_key4} > $src_port_threshold_value) {
# Trip the counter if threshold was met.
$was_anything_printed++;
# If this is our first time through and we met the threshold print the table heading.
if ($was_anything_printed == 1) {
print PFHTMLSTATS "<td><TABLE BORDER=\"1\">\n";
print PFHTMLSTATS "<tr><td colspan=\"2\"><b><a name=\"PORT:$tcud_dst_portct_key3\"></a>
<a href=\"http://isc.sans.org/port_details.php\?port=$tcud_dst_portct_key3\">
$tcud_dst_portct_key3</a></b></td></tr><tr><td><b>Source IP</b></td>
<td> <b># of blocks in</b></td></tr>\n";
}
# Print a table row.
print PFHTMLSTATS "<tr><td><a href=\"http://ws.arin.net/cgi-bin/whois.pl\?queryinput=$tcud_dst_portct_key4\">
$tcud_dst_portct_key4</a></td><td>
$tcud_dst_portct{$tcud_dst_portct_key3}{$tcud_dst_portct_key4}</td></tr>\n";
}
}
# If something was printed increase column counter
if ($was_anything_printed > 0) {
$porthostcount++;
print PFHTMLSTATS "</table></td><td> </td>\n\n";
}
# If host port counter has it its max (set above) close the whole row.
if ($porthostcount == $porthosttablerowcount) {
print PFHTMLSTATS "</tr><tr><td> </td></tr>\n\n";
}
$was_anything_printed = 0;
}
# If hash ended before row finished then close it.
if ($porthostcount < $porthosttablerowcount) {
print PFHTMLSTATS "<td> </td></tr>\n\n";
}
print PFHTMLSTATS "\n</table>\n</body></html>";
close(PFHTMLSTATS);
#####End: Output of HTML file.#####
#####Start: Subroutines.#####
sub push_tcp {
# Hash of a hash. Value is increasing counter.
$tcud_src_hstct{$line_src_host}{$line_dst_port}++;
$tcud_dst_portct{$line_dst_port}{$line_src_host}++;
}
sub push_udp {
# Hash of a hash. Value is increasing counter.
$tcud_src_hstct{$line_src_host}{$line_dst_port}++;
$tcud_dst_portct{$line_dst_port}{$line_src_host}++;
}
sub push_icmp {
#push(@icmp_src_hst_array, "$line_src_host");
#push(@icmp_dst_hst_array, "$line_dst_host");
}
sub hostnamelookup {
# Use the program "host" to do reverse lookups on ip's.
$key_hostname = $_[0];
$key_hostname = `host $key_hostname`;
if ($key_hostname =~ /pointer/) {
($key_hostname) = ($key_hostname =~ /pointer (.*)/);
$key_hostname = substr($key_hostname,0, -1);
} else {
$key_hostname = "No PTR Record";
}
}
sub format_epoch {
my ($fmt_second, $fmt_minute, $fmt_hour, $fmt_Day, $fmt_month, $fmt_year) = localtime($_[0]);
$fmt_month += 1;
$fmt_year += 1900;
if($fmt_month < 10) { $fmt_month = "0" . $fmt_month; }
if ($fmt_hour < 10) { $fmt_hour = "0" . $fmt_hour; }
if ($fmt_minute < 10) { $fmt_minute = "0" . $fmt_minute; }
if ($fmt_second < 10) { $fmt_second = "0" . $fmt_second; }
if($fmt_Day < 10) { $fmt_Day = "0" . $fmt_Day; }
return "$fmt_month-$fmt_Day-$fmt_year $fmt_hour:$fmt_minute:$fmt_second";
}
sub calc_low_epoch_sec {
if ($time_unit eq "min") { $time_amount *= 60; }
if ($time_unit eq "hour") { $time_amount *= 3600; }
if ($time_unit eq "day") { $time_amount *= 86400; }
return $low_epochsec = $up_epochsec - $time_amount;
}
#####End: Subroutines.#####
Changes
07-01-04 - First change log entry. Added date sort so filenames are read from
oldest to newest. Done via filenames numbers in order.
- Fixed date range date so it evaluates all lines not just block in lines.
- Added interface filter.
11-23-04 - Added port breakdown by IP address's.
- Added quick navigation to IP address port breakdown.
11-27-04 - Added IP breakdown by port numbers.
- Added quick navigation to ports IP address breakdown.
12-06-04 - Added IP and Port thresholds for output.
12-01-05 - Fixed pattern matching for PF logs in 3.8.
02-01-06 - Rewrote code to make it simpler and slightly faster.
Fixed thresholds so all counts reflect correct numbers.
10-25-06 - changed tcpdump switch to dump dates as unix epoch time.
- changed regex to use new date format.
- put in "next" lines to skip evaluation of rules we don't want
or have matched. This increased speed and resource usage.
10-26-06 - Added ability to set config settings from commandline. perl -s.
- Added basic module to convert DMYHMS to epoch seconds.
- Added ability to only process lines in a specific date range.
10-06-07 - Fixed html output so it is HTML 4.01 Strict.
11-12-07 - Fixed problem where if first line of a log file was not a matched
line the date would not be set correctly for the "Log data from" line.
10-31-08 - Fixed include exclude ports that was broken because I forgot to rename
some old vars to the new name.
Possible To Do's
1. Number of blocks in by days of the week or just date. Do a ascii graphic with symbol. 2. Do something with ICMP. 3. Keep cache of stats from last run to help with large log files. 4. Check log file extention. If it is bzip2 then use bzcat instead of zcat.
