pantz.org banner
Auditd crashes when overloaded
Posted on 10-06-2008 17:46:21 UTC | Updated on 10-06-2008 18:41:17 UTC
Section: /software/auditd/ | Permanent Link

On CentOS 5.2 I found out that the audit daemon will crash when put under heavy load with it's default settings. I was using the CIS Benchmarks to help lock down some machines and made some of the changes they suggest for auditd. After making the changes I ran one of the scripts to find files that did not belong to any user or group currently on the system. Here is that line.

for PART in $(grep -v '^#' /etc/fstab | awk '( $3 ~ "ext[23]" ) { print $2 }' );
 do
  find $PART -xdev -nouser -o -nogroup -print ;
 done

About halfway through the find the auditd would die with these messages in the syslog:

auditd[6135]: Cannot allocate audit reply
auditd[6135]: Cannot allocate audit reply, exiting
auditd[6135]: Error receiving audit netlink packet (No buffer space available)
auditd[6135]: Error setting audit daemon pid (No buffer space available)

The netlink error was because of the messages going to a syslog server getting cut off. That still did not explain why it died in the first place. After much googling of the error I found nothing. The only time I saw this error was from the source code of the original program. You know it's bad when that's the only spot the error shows up in.

After reading the man pages a few times for auditctl and bumping up the buffers (-b) to a massive number nothing would keep it from crashing on the find. Finally I saw the -r setting which is for rate limiting the messages. The -r setting in audit.rules file will set the limit in messages/sec. If the rate is non-zero and is exceeded, the failure flag is consulted by the kernel for action. The failure action could be do nothing, print a message, or just panic. Default panic action is to print a message. The default setting for rate limit is 0 which means no limit. With no limit the find must have been killing auditd. Setting the limit to 21000 (-r 21000) in the /etc/audit/audit.rules fixed it.

Just a warning. I believe this will keep some things from logging when it hits it's rate limit. So if your environment must have all audit logs then this might not be for you. The rate of 21000 is what I found to be high enough on my system so it would not crash auditd. You should experiment with bumping the limit higher if you can.

Reddit!

Related stories


RSS Feed RSS feed logo

About


3com

3ware

alsa

alsactl

alsamixer

amd

android

apache

areca

arm

ati

auditd

awk

badblocks

bash

bind

bios

bonnie

cable

carp

cat5

cdrom

cellphone

centos

chart

chrome

chromebook

cifs

cisco

cloudera

comcast

commands

comodo

compiz-fusion

corsair

cpufreq

cpufrequtils

cpuspeed

cron

crontab

crossover

cu

cups

cvs

database

dbus

dd

dd_rescue

ddclient

debian

decimal

dhclient

dhcp

diagnostic

diskexplorer

disks

dkim

dns

dos

dovecot

drac

dsniff

dvdauthor

e-mail

echo

editor

emerald

ethernet

expect

ext3

ext4

fat32

fedora

fetchmail

fiber

filesystems

firefox

firewall

flac

flexlm

floppy

flowtools

fonts

format

freebsd

ftp

gdm

gmail

gnome

google

greasemonkey

greylisting

growisofs

grub

hacking

hadoop

harddrive

hba

hex

hfsc

html

html5

http

https

hulu

idl

ie

ilo

intel

ios

iperf

ipmi

iptables

ipv6

irix

javascript

kde

kernel

kickstart

kmail

kprinter

krecord

kubuntu

kvm

lame

ldap

linux

logfile

lp

lpq

lpr

maradns

matlab

memory

mencoder

mhdd

mkinitrd

mkisofs

moinmoin

motherboard

mouse

movemail

mplayer

multitail

mutt

myodbc

mysql

mythtv

nagios

nameserver

netflix

netflow

nginx

nic

ntfs

ntp

nvidia

odbc

openbsd

openntpd

openoffice

openssh

openssl

openvpn

opteron

parted

partimage

patch

perl

pf

pfflowd

pfsync

photorec

php

pop3

pop3s

ports

postfix

power

procmail

proftpd

proxy

pulseaudio

putty

pxe

python

qemu

r-studio

raid

recovery

redhat

router

rpc

rsync

ruby

saltstack

samba

schedule

screen

scsi

seagate

seatools

sed

sendmail

sgi

shell

siw

smtp

snort

solaris

soundcard

sox

spam

spamd

spf

sql

sqlite

squid

srs

ssh

ssh.com

ssl

su

subnet

subversion

sudo

sun

supermicro

switches

symbols

syslinux

syslog

systemrescuecd

t1

tcpip

tcpwrappers

telnet

terminal

testdisk

tftp

thttpd

thunderbird

timezone

ting

tls

tools

tr

trac

tuning

tunnel

ubuntu

unbound

vi

vpn

wget

wiki

windows

windowsxp

wireless

wpa_supplicant

x

xauth

xfree86

xfs

xinearama

xmms

youtube

zdump

zeromq

zic

zlib