pantz.org banner
IPMI conflicts with RPC
Posted on 09-18-2008 02:46:52 UTC | Updated on 09-18-2008 03:38:13 UTC
Section: /hardware/ipmi/ | Permanent Link

I have a friend that had a really weird problem with a program that used RPC. Every once and a while the program that used RPC would open a port and tell the client to contact on the port it just opened. The client would try to do that and fail. Problem was this only happened randomly. Random problems are a pain to track down. The problem was eventually tracked down. The following is a description of what it was and the fix for it.

RPC uses a range of ports on Linux to allow clients to contact the server. By default these are 650-1023. RPC will pick a port at random from the range of ports. That is as long as port in the range is not already being used. Their servers have built in Baseband Management Controllers (BMC). BMC communicates with a BMC management utility (BMU) on a remote client using IPMI protocols. The BMC for the servers are in band and use the network controller on the motherboard to send and recieve data. The ports the BMC uses are 623 and 664.The thing is the controllers don't run any software in Linux to do this. So how do they work you ask? Well, they intercept the traffic on those ports before it gets into the OS's network stack. That means any data you send to those ports on those machines the OS will never see. You won't see it on any network sniffer on that machine at all. So any machine that RPC told to connect to it on 664 was failing. But it was random because that port would only come up from the pool of ports randomly.

Now that you know what the problem is how do you solve it? You could just run a dummy process out of xinetd to listen on port 664 (and 623 if your RPC uses ports that low) and be done with it.

You could also try to bump up your port range to start at 665. That can be done on some 2.6 linux kernels in proc.

echo 665 > /proc/sys/sunrpc/min_resvport

Restart your daemons that use RPC. If that works then you can put line sunrpc.min_resvport=665 in /etc/sysctl.conf so it sets on reboot. I have heard that some of the latest Linux kernels are going to up the port range to start after 665 to avoid this problem in the future.

RPC is most notably used by NFS. This can cause NFS mounts to fail randomly or just any other program that uses RPC.

Reddit!

Related stories


RSS Feed RSS feed logo

About


3com

3ware

alsa

alsactl

alsamixer

amd

android

apache

areca

arm

ati

auditd

awk

badblocks

bash

bind

bios

bonnie

cable

carp

cat5

cdrom

cellphone

centos

chart

chrome

chromebook

cifs

cisco

cloudera

comcast

commands

comodo

compiz-fusion

corsair

cpufreq

cpufrequtils

cpuspeed

cron

crontab

crossover

cu

cups

cvs

database

dbus

dd

dd_rescue

ddclient

debian

decimal

dhclient

dhcp

diagnostic

diskexplorer

disks

dkim

dns

dos

dovecot

drac

dsniff

dvdauthor

e-mail

echo

editor

emerald

ethernet

expect

ext3

ext4

fat32

fedora

fetchmail

fiber

filesystems

firefox

firewall

flac

flexlm

floppy

flowtools

fonts

format

freebsd

ftp

gdm

gmail

gnome

google

greasemonkey

greylisting

growisofs

grub

hacking

hadoop

harddrive

hba

hex

hfsc

html

html5

http

https

hulu

idl

ie

ilo

intel

ios

iperf

ipmi

iptables

ipv6

irix

javascript

kde

kernel

kickstart

kmail

kprinter

krecord

kubuntu

kvm

lame

ldap

linux

logfile

lp

lpq

lpr

maradns

matlab

memory

mencoder

mhdd

mkinitrd

mkisofs

moinmoin

motherboard

mouse

movemail

mplayer

multitail

mutt

myodbc

mysql

mythtv

nagios

nameserver

netflix

netflow

nginx

nic

ntfs

ntp

nvidia

odbc

openbsd

openntpd

openoffice

openssh

openssl

openvpn

opteron

parted

partimage

patch

perl

pf

pfflowd

pfsync

photorec

php

pop3

pop3s

ports

postfix

power

procmail

proftpd

proxy

pulseaudio

putty

pxe

python

qemu

r-studio

raid

recovery

redhat

router

rpc

rsync

ruby

saltstack

samba

schedule

screen

scsi

seagate

seatools

sed

sendmail

sgi

shell

siw

smtp

snort

solaris

soundcard

sox

spam

spamd

spf

spotify

sql

sqlite

squid

srs

ssh

ssh.com

ssl

su

subnet

subversion

sudo

sun

supermicro

switches

symbols

syslinux

syslog

systemd

systemrescuecd

t1

tcpip

tcpwrappers

telnet

terminal

testdisk

tftp

thttpd

thunderbird

timezone

ting

tls

tools

tr

trac

tuning

tunnel

ubuntu

unbound

vi

vpn

wget

wiki

windows

windowsxp

wireless

wpa_supplicant

x

xauth

xfree86

xfs

xinearama

xmms

youtube

zdump

zeromq

zic

zlib