pantz.org banner
Shell oneliners
Posted on 07-25-2007 00:13:00 UTC | Updated on 01-03-2010 22:02:22 UTC
Section: /software/shell/ | Permanent Link

Below are one off lines that perform different tasks at a command prompt. The commands can use any number of programs usually found on some type of Unix machine. Things like awk or perl.

Warning: These lines were tested on Linux with a bash shell using the GNU versions of each piece of software. Your mileage may vary with your version of *nix and your shell. The different programs (mostly GNU) used here may be different versions or not the GNU version on your system. This means they may work differently or use (or not have) different command line options. If need be check your man pages to adapt the commands below to your OS.

Any awk statements below assume the awk defaults which is spiting fields on spaces unless specified with -F. If using the awk statement and you want to work on a file instead of stdin then put a filename at the end of the awk statement. FYI: Awk refers to lines from files as "Records" and each "Record" has "fields". "Fields" are sections of characters (words) with delimiters between them.

Numbers and Counting

Print the total number of lines that have the name Bill.

awk '/Bill/{n++}; END {print n+0}' filename

Print line numbers using a tab instead of a space.

awk '{print FNR "\t" $0}' filename

Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be separated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.

awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head

Fields

Print the last field of each line of the file.

awk '{ print $NF }' filename

Count the lines in a file. Just like "wc -l".

awk 'END{print NR}' filename

Total the number of lines that contain the name Jose

awk '/Jose/{n++}; END {print n+0}' filename

Add up the numbers at the eighth position (field) of each line. Print the total.

awk '{ sum += $8 } END { print sum }' filename

If the fourth field starts with a number then print that fourth field.

awk '$4 ~ /^[0-9]/ { print $4 }' filename

Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

awk '{gsub(/dog|cat|bird,"pet");print}' filename

Find and replace "dog" with "cat" in every file with extension txt.

awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt

Find every line that begins with cat. In that line replace furry with nothing. Change the file called filename inline (-i).

sed -i '/^cat/{s/furry//}' filename

Find cat by itself on it's own line even if there are spaces or tabs before it or after it. Replace it with dog. Then print the line.

awk '{sub(/^[ \t]*cat .*$/,"dog");print}' filename

Find any line starting with the defined shell variable SHELLVAR (notice ' ' around it so it's evaluated). When the line is found substitute in foo or boo or coo with bar. Edit the file inline.

sed -i '/^'${SHELLVAR}'/s/\(foo\|boo\|coo\)/bar/' filename

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

awk '{sub(/\r$/,"");print}' filename

Remove all carriage returns from file and rewrite the edits back to same file. tr uses the octal form for cr.

tr -d "\015" < filename | tee > filename

From a unix os: Convert Unix newlines (LF) to DOS format.

awk '{sub(/$/,"\r");print} filename

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

awk '{sub(/^[ \t]+/, ""); print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Insert 6 blank spaces at beginning of each line.

awk '{sub(/^/, "      ");print}' filename

Substitute "dog" with "cat" on lines that don't contain the word "pet".

awk '!/pet/{gsub(/dog/, "cat")};{print}' filename

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

awk -F: '{print $1 " " $2}' filename

Swap the first 2 fields.

awk '{tmp = $1; $1 = $2; $2 = tmp; print}' filename

Remove the second field in each line and then print it.

awk '{ $1 = ""; print }' filename

Line Operations

Print the first 6 lines of a file.

awk 'NR <= 6' filename

Print the last line of a file

awk 'END{print}' filename

Print the lines matching the regular expression. (emulates grep).

awk '/regex_here/' filename

Print the lines that don't match the regular expression. (emulates grep -v).

awk '!/regex_here/' filename

Print the line before the regular expression match.

awk '/regex_here/{print x};{x=$0}' filename

Print the line after the regular expression match.

awk '/regex_here/{getline; print}' filename

Print the lines less than 50 characters.

awk 'length < 50' filename

Print the lines 20 through 30.

awk 'NR==20,NR==30' filename

Print the line 50.

awk 'NR==50 {print;exit}' filename

Print lines between the match starting at "Dog" and ending at "Cat".

awk '/Dog/,/Cat/' filename

File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or try the programs pkill or killall.

ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

dd if=/dev/zero of=/tmp/zero bs=1k count=2000

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

find / -type f -name "*.txt" -print | xargs rm

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

find . -inum 128128 | xargs rm

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for finding file names with spaces at the end.

ls -lA | cat -e

To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:

rm \ ^L

Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.

rsync -av --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/

Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is back grounded so all commands are executed one after another.

for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done

Set group id and sticky bits on a dir.

chmod g=swrx,+t DIR/

Use wget grab all of a certain type of file listed on a web page and put them in the current dir. The example will use jpeg's.

wget -r -l1 --no-parent -A "*.jpg" http://www.website.com/pictures/

Reddit!

Related stories


RSS Feed RSS feed logo

About


3com

3ware

alsa

alsactl

alsamixer

amd

android

apache

areca

arm

ati

auditd

awk

badblocks

bash

bind

bios

bonnie

cable

carp

cat5

cdrom

cellphone

centos

chart

chrome

chromebook

cifs

cisco

cloudera

comcast

commands

comodo

compiz-fusion

corsair

cpufreq

cpufrequtils

cpuspeed

cron

crontab

crossover

cu

cups

cvs

database

dbus

dd

dd_rescue

ddclient

debian

decimal

dhclient

dhcp

diagnostic

diskexplorer

disks

dkim

dns

dos

dovecot

drac

dsniff

dvdauthor

e-mail

echo

editor

emerald

encryption

ethernet

expect

ext3

ext4

fat32

fedora

fetchmail

fiber

filesystems

firefox

firewall

flac

flexlm

floppy

flowtools

fonts

format

freebsd

ftp

gdm

gmail

gnome

google

gpg

greasemonkey

greylisting

growisofs

grub

hacking

hadoop

harddrive

hba

hex

hfsc

html

html5

http

https

hulu

idl

ie

ilo

intel

ios

iperf

ipmi

iptables

ipv6

irix

javascript

kde

kernel

kickstart

kmail

kprinter

krecord

kubuntu

kvm

lame

ldap

linux

logfile

lp

lpq

lpr

maradns

matlab

memory

mencoder

mhdd

mkinitrd

mkisofs

moinmoin

motherboard

mouse

movemail

mplayer

multitail

mutt

myodbc

mysql

mythtv

nagios

nameserver

netflix

netflow

nginx

nic

ntfs

ntp

nvidia

odbc

openbsd

openntpd

openoffice

openssh

openssl

openvpn

opteron

parted

partimage

patch

perl

pf

pfflowd

pfsync

photorec

php

pop3

pop3s

ports

postfix

power

procmail

proftpd

proxy

pulseaudio

putty

pxe

python

qemu

r-studio

raid

recovery

redhat

router

rpc

rsync

ruby

saltstack

samba

schedule

screen

scsi

seagate

seatools

sed

sendmail

sgi

shell

siw

smtp

snort

solaris

soundcard

sox

spam

spamd

spf

spotify

sql

sqlite

squid

srs

ssh

ssh.com

ssl

su

subnet

subversion

sudo

sun

supermicro

switches

symbols

syslinux

syslog

systemd

systemrescuecd

t1

tcpip

tcpwrappers

telnet

terminal

testdisk

tftp

thttpd

thunderbird

timezone

ting

tls

tools

tr

trac

tuning

tunnel

ubuntu

unbound

vi

vpn

wget

wiki

windows

windowsxp

wireless

wpa_supplicant

x

xauth

xfree86

xfs

xinearama

xmms

youtube

zdump

zeromq

zic

zlib