Below are one off lines that perform different tasks at a command prompt. The commands can use any number of programs usually found on some type of Unix machine. Things like awk or perl.
Warning: These lines were tested on Linux with a bash shell using the GNU versions of each piece of software. Your mileage may vary with your version of *nix and your shell. The different programs (mostly GNU) used here may be different versions or not the GNU version on your system. This means they may work differently or use (or not have) different command line options. If need be check your man pages to adapt the commands below to your OS.
Any awk statements below assume the awk defaults which is spiting fields on spaces unless specified with -F. If using the awk statement and you want to work on a file instead of stdin then put a filename at the end of the awk statement. FYI: Awk refers to lines from files as "Records" and each "Record" has "fields". "Fields" are sections of characters (words) with delimiters between them.
Print the total number of lines that have the name Bill.
awk '/Bill/{n++}; END {print n+0}' filename
Print line numbers using a tab instead of a space.
awk '{print FNR "\t" $0}' filename
Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be separated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.
awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head
Print the last field of each line of the file.
awk '{ print $NF }' filename
Count the lines in a file. Just like "wc -l".
awk 'END{print NR}' filename
Total the number of lines that contain the name Jose
awk '/Jose/{n++}; END {print n+0}' filename
Add up the numbers at the eighth position (field) of each line. Print the total.
awk '{ sum += $8 } END { print sum }' filename
If the fourth field starts with a number then print that fourth field.
awk '$4 ~ /^[0-9]/ { print $4 }' filename
Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.
awk '{gsub(/dog|cat|bird,"pet");print}' filename
Find and replace "dog" with "cat" in every file with extension txt.
awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt
Find every line that begins with cat. In that line replace furry with nothing. Change the file called filename inline (-i).
sed -i '/^cat/{s/furry//}' filename
Find cat by itself on it's own line even if there are spaces or tabs before it or after it. Replace it with dog. Then print the line.
awk '{sub(/^[ \t]*cat .*$/,"dog");print}' filename
Find any line starting with the defined shell variable SHELLVAR (notice ' ' around it so it's evaluated). When the line is found substitute in foo or boo or coo with bar. Edit the file inline.
sed -i '/^'${SHELLVAR}'/s/\(foo\|boo\|coo\)/bar/' filename
From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).
awk '{sub(/\r$/,"");print}' filename
Remove all carriage returns from file and rewrite the edits back to same file. tr uses the octal form for cr.
tr -d "\015" < filename | tee > filename
From a unix os: Convert Unix newlines (LF) to DOS format.
awk '{sub(/$/,"\r");print} filename
Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.
awk '{sub(/^[ \t]+/, ""); print}' filename
Delete the trailing whitespace (spaces or tabs) from end of each line.
awk '{sub(/[ \t]+$/, "");print}' filename
Delete leading and trailing whitespace from each line.
awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}' filename
Delete the trailing whitespace (spaces or tabs) from end of each line.
awk '{sub(/[ \t]+$/, "");print}' filename
Insert 6 blank spaces at beginning of each line.
awk '{sub(/^/, " ");print}' filename
Substitute "dog" with "cat" on lines that don't contain the word "pet".
awk '!/pet/{gsub(/dog/, "cat")};{print}' filename
Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).
awk -F: '{print $1 " " $2}' filename
Swap the first 2 fields.
awk '{tmp = $1; $1 = $2; $2 = tmp; print}' filename
Remove the second field in each line and then print it.
awk '{ $1 = ""; print }' filename
Print the first 6 lines of a file.
awk 'NR <= 6' filename
Print the last line of a file
awk 'END{print}' filename
Print the lines matching the regular expression. (emulates grep).
awk '/regex_here/' filename
Print the lines that don't match the regular expression. (emulates grep -v).
awk '!/regex_here/' filename
Print the line before the regular expression match.
awk '/regex_here/{print x};{x=$0}' filename
Print the line after the regular expression match.
awk '/regex_here/{getline; print}' filename
Print the lines less than 50 characters.
awk 'length < 50' filename
Print the lines 20 through 30.
awk 'NR==20,NR==30' filename
Print the line 50.
awk 'NR==50 {print;exit}' filename
Print lines between the match starting at "Dog" and ending at "Cat".
awk '/Dog/,/Cat/' filename
Find a program by name from process listing that is not awk and kill it. Or try the programs pkill or killall.
ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill
Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.
dd if=/dev/zero of=/tmp/zero bs=1k count=2000
From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.
find / -type f -name "*.txt" -print | xargs rm
To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.
find . -inum 128128 | xargs rm
Mark the end of each line with a dollar sign so you can see where the filename ends. Good for finding file names with spaces at the end.
ls -lA | cat -e
To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:
rm \ ^L
Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.
rsync -av --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/
Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is back grounded so all commands are executed one after another.
for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done
Set group id and sticky bits on a dir.
chmod g=swrx,+t DIR/
Use wget grab all of a certain type of file listed on a web page and put them in the current dir. The example will use jpeg's.
wget -r -l1 --no-parent -A "*.jpg" http://www.website.com/pictures/
