Linux Commands

Useful Linux commands. Covering grep, tr, awk, sed, sort, comm, uniq, wc, find, curl, tar, tee, cut, paste

grep

Initial data, using -v to invert empty string and show everything between start ^ and end $

❯ grep -v '^$' location
United States
SINGAPORE
Malaysia
Papua New Guinea
South Korea

Find lines that match start and end using ^ and $

❯ grep '^South' location
South Korea
❯ grep 'Korea$' location
South Korea

Case insensitive search, using -i

❯ grep -i 'singapore' location
SINGAPORE

Count matches, case insensitive, using -c

❯ grep -ci 'singapore' location
1

Display file names that match pattern, case insensitive, using -l

❯ grep -li 'singapore' *
location

Look for the whole word in file. To search through all files and display result, use *. Add -n to show line number

❯ grep -wi 'singapore' *
location:SINGAPORE

❯ grep -nwi 'singapore' *
location:2:SINGAPORE

❯ grep -wi 'singapore' location
SINGAPORE

Invert result of pattern match, using -v

❯ grep -i 's' location
United States
SINGAPORE
Malaysia
South Korea

❯ grep -vi 's' location
Papua New Guinea

Show lines after, before, or before+after a pattern match, using -A, followed by the number of lines to display.

❯ grep -A1 '^Malay' location
Malaysia
Papua New Guinea

❯ grep -B1 '^Malay' location
SINGAPORE
Malaysia

❯ grep -C1 '^Malay' location
SINGAPORE
Malaysia
Papua New Guinea

Search recursively for a pattern in the directory

❯ grep -iR 'singapore' .
./location:SINGAPORE

tr

Used for translating or deleting characters. Supports transformations such as uppercase to lowercase, squeezing repeating characters, deleting specific characters and basic find and replace.

❯ cat location
United States
SINGAPORE
Malaysia
Papua New Guinea
South Korea

Translate lower to uppercase, vice versa

❯ cat location | tr '[:lower:]' '[:upper:]'
UNITED STATES
SINGAPORE
MALAYSIA
PAPUA NEW GUINEA
SOUTH KOREA
❯ cat location | tr '[:upper:]' '[:lower:]'
united states
singapore
malaysia
papua new guinea
south korea

❯ cat location | tr '[a-z]' '[A-Z]'
UNITED STATES
SINGAPORE
MALAYSIA
PAPUA NEW GUINEA
SOUTH KOREA

Translate whitespaces into tables. Use [:space:] if you want to match any whitespace character, including spaces, tabs, newlines, etc.

❯ cat location | tr ' ' '\t'
United	States
SINGAPORE
Malaysia
Papua	New	Guinea
South	Korea

❯ cat location | tr '[:space:]' '\t'
United	States	SINGAPORE	Malaysia	Papua	New	Guinea	South	Korea	%

Translate characters into other character

❯ cat location | tr 'SE' '*'
United *tates
*INGAPOR*
Malaysia
Papua New Guinea
*outh Korea

Remove repetitive characters using -s

❯ echo "Sentence    with  too many   spaces" | tr -s " "
Sentence with too many spaces

Delete specified characters using -d

❯ echo "Sentence    with  too many   spaces" | tr -s " " | tr -d e
Sntnc with too many spacs

Delete digits

❯ echo "My number is 1234 5588" | tr -d '[:digit:]'
My number is

❯ echo "My number is 1234 5588" | tr -d '0-9'
My number is

Use completement with -c to inverse the deleted characters

❯ echo "My number is 1234 5588" | tr -cd '[:digit:]'
12345588%

awk

sed

sort

Sort a file, arrange record in particular order. Sorts file assuming contents are ASCII.

-r : Sorts data in reverse order, descending
-k : Sorts table based on specific column number
-c : Check if file is already sorted and reports disorder
-u : Sorts and removes duplicate lines, providing unique sorted list
-R : Sorts by random hash of keys but groups identical keys together
-n : Sorts numeric fields by arithmetic value. A numeric field may contain leading blanks, an optional minus sign, decimal digits, thousands-separator characters, and an optional radix character. Numeric sorting of a field containing any nonnumeric character gives unpredictable results.
-d : Sorts using dictionary order. Only letters, digits, and spaces are considered in comparisons.
-M : Sorts by months (Jan-Dec)

Sort alphabetically, by default

❯ sort number_string
five
four
one
three
two

And in reverse order, using -r

❯ sort -r number_string
two
three
one
four
five

Sort by the 3rd column using -k

❯ paste number alphabet number_string
1	a	one
2	b	two
3	c	three
4	d	four
5	e	five
❯ paste number alphabet number_string | sort -k3
5	e	five
4	d	four
1	a	one
3	c	three
2	b	two

Check for sorting disorder using -c

❯ seq 5 | sort -r
5
4
3
2
1
❯ seq 5 | sort -r | sort -c
sort: -:2: disorder: 4

Sort and remove duplicates using -u

❯ seq 2 | cat - <(seq 2) | sort -r
2
2
1
1
❯ seq 2 | cat - <(seq 2) | sort -r | sort -u
1
2

Sort by random hash of keys but groups identical keys together using -R

❯ seq 3 | cat - <(seq 3) | sort -R
2
2
3
3
1
1

Sort by months (Jan-Dec), using -M, and using -k2 to reference 2nd column

❯ paste number months
1	Mar
2	Dec
3	Jan
4	Aug
5	Feb
❯ paste number months | sort -k2M
3	Jan
5	Feb
1	Mar
4	Aug
2	Dec

comm

uniq

wc

find

curl

tar

tee

Reads the standard input and writes to both standard output and one or more files. Command is named after the T-splitter used in plumbing. Breaks the output of a program so it can be displayed and saved in a file.

❯ seq 5
1
2
3
4
5
❯ seq 5 | tee five.txt
1
2
3
4
5
❯ cat five.txt
1
2
3
4
5

cut

Cutting out each sectins from each line of files and writting result to standard output. Can be used to cut parts of line by byte, character and field.

❯ cat location
United States
Singapore
Malaysia
Papua New Guinea
South Korea

Using -b (bytes), show first 3 characters

❯ cut -b 1,2,3 location
Uni
Sin
Mal
Pap
Sou

First few characters with ranges

❯ cut -b 1-3,5-7 location
Unied
Sinapo
Malysi
Papa N
Souh K

First x character onwards

❯ cut -b 3- location
ited States
ngapore
laysia
pua New Guinea
uth Korea

Can also use -c (character) to achieve the same result as -b

❯ cut -c 1,2,3 location
Uni
Sin
Mal
Pap
Sou

Split by whitespace and only take 1st item

❯ cut -d " " -f 1 location
United
Singapore
Malaysia
Papua
South

Can also use it to take the 2nd item. Will return 1st item if there is no 2nd item

❯ cut -d " " -f 2 location
States
Singapore
Malaysia
New
Korea

paste

Use to join files horizontally (parallel merging) by outputting lines consisting of lines from each specified file, separated by tab delimited (by default).

❯ cat alphabet
a
b
c
d
e

❯ cat number_string
one
two
three
four
five

Add number to display row number

❯ paste number alphabet number_string
1	a	one
2	b	two
3	c	three
4	d	four
5	e	five

Use -d to define delimiter

❯ paste -d "," number alphabet number_string
1,a,one
2,b,two
3,c,three
4,d,four
5,e,five

Transpose

❯ paste -s number alphabet number_string
1	2	3	4	5
a	b	c	d	e
one	two	three	four	five

Convert into 3 columns

❯ cat alphabet | paste - - -
a	b	c
d	e

Split by whitespace and only take first item. Pipe it to paste and add number

❯ cut -d " " -f 1 state | paste - number
Arunachal	1
Assam	2
Andhra	3
Bihar	4
Chhattisgrah	5