Useful Linux commands. Covering grep
, tr
, awk
, sed
, sort
, comm
, uniq
, wc
, find
, curl
, tar
, tee
, cut
, paste
grep
Initial data, using -v
to invert empty string and show everything between start ^
and end $
❯ grep -v '^$' location
United States
SINGAPORE
Malaysia
Papua New Guinea
South Korea
Find lines that match start and end using ^
and $
❯ grep '^South' location
South Korea
❯ grep 'Korea$' location
South Korea
Case insensitive search, using -i
❯ grep -i 'singapore' location
SINGAPORE
Count matches, case insensitive, using -c
❯ grep -ci 'singapore' location
1
Display file names that match pattern, case insensitive, using -l
❯ grep -li 'singapore' *
location
Look for the whole word in file. To search through all files and display result, use *
. Add -n
to show line number
❯ grep -wi 'singapore' *
location:SINGAPORE
❯ grep -nwi 'singapore' *
location:2:SINGAPORE
❯ grep -wi 'singapore' location
SINGAPORE
Invert result of pattern match, using -v
❯ grep -i 's' location
United States
SINGAPORE
Malaysia
South Korea
❯ grep -vi 's' location
Papua New Guinea
Show lines after, before, or before+after a pattern match, using -A
, followed by the number of lines to display.
❯ grep -A1 '^Malay' location
Malaysia
Papua New Guinea
❯ grep -B1 '^Malay' location
SINGAPORE
Malaysia
❯ grep -C1 '^Malay' location
SINGAPORE
Malaysia
Papua New Guinea
Search recursively for a pattern in the directory
❯ grep -iR 'singapore' .
./location:SINGAPORE
tr
Used for translating or deleting characters. Supports transformations such as uppercase to lowercase, squeezing repeating characters, deleting specific characters and basic find and replace.
❯ cat location
United States
SINGAPORE
Malaysia
Papua New Guinea
South Korea
Translate lower to uppercase, vice versa
❯ cat location | tr '[:lower:]' '[:upper:]'
UNITED STATES
SINGAPORE
MALAYSIA
PAPUA NEW GUINEA
SOUTH KOREA
❯ cat location | tr '[:upper:]' '[:lower:]'
united states
singapore
malaysia
papua new guinea
south korea
❯ cat location | tr '[a-z]' '[A-Z]'
UNITED STATES
SINGAPORE
MALAYSIA
PAPUA NEW GUINEA
SOUTH KOREA
Translate whitespaces into tables. Use [:space:]
if you want to match any whitespace character, including spaces, tabs, newlines, etc.
❯ cat location | tr ' ' '\t'
United States
SINGAPORE
Malaysia
Papua New Guinea
South Korea
❯ cat location | tr '[:space:]' '\t'
United States SINGAPORE Malaysia Papua New Guinea South Korea %
Translate characters into other character
❯ cat location | tr 'SE' '*'
United *tates
*INGAPOR*
Malaysia
Papua New Guinea
*outh Korea
Remove repetitive characters using -s
❯ echo "Sentence with too many spaces" | tr -s " "
Sentence with too many spaces
Delete specified characters using -d
❯ echo "Sentence with too many spaces" | tr -s " " | tr -d e
Sntnc with too many spacs
Delete digits
❯ echo "My number is 1234 5588" | tr -d '[:digit:]'
My number is
❯ echo "My number is 1234 5588" | tr -d '0-9'
My number is
Use completement with -c
to inverse the deleted characters
❯ echo "My number is 1234 5588" | tr -cd '[:digit:]'
12345588%
awk
sed
sort
Sort a file, arrange record in particular order. Sorts file assuming contents are ASCII.
-r : Sorts data in reverse order, descending
-k : Sorts table based on specific column number
-c : Check if file is already sorted and reports disorder
-u : Sorts and removes duplicate lines, providing unique sorted list
-R : Sorts by random hash of keys but groups identical keys together
-n : Sorts numeric fields by arithmetic value. A numeric field may contain leading blanks, an optional minus sign, decimal digits, thousands-separator characters, and an optional radix character. Numeric sorting of a field containing any nonnumeric character gives unpredictable results.
-d : Sorts using dictionary order. Only letters, digits, and spaces are considered in comparisons.
-M : Sorts by months (Jan-Dec)
Sort alphabetically, by default
❯ sort number_string
five
four
one
three
two
And in reverse order, using -r
❯ sort -r number_string
two
three
one
four
five
Sort by the 3rd column using -k
❯ paste number alphabet number_string
1 a one
2 b two
3 c three
4 d four
5 e five
❯ paste number alphabet number_string | sort -k3
5 e five
4 d four
1 a one
3 c three
2 b two
Check for sorting disorder using -c
❯ seq 5 | sort -r
5
4
3
2
1
❯ seq 5 | sort -r | sort -c
sort: -:2: disorder: 4
Sort and remove duplicates using -u
❯ seq 2 | cat - <(seq 2) | sort -r
2
2
1
1
❯ seq 2 | cat - <(seq 2) | sort -r | sort -u
1
2
Sort by random hash of keys but groups identical keys together using -R
❯ seq 3 | cat - <(seq 3) | sort -R
2
2
3
3
1
1
Sort by months (Jan-Dec), using -M
, and using -k2
to reference 2nd column
❯ paste number months
1 Mar
2 Dec
3 Jan
4 Aug
5 Feb
❯ paste number months | sort -k2M
3 Jan
5 Feb
1 Mar
4 Aug
2 Dec
comm
uniq
wc
find
curl
tar
tee
Reads the standard input and writes to both standard output and one or more files. Command is named after the T-splitter used in plumbing. Breaks the output of a program so it can be displayed and saved in a file.
❯ seq 5
1
2
3
4
5
❯ seq 5 | tee five.txt
1
2
3
4
5
❯ cat five.txt
1
2
3
4
5
cut
Cutting out each sectins from each line of files and writting result to standard output. Can be used to cut parts of line by byte, character and field.
❯ cat location
United States
Singapore
Malaysia
Papua New Guinea
South Korea
Using -b (bytes)
, show first 3 characters
❯ cut -b 1,2,3 location
Uni
Sin
Mal
Pap
Sou
First few characters with ranges
❯ cut -b 1-3,5-7 location
Unied
Sinapo
Malysi
Papa N
Souh K
First x character onwards
❯ cut -b 3- location
ited States
ngapore
laysia
pua New Guinea
uth Korea
Can also use -c (character)
to achieve the same result as -b
❯ cut -c 1,2,3 location
Uni
Sin
Mal
Pap
Sou
Split by whitespace and only take 1st item
❯ cut -d " " -f 1 location
United
Singapore
Malaysia
Papua
South
Can also use it to take the 2nd item. Will return 1st item if there is no 2nd item
❯ cut -d " " -f 2 location
States
Singapore
Malaysia
New
Korea
paste
Use to join files horizontally (parallel merging) by outputting lines consisting of lines from each specified file, separated by tab delimited (by default).
❯ cat alphabet
a
b
c
d
e
❯ cat number_string
one
two
three
four
five
Add number
to display row number
❯ paste number alphabet number_string
1 a one
2 b two
3 c three
4 d four
5 e five
Use -d
to define delimiter
❯ paste -d "," number alphabet number_string
1,a,one
2,b,two
3,c,three
4,d,four
5,e,five
Transpose
❯ paste -s number alphabet number_string
1 2 3 4 5
a b c d e
one two three four five
Convert into 3 columns
❯ cat alphabet | paste - - -
a b c
d e
Split by whitespace and only take first item. Pipe it to paste
and add number
❯ cut -d " " -f 1 state | paste - number
Arunachal 1
Assam 2
Andhra 3
Bihar 4
Chhattisgrah 5