Format Command Output In Linux

The short answer

In Unix-like operating systems such as Linux and macOS, you can scan patterns and process the output of commands using the [.inline-code]awk[.inline-code] command as follows:

$ <command> | awk '/<pattern>/ {<action>}'

Where:

• [.inline-code]<command>[.inline-code] is the command whose output you want to process or filter.
• [.inline-code]<pattern>[.inline-code] is a text string or a regex pattern used to filter the output of the specified command.
• [.inline-code]<action>[.inline-code] is the action to perform on the lines matching the specified pattern.

Alternatively, you can also pass one or more file paths as arguments:

$ awk '/<pattern>/ {<action>}' <file ...>

For example, the following command will print to the standard output all the lines of the [.inline-code]server.log[.inline-code] file containing the [.inline-code]error[.inline-code] string:

$ cat server.log | awk '/error/ {print}'

And the following command will print to the standard output all the lines of the [.inline-code]server.log[.inline-code] file starting with the [.inline-code]2024-05[.inline-code] string:

$ awk '/^2024-05/ {print}' server.log

[#split-the-output-into-columns] Splitting the output into columns [#split-the-output-into-columns]

By default, the [.inline-code]awk[.inline-code] command splits every line of its input into multiple columns, using the space character as a delimiter.

To print one or more specific columns instead of the entire line, you can use the following syntax:

$ <command> | awk '{print $<column>[,$<column>]}'

Where:

• [.inline-code]$<column>[,$<column>][.inline-code] is a list of comma-separated column numbers prefixed with a dollar sign ([.inline-code]$[.inline-code]).

For example, this command will only print the first column of the output generated by the [.inline-code]ps[.inline-code] command:

$ ps | awk '{print $1}'
PID
46686
46697
19849
5274

And this command will print the first and fourth columns of the output generated by the [.inline-code]ps[.inline-code] command:

$ ps | awk '{print $1,$4}'
PID CMD
46686 npm
46697 node
19849 awk
5274 bash

Note that you can quickly reference the last column using the pre-defined [.inline-code]$NF[.inline-code] variable as follows:

$ <command> | awk '{print $NF}'

[#use-custom-column-delimiters] Using custom delimiters [#use-custom-column-delimiters]

To split the output of commands into columns based a specific delimiter instead of the default space character, you can use the [.inline-code]-F[.inline-code] flag as follows:

$ <command> | awk -F '<delimiter>' '{<action>}'

Alternatively, you can specify multiple delimiters at once by encapsulating them in square brackets as follows:

$ <command> | awk -F '[<delimiter>...]' '{<action>}'

For example, considering the following [.inline-code]data.csv[.inline-code] file:

Name,Age,City
Sarah,32,New York
John,28,Chicago
Emily,35,Seattle

The following command will use the [.inline-code],[.inline-code] character to split each line of the [.inline-code]data.csv[.inline-code] file into separate columns and only output the third column:

$ cat data.csv | awk -F ',' '{print $3}'
City
New York
Chicago
Seattle

Summing up and counting columns

[#count-pattern-occurrences] Counting the occurrences of a pattern [#count-pattern-occurrences]

To count the occurrences of a specified pattern with [.inline-code]awk[.inline-code], you can increment a variable using the [.inline-code]++[.inline-code] operator as follows:

$ awk '/pattern/ { <variable>++ } END { print <variable> }' <file ...>

For example, considering the following [.inline-code]server.log[.inline-code] file:

ERROR | User authentication failed
INFO | Backup completed successfully
ERROR | User authentication failed
WARNING | High memory usage
INFO | Data export completed

This command will increment the [.inline-code]count[.inline-code] variable every time the [.inline-code]"ERROR"[.inline-code] string is found in the [.inline-code]server.log[.inline-code] file:

$ awk '/ERROR/ { count++ } END { print count }' server.log
2

[#sum-up-column-values] Summing up the data of columns [#sum-up-column-values]

To add up the values of a column and print their sum with [.inline-code]awk[.inline-code], you can use the following syntax:

$ <command> | awk '{<variable> += $<column>} END {print <variable>}'

Where:

• [.inline-code]<variable>[.inline-code] is the name of the variable the sum of the specified column will be accumulated in.

• [.inline-code]<column>[.inline-code] is the column number to add up.

• [.inline-code]END[.inline-code] is used to specify a command to execute once all the lines are processed.

For example, considering the following [.inline-code]data.csv[.inline-code] file:

bash
Name,Salary,Position
John,33000,Developer
Jack,36000,Manager

This command will print the sum of the values of the second column:

bash
$ cat data.csv | tail -n +2 | awk -F ',' '{sum += $2} END {print sum}'
69000

Where:

• [.inline-code]cat data.csv[.inline-code] is used to output the file's content.

• [.inline-code]tail -n +2[.inline-code] is used to remove the first line of the output.

• [.inline-code]awk -F ','[.inline-code] is used to split each line of the output into columns using [.inline-code],[.inline-code] as a separator.

• [.inline-code]{sum += $2}[.inline-code] is used to add the value of the second column to the [.inline-code]sum[.inline-code] variable.

• [.inline-code]END {print sum}[.inline-code] is used to print the [.inline-code]sum[.inline-code] variable.

Manipulating text

[#extracting-a-substring] Extracting a substring [#extracting-a-substring]

To extract a substring from a line with [.inline-code]awk[.inline-code], you can combine the [.inline-code]print[.inline-code] and [.inline-code]substr[.inline-code] functions as follows:

$ <command> | awk '{print substr($<column>, <start>, <length>)}'

Where: 

• [.inline-code]<column>[.inline-code] is the column number of the string you want to extract a substring from. Note that the full input string can be accessed through the column number [.inline-code]0[.inline-code].

• [.inline-code]<start>[.inline-code] is a number representing the starting position within the specified string.

• [.inline-code]<length>[.inline-code] is a number representing the length of the substring.

For example, this command will take as input the entire [.inline-code]"Hello World"[.inline-code] string and print [.inline-code]5[.inline-code] characters starting at position [.inline-code]7[.inline-code]:

$ echo "Hello World" | awk '{print substr($0, 7, 5)}'
World

And this command will take as input the second column and print [.inline-code]5[.inline-code] characters starting at position [.inline-code]0[.inline-code]:

$ echo "Hello World" | awk '{print substr($2, 0, 5)}'
World

[#replace-pattern-occurrences] Replacing the occurrences of a string or pattern [#replace-pattern-occurrences]

To replace all the occurrences of a string or pattern with [.inline-code]awk[.inline-code], you can use the [.inline-code]gsub[.inline-code] function as follows:

$ <command> | awk '{gsub(<pattern>, <string>, <column>); print}'

Where:

• [.inline-code]<pattern>[.inline-code] is the string or pattern you want to replace.

• [.inline-code]<string>[.inline-code] is the string you want to replace the matched [.inline-code]pattern[.inline-code] with.

For example, this command will replace all the occurrences of the letter [.inline-code]l[.inline-code] with [.inline-code]x[.inline-code]:

$ echo "Hello World" | awk '{gsub("l", "x", $0); print}'
Hexxo Worxd

[#easily-recall-syntax-with-ai] Easily retrieve this command using the Warp’s AI Command Suggestions [#easily-recall-syntax-with-ai]

If you’re using Warp as your terminal, you can easily retrieve this command using the Warp AI Command Suggestions feature:

Entering [.inline-code]awk replace all occurrences[.inline-code] into the AI Command Suggestions will prompt a [.inline-code]awk[.inline-code] command that can then be quickly inserted into your shell by doing [.inline-code]CMD+ENTER[.inline-code].

[#conditional-statements] Conditional statements [#conditional-statements]

To execute an action based on the evaluation of a condition with [.inline-code]awk[.inline-code], you can use the following syntax:

$ <command> | awk '{if (<condition>) {<action>} else if (<condition>) {<action>} else {<action>}}'

For example, let's consider the following [.inline-code]students.txt[.inline-code] file:

Alice 45
Bob 30
Charlie 25

This command will check if the value of the second column is greater or equal to [.inline-code]40[.inline-code] and print the first column with [.inline-code]“Passed”[.inline-code] if true, or [.inline-code]“Failed”[.inline-code] otherwise:

$ cat students.txt | awk '{if ($2 >= 40) { print $1, "Passed" } else { print $1, "Failed" }}'
Alice Passed
Bob Failed
Charlie Failed