Shell01: pipelines, text and the environment

What Shell01 asks of you

Most Shell01 exercises are a short script whose output is compared with the expected output byte for byte. That means two skills: knowing which tool does which job, and checking your output precisely — including spaces, separators and the final newline. The exercises also expect your scripts to run under /bin/sh, not only in your interactive shell.

Scripts that /bin/sh can run

A shell script is a text file of commands. To run it by name it needs execute permission, and its first line — the "shebang" — names the interpreter:

#!/bin/sh
echo "Build started"

/bin/sh is a POSIX shell. On some systems it is dash, on others a bash in compatibility mode, and features specific to bash (arrays, [[ ]], some echo options) may not work in it. Test with sh yourscript.sh to see what a strict shell makes of it.

Every command also returns an exit status: 0 for success, anything else for failure. echo $? shows the status of the last command, and pipelines and && / || use it to decide what runs next.

Streams, pipes and redirection

Every process starts with three streams: standard input (0), standard output (1) and standard error (2). The shell can connect them to files or to each other:

A pipeline is the heart of Shell01: each tool does one small transformation and passes text along. Because tools read standard input when no file is given, the same filter works on a file or at the end of a pipe.

Environment variables

Every process has an environment: a list of NAME=value pairs inherited from its parent. env prints it, and a script reads a variable with $NAME. A variable only reaches child processes once it is exported:

$ CITY=Lisbon
$ sh -c 'echo "City: $CITY"'      # prints "City: " — not exported
$ export CITY
$ sh -c 'echo "City: $CITY"'      # prints "City: Lisbon"

This is how many Shell01 scripts receive their input: the evaluator sets a variable, then runs your script, which must read it rather than hard-code a value. Always write "$NAME" in double quotes, so a value containing spaces stays one argument.

Asking the system about users and machines

Unix keeps information about users, groups and hardware that commands can report. id prints a user's numeric id and the groups they belong to; groups prints group names; man id lists options that select just one part of that output. Network interfaces, with their addresses — including hardware (MAC) addresses — are listed by ifconfig on macOS and BSD and by ip link on most Linux systems.

The catch is that the output format of these commands differs from one system to another. Before extracting a value from them, look at the raw output on the system you are graded on, and build your filter against that — not against an example from the internet.

Walking a tree with find

Shell01 uses find to produce lists rather than to act on files. A few behaviours to keep in mind (the Shell00 guide covers the expression syntax):

• Results come in filesystem order, not alphabetical order. If order matters, sort explicitly.
• Every path starts with the directory you gave — for find ., that is ./ — and the starting directory itself is one of the results.
• -type f and -type d separate files from directories; -maxdepth limits how deep the walk goes.

Counting is usually done by piping a list into wc -l, which counts lines. Whether a count should include the starting directory, or hidden files, is exactly the kind of detail to check against the subject.

Selecting and reshaping text

Most of Shell01 is picking out the right lines and reshaping them. The tools below cover nearly everything; each has a manual page worth reading in full. The examples use a small made-up file, fruit.csv:

apple,red,12
banana,yellow,7
cherry,red,30

These combine. For example, the names of the red fruits in capitals:

$ grep ',red,' fruit.csv | cut -d, -f1 | tr 'a-z' 'A-Z'
APPLE
CHERRY

Two things are easy to miss. sed selects lines by address as well as by pattern — a line number, a range, or a step — which is covered in its manual page. And head and tail together can cut any range of lines out of the middle of a stream.

Sorting and reversing

sort orders lines alphabetically by default; -n sorts numerically, -r reverses the order, and -t / -k sort by a chosen field. uniq collapses adjacent duplicates, so it is almost always used after sort. rev is something else entirely: it reverses the characters within each line.

Alphabetical order depends on the locale. The same input can sort differently depending on how upper case, lower case and punctuation are treated. If your output does not match, compare the ordering on the grading system, and read about LC_ALL in man sort.

Quoting

Before running a command the shell expands it: variables are replaced, * and ? become matching file names, and unquoted spaces split arguments. Quoting controls that:

• Single quotes '…' — nothing inside is expanded. What you type is what the command receives.
• Double quotes "…" — variables and $( ) are expanded, but the result stays one argument and wildcards are not expanded.
• Backslash \ — protects the single character after it.

$ touch 'a file with spaces'
$ ls a file with spaces        # four arguments: fails
$ ls 'a file with spaces'      # one argument: works

Names containing quotes, dollar signs or asterisks are a good test of whether you really understand the rules — build one in a scratch directory and check it with ls.

Numbers in other bases

A number written in base b uses b distinct symbols, and each position is worth a power of b. The symbols do not have to be 0123456789: any ordered set of distinct characters works, and a symbol's value is its position in that set.

Take the made-up alphabet xyz, so x = 0, y = 1, z = 2, in base 3. The number zxy is:

z x y  =  2×3² + 0×3¹ + 1×3⁰  =  18 + 0 + 1  =  19

Going the other way — from 19 back into that alphabet — divide repeatedly by the base and read the remainders from last to first. Being able to do both conversions by hand is what makes any tool that does them for you understandable. The same idea returns in the C modules, where you convert numbers between bases in code. 42CodeLens's free Number Base Conversions Lab lets you watch it happen.

Debugging a pipeline

• Build it one stage at a time. Run the first command, look at its output, then add the next pipe. Never write five stages and hope.
• Make invisible characters visible. cat -e shows line ends and od -c shows every byte, so a stray space or missing newline cannot hide.
• Compare, don't eyeball. Save the expected output to a file and diff it with yours.
• Trace the script. sh -x script.sh prints each command after expansion, which shows exactly what the shell made of your quoting.
• Try awkward input. An empty file, a name with spaces, a variable that is not set.

Common mistakes

• A bash-only feature in a script that must run under /bin/sh.
• An unquoted variable that splits into several arguments.
• An extra or missing newline, or a trailing separator, at the end of the output.
• Assuming find output is sorted.
• Building a filter against another system's output format.
• Hard-coding a value the script was supposed to read from the environment.

Where to go next