Basic information

Teacher: Petr Kučera <>, room S304
Schedule: Tuesday 10:40 — 12:10 SU2

Useful links

Lecture web pages
Web pages of the UNIX lab
Local copy of the last UNIX specification
The UNIX Specification
Exercise assignments in book “Shell v příkladech” by Libor Forst, translated to English.
Shell script checker
Literature:
- Classic Shell Scripting by Arnold Robbins, Nelson H. F. Beebe
- UNIX Shells by Example by Ellie Quigley
- Shell Programming in Unix, Linux and OS X: The Fourth Edition of Unix Shell Programming (4th Edition) (Developer's Library) by Stephen G. Kochan, Patrick Wood
- Shell Scripting Recipes: A Problem-Solution Approach by Chris Johnson

Credit requirements

At the end of each practical homework shall be assigned, I will expect to receive solutions to them by e-mail before the beginning of the next practical. If by the end of the semester you have at least 2/3 of possible number of points, you get a credit. Otherwise you will need to finish additional homework to get enough points. In case you submit solutions to less than 1/3 of homework assignments, you will not have a possibility to get a credit. Each set of homework exercises assigned after each practical will be worth 3 points, at the beginning of the semester that could mean three exercises while at the end it can be just a single but more difficult exercise.

If not stated otherwise, you are only allowed to use the commands, tools and structures we have used on practicals before the homework was assigned (including the one to which homework was assigned). In some cases the tools might be even more restricted. If I come across a solution which is just a copy of another one I will assign no points for this solution. If this case repeats there will be no possibility to obtain a credit from me.

Please, send me your solutions via an e-mail and do so before the beginning of the next practical. Either write your solution as a part of e-mail body, or append them in a form of plain text files, do not use pdf, word or similar file types. The homework assignments are appended below a list of exercises we have done at the corresponding practical (i.e. the current homework exercises are always at the bottom of this web page).

There have been 12 homework assignments and thus the number of points required to get the credit is 24. Additional homework exercises for those who do note have enough points will be added to the last exercise.

Some peculiarities of the Malá Strana UNIX lab

I think it is worth to know and bear in mind some specifics of the Malá Strana UNIX lab. To Czech students I recommend to check the web pages of the UNIX lab. I will warn you of the lab properties as we come across them. In this section I would to mention at least the most apparent of them:

Users home directories are placed in a remote AFS filesystem, where standard UNIX permissions do not apply and the standard UNIX commands for file permissions manipulations (mainly chmod) cannot be used or they can be used only in a very restricted way (there is a web page about the ACL which is the system in use, but it is only in Czech to the best of my knowledge).
This means that when you try the chmod command or anything which has some connection to the standard UNIX permission system, you should do so in local directories (such as /tmp in which you are allowed more or less anything). There the standard UNIX permissions apply.
As user accounts are stored remotely, local file /etc/passwd does not contain information about your account or accounts of other lab users. If you want to get your and other users account info in the format of /etc/passwd you can achieve that using a getent command as follows:
getent passwd
for other parameters and general help on this command use getent --help.
The same applies to /etc/group and getent group.
The find command (still most probably) is not in the best of terms with AFS filesystem — when used on network AFS drives in the lab such as your home dir you can get unexpected results. You can use find to search on local directories such as /tmp, /etc, /usr etc. The exercises usually suggest it.

Content of the practicals

1st practical (19th February, 2019)

Basic commands, shell command line edit.

More detailed:

Command line edit: Ctrl-a, Ctrl-e, Ctrl-h, Ctrl-r, arrows (left, right, up and down), Ctrl-d, Ctrl-k (deletes text from the cursor position to the end of line).
Commands: man, ls, cd, pwd, cat, more, less, head, tail, wc, mkdir, rmdir, cp, mv, rm, echo.
Redirecting and pipes: <, >, >>, 2>, 2>>, |, /dev/null.
Shell expansion: *, ?, [list], [!list], [[:class:]], apostrophes and quotation marks, filenames starting with a dot.

Exercises

In your home directory create a directory named DIR
Copy all files whose filenames satisfy the following conditions to ~/DIR. The files are in /usr/include directory, their names start with m, end with .h and contain a number.
Create a subdirectory called SUBDIR in your DIR directory.
The first five lines of each file you have copied from /usr/include copy to file ~/DIR/SUBDIR/firstfive.
The last lines of files in ~/DIR copy to file ~/DIR/SUBDIR/last.
Concatenate the two files in ~/DIR/SUBDIR into one file ~/DIR/SUBDIR/firstandlast
Delete the files in ~/DIR/SUBDIR except firstandlast.
Store the number of files and directories in ~/DIR into a file ~/DIR/SUBDIR/count
Output the long information on ~/DIR/SUBDIR directory. (Not its content, but information on it).
Delete the contents of ~/DIR/SUBDIR/firstandlast file without removing the file itself.
Add a line containing just a star sign (i.e. *) to file ~/DIR/SUBDIR/firstandlast.
Delete ~/DIR together with all the files it contains.
(Show solutions to preceding exercises)

Deprecated: Function create_function() is deprecated in /home/kucerap/public_html/include/geshi.php on line 4698
#!/bin/sh # In your home directory create a directory named DIR mkdir ~/DIR # Copy all files whose filenames satisfy the following conditions to # ~/DIR. The files are in /usr/include directory, # their names start with 'm', end with '.h' and contain a # number. cp /usr/include/m*[0-9]*.h ~/DIR # Create a subdirectory called SUBDIR in your DIR directory. mkdir ~/DIR/SUBDIR # The first five lines of each file you have copied from # /usr/include copy to file ~/DIR/SUBDIR/firstfive. head -n 5 ~/DIR/m*[0-9]*.h >~/DIR/SUBDIR/firstfive # The last lines of files in ~/DIR copy to file # ~/DIR/SUBDIR/last. tail -n 1 ~/DIR/m*[0-9]*.h >~/DIR/SUBDIR/last # Concatenate the two files in ~/DIR/SUBDIR into one file # ~/DIR/SUBDIR/firstandlast cat ~/DIR/SUBDIR/* >~/DIR/SUBDIR/firstandlast # Delete the files in ~/DIR/SUBDIR except firstandlast. rm ~/DIR/SUBDIR/firstfive rm ~/DIR/SUBDIR/last # Store the number of files and directories in ~/DIR into a # file ~/DIR/SUBDIR/count ls ~/DIR | wc -l >~/DIR/SUBDIR/count # Output the long information about ~/DIR/SUBDIR directory. # (Not its content, but information about it). ls -ld ~/DIR/SUBDIR # Delete the contents of ~/DIR/SUBDIR/firstandlast file # without removing the file itself. cp /dev/null ~/DIR/SUBDIR/firstandlast # Add a line containing just a star sign (i.e. *) to file # ~/DIR/SUBDIR/firstandlast. echo '*' >>~/DIR/SUBDIR/firstandlast # Delete ~/DIR together with all files the it contains. rm -r ~/DIR
Output lines number 11-20 from file /etc/passwd.
(Show solution)
#!/bin/sh head -n20 /etc/passwd | tail -n10

Homework

Write a command which lists all files in your home directory whose name starts with a dot (“.”) and contain at least two other characters. [1 point]
Write a command which prints the last line of each file in directory /usr/include whose names end with suffix “.h”. [1 point]
Create a directory INCL in your home directory. Write a command which copies files whose names do not start with a digit but contain a digit in the name from directory /usr/include to the above created directory INCL in your home directory. [1 point]

2nd practical (26th February, 2019)

Basic commands and shell usage.

Commands: ssh, scp, date, tee, touch.

Exercises

Finish the exercises from the last practical
Delete all files and directories (with subtrees) from the current working directory. Remove all files and directories including those whose names start with a dot.

Show solution

#!/bin/sh



rm -rf .* *

Create a file with name “-f” which contains a line with current date and time. Then delete this file.
(Show solution)
#!/bin/sh # Create a file -f with date and time: date >-f # Delete file -f: rm -- -f
Create a file named “fixed date” (i.e. “fixed<space>date”) with last date and time of modification set to 13:30, 1st February, 2009.
(Show solution)
#!/bin/sh touch -t 200902011330 "fixed date" # or touch -d "2009-02-01 13:30:00" "fixed date"
Copy the file /etc/passwd into your home directory. The copy should have name accounts and the last modification date and time should be preserved, i.e. should be the same as the last modification date and time of /etc/passwd.
(Show solution)
#!/bin/sh cp -p /etc/passwd ucty
Set the time and date of last modification of file accounts to the same time as the last modification time of the file /etc/group.
(Show solution)
#!/bin/sh touch -r /etc/group accounts
Write a command which outputs the list of files in directory /usr/bin sorted lexicographically in decreasing order. The output is written to files bina, binb, and to the screen. The screen output is viewed in pages (use less, or more).
(Show solution)
#!/bin/sh ls -r /usr/bin | tee bina binb | more
What happens during the execution of the following three commands? What is the difference betweeen them?
1. mv file /dev/null
2. cp file /dev/null
3. cat file >/dev/null
Write commands that output the ten biggest and the ten smallest files in directory /etc.
(Show solution)
#!/bin/sh ls -S /etc | head ls -rS /etc | head
Find a file in /usr/bin with the most recent modification date and time.
(Show solution)
#!/bin/sh ls -t /usr/bin | head -n 1

Homework

Write a command which determines the number of groups in the system (the number of lines in file /etc/group) [1 point]
Write a command which sets the last modification date and time of a file to 11th March, 1993, 15:04. [1 point]
Write command which prints the name of the second biggest file in directory /usr/bin. [1 point]

3rd practical (5th March, 2019)

Basic text processing utilities.

Commands: cut, paste, tr.

Exercises

Download file calories.csv, it is a file in a CSV format with semicolon (“;”) used as a field separator.
Replace quotation marks (") with apostrophes (') in file calories.csv
(Show solution)
#!/bin/sh tr \" \' <calories.csv >/tmp/calories.a.csv mv /tmp/calories.a.csv calories.csv
Remove apostrophes (') from file calories.csv. In this and the several following exercises we assume that the file is has been modified by preceding exercises, i.e. here after the previous exercise, quotation marks have been replaced with apostrophes.
(Show solution)
#!/bin/sh tr -d \' <calories.csv >/tmp/calories.a.csv mv /tmp/calories.a.csv calories.csv
Select the first column from file calories.csv, i.e. the column with food names.
(Show solution)
#!/bin/sh cut -f1 -d\; calories.csv
In file calories.csv replace upper case characters with their lower case counterparts, but only in the first column. (Cut the first columns using cut, replace capital letters with small ones using tr, then put them back using paste.)
(Show solution)
#!/bin/sh cut -f1 -d\; calories.csv >/tmp/first cut -f2- -d\; calories.csv >/tmp/others tr '[:upper:]' '[:lower:]' </tmp/first | paste -d\; - /tmp/others >calories.csv rm /tmp/first /tmp/others
Reorder the last three columns of file calories.csv in the opposite order, i.e. after reordering the first column contains amount of proteins, the second contains amount of carbohydrates and the third column contains the amount of fat. Use cut and paste.
(Show solution)
#!/bin/sh cut -f5 -d\; calories.csv >/tmp/fifth cut -f6 -d\; calories.csv >/tmp/sixth cut -f7 -d\; calories.csv >/tmp/seventh cut -f1-4 -d\; calories.csv | paste -d\; - /tmp/seventh /tmp/sixth /tmp/fifth >/tmp/novy mv /tmp/novy calories.csv rm /tmp/fifth /tmp/sixth /tmp/seventh
Select the 9 characters with permissions from the output of ls -l command.
(Show solution)
#!/bin/sh ls -l | tail -n +2 | cut -c2-10
Suppose you have three files, summand1, summand2 and sum, each of these files contains the same number of lines, each line contains just a number. Compose these files to just one output where each line has the following format:
summand1+summand2=sum.
(Show solution)
#!/bin/sh paste -d+= summand1 summand2 sum

Homework

In file calories.csv replace each food name with a “-” (minus) character. Other columns should remain unchanged. [1 point]
For each user in /etc/passwd output a pair uid:login (login means user login name). [1 point].
In the last column of /etc/group remove all characters except comma ',' [1 point].

4th practical (13th March, 2019)

Simple utilities.

Commands: sort, diff, comm, split.

Exercises

Write the names of files which occur in /usr/bin but which are not in /bin.
(Show solution)
#!/bin/sh ls /usr/bin | sort >/tmp/usr_bin ls /bin | sort | comm -13 - /tmp/usr_bin rm /tmp/usr_bin
Write the numbers of groups which are used in /etc/group but which are not used in /etc/passwd. That is find the numbers of groups which are not used as a primary group for any user.
(Show solution)
#!/bin/sh cut -d: -f4 </etc/passwd | sort > /tmp/uid cut -d: -f3 </etc/group | sort | comm -23 - /tmp/uid rm /tmp/uid
In file calories.csv replace commas (,) with hyphens (-), using diff then determine which lines have been changed.
(Show solution)
#!/bin/sh tr "," "-" <calories.csv | diff calories.csv -
Sort file calories.csv according to amount of calories in increasing order. The first header line has to remain at the beginning of the file.
(Show solution)
#!/bin/sh # Save the header head -n 1 calories.csv >/tmp/header # Sort the rest and append it to the header tail -n +2 calories.csv | sort -t\; -k4,4n >>/tmp/header mv /tmp/header calories.csv
Sort file calories.csv according to triple [protein, carbohydrates, fat] in an increasing order. The first header line has to remain at the beginning of the file.
(Show solution)
#!/bin/sh # Save the header head -n 1 calories.csv >/tmp/header # Sort the rest and append it to the header tail -n +2 calories.csv | sort -t\; -k7,7n -k6,6n -k5,5n >>/tmp/header mv /tmp/header calories.csv
Determine the number of different units used in the second column of file calories.csv. First consider two units with different number as different, then as the same, such as in case of “1 Cup” and “2 cup”.
(Show solution)
#!/bin/sh # In the first solution we consider two units with a different amount # as different, e.g. “1 Cup” and “2 Cup” are different units. tail -n +2 calories.csv | sort -t\; -k2,2 -u | wc -l # If we wish to consider two units with different amounts as the same # unit (i.e. we do not want to differentiate between “1 Cup” and “2 # Cup”, we can proceed in the following way: tail -n +2 calories.csv | cut -f2 -d\; | cut -f2- -d" " | sort -u | wc -l
It is known that a welfare crook lives in Los Angeles. You are given lists for 1) people receiving welfare, 2) Hollywood actors, and 3) residents of Beverly Hills. The lists are given as text files where each line contains one name of a person. A welfare crook is someone who appears in all three lists. Write a script to find at least one crook. Example files: social.txt, actor.txt, beverly_hills.txt.
(Show solution)
#!/bin/sh sort social.txt >/tmp/s_social.txt sort beverly_hills.txt >/tmp/s_beverly_hills.txt sort actor.txt | comm -12 /tmp/s_social.txt - | comm -12 - /tmp/s_beverly_hills.txt rm /tmp/s_social.txt /tmp/s_beverly_hills.txt.
Split a file (e.g. /etc/passwd) to parts consisting of five lines, then concatenate these parts back to one file.
(Show solution)
#!/bin/sh # We shall assume that the number of lines in /etc/passwd is not too # big, in particular that when using split the two letter suffix is # enough. split -l5 /etc/passwd passwdpart # The concatenated file will be stored in the current directory, we # naturally do not have the write permission to /etc directory. cat passwdpart* >passwd rm passwdpart*
Write the login names from /etc/passwd to ten lines, the login names on each line are separated with spaces.
(Show solution)
#!/bin/sh # We assume that /etc/passwd does not contain too much records and # that the default two letter suffix is enough in split. cut -d":" -f1 /etc/passwd | split -l10 - logins paste -d" " logins* rm logins*
Create a copy of /etc/group file in which each login name listed in the field with group members is replaced with a single character '@' (you may assume that a login name consists of alphanumeric symbols and '_'). The other columns remain unchanged.
(Show solution)
#/bin/sh cut -d: -f4 /etc/group | tr -sc ',\n' '[@*]' >/tmp/memb.$$ cut -d: -f-3 /etc/group | paste -d: - /tmp/memb.$$ >group rm /tmp/memb.$$
Determine the name of a group with the highest number of members assigned in the file /etc/group. This is a follow up to the previous exercise. In particular you may assume that you already have the file produced in the previous exercise.
(Show solution)
#/bin/sh sort -t: -k4,4r group | head -n 1 | cut -d: -f1

Homework

Determine the number of different countries which appear in file ip-by-country.csv. [1 point]
Determine the login name of the user with the highest uid in /etc/passwd. [1 point]
Based on files countrycodes_en.csv and kodyzemi_cz.csv list the names of states which are the same in English and in Czech. [1 point]

5th practical (19th March, 2019)

Permissions, links.

Commands: who, id, mail (mailx), chmod, chgrp, chown, file, ln, df, du, umask, find.

Attention, in Malá Strana UNIX lab try permissions in /tmp directory. Other directories are on afs file system which has a different system of permission handling.

Exercises

Write a command which outputs your current UID.
(Show solution)
#!/bin/sh id -u
Write a command that outputs the list of names of groups to which you belong as a user.
(Show solution)
#!/bin/sh id -Gn
Send the contents of file /etc/group by an e-mail using command mail (or mailx).
(Show solution)
#!/bin/sh mail -s "Obsah /etc/group" adresa </etc/group
In your home directory create a link to /etc/passwd. First try to create a hard link, then a soft link. Why a hard link cannot be created in lab? Call ls -l to find out how a soft link is displayed in its output.
(Show solution)
#!/bin/sh # Hard link ln /etc/passwd ~ # Soft link ln -s /etc/passwd ~
In directory /tmp create a hard link to another file in /tmp (you can copy some file to /tmp for this purpose, the hard link will naturally have a different name). Run ls -l on both files and observe how a number in the second column changed. Using ls -i check that both copies (the original file and its hard link) have the same inode number.
Create a simple script and assign it an execute permission.
(Show solution)
#!/bin/sh cat >script.sh <<EOF #!/bin/sh echo Hello world! EOF chmod a+x script.sh
Create a file which the owner cannot neither execute, read, or write to it, the members of file group can read and write to the file, and the others can just execute the and not read nor write to it.
(Show solution)
#!/bin/sh cat >script.sh <<EOF #!/bin/sh echo Hello world! EOF chmod 0761 /tmp/script.sh # Equivalently: chmod u+rwx,g=u-x,o=u-rw /tmp/script.sh # or chmod u=rwx,g=rw,o=x /tmp/script.sh
Create a script which cannot be read but has execute permissions. Try tu run this script. Why it is not possible to execute an unreadable shell script?
(Show solution)
#!/bin/sh cat >/tmp/script.sh <<EOF #!/bin/sh echo Hello world EOF chmod a+x-r /tmp/script.sh /tmp/script.sh # If a shell should run the script, it must be allowed to read it, too.
Create a directory to which anyone can go (by cd, can be used in path) and add files but only the directory owner can list its contents.
(Show solution)
#!/bin/sh mkdir /tmp/directory chmod 0755 /tmp/directory # Or equivalently chmod u+rwx,go=u-r /tmp/directory
Create a directory to which you can go, you cannot read its contents but you can add files to it. Create a file in this directory which can be only read (and not written to or executed). Delete this file using rm. Why you can delete a file which you cannot modify?
(Show solution)
#!/bin/sh mkdir /tmp/directory chmod 0300 /tmp/directory # Or equivalently chmod u+wx-r,go-rwx /tmp/directory touch /tmp/directory/file chmod 0400 /tmp/directory/file # Or equivalently chmod u+r-wx,go-rwx /tmp/directory/file rm -f /tmp/directory/file
Write a command which lists all files and directories in the subtree of current directory, whose names contain substring 'bin'.
(Show solution)
#!/bin/sh find . -name "*bin*" "(" -type d -o -type f ")"
Write a command which lists the number of directories in the subtree of /etc directory.
(Show solution)
#!/bin/sh find /etc -type d | wc -l
List all symbolic (or soft) links in a directory /usr. List only links directly present in the directory, not deeper in its subtree.
(Show solution)
#!/bin/sh # The following can be quite slow find /usr -type l '!' -path "/usr/*/*" # or faster find /usr/* -prune -type l # or even better find /usr $ -path "/usr/*/*" -prune $ -o $ -type l -print $
Find all files in subtree of directory /tmp, which have size bigger than 100kB and which are readable by anyone.
(Show solution)
#!/bin/sh find /tmp -type f -size +200 -perm -a+r
In the subtree of /usr/include find the files with which are in depth at least 2 and at most 3 in the subtree.
(Show solution)
#!/bin/sh find /usr/include -path "/usr/include/*/*" ! -path "/usr/include/*/*/*/*" # A more effective solution with prune: find /usr/include -path "/usr/*/*/*" -prune -o -path "/usr/*/*"
In the subtree of directory /etc find all files which are newer than /etc/passwd.
(Show solution)
#!/bin/sh find /etc -newer /etc/passwd
In the subtree of directory /bin find all files which are owned by root and which can be executed by the owner and group members, but not by others.
(Show solution)
#!/bin/sh find /bin -user root -perm -ug+x ! -perm -o+x
Find all files in directory /etc which belong to group stunnel and write long information using ls -l on each of these files.
(Show solution)
#!/bin/sh find /etc -group stunnel -exec ls -l {} +

Homework

Write a command, which adds a group read permission to all files in a subtree of the directory /tmp/dir. Use only one call to chmod [1 point]
Create a file and set its permissions so that the owner can execute this file, read and write to it, group members can execute and read it, and the others can only read the file. In chmod command use a symbolic way of specifying the permissions and also the octal numeric way [1 point].
Find all files in the subtree of /etc which are not newer than /etc/group. [1 point]

6th practical (26th March, 2019)

Commands: xargs, join.

Exercises

Try the difference between echo $PATH; echo "$PATH"; echo '$PATH';
Try the difference between echo *; echo "*"; echo '*';
List the paths contained in variable $PATH so that each of them will be written on a separate line and there will be no separators “:”.
(Show solution)
#!/bin/sh echo "$PATH" | tr ":" "\n"
Output the long information on all directories contained in variable $PATH (not their contents, just the long info).
(Show solution)
#!/bin/sh echo "$PATH" | tr ":" "\n" | xargs -I{} ls -ld {}
Create a file with the following three lines:
```
a b c
d e f
g h i
```
and try to run the following commands on this file (here named “file”):
```
cat file | xargs echo
cat file | xargs echo -
cat file | xargs -n 2 echo -
cat file | xargs -I{} echo "-{}-" "-{}-"
cat file | xargs -I{} -n 2 echo "-{}-" "-{}-"
```
Observe what happens when different options are passed to xargs.
Split the file /etc/passwd to parts consisting of five lines each and then reorder these parts in the opposite order (that is assuming /etc/passwd has n lines where n is divisible by five, the lines are now ordered as: n-4, n-3, n-2, n-1, n, n-9, n-8, n-7, n-6, n-5, ..., 1, 2, 3, 4, 5).
(Show solution)
#!/bin/sh split -l5 /etc/passwd passwdparts ls -r passwdparts* | xargs cat >passwd rm passwdparts*
Find out which directories stored in variable $PATH contain a program named awk.
(Show solution)
#!/bin/sh echo "$PATH" | tr ':' '\n' | xargs -I{} find {}/awk -prune 2>/dev/null
Write commands which print a list of pairs login:group, where login is the login of a user and group is the name of primary group of user with given login. To produce this list, use join on files /etc/passwd and /etc/group (which are preprocessed as necessary).
(Show solution)
#!/bin/sh sort -t: -k3,3 /etc/group >skupiny sort -t: -k4,4 /etc/passwd | join -1 3 -2 4 -t: -o 2.1,1.1 skupiny - rm skupiny
Download files countrycodes_en.csv and kodyzemi_cz.csv. Both of these files are in a CSV format with semicolon (;) used as a separator.
Use files countrycodes_en.csv and kodyzemi_cz.csv to create a list of countries with two columns separated with an equality character (=) where each line has the form of
Czech name=English name.
(Show solution)
#!/bin/sh tr -d '"' <kodyzemi_cz.csv | tail -n +2 | sort -t \; -k1,1 >/tmp/kody_s tr -d '"' <countrycodes_en.csv | tail -n +2 | sort -t \; -k4,4 >/tmp/codes_s join -t\; -1 1 -2 4 -o 1.4,2.1 /tmp/kody_s /tmp/codes_s | tr \; = >translation rm /tmp/kody_s /tmp/codes_s

Homework

Write commands which remove all files in the subtree of current working directory which have size 0B. [1 point]
Output the list of lines which contains the first line of each file in in the subtree of /usr/include, whose names start with “std” but do not end with “.h” (simply call head on each file separately). [1 point]
For each user from /etc/passwd write a line
<login name>:<groups in which the user appears>
(Hint: You can run id with suitable parameters for each user and paste (using paste) it with the list of the users.) [1 point]

7th practical (2nd April, 2019)

Commands: grep, sed

Exercises

From file calories.csv select lines in which the food name contains a character which is not alphanumeric (i.e. does not belong to [:alnum:]).
(Show solution)
#!/bin/sh grep '^"[^;]*[^[:alnum:];][^;]*";' calories.csv # Případně grep -v '^"[[:alnum:]]*";' calories.csv
From file calories.csv select lines in which amount is measured in units “Piece” or “Cake”.
(Show solution)
#!/bin/sh grep -e '^[^;]*;[[:digit:].]* Piece;' \ -e '^[^;]*;[[:digit:].]* Cake;' calories.csv # Respectively grep -E '^[^;]*;[[:digit:].]* (Piece|Cake);' calories.csv
Find files in the subtree of /usr/include whose names have length 5-7 characters not including suffix which is required to by “.h”. (Use grep on the output of find command.)
(Show solution)
#!/bin/sh find /usr/include -name "*.h" | grep '/[^/]\{5,7\}\.h'
Select lines from the output of ls -l in which owner has the same permissions as group members and as others. (Create a file satisfying these condition in say /tmp if necessary.)
(Show solution)
#!/bin/sh ls -l | grep '^.$...$\1\1'
Find files in the subtree of /usr/include which contain a string “printf” in them.
(Show solution)
#!/bin/sh find /usr/include -exec grep -q printf {} \; -print # or find /usr/include | xargs grep -l printf # or find /usr/include -exec grep -l printf {} +
Using sed select only the column with the file size from the output of ls -l.
(Show solution in sed)
#!/bin/sh ls -l | sed '1d; s/^$[^ ]* *$\{5\} .*$/\1/'

(Show solution in ed)
#!/bin/sh # Backslash in front of END prevents shell from performing any # expansions in the text before END. ls -l > /tmp/lsl.$$ ed /tmp/lsl.$$<<\END 2,$g/^/s/^$[^ ]* *$\{5\} .*$/\1/ %p Q END rm /tmp/lsl.$$
Using sed transform the lines of /etc/passwd to the form of uid (login).
(Show solution in sed)
#!/bin/sh sed 's/^$[^:]*$:[^:]*:$[^:]*$.*$/\2 (\1)/' /etc/passwd

(Show solution in ed)
#!/bin/sh # Backslash in front of END prevents shell from performing any # expansions in the text before END. # If wish to store the results into the input file, # we can add commmands w and q. cp /etc/passwd /tmp/passwd.$$ ed /tmp/passwd.$$<<\END g/^/s/^$[^:]*$:[^:]*:$[^:]*$.*$/\2 (\1)/ %p Q END rm /tmp/passwd.$$
Using sed put character # to the beginning of lines 10 to 20 in file /etc/passwd.
(Show solution in sed)
#!/bin/sh sed '10,20s/^/#/' /etc/passwd

(Show solution in ed)
#!/bin/sh ed /etc/passwd <<\END 10,20s/^/#/ %p Q END
Using sed put character “o” to the beginning of each odd line in file /etc/passwd and “e” to the beginning of each even line in this file.
(Show solution in sed)
#!/usr/bin/sed -f # This is a sed script, use with sed -f # Usage: sed -f <script> /etc/passwd s/^/o/ n s/^/e/

(Show solution in ed)
#!/bin/sh # The modified file is not written (it would not be a good idea in # case of /etc/passwd). # We first add an empty line at the end of the file, so that address $-1 refers # to the last line of the original file. # The last line of the file is then deleted. ed /etc/passwd <<\END $a . 1,$-1g/^/s//o/\ +s//e/ $d %p Q END
Using sed insert an empty line between each two lines of /etc/passwd.
(Show solution in sed)
#!/bin/sh # The $b command ensures that nothing will be added after the last # line. # Command a performs the actual empty line append. sed '$b; a\ ' /etc/passwd # Equivalently we can do it in the following way with i (the addresses ensure that nothing is added before the first line): sed '2,$i\ ' /etc/passwd

(Show solution in ed)
#!/bin/sh # Backslash in front of END prevents shell from performing any # expansions in the text before END. # # The backslash after i command is there because it is a command # within global g command, otherwise there would be no backslash after # i. ed /etc/passwd <<\END 2,$g/.*/i\ \ . %p Q END
Using sed reverse the order of lines in the input.
(Show solution in sed)
#!/bin/sh # We shall try it with nl /etc/passwd to see the difference. nl /etc/passwd | sed -n 'G; $p; h'

(Show solution in ed)
#!/bin/sh # Backslash in front of END prevents shell from performing any # expansions in the text before END. ed /etc/passwd <<\END g/^/m0 %p Q END
Using sed print only even lines from file /etc/passwd.
(Show solution in sed)
sed -n 'n; p' /etc/passwd

(Show solution in ed)
#!/bin/sh # Observe the backslash before string END which prevents shell # expansion in the text. # Command g performs the following commands on each line. # 1) +1p prints the next line and # 2) .s/^// unmarks (the next) line so that g skips it. # the last line is omitted (we assume that the file contains at least # two lines), because for the last line +1 would be behind the end of # file. # # If we would use s///, then instead of an empty regexp g would fill # in its own regexp and even lines would be deleted. If the changes # are not about to be saved, this is not an issue. nl /etc/passwd > /tmp/passwd.$$ ed /tmp/passwd<<\END 1,$-1g/^/+1p\ .s/^// Q END rm /tmp/passwd.$$
Using sed insert
#!/bin/sh
as the first line of a given file.
(Show solution in sed)
#!/usr/bin/sed -f # This is a sed script, use with sed -f 1i\ #!/bin/sh

(Show solution in ed)
#!/bin/sh ed inputfile<<\END 1i #!/bin/sh . w q END
Write a sed script which removes comments from a shell script, in particular:
1. If the first line has a special form of ^#!.*, then it should not be removed.
2. Lines which start with # (or have only white characters before the first #) should be deleted from the output.
3. Lines which contain some other characters before # should be modified to keep only the part before the # character.
4. For simplicity ignore the fact that in a shell script the # character can occur escaped, in quotation marks, apostrophes or in other contexts. Assume that each # in the input indeed starts a comment.
(Show solution in sed)
#!/usr/bin/sed -f # This is a sed script, use with sed -f # Skip the first line if it has a special form of #! 1{ /^#!/n } # Delete lines starting with # /^[[:space:]]*#/d # Deletes comments from the remaining lines (ignores the fact that # # can be a part of a string in quotation marks, apostrophes etC.). s/#.*$//

(Show solution in ed)
#!/bin/sh # We first deal with all lines except the first one. # - The first command deletes the lines having nothing before the # comment # character. # - The second command deletes everything from # to the end of line # from each of remaining lines. # Then we resolve the first line. # - We first delete the first line if it contains a comment having # nothing before # and does not have ! after #. # - We delete the first line if it contains only # and nothing else. # - Finally we remove everything from # to the end if there is not #! # at the beginning of line. ed inputfile.sh <<\END 2,$g/^[[:space:]]*#/d 2,$g/#/s/#.*// 1g/^[[:space:]]*#[^!]/d 1g/^[[:space:]]*#$/d 1v/^#!/s/#.*$// w q END

Extra assignments (without solutions)

Write a sed script which replaces all C style comments (i.e. /* */) to C++ style comments (i.e. //). Be aware of the following facts:
- C style comments can be on a single line, but they can be multi-line as well.
- If there is something after a C style comment, you have to move it to the next line.
- There can be multiple C style comments on a single line.
You may assume that the input file does not contain other occurrences of /* and */ except those which delimit a comment, that the comments are not nested and that the comment delimiters are well paired. For example
```
aaa /* bbb */ ccc /* ddd */ eee /* fff
ggg */ hhh /* iii */ jjj
kkk
/* lll
   mmm
   nnn */
ooo
```
would be turned into
```
aaa // bbb
ccc // ddd
eee // fff
// ggg
hhh // iii
jjj
kkk
// lll
// mmm
// nnn
ooo
```
The output of your script may include additional spaces (or newlines), but the output should be valid.
We define a number in a text file as a string which consists of digits 0-9 with possible leading sign + or - which is delimited with a blank character (a space, tab, begining of line, end of line, etc.). I.e. line
```
12 -31 aaaa b01 001 a-13
```
contains three numbers 12, -31 and 1 (leading zeros are ignored). Write shell commands which count the number of pairwise different numbers in a given file in absolute value (i.e. leading zeros are ignored, signs are ignored, in particular -31, 31, 0031, +0000031 all count as the same number in absolute value 31). Motivation: A CNF file such as formula.cnf consists of a header line and a list of clauses where numbers denote number of variables in a conjuctive normal form (CNF) formula. The commands from exercise should count the number of different variables in the formula. Exercise is a little bit more general.
Write a sed script which reads a decimal number n from the input and writes sequence 0, 1, ..., n to the output. Each number is on a separate line.
Write a sed script which outputs the last ten lines of the input.

Homework:

Using grep select those lines from countrycodes_en.csv where the first two characters in the three character code (Alpha-3 code, 3rd column) are not the same as the two characters in the two character code (Alpha-2 code, 2nd column). (Hint: use grep -v) [1 point]
Using sed replace in the input text file all occurrences of characters '&', '<', '>' with their HTML entities (i.e. replace '&' with '&', '<' with '<', and '>' with '>'). Call sed just once. [1 point]
Assume that the input lines contain only characters '(' and ')', write a sed script which checks whether each input line contains string of well paired brackets. The correct lines are replaced with 'ok' string, the wrong lines with 'wrong' string. [1 point]

8th practical (9th April, 2019)

Command: ed.

Using ed output the last ten lines of /etc/passwd.
(Show solution)
#!/bin/sh ed /etc/passwd <<\END $-9,$p Q END
Solve the exercises 6-14 from the 7th practical also in ed. For solutions see each of these exercises.
Run vimtutor

Homework

Write a script for ed which moves all lines starting with character '#' to the end of the input file. [1 point]
Consider a text file in which paragraphs are separated with an empty line. Write a script for sed or ed which joins the lines of each paragraph to a single line (with space as a separator). [1 point]
Write a sed script which reverses the order of characters on each line. The only character which cannot be part of a line is a newline character. [1 point]

9th practical (18th April, 2018)

Commands: read, control structures (for, while, if, case, etc.), printf, expr.

Other: Variables and expansion, redirection, positional and special parameters/variables ($#, $0, $n, ${n}, shift, "$@", set -, $?, $$, $!), command combination using &&, command grouping and subshell invokation, back apostrophes and $() expansion.

Write a shell script which expects two parameters, a file name and a number. The script deletes the line with given number from given file.
(Show solution)
#!/bin/sh ed "$1" <<END ${2}d w q END
Write a script which expects n+1 parameters, the first parameter contains a number x between 1 and n, the script then outputs the (x+1)st parameter. For instance script 2 a b c outputs b, i.e. the third parameter (or the second if we count starting from a. (Use shift with a parameter.)
(Show solution)
#!/bin/sh shift "$1" echo "$1"
Write script reverse which writes its parameters to standard output in reversed order. For example reverse a b c d outputs d c b a.
(Show solution)
#!/bin/sh for x do y="$x $y" done echo "$y"
Write a script max which selects the maximum of its numeric parameters. For example max 1 15 7 19 11 prints 19.
(Show solution)
#!/bin/sh # Without test, using sort for x do echo "$x" done | sort -nr | head -n 1 # Using sort, now we assume that the parameters are numbers without spaces: echo "$@" | tr " " '\n' | sort -nr | head -n 1 # Using test m="$1" shift for x do if [ "$x" -gt "$m" ] then m="$x" fi done echo "$m" # Using shell arithmetic expansion m="$1" shift for x do m=$((x>m ? x : m)) done echo "$m"
Write a script which expects one or two number parameters. The script then pads the first number to number of digits given in the second parameter. If the second parameter is not specified, it is considered to be 5. (Use printf and expansion ${variable:-word}.)
(Show solution)
#!/bin/sh printf "%${2:-5}d\n" $1
What the following lines write to the screen?
a="hello"; { echo $a ; a="hi" ; echo $a ; } ; echo $a b="hello"; ( echo $b ; b="hi" ; echo $b ; ) ; echo $b x="hello"; x="hi" sh -c 'echo $x'; echo $x y="hello"; y="hi"; sh -c 'echo $y'; echo $y z="hello"; sh -c 'echo $z' export u="hello"; sh -c 'echo $u'
What is the following command doing?
a=1;b=2; { a=LEFT; echo $a >&2; } |\ { b=RIGHT; echo $b; tr 'A-Z' 'a-z';}; echo "A=$a,B=$b"
And how is it different from:
a=1;b=2; { a=LEFT; echo $a; } |\ { b=RIGHT; echo $b; tr 'A-Z' 'a-z';}; echo "A=$a,B=$b"
What shall the following command write to out and err?
{ { { echo 1aaa;echo 2bbb >&2;} 3>&1 1>&2 2>&3 |\ { tr 'a-z' 'A-Z'; echo 2tr >&2;} 2>&1; } \ 3>&1 1>&2 2>&3; }>out 2>err
What shall the following commands write on screen?
sh -c "exit 1" | sh -c "exit 2" echo $? { { { sh -c "exit 1"; echo $? >&3; } |\ sh -c "exit 2" >&4; } 3>&1 | { read e; exit $e; }; } 4>&1 echo $? { { { sh -c "exit 1"; e=$?; } |\ sh -c "exit 2" >&4; } 3>&1 | { exit $e; }; } 4>&1 echo $?

Extra assignments (without solutions)

Extra assignments can use any commands

Assume the input file is in form of a csv file where columns are separated with a comma. Write a shell script which accepts two parameters, a file and a number of a column. The script then outputs a median of the numeric values in the given column within the given file. You can also enhance the script to compute average and standard deviation of the numeric values in the column.
We define Fibonacci words as follows: Word $S_0=0$, word $S_1=01$, then word $S_n=S_{n-1}S_{n-2}$ (i.e. concatenation of words $S_{n-1}$ and $S_{n-2}$). Write a shell script which given a number $n$ writes $n$-th Fibonacci word. Allow two optional parameters which specify characters to be used instead of 0 and 1.
Write a shell script which reads a file with a number on each line and sorts it. Each number from the input appears only once in the output, but the script computes the number of occurrences of each number within the input file and appends this frequency to each number as another column of the output file.

Homework

Write a script which works as cp but with reversed order of parameters (the target is the first parameter, note that cp accepts more than two parameters). [1 point]
Write a script which expects three parameters, a file name, a number n and a character c. The script then performs a circular shift of each input line (input is in the file specified in the first parameter). The lines are considered to be split into fields separated with character c and the number of elements to shift is given in number n. For example if the script is called like
script /etc/passwd 2 :
then each line of /etc/passwd is transformed from the initial format
login:*:uid:gid:full name:home dir:shell
to the following form:
uid:gid:full name:home dir:shell:login:*
The modified contents of the input file is written to the standard output, the file itself is left untouched. [2 points]

10th practical (23th April, 2019)

Commands: read, expr, dirname, basename.

Write a shell script which expects file paths in its parameters. The paths in parameters can be relative. For each path in parameter the script outputs the full path of the file.
(Show solution)
#!/bin/sh for x do directory=$(dirname "$x") directory=$(cd $directory && pwd) echo $directory/$(basename $x) done
Will the following script correctly compute the number of lines in /etc/passwd?
count=0; cat /etc/passwd | while read x do count=`expr $count + 1` done echo $count
And what about the following script?
cat /etc/passwd | { count=0 while read x do count=`expr $count + 1` done echo $count }
And what about the following script?
count=0 while read x do count=`expr $count + 1` done </etc/passwd echo $count
And finally, what about the following script?
count=0 while read x </etc/passwd do count=`expr $count + 1` done echo $count
What is the difference between all these scripts?
Write a shell script which renames all files in the current directory to files with the same name but where the upper case characters are changed into lower case. (E.g. file BACKUP.ZIP would be renamed to backup.zip.)
(Show solution)
#!/bin/sh for x in *[A-Z]* do mv "$x" "`echo $x | tr '[:upper:]' '[:lower:]'`" done
Write a shell script which renames all files in the current directory with suffix “.jpeg” to the files with the same name but with their suffix changed to “.jpg”.
(Show solution)
#!/bin/sh # A variant using special variable expansion (not for too long in # the standard though). for x in *.jpeg do mv "$x" "${x%jpeg}jpg" done # A variant using sed: for x in *.jpeg do mv "$x" "$(echo $x | sed 's/\.jpeg$/.jpg/')" done # A variant using basename and dirname. for x in *.jpeg do mv "$x" "$(dirname "$x")/$(basename "$x" .jpeg).jpg" done
Write a shell script which receives three parameters, all of them are numbers. The script outputs a list of numbers separated with spaces. The numbers start with $1, end with $2 and they are increased with step given in $3. If the third parameter is missing, then step 1 is considered by default.
(Show solution)
#!/bin/sh if [ $# -lt 2 ] || [ $# -gt 3 ] then echo Bad number of parameters. exit 1 fi step=${3:-1} x=$1 while [ $x -le $2 ] do printf "$x " x=`expr $x + $step` done
The same as the previous exercise, but now each number is on a separate line and they are justified to the number of digits of the longest number (i.e. $2). The numbers are justified with leading zeros, i.e.
script 0 10 2
would output:
```
00
02
04
06
08
10
```
(Show solution)
#!/bin/sh if [ $# -lt 2 ] || [ $# -gt 3 ] then echo Bad number of parameters exit 1 fi step=${3:-1} x=$1 len=${#2} while [ $x -le $2 ] do printf "%0${len}d\n" $x x=`expr $x + $step` done
Write a shell script which renames all files in the current directory having suffix “.jpg” to files with the same suffix but with numbers instead of names. The numbers are justified to the number of digits as necessary (given by the number of files). (Use the exercises of this practical). For example if the current directory would contain files alice.jpg, bob.jpg, cyril.jpg, daniel.jpg, they would be renamed to 1.jpg, 2.jpg, 3.jpg, and 4.jpg. If there would be at least 10 but at most 99 files having suffix “.jpg”, they would get names 01.jpg, 02.jpg, 03.jpg, 04.jpg, ...
(Show solution)
#!/bin/sh # If by any chance there is a file with name with the number in the # required output format, then this script might cause some problems. # In that case it is necessary to first move the files (under the new # names) to a temporary directory and back. x=1 count=`ls *.jpg | wc -l | tr -d " "` for name in *.jpg do mv "$name" `printf "%0${#count}d\n" $x`.jpg x=`expr $x + 1` done
Write a shell script which reads the lines from the standard input of the form:
a+b=c
where a, b, and c are numbers. The script reads the numbers on each line and checks if the equality really holds. The lines not satisfying the equality are then reported to standard output. At the end a total number of incorrect lines is written to standard output.
Hint: Use read in combination with IFS="+=".
(Show solution)
#!/bin/sh IFS="+=" line=0 while read a b c do line=$(expr $line + 1) if expr "$a" + "$b" != "$c" then printf "Error on line %d (%d+%d!=%d).\n" $line "$a" "$b" "$c" fi done printf "Number of processed lines: %d\n" $line

Homework

Write script average which computes the average value (rounded to an integer as computed with expr) of the numeric values given in the parameters parameters and writes the result to standard output. For example average 4 13 112 7 outputs 34. [1 point]
Write a script rmexcept, which expects n+1 parameters where the first parameter is a directory and the rest of parameters contain shell masks specifying file names. The script removes all files whose names do not match any of the given shell masks from the given directory. The files whose names match one of the masks, are left untouched (in particular they cannot be moved anywhere else and then back). E.g. calling rmexcept . '*.jpg' '*.png'
would remove all files whose suffix is neither jpg nor png from the current working directory. [2 points]

11th practical (30th April, 2019)

Command: awk.

Write an awk script which outputs each tenth line of the input.
(Show solution)
#!/bin/awk -f (NR % 10) == 0
Write an awk script which changes the characters in login field of /etc/passwd file to upper case.
(Show solution)
#!/bin/awk -f BEGIN { FS=":"; OFS=":" } { $1=toupper($1); print }
Write an awk script, which reverses the order of words at each line of the input (writes the words in opposite order).
(Show solution)
#!/bin/awk -f { for (i = NF; i > 1; i --) { printf ("%s" OFS, $i); } printf ("%s" ORS, $1); }
Write an awk script which numbers the lines of the input (similarly to nl).
(Show solution)
#!/bin/awk -f { print NR " " $0; }
Write an awk script which lists random numbers between 0 and 1, the number of random number to be output is given in the parameter.
(Show solution)
#!/bin/sh # This is a shell script which encapsulates the awk script. # The number of random numbers to be output is passed as the value of n. awk -v n="$1" 'BEGIN { srand(); for (i=0; i<n; ++i) { print rand (); } }'
Write an awk script which prints the number of seconds since beginning of the UNIX era and terminates. Hint: Use srand().
(Show solution)
#!/bin/awk -f BEGIN { srand(); printf("%d\n", srand()); }
Write an awk script which computes the number of HTML links in an HTML file (i.e. counts the tags <a).
(Show solution)
#!/bin/awk -f # RS will be "<", because that is how HTML is separated. BEGIN { RS = "<"; cnt = 0; } ($1 ~ /^a$/) { ++ cnt; } END { print cnt; }
Write an awk script which expects an HTML file at the input and lists the web addresses which the file references (using tag <a href="address">. Note that the tag can be split over several lines and that there can be spaces around a and href.
Hint: Use suitable RS and FS values.
(Show solution)
#!/bin/awk -f # The solution is a bit complicated so that it works even if there are # other parameters to a between a and href. Otherwise it would be # enough to take i=1. BEGIN { RS = "<"; FS = "="; } $1 ~ /^[[:blank:]]*a[[:blank:]]+/ { i=1 while (i <= NF) { if ($i ~ /[[:blank:]]+href[[:blank:]]*$/) { split ($(i+1), a, "[[:blank:]]*\"[[:blank:]]*"); print a[2] } else if ($i ~ />/) { break; } ++i; } }
Write an awk scripts which outputs the last 10 lines of the input file.
(Show solution)
#!/bin/awk -f BEGIN { konecBufferu = 0; } { buffer [konecBufferu] = $0; konecBufferu = (konecBufferu + 1) % 10; } END { if (NR > 0) { if (NR < 10) { indexVBufferu = 0; } else { indexVBufferu = konecBufferu; } do { print buffer[indexVBufferu]; indexVBufferu = (indexVBufferu + 1) % 10; } while (indexVBufferu != konecBufferu) } }
Let us define a word as a maximal sequence of alphanumeric characters (from class [:alnum:]). Write an awk script which reads a text and after it finishes the reading, it prints for each word in the text file the number of its occurrences within the input text.
(Show solution)
#!/bin/awk -f BEGIN { FS="[^[:alnum:]]" } { for(i=1; i<=NF; ++i) { if($i!="") { ++count[$i] } } } END { for(word in count) { printf("%s: %d\n", word, count[word]) } }
Write an awk script which expects a file in the csv format at the input. This means that each line of the input line consists of several fields separated with a comma ",". The first line is the header line, it contains the column names. The first column of the subsequent lines contains a name, the rest of columns contain numbers. The script adds a new column called “Sum” (in the header line), which on each line contains the sum of the number fields. For example in case of input
```
Name,pts1,pts2,pts3
Hynek,3,12,9
Jarmila,7,34,1
Vilém,8,27,0
```
the following output is produced by the awk script:
```
Name,pts1,pts2,pts3,Sum
Hynek,3,12,9,24
Jarmila,7,34,1,42
Vilém,8,27,0,35
```
(Show solution)
#!/usr/bin/awk -f BEGIN { FS=","; OFS="," } FNR == 1 { $(NF+1)="Sum"; print } FNR > 1 { $(NF+1)=0 for (i=1; i < NF; i++) { $NF+=$i } print }

Homework:

When you set the record separator to the empty string, it means that the records are separated with an empty line.

Write an awk which reads a text and if it encounters a line starting with # (i.e. a comment line), it will add # followed by a space to the beginning of each following line until the end of the paragraph (paragraphs are separated with an empty line). The lines which already start with # are not modified. [1 point]
E.g. the following input
```
a b c
d e f
# g h i
j k l
m n o

p q r
s t u

# v w x
# y z 1
2 3 4
# 5 6 7
8 9 0
```
would be modified into
```
a b c
d e f
# g h i
# j k l
# m n o

p q r
s t u

# v w x
# y z 1
# 2 3 4
# 5 6 7
# 8 9 0
```
Write an awk script which formats the lines of each paragraph in a file to a given number of characters (without splitting the words). The awk script takes a numeric parameter n (e.g. using -v n=...). The rest of parameters are files to be processed. The script considers the input files to be text files where paragraphs are separated with an empty line. Within each paragraph the lines are formated to be at most n characters long. Note that if you set RS to an empty string, the records will be exactly the paragraphs.[2 points]

12th practical (8th May, 2019)

Commands: trap, ps, kill, sleep, date.

Write a script which prints every line it reads from the standard input. If the script receives signal KILL (9), then it naturally stops. If the script receives one of signals TERM (15), QUIT (3), INT (2) (the one after CTRL-C), the script does not stop, but it only prints out the number of singal it received. In case of INT (2) the script moreover prints out the last line which was read from the input.
(Show solution)
#!/bin/sh trap "echo 15" TERM trap "echo 3" QUIT trap 'echo 2; echo $x' INT while read x do echo $x done
Write a script which expects two parameters, a signal and a program name. The script then sends the signal to all processes of the program. You can assume that the program name does not contain any weird characters (it can contain a space, however). That is, the aim is to create something like killall, but killall itself cannot be used as it is not a standard command. I suggest to look at parameter -o of command ps.
(Show solution)
#!/bin/sh [ $# -eq 2 ] || { echo "Expecting two parameters, a signal and a program name."; exit 1 } signal="$1" program="$2" ps -e -o pid= -o comm= |\ sed -n 's/^ *$[[:digit:]]\{1,\}$ \{1,\}'"$program"'$/\1/p' |\ xargs kill -$signal # The same can be achieved using a while cycle. ps -e -o pid= -o comm= | while read pid cmd do if [ "$cmd" == "$2" ] then kill -$signal $pid fi done
Write a script which expects one parameter specifying number of seconds. The script then waits for given number of seconds before it finishes. When the script receives signal SIGINT (2 - Ctrl-C), then it writes down how many seconds did it wait until now. When the script receives signal SIGQUIT (3 - Ctrl-\), then it writes down how many seconds remain. When the script receives SIGTERM (15), then it writes down how many seconds did it wait, how many seconds remain and stops. The exit code of the script should be 0 if it finishes in the required time, or the remaining number seconds if it was terminated earlier (due to SIGTERM (15)).
(Show solution)
#!/bin/sh die() { echo "$1" exit 1; } now() { awk 'BEGIN { srand(); print srand(); }' } [ $# -eq 1 ] || die "Wrong number of parameters" expr "$1" : '^[[:digit:]]*$' >/dev/null || die "Parameter is not a number" start=$(now) stop=$((start + $1)) trap 'echo $(($(now) - start))' 2 trap 'echo $((stop - $(now)))' 3 trap 'echo $(($(now) - start)); kill $! 2>/dev/null; exit' 15 while [ "$(now)" -lt "$stop" ] do sleep $((stop - $(now) )) & wait $! [ $? -gt 128 ] && kill $! 2>/dev/null done # It would be sufficient to use sleep 1 in the above cycle, but then # the process would wake up each second. # sleep $((stop - $(now) )) is not enough by itself because of SIGTERM, # if the shell process running the script receives SIGTERM when sleep # is running, then it first lets sleep to finish and then it processes # the signal using trap. (This is not the case of Ctrl-C as there the # signal is received by sleep.) That is why sleep is being run on # background and then it is waited for the child process to finish. # After wait the signal is processed and SIGTERM is passed to the # sleep process if necessary. Wait returns 128 if it was terminated # due to a signal.
Sample exam question with a solution

Homework

Write a script which will indent lines in the input files depending on how deep the brackets are nested. The parameters to the script are an indentation character c and a number of characters per level n, these parameters are followed by a list of files (if no files are provided, standard input is used). The script then reads line by line and on each line it removes the white characters at the beginning of line and replaces them with k*n characters c where k is the level of bracket nesting. Consider normal brackets (), curly brackets {} and square brackets []. For example the input file
```
a(b
     c d[ e]f [
  g h { j (
            k)}l m
     n ]o )p
q r
```
would be modified as follows if the script is run with parameters c='.' and n=1:
```
a(b
.c d[ e]f [
..g h { j (
....k)}l m
..n ]o ) p
q r
```
You can assume that the input has well paired brackets. A space or a tab character must be allowed as character c. [3 points]

13th practical (20th May, 2019)

Commands: eval

Write a script which accepts names of shell variables in parameters. For each variable name passed as a parameter the script determines and outputs its state. The state of a variable can be one of the following three values:

NONEMPTY
The value of variable is defined and nonempty.

NULL
The value of variable is defined and empty.

UNSET
The value of variable has not been set (it is undefined).

To test a state of a variable you can use expansion ${x+word} and checking nonemptiness of a string using test.
(Show solution)
#!/bin/sh for x do if [ "$(eval echo '${'"$x"'+set}')" != set ] then echo "$x UNSET" elif [ "$(eval echo \$"$x")" ] then echo "$x NONEMPTY" else echo "$x EMPTY" fi done
Write a script which receives a file name f in a parameter. The script then accepts a sequence of numbers on the standard input. For each such number i the script then prints line number i from file f. Implement the script in such a way that it first reads all lines into an "array" which is implemented using eval (i.e. eval '$'ARRAY_$INDEX). Then the script answers the queries by acquiring the lines from the array.
(Show solution)
#!/bin/sh [ $# -eq 1 ] || { echo "Wrong number of parameters"; exit 1; } count=0 while read Lines_$pocet do count=$((count+1)) done <"$1" while read index do eval echo '$'Lines_$index done
Write script cutreorder which behaves similarly to cut, it also has parameters -d and -f with the same format, meaning, and default values as cut. The rest of the parameters contains file names (if missing, standard input is used). The difference from cut is that it reorders the columns in the output so that they are in the order specified with parameters -f. (Recall that in cut the columns in the output are in the same order as in the input file.) For instance:
- cutreorder -d : -f 3,1 /etc/passwd would print /etc/passwd in two columns uid:login.
- cutreorder -d : -f 3,3-5,1 /etc/passwd would print columns uid:uid:gid:full name:login.
(Show solution)
#!/bin/sh # Processing parameters # The default delimiter is tab (put here literaly between the quotation marks. delim=" " fields="" while getopts f:d: name do case $name in f) fields="$OPTARG";; d) delim="$OPTARG";; ?) printf "Use: %s ...\n" $0;; esac done shift $((OPTIND-1)) if [ -z "$fields" ] then # Nothing to output exit 0 fi # Check if the standard input should not be used if [ $# -eq 0 ] then set -- - fi # Modify the fields into a form suitbale to be used in a for cycle fields="`echo $fields | tr , " "`" # Process the files cat "$@" | while read line do result="" # Process the fields in the input and output for f in $fields do if [ "$result" ] then result="$result$delim" fi result="$result$(echo "$line" | cut -d $delim -f $f)" done echo "$result" done

Additional homework (if you do not have enough points).

Further exercises will be added as needed.

Write a shell script which selects n longest lines from the files passed to it in parameters. Each output line is prepended with its number within the file (format line number: line content). The lines in the output are in the same order as they are in the input file, i.e. as if sorted by their line number. Parameter n is 5 by default but it can be specified in the first parameter as -n10 or in the first two parameters as -n 10 (this is for the case when n=10). The syntax is thus same as in case of head. If the file names are missing in parameters, standard input is used. If at least two files are specified in parameters, each list of lines is preceded by the corresponding file name. [3 points]
Write a script csvcut which accepts two or more parameters with the following meaning: The 1st parameter is a delimiter character delim, the 2nd parameter is a string specifying a column in a CSV file, the 3rd and following parameter specify file names (if these names are missing, the script reads standard input). The script assumes that the input files are CSV files with delim as a delimiter character. Moreover the script assumes that the first line of each of these files contains a header. The script cuts the specified column from the input files, i.e. it cuts the column whose name is equal to the 2nd parameter of the script, regardless the possible quotation marks. For example
- csvcut ';' Measure calories.csv
  cuts the second column from file calories.csv that is it would be equivalent to cut -d ';' -f 2 calories.csv).
- csvcut ';' Food calories.csv
  cuts the first column from the same file although the name is surrounded with quotation marks.
Write a script which receives number of seconds sec, a program name prg followed by its arguments. The script runs program prg with its arguments and then waits for it to finish. If the program does not finish within sec seconds, the script sends signal TERM to prg. If after at most another 5 seconds the program still does not finished, the script kills it by sending signal KILL to it. [3 points]
Let us define a word to be a maximal nonempty sequence of alphanumeric characters and underscores (i.e. matching BRE [[:alnum:]_]\{1,\}). Consider the word to name a variable (function, type, etc.) in a programming language. We say that the word is in camel case if the subwords are introduced with an upper case letter (such as MyVerySpecialVariable). The word can start with an upper case character (UpperCamelCase) or a lower case character (lowerCamelCase). For simplicity, we assume underscores are not present in camel case. Another common naming style is snake case which are in lower case only and subwords are separated with underscores (such as my_very_special_variable, snake_case). Your task is to write two scripts which translate one style to another (camel2snake and snake2camel). In particular the scripts work as filters, i.e. they read the standard input and write to the standard output.
- Script camel2snake translate all words which are in UpperCamelCase or lowerCamelCase are translated to snake case (words containing underscores are left untouched).
- On the other hand script snake2camel translate every word in snake case into camel case. Default is UpperCamelCase. If option -l is present, then it translates into lowerCamelCase. Words not in snake case are left untouched (those containing capital characters).
[4 points, 2 for each script].
Ceasar cipher. In a Ceasar cipher we shift every alphabet character by a given number. Write a script which can encrypt and decrypt a text using Ceasar cipher. It accepts two parameters, the first is the action (d for decrypt, e for encrypt), the second is a number n. The script then reads a text from the standard input and either encrypts, or decrypts the text by shifting each English alphabet letter by n positions (forward, or backward), the cases of characters must be preserved. The output is written to the standard output. [3 points]
Write two scripts for translating roman numbers into decimal and back. Each script takes as a single parameter a string and writes the result to the standard output. Each direction (roman to decimal and decimal to roman) is for 2 points. See also how to convert number to roman numerals and how to convert roman numerals to number.

Practicals to Introduction to UNIX, summer semester 2018/19

Petr Kučera, KTIML MFF UK

(Česká verze)

Basic information

Useful links

Credit requirements

Some peculiarities of the Malá Strana UNIX lab

Content of the practicals

1st practical (19th February, 2019)

Exercises

Homework

2nd practical (26th February, 2019)

Exercises

Homework

3rd practical (5th March, 2019)

Exercises

Homework

4th practical (13th March, 2019)

Exercises

Homework

5th practical (19th March, 2019)

Exercises

Homework

6th practical (26th March, 2019)

Exercises

Homework

7th practical (2nd April, 2019)

Exercises

Extra assignments (without solutions)

Homework:

8th practical (9th April, 2019)

Homework

9th practical (18th April, 2018)

Extra assignments (without solutions)

Homework

10th practical (23th April, 2019)

Homework

11th practical (30th April, 2019)

Homework:

12th practical (8th May, 2019)

Homework

13th practical (20th May, 2019)

Additional homework (if you do not have enough points).