Practicals to Introduction to UNIX, summer semester 2017/18

Petr Kučera, KTIML MFF UK

Basic information

Useful links

Credit requirements

At the end of each practical homework shall be assigned, I will expect to receive solutions to them by e-mail before the beginning of the next practical. If by the end of the semester you have at least 2/3 of possible number of points, you get a credit. Otherwise you will need to finish additional homework to get enough points. In case you submit solutions to less than 1/3 of homework assignments, you will not have a possibility to get a credit. Each set of homework exercises assigned after each practical will be worth 3 points, at the beginning of the semester that could mean three exercises while at the end it can be just a single but more difficult exercise.

If not stated otherwise, you are only allowed to use the commands, tools and structures we have used on practicals before the homework was assigned (including the one to which homework was assigned). In some cases the tools might be even more restricted. If I come across a solution which is just a copy of another one I will assign no points for this solution. If this case repeats there will be no possibility to obtain a credit from me.

Please, send me your solutions via an e-mail and do so before the beginning of the next practical. Either write your solution as a part of e-mail body, or append them in a form of plain text files, do not use pdf, word or similar file types. The homework assignments are appended below a list of exercises we have done at the corresponding practical (i.e. the current homework exercises are always at the bottom of this web page).

Some peculiarities of the Malá Strana UNIX lab

I think it is worth to know and bear in mind some specifics of the Malá Strana UNIX lab. To Czech students I recommend to check the web pages of the UNIX lab. I will warn you of the lab properties as we come across them. In this section I would to mention at least the most apparent of them:

  1. Users home directories are placed in a remote AFS filesystem, where standard UNIX permissions do not apply and the standard UNIX commands for file permissions manipulations (mainly chmod) cannot be used or they can be used only in a very restricted way (there is a web page about the ACL which is the system in use, but it is only in Czech to the best of my knowledge).
  2. This means that when you try the chmod command or anything which has some connection to the standard UNIX permission system, you should do so in local directories (such as /tmp in which you are allowed more or less anything). There the standard UNIX permissions apply.
  3. As user accounts are stored remotely, local file /etc/passwd does not contain information about your account or accounts of other lab users. If you want to get your and other users account info in the format of /etc/passwd you can achieve that using a getent command as follows:
    getent passwd
    for other parameters and general help on this command use getent --help.
  4. The same applies to /etc/group and getent group.
  5. The find command (still most probably) is not in the best of terms with AFS filesystem — when used on network AFS drives in the lab such as your home dir you can get unexpected results. You can use find to search on local directories such as /tmp, /etc, /usr etc. The exercises usually suggest it.

Content of the practicals

1st practical (26th February, 2016)

Basic commands, shell command line edit.

More detailed:

Exercises

  1. In your home directory create a directory named DIR
  2. Copy all files whose filenames satisfy the following conditions to ~/DIR. The files are in /usr/include directory, their names start with m, end with .h and contain a number.
  3. Create a subdirectory called SUBDIR in your DIR directory.
  4. The first five lines of each file you have copied from /usr/include copy to file ~/DIR/SUBDIR/firstfive.
  5. The last lines of files in ~/DIR copy to file ~/DIR/SUBDIR/last.
  6. Concatenate the two files in ~/DIR/SUBDIR into one file ~/DIR/SUBDIR/firstandlast
  7. Delete the files in ~/DIR/SUBDIR except firstandlast.
  8. Store the number of files and directories in ~/DIR into a file ~/DIR/SUBDIR/count
  9. Output the long information on ~/DIR/SUBDIR directory. (Not its content, but information on it).
  10. Delete the contents of ~/DIR/SUBDIR/firstandlast file without removing the file itself.
  11. Add a line containing just a star sign (i.e. *) to file ~/DIR/SUBDIR/firstandlast.
  12. Delete ~/DIR together with all the files it contains.
  13. (Show solutions to preceding exercises)
    #!/bin/sh

    # In your home directory create a directory named DIR
    mkdir ~/DIR

    # Copy all files whose filenames satisfy the following conditions to
    # ~/DIR. The files are in /usr/include directory,
    # their names start with 'm', end with '.h' and contain a
    # number.
    cp /usr/include/m*[0-9]*.h ~/DIR

    # Create a subdirectory called SUBDIR in your DIR directory.
    mkdir ~/DIR/SUBDIR

    # The first five lines of each file you have copied from
    # /usr/include copy to file ~/DIR/SUBDIR/firstfive.
    head -n 5 ~/DIR/m*[0-9]*.h >~/DIR/SUBDIR/firstfive

    # The last lines of files in ~/DIR copy to file
    # ~/DIR/SUBDIR/last.
    tail -n 1 ~/DIR/m*[0-9]*.h >~/DIR/SUBDIR/last

    # Concatenate the two files in ~/DIR/SUBDIR into one file
    # ~/DIR/SUBDIR/firstandlast
    cat ~/DIR/SUBDIR/* >~/DIR/SUBDIR/firstandlast

    # Delete the files in ~/DIR/SUBDIR except firstandlast.
    rm ~/DIR/SUBDIR/firstfive
    rm ~/DIR/SUBDIR/last

    # Store the number of files and directories in ~/DIR into a
    # file ~/DIR/SUBDIR/count
    ls ~/DIR | wc -l >~/DIR/SUBDIR/count

    # Output the long information about ~/DIR/SUBDIR directory.
    # (Not its content, but information about it).
    ls -ld ~/DIR/SUBDIR

    # Delete the contents of ~/DIR/SUBDIR/firstandlast file
    # without removing the file itself.
    cp /dev/null ~/DIR/SUBDIR/firstandlast

    # Add a line containing just a star sign (i.e. *) to file
    # ~/DIR/SUBDIR/firstandlast.
    echo '*' >>~/DIR/SUBDIR/firstandlast

    # Delete ~/DIR together with all files the it contains.
    rm -r ~/DIR
  14. Output lines number 11-20 from file /etc/passwd.
    (Show solution)
    #!/bin/sh

    head -n20 /etc/passwd | tail -n10

Homework

  1. Write a command which lists all files in your home directory whose name starts with a dot (“.”) and contain at least two other characters. [1 point]
  2. Write a command which prints the last line of each file in directory /usr/include whose names end with suffix “.h”. [1 point]
  3. Create a directory INCL in your home directory. Write a command which copies files whose names do not start with a digit but contain a digit in the name from directory /usr/include to the above created directory INCL in your home directory. [1 point]

2nd practical (28th February, 2018)

Basic commands and shell usage.

Commands: date, tee, touch, cut, paste, tr.

Exercises

  1. Create a file with name “-f” which contains a line with current date and time. Then delete this file.
    (Show solution)
    #!/bin/sh

    # Create a file -f with date and time:
    date >-f
    # Delete file -f:
    rm -- -f
  2. Create a file named “fixed date” (i.e. “fixed<space>date”) with last date and time of modification is set to 13:30, 1st February, 2009.
    (Show solution)
    #!/bin/sh

    touch -t 200902011330 "fixed date"
    # or
    touch -d "2009-02-01 13:30:00" "fixed date"
  3. Copy the file /etc/passwd into your home directory. The copy should have name accounts and the last modification date and time should be preserved, i.e. should be the same as the last modification date and time of /etc/passwd.
    (Show solution)
    #!/bin/sh

    cp -p /etc/passwd ucty
  4. Set the time and date of last modification of file accounts to the same time as the last modification time of the file /etc/group.
    (Show solution)
    #!/bin/sh

    touch -r /etc/group accounts
  5. Write a command which outputs the list of files in directory /usr/bin sorted lexicographically in decreasing order. The output is written to files bina, binb, and to the screen. The screen output is viewed in pages (use less, or more).
    (Show solution)
    #!/bin/sh

    ls -r /usr/bin | tee bina binb | more
  6. What happens during the execution of the following three commands? What is the difference betweeen them?
    1. mv file /dev/null
    2. cp file /dev/null
    3. cat file >/dev/null
  7. Write commands that output the ten biggest and the ten smallest files in directory /etc.
    (Show solution)
    #!/bin/sh

    ls -S /etc | head
    ls -rS /etc | head
  8. Find a file in /usr/bin with the most recent modification date and time.
    (Show solution)
    #!/bin/sh

    ls -t /usr/bin | head -n 1
  9. Download file calories.csv, it is a file in a CSV format with semicolon (“;”) used as a field separator.
  10. Replace quotation marks (") with apostrophes (') in file calories.csv
    (Show solution)
    #!/bin/sh

    tr \" \' <calories.csv >/tmp/calories.a.csv
    mv /tmp/calories.a.csv calories.csv
  11. Remove apostrophes (') from file calories.csv. In this and the several following exercises we assume that the file is has been modified by preceding exercises, i.e. here after the previous exercise, quotation marks have been replaced with apostrophes.
    (Show solution)
    #!/bin/sh

    tr -d \' <calories.csv >/tmp/calories.a.csv
    mv /tmp/calories.a.csv calories.csv
  12. Select the first column from file calories.csv, i.e. the column with food names.
    (Show solution)
    #!/bin/sh

    cut -f1 -d\; calories.csv
  13. In file calories.csv replace upper case characters with their lower case counterparts, but only in the first column. (Cut the first columns using cut, replace capital letters with small ones using tr, then put them back using paste.)
    (Show solution)
    #!/bin/sh

    cut -f1 -d\; calories.csv >/tmp/first
    cut -f2- -d\; calories.csv >/tmp/others
    tr '[:upper:]' '[:lower:]' </tmp/first | paste -d\; - /tmp/others >calories.csv
    rm /tmp/first /tmp/others
  14. Reorder the last three columns of file calories.csv in the opposite order, i.e. after reordering the first column contains amount of proteins, the second contains amount of carbohydrates and the third column contains the amount of fat. Use cut and paste.
    (Show solution)
    #!/bin/sh

    cut -f5 -d\; calories.csv >/tmp/fifth
    cut -f6 -d\; calories.csv >/tmp/sixth
    cut -f7 -d\; calories.csv >/tmp/seventh
    cut -f1-4 -d\; calories.csv | paste -d\; - /tmp/seventh /tmp/sixth /tmp/fifth >/tmp/novy
    mv /tmp/novy calories.csv
    rm /tmp/fifth /tmp/sixth /tmp/seventh
  15. Select the 9 characters with permissions from the output of ls -l command.
    (Show solution)
    #!/bin/sh

    ls -l | tail -n +2 | cut -c2-10
  16. Suppose you have three files, summand1, summand2 and sum, each of these files contains the same number of lines, each line contains just a number. Compose these files to just one output where each line has the following format:
    summand1+summand2=sum.
    (Show solution)
    #!/bin/sh

    paste -d+= summand1 summand2 sum

Homework

  1. Write a command which sets the last modification date and time of a file to 11th March, 1993, 15:04. [1 point]
  2. In file calories.csv replace each food name with a “-” (minus) character. Other columns should remain unchanged. [1 point]
  3. For each user in /etc/passwd output a pair uid:login. [1 point].

3rd practical (7th March, 2018)

Permissions, links, communication utilities.

Commands: ssh, scp, who, id, mail (mailx), last (not a standard command), chmod, chgrp, chown, file, ln, df, du, umask.

Attention, in Malá Strana UNIX lab try permissions in /tmp directory. Other directories are on afs file system which has a different system of permission handling.

Exercises

  1. Output the list of files in the current directory in a form of “size filename”. The list should be created by filtering the outputs of ls -l and ls commands.
    (Show solution)
    #!/bin/sh

    ls -l | tail -n +2 | tr -s " " | cut -d" " -f5 >/tmp/lsl
    ls | paste -d" " /tmp/lsl -
    rm /tmp/lsl
  2. Output the logins from file /etc/passwd, each line contains 5 logins separated with a comma (,).
    (Show solution)
    #!/bin/sh

    cut -d: -f1 </etc/passwd | paste -d"," - - - - -

    # Alternatively and equivalently:
    cut -d: -f1 </etc/passwd | paste -s -d",,,,\n" -
  3. Write a command which outputs your current UID.
    (Show solution)
    #!/bin/sh

    id -u
  4. Write a command that outputs the list of names of groups to which you belong as a user.
    (Show solution)
    #!/bin/sh

    id -Gn
  5. Send the contents of file /etc/group by an e-mail using command mail (or mailx).
    (Show solution)
    #!/bin/sh

    mail -s "Obsah /etc/group" adresa </etc/group
  6. Try the command df which determines amount of space used on a disk. Try different options (such as -k and -P, you can also try the other ones but note that only other standard parameter is -t).
  7. Try the command du on directory /tmp or on your home directory with different options (-k, -s, -x), find out more about these options in the specification.
  8. In your home directory create a link to /etc/passwd. First try to create a hard link, then a soft link. Why a hard link cannot be created in lab? Call ls -l to find out how a soft link is displayed in its output.
    (Show solution)
    #!/bin/sh

    # Hard link
    ln /etc/passwd ~
    # Soft link
    ln -s /etc/passwd ~
  9. In directory /tmp create a hard link to another file in /tmp (you can copy some file to /tmp for this purpose, the hard link will naturally have a different name). Run ls -l on both files and observe how a number in the second column changed. Using ls -i check that both copies (the original file and its hard link) have the same inode number.
  10. Create a simple script and assign it an execute permission.
    (Show solution)
    #!/bin/sh

    cat >script.sh <<EOF
    #!/bin/sh

    echo Hello world!
    EOF


    chmod a+x script.sh
  11. Create a file which the owner cannot neither execute, read, or write to it, the members of file group can read and write to the file, and the others can just execute the and not read nor write to it.
    (Show solution)
    #!/bin/sh
    cat >script.sh <<EOF
    #!/bin/sh

    echo Hello world!
    EOF


    chmod 0761 /tmp/script.sh

    # Equivalently:
    chmod u+rwx,g=u-x,o=u-rw /tmp/script.sh

    # or
    chmod u=rwx,g=rw,o=x /tmp/script.sh
  12. Create a script which cannot be read but has execute permissions. Try tu run this script. Why it is not possible to execute an unreadable shell script?
    (Show solution)
    #!/bin/sh
    cat >/tmp/script.sh <<EOF
    #!/bin/sh

    echo Hello world
    EOF


    chmod a+x-r /tmp/script.sh
    /tmp/script.sh
    # If a shell should run the script, it must be allowed to read it, too.
  13. Create a directory to which anyone can go (by cd, can be used in path) and add files but only the directory owner can list its contents.
    (Show solution)
    #!/bin/sh

    mkdir /tmp/directory
    chmod 0755 /tmp/directory

    # Or equivalently
    chmod u+rwx,go=u-r /tmp/directory
  14. Create a directory to which you can go, you cannot read its contents but you can add files to it. Create a file in this directory which can be only read (and not written to or executed). Delete this file using rm. Why you can delete a file which you cannot modify?
    (Show solution)
    #!/bin/sh
    mkdir /tmp/directory
    chmod 0300 /tmp/directory
    # Or equivalently
    chmod u+wx-r,go-rwx /tmp/directory

    touch /tmp/directory/file
    chmod 0400 /tmp/directory/file
    # Or equivalently
    chmod u+r-wx,go-rwx /tmp/directory/file

    rm -f /tmp/directory/file
  15. Make a copy of /etc/passwd in /tmp using cp without and then with -p option. Observe the difference (with the help of ls -l).

Homework

  1. Write a command, which adds a group read permission to all files in a subtree of the directory /tmp/dir. Use only one call to chmod [1 point]
  2. Create a file and set its permissions so that the owner can execute this file, read and write to it, group members can execute and read it, and the others can only read the file. In chmod command use a symbolic way of specifying the permissions [1 point] and also the octal numeric way [1 point].

4th practical (14th March, 2018)

Simple utilities.

Commands: sort, diff, comm, split.

Exercises

  1. Write the names of files which occur in /usr/bin but which are not in /bin.
    (Show solution)
    #!/bin/sh

    ls /usr/bin | sort >/tmp/usr_bin
    ls /bin | sort | comm -13 - /tmp/usr_bin
    rm /tmp/usr_bin
  2. Write the numbers of groups which are used in /etc/group but which are not used in /etc/passwd. That is find the numbers of groups which are not used as a primary group for any user.
    (Show solution)
    #!/bin/sh

    cut -d: -f4 </etc/passwd | sort > /tmp/uid
    cut -d: -f3 </etc/group | sort | comm -23 - /tmp/uid
    rm /tmp/uid
  3. In file calories.csv replace commas (,) with hyphens (-), using diff then determine which lines have been changed.
    (Show solution)
    #!/bin/sh

    tr "," "-" <calories.csv | diff calories.csv -
  4. Sort file calories.csv according to amount of calories in increasing order. The first header line has to remain at the beginning of the file.
    (Show solution)
    #!/bin/sh

    # Save the header
    head -n 1 calories.csv >/tmp/header
    # Sort the rest and append it to the header
    tail -n +2 calories.csv | sort -t\; -k4,4n >>/tmp/header
    mv /tmp/header calories.csv
  5. Sort file calories.csv according to triple [protein, carbohydrates, fat] in an increasing order. The first header line has to remain at the beginning of the file.
    (Show solution)
    #!/bin/sh

    # Save the header
    head -n 1 calories.csv >/tmp/header
    # Sort the rest and append it to the header
    tail -n +2 calories.csv | sort -t\; -k7,7n -k6,6n -k5,5n >>/tmp/header
    mv /tmp/header calories.csv
  6. Determine the number of different units used in the second column of file calories.csv. First consider two units with different number as different, then as the same, such as in case of “1 Cup” and “2 cup”.
    (Show solution)
    #!/bin/sh

    # In the first solution we consider two units with a different amount
    # as different, e.g. “1 Cup” and “2 Cup” are different units.
    tail -n +2 calories.csv | sort -t\; -k2,2 -u | wc -l

    # If we wish to consider two units with different amounts as the same
    # unit (i.e. we do not want to differentiate between “1 Cup” and “2
    # Cup”, we can proceed in the following way:
    tail -n +2 calories.csv | cut -f2 -d\; | cut -f2- -d" " | sort -u | wc -l
  7. It is known that a welfare crook lives in Los Angeles. You are given lists for 1) people receiving welfare, 2) Hollywood actors, and 3) residents of Beverly Hills. The lists are given as text files where each line contains one name of a person. A welfare crook is someone who appears in all three lists. Write a script to find at least one crook. Example files: social.txt, actor.txt, beverly_hills.txt.
    (Show solution)
    #!/bin/sh

    sort social.txt >/tmp/s_social.txt
    sort beverly_hills.txt >/tmp/s_beverly_hills.txt
    sort actor.txt | comm -12 /tmp/s_social.txt - | comm -12 - /tmp/s_beverly_hills.txt
    rm /tmp/s_social.txt /tmp/s_beverly_hills.txt.
  8. Split a file (e.g. /etc/passwd) to parts consisting of five lines, then concatenate these parts back to one file.
    (Show solution)
    #!/bin/sh

    # We shall assume that the number of lines in /etc/passwd is not too
    # big, in particular that when using split the two letter suffix is
    # enough.
    split -l5 /etc/passwd passwdpart
    # The concatenated file will be stored in the current directory, we
    # naturally do not have the write permission to /etc directory.
    cat passwdpart* >passwd
    rm passwdpart*
  9. Write the logins from /etc/passwd to ten lines, the logins on each line are separated with spaces.
    (Show solution)
    #!/bin/sh

    # We assume that /etc/passwd does not contain too much records and
    # that the default two letter suffix is enough in split.
    cut -d":" -f1 /etc/passwd | split -l10 - logins
    paste -d" " logins*
    rm logins*
  10. Create a copy of /etc/group file in which each login listed in the field with group members is replaced with a single character '@' (you may assume that a login consists of alphanumeric symbols and '_'). The other columns remain unchanged.
    (Show solution)
    #/bin/sh

    cut -d: -f4 /etc/group | tr -sc ',\n' '[@*]' >/tmp/memb.$$
    cut -d: -f-3 /etc/group | paste -d: - /tmp/memb.$$ >group
    rm /tmp/memb.$$
  11. Determine the name of a group with the highest number of members assigned in the file /etc/group. This is a follow up to the previous exercise. In particular you may assume that you already have the file produced in the previous exercise.
    (Show solution)
    #/bin/sh

    sort -t: -k4,4r group | head -n 1 | cut -d: -f1

Homework

  1. Determine the login of the user with the highest uid in /etc/passwd. [1 point]
  2. Determine the number of different countries which appear in file ip-by-country.csv. [1 point]
  3. Based on files countrycodes_en.csv and kodyzemi_cz.csv list the names of states which are the same in English and in Czech. [1 point]

5th practical (21th March, 2018)

Commands: find, xargs.

Exercises:

  1. Write a command which lists all files and directories in the subtree of current directory, whose names contain substring 'bin'.
    (Show solution)
    #!/bin/sh

    find . -name "*bin*" "(" -type d -o -type f ")"
  2. Write a command which lists the number of directories in the subtree of /etc directory.
    (Show solution)
    #!/bin/sh

    find /etc -type d | wc -l
  3. List all symbolic (or soft) links in a directory /usr. List only links directly present in the directory, not deeper in its subtree.
    (Show solution)
    #!/bin/sh

    # The following can be quite slow

    find /usr -type l '!' -path "/usr/*/*"

    # or faster

    find /usr/* -prune -type l

    # or even better

    find /usr \( -path "/usr/*/*" -prune \) -o \( -type l -print \)
  4. List the files with at least three hard links pointing to them in the subtree in directory /usr/bin.
    (Show solution)
    #!/bin/sh

    find /usr/bin -links +2
  5. Find all files in subtree of directory /tmp, which have size bigger than 100kB and which are readable by anyone.
    (Show solution)
    #!/bin/sh

    find /tmp -type f -size +200 -perm -a+r
  6. In the subtree of /usr/include find the files with which are in depth at least 2 and at most 3 in the subtree.
    (Show solution)
    #!/bin/sh

    find /usr/include -path "/usr/include/*/*" ! -path "/usr/include/*/*/*/*"

    # A more effective solution with prune:
    find /usr/include -path "/usr/*/*/*" -prune -o -path "/usr/*/*"
  7. In the subtree of directory /etc find all files which are newer than /etc/passwd.
    (Show solution)
    #!/bin/sh

    find /etc -newer /etc/passwd
  8. In the subtree of directory /bin find all files which are owned by root and which can be executed by the owner and group members, but not by others.
    (Show solution)
    #!/bin/sh

    find /bin -user root -perm -ug+x ! -perm -o+x
  9. Find all files in directory /etc which belong to group stunnel and write long information using ls -l on each of these files.
    (Show solution)
    #!/bin/sh

    find /etc -group stunnel -exec ls -l {} +
  10. Try the difference between echo $PATH; echo "$PATH"; echo '$PATH';
  11. Try the difference between echo *; echo "*"; echo '*';
  12. List the paths contained in variable $PATH so that each of them will be written on a separate line and there will be no separators “:”.
    (Show solution)
    #!/bin/sh

    echo "$PATH" | tr ":" "\n"
  13. Output the long information on all directories contained in variable $PATH (not their contents, just the long info).
    (Show solution)
    #!/bin/sh

    echo "$PATH" | tr ":" "\n" | xargs -I{} ls -ld {}
  14. Create a file with the following three lines:
    a b c
    d e f
    g h i
    
    and try to run the following commands on this file (here named “file”):
    cat file | xargs echo
    cat file | xargs echo -
    cat file | xargs -n 2 echo -
    cat file | xargs -I{} echo "-{}-" "-{}-"
    cat file | xargs -I{} -n 2 echo "-{}-" "-{}-"
    
    Observe what happens when different options are passed to xargs.
  15. Split the file /etc/passwd to parts consisting of five lines each and then reorder these parts in the opposite order (that is assuming /etc/passwd has n lines where n is divisible by five, the lines are now ordered as: n-4, n-3, n-2, n-1, n, n-9, n-8, n-7, n-6, n-5, ..., 1, 2, 3, 4, 5).
    (Show solution)
    #!/bin/sh

    split -l5 /etc/passwd passwdparts
    ls -r passwdparts* | xargs cat >passwd
    rm passwdparts*
  16. Find out which directories stored in variable $PATH contain a program named awk.
    (Show solution)
    #!/bin/sh

    echo "$PATH" | tr ':' '\n' | xargs -I{} find {}/awk -prune 2>/dev/null

Homework:

  1. Find all files in the subtree of /etc which are not newer than /etc/group. [1 point]
  2. Find the files in the subtree of current working directory which have size 0B. [1 point]
  3. Find the files in the subtree of /usr/include the files whose name start with “std” but do not end with “.h”. [1 point]

6th practical (28th March, 2018)

Commands: join, grep, sed.

  1. Write commands which print a list of pairs login:group, where login is the login of a user and group is the name of primary group of user with given login. To produce this list, use join on files /etc/passwd and /etc/group (which are preprocessed as necessary).
    (Show solution)
    #!/bin/sh

    sort -t: -k3,3 /etc/group >skupiny
    sort -t: -k4,4 /etc/passwd | join -1 3 -2 4 -t: -o 2.1,1.1 skupiny -
    rm skupiny
  2. Download files countrycodes_en.csv and kodyzemi_cz.csv. Both of these files are in a CSV format with semicolon (;) used as a separator.
  3. Use files countrycodes_en.csv and kodyzemi_cz.csv to create a list of countries with two columns separated with an equality character (=) where each line has the form of
    Czech name=English name.
    (Show solution)
    #!/bin/sh

    tr -d '"' <kodyzemi_cz.csv | tail -n +2 | sort -t \; -k1,1 >/tmp/kody_s
    tr -d '"' <countrycodes_en.csv | tail -n +2 | sort -t \; -k4,4 >/tmp/codes_s
    join -t\; -1 1 -2 4 -o 1.4,2.1 /tmp/kody_s /tmp/codes_s | tr \; = >translation
    rm /tmp/kody_s /tmp/codes_s
  4. From file calories.csv select lines in which the food name contains a character which is not alphanumeric (i.e. does not belong to [:alnum:]).
    (Show solution)
    #!/bin/sh

    grep '^"[^;]*[^[:alnum:];][^;]*";' calories.csv

    # Případně

    grep -v '^"[[:alnum:]]*";' calories.csv
  5. From file calories.csv select lines in which amount is measured in units “Piece” or “Cake”.
    (Show solution)
    #!/bin/sh

    grep -e '^[^;]*;[[:digit:].]* Piece;' \
         -e '^[^;]*;[[:digit:].]* Cake;' calories.csv

    # Případně

    grep -E '^[^;]*;[[:digit:].]* (Piece|Cake);' calories.csv
  6. Find files in the subtree of /usr/include whose names have length 5-7 characters not including suffix which is required to by “.h”. (Use grep on the output of find command.)
    (Show solution)
    #!/bin/sh

    find /usr/include -name "*.h" | grep '/[^/]\{5,7\}\.h'
  7. Select lines from the output of ls -l in which owner has the same permissions as group members and as others. (Create a file satisfying these condition in say /tmp if necessary.)
    (Show solution)
    #!/bin/sh

    ls -l | grep '^.\(...\)\1\1'
  8. Find files in the subtree of /usr/include which contain a string “printf” in them.
    (Show solution)
    #!/bin/sh

    find /usr/include -exec grep -q printf {} \; -print

    # or

    find /usr/include | xargs grep -l printf

    # or

    find /usr/include -exec grep -l printf {} +
  9. Using sed select only the column with the file size from the output of ls -l.
    (Show solution in sed)
    #!/bin/sh

    ls -l | sed '1d; s/^\([^ ]* *\)\{5\} .*$/\1/'

    (Show solution in ed)
    #!/bin/sh

    # Backslash in front of END prevents shell from performing any
    # expansions in the text before END.

    ls -l > /tmp/lsl.$$
    ed /tmp/lsl.$$<<\END
    2,$g/^/s/^\([^ ]* *\)\{5\} .*$/\1/
    %p
    Q
    END
    rm /tmp/lsl.$$
  10. Using sed transform the lines of /etc/passwd to the form of uid (login).
    (Show solution in sed)
    #!/bin/sh

    sed 's/^\([^:]*\):[^:]*:\([^:]*\).*$/\2 (\1)/' /etc/passwd

    (Show solution in ed)
    #!/bin/sh

    # Backslash in front of END prevents shell from performing any
    # expansions in the text before END.

    # If wish to store the results into the input file,
    # we can add commmands w and q.

    cp /etc/passwd /tmp/passwd.$$
    ed /tmp/passwd.$$<<\END
    g/^/s/^\([^:]*\):[^:]*:\([^:]*\).*$/\2 (\1)/
    %p
    Q
    END
    rm /tmp/passwd.$$

Homework:

  1. Using grep select those lines from countrycodes_en.csv where the first two characters in the three character code (Alpha-3 code, 3rd column) are not the same as the two characters in the two character code (Alpha-2 code, 2nd column). [1 point]
  2. In an HTML file (take this web page for example) titles are enclosed by a pair of <hi>, </hi> tags, where i is a digit 1 to 6. Find all lines with this pair of tags (the digit has to be same in both the opening and closing tag). [1 point]
  3. Using sed replace in the input text file all occurrences of characters '&', '<', '>' with their HTML entities (i.e. replace '&' with '&amp;', '<' with '&lt;', and '>' with '&gt;'). Call sed just once. [1 point]

7th practical (4th April, 2018)

Commands: sed, nl.

  1. Using sed put character # to the beginning of lines 10 to 20 in file /etc/passwd.
    (Show solution in sed)
    #!/bin/sh

    sed '10,20s/^/#/' /etc/passwd

    (Show solution in ed)
    #!/bin/sh

    ed /etc/passwd <<\END
    10,20s/^/#/
    %p
    Q
    END
  2. Using sed put character “o” to the beginning of each odd line in file /etc/passwd and “e” to the beginning of each even line in this file.
    (Show solution in sed)
    #!/usr/bin/sed -f
    # This is a sed script, use with sed -f
    # Usage: sed -f <script> /etc/passwd

    s/^/o/
    n
    s/^/e/

    (Show solution in ed)
    #!/bin/sh

    # The modified file is not written (it would not be a good idea in
    # case of /etc/passwd).

    ed /etc/passwd <<\END
    g/./s/^/o/\
    +1s/^/e/
    %p
    Q
    END
  3. Using sed insert an empty line between each two lines of /etc/passwd.
    (Show solution in sed)
    #!/bin/sh

    # The $b command ensures that nothing will be added after the last
    # line.
    # Command a performs the actual empty line append.

    sed '$b; a\
    '
    /etc/passwd

    #  Equivalently we can do it in the following way with i (the
    addresses ensure that nothing is added before the first line):

    sed '2,$i\
    '
    /etc/passwd

    (Show solution in ed)
    #!/bin/sh

    # Backslash in front of END prevents shell from performing any
    # expansions in the text before END.
    #
    # The backslash after i command is there because it is a command
    # within global g command, otherwise there would be no backslash after
    # i.

    ed /etc/passwd <<\END
    2,$g/.*/i\
    \
    .
    %p
    Q
    END
  4. Using sed reverse the order of lines in the input.
    (Show solution in sed)
    #!/bin/sh

    # We shall try it with nl /etc/passwd to see the difference.
    nl /etc/passwd | sed -n 'G; $p; h'

    (Show solution in ed)
    #!/bin/sh

    # Backslash in front of END prevents shell from performing any
    # expansions in the text before END.

    # The backslash after i command is needed because it is part of g
    # command.

    ed /etc/passwd <<\END
    g/^/m0
    %p
    Q
    END
  5. Using sed print only even lines from file /etc/passwd.
    (Show solution in sed)
    sed -n 'n; p' /etc/passwd

    (Show solution in ed)
    #!/bin/sh

    # Observe the backslash before string END which prevents shell
    # expansion in the text.

    # Command g performs the following commands on each line.
    #    1) +1p prints the next line and
    #    2) .s/^// unmarks (the next) line so that g skips it.
    # the last line is omitted (we assume that the file contains at least
    # two lines), because for the last line +1 would be behind the end of
    # file.
    #
    # If we would use s///, then instead of an empty regexp g would fill
    # in its own regexp and even lines would be deleted. If the changes
    # are not about to be saved, this is not an issue.

    nl /etc/passwd > /tmp/passwd.$$
    ed /tmp/passwd<<\END
    1,$-1g/^/+1p\
    .s/^//
    Q
    END
    rm /tmp/passwd.$$
  6. Using sed insert
    #!/bin/sh
    as the first line of a given file.
    (Show solution in sed)
    #!/usr/bin/sed -f
    # This is a sed script, use with sed -f

    1i\
    #!/bin/sh

    (Show solution in ed)
    #!/bin/sh

    ed inputfile<<\END
    1i
    #!/bin/sh
    .
    w
    q
    END
  7. Write a sed script which removes comments from a shell script, in particular:
    1. If the first line has a special form of ^#!.*, then it should not be removed.
    2. Lines which start with # (or have only white characters before the first #) should be deleted from the output.
    3. Lines which contain some other characters before # should be modified to keep only the part before the # character.
    4. For simplicity ignore the fact that in a shell script the # character can occur escaped, in quotation marks, apostrophes or in other contexts. Assume that each # in the input indeed starts a comment.
    (Show solution in sed)
    #!/usr/bin/sed -f
    # This is a sed script, use with sed -f

    # Skip the first line if it has a special form of #!
    1{
    /^#!/n
    }

    # Delete lines starting with #
    /^[[:space:]]*#/d

    # Deletes comments from the remaining lines (ignores the fact that #
    # can be a part of a string in quotation marks, apostrophes etC.).
    s/#.*$//

    (Show solution in ed)
    #!/bin/sh

    # We first deal with all lines except the first one.
    # - The first command deletes the lines having nothing before the
    #   comment # character.
    # - The second command deletes everything from # to the end of line
    #   from each of remaining lines.
    # Then we resolve the first line.
    # - We first delete the first line if it contains a comment having
    #   nothing before # and does not have ! after #.
    # - We delete the first line if it contains only # and nothing else.
    # - Finally we remove everything from # to the end if there is not #!
    # at the beginning of line.

    ed inputfile.sh <<\END
    2,$g/^[[:space:]]*#/d
    2,$g/#/s/#.*//
    1g/^[[:space:]]*#[^!]/d
    1g/^[[:space:]]*#$/d
    1v/^#!/s/#.*$//
    w
    q
    END

Extra assignments (without solutions)

  1. Write a sed script which replaces all C style comments (i.e. /* */) to C++ style comments (i.e. //). Be aware of the following facts:
    • C style comments can be on a single line, but they can be multi-line as well.
    • If there is something after a C style comment, you have to move it to the next line.
    • There can be multiple C style comments on a single line.
    You may assume that the input file does not contain other occurrences of /* and */ except those which delimit a comment, that the comments are not nested and that the comment delimiters are well paired. For example
    aaa /* bbb */ ccc /* ddd */ eee /* fff
    ggg */ hhh /* iii */ jjj
    kkk
    /* lll
       mmm
       nnn */
    ooo
    
    would be turned into
    aaa // bbb
    ccc // ddd
    eee // fff
    // ggg
    hhh // iii
    jjj
    kkk
    // lll
    // mmm
    // nnn
    ooo
    
    The output of your script may include additional spaces (or newlines), but the output should be valid.
  2. We define a number in a text file as a string which consists of digits 0-9 with possible leading sign + or - which is delimited with a blank character (a space, tab, begining of line, end of line, etc.). I.e. line
    12 -31 aaaa b01 001 a-13
    
    contains three numbers 12, -31 and 1 (leading zeros are ignored). Write shell commands which count the number of pairwise different numbers in a given file in absolute value (i.e. leading zeros are ignored, signs are ignored, in particular -31, 31, 0031, +0000031 all count as the same number in absolute value 31). Motivation: A CNF file such as formula.cnf consists of a header line and a list of clauses where numbers denote number of variables in a conjuctive normal form (CNF) formula. The commands from exercise should count the number of different variables in the formula. Exercise is a little bit more general.
  3. Write a sed script which reads a decimal number n from the input and writes sequence 0, 1, ..., n to the output. Each number is on a separate line.
  4. Write a sed script which outputs the last ten lines of the input.

Homework

  1. Assume that the input lines contain only characters '(' and ')', write a sed script which checks whether each input line contains string of well paired brackets. The correct lines are replaced with 'ok' string, the wrong lines with 'wrong' string. [1 points]
  2. Consider a text file in which paragraphs are separated with an empty line. Write a script for sed which joins the lines of each paragraph to a single line (with space as a separator). [1 points]
  3. Write a sed script which selects the middle character from each line. If the number of characters on a line is odd, it is replaced with the middle character. If the number of characters on a line is even, the line is replaced with one of the two middle characters. [1 point]

8th practical (11th April, 2018)

Command: ed.

  1. Using ed output the last ten lines of /etc/passwd.
    (Show solution)
    #!/bin/sh

    ed /etc/passwd <<\END
    $-9,$p
    Q
    END
  2. Solve the following exercises also in ed: 9 and 10 in 6th practical, 1 to 7 in 7th practical. For solutions see each of these exercises.
  3. Run vimtutor

Extra assignments (without solutions)

  1. Assume the input file contains words separated with spaces where each word consists only of characters in English alphabet ([:alpha:]) Write a sed script which reverses the order of words on each line.

Homework

  1. Write a script for ed which moves all lines starting with character '#' to the end of the input file. [1 point]
  2. File morse.csv contains a translation of English alphabet into the Morse code. Write a script for sed or ed which translates this file into another file translate.sed which is a sed script translating an English text into the Morse code. In other words, the input to your script is the file morse.csv. The output of your script is the file translate.sed. We can use translate.sed as a script for sed to translate an English text into the Morse code as follows:
    sed -f translate.sed input >output
    You can expect that the text to be translated is written only using letters of English alphabet (lower or upper case) and words are separated with spaces. When translated into the Morse code, the characters are separated with '/' (a slash) and words are separated with '//' (two slashes). [2 points]

9th practical (18th April, 2018)

Commands: read, control structures (for, while, if, case, etc.), printf, expr.

Other: Variables and expansion, redirection, positional and special parameters/variables ($#, $0, $n, ${n}, shift, "$@", set -, $?, $$, $!), command combination using &&, command grouping and subshell invokation, back apostrophes and $() expansion.

  1. Write a shell script which expects two parameters, a file name and a number. The script deletes the line with given number from given file.
    (Show solution)
    #!/bin/sh
     ed "$1" <<END
     ${2}d
    w
    q
    END
  2. Write a script which expects n+1 parameters, the first parameter contains a number x between 1 and n, the script then outputs the (x+1)st parameter. For instance script 2 a b c outputs b, i.e. the third parameter (or the second if we count starting from a. (Use shift with a parameter.)
    (Show solution)
    #!/bin/sh

    shift "$1"
    echo "$1"
  3. Write script reverse which writes its parameters to standard output in reversed order. For example reverse a b c d outputs d c b a.
    (Show solution)
    #!/bin/sh

    for x
    do
       y="$x $y"
    done
    echo "$y"
  4. Write a script max which selects the maximum of its numeric parameters. For example max 1 15 7 19 11 prints 19.
    (Show solution)
    #!/bin/sh

    # Without test, using sort
    for x
    do
       echo "$x"
    done | sort -nr | head -n 1

    # Using sort, now we assume that the parameters are numbers without spaces:
    echo "$@" | tr " " '\n' | sort -nr | head -n 1

    # Using test
    m="$1"
    shift
    for x
    do
       if [ "$x" -gt "$m" ]
       then
          m="$x"
       fi
    done
    echo "$m"

    # Using shell arithmetic expansion
    m="$1"
    shift
    for x
    do
       m=$((x>m ? x : m))
    done
    echo "$m"
  5. Write a script which expects one or two number parameters. The script then pads the first number to number of digits given in the second parameter. If the second parameter is not specified, it is considered to be 5. (Use printf and expansion ${variable:-word}.)
    (Show solution)
    #!/bin/sh

    printf "%${2:-5}d\n" $1
  6. What the following lines write to the screen?
    a="hello"; { echo $a ; a="hi" ; echo $a ; } ; echo $a
    b="hello"; ( echo $b ; b="hi" ; echo $b ; ) ; echo $b
    x="hello"; x="hi" sh -c 'echo $x'; echo $x
    y="hello"; y="hi"; sh -c 'echo $y'; echo $y
    z="hello"; sh -c 'echo $z'
    export u="hello"; sh -c 'echo $u'
  7. What is the following command doing?
    a=1;b=2; { a=LEFT; echo $a >&2; } |\
    { b=RIGHT; echo $b; tr 'A-Z' 'a-z';}; echo "A=$a,B=$b"
    And how is it different from:
    a=1;b=2; { a=LEFT; echo $a; } |\
    { b=RIGHT; echo $b; tr 'A-Z' 'a-z';}; echo "A=$a,B=$b"

Extra assignments (without solutions)

Extra assignments can use any commands

  1. Assume the input file is in form of a csv file where columns are separated with a comma. Write a shell script which accepts two parameters, a file and a number of a column. The script then outputs a median of the numeric values in the given column within the given file. You can also enhance the script to compute average and standard deviation of the numeric values in the column.
  2. We define Fibonacci words as follows: Word \(S_0=0\), word \(S_1=01\), then word \(S_n=S_{n-1}S_{n-2}\) (i.e. concatenation of words \(S_{n-1}\) and \(S_{n-2}\)). Write a shell script which given a number \(n\) writes \(n\)-th Fibonacci word. Allow two optional parameters which specify characters to be used instead of 0 and 1.
  3. Write a shell script which reads a file with a number on each line and sorts it. Each number from the input appears only once in the output, but the script computes the number of occurrences of each number within the input file and appends this frequency to each number as another column of the output file.

Homework

  1. Write a script which works as cp but with reversed order of parameters (the target is the first parameter, note that cp accepts more than two parameters). [1 point]
  2. Write a script which expects three parameters, a file name, a number n and a character c. The script then performs a circular shift of each input line (input is in the file specified in the first parameter). The lines are considered to be split into fields separated with character c and the number of elements to shift is given in number n. For example if the script is called like
    script /etc/passwd 2 :
    then each line of /etc/passwd is transformed from the initial format
    login:*:uid:gid:full name:home dir:shell
    to the following form:
    uid:gid:full name:home dir:shell:login:*
    The modified contents of the input file is written to the standard output, the file itself is left untouched. [2 points]

10th practical

Commands: read, expr, dirname, basename.

  1. Write a shell script which expects file paths in its parameters. The paths in parameters can be relative. For each path in parameter the script outputs the full path of the file.
  2. Will the following script correctly compute the number of lines in /etc/passwd?
    count=0;
    cat /etc/passwd | while read x
    do
       count=`expr $count + 1`
    done
    echo $count
    And what about the following script?
    cat /etc/passwd | {
       count=0
       while read x
       do
          count=`expr $count + 1`
       done
       echo $count
    }
    And what about the following script?
    count=0
    while read x
    do
       count=`expr $count + 1`
    done </etc/passwd
    echo $count
    And finally, what about the following script?
    count=0
    while read x </etc/passwd
    do
       count=`expr $count + 1`
    done
    echo $count
    What is the difference between all these scripts?
  3. Write a shell script which renames all files in the current directory to files with the same name but where the upper case characters are changed into lower case. (E.g. file BACKUP.ZIP would be renamed to backup.zip.)
  4. Write a shell script which renames all files in the current directory with suffix “.jpeg” to the files with the same name but with their suffix changed to “.jpg”.