# Lab 6: Processes and signals

## Exercise 1: Managing processes

### Task 1: Process id and jobs

- We will start a few processes and manage them through the command line. Open a command shell and change directory to your home. Start the top command and put it into the background. Use `&` to put the process in the background

      $ top &

- Then start a background process called `yes` and redirect its out to `/dev/null` (the bit bucket).

      $ yes > /dev/null &

- Now let’s start an `md5sum` process to calculate the md5 hash of the first drive on the system. Notice how this hangs the prompt; it should take a long time to complete this task.

      $ md5sum /dev/sda

- Let’s stop the process and run it to the background. To stop the process push `CTRL+Z`.
- Now restart the job in the background. To see the job numbers, use the following. You can see that `top` process also was stopped in the background

      $ jobs

> Starting full screen programs like “nano” or “top” in background are immediately stopped (by a SIGTTIN signal - when they try to read from the tty, or by a SIGTTOU signal when they try to change the tty parameters). Many programs handle that badly.
> About tty and signals: http://www.linusakesson.net/programming/tty/index.php

- Identify `id` number of the job `md5sum` from the previous command and run it with `bg` command

      $ bg 3

- Now list the current jobs running and stopped, see the changes

      $ jobs

- We can also bring specific jobs to the terminal screen. Afterwards you can terminate the process by `CTRL+C`

      $ fg 3

> `CTRL+C` – sends to a process by its controlling terminal (by the TTY driver) SIGINT signal to the current foreground job.

- To list the process IDs of the current processes running in the current shell

      $ ps

- The fundamental way of controlling processes in Linux is by sending signals to them. There are multiple signals that you can send to a process. To view all the signals, run:

      $ kill -l

- Identify the process ID for the `yes` process, in this example its ID is `27522`. To kill this process with a SIGTERM (-15)

      $ kill 27522

- If that failed, you can use a SIGKILL (-9)

      $ kill -9 27522

- To list all process running on the system, issue the following command

      $ ps -ef

- To find the process ID of a specific process named `bash`

      $ ps -ef | grep bash

- Another useful command is the pstree command which shows a tree structure of the cascading process IDs (-p).

      $ pstree -p

- When you press the `CTRL+C` or Break key at your terminal during execution of a shell program, normally that program is immediately terminated, and your command prompt returns. This may not always be desirable. For instance, you may end up leaving a bunch of temporary files that won’t get cleaned up.
- Trapping these signals is quite easy, and the trap command has the following syntax:

      trap "commands" signals

  Here command can be any valid Unix command, or even a user-defined function, and signal can be a list of any number of signals you want to trap.
  There are two common uses for trap in shell scripts:
  - Clean up temporary files
  - Ignore signals
- Let’s create a script with a trap `SIGINT`. Save the script as `sleeper.sh`

      #!/bin/bash

      trap "echo SIGINT encountered, Goodbye forever!" SIGINT
      echo Hello, I am now going to sleep
      sleep infinity

  > The command to execute when the trap is encountered must be in quotes.

- Now run `sleeper.sh`

      $ bash sleeper.sh
      Hello, I am now going to sleep

- Send a `SIGINT` by pressing `CTRL+C` on the keyboard. You should have the following output:

      $ bash sleeper.sh
      Hello, I am now going to sleep
      ^CSIGINT encountered, Goodbye forever!

  > Remember, you can also find the process ID and then use `kill` to send the signal in the form `$ kill -signal pid`

- You can also use trap to ensure the user cannot interrupt the script execution. This feature is important when executing sensitive commands whose interruption may permanently damage the system. The syntax for disabling a signal is:

      trap "command" [signal]

  Double quotation marks mean that no command will be executed when the signal is received. For example, to trap the SIGINT and SIGABRT signals, type:

      trap "" SIGINT SIGABRT

### Task 2: The proc file system

The `/proc/` directory — also called the proc file system — contains a hierarchy of special files which represent the current state of the kernel — allowing applications and users to peer into the kernel’s view of the system.
Within the `/proc/` directory, one can find a wealth of information detailing the system hardware and any processes currently running. In addition, some of the files within the `/proc/` directory tree can be manipulated by users and applications to communicate configuration changes to the kernel.

- You can view the `/proc/` virtual files with the command line file readers. For example, view `/proc/cpuinfo`

      $ cat /proc/cpuinfo

  You should receive output similar to the following:

      processor	: 0
      vendor_id	: AuthenticAMD
      cpu family	: 25
      model		: 80
      model name	: AMD Ryzen 5 5600H with Radeon Graphics
      stepping	: 0
      microcode	: 0xffffffff
      cpu MHz		: 3293.695
      cache size	: 512 KB
      physical id	: 0
      siblings	: 2
      core id		: 0
      cpu cores	: 2
      apicid		: 0
      initial apicid	: 0
      fpu		: yes
      fpu_exception	: yes
      cpuid level	: 13
      wp		: yes

  When viewing different virtual files in the `/proc/` file system, some of the information are easily understandable while some are not human-readable. This is in part why utilities exist to pull data from virtual files and display it in a useful way. Examples of these utilities include `lspci`, `apm`, `free`, and `top`.

- Most virtual files within the `/proc/` directory are read-only. However, some can be used to adjust settings in the kernel. This is especially true for files in the `/proc/sys/` subdirectory.

- To change the value of a virtual file, use the `echo` command and redirect (>) the new value to the file. For example, to change the hostname on the fly, type:

      echo SNALabPC > /proc/sys/kernel/hostname

- Other files act as binary or Boolean switches. Typing `$ cat /proc/sys/net/ipv4/ip_forward` returns either a 0 or a 1. A `0` indicates that the kernel is not forwarding network packets. Using the echo command to change the value of the `ip_forward` file to `1` immediately turns packet forwarding on.

- On multi-user systems, it is often useful to secure the process directories stored in `/proc/` so that they can be viewed only by the `root` user. You can restrict the access to these directories with the use of the `hidepid` option.

- To change the file system parameters, you can use the `mount` command with the `-o` remount option.

      $ sudo mount -o remount,hidepid=value /proc

  Here, value passed to hidepid is one of:

  - `0` (default) — every user can read all world-readable files stored in a process directory.
  - `1` — users can access only their own process directories. This protects the sensitive files like cmdline, sched, or status from access by non-root users. This setting does not affect the actual file permissions.
  - `2` — process files are invisible to non-root users. The existence of a process can be learned by other means, but its effective UID and GID is hidden. Hiding these IDs complicates an intruder’s task of gathering information about running processes.

- To make process files accessible only to the root user, type:

      $ sudo mount -o remount,hidepid=1 /proc

  With `hidepid=1`, a non-root user cannot access the contents of process directories. An attempt to do so fails with the following message:

      $ ls /proc/1/
      ls: /proc/1/: Operation not permitted

  With hidepid=2 enabled, process directories are made invisible to non-root users:

      $ ls /proc/1/
      ls: /proc/1/: No such file or directory

- Also, you can specify a user group that will have access to process files even when `hidepid` is set to 1 or 2. To do this, use the gid option.

      $ sudo mount -o remount,hidepid=value,gid=gid /proc

  > You can find system groups and their respective group IDs in `/etc/group`
  > Replace `gid` with the specific group id. For members of selected group, the process files will act as if `hidepid` was set to `0`. However, users which are not supposed to monitor the tasks in the whole system should not be added to the group.

### Task 3: `top`

- Open a command shell run the `top` command

      $ top

> This opens up a tool that shows the top processes running on your system. This tool can be used to kill processes, renice processes, sort and various other process management. Press the h command to get a list of help.
> Read material:
> https://www.guru99.com/managing-processes-in-linux.html

- By default, top sorts the process list using the %CPU column. To sort processes using a different column, press one of the following keys.

  - `M` Sort by the `%MEM` column.
  - `N` Sort by `PID` column.
  - `T` Sort by the `TIME+` column.
  - `P` Sort by the `%CPU` column.

- To show the process command line instead of just the process name, press `c`.

- The filter feature allows using a filter expression to limit which processes to see in the list. Activate the filter option by pressing `o`. The program prompts you to enter a filter expression. You can enter the following to filter processes using more than 1% CPU.

      %CPU>1.0

- Clear the filters by pressing `=`

- To filter processes by a specific user, specify the `-u` option when you run the top command

      $ top -u root

- The first five lines of the output show some useful statistics

  - `top` displays uptime information
  - `Tasks` displays process status information
  - `%Cpu(s)` displays various processor values
  - `MiB Mem` displays physical memory utilization
  - `MiB Swap` displays virtual memory utilization

Uptime
Top’s first line, `top`, shows the same information as the uptime command. The first value is the system time. The second value represents how long the system has been up and running, while the third value indicates the current number of users on the system. The final values are the load average for the system.

The load average is broken down into three time increments. The first shows the load for the last one minute, the second for the last five minutes, and the final value for the last 15 minutes. The results are a percentage of CPU load between 0 and 1.0. The processor is likely overworked if 1.0 (or higher) is displayed.

    top - 00:49:59 up 1 day, 12:12,  3 users,  load average: 0,63, 0,66, 0,64

Tasks
The second line is the `Tasks` output, and it’s broken down into five states. These five states display the status of processes on the system:

- `total` shows the sum of the processes from any state.
- `running` shows how many processes are handling requests, executing normally, and have CPU access.
- `sleeping` indicates processes awaiting resources, which is a normal state.
- `stopped` reports processes exiting and releasing resources; these send a termination message to the parent process.
- `zombie` refers to a process waiting for its parent process to release it; it may become orphaned if the parent exits first.
  Zombie processes usually mean an application or service didn’t exit gracefully. A few zombie processes on a long-running system are not usually a problem.

    Tasks: 386 total,   1 running, 384 sleeping,   1 stopped,   0 zombie

%Cpu(s)
Values related to processor utilization are displayed on the third line. They provide insight into exactly what the CPUs are doing.

- `us` is the percent of time spent running user processes.
- `sy` is the percent of time spent running the kernel.
- `ni` is the percent of time spent running processes with manually configured nice values.
- `id` is the percent of time idle (if high, CPU may be overworked).
- `wa` is the percent of wait time (if high, CPU is waiting for I/O access).
- `hi` is the percent of time managing hardware interrupts.
- `si` is the percent of time managing software interrupts.
- `st` is the percent of virtual CPU time waiting for access to physical CPU.
  Values such as `id`, `wa`, and `st` help identify whether the system is overworked.

    %Cpu(s):  4,1 us,  0,4 sy,  0,0 ni, 95,3 id,  0,0 wa,  0,0 hi,  0,1 si,  0,0 st

MiB Memory
The final two lines of top’s output provide information on memory utilization. The first line—`MiB Mem`—displays physical memory utilization. This value is based on the total amount of physical RAM installed on the system.

    MiB Mem :  15967,8 total,    260,9 free,   2749,7 used,  12957,2 buff/cache

> Note: The term mebibyte (and similar units, such as kibibytes and gibibytes) differs slightly from measurements such as megabytes. Mebibytes are based on 1024 units, and megabytes are based on 1000 units (decimal). Most users are familiar with the decimal measurement, but it is not as accurate as the binary form. The top utility reports memory consumption in decimal.

> - `total` shows total installed memory.
> - `free` shows available memory.
> - `used` shows consumed memory.
> - `buff/cache` shows the amount of information buffered to be written.

MiB Swap
Linux can take advantage of virtual memory when physical memory space is consumed by borrowing storage space from storage disks. The process of swapping data back and forth between physical RAM and storage drives is time-consuming and uses system resources, so it’s best to minimize the use of virtual memory.

    MiB Swap:   2048,0 total,   2047,5 free,      0,5 used.  12739,8 avail Mem

- `total` shows total swap space.
- `free` shows available swap space.
- `used` shows consumed swap space.
- `buff/cache` shows the amount of information cached for future reads.

In general, a high amount of swap utilization indicates the system does not have enough memory installed for its tasks. The solution is to either increase RAM or decrease the workload.

### Task 4: `free`

`free` is a popular command used by system administrators on Unix/Linux platforms. It’s a powerful tool that gives insight into the memory usage in human-readable format.
The man page for this command states that free displays the total amount of free and used memory on the system, including physical and swap space, as well as the buffers and caches used by the kernel. The information is gathered by parsing `/proc/meminfo`.

- Run `free` with the `-h` option for human-readable output

      free -h
                     total        used        free      shared  buff/cache   available
      Mem:            15Gi       2,7Gi       265Mi       149Mi        12Gi        12Gi
      Swap:          2,0Gi       0,0Ki       2,0Gi

- `free` provides options to display amount of memory in various units. free `-b`, `-k`, `-m`, `-g` display the amount of memory in bytes, kilobytes, megabytes, gigabytes respectively.
- The various columns, displayed by the various releases above, seek to identify the Total, used, free, shared memory. It also seeks to display the memory held in cache and buffers as well.

## Questions to answer

1. What are zombie processes? How can you find and kill them?
2. What are the differences between `kill`, `killall`, and `pkill`?
3. Run the `top` command on your system and annotate the data in the `Tasks` and `%Cpu(s)` lines of your output. Provide single sentence explanations for each of the data presented in these two lines.
4. Execute the following bash command:

       $ bash -c "exec -a fun${RANDOM}process sleep infinity" &

   - Assume that there are multiple of such processes. To simulate this, you can run the command more than once.
   - Write a bash script that will locate and kill all the processes created by this command.
   - Display status messages when one of such processes is found, and when the process is killed. Additionally, display a message when the process is not found.
   - Your script should work on any machine it is executed on.
   - Be extremely careful and be as accurate as possible when finding this process. You don’t want to kill the wrong process.

   > Show test results in the form of screenshots.

5. Write a bash script that loops infinitely and prints “Hello world!” every ten seconds. It should print “Interrupt received” when it receives `SIGUSR1`.
   - Show the script in your report, and show how you’re sending the signal to it.
6. Write a bash script to monitor CPU usage, memory usage, and disk space usage.
   - For testing purposes, the check should execute every 15 seconds.
   - The usage statistics should be saved to a log file `/var/log/system_utilization.log`.
   - One line of log should contain the timestamp, the % of CPU in use, the % of memory in use, and the % of disk space used.
   - The log should contain descriptive information that will make it easy to understand.

   > Show log samples created by this script in your report.