# Lab 10: Logging and auditing

## Task 1: journald

- Journald rate limits log messages and will drop all messages from a service if it passes certain limits. These limits can be configured via `RateLimitBurst` and `RateLimitIntervalSec`, which default to 10000 and 30s respectively.

      cat /etc/systemd/journald.conf | grep RateLimit

  The rate limiting feature is very handy when you have services that generate a lot of logs.
- `journalctl` is the main tool for interacting with the journal. Run this tool

      $ journalctl

- View all the options you can use with journalctl.

      man journalctl

- Show only the last 10 lines of the journal with the `-n` option.

      $ journalctl -n 10

- Follow the logs with the `-f` option.

      $ journalctl -f

- View the journal in reverse order. From the newest to the oldest.

      $ journalctl --reverse

- The `--output` in journalctl can be used to format the output of the journal in various forms.
- View the journal in json format with the option `--output=json-pretty` or just `--output=json` if you want it compact.

      $ journalctl --output=json-pretty

- Filter journal for specific systemd unit with the command `journalctl -u <systemd-unit>`.

      $ journalctl -u multiuser.target

- You can also filter with the `-p` option to specify event severity levels that should be displayed.
- Show only kernel messages.

      $ journalctl --dmesg

- You can filter by time with the `--since` and `--until` options.

      $ journalctl --since="2022-10-30 23:00:00"

      OR

      $ journalctl --since=yesterday --until=now

- You can view the time that the system was reboot in the journal

      $ journalctl MESSAGE="Server listening on 0.0.0.0 port 22."

## Task 2: rsyslog

- Use the `logger` utility to generate logs.

      $ logger Test

- View the log in `/varlog/syslog`

      $ tail /var/log/syslog

  You should see a line of log similar to the following.

      Oct 30 02:06:04 sna-vm root: Test

Rsyslog rules typically have the facility and the level. These two combined in the form `facility.level` define the priority of the log message.

> More about facility and level: https://success.trendmicro.com/dcx/s/solution/TP000086250

- Create an rsyslog rule that stores all logs with `mail.emerg` priority to a log file `/var/log/test.log`.
- Do this by adding the following rule to `/etc/rsyslog.conf`.

      user.emerg /var/log/test.log

- Restart rsyslog to apply the changes.

      $ systemctl restart rsyslog

- Open a new terminal in another tab where you will run the `logger` utility to test this rule and execute the following command.

      $ logger -p user.emerg "test log"

- Go to the previous terminal tab, you should see a message similar to the following:
  By default, there is a broadcast message everytime an emergency log is received.
- View the log file where the rule we created will save logs of this priority to.

      $ tail /var/log/test.log

  You should get output similar to the following:

      Oct 30 03:24:34 sna-vm user1: test log

### Logging rsyslog to a remote server

- The rsyslog configuration file is at `/etc/rsyslog.conf`.
- Open the configuration file and add the following information. `IP` is the IP address of the remote log server.

      *.* @IP

  > If you’re using TCP, the syntax is `*.* @@IP`.
  > `*.*` means we are forwarding logs of all facility, and all level to the remote server. Essentially forwarding everything.
  > A remote syslog server IP address will be provided in the lab for testing.

- Restart the `rsyslog` service to apply the changes.

      $ systemctl restart rsyslog

## Task 3: Logrotate

We have a log file `/var/log/lab10.log` that we need to rotate periodically.

- List all configurations created for logrotate to apply on various log files.

      $ ls -lah /etc/logrotate.d/

- Let’s create a custom log rotate configuration. First create the directory:

      $ mkdir /etc/lab10-rotate.d

- Create the file `/etc/lab10-rotate.d/lab10log`:

      /var/log/lab10.log {
             su root syslog
             rotate 1
             maxsize 1M
             compress
             delaycompress
         }

- Change the file permissions:

      $ sudo chmod 644 /etc/lab10-rotate.d/lab10log

- Include this new configuration to the `logrotate` configuration file.

      $ cat << EOF | sudo tee /etc/lab10-rotate.conf
      # Log rotation information for lab10log
      include /etc/lab10-rotate.d
      EOF

- Change the file permissions:

      $ sudo chmod 644 /etc/lab10-rotate.conf

- Create a cron job that will run `logrotate`. Edit crontab as a root user:

      $ sudo su
      $ crontab -e

  Add the following job to cron

      #run command at every 2nd minute
      */2 * * * * logrotate /etc/lab10-rotate.d/lab10log

> You can test and troubleshoot your logrotate configuration by running `$ logrotate -s logstatus /etc/lab10-rotate.d/lab10log`.
> If no message is displayed on the terminal after running the command, it means that the configuration is good.
> You can look in the target log file directory to verify that the log has been rotated.

- If you are rotating a running service, it is good practice to restart the service in the logrotate configuration. The format is shown below:

      postrotate
          <command to restart the service>
      endscript

## Task 4: User authentication activities

- System login activity are saved in the log file `/var/log/auth.log`. You can see user sessions opened and other events such as authentication failure.
- Let’s create a new user `testuser` which we will use to simulate failed login.

      $ sudo adduser testuser

- Try switching to the `testuser` with the `su` utility. Enter the wrong password on the prompt.

      $ su testuser
      Password:
      su: Authentication failure

- Check the log file `/var/log/auth.log` to view this failed login activity.

      $ tail /var/log/auth.log

  You should see lines of log similar to the following two. These are the failed login events.

      Oct 30 20:03:07 sna-vm su: pam_unix(su:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=user1 rhost=  user=testuser
      Oct 30 20:03:09 sna-vm su: FAILED SU (to testuser) user1 on pts/0

- Try switching to the `root` user. This time, enter three consequtive wrong passwords in the prompt.

      $ sudo su
      [sudo] password for user1:
      Sorry, try again.
      [sudo] password for user1:
      Sorry, try again.
      [sudo] password for user1:
      sudo: 3 incorrect password attempts

- View the authentication log file again.

      $ tail /var/log/auth.log

  You should see the following lines in the output.

      Oct 30 20:09:05 sna-vm sudo: pam_unix(sudo:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/1 ruser=user1 rhost=  user=user1
      Oct 30 20:09:13 sna-vm sudo:    user1 : 3 incorrect password attempts ; TTY=pts/1 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/su

- The audit log file contains several other events. We can filter for failed log in attempts with the text filtering editors.

      $ cat /var/log/auth.log | grep -P "(?i)authentication failure|incorrect password"

- `journalctl` can also display such logs. Let’s run journalctl and filter for authentication failure.

      $ journalctl | grep -P "(?i)authentication failure|incorrect password"

- Create a script that automatically extracts these failed authentication logs and save them to another file. This makes it easier to quickly detect security violations.
  Create the script `track_auth_fail.sh`

      $ vim track_auth_fail.sh

  Add the following content to the script:

      #!/bin/bash

      tail -n0 -f /var/log/auth.log | \
      grep -P --line-buffered "authentication failure|incorrect password" |\
      tee /var/log/failed_auth.log

- Run the script in the background and suppress it’s output.

      sudo bash track_auth_fail.sh > /dev/null 2>&1 &

- Simulate a failed login for `testuser`.

      $ su testuser
      Password:
      su: Authentication failure

- Check the new log file with `$ tail /var/log/failed_auth.log` and you should see log similar to the following:

      Oct 30 21:15:37 sna-vm su: pam_unix(su:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=user1 rhost=  user=testuser

## Task 5: User sessions and sudo usage

- We can run commands as other users from our login session. When we do this, a new user session is opened. Run a command as `testuser`.

      $ su - testuser -c "whoami"
      Password:
      testuser

- Now check the auth log file to see if this activity is logged.

      $ tail  /var/log/auth.log

  You should get output similar to the following:

      Oct 30 21:55:49 sna-vm su: (to testuser) user1 on pts/1
      Oct 30 21:55:49 sna-vm su: pam_unix(su-l:session): session opened for user testuser(uid=1001) by (uid=1000)
      Oct 30 21:55:50 sna-vm su: pam_unix(su-l:session): session closed for user testuser

  - The first line of log above shows that `user1` first attempted to use the `su` utility to switch to `testuser`.
  - The second line shows that the session was opened.
  - The third line shows that the session was closed. This happened after the command finished executing.
- Sudo usage logs are also saved in `/var/log/auth.log`. When a user runs `sudo`, they open a sudo session.
- Filter the log file for `sudo`. Run the following command:

      $ sudo ip a

- View the log file

      $ tail  /var/log/auth.log

  You should get output siilar to the following:

      Oct 30 22:03:20 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
      Oct 30 22:03:20 sna-vm sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
      Oct 30 22:03:20 sna-vm sudo: pam_unix(sudo:session): session closed for user root

  When a user attempts to use sudo to execute a command as `root`, that command is logged to the auth file as seen in the first line of the output above.
- Filter the log to view all attempts made to execute commands as `root`:

      $ cat /var/log/auth.log | grep -P "USER=root.*COMMAND="

  You should get output similar to the following:

      Oct 30 21:15:23 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/bash track_auth_fail.sh
      Oct 30 21:47:54 sna-vm sudo:    user1 : TTY=pts/1 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
      Oct 30 22:03:20 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
      Oct 30 22:07:25 sna-vm sudo:    user1 : 3 incorrect password attempts ; TTY=pts/4 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a

## Questions to answer

1. What security monitoring tool will you forward your system logs to for security event detection? Give reasons for your choice.
2. Configure rsyslogd by adding a rule to the newly created configuration file `/etc/rsyslog.d/auth-errors.conf` to log all security and authentication messages with the priority alert and higher to the `/var/log/auth-errors` file. Test the newly added log directive with the logger command. Verify it from rsyslog and journald perspectives by filtering the output.
3. Install Apache web server and configure log rotate to rotate its web access log every six hours. Compress the rotated log files, and ensure that log rotate restarts the web server after rotating the logs. Manually execute the logrotate utility to test your configuration and show results.
4. Create a bash script that continuously monitors the `/var/log/auth.log` file and triggers an alarm if there are three or more “authentication failure” in 30 seconds. The text `Three or more authentication failure in 30 seconds` should be appended to a log file `/var/log/alarm.log` everytime the alarm is triggered. Show test use case and results.
5. How can you log all commands executed by every user on Linux systems. What utility will you use for this. Show how you configure this tool, and show the logs generated.

### Bonus

6. Set up a centralized journald logging server `systemd-journal-remote`. Configure another machine as a client to forward its journal to the logging server.
   - Test your setup by running the `logger` utility on the client system and show the logs generated on the logging server.