Lab 10: Logging and auditing

Task 1: journald

Journald rate limits log messages and will drop all messages from a service if it passes certain limits. These limits can be configured via RateLimitBurst and RateLimitIntervalSec, which default to 10000 and 30s respectively.
```
cat /etc/systemd/journald.conf | grep RateLimit
```
The rate limiting feature is very handy when you have services that generate a lot of logs.
journalctl is the main tool for interacting with the journal. Run this tool
```
$ journalctl
```
View all the options you can use with journalctl.
```
man journalctl
```
Show only the last 10 lines of the journal with the -n option.
```
$ journalctl -n 10
```
Follow the logs with the -f option.
```
$ journalctl -f
```
View the journal in reverse order. From the newest to the oldest.
```
$ journalctl --reverse
```
The --output in journalctl can be used to format the output of the journal in various forms.
View the journal in json format with the option --output=json-pretty or just --output=json if you want it compact.
```
$ journalctl --output=json-pretty
```
Filter journal for specific systemd unit with the command journalctl -u <systemd-unit>.
```
$ journalctl -u multiuser.target
```
You can also filter with the -p option to specify event severity levels that should be displayed.
Show only kernel messages.
```
$ journalctl --dmesg
```

You can filter by time with the --since and --until options.

$ journalctl --since="2022-10-30 23:00:00"

OR

$ journalctl --since=yesterday --until=now

You can view the time that the system was reboot in the journal
```
$ journalctl MESSAGE="Server listening on 0.0.0.0 port 22."
```

Task 2: rsyslog

Use the logger utility to generate logs.
```
$ logger Test
```
View the log in /varlog/syslog
```
$ tail /var/log/syslog
```
You should see a line of log similar to the following.
```
Oct 30 02:06:04 sna-vm root: Test
```

Rsyslog rules typically have the facility and the level. These two combined in the form facility.level define the priority of the log message.

More about facility and level: https://success.trendmicro.com/dcx/s/solution/TP000086250

Create an rsyslog rule that stores all logs with mail.emerg priority to a log file /var/log/test.log.
Do this by adding the following rule to /etc/rsyslog.conf.
```
user.emerg /var/log/test.log
```
Restart rsyslog to apply the changes.
```
$ systemctl restart rsyslog
```
Open a new terminal in another tab where you will run the logger utility to test this rule and execute the following command.
```
$ logger -p user.emerg "test log"
```
Go to the previous terminal tab, you should see a message similar to the following:

By default, there is a broadcast message everytime an emergency log is received.
View the log file where the rule we created will save logs of this priority to.
```
$ tail /var/log/test.log
```
You should get output similar to the following:
```
Oct 30 03:24:34 sna-vm user1: test log
```

Logging rsyslog to a remote server

The rsyslog configuration file is at /etc/rsyslog.conf.
Open the configuration file and add the following information. IP is the IP address of the remote log server.
```
*.* @IP
```
If you’re using TCP, the syntax is *.* @@IP.
*.* means we are forwarding logs of all facility, and all level to the remote server. Essentially forwarding everything.
A remote syslog server IP address will be provided in the lab for testing.
Restart the rsyslog service to apply the changes.
```
$ systemctl restart rsyslog
```

Task 3: Logrotate

We have a log file /var/log/lab10.log that we need to rotate periodically.

List all configurations created for logrotate to apply on various log files.
```
$ ls -lah /etc/logrotate.d/
```
Let’s create a custom log rotate configuration. First create the directory:
```
$ mkdir /etc/lab10-rotate.d
```

Create the file /etc/lab10-rotate.d/lab10log:

/var/log/lab10.log {
       su root syslog
       rotate 1
       maxsize 1M
       compress
       delaycompress
   }

Change the file permissions:

$ sudo chmod 644 /etc/lab10-rotate.d/lab10log

Include this new configuration to the logrotate configuration file.

$ cat << EOF | sudo tee /etc/lab10-rotate.conf
# Log rotation information for lab10log
include /etc/lab10-rotate.d
EOF

Change the file permissions:

$ sudo chmod 644 /etc/lab10-rotate.conf

Create a cron job that will run logrotate. Edit crontab as a root user:

$ sudo su
$ crontab -e

Add the following job to cron

#run command at every 2nd minute
*/2 * * * * logrotate /etc/lab10-rotate.d/lab10log

You can test and troubleshoot your logrotate configuration by running $ logrotate -s logstatus /etc/lab10-rotate.d/lab10log.
If no message is displayed on the terminal after running the command, it means that the configuration is good.
You can look in the target log file directory to verify that the log has been rotated.

If you are rotating a running service, it is good practice to restart the service in the logrotate configuration. The format is shown below:
```
postrotate
    <command to restart the service>
endscript
```

Task 4: User authentication activities

System login activity are saved in the log file /var/log/auth.log. You can see user sessions opened and other events such as authentication failure.
Let’s create a new user testuser which we will use to simulate failed login.
```
$ sudo adduser testuser
```
Try switching to the testuser with the su utility. Enter the wrong password on the prompt.
```
$ su testuser
Password: 
su: Authentication failure
```

Check the log file /var/log/auth.log to view this failed login activity.

$ tail /var/log/auth.log

You should see lines of log similar to the following two. These are the failed login events.

Oct 30 20:03:07 sna-vm su: pam_unix(su:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=user1 rhost=  user=testuser
Oct 30 20:03:09 sna-vm su: FAILED SU (to testuser) user1 on pts/0

Try switching to the root user. This time, enter three consequtive wrong passwords in the prompt.

$ sudo su
[sudo] password for user1: 
Sorry, try again.
[sudo] password for user1: 
Sorry, try again.
[sudo] password for user1: 
sudo: 3 incorrect password attempts

View the authentication log file again.

$ tail /var/log/auth.log

You should see the following lines in the output.

Oct 30 20:09:05 sna-vm sudo: pam_unix(sudo:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/1 ruser=user1 rhost=  user=user1
Oct 30 20:09:13 sna-vm sudo:    user1 : 3 incorrect password attempts ; TTY=pts/1 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/su

The audit log file contains several other events. We can filter for failed log in attempts with the text filtering editors.
```
$ cat /var/log/auth.log | grep -P "(?i)authentication failure|incorrect password"
```
journalctl can also display such logs. Let’s run journalctl and filter for authentication failure.
```
$ journalctl | grep -P "(?i)authentication failure|incorrect password"
```
Create a script that automatically extracts these failed authentication logs and save them to another file. This makes it easier to quickly detect security violations.
Create the script track_auth_fail.sh
```
$ vim track_auth_fail.sh
```
Add the following content to the script:
```
#!/bin/bash

tail -n0 -f /var/log/auth.log | \
grep -P --line-buffered "authentication failure|incorrect password" |\
tee /var/log/failed_auth.log
```
Run the script in the background and suppress it’s output.
```
sudo bash track_auth_fail.sh > /dev/null 2>&1 &
```

Simulate a failed login for testuser.

$ su testuser
Password: 
su: Authentication failure

Check the new log file with $ tail /var/log/failed_auth.logand you should see log similar to the following:

Oct 30 21:15:37 sna-vm su: pam_unix(su:auth): authentication failure; logname= uid=1000 euid=0 tty=/dev/pts/0 ruser=user1 rhost=  user=testuser

Task 5: User sessions and sudo usage

We can run commands as other users from our login session. When we do this, a new user session is opened. Run a command as testuser.
```
$ su - testuser -c "whoami"
Password: 
testuser
```
Now check the auth log file to see if this activity is logged.
```
$ tail  /var/log/auth.log
```
You should get output similar to the following:
```
Oct 30 21:55:49 sna-vm su: (to testuser) user1 on pts/1
Oct 30 21:55:49 sna-vm su: pam_unix(su-l:session): session opened for user testuser(uid=1001) by (uid=1000)
Oct 30 21:55:50 sna-vm su: pam_unix(su-l:session): session closed for user testuser
```
- The first line of log above shows that user1 first attempted to use the su utility to switch to testuser.
- The second line shows that the session was opened.
- The third line shows that the session was closed. This happened after the command finished executing.
Sudo usage logs are also saved in /var/log/auth.log. When a user runs sudo, they open a sudo session.
Filter the log file for sudo. Run the following command:
```
$ sudo ip a
```

View the log file

$ tail  /var/log/auth.log

You should get output siilar to the following:

Oct 30 22:03:20 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
Oct 30 22:03:20 sna-vm sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Oct 30 22:03:20 sna-vm sudo: pam_unix(sudo:session): session closed for user root

When a user attempts to use sudo to execute a command as root, that command is logged to the auth file as seen in the first line of the output above.

Filter the log to view all attempts made to execute commands as root:

$ cat /var/log/auth.log | grep -P "USER=root.*COMMAND="

You should get output similar to the following:

Oct 30 21:15:23 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/bash track_auth_fail.sh
Oct 30 21:47:54 sna-vm sudo:    user1 : TTY=pts/1 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
Oct 30 22:03:20 sna-vm sudo:    user1 : TTY=pts/0 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a
Oct 30 22:07:25 sna-vm sudo:    user1 : 3 incorrect password attempts ; TTY=pts/4 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/sbin/ip a

Questions to answer

What security monitoring tool will you forward your system logs to for security event detection? Give reasons for your choice.
Configure rsyslogd by adding a rule to the newly created configuration file /etc/rsyslog.d/auth-errors.conf to log all security and authentication messages with the priority alert and higher to the /var/log/auth-errors file. Test the newly added log directive with the logger command. Verify it from rsyslog and journald perspectives by filtering the output.
Install Apache web server and configure log rotate to rotate its web access log every six hours. Compress the rotated log files, and ensure that log rotate restarts the web server after rotating the logs. Manually execute the logrotate utility to test your configuration and show results.
Create a bash script that continuously monitors the /var/log/auth.log file and triggers an alarm if there are three or more “authentication failure” in 30 seconds. The text Three or more authentication failure in 30 seconds should be appended to a log file /var/log/alarm.log everytime the alarm is triggered. Show test use case and results.
How can you log all commands executed by every user on Linux systems. What utility will you use for this. Show how you configure this tool, and show the logs generated.

Bonus

Set up a centralized journald logging server systemd-journal-remote. Configure another machine as a client to forward its journal to the logging server.
- Test your setup by running the logger utility on the client system and show the logs generated on the logging server.