linux-server-setup/exercises/h5.md

Linux servers - Exercice 5
==============

*Disclaimer:*
--------------

This exercise is a part of [Linux servers (ICT4TN021, spring 2018) // Linux-palvelimet (ICT4TN021, kevät 2018)](http://www.haaga-helia.fi/fi/opinto-opas/opintojaksokuvaukset/ICT4TN021) school course organized as a part of Information Technology studies in Haaga-Helia university of Applied Sciences, Helsinki, Finland. Course lecturer [Tero Karvinen](http://terokarvinen.com/) has defined the original assignment descriptions in Finnish presented in this document in English. Answers and translations have been written by Pekka Helenius (me, ~ Fincer).

**a)** Install SSH server daemon
--------------

**Answer:**

SSH daemon has already been installed on the virtual server machine (you can't establish a connection to the server with a SSH client program otherwise). However, the installation on Debian-based systems goes as follows:

```
phelenius@my-machine:~$ sudo apt-get update && sudo apt-get install openssh-server
```

Start SSH daemon and check whether it is active or not:

```
phelenius@my-machine:~$ sudo systemctl start sshd.service
phelenius@my-machine:~$ [[ $(systemctl is-active sshd | grep ^active | wc -w) -eq 1 ]] && echo "REPLY: SSH server daemon is active" || echo "REPLY: SSH server daemon is not active"
REPLY: SSH server daemon is active
```

SSH server daemon has been installed and is active on the server computer. Let's take a look-up into Systemd process tree:

```
phelenius@my-machine:~$ systemctl status
...
   CGroup: /
           ├─init.scope
           │ └─1 /lib/systemd/systemd --system --deserialize 20
           ├─system.slice
           ...
           │ ├─ssh.service
           │ │ ├─1534 /usr/sbin/sshd -D
           │ │ ├─2365 sshd: [accepted]    
           │ │ └─2366 sshd: [net]
```

Executable root binary file (sbin) is `/usr/sbin/sshd_ with command parameter _-D` (Process Identifier/Process ID/PID is 1534).

**NOTE:** The following is stated about `-D` parameter in sshd manual:

```
phelenius@my-machine:~$ man sshd | grep -E "\-D" -C1

     -D      When this option is specified, sshd will not detach and does not become a daemon.  This allows easy monitoring of sshd.
```

`-D` parameter definition by [SSH Communications Security](https://www.ssh.com/ssh/sshd/):

> -D Do not detach and become daemon. This is often used when sshd is run using systemd. This allows easier monitoring of the process in such environments. Without this option, the SSH server forks and detaches from terminal, making itself a background daemon process. The latter has been the traditional way to run the SSH server until recently. Many embedded systems would still use the latter.

About relevance of the `-D` parameter has been discussed, for instance, on [superuser.com (Differences between ssh -L to -D](https://superuser.com/questions/408031/differences-between-ssh-l-to-d)).


**b)** Establish a firewall protection to the server computer (Note: allow SSH traffic before that)
--------------

**Answer:**

The firewall protection is done by using Linux kernel ip_table module(s). Firewall rules can be modified in User Space with the corresponding iptables command or with the simplified Python 3 based program/script 'Uncomplicated Firewall' (`ufw`). Other firewall solutions also exists on Linux, please see ['Other firewall solutions'](https://github.com/Fincer/linux_server_setup/blob/master/exercises/h5.md#other-firewall-solutions) below. 

We can check which loadable kernel modules have been enabled in Linux kernel with the kernel-related lsmod command.

```
phelenius@my-machine:~$ man lsmod | sed -n '/NAME/{n;p}' | sed 's/\s*//'
lsmod - Show the status of modules in the Linux Kernel
...
phelenius@my-machine:~$ lsmod |grep filter
ip6table_filter        16384  1
ip6_tables             28672  1 ip6table_filter
iptable_filter         16384  1
ip_tables              24576  1 iptable_filter
x_tables               36864  14 ip6table_filter,xt_hl,xt_recent,ip_tables,xt_tcpudp,xt_limit,xt_conntrack,xt_LOG,iptable_filter,ip6t_rt,ipt_REJECT,ip6_tables,xt_addrtype,ip6t_REJECT
```

Source codes of these modules can be found on git.kernel.org: [ipv6 netfilter](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv6/netfilter), [ipv6 netfilter core](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv6/netfilter.c), [ipv4 netfilter](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/netfilter), [ipv4 netfilter core](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/netfilter.c)

### UFW configuration

By default, python-based Uncomplicated Firewall (`ufw`) is usually pre-installed on many Linux distributions, including Debian-based systems. It's usually the default firewall front-end for `iptables` on Linux systems. Let's confirm that:

```
phelenius@my-machine:~$ if [[ $(dpkg --get-selections |grep ufw | awk '{print $1}' | wc -l) -eq 0 ]]; then sudo apt-get update && sudo apt-get -y install ufw && echo "REPLY: UFW is installed now"; else echo "REPLY: UFW has already been installed"; fi
REPLY: UFW has already been installed
```

Therefore UFW firewall is installed. Otherwise it would be installed on the system with `apt-get install ufw` command.

By default, the Linux firewall blocks SSH input traffic (default port `22`). This input port can be opened by applying the following ufw command:

```
phelenius@my-machine:~$ sudo ufw allow 22/tcp
```

or with the command (however, not specific port number is defined here):

```
phelenius@my-machine:~$ sudo ufw allow ssh
```

Afterwards, UFW can be re-enabled with the following command:

```
phelenius@my-machine:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)?
```

Your answer should be `y` (yes). The firewall does not close (possibly already established) SSH connection because we have opened port `22` TCP input traffic. We get a notification, stating:

```
Firewall is active and enabled on system startup
```

**NOTE:** The message is bit unclear, because according to some testing, the ufw firewall may still not be enabled on system boot. Automated start of ufw during system boot can foolproofly be enabled with:

```
phelenius@my-machine:~$ sudo systemctl enable ufw.service
Created symlink /etc/systemd/system/multi-user.target.wants/ufw.service -> /usr/lib/systemd/system/ufw.service.
```

Alternatively, the following message may appear:

```
Synchronizing state of ufw.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable ufw
```

### iptables configuration

**1.** Remove UFW from the Linux system, and remove all relevant UFW entries from iptables firewall rule list.

**NOTE:** Warning: (May) delete other important iptables rules configured by system administration!

**NOTE:** Be sure that you have direct access (local terminal, without network) to the target machine in a case of failure. Otherwise you may lock yourself out from the server.

```
sudo systemctl stop ufw.service && sudo systemctl disable ufw.service
sudo apt-get purge --remove ufw
sudo iptables --flush && sudo iptables --delete-chains
```

**2.** Confirm deletion of ufw entries from iptables firewall rule list with

```
sudo iptables -S
```

Output should be:

```
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
```

**3.** Add new firewall rules to iptables and keep already established connections open:

```
sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
```

**4.** Drop all connections, excluding already established connections like SSH:

```
sudo iptables -A INPUT -j DROP
sudo iptables -A FORWARD -j DROP
```

**5.** Allow SSH traffic (port number: `22`, protocol: `tcp`, etc.):

```
sudo iptables -I INPUT -p tcp --dport 22 -j ACCEPT
```

**NOTE:** This applies only to IPv4 connections. For IPv6, check [iptables6](https://www.linux.com/learn/intro-to-linux/2017/8/iptables-rules-ipv6).

**NOTE:** The forementioned command can be written so that connections only from a single IP is allowed, by using `eth0` network interface. For instance:

```
sudo iptables -I INPUT -p tcp -i eth0 -s 231.123.24.24 --dport 22 -j ACCEPT
```

**NOTE:** Be sure you have enabled 'net.ifnames=0' in your udev rules to get network interface names like `eth0`, `wlan0` etc.!

More about setting iptables firewall rules, take a look on [DigitalOcean website](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands) and [iptables - Append vs. Insert - website](https://serverfault.com/questions/472258/difference-between-iptables-a-and-i-option), for instance.

**6.** Confirm proper iptables firewall rules with `sudo iptables -S` command:

```
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j DROP
-A FORWARD -j DROP
-I INPUT -p tcp --dport 22 -j ACCEPT
```

**7.** Save the rules so that they won't be lost after rebooting:

```
sudo iptables-save | sudo tee /etc/iptables/iptables.save > /dev/null
cat /etc/iptables/iptables.save
```

**8.** Apply and enable the rules. You can restart iptables service (Packet Filtering Framework) if you prefer to do so:

```
sudo iptables-restore && sudo systemctl restart iptables.service
```

**9.** Confirm the current iptables firewall rules with the command `sudo iptables -S`.

More about iptables:

- [DigitalOcean - Iptables Essentials: Common Firewall Rules and Commands, SSH service](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands#service-ssh)

- [DigitalOcean - Iptables Essentials: Common Firewall Rules and Commands, Saving rules](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands#saving-rules).

- [The Geek Stuff - 25 Most Frequently Used Linux IPTables Rules Examples](https://www.thegeekstuff.com/2011/06/iptables-rules-examples)

### Other firewall solutions

In addition to UFW, other iptables-based firewall solutions have been developed on Linux. Take a look on [Firestarter](http://www.fs-security.com/), [Firewalld](https://fedoraproject.org/wiki/Firewalld?rd=FirewallD), [PeerGuardian](https://sourceforge.net/projects/peerguardian/?source=navbar), [FWBuilder](http://www.fwbuilder.org/), etc.

------------------------------------------------

### EXTRA - root account: more restrictions

In addition to previous root restrictions, the following extra restrictions were applied, too:

- root default shell were changed to `/sbin/nologin` in `/etc/passwd` file

- root account were locked by applying command `sudo usermod root --lock` (this was done already, it sets up an exclamation mark prefix to the root account password in file `/etc/shadow`

- root access removal of TTY sessions by commenting out relevant lines in file `/etc/securetty`

- SSH: setting `PermitRootLogin no` were applied in SSH server configuration file `/etc/ssh/sshd_config`

PAM configuration files (Pluggable Authentitation Modules) were not altered in `/etc/pam.d/` although security can be improved by applying some special system-spefific authentication rules there. PAM rules can restrict root permissions as well, including executable daemon processes and commands (i.e. you can restrict even root execution rights in the target system, in contradiction what is usually told for new Linux users that "root can do anything". With PAM, you can restrict that "anything" privilege.)

Reference: [Red Hat - 4.4.2. Disallowing Root Access](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Security_Guide/s2-wstation-privileges-noroot.html)

**c)** Transfer files using SSH protocol
--------------

**Answer:**

Let's use [Secure Copy (scp)](https://en.wikipedia.org/wiki/Secure_copy) command for file transfers.

The assignment has been done in the following command sequence. The following steps are executed:

**1.** Create new subdirectory `httpd` into the current user's home directory on the local computer

**2.** Download related control, description and patch files of debian package of Apache web server, version `2.4.18-2ubuntu3.5` from archive.ubuntu.com ([packages.ubuntu.com](https://packages.ubuntu.com/xenial/apache2)). Target directory for the download is the recently created subdirectory `httpd`

**3.** Extract the downloaded archive files into `$HOME/httpd`. Subdirectory `debian` will be created in `httpd` directory.

**4.** Establish a new SSH connection to the server computer with server user `newuser`, use port `1234` and create directory `$HOME/patches` for this user.

**5.** Transfer all local files (and directories) which 1) start with any character 2) end with suffix `.diff` and `.patch` 3) locate at the current user's `./httpd/debian/patches/` subdirectory

to the directory `$HOME/patches` located at the server computer (where `$HOME` is `/home/newuser/`). SSH connection port is `1234`. Use `scp` command.

For more information about patches, check 

- [What is the format of a patch file - stackoverflow.com](https://stackoverflow.com/questions/987372/what-is-the-format-of-a-patch-file)

- [Patch command examples - thegeekstuff.com](https://www.thegeekstuff.com/2014/12/patch-command-examples))

**6.** List contents of the remote subdirectory `$HOME/patches` with SSH connection, use port `1234`

**7.** Delete current user's local subdirectory `httpd` and its contents. `-R` or `-r` stands for recursive deletion, `-f` stands for forceful removal. Remote files in the server computer's directory `/home/newuser/patches/` will be kept intact.

```
[20/02/2018 13:24:40 - fincer: ~ ]$ mkdir httpd
[20/02/2018 13:24:54 - fincer: ~ ]$ wget http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.18-2ubuntu3.5.debian.tar.xz -P httpd
--2018-02-20 13:25:13--  http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.18-2ubuntu3.5.debian.tar.xz
Resolving archive.ubuntu.com... 2001:67c:1560:8001::11, 2001:67c:1360:8001::17, 2001:67c:1560:8001::14, ...
Connecting to archive.ubuntu.com|2001:67c:1560:8001::11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 387448 (378K) [application/x-xz]
Saving to: ‘httpd/apache2_2.4.18-2ubuntu3.5.debian.tar.xz’

apache2_2.4.18-2ubuntu3.5.debian.tar.xz   100%[==================================================================================>] 378.37K   450KB/s    in 0.8s    

2018-02-20 13:25:15 (450 KB/s) - ‘httpd/apache2_2.4.18-2ubuntu3.5.debian.tar.xz’ saved [387448/387448]

[20/02/2018 13:25:15 - fincer: ~ ]$ tar xf apache2_2.4.18-2ubuntu3.5.debian.tar.xz -C httpd
[20/02/2018 13:25:35 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p 1234 mkdir patches
newuser@174.138.2.190's password: 
[20/02/2018 13:26:12 - fincer: ~ ]$ scp -P 1234 ./httpd/debian/patches/*.{diff,patch} newuser@174.138.2.190:./patches/
newuser@174.138.2.190's password: 
hostnames_with_underscores.diff                                                                                                    100%  479   114.6KB/s   00:00    
reproducible_builds.diff                                                                                                           100% 1710   427.2KB/s   00:00    
build_suexec-custom.patch                                                                                                          100% 1981   473.6KB/s   00:00    
customize_apxs.patch                                                                                                               100% 9316   888.9KB/s   00:00    
CVE-2016-0736.patch                                                                                                                100%   12KB 867.4KB/s   00:00    
CVE-2016-2161.patch                                                                                                                100% 4787   493.1KB/s   00:00    
CVE-2016-5387.patch                                                                                                                100%  666   287.9KB/s   00:00    
CVE-2016-8743.patch                                                                                                                100%   80KB   1.0MB/s   00:00    
CVE-2017-3167.patch                                                                                                                100% 8922   698.9KB/s   00:00    
CVE-2017-3169.patch                                                                                                                100% 4663   458.1KB/s   00:00    
CVE-2017-7668.patch                                                                                                                100% 1746   482.5KB/s   00:00    
CVE-2017-7679.patch                                                                                                                100% 1925   527.4KB/s   00:00    
CVE-2017-9788.patch                                                                                                                100% 2252   582.1KB/s   00:00    
CVE-2017-9798.patch                                                                                                                100% 1172   381.3KB/s   00:00    
fhs_compliance.patch                                                                                                               100% 2541   635.6KB/s   00:00    
no_LD_LIBRARY_PATH.patch                                                                                                           100%  444   199.8KB/s   00:00    
prefork_single_process_crash.patch                                                                                                 100%  773   359.0KB/s   00:00    
suexec-custom.patch                                                                                                                100% 5785   792.4KB/s   00:00    
suexec-CVE-2007-1742.patch                                                                                                         100% 2356   589.3KB/s   00:00    
suexec_is_shared.patch                                                                                                             100%  622   194.1KB/s   00:00    
[20/02/2018 13:28:03 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p 1234 ls -l ./patches
newuser@174.138.2.190's password: 
total 196
-rw-r--r-- 1 newuser newuser 12446 Feb 20 11:38 CVE-2016-0736.patch
-rw-r--r-- 1 newuser newuser  4787 Feb 20 11:38 CVE-2016-2161.patch
-rw-r--r-- 1 newuser newuser   666 Feb 20 11:38 CVE-2016-5387.patch
-rw-r--r-- 1 newuser newuser 81900 Feb 20 11:38 CVE-2016-8743.patch
-rw-r--r-- 1 newuser newuser  8922 Feb 20 11:38 CVE-2017-3167.patch
-rw-r--r-- 1 newuser newuser  4663 Feb 20 11:38 CVE-2017-3169.patch
-rw-r--r-- 1 newuser newuser  1746 Feb 20 11:38 CVE-2017-7668.patch
-rw-r--r-- 1 newuser newuser  1925 Feb 20 11:38 CVE-2017-7679.patch
-rw-r--r-- 1 newuser newuser  2252 Feb 20 11:38 CVE-2017-9788.patch
-rw-r--r-- 1 newuser newuser  1172 Feb 20 11:38 CVE-2017-9798.patch
-rw-r--r-- 1 newuser newuser  1981 Feb 20 11:38 build_suexec-custom.patch
-rw-r--r-- 1 newuser newuser  9316 Feb 20 11:38 customize_apxs.patch
-rw-r--r-- 1 newuser newuser  2541 Feb 20 11:38 fhs_compliance.patch
-rw-r--r-- 1 newuser newuser   479 Feb 20 11:38 hostnames_with_underscores.diff
-rw-r--r-- 1 newuser newuser   444 Feb 20 11:38 no_LD_LIBRARY_PATH.patch
-rw-r--r-- 1 newuser newuser   773 Feb 20 11:38 prefork_single_process_crash.patch
-rw-r--r-- 1 newuser newuser  1710 Feb 20 11:38 reproducible_builds.diff
-rw-r--r-- 1 newuser newuser  2356 Feb 20 11:38 suexec-CVE-2007-1742.patch
-rw-r--r-- 1 newuser newuser  5785 Feb 20 11:38 suexec-custom.patch
-rw-r--r-- 1 newuser newuser   622 Feb 20 11:38 suexec_is_shared.patch
[20/02/2018 13:32:01 - fincer: ~ ]$ rm -Rf httpd
```

**d)** Automate SSH login with public key method
--------------

**Answer:**

**1.** Let's generate a key pair on the client computer with a user who is wanted to have the public key authentication method for SSH connections. Both private and public keys are generated with `ssh-keygen` command. Accept the default settings unless you have any requirements to customize them.

**2.** Copy the **public key** to the server computer with your SSH client using command `ssh-copy-id -i $HOME/.ssh/id_rsa.pub username@server-ip-or-name`. The command copies your public key to the subdirectory `.ssh/authorized_keys` located at the server computer.

**NOTE:** Copy public key only, do **NOT** copy the private key!!

More about the topic:

[ssh.com - Key Pair - Public and Private](https://www.ssh.com/ssh/public-key-authentication#sec-Key-Pair-Public-and-Private)

**3.** Let confirm that SSH server daemon configuration file (`/etc/ssh/sshd_config` by default) has the following settings in place (either use `sudo nano /etc/ssh/sshd_config` or `sudoedit /etc/ssh/sshd_config`):

```
AuthenticationMethods   publickey
PubkeyAuthentication    yes
AuthorizedKeysFile      .ssh/authorized_keys
```

**Extra hint 1:** SSH client settings are generally defined in `/etc/ssh/ssh_config`

**Extra hint 2:** You can use ssh command with `-v` parameter to gain more information about used authentication methods in unclear cases or for debugging purposes. The `-v` parameter is defined as follows:

> -v      Verbose mode. Causes ssh to print debugging messages about its progress.  This is helpful in debugging connection, authentication, and configuration problems.  Multiple -v options increase the verbosity. The maximum is 3.

**4.** We shall restart server SSH daemon with `sudo systemctl restart sshd.service` command (note! this interrupts any existing & opened SSH connections)

**5.** We shall test public key authentication from the client computer. Use same user account which you previously used for executing `ssh-keygen` command on the client computer. The client computer user should be able to login into SSH server without a password.

**EXTRA: Modifying welcome banner**

Let's modify default Ubuntu login/SSH banner message by executing the following bash script:

```
#!/bin/bash

# Modify SSH login banner message

## 00-header file:
#
sudo sed -i 's?printf "Welcome to %s (%s %s %s)\\n" "$DISTRIB_DESCRIPTION" "$(uname -o)" "$(uname -r)" "$(uname -m)"?printf "Welcome to %s\\nUptime: 
%s\\n\\n" "Server Computer" "$(uptime)"?' /etc/update-motd.d/00-header

## File extensions of the following files will be modified
## We could also remove these files with 'sudo rm <file>'
#
FILES=( 10-help-text 51-cloudguest 90-updates-available 91-release-upgrade )

cd /etc/update-motd.d/
for file in ${FILES[@]}; do sudo mv $file $file.old; done
cd
```

**NOTE:** Location of the banner file varies between different Linux distributions. For instance, Arch Linux uses banner file `/etc_motd`.

**e)** Install, configure and start sysstat. Use sar command to confirm whether the sysstat package services have been enabled (for instance, log entry "Linux reboot..." exists). Run sysstat a day or two. Afterwards, check computer workload history with sysstat commands sar, iostat, pidstat etc. Analyze the results, i.e. explain the results in detail.
--------------

**Answer:**

**1.** We shall connect to the server computer (running Ubuntu) and then install the required package with command sequence `sudo apt-get update && sudo apt-get install -y install sysstat`

**2.** Run [shell-based sysstat script](https://github.com/Fincer/linux_server_setup/blob/master/scripts/sysstat_command.sh) which runs `sar` and `pidstat` commands for two days with 20 second intervals.

**NOTE:** The script can be opmitized more/adapted better to suit the real requirements for the server environment (statistics interval, collection period, etc.) 

**3.** The script described in step 2 has generated sar analytics file `sar-stats_2018-02-24_2018-02-26.file` into path `/home/newuser/sar_statistics/` on the server computer. 

Time period: 24. - 26.02.2018

The following sums up some command samples which can be applied to the analytics file.

| File | Description |
|------------------------------------|-------------------------------------------------|
| sar -u -f <analytics-file> | Processor workload statistics |
| sar -v -f <analytics-file> | Inode and file statistics |
| sar -r -f <analytics-file> | Memory consumption statistics |
| sar -n DEV -f <analytics-file> | Network stats: devices |
| sar -n EDEV -f <analytics-file> | Network stats: errors in devices |
| sar -n IP -f <analytics-file> | Network stats: IPv4 traffic |
| sar -n EIP -f <analytics-file> | Network stats: errors in IPv4 traffic |
| sar -n IP6 -f <analytics-file> | Network stats: IPv6 traffic |
| sar -n EIP6 -f <analytics-file> | Network stats: errors in IPv6 traffic |
| sar -n SOCK -f <analytics-file> | Network stats: IPv4 socket |
| sar -n SOCK6 -f <analytics-file> | Network stats: IPv6 socket |
| sar -n TCP -f <analytics-file> | Network stats: TCPv4 protocol traffic |
| sar -n ETCP -f <analytics-file> | Network stats: errors in TCPv4 protocol traffic |
| sar -S -f <analytics-file> | Swap memory consumption |
| sar -w -f <analytics-file> | Process statistics |
| sar -F MOUNT / -f <analytics-file> | File system which is mounted at / |

Statistics can be combined etc, as you can find out with 'man sar' command:

```
sar -u 2 5
    Report CPU utilization for each 2 seconds. 5 lines are displayed.

sar -I 14 -o int14.file 2 10
    Report statistics on IRQ 14 for each 2 seconds. 10 lines are displayed.  Data are stored in a file called int14.file.

sar -r -n DEV -f /var/log/sysstat/sa16
    Display memory and network statistics saved in daily data file 'sa16'.

sar -A
    Display all the statistics saved in current daily data file.
```

Or [The Geek Stuff - 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring](https://www.thegeekstuff.com/2011/03/sar-examples/?utm_source=feedburner)

What are inode and swap? Check
 
- [inode - 1](https://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html)

- [inode - 2](https://unix.stackexchange.com/questions/117093/find-where-inodes-are-being-used)

- [Swap](https://wiki.archlinux.org/index.php/swap)

Additionally, the following pidstat files were generated:

| File | Description |
|----------------------------|------------------------------------|
| pidstat_stats_cpu-tasks | Processor workload statistics |
| pidstat_stats_io | I/O statistics |
| pidstat_stats_kerneltables | Statistics of Linux kernel tables  |
| pidstat_stats_pagefaults | Page fault statistics |
| pidstat_stats_stacks | Process stack statistics |

(Check. [Stacks](https://stackoverflow.com/questions/8905271/what-is-the-linux-stack), [Page fault](https://en.wikipedia.org/wiki/Page_fault))

Additionally, iostat command was run on the background.

**4.** Let's take a closer look on two sar analytics files and I/O statistics file. The following commands open analytics files in more detailed view.

------------------------------------------------

### SAR network statistics - IPv4 traffic

**command: sar -n IP -f sar-stats_2018-02-24_2018-02-26.file**

![sar-stats-ipv4](https://github.com/Fincer/linux_server_setup/blob/master/images/sar-stats_ipv4.png)

**Observation period:** 24.-26.02.2018

| Field | Description | Average |
|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
| Left column | Record time |  |
| irec/s | The total number of input datagrams received from interfaces per second, including those received in error. | 1.33 |
| fwddgm/s | The number of input datagrams per second, for which this entity was not their final IP destination, as a result of which an attempt was made to find a route to forward them to that final destination. | 0.00 |
| idel/s | The total number of input datagrams,successfully,delivered,per,second,to,IP user-protocols (including ICMP). | 1.30 |
| orq/s | The total number of IP datagrams which local IP user-protocols (including ICMP) supplied per second to IP in requests for,transmission. Note that this,counter,does,not include any datagrams counted in fwddgm/s. | 1.45 |
| asmrq/s | The number of IP fragments received per second which needed to be reassembled at this entity. | 0.00 |
| asmok/s | The number of IP datagrams successfully re-assembled per second. | 0.00 |
| fragok/s | The number of IP datagrams that,have,been,successfully fragmented at this entity per second. | 0.00 |
| fragcrt/s | The number of IP datagram fragments that have been generated per second as a result of fragmentation at this entity. | 0.00 |

**NOTE:** Descriptions are provided in sysstat package (manpages).

**Analysis - IPV4**

The server computer has not forwarded a single datagram in the observation period. Input network traffic has been received which can be concluded by observing `irec/s`, `idel/s` and `orq/s` field values. The current workload of the server computer is very low, including only HTTP daemon (Apache web server) and SSH server daemon. No webpages are available on the server which could increase input HTTP/HTTPS protocol based traffic. HTTPS was not enabled in the server computer.

By default, the server computer allows `128` simultaneous input network connections (command `cat /proc/sys/net/core/somaxconn`. **NOTE:** The value can be changed by issuing `echo 1024 | sudo tee /proc/sys/net/core/somaxconn`, for instance).

MTU value of the web server computer for network interface `eth0` was `1500` (you can check it with `ifconfig` command). MTU stands for **Maximum Transmission Unit** which determines the highest PDU unit ([Protocol Data Unit](https://en.wikipedia.org/wiki/Protocol_data_unit)) that can be transferred over the network at once.

Check also

- [MTU - hyperlink 1](http://www.tcpipguide.com/free/t_IPDatagramSizeMaximumTransmissionUnitMTUFragmentat.htm)

- [MTU - hyperlink 2](https://en.wikipedia.org/wiki/Maximum_transmission_unit)

- [Stack Overflow - What's the practical limit on the size of single packet transmitted over domain socket?](https://stackoverflow.com/questions/21856517/whats-the-practical-limit-on-the-size-of-single-packet-transmitted-over-domain)

- [Stack Overflow - What is the default size of datagram queue length in Unix Domain Sockets (AF_UNIX)? Is it configurable?](https://stackoverflow.com/questions/21448960/what-is-the-default-size-of-datagram-queue-length-in-unix-domain-sockets-af-uni)

- [IP fragmentation](https://en.wikipedia.org/wiki/IP_fragmentation)

- Related article: [Linux Tune Network Stack (Buffers Size) To Increase Networking Performance](https://www.cyberciti.biz/faq/linux-tcp-tuning/)

**NOTE:** About datagrams, [quoted from Wikipedia](https://en.wikipedia.org/wiki/Datagram#Internet_Protocol):

> The term datagram is often considered synonymous to packet but there are some nuances. The term datagram is generally reserved for packets of an unreliable service, which cannot notify the sender if delivery fails, while the term packet applies to any packet, reliable or not. Datagrams are the IP packets that provide a quick and unreliable service like UDP, and all IP packets are datagrams; however, at the TCP layer, what is termed a TCP segment is the sometimes necessary IP fragmentation of a datagram, but those are referred to as "packets".

------------------------------------------------

### SAR - memory consumption statistics - RAM & Swap

**command: sar -r -f sar-stats_2018-02-24_2018-02-26.file**
**command: sar -S -f sar-stats_2018-02-24_2018-02-26.file**

![sar-stats-memusage](https://github.com/Fincer/linux_server_setup/blob/master/images/sar-stats_memusage.png)

**Observation period:** 24.-26.02.2018

**NOTE:** Average values are not visible in the attached picture!

| Field | Description | Average |
|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------|
| Left column | Record time |  |
| kbmemfree | Amount of free memory available in kilobytes. | 87631 KB (~ 87 MB) |
| kbmemused | Amount,of,used,memory in kilobytes. This does not take into account memory used by the kernel itself. | 928 457 KB (~ 928 MB) |
| %memused | Percentage of used memory. | 91.38 % |
| kbbuffers | Amount of memory used as buffers by the kernel in kilobytes. | 77 746 KB (~ 77 MB) |
| kbcached | Amount of memory used to cache data by the kernel in kilobytes. | 644 777 KB (~ 644 MB) |
| kbcommit | Amount of memory in kilobytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory. | 470 619 KB (~ 470 MB) |
| %commit | Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap).,This number may be greater than 100% because the,kernel,usually overcommits memory. | 46.32 % |
| kbactive | Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary). | 468 703 KB (~ 468 MB) |
| kbinact | Amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes). | 301 098 KB (~ 301 MB) |
| kbdirty | Amount of memory in kilobytes waiting to get written back to the disk. | 254 KB (0.254 MB) |

**NOTE:** Descriptions are provided in `sysstat` package (manpages).

**Analysis - Memory Statistics**

The target server memory size:

```
[newuser@goauldhost: ~ ]$ grep -i memtotal /proc/meminfo
MemTotal:        1016088 kB
```

i.e. approximately ~ 1016 MB.

Great part of this memory were in use during the observation period. There were multiple active processes running on the server computer during the period, thus increased workload was intended and meant for statistics collection.

The web server didn't have Swap partition or Swap file. This can be found out by

- executing command `for i in "cat /etc/fstab" "sudo fdisk -l"; do $i | grep -i swap; echo $?; done` which returns value 1 two times  (meaning returning of 'false' boolean value to shell two separate times) 

- Or, for instance:

```
[newuser@goauldhost: ~ ]$ cat /etc/fstab
LABEL=cloudimg-rootfs	/	 ext4	defaults	0 0
LABEL=UEFI	/boot/efi	vfat	defaults	0 0

[newuser@goauldhost: ~ ]$ sudo fdisk -l
[sudo] password for newuser: 
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 39DFE5D0-C8FB-44D8-93F8-EBB37A54BDF8

Device      Start      End  Sectors  Size Type
/dev/vda1  227328 52428766 52201439 24.9G Linux filesystem
/dev/vda14   2048    10239     8192    4M BIOS boot
/dev/vda15  10240   227327   217088  106M Microsoft basic data
```

It may not be wise to collect Swap statistics (although Linux kernel [Swappiness value](https://en.wikipedia.org/wiki/Swappiness) has default value `60` defined in `/proc/sys/vm/swappiness` on DigitalOcean virtual servers).

------------------------------------------------

### SAR - I/O statistics

![sar-iostat](https://github.com/Fincer/linux_server_setup/blob/master/images/sar-iostats.png)

Main command: `iostat -dmtx 20`

```
-d     Display the device utilization report.
-m     Display statistics in megabytes per second.
-t     Print the time for each report displayed.
-x     Display extended statistics.
20     20 sec interval.
```

| Field | Description |
|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Device | Device or partition defined in system directory '/dev'. |
| rrqm/s | The number of read requests merged per second that were queued to the device. |
| wrqm/s | The percentage of write requests merged together before being sent to the device. |
| r/s | The number (after merges) of read requests completed per second for the device. |
| w/s | The number (after merges) of write requests completed per second for the device. |
| rMB/s | The number of sectors (KB, MB) read from the device per second. |
| wMB/s | The number of sectors (KB, MB) write from the device per second. |
| avgrq-sz (areq-sz) | The average size (in kilobytes) of the I/O requests,that were issued to the device. |
| avgqu-sz (aqu-sz) | The average queue length of the requests that were issued to the device. |
| await | The average time (ms) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. |
| r_await | The average time (ms) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. |
| w_await | The average time (ms),for,write,requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. |
| svctm | The average service time (ms) for I/O requests that were issued to the device. This field will be removed in a future sysstat version. |
| %util | Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits. |

**NOTE:** Descriptions are provided in sysstat package (manpages).

**Analysis - I/O Statistics**

Virtual partition `/dev/vda` were hibernated most of the time during the observation period. Little variation can be seen in writing requests (`wrqm/s`), in succeeded writing requests (`w/s`) and in written sector count (`wMB/s`). Average sector size is 10-20 sectors (`avgrq-sz`) which is quite little. I/O and read request service times are mostly 0.0. Workload of the virtual hard drive is low during the observation period.

------------------------------------------------

In addition to sysstat, `top` command could have been used in some TTY session in order to observe processes in real time, and with [inotify](https://en.wikipedia.org/wiki/Inotify), some file system changes could have been reported. Sysstat's advantage is that programs which are included in it can record and save analytics in binary format for further and detailed inspection.

**f)** (optional) Change sshd (SSH server process) port
--------------

**Answer:**

Let the following shell script do the job...

```
#!/bin/bash

# SSH server daemon configuration file in the system
SSH_CONFIG=/etc/ssh/sshd_config

# New SSH server daemon input port as user input value.
NEW_SSH_PORT=$1

[[ -f $SSH_CONFIG ]] \
&& sed -i "s/.*Port.*/Port $NEW_SSH_PORT/" $SSH_CONFIG && echo "SSH server: new port $NEW_SSH_PORT is set." \
|| echo "SSH server configuration file could not be found!"

if [[ $(cat $SSH_CONFIG | grep -i port | awk '{print $2}') == $NEW_SSH_PORT ]]; then
    echo -e "SSH server input port has been changed to $NEW_SSH_PORT.\n \
Restarting SSH server daemon in order to apply the changes (root required)."

    sudo systemctl restart sshd.service

    if [[ $? == 0 ]]; then
        echo "SSH server daemon restarted, new input port is $NEW_SSH_PORT."
    else
        echo "Something went wrong while restarting SSH server daemon. Execute 'systemctl status sshd.service' \
to get more information about the problem."
    fi
fi
```

Save the above script code in file `$HOME/ssh-port.sh`, for instance. Change the port with command `bash $HOME/ssh-port.sh 4312` where the number value is your new SSH port (`4312` in this case).

------------------------------------------------

### EXTRA - Using new port address of SSH server daemon when connecting with a client computer/program

Changing SSH server input port on the server computer must be taken into account while establishing connection with a client computer. Because the default SSH port `22` is not used anymore, the following syntax must be applied while connecting to the SSH server computer:

```
[19/02/2018 23:23:49 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p <new-port-number>
```

------------------------------------------------

### EXTRA - detecting SSH port change with port scanning techniques (nmap)

Enable log level `VERBOSE` in `/etc/ssh/sshd_config` configuration file (`LogLevel VERBOSE`), restart SSH server daemon with command `sudo systemctl restart sshd.service` and try port scanning with another computer.

Port scanning:

```
nmap -sS -F -v -O 174.138.2.190
```

nmap output when port scanning is applied with another computer using target IP `174.138.2.190` and port `1234` (**NOTE:** This scanning has been applied to my own server only **once** as an experiment, not in bad intentions! Make sure you have permission to proceed with your scanning and do not attack to any server just for fun, to avoid any problems!):

```
phelenius@my-machine:~$ sudo nmap -sS -p 1234 -O -v 174.138.2.190

Starting Nmap 7.01 ( https://nmap.org ) at 2018-02-19 20:02 EET
Initiating Ping Scan at 20:02
Scanning 174.138.2.190 [4 ports]
Completed Ping Scan at 19:59, 0.20s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 20:02
Completed Parallel DNS resolution of 1 host. at 20:02, 0.01s elapsed
Initiating SYN Stealth Scan at 20:02
Scanning 174.138.2.190 [1 port]
Completed SYN Stealth Scan at 20:02, 0.20s elapsed (1 total ports)
Initiating OS detection (try #1) against 174.138.2.190
adjust_timeouts2: packet supposedly had rtt of -125997 microseconds.  Ignoring time.
adjust_timeouts2: packet supposedly had rtt of -125997 microseconds.  Ignoring time.
Retrying OS detection (try #2) against 174.138.2.190
adjust_timeouts2: packet supposedly had rtt of -150518 microseconds.  Ignoring time.
adjust_timeouts2: packet supposedly had rtt of -150518 microseconds.  Ignoring time.
WARNING: OS didn't match until try #2
Nmap scan report for 174.138.2.190
Host is up (0.00041s latency).
PORT   STATE    SERVICE
1234/tcp open  unknown
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: switch|general purpose|media device
Running: Cisco CatOS 7.X|8.X, HP Tru64 UNIX 5.X, Vantage embedded
OS CPE: cpe:/h:cisco:catalyst_ws-c6506 cpe:/o:cisco:catos:7.6 cpe:/o:cisco:catos:8.3 cpe:/o:hp:tru64:5.1a cpe:/h:vantage:hd7100s
OS details: Cisco Catalyst WS-C6506 switch (CatOS 7.6(16)), Cisco Catalyst switch (CatOS 8.3(2)), HP Tru64 UNIX 5.1A, Vantage HD7100S satellite receiver

Read data files from: /usr/bin/../share/nmap
OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 3.39 seconds
           Raw packets sent: 44 (6.312KB) | Rcvd: 18 (1.820KB)
```

If SSH server uses the default input port `22`, the following is likely to be presented in output of `nmap` command (unless you've defined a particular port to be scanned in your `nmap` command, like port `1234`):

```
PORT   STATE    SERVICE
22/tcp filtered ssh
```

Corresponding log entries of targeted SSH server during the nmap scanning (`/var/log/auth.log`):

```
Feb 19 18:02:46 goauldhost sshd[30057]: Connection from XXX.XXX.XXX.XXX port 6967 on 174.138.2.190 port 1234
Feb 19 18:02:46 goauldhost sshd[30057]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:46 goauldhost sshd[30058]: Connection from XXX.XXX.XXX.XXX port 52205 on 174.138.2.190 port 1234
Feb 19 18:02:46 goauldhost sshd[30058]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30059]: Connection from XXX.XXX.XXX.XXX port 25326 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30059]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30060]: Connection from XXX.XXX.XXX.XXX port 32812 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30060]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30061]: Connection from XXX.XXX.XXX.XXX port 17024 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30061]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30062]: Connection from XXX.XXX.XXX.XXX port 53268 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30062]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30063]: Connection from XXX.XXX.XXX.XXX port 34923 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30063]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30064]: Connection from XXX.XXX.XXX.XXX port 14489 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30064]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30065]: Connection from XXX.XXX.XXX.XXX port 40086 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30065]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30066]: Connection from XXX.XXX.XXX.XXX port 38147 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30066]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30067]: Connection from XXX.XXX.XXX.XXX port 49215 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30067]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30068]: Connection from XXX.XXX.XXX.XXX port 34445 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30068]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30069]: Connection from XXX.XXX.XXX.XXX port 4600 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30069]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30070]: Connection from XXX.XXX.XXX.XXX port 59405 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30070]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30071]: Connection from XXX.XXX.XXX.XXX port 7848 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30071]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:49 goauldhost sshd[30072]: Connection from XXX.XXX.XXX.XXX port 5206 on 174.138.2.190 port 1234
Feb 19 18:02:49 goauldhost sshd[30072]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30073]: Connection from XXX.XXX.XXX.XXX port 5517 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30073]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30074]: Connection from XXX.XXX.XXX.XXX port 3970 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30074]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30075]: Connection from XXX.XXX.XXX.XXX port 38690 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30075]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30076]: Connection from XXX.XXX.XXX.XXX port 50572 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30076]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30077]: Connection from XXX.XXX.XXX.XXX port 27830 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30077]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30078]: Connection from XXX.XXX.XXX.XXX port 49371 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30078]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30079]: Connection from XXX.XXX.XXX.XXX port 36802 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30079]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30080]: Connection from XXX.XXX.XXX.XXX port 50546 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30080]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30081]: Connection from XXX.XXX.XXX.XXX port 43542 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30081]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30082]: Connection from XXX.XXX.XXX.XXX port 56108 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30082]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30083]: Connection from XXX.XXX.XXX.XXX port 6399 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30083]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30084]: Connection from XXX.XXX.XXX.XXX port 55980 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30084]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30085]: Connection from XXX.XXX.XXX.XXX port 12713 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30085]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30086]: Connection from XXX.XXX.XXX.XXX port 5026 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30086]: Did not receive identification string from XXX.XXX.XXX.XXX

```

The server computer has logged invalid nmap connection requests. In this case, we have defined the known SSH connection port number `1234` in our `nmap` command on the client computer (which is INPUT port at the server end, OUTPUT port at the client end).

You may check suggested countermeasures against port scanners on [Unix & Linux Stack Exchange](https://unix.stackexchange.com/questions/345114/how-to-protect-against-port-scanners/407904#407904). Does these measures interfere negatively to other network traffic in busy server environments? This depends on which ports (read: server daemon programs/services) are targets of your countermeasures.

Another `nmap` command shows us the following:

```
phelenius@my-machine:~$ sudo nmap 174.138.2.190 -sV

Starting Nmap 7.01 ( https://nmap.org ) at 2018-02-20 14:50 EET
Nmap scan report for 174.138.2.190
Host is up (0.33s latency).
Not shown: 998 filtered ports
PORT     STATE SERVICE VERSION
1234/tcp open  ssh     OpenSSH 7.6 (protocol 2.0)
80/tcp   open  http
...
```

... which reveals the actual service behind our new-defined port 1234/tcp to an attacker. In addition, the attacker can get more information about the server system, especially on Debian-based Linux distributions. For instance:

```
...
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 7.2p2 Ubuntu 4ubuntu2.4 (Ubuntu Linux; protocol 2.0)
...
```

This information can be hidden by adding line `DebianBanner no` in `/etc/ssh/sshd_config` on Debian-based Linux systems and restarting the SSH server daemon with command `sudo systemctl restart sshd.service` afterwards

**NOTE:** Port scanning does not leave any log traces behind in Apache's access.log file (`/var/log/apache/access.log`)!

Check also
 
- [MyPapit GNU/Linux - How to Hide OpenSSH Ubuntu version from Nmap and other scanners](https://blog.mypapit.net/2015/08/how-to-hide-openssh-ubuntu-release-from-nmap-and-other-scanners.html)

- [serverfault.com - How to hide web server name and openssh version on linux when scanning server ports?](https://serverfault.com/questions/81690/how-to-hide-web-server-name-and-openssh-version-on-linux-when-scanning-server-po/81697#81697)

- [Unix & Linux Stack Exchange - Change SSH banner which is grabbed by netcat](https://unix.stackexchange.com/questions/269024/change-ssh-banner-which-is-grabbed-by-netcat/269027#269027)

- [GitHub - metacloud/openssh - Include the Debian version in our identification](https://github.com/metacloud/openssh/blob/master/debian/patches/package-versioning.patch)

------------------------------------------------

### EXTRA - Using Port Knocking technique against port scanning

Nmap requests are targeted to layer 3 (Network Layer) in OSI model. Additional security measures can be taken by applying [Port Knocking login techniques](https://wiki.archlinux.org/index.php/Port_knocking) on the server computer.

**WARNING!** The following **Knockd daemon** may break easily. Thus it can not be recommended for critical or rapidly updating server environments. For instance, dhcpcd version 7 or above breaks `knockd` daemon. Many users have had problems to start the daemon service during server boot-up process. Knockd assumes that you have a preconfigured network interface with established network connection (this is not 100% valid information but this is my best guess).

[DigitalOcean, "Port Knocking" - Ubuntu, knockd](https://www.digitalocean.com/community/tutorials/how-to-use-port-knocking-to-hide-your-ssh-daemon-from-attackers-on-ubuntu):

Dynamic firewall rules are manually applied to the server (iptables, for instance) or separate daemon process ([for instance, knockd](http://www.zeroflux.org/projects/knock)) is set up on the server computer. Daemon process is used to open a pre-defined port only if a client program has knocked correct server port numbers in right order and using correct protocol, either TCP or UDP, and this all has been done in predefined time limit. Additionally, limited timeout for user log-in can be set. For instance, a correctly authenticated user must log in in 10 seconds ater Knock daemon has opened a SSH port for that time period. When the time limit is reached, Knock daemon closes the port for any further connection attempts unless the knock port sequence is re-applied.

Knocking technique must be configured so that firewall does not close already established connections (like SSH in our case), but refuses any further connection attempts so that the port can not be detected with nmap port scanning techniques after the time limit has been reached.

[DigitalOcean, "Port Knocking" - FWKnop - Single packet authentication](https://www.digitalocean.com/community/tutorials/how-to-use-fwknop-to-enable-single-packet-authentication-on-ubuntu-12-04): 

Some more or less secure implementations for port knocking exist. Various encryption methods can be applied to the knocking technique, making it more difficult for an attacker to sniff network packets between a server and a client. One solution using encyption methods is [fwknop](http://www.cipherdyne.org/fwknop/) which uses `Single Packet Authorization` (SPA) method instead of insecure port sequences.

More about Port Knocking technique:

- [Improved Port Knocking with Strong Authentication - Rennie deGraaf, John Aycock, and Michael Jacobson, Jr.∗ (Department of Computer Science - University of Calgary)](https://www.acsac.org/2005/papers/156.pdf)

- [OpenWRT website](https://wiki.openwrt.org/doc/howto/portknock.server)

- [portknocking.org](http://www.portknocking.org/) and practical solutions on [Implementations sub-page](http://www.portknocking.org/view/implementations)

- [Information Security Stack Exchange - Port Knocking is it a good idea?](https://security.stackexchange.com/questions/1194/port-knocking-is-it-a-good-idea/1196#1196)

**NOTE:** Port Knocking technique must not be applied to block ports of server daemon processes which are required to be always reachable from a client program (take Apache server for an example). For SSH which does not require 24/7/365 connection, port knocking techniques can be applied more reliably. In addition to public key, password and PAM authentication methods, port knocking adds one more security layer to the stack. However, you must not apply these security countermeasures individually but use them together, instead. **The goal is to make it more difficult for an attacker to penetrate your network security.**

------------------------------------------------

### EXTRA - ARP Scan and spoofing your MAC address

Program [arp-scan](https://www.blackmoreops.com/2015/12/31/use-arp-scan-to-find-hidden-devices-in-your-network/) can be used in limited scale to scan a MAC address (OSI model layer 2, Data Link Layer) in a network.

Unique MAC address of a network interface (network card) can programmatically be spoofed with [these Arch Wiki instructions](https://wiki.archlinux.org/index.php/MAC_address_spoofing#Method_1:_systemd-networkd) or with my [Spoof MAC Address shell script](https://github.com/Fincer/linux_server_setup/blob/master/scripts/spoof_mac_address.sh).

Sample results of ARP Scan.

BEFORE:

```
[20/02/2018 18:52:35 - fincer: ~ ]$ sudo arp-scan --interface=wlan0 --localnet
[sudo] password for fincer: 
Interface: wlan0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
A.B.C.D         f4:0e:22:ad:b8:7d       (Unknown)
1.2.3.4         18:a9:05:4b:61:58       Hewlett-Packard Company
```

AFTER - MAC address of a computer having IP address 1.2.3.4 has been changed:

```
[20/02/2018 18:54:28 - fincer: ~ ]$ sudo arp-scan --interface=wlan0 --localnet
Interface: wlan0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
A.B.C.D         f4:0e:22:ad:b8:7d       (Unknown)
1.2.3.4         aa:0c:9a:fa:7b:d4       (Unknown)
```

The following warning messages may be expected after spoofing a MAC address and when you establish SSH connection from a local computer to your server:

```
Warning: Permanently added the ECDSA host key for IP address '[1.2.3.4]:22' to the list of known hosts.
```

**g)** (optional) Allow SSH login only for users in group 'sshers'. Add your account to this group.
--------------

**Answer:**

**1.** Log in to the server computer (`ssh username@server-ip -p <ssh-port>`)

**2.** Add group `sshers` with GID number `876` (you don't have to define GID here)

```
sudo groupadd -g 876 sshers
```

**3.** You can confirm existence of the created group with command `grep sshers /etc/group`. Output:

```
sshers:x:876:
```

**4.** Add the current user (read: yourself) `newuser` into this group

```
sudo usermod -aG sshers newuser
```

**5.** Command `grep sshers /etc/group` output now:

```
sshers:x:876:newuser
```

You can alternatively check groups of `newuser` by executing command `groups` while being that user:

```
[27/02/2018 10:39:58 - newuser@goauldhost: ~ ]$ groups
sshers newuser sudo
```

or using command `sudo -u newuser groups` as any system user who can execute commands with sudo.

**NOTE:** `groups` command gives you correct output only after re-login.

**6.** Allow SSH login only to members of the group `sshers`.

Manual page of `sshd_config` (`man sshd_config`) describes `AllowGroups` option as follows:

> AllowGroups
> This keyword can be followed by a list of group name patterns, separated by spaces.  If specified, login is allowed only for users whose primary group or supplementary group list matches one of the patterns.  Only group names are valid; a numerical group ID is not recognized. By default, login is allowed for all groups.  The allow/deny directives are processed in the following order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups.

Let's apply the following information into SSH server daemon configuration file `/etc/ssh/sshd_config`:

```
PermitRootLogin no
AllowGroups sshers
```

And comment out the following lines (be careful here!):

```
# DenyUsers <user accounts>
# AllowUsers <user accounts>
# DenyGroups <group names>
```

If those lines are not defined in the configuration file, it is OK.

You can add multiple groups and users after those keywords, as you like.

For additional security, the following options can be applied as well (into `/etc/ssh/sshd_config` file), depending on your requests for SSH authentication policy:

```
IgnoreRhosts yes
PermitEmptyPasswords no
ChallengeResponseAuthentication yes
UsePAM yes
MaxAuthTries 3
```

etc. More options and configurations can be found with commands `man sshd_config` and `man sshd`

**7.** Let's save this configuration and restart SSH server daemon by applying command `sudo systemctl restart sshd.service` (NOTE! Disconnects already-established/existing SSH connections!)

**8.** Test SSH login with a client computer using remote user `newuser` (belonging to the remote group `sshers`). Command syntax is: `ssh newuser@server-ip -p <server-port>`

**h)** (optional) Attach a remote network directory with sshfs.
--------------

**Answer:**

SSHFS stands for SSH File System. Corresponding program `sshfs` has been developed to attach SSH server file systems (folders/files) remotely on a client computer so that all attached file systems can locally be browsed on the client computer. SSHFS uses encrypted SFTP protocol (SSH/Secure File Transfer Protocol) for file tranfers.

**1.** Install `sshfs` with command sequence `sudo apt-get update && sudo apt-get -y install sshfs` on a Debian-based Linux system.

**2.** Mount remote folder `/home/newuser/public_html/` (server), to path `/home/<current-user>/public_html_ssh_remote` on your client computer. 

```
mkdir $HOME/public_html_ssh_remote && sshfs newuser@174.138.2.190:./public_html $HOME/public_html_ssh_remote -p <ssh port>
```

**NOTE:** If you use public key SSH authentication method, you are not asked for any password.

**3.** Check that you have successfully mounted your remote server folder `/home/newuser/public_html` to the client computer (command `ls ~/public_html_ssh_remote/`). Output should be as follows:

```
index.html
```

Additionally, check output of command `mount | grep public_html_ssh_remote`:

```
newuser@174.138.2.190:./public_html on /home/phelenius/public_html_ssh_remote type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
```

**4.** Unmount the remote folder by issuing command `sudo umount /home/phelenius/public_html_ssh_remote`. On the client end, `ls` command should have empty output for directory `public_html_ssh_remote`.