linux-server-setup/exercises/h5.md

Linux servers - Exercise 5

Disclaimer:

This exercise is a part of Linux Server Administration (ICT4TN021, spring 2018) // Linux-palvelimet (ICT4TN021, kevät 2018) school course organized as a part of Information Technology studies in Haaga-Helia university of Applied Sciences, Helsinki, Finland. Course lecturer Tero Karvinen has defined the original assignment descriptions in Finnish presented in this document in English. Answers and translations have been written by Pekka Helenius (me, ~ Fincer).

Table of contents:

• a) SSH Daemon Install SSH server daemon

• b) Firewall Establish a firewall protection to the server computer (Note: allow SSH traffic before that)

  • UFW configuration

  • iptables configuration

  • Other firewall solutions

  • EXTRA - root account: more restrictions

• c) File transfer Transfer files using SSH protocol

• d) SSH auto-login Automate SSH login with public key method

  • EXTRA: Modifying welcome banner

• e) Analysis of server processes Install, configure and start sysstat. Use sar command to confirm whether the sysstat package services have been enabled (for instance, log entry "Linux reboot..." exists). Run sysstat a day or two. Afterwards, check computer workload history with sysstat commands sar, iostat, pidstat etc. Analyze the results, i.e. explain the results in detail.

  • SAR - network statistics, IPv4 traffic

  • SAR - memory consumption statistics, RAM & Swap

  • SAR - I/O statistics

• f) SSH port configuration (optional) Change sshd (SSH server process) port

  • EXTRA - Using new port address of SSH server daemon when connecting with a client computer/program

  • EXTRA - detecting SSH port change with port scanning techniques (nmap)

  • EXTRA - Using Port Knocking technique against port scanning

  • EXTRA - ARP Scan and spoofing your MAC address

• g) SSH configuration - groups (optional) Allow SSH login only for users in group 'sshers'. Add your account to this group.

• h) SSH remote dir mounting (optional) Attach a remote network directory with sshfs.

---

a) Install SSH server daemon

Answer:

SSH daemon has already been installed on the virtual server machine (you can't establish a connection to the server with a SSH client program otherwise). However, the installation on Debian-based systems goes as follows:

phelenius@my-machine:~$ sudo apt-get update && sudo apt-get install openssh-server

Start SSH daemon and check whether it is active or not:

phelenius@my-machine:~$ sudo systemctl start sshd.service
phelenius@my-machine:~$ [[ $(systemctl is-active sshd | grep ^active | wc -w) -eq 1 ]] && echo "REPLY: SSH server daemon is active" || echo "REPLY: SSH server daemon is not active"
REPLY: SSH server daemon is active

SSH server daemon has been installed and is active on the server computer. Let's take a look-up into Systemd process tree:

phelenius@my-machine:~$ systemctl status
...
CGroup: /
├─init.scope
│ └─1 /lib/systemd/systemd --system --deserialize 20
├─system.slice
...
│ ├─ssh.service
│ │ ├─1534 /usr/sbin/sshd -D
│ │ ├─2365 sshd: [accepted]    
│ │ └─2366 sshd: [net]

Executable root binary file (sbin) is /usr/sbin/sshd_ with command parameter _-D (Process Identifier/Process ID/PID is 1534).

NOTE: The following is stated about -D parameter in sshd manual:

phelenius@my-machine:~$ man sshd | grep -E "\-D" -C1
-D      When this option is specified, sshd will not detach and does not become a daemon.  This allows easy monitoring of sshd.

-D parameter definition by SSH Communications Security:

-D Do not detach and become daemon. This is often used when sshd is run using systemd. This allows easier monitoring of the process in such environments. Without this option, the SSH server forks and detaches from terminal, making itself a background daemon process. The latter has been the traditional way to run the SSH server until recently. Many embedded systems would still use the latter.

About relevance of the -D parameter has been discussed, for instance, on superuser.com (Differences between ssh -L to -D).

b) Establish a firewall protection to the server computer (Note: allow SSH traffic before that)

Answer:

The firewall protection is done by using Linux kernel ip_table module(s). Firewall rules can be modified in User Space with the corresponding iptables command or with the simplified Python 3 based program/script 'Uncomplicated Firewall' (ufw). Other firewall solutions also exists on Linux, please see 'Other firewall solutions' below.

We can check which loadable kernel modules have been enabled in Linux kernel with the kernel-related lsmod command.

phelenius@my-machine:~$ man lsmod | sed -n '/NAME/{n;p}' | sed 's/\s*//'
lsmod - Show the status of modules in the Linux Kernel
...
phelenius@my-machine:~$ lsmod |grep filter
ip6table_filter        16384  1
ip6_tables             28672  1 ip6table_filter
iptable_filter         16384  1
ip_tables              24576  1 iptable_filter
x_tables               36864  14 ip6table_filter,xt_hl,xt_recent,ip_tables,xt_tcpudp,xt_limit,xt_conntrack,xt_LOG,iptable_filter,ip6t_rt,ipt_REJECT,ip6_tables,xt_addrtype,ip6t_REJECT

Source codes of these modules can be found on git.kernel.org: ipv6 netfilter, ipv6 netfilter core, ipv4 netfilter, ipv4 netfilter core

UFW configuration

By default, python-based Uncomplicated Firewall (ufw) is usually pre-installed on many Linux distributions, including Debian-based systems. It's usually the default firewall front-end for iptables on Linux systems. Let's confirm that:

phelenius@my-machine:~$ if [[ $(dpkg --get-selections |grep ufw | awk '{print $1}' | wc -l) -eq 0 ]]; then sudo apt-get update && sudo apt-get -y install ufw && echo "REPLY: UFW is installed now"; else echo "REPLY: UFW has already been installed"; fi
REPLY: UFW has already been installed

Therefore UFW firewall is installed. Otherwise it would be installed on the system with apt-get install ufw command.

By default, the Linux firewall blocks SSH input traffic (default port 22). This input port can be opened by applying the following ufw command:

phelenius@my-machine:~$ sudo ufw allow 22/tcp

or with the command (however, not specific port number is defined here):

phelenius@my-machine:~$ sudo ufw allow ssh

Afterwards, UFW can be re-enabled with the following command:

phelenius@my-machine:~$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)?

Your answer should be y (yes). The firewall does not close (possibly already established) SSH connection because we have opened port 22 TCP input traffic. We get a notification, stating:

Firewall is active and enabled on system startup

NOTE: The message is bit unclear, because according to some testing, the ufw firewall may still not be enabled on system boot. Automated start of ufw during system boot can foolproofly be enabled with:

phelenius@my-machine:~$ sudo systemctl enable ufw.service
Created symlink /etc/systemd/system/multi-user.target.wants/ufw.service -> /usr/lib/systemd/system/ufw.service.

Alternatively, the following message may appear:

Synchronizing state of ufw.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable ufw

iptables configuration

---

NOTE: A iptables ruleset for a simple SSH+HTTP server setup has been uploaded to this repository, you may want to check it out: Fincer/linux-server-setup - iptables.rules.

These rules do the following:

1. Do not respond to ping echoes by clients (possibly reduce spambots)

2. Reject connection if too intense attempts. Useful against port scanners such as Nmap and other brute force scanners such as Dirbuster.

3. Drop all incoming connections, apply only SSH, HTTP and HTTPS

The ruleset file has basic installation steps in the header section.

---

A basic iptables configuration for incoming SSH requests is as follows:

1. Remove UFW from the Linux system, and remove all relevant UFW entries from iptables firewall rule list.

NOTE: Warning: (May) delete other important iptables rules configured by system administration!

NOTE: Be sure that you have direct access (local terminal, without network) to the target machine in a case of failure. Otherwise you may lock yourself out from the server.

sudo systemctl stop ufw.service && sudo systemctl disable ufw.service
sudo apt-get purge --remove ufw
sudo iptables --flush && sudo iptables --delete-chain

2. Confirm deletion of ufw entries from iptables firewall rule list with

sudo iptables -S

Output should be:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

3. Add new firewall rules to iptables and keep already established connections open:

sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

4. Drop all connections, excluding already established connections like SSH:

sudo iptables -A INPUT -j DROP
sudo iptables -A FORWARD -j DROP

5. Allow SSH traffic (port number: 22, protocol: tcp, etc.):

sudo iptables -I INPUT -p tcp --dport 22 -j ACCEPT

NOTE: This applies only to IPv4 connections. For IPv6, check iptables6.

NOTE: The forementioned command can be written so that connections only from a single IP is allowed, by using eth0 network interface. For instance:

sudo iptables -I INPUT -p tcp -i eth0 -s 231.123.24.24 --dport 22 -j ACCEPT

NOTE: Be sure you have enabled 'net.ifnames=0' in your udev rules to get network interface names like eth0, wlan0 etc.!

More about setting iptables firewall rules, take a look on DigitalOcean website and iptables - Append vs. Insert - website, for instance.

6. Confirm proper iptables firewall rules with sudo iptables -S command:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j DROP
-A FORWARD -j DROP
-I INPUT -p tcp --dport 22 -j ACCEPT

7. Save the rules so that they won't be lost after rebooting:

sudo iptables-save | sudo tee /etc/iptables/iptables.save > /dev/null
cat /etc/iptables/iptables.save

8. Apply and enable the rules. You can restart iptables service (Packet Filtering Framework) if you prefer to do so:

sudo iptables-restore && sudo systemctl restart iptables.service

9. Confirm the current iptables firewall rules with the command sudo iptables -S.

More about iptables:

• DigitalOcean - Iptables Essentials: Common Firewall Rules and Commands, SSH service

• DigitalOcean - Iptables Essentials: Common Firewall Rules and Commands, Saving rules.

• The Geek Stuff - 25 Most Frequently Used Linux IPTables Rules Examples

Other firewall solutions

In addition to UFW, other iptables-based firewall solutions have been developed on Linux. Take a look on Firestarter, Firewalld, PeerGuardian, FWBuilder, etc.

---

EXTRA - root account: more restrictions

In addition to previous root restrictions, the following extra restrictions were applied, too:

• root default shell were changed to /sbin/nologin in /etc/passwd file

• root account were locked by applying command sudo usermod root --lock (this was done already, it sets up an exclamation mark prefix to the root account password in file /etc/shadow

• root access removal of TTY sessions by commenting out relevant lines in file /etc/securetty

• SSH: setting PermitRootLogin no were applied in SSH server configuration file /etc/ssh/sshd_config

PAM configuration files (Pluggable Authentitation Modules) were not altered in /etc/pam.d/ although security can be improved by applying some special system-spefific authentication rules there. PAM rules can restrict root permissions as well, including executable daemon processes and commands (i.e. you can restrict even root execution rights in the target system, in contradiction what is usually told for new Linux users that "root can do anything". With PAM, you can restrict that "anything" privilege.)

Reference: Red Hat - 4.4.2. Disallowing Root Access

c) Transfer files using SSH protocol

Answer:

Let's use Secure Copy (scp) command for file transfers.

The assignment has been done in the following command sequence. The following steps are executed:

1. Create new subdirectory httpd into the current user's home directory on the local computer

2. Download related control, description and patch files of debian package of Apache web server, version 2.4.18-2ubuntu3.5 from archive.ubuntu.com (packages.ubuntu.com). Target directory for the download is the recently created subdirectory httpd

3. Extract the downloaded archive files into $HOME/httpd. Subdirectory debian will be created in httpd directory.

4. Establish a new SSH connection to the server computer with server user newuser, use port 1234 and create directory $HOME/patches for this user.

5. Transfer all local files (and directories) which 1) start with any character 2) end with suffix .diff and .patch 3) locate at the current user's ./httpd/debian/patches/ subdirectory

to the directory $HOME/patches located at the server computer (where $HOME is /home/newuser/). SSH connection port is 1234. Use scp command.

For more information about patches, check

• What is the format of a patch file - stackoverflow.com

• Patch command examples - thegeekstuff.com)

6. List contents of the remote subdirectory $HOME/patches with SSH connection, use port 1234

7. Delete current user's local subdirectory httpd and its contents. -R or -r stands for recursive deletion, -f stands for forceful removal. Remote files in the server computer's directory /home/newuser/patches/ will be kept intact.

[20/02/2018 13:24:40 - fincer: ~ ]$ mkdir httpd
[20/02/2018 13:24:54 - fincer: ~ ]$ wget http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.18-2ubuntu3.5.debian.tar.xz -P httpd
--2018-02-20 13:25:13--  http://archive.ubuntu.com/ubuntu/pool/main/a/apache2/apache2_2.4.18-2ubuntu3.5.debian.tar.xz
Resolving archive.ubuntu.com... 2001:67c:1560:8001::11, 2001:67c:1360:8001::17, 2001:67c:1560:8001::14, ...
Connecting to archive.ubuntu.com|2001:67c:1560:8001::11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 387448 (378K) [application/x-xz]
Saving to: ‘httpd/apache2_2.4.18-2ubuntu3.5.debian.tar.xz’
apache2_2.4.18-2ubuntu3.5.debian.tar.xz   100%[==================================================================================>] 378.37K   450KB/s    in 0.8s    
2018-02-20 13:25:15 (450 KB/s) - ‘httpd/apache2_2.4.18-2ubuntu3.5.debian.tar.xz’ saved [387448/387448]
[20/02/2018 13:25:15 - fincer: ~ ]$ tar xf apache2_2.4.18-2ubuntu3.5.debian.tar.xz -C httpd
[20/02/2018 13:25:35 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p 1234 mkdir patches
newuser@174.138.2.190's password: 
[20/02/2018 13:26:12 - fincer: ~ ]$ scp -P 1234 ./httpd/debian/patches/*.{diff,patch} newuser@174.138.2.190:./patches/
newuser@174.138.2.190's password: 
hostnames_with_underscores.diff                                                                                                    100%  479   114.6KB/s   00:00    
reproducible_builds.diff                                                                                                           100% 1710   427.2KB/s   00:00    
build_suexec-custom.patch                                                                                                          100% 1981   473.6KB/s   00:00    
customize_apxs.patch                                                                                                               100% 9316   888.9KB/s   00:00    
CVE-2016-0736.patch                                                                                                                100%   12KB 867.4KB/s   00:00    
CVE-2016-2161.patch                                                                                                                100% 4787   493.1KB/s   00:00    
CVE-2016-5387.patch                                                                                                                100%  666   287.9KB/s   00:00    
CVE-2016-8743.patch                                                                                                                100%   80KB   1.0MB/s   00:00    
CVE-2017-3167.patch                                                                                                                100% 8922   698.9KB/s   00:00    
CVE-2017-3169.patch                                                                                                                100% 4663   458.1KB/s   00:00    
CVE-2017-7668.patch                                                                                                                100% 1746   482.5KB/s   00:00    
CVE-2017-7679.patch                                                                                                                100% 1925   527.4KB/s   00:00    
CVE-2017-9788.patch                                                                                                                100% 2252   582.1KB/s   00:00    
CVE-2017-9798.patch                                                                                                                100% 1172   381.3KB/s   00:00    
fhs_compliance.patch                                                                                                               100% 2541   635.6KB/s   00:00    
no_LD_LIBRARY_PATH.patch                                                                                                           100%  444   199.8KB/s   00:00    
prefork_single_process_crash.patch                                                                                                 100%  773   359.0KB/s   00:00    
suexec-custom.patch                                                                                                                100% 5785   792.4KB/s   00:00    
suexec-CVE-2007-1742.patch                                                                                                         100% 2356   589.3KB/s   00:00    
suexec_is_shared.patch                                                                                                             100%  622   194.1KB/s   00:00    
[20/02/2018 13:28:03 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p 1234 ls -l ./patches
newuser@174.138.2.190's password: 
total 196
-rw-r--r-- 1 newuser newuser 12446 Feb 20 11:38 CVE-2016-0736.patch
-rw-r--r-- 1 newuser newuser  4787 Feb 20 11:38 CVE-2016-2161.patch
-rw-r--r-- 1 newuser newuser   666 Feb 20 11:38 CVE-2016-5387.patch
-rw-r--r-- 1 newuser newuser 81900 Feb 20 11:38 CVE-2016-8743.patch
-rw-r--r-- 1 newuser newuser  8922 Feb 20 11:38 CVE-2017-3167.patch
-rw-r--r-- 1 newuser newuser  4663 Feb 20 11:38 CVE-2017-3169.patch
-rw-r--r-- 1 newuser newuser  1746 Feb 20 11:38 CVE-2017-7668.patch
-rw-r--r-- 1 newuser newuser  1925 Feb 20 11:38 CVE-2017-7679.patch
-rw-r--r-- 1 newuser newuser  2252 Feb 20 11:38 CVE-2017-9788.patch
-rw-r--r-- 1 newuser newuser  1172 Feb 20 11:38 CVE-2017-9798.patch
-rw-r--r-- 1 newuser newuser  1981 Feb 20 11:38 build_suexec-custom.patch
-rw-r--r-- 1 newuser newuser  9316 Feb 20 11:38 customize_apxs.patch
-rw-r--r-- 1 newuser newuser  2541 Feb 20 11:38 fhs_compliance.patch
-rw-r--r-- 1 newuser newuser   479 Feb 20 11:38 hostnames_with_underscores.diff
-rw-r--r-- 1 newuser newuser   444 Feb 20 11:38 no_LD_LIBRARY_PATH.patch
-rw-r--r-- 1 newuser newuser   773 Feb 20 11:38 prefork_single_process_crash.patch
-rw-r--r-- 1 newuser newuser  1710 Feb 20 11:38 reproducible_builds.diff
-rw-r--r-- 1 newuser newuser  2356 Feb 20 11:38 suexec-CVE-2007-1742.patch
-rw-r--r-- 1 newuser newuser  5785 Feb 20 11:38 suexec-custom.patch
-rw-r--r-- 1 newuser newuser   622 Feb 20 11:38 suexec_is_shared.patch
[20/02/2018 13:32:01 - fincer: ~ ]$ rm -Rf httpd

d) Automate SSH login with public key method

Answer:

1. Let's generate a key pair on the client computer with a user who is wanted to have the public key authentication method for SSH connections. Both private and public keys are generated with ssh-keygen command. Accept the default settings unless you have any requirements to customize them.

2. Copy the public key to the server computer with your SSH client using command ssh-copy-id -i $HOME/.ssh/id_rsa.pub username@server-ip-or-name. The command copies your public key to the subdirectory .ssh/authorized_keys located at the server computer.

NOTE: Copy public key only, do NOT copy the private key!!

More about the topic:

ssh.com - Key Pair - Public and Private

3. Let confirm that SSH server daemon configuration file (/etc/ssh/sshd_config by default) has the following settings in place (either use sudo nano /etc/ssh/sshd_config or sudoedit /etc/ssh/sshd_config):

AuthenticationMethods   publickey
PubkeyAuthentication    yes
AuthorizedKeysFile      .ssh/authorized_keys

Extra hint 1: SSH client settings are generally defined in /etc/ssh/ssh_config

Extra hint 2: You can use ssh command with -v parameter to gain more information about used authentication methods in unclear cases or for debugging purposes. The -v parameter is defined as follows:

-v Verbose mode. Causes ssh to print debugging messages about its progress. This is helpful in debugging connection, authentication, and configuration problems. Multiple -v options increase the verbosity. The maximum is 3.

4. We shall restart server SSH daemon with sudo systemctl restart sshd.service command (note! this interrupts any existing & opened SSH connections)

5. We shall test public key authentication from the client computer. Use same user account which you previously used for executing ssh-keygen command on the client computer. The client computer user should be able to login into SSH server without a password.

EXTRA: Modifying welcome banner

Let's modify default Ubuntu login/SSH banner message by executing the following bash script:

#!/bin/bash
# Modify SSH login banner message
## 00-header file:
#
sudo sed -i 's?printf "Welcome to %s (%s %s %s)\\n" "$DISTRIB_DESCRIPTION" "$(uname -o)" "$(uname -r)" "$(uname -m)"?printf "Welcome to %s\\nUptime: 
%s\\n\\n" "Server Computer" "$(uptime)"?' /etc/update-motd.d/00-header
## File extensions of the following files will be modified
## We could also remove these files with 'sudo rm <file>'
#
FILES=( 10-help-text 51-cloudguest 90-updates-available 91-release-upgrade )
cd /etc/update-motd.d/
for file in ${FILES[@]}; do sudo mv $file $file.old; done
cd

NOTE: Location of the banner file varies between different Linux distributions. For instance, Arch Linux uses banner file /etc_motd.

e) Install, configure and start sysstat. Use sar command to confirm whether the sysstat package services have been enabled (for instance, log entry "Linux reboot..." exists). Run sysstat a day or two. Afterwards, check computer workload history with sysstat commands sar, iostat, pidstat etc. Analyze the results, i.e. explain the results in detail.

Answer:

1. We shall connect to the server computer (running Ubuntu) and then install the required package with command sequence sudo apt-get update && sudo apt-get install -y install sysstat

2. Run shell-based sysstat script which runs sar and pidstat commands for two days with 20 second intervals.

NOTE: The script can be opmitized more/adapted better to suit the real requirements for the server environment (statistics interval, collection period, etc.)

3. The script described in step 2 has generated sar analytics file sar-stats_2018-02-24_2018-02-26.file into path /home/newuser/sar_statistics/ on the server computer.

Time period: 24. - 26.02.2018

The following sums up some command samples which can be applied to the analytics file.

File	Description
sar -u -f	Processor workload statistics
sar -v -f	Inode and file statistics
sar -r -f	Memory consumption statistics
sar -n DEV -f	Network stats: devices
sar -n EDEV -f	Network stats: errors in devices
sar -n IP -f	Network stats: IPv4 traffic
sar -n EIP -f	Network stats: errors in IPv4 traffic
sar -n IP6 -f	Network stats: IPv6 traffic
sar -n EIP6 -f	Network stats: errors in IPv6 traffic
sar -n SOCK -f	Network stats: IPv4 socket
sar -n SOCK6 -f	Network stats: IPv6 socket
sar -n TCP -f	Network stats: TCPv4 protocol traffic
sar -n ETCP -f	Network stats: errors in TCPv4 protocol traffic
sar -S -f	Swap memory consumption
sar -w -f	Process statistics
sar -F MOUNT / -f	File system which is mounted at /

Statistics can be combined etc, as you can find out with 'man sar' command:

sar -u 2 5
Report CPU utilization for each 2 seconds. 5 lines are displayed.
sar -I 14 -o int14.file 2 10
Report statistics on IRQ 14 for each 2 seconds. 10 lines are displayed.  Data are stored in a file called int14.file.
sar -r -n DEV -f /var/log/sysstat/sa16
Display memory and network statistics saved in daily data file 'sa16'.
sar -A
Display all the statistics saved in current daily data file.

Or The Geek Stuff - 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring

What are inode and swap? Check

• inode - 1

• inode - 2

• Swap

Additionally, the following pidstat files were generated:

File	Description
pidstat_stats_cpu-tasks	Processor workload statistics
pidstat_stats_io	I/O statistics
pidstat_stats_kerneltables	Statistics of Linux kernel tables
pidstat_stats_pagefaults	Page fault statistics
pidstat_stats_stacks	Process stack statistics

(Check. Stacks, Page fault)

Additionally, iostat command was run on the background.

4. Let's take a closer look on two sar analytics files and I/O statistics file. The following commands open analytics files in more detailed view.

---

SAR - network statistics, IPv4 traffic

command: sar -n IP -f sar-stats_2018-02-24_2018-02-26.file

Observation period: 24.-26.02.2018

Field	Description	Average
Left column	Record time
irec/s	The total number of input datagrams received from interfaces per second, including those received in error.	1.33
fwddgm/s	The number of input datagrams per second, for which this entity was not their final IP destination, as a result of which an attempt was made to find a route to forward them to that final destination.	0.00
idel/s	The total number of input datagrams,successfully,delivered,per,second,to,IP user-protocols (including ICMP).	1.30
orq/s	The total number of IP datagrams which local IP user-protocols (including ICMP) supplied per second to IP in requests for,transmission. Note that this,counter,does,not include any datagrams counted in fwddgm/s.	1.45
asmrq/s	The number of IP fragments received per second which needed to be reassembled at this entity.	0.00
asmok/s	The number of IP datagrams successfully re-assembled per second.	0.00
fragok/s	The number of IP datagrams that,have,been,successfully fragmented at this entity per second.	0.00
fragcrt/s	The number of IP datagram fragments that have been generated per second as a result of fragmentation at this entity.	0.00

NOTE: Descriptions are provided in sysstat package (manpages).

Analysis - IPV4

The server computer has not forwarded a single datagram in the observation period. Input network traffic has been received which can be concluded by observing irec/s, idel/s and orq/s field values. The current workload of the server computer is very low, including only HTTP daemon (Apache web server) and SSH server daemon. No webpages are available on the server which could increase input HTTP/HTTPS protocol based traffic. HTTPS was not enabled in the server computer.

By default, the server computer allows 128 simultaneous input network connections (command cat /proc/sys/net/core/somaxconn. NOTE: The value can be changed by issuing echo 1024 | sudo tee /proc/sys/net/core/somaxconn, for instance).

MTU value of the web server computer for network interface eth0 was 1500 (you can check it with ifconfig command). MTU stands for Maximum Transmission Unit which determines the highest PDU unit (Protocol Data Unit) that can be transferred over the network at once.

Check also

• MTU - hyperlink 1

• MTU - hyperlink 2

• Stack Overflow - What's the practical limit on the size of single packet transmitted over domain socket?

• Stack Overflow - What is the default size of datagram queue length in Unix Domain Sockets (AF_UNIX)? Is it configurable?

• IP fragmentation

• Related article: Linux Tune Network Stack (Buffers Size) To Increase Networking Performance

NOTE: About datagrams, quoted from Wikipedia:

The term datagram is often considered synonymous to packet but there are some nuances. The term datagram is generally reserved for packets of an unreliable service, which cannot notify the sender if delivery fails, while the term packet applies to any packet, reliable or not. Datagrams are the IP packets that provide a quick and unreliable service like UDP, and all IP packets are datagrams; however, at the TCP layer, what is termed a TCP segment is the sometimes necessary IP fragmentation of a datagram, but those are referred to as "packets".

---

SAR - memory consumption statistics, RAM & Swap

command: sar -r -f sar-stats_2018-02-24_2018-02-26.file command: sar -S -f sar-stats_2018-02-24_2018-02-26.file

Observation period: 24.-26.02.2018

NOTE: Average values are not visible in the attached picture!

Field	Description	Average
Left column	Record time
kbmemfree	Amount of free memory available in kilobytes.	87631 KB (~ 87 MB)
kbmemused	Amount,of,used,memory in kilobytes. This does not take into account memory used by the kernel itself.	928 457 KB (~ 928 MB)
%memused	Percentage of used memory.	91.38 %
kbbuffers	Amount of memory used as buffers by the kernel in kilobytes.	77 746 KB (~ 77 MB)
kbcached	Amount of memory used to cache data by the kernel in kilobytes.	644 777 KB (~ 644 MB)
kbcommit	Amount of memory in kilobytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory.	470 619 KB (~ 470 MB)
%commit	Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap).,This number may be greater than 100% because the,kernel,usually overcommits memory.	46.32 %
kbactive	Amount of active memory in kilobytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).	468 703 KB (~ 468 MB)
kbinact	Amount of inactive memory in kilobytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).	301 098 KB (~ 301 MB)
kbdirty	Amount of memory in kilobytes waiting to get written back to the disk.	254 KB (0.254 MB)

NOTE: Descriptions are provided in sysstat package (manpages).

Analysis - Memory Statistics

The target server memory size:

[newuser@goauldhost: ~ ]$ grep -i memtotal /proc/meminfo
MemTotal:        1016088 kB

i.e. approximately ~ 1016 MB.

Great part of this memory were in use during the observation period. There were multiple active processes running on the server computer during the period, thus increased workload was intended and meant for statistics collection.

The web server didn't have Swap partition or Swap file. This can be found out by

• executing command for i in "cat /etc/fstab" "sudo fdisk -l"; do $i | grep -i swap; echo $?; done which returns value 1 two times (meaning returning of 'false' boolean value to shell two separate times)

• Or, for instance:

[newuser@goauldhost: ~ ]$ cat /etc/fstab
LABEL=cloudimg-rootfs	/	 ext4	defaults	0 0
LABEL=UEFI	/boot/efi	vfat	defaults	0 0
[newuser@goauldhost: ~ ]$ sudo fdisk -l
[sudo] password for newuser: 
Disk /dev/vda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 39DFE5D0-C8FB-44D8-93F8-EBB37A54BDF8
Device      Start      End  Sectors  Size Type
/dev/vda1  227328 52428766 52201439 24.9G Linux filesystem
/dev/vda14   2048    10239     8192    4M BIOS boot
/dev/vda15  10240   227327   217088  106M Microsoft basic data

It may not be wise to collect Swap statistics (although Linux kernel Swappiness value has default value 60 defined in /proc/sys/vm/swappiness on DigitalOcean virtual servers).

---

SAR - I/O statistics

command: iostat -dmtx 20

-d     Display the device utilization report.
-m     Display statistics in megabytes per second.
-t     Print the time for each report displayed.
-x     Display extended statistics.
20     20 sec interval.

Field	Description
Device	Device or partition defined in system directory '/dev'.
rrqm/s	The number of read requests merged per second that were queued to the device.
wrqm/s	The percentage of write requests merged together before being sent to the device.
r/s	The number (after merges) of read requests completed per second for the device.
w/s	The number (after merges) of write requests completed per second for the device.
rMB/s	The number of sectors (KB, MB) read from the device per second.
wMB/s	The number of sectors (KB, MB) write from the device per second.
avgrq-sz (areq-sz)	The average size (in kilobytes) of the I/O requests,that were issued to the device.
avgqu-sz (aqu-sz)	The average queue length of the requests that were issued to the device.
await	The average time (ms) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
r_await	The average time (ms) for read requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
w_await	The average time (ms),for,write,requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
svctm	The average service time (ms) for I/O requests that were issued to the device. This field will be removed in a future sysstat version.
%util	Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.

NOTE: Descriptions are provided in sysstat package (manpages).

Analysis - I/O Statistics

Virtual partition /dev/vda were hibernated most of the time during the observation period. Little variation can be seen in writing requests (wrqm/s), in succeeded writing requests (w/s) and in written sector count (wMB/s). Average sector size is 10-20 sectors (avgrq-sz) which is quite little. I/O and read request service times are mostly 0.0. Workload of the virtual hard drive is low during the observation period.

---

In addition to sysstat, top command could have been used in some TTY session in order to observe processes in real time, and with inotify, some file system changes could have been reported. Sysstat's advantage is that programs which are included in it can record and save analytics in binary format for further and detailed inspection.

f) (optional) Change sshd (SSH server process) port

Answer:

Let the following shell script do the job...

#!/bin/bash
# SSH server daemon configuration file in the system
SSH_CONFIG=/etc/ssh/sshd_config
# New SSH server daemon input port as user input value.
NEW_SSH_PORT=$1
[[ -f $SSH_CONFIG ]] \
&& sed -i "s/.*Port.*/Port $NEW_SSH_PORT/" $SSH_CONFIG && echo "SSH server: new port $NEW_SSH_PORT is set." \
|| echo "SSH server configuration file could not be found!"
if [[ $(cat $SSH_CONFIG | grep -i port | awk '{print $2}') == $NEW_SSH_PORT ]]; then
echo -e "SSH server input port has been changed to $NEW_SSH_PORT.\n \
Restarting SSH server daemon in order to apply the changes (root required)."
sudo systemctl restart sshd.service
if [[ $? == 0 ]]; then
echo "SSH server daemon restarted, new input port is $NEW_SSH_PORT."
else
echo "Something went wrong while restarting SSH server daemon. Execute 'systemctl status sshd.service' \
to get more information about the problem."
fi
fi

Save the above script code in file $HOME/ssh-port.sh, for instance. Change the port with command bash $HOME/ssh-port.sh 4312 where the number value is your new SSH port (4312 in this case).

---

EXTRA - Using new port address of SSH server daemon when connecting with a client computer/program

Changing SSH server input port on the server computer must be taken into account while establishing connection with a client computer. Because the default SSH port 22 is not used anymore, the following syntax must be applied while connecting to the SSH server computer:

[19/02/2018 23:23:49 - fincer: ~ ]$ ssh newuser@174.138.2.190 -p <new-port-number>

---

EXTRA - detecting SSH port change with port scanning techniques (nmap)

Enable log level VERBOSE in /etc/ssh/sshd_config configuration file (LogLevel VERBOSE), restart SSH server daemon with command sudo systemctl restart sshd.service and try port scanning with another computer.

Port scanning:

nmap -sS -F -v -O 174.138.2.190

nmap output when port scanning is applied with another computer using target IP 174.138.2.190 and port 1234 (NOTE: This scanning has been applied to my own server only once as an experiment, not in bad intentions! Make sure you have permission to proceed with your scanning and do not attack to any server just for fun, to avoid any problems!):

phelenius@my-machine:~$ sudo nmap -sS -p 1234 -O -v 174.138.2.190
Starting Nmap 7.01 ( https://nmap.org ) at 2018-02-19 20:02 EET
Initiating Ping Scan at 20:02
Scanning 174.138.2.190 [4 ports]
Completed Ping Scan at 19:59, 0.20s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 20:02
Completed Parallel DNS resolution of 1 host. at 20:02, 0.01s elapsed
Initiating SYN Stealth Scan at 20:02
Scanning 174.138.2.190 [1 port]
Completed SYN Stealth Scan at 20:02, 0.20s elapsed (1 total ports)
Initiating OS detection (try #1) against 174.138.2.190
adjust_timeouts2: packet supposedly had rtt of -125997 microseconds.  Ignoring time.
adjust_timeouts2: packet supposedly had rtt of -125997 microseconds.  Ignoring time.
Retrying OS detection (try #2) against 174.138.2.190
adjust_timeouts2: packet supposedly had rtt of -150518 microseconds.  Ignoring time.
adjust_timeouts2: packet supposedly had rtt of -150518 microseconds.  Ignoring time.
WARNING: OS didn't match until try #2
Nmap scan report for 174.138.2.190
Host is up (0.00041s latency).
PORT   STATE    SERVICE
1234/tcp open  unknown
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: switch|general purpose|media device
Running: Cisco CatOS 7.X|8.X, HP Tru64 UNIX 5.X, Vantage embedded
OS CPE: cpe:/h:cisco:catalyst_ws-c6506 cpe:/o:cisco:catos:7.6 cpe:/o:cisco:catos:8.3 cpe:/o:hp:tru64:5.1a cpe:/h:vantage:hd7100s
OS details: Cisco Catalyst WS-C6506 switch (CatOS 7.6(16)), Cisco Catalyst switch (CatOS 8.3(2)), HP Tru64 UNIX 5.1A, Vantage HD7100S satellite receiver
Read data files from: /usr/bin/../share/nmap
OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 3.39 seconds
Raw packets sent: 44 (6.312KB) | Rcvd: 18 (1.820KB)

If SSH server uses the default input port 22, the following is likely to be presented in output of nmap command (unless you've defined a particular port to be scanned in your nmap command, like port 1234):

PORT   STATE    SERVICE
22/tcp filtered ssh

Corresponding log entries of targeted SSH server during the nmap scanning (/var/log/auth.log):

Feb 19 18:02:46 goauldhost sshd[30057]: Connection from XXX.XXX.XXX.XXX port 6967 on 174.138.2.190 port 1234
Feb 19 18:02:46 goauldhost sshd[30057]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:46 goauldhost sshd[30058]: Connection from XXX.XXX.XXX.XXX port 52205 on 174.138.2.190 port 1234
Feb 19 18:02:46 goauldhost sshd[30058]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30059]: Connection from XXX.XXX.XXX.XXX port 25326 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30059]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30060]: Connection from XXX.XXX.XXX.XXX port 32812 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30060]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30061]: Connection from XXX.XXX.XXX.XXX port 17024 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30061]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30062]: Connection from XXX.XXX.XXX.XXX port 53268 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30062]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30063]: Connection from XXX.XXX.XXX.XXX port 34923 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30063]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30064]: Connection from XXX.XXX.XXX.XXX port 14489 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30064]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30065]: Connection from XXX.XXX.XXX.XXX port 40086 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30065]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:47 goauldhost sshd[30066]: Connection from XXX.XXX.XXX.XXX port 38147 on 174.138.2.190 port 1234
Feb 19 18:02:47 goauldhost sshd[30066]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30067]: Connection from XXX.XXX.XXX.XXX port 49215 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30067]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30068]: Connection from XXX.XXX.XXX.XXX port 34445 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30068]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30069]: Connection from XXX.XXX.XXX.XXX port 4600 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30069]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30070]: Connection from XXX.XXX.XXX.XXX port 59405 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30070]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:48 goauldhost sshd[30071]: Connection from XXX.XXX.XXX.XXX port 7848 on 174.138.2.190 port 1234
Feb 19 18:02:48 goauldhost sshd[30071]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:49 goauldhost sshd[30072]: Connection from XXX.XXX.XXX.XXX port 5206 on 174.138.2.190 port 1234
Feb 19 18:02:49 goauldhost sshd[30072]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30073]: Connection from XXX.XXX.XXX.XXX port 5517 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30073]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30074]: Connection from XXX.XXX.XXX.XXX port 3970 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30074]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30075]: Connection from XXX.XXX.XXX.XXX port 38690 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30075]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30076]: Connection from XXX.XXX.XXX.XXX port 50572 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30076]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30077]: Connection from XXX.XXX.XXX.XXX port 27830 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30077]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:50 goauldhost sshd[30078]: Connection from XXX.XXX.XXX.XXX port 49371 on 174.138.2.190 port 1234
Feb 19 18:02:50 goauldhost sshd[30078]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30079]: Connection from XXX.XXX.XXX.XXX port 36802 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30079]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30080]: Connection from XXX.XXX.XXX.XXX port 50546 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30080]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30081]: Connection from XXX.XXX.XXX.XXX port 43542 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30081]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30082]: Connection from XXX.XXX.XXX.XXX port 56108 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30082]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30083]: Connection from XXX.XXX.XXX.XXX port 6399 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30083]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30084]: Connection from XXX.XXX.XXX.XXX port 55980 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30084]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30085]: Connection from XXX.XXX.XXX.XXX port 12713 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30085]: Did not receive identification string from XXX.XXX.XXX.XXX
Feb 19 18:02:51 goauldhost sshd[30086]: Connection from XXX.XXX.XXX.XXX port 5026 on 174.138.2.190 port 1234
Feb 19 18:02:51 goauldhost sshd[30086]: Did not receive identification string from XXX.XXX.XXX.XXX

The server computer has logged invalid nmap connection requests. In this case, we have defined the known SSH connection port number 1234 in our nmap command on the client computer (which is INPUT port at the server end, OUTPUT port at the client end).

You may check suggested countermeasures against port scanners on Unix & Linux Stack Exchange. Does these measures interfere negatively to other network traffic in busy server environments? This depends on which ports (read: server daemon programs/services) are targets of your countermeasures.

Another nmap command shows us the following:

phelenius@my-machine:~$ sudo nmap 174.138.2.190 -sV
Starting Nmap 7.01 ( https://nmap.org ) at 2018-02-20 14:50 EET
Nmap scan report for 174.138.2.190
Host is up (0.33s latency).
Not shown: 998 filtered ports
PORT     STATE SERVICE VERSION
1234/tcp open  ssh     OpenSSH 7.6 (protocol 2.0)
80/tcp   open  http
...

... which reveals the actual service behind our new-defined port 1234/tcp to an attacker. In addition, the attacker can get more information about the server system, especially on Debian-based Linux distributions. For instance:

...
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 7.2p2 Ubuntu 4ubuntu2.4 (Ubuntu Linux; protocol 2.0)
...

This information can be hidden by adding line DebianBanner no in /etc/ssh/sshd_config on Debian-based Linux systems and restarting the SSH server daemon with command sudo systemctl restart sshd.service afterwards

NOTE: Port scanning does not leave any log traces behind in Apache's access.log file (/var/log/apache/access.log)!

Check also

• MyPapit GNU/Linux - How to Hide OpenSSH Ubuntu version from Nmap and other scanners

• serverfault.com - How to hide web server name and openssh version on linux when scanning server ports?

• Unix & Linux Stack Exchange - Change SSH banner which is grabbed by netcat

• GitHub - metacloud/openssh - Include the Debian version in our identification

---

EXTRA - Using Port Knocking technique against port scanning

Nmap requests are targeted to layer 3 (Network Layer) in OSI model. Additional security measures can be taken by applying Port Knocking login techniques on the server computer.

WARNING! The following Knockd daemon may break easily. Thus it can not be recommended for critical or rapidly updating server environments. For instance, dhcpcd version 7 or above breaks knockd daemon. Many users have had problems to start the daemon service during server boot-up process. Knockd assumes that you have a preconfigured network interface with established network connection (this is not 100% valid information but this is my best guess).

DigitalOcean, "Port Knocking" - Ubuntu, knockd:

Dynamic firewall rules are manually applied to the server (iptables, for instance) or separate daemon process (for instance, knockd) is set up on the server computer. Daemon process is used to open a pre-defined port only if a client program has knocked correct server port numbers in right order and using correct protocol, either TCP or UDP, and this all has been done in predefined time limit. Additionally, limited timeout for user log-in can be set. For instance, a correctly authenticated user must log in in 10 seconds ater Knock daemon has opened a SSH port for that time period. When the time limit is reached, Knock daemon closes the port for any further connection attempts unless the knock port sequence is re-applied.

Knocking technique must be configured so that firewall does not close already established connections (like SSH in our case), but refuses any further connection attempts so that the port can not be detected with nmap port scanning techniques after the time limit has been reached.

DigitalOcean, "Port Knocking" - FWKnop - Single packet authentication:

Some more or less secure implementations for port knocking exist. Various encryption methods can be applied to the knocking technique, making it more difficult for an attacker to sniff network packets between a server and a client. One solution using encyption methods is fwknop which uses Single Packet Authorization (SPA) method instead of insecure port sequences.

More about Port Knocking technique:

• Improved Port Knocking with Strong Authentication - Rennie deGraaf, John Aycock, and Michael Jacobson, Jr.∗ (Department of Computer Science - University of Calgary)

• OpenWRT website

• portknocking.org and practical solutions on Implementations sub-page

• Information Security Stack Exchange - Port Knocking is it a good idea?

NOTE: Port Knocking technique must not be applied to block ports of server daemon processes which are required to be always reachable from a client program (take Apache server for an example). For SSH which does not require 24/7/365 connection, port knocking techniques can be applied more reliably. In addition to public key, password and PAM authentication methods, port knocking adds one more security layer to the stack. However, you must not apply these security countermeasures individually but use them together, instead. The goal is to make it more difficult for an attacker to penetrate your network security.

---

EXTRA - ARP Scan and spoofing your MAC address

Program arp-scan can be used in limited scale to scan a MAC address (OSI model layer 2, Data Link Layer) in a network.

Unique MAC address of a network interface (network card) can programmatically be spoofed with these Arch Wiki instructions or with my Spoof MAC Address shell script.

Sample results of ARP Scan.

BEFORE:

[20/02/2018 18:52:35 - fincer: ~ ]$ sudo arp-scan --interface=wlan0 --localnet
[sudo] password for fincer: 
Interface: wlan0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
A.B.C.D         f4:0e:22:ad:b8:7d       (Unknown)
1.2.3.4         18:a9:05:4b:61:58       Hewlett-Packard Company

AFTER - MAC address of a computer having IP address 1.2.3.4 has been changed:

[20/02/2018 18:54:28 - fincer: ~ ]$ sudo arp-scan --interface=wlan0 --localnet
Interface: wlan0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
A.B.C.D         f4:0e:22:ad:b8:7d       (Unknown)
1.2.3.4         aa:0c:9a:fa:7b:d4       (Unknown)

The following warning messages may be expected after spoofing a MAC address and when you establish SSH connection from a local computer to your server:

Warning: Permanently added the ECDSA host key for IP address '[1.2.3.4]:22' to the list of known hosts.

g) (optional) Allow SSH login only for users in group 'sshers'. Add your account to this group.

Answer:

1. Log in to the server computer (ssh username@server-ip -p <ssh-port>)

2. Add group sshers with GID number 876 (you don't have to define GID here)

sudo groupadd -g 876 sshers

3. You can confirm existence of the created group with command grep sshers /etc/group. Output:

sshers:x:876:

4. Add the current user (read: yourself) newuser into this group

sudo usermod -aG sshers newuser

5. Command grep sshers /etc/group output now:

sshers:x:876:newuser

You can alternatively check groups of newuser by executing command groups while being that user:

[27/02/2018 10:39:58 - newuser@goauldhost: ~ ]$ groups
sshers newuser sudo

or using command sudo -u newuser groups as any system user who can execute commands with sudo.

NOTE: groups command gives you correct output only after re-login.

6. Allow SSH login only to members of the group sshers.

Manual page of sshd_config (man sshd_config) describes AllowGroups option as follows:

AllowGroups This keyword can be followed by a list of group name patterns, separated by spaces. If specified, login is allowed only for users whose primary group or supplementary group list matches one of the patterns. Only group names are valid; a numerical group ID is not recognized. By default, login is allowed for all groups. The allow/deny directives are processed in the following order: DenyUsers, AllowUsers, DenyGroups, and finally AllowGroups.

Let's apply the following information into SSH server daemon configuration file /etc/ssh/sshd_config:

PermitRootLogin no
AllowGroups sshers

And comment out the following lines (be careful here!):

# DenyUsers <user accounts>
# AllowUsers <user accounts>
# DenyGroups <group names>

If those lines are not defined in the configuration file, it is OK.

You can add multiple groups and users after those keywords, as you like.

For additional security, the following options can be applied as well (into /etc/ssh/sshd_config file), depending on your requests for SSH authentication policy:

IgnoreRhosts yes
PermitEmptyPasswords no
ChallengeResponseAuthentication yes
UsePAM yes
MaxAuthTries 3

etc. More options and configurations can be found with commands man sshd_config and man sshd

7. Let's save this configuration and restart SSH server daemon by applying command sudo systemctl restart sshd.service (NOTE! Disconnects already-established/existing SSH connections!)

8. Test SSH login with a client computer using remote user newuser (belonging to the remote group sshers). Command syntax is: ssh newuser@server-ip -p <server-port>

h) (optional) Attach a remote network directory with sshfs.

Answer:

SSHFS stands for SSH File System. Corresponding program sshfs has been developed to attach SSH server file systems (folders/files) remotely on a client computer so that all attached file systems can locally be browsed on the client computer. SSHFS uses encrypted SFTP protocol (SSH/Secure File Transfer Protocol) for file tranfers.

1. Install sshfs with command sequence sudo apt-get update && sudo apt-get -y install sshfs on a Debian-based Linux system.

2. Mount remote folder /home/newuser/public_html/ (server), to path /home/<current-user>/public_html_ssh_remote on your client computer.

mkdir $HOME/public_html_ssh_remote && sshfs newuser@174.138.2.190:./public_html $HOME/public_html_ssh_remote -p <ssh port>

NOTE: If you use public key SSH authentication method, you are not asked for any password.

3. Check that you have successfully mounted your remote server folder /home/newuser/public_html to the client computer (command ls ~/public_html_ssh_remote/). Output should be as follows:

index.html

Additionally, check output of command mount | grep public_html_ssh_remote:

newuser@174.138.2.190:./public_html on /home/phelenius/public_html_ssh_remote type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

4. Unmount the remote folder by issuing command sudo umount /home/phelenius/public_html_ssh_remote. On the client end, ls command should have empty output for directory public_html_ssh_remote.