In this blog post, I collect and share a few techniques commonly used to defend Apache2 web servers. Specifically, I will cover how to (1) mitigate banner grabbing attacks, (2) disable directory listing, (3) set-up a web application firewall, (4) enable HTTPS, and (5) configure an Intrusion Protection System. All steps outlined here are under the assumption you have an actively running Apache2 instance with a default (Ubuntu-based) configuration.

How to Mitigate Banner Grabbing Attacks

Attackers follow a methodology that begins with reconnaissance. In order to deny them actionable intelligence, we can first limit a few settings that share information about your Apache2 instance. Before securing these settings, let me prove to you this is a vulnerability.

From a terminal on your web server, run the following command.

curl localhost/foo

You should get a response similar to the one below.

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /foo was not found on this server.</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at localhost Port 80</address>
</body></html>

The second to the last line identifies what software version, operating system (OS), and port we’re using to run Apache2 (bad). To stop this from happening, we’re going to modify two parameters in our main .conf file: ServerTokens and ServerSignature. ServerTokens can be set to one of multiple options with Prod (Production) being the least verbose. This parameter addresses what is returned in Server HTTP responses while ServerSignature specifies whether or not to share your Apache2 version.

sudo vim /etc/apache2/apache2.conf
# append the two lines below to your apache2.conf file
ServerTokens Prod
ServerSignature Off 

Reload your modified .conf file using the command below.

sudo systemctl reload apache2

Now, if we execute the same web request, we get back less information (good).

curl localhost/foo
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /foo was not found on this server.</p>
</body></html>

Another way to see how much information is being shared in Server HTTP responses is to monitor outgoing traffic using tcpdump. For example, the command sentence below will tell tcpdump to use a local loopback (lo) interface (-i) to dump (in hexadecimal and ASCII format; -XX) packets with the source port of 80 (src port 80; modify if you’re using a non-default port for your web server).

sudo tcpdump -i lo src port 80 -XX

How to Disable Directory Listing

Another simple way attackers can foot-print your Apache2 web server is Directory Listing. For example, in the absence of an index.html file, your server will resort to listing all files in the directory being queried (bad).

To replicate this vulnerability, let’s temporarily move our index.html file to another directory and position a few junk files in its place.

sudo mv /var/www/html/index.html /home/

Use the command sentence (for loop) below to create (touch) three new, empty files under the directory Apache2 uses to serve web content by default.

for number in {1..3}; do sudo touch /var/www/html/file$number; done

Again, we can use a terminal on our web server to perform a simple web request. Since index.html is absent, our response will be a directory listing.

curl localhost
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>
   <tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="file1">file1</a></td><td align="right">2019-02-01 12:32  </td><td align="right">  0 </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="file2">file2</a></td><td align="right">2019-02-01 12:32  </td><td align="right">  0 </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="file3">file3</a></td><td align="right">2019-02-01 12:32  </td><td align="right">  0 </td><td>&nbsp;</td></tr>
   <tr><th colspan="5"><hr></th></tr>
</table>
</body></html>

Now, that we’ve seen this vulnerability exploited, let’s modify our main .conf file to mitigate it from happening. Specifically, we will need to turn-off the setting that allows Indexing under our default directory (/var/www/html/).

sudo vim /etc/apache2/apache2.conf

First, find the section addressing the directory in question.

<Directory /var/www/>
	Options Indexes FollowSymLinks
</Directory>

Then, comment-out the default setting and append a similar, but modified one like below.

<Directory /var/www/>
	# Options Indexes FollowSymLinks
	Options -Indexes +FollowSymLinks
</Directory>

Here, we’re telling Apache2 to disable (-) Indexing under the /var/www/ directory while still enabling (+) Symbolic Links to be followed (think shortcuts, like in Microsoft Windows environments). You may get an error if you only use - so ensure to use the same method for all options when identifying what you want enabled and disabled.

After you save the modified .conf file, reload Apache2 again and repeat your previous web request.

sudo systemctl reload apache2
curl localhost
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access / on this server.<br />
</p>
</body></html>

How to Set-up a Web Application Firewall

Modsecurity is an Apache2 module commonly used as a Web Application Firewall (WAF). To install it, use the command below (the exact package name will vary based on your OS).

sudo apt-get install libapache2-modsecurity

Use the following command sentence to verify it was installed.

sudo apachectl -M | grep security
security2_module (shared)	# <-- this is Modsecurity

To explain, you’re using the Apache2 control (apachectl) interface to dump/list all modules (-M) onto the screen and then, using grep to Globally search for the Regular Expression security and Print the results. Next, change directories to start configuring Modsecurity.

cd /etc/modsecurity/

We need to first rename the provided .conf file so Modsecurity recognizes it as our main source of configuration. We can do that by simply making a copy, but without the -recommended suffix in the file-name.

sudo cp modsecurity.conf-recommended modsecurity.conf

Now, we need to download the Core Rule Set (CRS) Modsecurity will use to perform it’s firewall duties. Execute the commands below.

sudo git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git
sudo cp owasp-modsecurity-crs.git/crs-setup.conf.example /etc/modsecurity/crs-setup.conf
sudo mv owasp-modsecurity-crs/rules/ .

To explain, we cloned (git clone) the CRS archive hosted on the Open Web Application Security Project (OWASP) team’s Github page. Then, we made a copy of the provided .conf file, naming it crs-setup.conf. Finally, we moved (mv) all the provided rules into our current working directory (.) which should be /etc/modsecurity/.

By default, our “Rule Engine” is set to operate in DetectionOnly mode. We can ask it to also enforce these rules by appending the argument SecRuleEngine On like below.

sudo vim /etc/modsecurity/modsecurity.conf
SecRuleEngine DetectionOnly
SecRuleEngine On

Lastly, we need to make sure when Modsecurity starts running on behalf of Apache2, it includes the rules we downloaded from the OWASP team. We can do so by editing the .conf file Apache2 has for Modsecurity and appending Include /etc/modsecurity/rules*.conf as seen below.

sudo vim /etc/apache2/mods-available/security2.conf
<IfModule security2_module>
        # Default Debian dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        # Include all the *.conf files in /etc/modsecurity.
        # Keeping your local configuration in that directory
        # will allow for an easy upgrade of THIS file and
        # make your life easier
        IncludeOptional /etc/modsecurity/*.conf
	Include /etc/modsecurity/rules/*.conf # <-- add this line
</IfModule>

Once complete, restart Apache2.

sudo systemctl restart apache2

To monitor Modsecurity alerts in real-time, put a tail on the Apache2 error.log like below.

sudo tail -f /var/log/apache2/error.log

Then, execute the following command sentence in a separate terminal on your web server to test Modsecurity.

curl localhost/index.html?exec=/bin/bash

You should get a few alerts that look similar to the one below.

[Sat Feb 02 05:27:40.811869 2019] [:error] [pid 5100:tid 140073513252608] [client 127.0.0.1] ModSecurity: Warning. Matched phrase "bin/bash" at ARGS:exec. [file "/etc/modsecurity/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "518"] [id "932160"] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "CRITICAL"] [ver "OWASP_CRS/3.1.0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "localhost"] [uri "/index.html"] [unique_id "XFUqTH8AAQEAABPs@5YAAABF"]

To address false positives, you can disable rules based on their message. Simply add an argument to the same .conf file we just modified. Below is an example disabling a rule that generates the message PHP Information Leakage.

sudo vim /etc/apache2/mods-available/security2.conf
<IfModule security2_module>
        # Default Debian dir for modsecurity's persistent data
        SecDataDir /var/cache/modsecurity

        # Include all the *.conf files in /etc/modsecurity.
        # Keeping your local configuration in that directory
        # will allow for an easy upgrade of THIS file and
        # make your life easier
        IncludeOptional /etc/modsecurity/*.conf
	Include /etc/modsecurity/rules/*.conf

	# Disabling a rule generating false positive alerts
	SecRuleRemoveByMsg 'PHP Information Leakage' # <-- add this line
</IfModule>

As before, make sure to restart Apache2 afterwards.

sudo systemctl restart apache2

How to Enable HTTPS

Hyper Text Transfer Protocol Secure, or HTTPS, is a confidential way of exchanging web requests & content. You are able to invoke HTTPS because of entities called Certificate Authorities (CAs). CAs generate “digital certificates” which are then used to faciliate encryption, decryption, and mutual authentication between web servers and their clients.

One free & automated CA is Let’s Encrypt. Let’s Encrypt makes enabling HTTPS painless, mostly because of a script it provides called certbot. certbot works by first, evaluating your web server configuration. Then, it performs a series of challenges Let’s Encrypt requires in order to receive a digital certificate. For Apache2 instances, certbot uses your virtual host configuration for majority of the process. To demonstrate, let’s go through these motions as if we’re preparing a website called https://www.sky.net.

First, we need to (1) make the parent directory we plan to serve our content from, (2) configure our permissions, and then, (3) create a simple home page.

Making a Directory to Serve Web Content

sudo mkdir -p /var/www/sky.net/html/

Configuring Permissions

# this command depends on the existence of a root account & group
sudo chown -R root:root /var/www/sky.net/html/
sudo chmod -R 755 /var/www/sky.net/

Creating a Simple Home Page

sudo vim /var/www/sky.net/index.html
<html>
<head>
<title>Welcome to Sky.net!</title>
</head>
<body>
<h1>Hello world!</h1>	
</body>
</html>

Next, we need to (1) create a virtual host configuration file for our site, (2) enable our site, (3) disable the default configuration, (4) run a configtest to verify our parameters, and then, (5) restart our Apache2 server.

How to Create a Virtual Host Configuration File

sudo vim /etc/apache2/sites-available/sky.net.conf
<VirtualHost *:80>
	ServerAdmin alex@sky.net
	ServerName sky.net
	ServerAlias www.sky.net
	DocumentRoot /var/www/sky.net/html
	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

How to Enable a Site

sudo a2ensite sky.net.conf

How to Disable a Site

# not running this command prevented me from obtaining an SSL cert!
sudo a2dissite 000-default.conf 
Enabling site sky.net.
To activate the new configuration, you need to run:
  service apache2 reload

How to Verify Your Virtual Host Configuration File Parameters

sudo apache2ctl configtest
Syntax OK

How to Restart Your Apache2 Server

sudo systemctl restart apache2

Now that we have our website setup, let’s (1) add the certbot repository to our package manager, (2) download certbot, and then, (3) tell certbot to help us generate digital certificates.

How to Add a Repo to Your Package Manager

sudo add-apt-repository ppa:certbot/certbot
 This is the PPA for packages prepared by Debian Lets Encrypt Team and backported for Ubuntu(s).
 More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpldp1mk7v/secring.gpg' created
gpg: keyring `/tmp/tmpldp1mk7v/pubring.gpg' created
gpg: requesting key 75BCA694 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpldp1mk7v/trustdb.gpg: trustdb created
gpg: key 75BCA694: public key "Launchpad PPA for certbot" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

How to Download cerbot

sudo apt-get update && sudo apt-get install python-certbot-apache

How to Generate Digital Certificates Using certbot
Beware: the following command will enable the mod_rewrite Apache2 module. Also, this command will fail if you did not create a Host A record for your root domain and CNAME record for your sub-domain (www).

sudo certbot --apache -d sky.net -d www.sky.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.sky.net
http-01 challenge for sky.net
Enabled Apache rewrite module
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/sky.net-le-ssl.conf
Enabled Apache socache_shmcb module
Enabled Apache ssl module
Deploying Certificate to VirtualHost /etc/apache2/sites-available/sky.net-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/sky.net-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/sky.net-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you are confident your site works on HTTPS. You can undo this
change by editing your web server configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Enabled Apache rewrite module
Redirecting vhost in /etc/apache2/sites-enabled/sky.net.conf to ssl vhost in /etc/apache2/sites-available/sky.net-le-ssl.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://sky.net and
https://www.sky.net

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=sky.net
https://www.ssllabs.com/ssltest/analyze.html?d=www.sky.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/sky.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/sky.net/privkey.pem
   Your cert will expire on 2019-05-09. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Lets Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If all went well, you should now also have a scheduled task (a.k.a cronjob) telling certbot to renew your digital certificates if they’re about to expire.

sudo cat /etc/cron.d/cerbot
# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
#
# Important Note!  This cronjob will NOT be executed if you are
# running systemd as your init system.  If you are running systemd,
# the cronjob.timer function takes precedence over this cronjob.  For
# more details, see the systemd.timer manpage, or use systemctl show
# certbot.timer.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

How to Configure an Intrusion Protection System

Fail2ban prevents malicious activity by reading your logs, and updating your firewall automatically. Out of the box, it is able to protect against brute force attacks against common services like Apache and SSH. Use the command below to install it.

sudo apt-get install fail2ban

The primary elements of Fail2ban are Actions, Filters, and Jails. Actions represent what can be done about the IP addresses responsible for malicious activity. Filters are the parameters Fail2ban uses to search for said activity. Jails define the criteria for getting banned, the ultimate fix-action for folks violating your security policy. If you chose to modify or customize any of these aspects, it is recommended you create a .local file in the appropiate directory as opposed to editing the monolithic .conf files.

/etc/fail2ban/action.d/
/etc/fail2ban/filter.d/
/etc/fail2ban/jail.d/

As an example, I prefer to keep malicious IP addresses in jail forever (cannot be unbanned, unblocked, re-enabled, etc.). So I created a .local file specifying the bantime as -1, meaning malicious IPs will unbanned minus one second before they are banned (a paradox).

sudo vim /etc/fail2ban/jail.d/turn_off_unban.local
[DEFAULT]
bantime = -1

Again, your bantime is just one configurable setting for your jails. I specified this setting to be default, or applied to all services. You can specify unique jail criteria for each service you have running. For instance, let’s create a .local file to define custom jail criteria for those who attack your Apache2 service.

sudo vim /etc/fail2ban/jail.d/apache2.local

Copy & paste the following into your /etc/fail2ban/jail.d/apache2.local file.

[apache]
enabled = true
port = http,https
filter = apache-auth
logpath = /var/log/apache2/*error.log
maxretry = 3

[apache-overflows]
enabled = true
port = http,https
filter = apache-overflows
logpath = /var/log/apache2/*error.log

[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache2/*error.log

[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
logpath = /var/log/apache2/*error.log

To explain, we’re telling Fail2ban to enable four jails for IP addresses attacking our HTTP & HTTPS ports (80, 443). Malicious activity found in any error.log under the /var/log/apache2/ directory matching our either one of our four predefined filters (apache-auth,apache-overflows,apache-noscript,apache-badbots) will be banned.

As another demonstration, let’s ask Fail2ban to defend against Denial of Service attacks aimed at our Apache2 instance. First, create the following filter.

sudo vim /etc/fail2ban/filter.d/apache-dos.conf
[Definition]
failregex = ^<HOST> -.*"(GET|POST).*
ignoreregex =

Then, append the following jail criteria to your /etc/fail2ban/jail.d/apache.local file.

[apache-dos]
enabled = true
port = http,https
filter = apache-dos
logpath = /var/log/apache2/access.log
maxretry = 3
findtime = 6
action = iptables[name=HTTP, port=http, protocol=tcp]

Finally, reload Fail2ban.

sudo systemctl reload fail2ban

To explain, we just enabled a new jail, called [apache-dos]. Any IP address sending more than three HTTP GET and/or POST requests will be banned. This works thanks to the broad failregex (Fail2ban Regular Expression) we defined as our filter and firewall rule (iptables=[name=HTTP, port=http, protocol=tcp) we included as our action.

Let’s test this filter by first putting a tail on our web server’s /var/log/fail2ban.log file.

sudo tail -f /var/log/fail2ban.log

Next, execute the following command from system separate from your web server. Replace https://www.sky.net/ with the IP address or URL of your web server.

sudo ab -n 500 -c 10 https://www.sky.net/

To explain, we’re using the Apache HTTP Server Benchmark tool (ab) to send 500 HTTP requests across 10 connections to https://www.sky.net/. Immediately, you should get output similar to below in the terminal window following your fail2ban.log file.

2019-02-16 13:22:58,960 fail2ban.filter         [16812]: INFO    [apache-dos] Found 12.34.56.78 - 2019-02-16 13:22:58
2019-02-16 13:22:58,999 fail2ban.actions        [16812]: NOTICE  [apache-dos] Ban 12.34.56.78

To manually unban an IP address, use the following command.

sudo fail2ban-client unban 12.34.56.78

Summary

To summarize, I just explained how to harden an Apache2 configuration, how to use Modsecurity as a WAF, how to facilitate HTTPS using Let’s Encrypt, and how to put malicious actors in cyber-jail using Fail2ban. Granted, these steps are not all that can be done to protect your web server. Yet, it should be enough to inspire you to determine how you’re being compromised, develop countermeasures, and then, defend your web presence.

References