Diagnosing and Mitigating Layer 7 (Application Layer) Distributed Denial of Service Attacks (Microcache/F2B/PHP5-FPM)

So, recently I began researching ways of hosting a CMS controlled site manually to cope with a high legitimate request load or a Layer 7 DDoS attack (Application Layer of the OSI Model). I have been the recipient of such attacks before which instantly gave out the “Resource Limit Exceeded” error on any shared host I hosted this site on.

Finding hosts with Layer 7 mitigation already in place is very, very rare. Transits like InterNap or Layer3 only provide Layer 4 mitigation and load balancing. BlazingFast is one company that has a reputable 300Gbps network in Europe (AS60033 – http://blazingfast.io).

As an interim sweet spot I decided to set up my manual hosting on a France OVH 2gb VPS with 8 vCores (http://ovh.com).

 

For the following set-up and testing, I utilised both WordPress and Joomla reflection scripts and http://blitz.io performance load testing methods.

First steps was to try Apache2 with Mod_Security, however even after blocking common bot User-Agents along with the Reflected Layer 7 Attacks such as WordPress and Joomla (User-Agents of “WordPress/*.*.*” and nothing respectively) I found the mysqld and apache2 instances going through the roof, causing RAM to spike and crash the system after around 1000 concurrent active connections.

Similar to this illustration of a default LAMP stack on a 1gb VPS (Image by ReviewSignals):

 

So Apache was a no go, which brought to mind Litespeed or Nginx and since nginx is more community driven, I utilised that. The following carried out on my OVH Ubuntu 14.04 LTS VPS:

  • apt-get install mysql-server mysql-client
  • apt-get update;apt-get upgrade
  • apt-get install nginx php5 php5-fpm php5-mysql php5-curl php5-gd php5-intl php-pear php5-imagick php5-imap php5-mcrypt php5-memcache php5-ming php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl php5-apc

To minimise CPU and RAM usage I wanted to install php5-fpm (FastCGI Process Manager) with microcaching of fastcgi_cache. I followed a very helpful script by Kevin Ohashi from ReviewSignals on GitHub:

https://github.com/kevinohashi/WordPressVPS/blob/master/setup-nginx-php-fpm-microcache.sh

After this was set up, I began setting up a custom fail2ban rule to block flood attacks by IP and blackhole each ip without a timer (uses cron which utilises too much RAM) using IPTables automatically:

  • apt-get install fail2ban nano
  • nano /etc/fail2ban/jail.conf

Paste the following:

[httpflood-prot] enabled = true
port = http,https
filter = httpflood-prot
logpath = /var/log/nginx/access.log
maxretry = 50
findtime = 10
bantime = -1
action = route

Crtl+X, Y, Return(ENTER) to save.

Now run:

  • nano /etc/fail2ban/action.d/route.conf

Change blocktype to blackhole and make sure the following is present:

  • actionban = ip route add blackhole
  • actionunban = ip route del blackhole

 

For the finishing touches, and depending on how many cores are available to you, you can set nginx to have 1 worker per thread at 1500 threads in /etc/nginx/nginx.conf.

Edit mysqld max connections to 4 billion, and finally restart php5-fpm, nginx, fail2ban and mysql services.

Your Layer 7 protected box should now be ready to become live!

Leave a Reply

Your email address will not be published.