Friday, September 3, 2010

EC2 and its goodness

Recently, together with my favourite system administrator Emanuele, I had the occasion to migrate a small size structure to EC2. Actually Emanuele did most of the work but I believe my contribute was important too :-) .

We wanted to keep the base costs as low as possible and, at the same time, have a system able to scale on traffic peaks. Moreover we wanted a system easy to upgrade and adapt to the changing requirements. Not much original indeed.

I believe that the final result is quite interesting and I want to summarize it here.

We have an ELB and a main instance behind it. The instance is able to serve the web sites alone but when necessary the system can add additional application servers to help it.

System configuration is done with chef and poolparty so we can terminate it and create a new one in minutes. That's especially useful to create a temporary staging environment to test new upgrades.
Applications deployments is done in the same way and we can host many distinct applications on the same cloud.

Scaling triggers didn't fulfill well our requirements, so we fall back implementing a monitor script that leverages AWS service and controls how we scale up and down the system. This script allows us non-linear scaling (i.e. scaling up or down a variable number of instances depending on situation) and to consider many metrics in scaling conditions.

Although AWS has added support for configurable reverse DNS records since March 2010 we still think it is advantageous to delegate emails delivery to Google Apps. Postfix relays outgoing emails to Google smtp servers using the credentials associated to the sender of the message.
To cope with Google Apps limits we record the number of messages sent by an address and we can switch to another address of the same type in case we reach a predefined quota. For example we can send thousand messages with no-reply@example.org and then send other thousand messages with no-reply1@example.org and so on.

To speed up the initialization of a new application server we can create and use a custom AMIs which are almost ready to run. That doesn't conflict with chef which is able to skip already done
configuration steps and to quickly complete the setup.
Updating the auto-scaling group in order to use the new AMI is a bit tricky but can be done without affecting the production system or the ELB and related CNAME DNS records.

System backups are based on EBS snapshots. We create monthly, weekly, daily and 3-hourly snapshots and we prune the oldest ones of any type independently.

That's all folks, add a comment if you want to know more.