High Performance Jenkins CI on Amazon EC2

We use Jenkins with Amazon EC2 to enable very low cost, but high-preformance continuous integration.

Our Setup

EC2 micro instances are extremely cost effective and more than sufficient to power Jenkins itself, unfortunately Amazon extremely heavily throttles the CPU usage of micro instances, making them completely unsuitable for building or testing projects. http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/

So, we run Jenkins on a reserved micro instance, but configured to start up a small instance to preform the actual builds. Our setup builds at each push and at the moment we average one push per day, so we pay roughly extra $3/mo. for our small instance use.

Important Note

We’re going to enable SSH login for the root user on the slave instances. This is generally an extremely bad idea, only slightly less insane because we’re using keys instead of passwords. Unfortunately, it seems that the current EC2 plugin for Jenkins (1.11) does not really support non-root users in conjunction with the default Amazon linux images, so we hack around that.

Configuring Jenkins Master

We use a 64 bit micro image in case we’d like to scale up in the future (but note that presently small EC2 instances require a 32 bit image).

Be sure to open port 80 for HTTP (and 22, for SSH) in the security groups when provisioning your master instance.

Basic setup for the machine:

sudo yum update
sudo yum install nginx ImageMagick graphviz

We install ImageMagick and graphviz so Jenkins can create plots.

We also installed lynx, mlocate, mercurial, postgresql, but they’re not necessary.

Now we install jenkins using yum:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins

We start jenkins:

sudo service jenkins start

Now we need to configure nginx to proxy requests to jenkins. We edit /etc/nginx/nginx.conf in the http section to enable proxying: (leave the other sections be)

http {
    # allow long server names
    server_names_hash_bucket_size 64;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

    access_log          /var/log/nginx/access.log;

    # spool uploads to disk instead of clobbering downstream servers
    client_body_temp_path /var/spool/nginx-client-body 1 2;
    client_max_body_size 32m;
    client_body_buffer_size    128k;

    server_tokens       off;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         off;

    keepalive_timeout   5;

    ## Compression
    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_min_length  1100;
    gzip_buffers 16 8k;
    gzip_types text/plain text/html text/css application/x-javascript \
        text/xml application/xml application/xml+rss text/javascript;
    # Some version of IE 6 don't handle compression well on some mime-types,
    # so just disable for them
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    # Set a vary header so downstream proxies don't send cached gzipped
    # content to IE6
    gzip_vary on;

    # proxy settings
    proxy_redirect     off;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_max_temp_file_size 0;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;

    include             /etc/nginx/sites-enabled/*;

}

And we create a /etc/nginx/sites-enabled/jenkins.conf containing just this configuration:

server {
    listen       80;
    server_name  dev.testproject.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://127.0.0.1:8080/;
        proxy_redirect     off;

        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_max_temp_file_size 0;

        client_max_body_size       10m;
        client_body_buffer_size    128k;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
    }
}

Now we reload nginx:

sudo service nginx restart

If you browse to your EC2 instance’s DNS you should see the jenkins install. We’re almost done here.

We want to make sure jenkins and nginx both start if the instance gets rebooted:

sudo chkconfig nginx --add
sudo chkconfig jenkins --add
sudo chkconfig nginx on
sudo chkconfig jenkins on

We’ll have to preform some more setup in a little bit, but now let’s move onto the slaves.

Configuring the Slave Image

What we’re going to do is create an instance, set it up so it has all the required libraries and external packages required to preform your builds and run your tests (like a database) and then we’re going to create an image from it which Jenkins will use to create future instances from.

Since we wanted to use a Small EC2 instance, we started with the 32 bit Amazon Linux AMI. For this guy we only need SSH available through the security group (typically the default, but double check, otherwise you can’t get in to the instance).

We preform the usual update:

sudo yum update

And install jenkins:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins

As well as any versioning system you may need. (mercurial in our case).

In our case we also installed and configured PostgreSQL.

Now, as we mentioned before, we have to enable root login through SSH so Jenkins plays better. This is relatively straightforward.

First we copy over the authorized SSH keys:

sudo cp -rv /home/ec2-user/.ssh /root/

And now we enable root login. Edit /etc/ssh/sshd_config, changing the line starting:

PermitRootLogin forced-commands-only

To:

PermitRootLogin yes

And reload the SSH configuration:

sudo service sshd reload

Okay. So we’re done with configuring the slave. Create a new AMI image through the AWS Management Console (right click on the small EC2 instance and click ‘Create Image’). This will shutdown the instance for imaging.

While that works. Let’s configure Jenkins on the master.

Configuring Jenkins on the Master

Start by installing whatever plugins you need (maven, mercurial, etc.) to build your project. You should try to build your project on the master just to make sure everything is working from that end.

Once you have a working build, it’s time to setup the EC2 plugin.

Start by installing it. “Hudson Amazon EC2 plugin”

Under “Manage Jenkins”, “Configure System”, at the bottom there should be an “add cloud” button through which you can add a new EC2 cloud.

The first part is relatively straightforward, just be sure to use “test connection”.

Now we’ll add the client image that should be done creating. Click “Add AMI” and enter the AMI id, don’t bother with the Check AMI “feature” since it will give false negatives.

The default instance type is small, which is probably what you want. Leave remote user blank to use root, and set “Remote FS Root” to /var/lib/jenkins.

Since you created a custom AMI you don’t need an init script.

Save the configuration, and now to test. Click “Build Executor Status” on the left, and then the “Provision via EC2” button that should be there now. It’ll take a few minutes, but you should have a sparkly new executor available. (see below if it doesn’t work)

Finally, go back to Jenkins settings and set the number of executors to 0 since we don’t want to use the micro node for anything.

And now it’s time to try a build of your project!

Gotchas

There are a crazy amount of gotchas to watch out for, but here are a few:

Master cannot provision a slave

  • Most of the error messages are reasonably useful. (if you screw up the AMI, for example)
  • Be sure the AMI is of the right platform (32 bit for a small, 64bit for most others).
  • If you’ve set the instance limiter in the Jenkins EC2 plugin, realize it’s for the total number of EC2 instance, including ones not created by Jenkins.

Master cannot connect to the slave

  • Slaves are created with the “default” security group, which might not be allowing incoming connections from port 22. Be sure to enable that.
  • Try to connect using SSH to the slave yourself. Be sure to login as the root user, since that’s what jenkins is trying ot do.

You forgot to install something on the image

  • Well, that sucks. Unfortunately, all you can do is SSH into a slave, add whatever you want, and create a new image from that.
  1. wikicook reblogged this from wkmacura
  2. wkmacura posted this