High Performance Jenkins CI on Amazon EC2
29 ♥We use Jenkins with Amazon EC2 to enable very low cost, but high-preformance continuous integration.
Our Setup
EC2 micro instances are extremely cost effective and more than sufficient to power Jenkins itself, unfortunately Amazon extremely heavily throttles the CPU usage of micro instances, making them completely unsuitable for building or testing projects. http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/
So, we run Jenkins on a reserved micro instance, but configured to start up a small instance to preform the actual builds. Our setup builds at each push and at the moment we average one push per day, so we pay roughly extra $3/mo. for our small instance use.
Important Note
We’re going to enable SSH login for the root user on the slave instances. This is generally an extremely bad idea, only slightly less insane because we’re using keys instead of passwords. Unfortunately, it seems that the current EC2 plugin for Jenkins (1.11) does not really support non-root users in conjunction with the default Amazon linux images, so we hack around that.
Configuring Jenkins Master
We use a 64 bit micro image in case we’d like to scale up in the future (but note that presently small EC2 instances require a 32 bit image).
Be sure to open port 80 for HTTP (and 22, for SSH) in the security groups when provisioning your master instance.
Basic setup for the machine:
sudo yum update
sudo yum install nginx ImageMagick graphviz
We install ImageMagick and graphviz so Jenkins can create plots.
We also installed lynx, mlocate, mercurial, postgresql, but they’re not necessary.
Now we install jenkins using yum:
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
We start jenkins:
sudo service jenkins start
Now we need to configure nginx to proxy requests to jenkins. We edit /etc/nginx/nginx.conf in the http section to enable proxying: (leave the other sections be)
http {
# allow long server names
server_names_hash_bucket_size 64;
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log;
# spool uploads to disk instead of clobbering downstream servers
client_body_temp_path /var/spool/nginx-client-body 1 2;
client_max_body_size 32m;
client_body_buffer_size 128k;
server_tokens off;
sendfile on;
tcp_nopush on;
tcp_nodelay off;
keepalive_timeout 5;
## Compression
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_types text/plain text/html text/css application/x-javascript \
text/xml application/xml application/xml+rss text/javascript;
# Some version of IE 6 don't handle compression well on some mime-types,
# so just disable for them
gzip_disable "MSIE [1-6].(?!.*SV1)";
# Set a vary header so downstream proxies don't send cached gzipped
# content to IE6
gzip_vary on;
# proxy settings
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
include /etc/nginx/sites-enabled/*;
}
And we create a /etc/nginx/sites-enabled/jenkins.conf containing just this configuration:
server {
listen 80;
server_name dev.testproject.com;
access_log off;
error_log off;
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
}
Now we reload nginx:
sudo service nginx restart
If you browse to your EC2 instance’s DNS you should see the jenkins install. We’re almost done here.
We want to make sure jenkins and nginx both start if the instance gets rebooted:
sudo chkconfig nginx --add
sudo chkconfig jenkins --add
sudo chkconfig nginx on
sudo chkconfig jenkins on
We’ll have to preform some more setup in a little bit, but now let’s move onto the slaves.
Configuring the Slave Image
What we’re going to do is create an instance, set it up so it has all the required libraries and external packages required to preform your builds and run your tests (like a database) and then we’re going to create an image from it which Jenkins will use to create future instances from.
Since we wanted to use a Small EC2 instance, we started with the 32 bit Amazon Linux AMI. For this guy we only need SSH available through the security group (typically the default, but double check, otherwise you can’t get in to the instance).
We preform the usual update:
sudo yum update
And install jenkins:
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
As well as any versioning system you may need. (mercurial in our case).
In our case we also installed and configured PostgreSQL.
Now, as we mentioned before, we have to enable root login through SSH so Jenkins plays better. This is relatively straightforward.
First we copy over the authorized SSH keys:
sudo cp -rv /home/ec2-user/.ssh /root/
And now we enable root login. Edit /etc/ssh/sshd_config, changing the line starting:
PermitRootLogin forced-commands-only
To:
PermitRootLogin yes
And reload the SSH configuration:
sudo service sshd reload
Okay. So we’re done with configuring the slave. Create a new AMI image through the AWS Management Console (right click on the small EC2 instance and click ‘Create Image’). This will shutdown the instance for imaging.
While that works. Let’s configure Jenkins on the master.
Configuring Jenkins on the Master
Start by installing whatever plugins you need (maven, mercurial, etc.) to build your project. You should try to build your project on the master just to make sure everything is working from that end.
Once you have a working build, it’s time to setup the EC2 plugin.
Start by installing it. “Hudson Amazon EC2 plugin”
Under “Manage Jenkins”, “Configure System”, at the bottom there should be an “add cloud” button through which you can add a new EC2 cloud.
The first part is relatively straightforward, just be sure to use “test connection”.
Now we’ll add the client image that should be done creating. Click “Add AMI” and enter the AMI id, don’t bother with the Check AMI “feature” since it will give false negatives.
The default instance type is small, which is probably what you want. Leave remote user blank to use root, and set “Remote FS Root” to /var/lib/jenkins.
Since you created a custom AMI you don’t need an init script.
Save the configuration, and now to test. Click “Build Executor Status” on the left, and then the “Provision via EC2” button that should be there now. It’ll take a few minutes, but you should have a sparkly new executor available. (see below if it doesn’t work)
Finally, go back to Jenkins settings and set the number of executors to 0 since we don’t want to use the micro node for anything.
And now it’s time to try a build of your project!
Gotchas
There are a crazy amount of gotchas to watch out for, but here are a few:
Master cannot provision a slave
- Most of the error messages are reasonably useful. (if you screw up the AMI, for example)
- Be sure the AMI is of the right platform (32 bit for a small, 64bit for most others).
- If you’ve set the instance limiter in the Jenkins EC2 plugin, realize it’s for the total number of EC2 instance, including ones not created by Jenkins.
Master cannot connect to the slave
- Slaves are created with the “default” security group, which might not be allowing incoming connections from port 22. Be sure to enable that.
- Try to connect using SSH to the slave yourself. Be sure to login as the root user, since that’s what jenkins is trying ot do.
You forgot to install something on the image
- Well, that sucks. Unfortunately, all you can do is SSH into a slave, add whatever you want, and create a new image from that.
