AutoScaling Builds

Triggering auto-scaling builds on Jenkins

As a best practice for DevOps, the CI pipeline would be configured to be triggered by every commit, which in turn would create a build job on Jenkins. During a release cycle, the rate of commits may vary and at some point, the rate may be too high, causing a build up the queue jobs on Jenkins. This may significantly impact the performance and the effective throughput of the Jenkins pipeline. We will utilize the ECS auto-scaling capability to spawn multiple containers for concurrent build processing, however, if the ECS infrastructure, on which the containers are spawned gets overutilized, the CloudWatch rules would automatically trigger additional ECS Instances to handle the scale. Once the rate of build job decreases, the infrastructure will be scaled-in and additional container instances will be terminated.

Now we will trigger a build on Jenkins to demonstrate auto-scaling capability offered by the ECS Container Instances.

Go to the Jenkins dashboard and select the build job: Run-All. Click Build Now and observe: dashboard

  • This job will trigger 10 build jobs, each taking around 4 minutes to complete

  • The jobs will be first queued in the Build Queue and the build workers will be seen triggerred under Build Executor Status

  • The build workers will initially be in the Launching and Suspended state dashboard

  • During this time, new task definitions are created for the ECS Cluster, which will be used to spawn the Build Workers: dashboard

  • After ~ 2 minutes, one of the build workers should change status as shown below to indicate the build job progress as shown below: dashboard

  • During this time, check the ECS service and you would see new tasks being launched. However, with the increased CPU usage, the Scale-UP Alarm Policy for CloudWatch is kicked off as shown below: dashboard

  • This will trigger additional container instances with containers as a part of Auto-Scaling Group triggerred by the Scale-UP Alarm: dashboard dashboard

  • New tasks launched can be tracked here as shown below: dashboard

  • After a period of time, the build job will be completed and all build jobs will have a blue-ball in front of their names.