As you move services and data to the cloud and the complexity of your infrastructure grows, you can get stuck spending large amounts of time doing menial tasks that prevent you from getting the maximum benefit from your system. To avoid this, you’ll likely want to automate as many of your routine or tedious tasks as you can.
Fortunately, AWS offers several integrated ways to add automation as well as some options for automation through third-party configuration. Here, we’ll look at a few of the ways you can include automation into your workflows but keep in mind that most of these methods can be modified for other purposes as well.
AWS Systems Manager Automation is a great resource for automating common maintenance and deployment tasks related to EC2 instances or other AWS resources and can even be used to integrate on-premise systems with EC2 systems. With Systems Manager, you can manage instance states according to a set schedule based on tags or performance thresholds, create AMIs or recover unreachable instances, manage user permissions, and create custom workflows.
Before you can use Systems Manager, you will need to configure roles for the automation tasks, either through CloudFormation or Identity and Access Management (IAM).
One example of the automation you can accomplish is patching for Linux systems. Systems Manager provides a document called AWS-UpdateLinuxAmi, based on JSON or YAML, which lets you upgrade distribution packages and Amazon software on Linux, Red Hat, Ubuntu, SLES, or CentOS AMIs.
It also lets you install or upgrade other software packages on the instances. This process uses the SSM agent to download and apply your defined updates, including any pre or post-update scripts you specify before creating a new AMI with the updated software versions.
Amazon CloudWatch is a monitoring and management service that provides performance and operational data on other AWS services. CloudWatch provides information and offers automated actions for when a threshold has been reached on a certain metric. For example, when CloudWatch detects an EC2 instance that is not fully utilized, you can use a CloudWatch action to shut down the machine.
Using CloudWatch automated actions in combination with Amazon Lambda, a serverless computing service, can provide even greater options. Lambda lets you run code to perform complex activities triggered by a CloudWatch action, without worrying about managing an infrastructure to run that code (everything is handled transparently by Lambda).
In combination with Cron expressions, you can fully automate a variety of events according to the CloudWatch trigger you set, limited only by the available API or SDK options.
Many AWS services include a few built-in automation features, such as alerting, scheduling, or diagnostics. These are typically managed through the tagging of volumes or resources and the application of policies that define when automated tasks should be done, under what conditions, or by whom. The creation of backups is one example in which this automation is particularly useful.
AWS allows you to store incremental backups of EC2 instances as EBS snapshots from which your instances can be restored. These AWS snapshots are comparable to those available in Azure and in of themselves are not too difficult to manage.
The problem starts when you need to backup large numbers of EC2 instances and EBS volumes on an ongoing basis. The automation features included in Lifecycle Manager, however, can solve this issue by allowing you to automate your backups on a large scale with the previously mentioned tags and policies. AWS Backup grants similar functionality to Amazon’s other storage services.
AWS CloudFormation is a service used to model and configure your AWS resources using JSON templates to launch resources into “stacks” of services or applications. If you only need to create a few stacks, manual creation may be an option but for large quantities, you’ll want to automate.
One way of doing this is to write a script directing CloudFormation through SDK. In this method, you set the parameters for the CloudFormation template you want to use and point your script to reference that template. Check out this post to see an example of this process using Python.
Another option you can use is to include CloudFormation as a deployment action in your CodePipeline workflows. This action can be included at any stage of your pipeline and allows you to create and delete stacks as well as create or execute changesets, which allows you to test how changes will impact your resources before they are implemented. This combination is ideal if you’re already employing a continuous delivery model, as it can also be used to automatically deploy and terminate testing or pre-production environments for your applications.
Third-party tools are also an available option for automation in AWS. If you use Jenkins as a Continuous Integration server, you can add Systems Manager Amazon automation as a post-build step, to install your release into an AMI. Or you can use Jenkins scheduling to automatically patch your operating system or other software, as described above.
To enable Jenkins to access your EC2 instances, you can create a user for it in AWS IAM, enabling Programmatic Access. You then install the AWS CLI on your Jenkins server, configure it using the user credentials you defined, and then add a build step in Jenkins that executes a shell or Windows batch command like this:
The ways in which you can automate AWS are not limited to those covered here, but this list should give you a good idea of the sort of tasks you can accomplish and what types of tools are available. For certain, to get the most out of AWS you need to consider including automation whenever reasonable.
Although it isn’t always simple to do so, starting with the built-in automation functionalities and slowly expanding to features requiring more expertise knowledge can significantly increase your productivity and the benefits you see from your system.