Vanshika Sharma

AWS EC2 Automation

Vanshika Sharma's photo
Vanshika Sharma
·

19 min read

Table of contents

Automation in EC2:

Automation in Amazon EC2 (Elastic Compute Cloud) refers to using tools and scripts to manage the lifecycle of EC2 instances, deployments, scaling, monitoring, and backups without manual intervention. Here are the key aspects of automation in EC2:

1. Auto Scaling

  • Amazon EC2 Auto Scaling automatically adjusts the number of EC2 instances based on traffic and resource utilization.

  • Ensures high availability and optimal performance.

  • Can be configured with scaling policies and CloudWatch alarms.

2. Elastic Load Balancing (ELB) Integration

  • Automatically distributes incoming traffic across multiple instances.

  • Ensures fault tolerance and minimizes manual traffic management.

3. Instance Launch Templates

  • In templates, define instance configurations (AMI, instance type, security groups, and key pairs).

  • Allows consistent instance launches and integrates with Auto Scaling for automation.

4. AWS Systems Manager Automation

  • Run predefined or custom automation documents (SSM documents) for common tasks like patching instance termination, and updates.

  • Can automate operational tasks across multiple EC2 instances.

5. AWS CloudFormation

  • Automates the provisioning and management of EC2 instances along with associated infrastructure using Infrastructure as Code (IaC).

  • Ensures repeatable and version-controlled deployments.

6. AWS Lambda for Automation

  • Trigger Lambda functions on EC2 events to automate actions like instance clean-up, backups, or state changes.

  • Can be used with AWS Config for compliance enforcement.

7. Amazon EventBridge

  • Automate responses to EC2 instance state changes (e.g., starting, stopping, or terminating).

  • Integrates with other AWS services for complex workflows.

8. EC2 Auto Recovery

  • Automatically recover unhealthy EC2 instances by restarting them or replacing failed hardware.

9. Backup and Restore Automation

  • Use AWS Backup to automate EC2 volume snapshot creation and retention policies.

  • Schedule regular snapshots for disaster recovery.

10. CI/CD Integration

  • Automate the deployment and configuration of EC2 instances through Jenkins, GitLab CI/CD, or AWS CodeDeploy.

  • Ensures consistent and frequent deployments with rollback options.

11. CloudWatch Alarms and Actions

  • Automate instance state changes or scaling based on CloudWatch metrics (e.g., CPU usage, disk I/O).

  • Trigger automated workflows to handle anomalies or high usage.

Benefits of EC2 Automation

  • Consistency: Reduces human error by standardizing operations.

  • Scalability: Ensures resources are allocated based on demand.

  • Cost-efficiency: Automatically stops or terminates unused instances to optimize costs.

  • Resilience: Enables quick recovery from failures and improves uptime.

Launch template in AWS EC2:

A Launch Template in AWS EC2 is a resource that captures all the settings and configurations necessary to launch an EC2 instance. It simplifies the instance creation process, ensures consistency across instances, and, when integrated with services like Auto Scaling and Spot Fleet, allows for easier scaling and automation.

Key Components of a Launch Template

  1. AMI (Amazon Machine Image)

    • The OS and application configuration for the instance.

  2. Instance Type

    • Defines the hardware (CPU, memory, storage) for the instance.

  3. Key Pair

    • The public key is used for secure SSH access to the instance.

  4. Security Groups

    • Define firewall rules to control traffic to and from the instance.

  5. Block Device Mapping

    • Configure the storage volumes attached to the instance.

  6. Instance Profile (IAM Role)

    • Attach an IAM role to grant the instance permissions for other AWS services.

  7. Network Settings

    • Specify the VPC, subnet, public IP assignment, and placement options.

  8. User Data

    • Scripts or cloud-init instructions for instance initialization (e.g., software installation).

  9. Elastic IP

    • Associate an Elastic IP address for static, public-facing access.

  10. Advanced Features

    • Include metadata options, monitoring settings, tenancy (shared or dedicated hardware), and more.

Benefits of Using Launch Templates

  1. Consistency and Reusability

    • Ensures all instances are launched with uniform configurations.

  2. Versioning Support

    • Allows multiple versions of the template, enabling rollbacks or gradual updates to configurations.

  3. Integration with Auto Scaling

    • Auto Scaling groups can use launch templates to scale EC2 instances based on demand.

  4. Cost Efficiency

    • Can specify Spot Instances and their parameters, making it easier to manage on-demand and spot instance strategies.

  5. Tagging

    • Automatically apply tags to all instances launched from the template.

How to Create a Launch Template (AWS Console)

  1. Navigate to EC2 Dashboard:
    Open the AWS Management Console and go to the EC2 service.

  2. Create a New Launch Template:
    Select "Launch Templates" in the EC2 Dashboard and click "Create Launch Template."

  3. Provide Template Details:

    • Name and Description

    • Base AMI

    • Instance Type

    • Key Pair and Security Groups

    • Storage and Advanced Options

  4. Review and Create:
    Verify the configuration and click "Create Launch Template."

Using a Launch Template

  • Launching EC2 Instances:
    Select the launch template from the EC2 launch wizard.

  • Auto Scaling:
    Attach the launch template to an Auto Scaling group to scale instances automatically.

  • Spot Fleet:
    Use the launch template to manage spot requests and bidding.

Instance Types:

AWS EC2 Instance Types define the underlying hardware for virtual machines (instances) in AWS. They are categorized based on various workloads like general-purpose, compute-optimized, memory-optimized, and more. Below are the key categories and popular instance types:

1. General-Purpose Instances

  • Use Case: Balanced compute, memory, and network resources for a wide range of workloads.

  • Popular Types:

    • t3, t3a, t4g: Burstable instances with lower cost, ideal for intermittent workloads.

    • m6i, m6a, m7g: Versatile for databases, web servers, and development environments.

  • Key Benefit: Good performance at a lower price.

2. Compute-Optimized Instances

  • Use Case: Applications requiring high-performance processing and low-latency networking.

  • Popular Types:

    • c6i, c6g, c7g: High-performance instances for gaming, web servers, and analytics.

    • c5n: High networking throughput, ideal for machine learning or HPC.

  • Key Benefit: High compute power with predictable performance.

3. Memory-Optimized Instances

  • Use Case: Memory-intensive applications like in-memory databases, data analytics, and caching.

  • Popular Types:

    • r6i, r6g, r7g: Optimized for memory-intensive workloads.

    • x2gd: High memory-to-vCPU ratio for large data analytics.

    • z1d: High clock speed and memory for financial applications.

  • Key Benefit: Larger memory footprint for intensive tasks.

4. Storage-Optimized Instances

  • Use Case: High I/O and local storage for NoSQL databases, Hadoop, and data warehousing.

  • Popular Types:

    • i4i, i3: Optimized for high IOPS and large SSD storage.

    • d2, d3: Dense storage instances for large datasets.

    • h1: Balanced storage and computing for data warehousing and log processing.

  • Key Benefit: High-performance storage for fast data processing.

5. Accelerated Computing (GPU-Optimized) Instances

  • Use Case: Machine learning, AI, HPC, and 3D rendering workloads.

  • Popular Types:

    • p4, p3: Designed for deep learning training and inferencing.

    • g5, g4dn: High-performance GPUs for AI inferencing and graphic-intensive tasks.

    • inf1: Cost-effective for AI/ML inference workloads.

  • Key Benefit: Leverage GPUs for high parallelism in ML, AI, or visualization.

6. Arm-Based Instances

  • Use Case: Cost-effective and energy-efficient workloads with AWS Graviton processors.

  • Popular Types:

    • t4g: General-purpose burstable instances.

    • c6g: Compute-optimized.

    • r6g: Memory-optimized for Graviton-powered workloads.

  • Key Benefit: Better price/performance with AWS Graviton chips.

7. High Memory Instances

  • Use Case: Large-scale in-memory databases (e.g., SAP HANA).

  • Popular Types:

    • u-6tb1, u-9tb1, u-12tb1: Large memory instances with up to 12 TB of RAM.

  • Key Benefit: Ultra-high memory for enterprise applications.

8. Bare Metal Instances

  • Use Case: Full control over hardware with no hypervisor. Suitable for legacy applications or those requiring specific hardware features.

  • Popular Types:

    • m5.metal, c5.metal, r5.metal

  • Key Benefit: Direct hardware access for specialized workloads.

Choosing the Right Instance Type

When selecting an instance type, consider:

  • Workload Needs: Match compute, memory, or storage to the application.

  • Scalability: Use Auto Scaling with a suitable instance type for growth.

  • Cost Efficiency: Balance performance with budget constraints.

AMI:

An Amazon Machine Image (AMI) is a pre-configured template in AWS EC2 that provides the information required to launch instances. It acts as the "blueprint" for an EC2 instance and includes the operating system, application server, and other configurations necessary for launching and running workloads.

Key Components of an AMI

  1. Operating System (OS)

    • The AMI includes the OS, such as Linux distributions (Amazon Linux, Ubuntu, Red Hat, etc.) or Windows Server.

  2. Application Software

    • Custom AMIs can include pre-installed software like databases, web servers, or development tools.

  3. Root Volume

    • Specifies the root volume configuration (EBS or instance store).

  4. Launch Permissions

    • Controls which AWS accounts or users can launch instances from the AMI.

  5. Block Device Mapping

    • Defines the storage volumes attached to instances launched from the AMI.

Types of AMIs

  1. AWS-Provided AMIs

    • Amazon Linux and other standard Linux/Windows AMIs are provided by AWS for public use.

  2. Marketplace AMIs

    • Pre-configured AMIs with commercial software available in the AWS Marketplace (may involve additional costs).

  3. Community AMIs

    • AMIs created by the AWS community, are often free but not officially supported by AWS.

  4. Custom AMIs

    • Users can create their own AMIs to include specific configurations, software, and security patches.

Creating a Custom AMI

  1. Launch and Configure EC2:
    Launch an EC2 instance with the desired configuration and install any required software.

  2. Stop the Instance:
    Stop the instance to ensure data consistency before creating the AMI.

  3. Create the AMI:

    • Navigate to the EC2 dashboard.

    • Select the instance and choose "Create Image."

    • Provide a name and description, and configure any additional storage volumes.

  4. Register the AMI:
    The AMI will be registered and available for future launches.

Using an AMI

  • Launching Instances:
    During instance creation, you can choose an AMI to define the instance's configuration.

  • Backup and Disaster Recovery:
    Use AMIs to create snapshots of an instance, allowing recovery in case of failure.

  • Scaling and Consistency:
    AMIs ensure new instances have the same configuration, making scaling and replication consistent.

Benefits of Using AMIs

  1. Consistency:
    Launch identical instances for load balancing, Auto Scaling, or deployments.

  2. Efficiency:
    Avoid repetitive installation and configuration by creating pre-configured AMIs.

  3. Backup and Recovery:
    AMIs act as backups for entire instances, allowing quick recovery or replication.

  4. Customizability:
    Create tailored AMIs for specific workloads or security policies.

Task1:

  • Create a launch template with Amazon Linux 2 AMI and t2.micro instance type with Jenkins and Docker setup (You can use the Day 39 User data script for installing the required tools.

  • Create 3 Instances using Launch Template, there must be an option that shows the number of instances to be launched

Here’s how you can create a launch template with Amazon Linux 2 AMI and t2.micro instance type, with Jenkins and Docker installation, and launch 3 instances using the AWS Management Console or CLI.

Steps to Create a Launch Template (AWS Console)

  1. Navigate to EC2 Dashboard:

  2. Create a New Launch Template:

    • Under the EC2 menu, select Launch Templates and click Create Launch Template.

  3. Fill in Template Details:

    • Launch Template Name: Give it a meaningful name (e.g., jenkins-docker-template).

    • Amazon Machine Image (AMI): Choose Amazon Linux 2.

    • Instance Type: Select t2.micro.

  4. Add User Data Script:

    • In the Advanced Details section, add the following User Data script (modify as per your Day 39 script):

        #!/bin/bash
  # Update the system
  sudo yum update -y

  # Install Docker
  sudo yum install docker -y
  sudo systemctl start docker
  sudo systemctl enable docker

  # Install Jenkins
  sudo yum install java-11-openjdk -y
  sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
  sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
  sudo yum install jenkins -y
  sudo systemctl start jenkins
  sudo systemctl enable jenkins

  # Add ec2-user to docker group to avoid permission issues
  sudo usermod -aG docker ec2-user

  5. Key Pair:

    • Select an existing key pair or create a new one for SSH access to the instances.

  6. Network Settings:

    • Select the appropriate VPC and security group (allow HTTP, SSH, and Jenkins ports like 8080).

  7. Storage:

    • Set storage size and type based on your requirements.

  8. Review and Create:

    • Click Create Launch Template to finalize the process.

Launching Instances Using the Launch Template

  1. Go to the Launch Template Section:

    • Select the newly created launch template.

  2. Launch Instances:

    • Click Actions → Launch Instance from Template.

  3. Set Number of Instances:

    • In the "Launch Instances" window, under Number of Instances, set the count to 3.

  4. Review and Launch:

    • Click Launch Instances to spin up 3 instances using the launch template.

Verifying the Setup

  1. Check EC2 Instances:

    • Ensure all 3 instances are running.

  2. Verify Jenkins and Docker:

    • Connect to each instance via SSH and verify:

      • Docker: docker --version

      • Jenkins: Visit http://<instance-public-ip>:8080.

Setting up an Application Load Balancer with AWS EC2

What is Load Balancing?

Load Balancing is the process of distributing incoming network traffic across multiple servers or resources to ensure no single server becomes overwhelmed. By managing workloads efficiently, load balancing enhances application availability, reliability, and performance.

Key Functions of Load Balancing

  1. Distributes Traffic
    Spreads incoming requests across multiple instances or servers to balance the load.

  2. High Availability
    Ensures continuous availability by routing traffic to healthy instances if one fails.

  3. Improved Performance
    Reduces latency and response time by sending requests to less busy servers.

  4. Scalability
    Supports dynamic scaling by distributing traffic evenly as new instances are added.

  5. Health Monitoring
    Continuously checks the health of servers and routes traffic only to healthy instances.

Types of Load Balancing

  1. Application Load Balancing (Layer 7)

    • Works at the application layer (HTTP/HTTPS).

    • Can route traffic based on request content (e.g., URL, headers).

    • Example: AWS Application Load Balancer (ALB).

  2. Network Load Balancing (Layer 4)

    • Works at the transport layer (TCP/UDP).

    • Suitable for high-throughput and low-latency applications.

    • Example: AWS Network Load Balancer (NLB).

  3. Classic Load Balancing

    • Legacy AWS load balancer that operates at both Layer 4 and Layer 7.

    • Supports basic routing rules.

  4. Global Load Balancing

    • Distributes traffic across different geographic regions for disaster recovery and improved user experience.

    • Example: AWS Global Accelerator.

Benefits of Load Balancing

  1. Fault Tolerance
    Automatically removes failed servers from the pool and redirects traffic.

  2. Enhanced User Experience
    Reduces server bottlenecks and optimizes response times for users.

  3. Efficient Resource Utilization
    Ensures all servers are utilized efficiently, avoiding idle or overloaded resources.

  4. Seamless Scaling
    Allows dynamic scaling by balancing traffic as new servers are added or removed.

Common Use Cases

  • Web Applications: Ensure even distribution of web requests across multiple backend servers.

  • Microservices: Route specific API requests to appropriate backend services.

  • Disaster Recovery: Automatically route traffic to healthy servers in case of failure.

  • Global Applications: Balance traffic across multiple regions for faster access.

Elastic Load Balancing:

Elastic Load Balancing (ELB) is an AWS service that automatically distributes incoming application traffic across multiple EC2 instances, containers, or other resources to ensure high availability, scalability, and fault tolerance. It is a fully managed service that adapts to changes in incoming traffic.

Types of Elastic Load Balancers

  1. Application Load Balancer (ALB)

    • Operates at Layer 7 (Application Layer).

    • Supports advanced routing, such as path-based or host-based routing.

    • Ideal for HTTP and HTTPS traffic.

    • Features:

      • WebSocket and HTTP/2 support.

      • Host-based routing (e.g., api.example.com vs www.example.com).

      • Path-based routing (e.g., /api to one target group and /app to another).

      • SSL Termination.

  2. Network Load Balancer (NLB)

    • Operates at Layer 4 (Transport Layer).

    • Handles TCP and UDP traffic with ultra-low latency.

    • Designed for high-throughput workloads.

    • Features:

      • Supports static IP addresses.

      • Preserves the source IP of the client.

      • Provides extreme performance and low latency.

  3. Gateway Load Balancer (GWLB)

    • Operates at Layer 3 (Network Layer).

    • Allows you to deploy, manage, and scale virtual appliances (e.g., firewalls, intrusion detection systems).

    • Features:

      • Combines load balancing and transparent network traffic inspection.

      • Simplifies deployment of third-party security appliances.

  4. Classic Load Balancer (CLB) (Legacy)

    • Operates at both Layer 4 and Layer 7.

    • Supports both HTTP/HTTPS and TCP traffic.

    • Limited routing capabilities compared to ALB and NLB.

    • AWS recommends using ALB or NLB for new workloads.

Key Features of Elastic Load Balancing

  1. Automatic Scaling

    • ELB adjusts to handle changes in incoming traffic by distributing traffic across healthy resources.

  2. Health Checks

    • Periodically checks the health of backend instances and only routes traffic to healthy instances.

  3. High Availability

    • Ensures fault tolerance by distributing traffic across multiple Availability Zones.

  4. SSL Termination

    • Offloads SSL/TLS decryption to the load balancer, reducing the workload on backend instances.

  5. Sticky Sessions

    • Ensures a client’s requests are consistently routed to the same instance.

  6. Logging and Monitoring

    • Provides detailed logs for requests through Access Logs and metrics via Amazon CloudWatch.

  7. IP Address Targeting

    • Allows routing to targets based on IP addresses, providing flexibility for hybrid cloud setups.

Use Cases of Elastic Load Balancing

  1. Web Applications

    • Balance HTTP/HTTPS traffic across multiple servers for improved performance and availability.

  2. Microservices

    • Route API requests to different backend services based on path or host.

  3. Gaming and IoT

    • Use NLB for low-latency TCP/UDP traffic for gaming applications or IoT data streams.

  4. Security Services

    • Deploy security appliances with GWLB for traffic inspection and intrusion detection.

Benefits of Elastic Load Balancing

  1. Improved Fault Tolerance
    Automatically detects unhealthy instances and redirects traffic to healthy instances.

  2. Increased Application Performance
    Distributes traffic evenly to avoid overloading any single instance.

  3. Scalability
    Seamlessly scales to handle sudden spikes in traffic.

  4. Ease of Management
    Fully managed by AWS, so you don’t need to maintain the underlying infrastructure.

Task 2:

  • Launch 2 EC2 instances with an Ubuntu AMI and use User Data to install the Apache Web Server.

  • Modify the index.html file to include your name so that when your Apache server is hosted, it will display your name also do it for 2nd instance which includes " DevOps Community is Super Awesome".

  • Copy the public IP address of your EC2 instances.

  • Open a web browser and paste the public IP address into the address bar.

  • You should see a webpage displaying information about your PHP installation.

Task 2: Launch 2 EC2 Instances with Apache Web Server Installation and Custom Web Pages

Below are the detailed steps to perform this task using the AWS Management Console.

Step 1: Launch Two EC2 Instances (Ubuntu AMI)

  1. Navigate to EC2 Dashboard:

    • Open the AWS Management Console and go to the EC2 Dashboard.

  2. Launch an EC2 Instance:

    • Choose Ubuntu AMI (e.g., Ubuntu Server 20.04 LTS).

    • Select t2.micro instance type (for free tier).

  3. Configure Instance Details:

    • Ensure proper network settings (VPC, subnet, etc.).

    • Enable Auto-assign Public IP to access the instance via its public IP address.

  4. Add User Data:
    Under the Advanced Details section, add the following User Data script to automatically install Apache and modify the web page.

    For the 1st Instance (Your Name)

     #!/bin/bash
 # Update packages and install Apache
 sudo apt update -y
 sudo apt install apache2 -y

 # Create a custom index.html file
 echo "<html><body><h1>Welcome! This server is managed by [Your Name]</h1></body></html>" | sudo tee /var/www/html/index.html

 # Start Apache
 sudo systemctl start apache2
 sudo systemctl enable apache2

    For the 2nd Instance (DevOps Community Message)

     #!/bin/bash
 # Update packages and install Apache
 sudo apt update -y
 sudo apt install apache2 -y

 # Create a custom index.html file
 echo "<html><body><h1>DevOps Community is Super Awesome!</h1></body></html>" | sudo tee /var/www/html/index.html

 # Start Apache
 sudo systemctl start apache2
 sudo systemctl enable apache2

  5. Add Storage:

    • Keep the default storage configuration or adjust it as needed.

  6. Configure Security Group:

    • Create a new security group or modify an existing one to allow HTTP (port 80) traffic from anywhere (0.0.0.0/0).

  7. Launch the Instance:

    • Launch both instances using the respective User Data scripts.

Step 2: Access the Web Pages

  1. Get the Public IP Addresses:

    • Go to the EC2 Dashboard, select each instance, and copy their public IP addresses.

  2. Open a Web Browser:

    • Paste the public IP address of the first instance. You should see a webpage with your name displayed.

    • Paste the public IP address of the second instance. You should see the message "DevOps Community is Super Awesome!" displayed.

Verifying Apache Installation

  • If the webpage is not loading, check the following:

    • Ensure the security group allows inbound traffic on port 80 (HTTP).

    • Ensure Apache is running (sudo systemctl status apache2).

Task 3:

  • Create an Application Load Balancer (ALB) in EC2 using the AWS Management Console.

  • Add EC2 instances which you launch in task-1 to the ALB as target groups.

  • Verify that the ALB is working properly by checking the health status of the target instances and testing the load-balancing capabilities.

Task 3: Create an Application Load Balancer (ALB) in EC2 and Add Target EC2 Instances

Below are the step-by-step instructions to create an ALB and attach the EC2 instances from Task 1 as targets.

Step 1: Navigate to EC2 Dashboard

  1. Open the AWS Management Console and go to the EC2 Dashboard.

  2. Under the left-hand menu, select Load Balancers.

Step 2: Create an Application Load Balancer (ALB)

  1. Click on "Create Load Balancer":

    • Choose Application Load Balancer.

  2. Configure the Load Balancer:

    • Name: Enter a descriptive name for your ALB (e.g., task1-alb).

    • Scheme: Choose internet-facing.

    • IP Address Type: Choose IPv4.

    • Listeners: Ensure there is a listener for HTTP on the port 80.

  3. Availability Zones:

    • Select the VPC where your EC2 instances were launched.

    • Choose multiple availability zones for better fault tolerance (make sure your instances are in the selected subnets).

Step 3: Configure Target Group

  1. Target Group Type:

    • Choose Instances the target type.

    • Enter a target group name (e.g., task1-target-group).

    • Choose HTTP as the protocol and port 80.

    • Health check protocol: HTTP.

    • Health check path: Leave as the default (/).

  2. Register Targets:

    • Click "Register targets".

    • Select the two EC2 instances you created in Task 1.

    • Click Add to registered → Create target group.

Step 4: Finalize the Load Balancer

  1. Under the "Register Targets" section, choose the target group you just created.

  2. Click "Create Load Balancer" to finalize the setup.

Step 5: Verify the Health of Target Instances

  1. Once the ALB is created, navigate to Target Groups in the EC2 dashboard.

  2. Select your target group and check the Health Status of the registered instances.

    • The instances should show as "healthy" if they are properly configured and the Apache web server is running.

Step 6: Test the Load Balancer

  1. Copy the DNS name of your Application Load Balancer from the EC2 dashboard (found under Description → DNS name).

  2. Paste the DNS name into your web browser.

    • You should see the custom web pages from Task 1:

      • One page displaying your name.

      • Another page displaying "DevOps Community is Super Awesome!".

  3. Load Balancing Check:

    • Refresh the page multiple times to see traffic being distributed across both instances.

    • You should see both web pages at different times, verifying that the load balancer is distributing traffic correctly.

Troubleshooting Tips

  • Health Check Failures:

    • Ensure both EC2 instances are running Apache (sudo systemctl status apache2).

    • Check that port 80 is allowed in the security group for both the instances and the ALB.

  • No Response from ALB:

    • Ensure the target group is properly associated with the ALB.

    • Ensure the correct DNS name is used in the browser.

automation in EC2Launch template in aws EC2Instance typesLoad Balancing in awsaws ec2 automationAMIElastic Load Balancing