Automation of Cloud Infrastructure is a norm that every company follows. Whenever we think of Cloud platforms, the first thing that comes to our mind is what IAAC (Infrastructure as a Code) tooling to be used. Well all the Cloud platforms out there comes up with their own toolset, like -

  1. AWS - Cloudformation
  2. Azure - ARM Templates
  3. GCP - Deployment Manager

Apart from these one can use Chef, Ansible or Puppet as well to achieve the same functionalities along with simply using Cloud SDK. However, most organizations out there prefer to choose Terraform as their Cloud Automation service. The major advantage of using Terraform is, it’s totally free, comes with a huge community support, provides support for all major cloud providers along with many other things.

In this article, I’ll walk you through what Terraform is, why it’s being used, and how to set it up for your Cloud Account. We will also see how to manage cloud resources with it.

Terraform by Hashicorp

Terraform is the infrastructure as code offering from HashiCorp. It is a tool for building, changing, and managing infrastructure in a safe, repeatable way. SRE and DevOps teams can use Terraform to manage environments with a configuration language called the HashiCorp Configuration Language (HCL) for human-readable, automated deployments.

Install Terraform

To start with, firstly we need to download, install and setup Terraform by following link - Terraform Install


Once setup is done, you can simply check Terraform version as -
prateek~$terraform --version
Terraform v0.12.24
Your version of Terraform is out of date! The latest version
is 0.12.28. You can update by downloading from https://www.terraform.io/downloads.html

Terraform State

Terraform manages your cloud infrastructure by maintaining a state file. This state file consists of the actual state of your resources at a given point of time. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.

By default Terraform stores state in a local file named "terraform.tfstate", but it can also be stored remotely, which works better in a team environment.

In an organization it’s always recommended to store Terraform State remotely. In this article I’ll show you how to store your state on AWS - S3, Azure - ADLS Gen2 and GCP - Cloud Storage.

Terraform uses this state to create plans and make changes to your infrastructure. Prior to any operation, Terraform does a refresh to update the state with the real infrastructure. Terraform will showcase you the desired state when you run “terraform plan” by comparing it with the actual state which is present in your state file.

Let’s now jump straight into the code.

AWS

Firstly, setup AWS CLI and configure it to access your AWS account locally. We need AWS Access Keys and Secrets to access our AWS account which in further would be used by Terraform as well to generate “terraform plan” and apply changes.

Install and Configure AWS CLI - AWS CLI

Now, we need a S3 bucket. We use S3 as our Backend to store Terraform State Files. Once our S3 bucket is created, we can start using Terraform to create AWS resources. I’ll showcase how to create an IAM User, Group, Custom Policies and S3 Bucket using my modules.

Terraform GitHub Repo - Github Repo

I prefer creating Modules for my resources, so that they’re repeatable and can be used going forward to spin up the same resources again. Terraform Modules work similar to Functions, which are repeatable in nature and come with a modular structure.

Modules

Let’s go through the S3 module for this article.

Github Repo - S3 Module

Whenever we create a Module for our resource, we need to create 3 files -

  • main.tf - Contains the definition of our resource
  • variables.tf - Contains the variable definitions which will be used as parameters in our main.tf
  • output.tf - We can publish the output value of our AWS resources so that we can use it as an input in some other module. AWS outputs are ARN of the resources

Let’s try going through our “main.tf”. I’m creating a S3 bucket using this module and we can use “aws_s3_bucket” resource type from Terraform AWS providers for this.


main.tf
resource "aws_s3_bucket" "s3_bucket" {
  bucket = var.bucket_name  # Inputs the bucket name which is passed via variable bucket_name
  acl    = "private" # Bucket is set as Private means it won’t be accessible publicly 
  region = var.aws_region # This is AWS region where you bucket should reside

  lifecycle {
    prevent_destroy = true # Lifecycle policy set to true for prevent_destroy will help us to avoid any accidental deletion of our bucket
  }

  tags = {
    Name    = var.bucket_name # We can also pass TAGs for our bucket
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256" # We can define encryption type for our bucket 
      }
    }
  }
}

We can see above how our main.tf is completely configurable and just accepts variables as input and is repeatable in nature. Now, let’s straight away jump through output.tf and variables.tf

output.tf
output "s3_bucket_arn" {
  value = aws_s3_bucket.s3_bucket.arn # Output ARN of our S3 Bucket
}
variables.tf
variable "aws_region" { # Variable for AWS region
  type        = string
/* I have set the region as Singapore (ap-southeast-1), you can set it anything as per your AWS region preference. To find AWS region codes kindly check https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html */
  default     = "ap-southeast-1" 
  description = "AWS region"
}

variable "bucket_name" { # Variable for S3 bucket
  type        = string
  description = "S3 Bucket name"
}

Now, I believe you must have understood how we create modules for our AWS resources using Terraform. Similarly, we can create modules for any AWS resource. To check the complete list of resources supported by Terraform, please visit - Terraform Providers


Resources

Let’s see now how we actually call our modules to create our Terraform resource. For this, I have kept the code at -

S3 Resource

To create our AWS resource we need to create 2 files -

  • providers.tf - Used to define Terraform AWS provider
  • backend.tf - Used to define the Backend store to keep our Terraform State file

backend.tf
terraform {    
    required_version = ">= 0.12.0"  # Terraform version to be used
    backend "s3" {
        bucket  = "dataengineeringe2e" # S3 bucket to be used to store Terraform State file
        key     = "tfstate/state.tfstate" # Folder under your S3 bucket where state file will reside
        region  = "ap-southeast-1" # AWS region of S3 bucket, in this case Singapore
    }
}
providers.tf
provider "aws" { # Terraform AWS provider
    region  = "ap-southeast-1" # AWS region 
    version = "~> 2.7" # Terraform AWS provider version
}

Once these 2 files are created, we can simply now create our AWS resources by creating a HCL file and call modules as - AWS Resource


You can see in this file, how I’m calling a sub-module within the folder which in further is calling our main module which we created above. Once, terraform files are ready we simply need to run following 3 commands -

  • terraform init - Used to initialize a working directory containing Terraform configuration files.
  • terraform plan - Used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.
  • terraform apply - Used to apply the changes required to reach the desired state of the configuration.

Post successful run of all above 3 commands, you can go and check your AWS account, your resources should be ready for usage.

Azure

Firstly, setup Azure CLI and configure it to access your Azure account locally.

Install and Configure Azure CLI - Azure CLI


Now, we need an ADLS Gen2 or Blob storage account to store our state file. Once our storage account is created, we can start using Terraform to create Azure resources. I have created modules to create Storage Account, Resource group and VNets.

Terraform GitHub Repo - Github Repo


In Azure we’re making use of ADLS Gen2 as our backend store. It’s defined in backend.tf as - Link


terraform {

  required_version = ">= 0.12"
  backend "azurerm" {
    resource_group_name  = "dataengineeringe2e"
    storage_account_name = "dataengineeringe2e"
    container_name       = "tfstate"
    key                  = "state.tfstate"
  }
}

Do walkthrough the modules and feel free to use them. Do let me know if you want me to add more modules related to Azure in it.

GCP

Firstly, setup Google Cloud CLI and SDK and configure it to access your GCP account locally.

Install and Configure GCP CLI - GCP CLI


Now, we need Google Cloud Storage to store our state file. Once our storage is created, we can start using Terraform to create GCP resources. For GCP I have only created a single module for now which will showcase how we can create a Storage in Google.

Terraform GitHub Repo - Github Repo


In GCP we’re making use of Google Cloud Storage as our backend store. It’s defined in backend.tf as - Link


terraform {

  required_version = ">= 0.12"
  backend "gcs" { # Google Cloud Storage as backend store
    bucket  = "dataengineeringe2e" # GCS name
    prefix  = "tfstate/state.tfstate" # Folder within our GCS
  }
}

At the end, I hope you must have learned how we use Terraform with AWS, Azure and GCP and the benefits of it. Terraform is used to manage multiple technologies and resources along with Cloud. It’s adoption by organizations is increasing day by day at a rapid pace and I’m sure learning it will add a niche skill to your CV 😊