When working with Terraform, the state file (terraform.tfstate) is a critical component that tracks the current infrastructure setup. Managing this file efficiently, especially in collaborative environments, is key to ensuring smooth infrastructure provisioning. One of the most common practices for managing Terraform state files is using remote backends, and Amazon S3 (Simple Storage Service) is often chosen as the backend for this purpose. In this article, we’ll explore the concept of Terraform remote backends, why we use an S3 bucket, and walk through an example to illustrate how it works.
What is a Terraform Remote Backend?
In Terraform, the state refers to a snapshot of your infrastructure, and the state file is what Terraform uses to keep track of all the resources it manages. By default, Terraform stores this state file locally (in the same directory where you run your Terraform commands). However, this can be problematic when working in teams or when using Terraform in production environments. If multiple users are making changes to the infrastructure simultaneously, it can result in conflicts or loss of data.
A remote backend allows Terraform to store its state file in a centralized location that can be accessed by multiple users or automated systems. This enables collaboration, provides secure access control, and ensures that the state file is safely backed up and stored. The remote backend also supports features like state locking to prevent multiple processes from modifying the state at the same time.
In this article, we will focus on using an S3 bucket as a remote backend for storing Terraform state.
Why Use S3 for Terraform Remote Backends?
Amazon S3 is a highly reliable and scalable storage service that is perfect for storing files, including Terraform’s state files. Here are a few reasons why S3 is commonly used as a backend for Terraform:
1. Scalability: S3 is designed to scale with your needs. Whether you’re managing a small project or a large infrastructure setup, S3 can handle your Terraform state files efficiently.
2. Reliability: S3 provides high durability, ensuring that your state file is safely stored. AWS S3 offers 99.999999999% durability, meaning your state file is unlikely to be lost.
3. Collaboration: With S3, multiple team members can access and modify the state file concurrently, ensuring that everyone is working with the most up-to-date infrastructure data.
4. Security: S3 offers fine-grained access control using AWS IAM (Identity and Access Management) policies. This allows you to control who can read from or write to the S3 bucket, providing robust security for your state files.
5. State Locking: By using a combination of S3 with DynamoDB (for state locking), you can prevent concurrent modifications to the state file, avoiding potential issues with conflicting changes.
GitHub Repository
You can find all the code examples used in this guide in my GitHub repository:
Setting Up a Remote Backend with S3
To set up an S3 bucket as the backend for Terraform, you need to specify the backend configuration in your Terraform code. Here’s a step-by-step walkthrough using an example setup.
Step 1: Create an S3 Bucket (if not already created)
In the example, we are using an existing S3 bucket called dhanikaa-remote-backend to store the state file. If you haven’t created an S3 bucket yet, you can do so manually via the AWS console or with Terraform itself.
Example of creating an S3 bucket in Terraform:
resource "aws_s3_bucket" "remote_backend" {
bucket = "dhanikaa-remote-backend"
region = "eu-north-1"
}Step 2: Configure Terraform Backend
The next step is to configure the S3 backend. This is done in the backend.tf file. Here’s how you define the backend configuration:
terraform {
backend "s3" {
bucket = "dhanikaa-remote-backend" # Name of the existing S3 bucket
region = "eu-north-1" # The region where the bucket is located
key = "terraform.tfstate" # The key (path) where the state file will be stored
}
}In this configuration:
• bucket: The name of the S3 bucket that will store the state file.
• region: The AWS region where the S3 bucket is located.
• key: The path in the bucket where the state file will be stored (e.g., terraform.tfstate).
Step 3: Initialize the Backend
Once you’ve defined the backend configuration, the next step is to initialize the Terraform project with the remote backend:
terraform init
This command will initialize the configuration and set up the connection to the S3 bucket as the backend. If the bucket doesn’t exist, Terraform will provide instructions on how to create it.
Step 4: Using the S3 Backend with Terraform Resources
Now that your backend is set up, you can proceed with creating and managing infrastructure resources with Terraform. Here’s an example main.tf file that defines a simple AWS EC2 instance:
provider "aws" {
region = "eu-north-1"
}
resource "aws_instance" "remote_backend_example" {
ami = "ami-02db68a01488594c5" # Amazon Linux 2 AMI
instance_type = "t3.micro" # Instance type
tags = {
Name = "remote-backend-example"
}
}When you run terraform apply, the state file (terraform.tfstate) will be saved in the specified S3 bucket.
How It Works: Step-by-Step Breakdown
1. Local Device Requests State: When you run any Terraform command like terraform plan or terraform apply, your local device (or the CI/CD system you’re using) requests the Terraform state from the remote backend.
2. Query Root and TLD Servers: If the S3 bucket and the state file are set up correctly, Terraform will find the state file from the S3 bucket in the region you specified (e.g., eu-north-1).
3. Manage Infrastructure: Terraform applies any changes to the infrastructure (like launching a new EC2 instance, as shown in the example). It then updates the state file and saves it back to the S3 bucket.
4. Consistency: Since the state file is stored remotely in S3, it ensures that all users working with the same Terraform configuration file are using the most up-to-date version of the state file. Additionally, it is recommended to use DynamoDB for state locking to avoid race conditions.
Conclusion
Terraform remote backends, particularly with Amazon S3, provide a scalable and secure solution for managing your Terraform state files. By using S3, you can ensure that your state files are centrally stored, accessible by multiple team members, and protected with strong security controls. This setup helps eliminate issues like state file conflicts and provides an efficient way to manage infrastructure in team-based environments.
The example above shows how easy it is to set up a remote backend with S3, making Terraform more collaborative and robust for your infrastructure management needs.
🚀 Let’s Connect!
If you found this guide helpful, follow me for more DevOps and Cloud Engineering content:
🔗 GitHub → github.com/Dhanika-Kumarasiri
🔗 Medium → medium.com/@dhanika-kumarasiri
Have questions? Drop them in the comments! Let’s automate AWS the smart way! 🚀