Terraform and CloudFormation are provisioning tools designed to provision infrastructure, whereas Ansible, Saltstack, Chef, and Puppet are configuration management tools designed to install and manage software on existing servers.Terraform can perform some degree of configuration management, but its primary focus is on provisioning.
Terraform’s language is declarative, describing the intended goal rather than the steps needed to achieve it. In contrast, imperative tools specify step-by-step instructions for achieving the desired end state. Terraform works well on Windows, Linux, macOS, and BSD.
There are two ways to install Terraform: using a binary package by downloading it from the Terraform website or using a package manager.
After installing Terraform, you can run the command terraform -install -install-autocomplete to add the line complete -C /usr/local/bin/terraform terraform to your .bashrc file for auto-completion.
When implementing a new application, one of the critical decisions is whether to choose a cloud-based infrastructure or an on-premises solution. On-premises infrastructure refers to the traditional approach where everything is installed locally on the company’s servers, requiring procurement, management, and maintenance of hardware and software resources. Cloud computing offers a different approach, providing on-demand access to networking resources without the need to procure and manage infrastructure in advance.
One of the key benefits of cloud computing is its scalability, allowing businesses to instantly spin up hundreds or thousands of servers in just a few minutes. This level of agility is not possible with on-premises infrastructure, where procurement and deployment of new hardware and software can take weeks or months. However, on-premises infrastructure still has its advantages, such as full control over infrastructure, specific security or compliance requirements, and customization.
Terraform is a DevOps tool that allows engineers to automate and manage data center infrastructure, platforms, and services. It’s an open-source infrastructure as code tool created by Hashicorp, enabling businesses to define their infrastructure in a human-readable configuration file and create, modify, and version control their infrastructure across various cloud and on-premises environments.
Provisioning refers to the process of acquiring and setting up the physical components required to run specific applications. In the context of cloud computing, provisioning can be done automatically using tools like Terraform, allowing businesses to quickly and efficiently deploy the resources they need to run their applications.
In summary, when deciding between cloud and on-premises infrastructure, businesses need to consider factors such as scalability, control, security, and compliance. Terraform is a powerful tool that can help businesses automate and manage their infrastructure, whether they choose a cloud-based or on-premises solution.
In Terraform, a module is a directory containing one or more configuration files (.tf) that define a set of resources and their dependencies. The root module is the main module containing the top-level configuration for your infrastructure. When you run Terraform, it searches for configuration files with the .tf extension in the working directory and its subdirectories. All the configuration files in the working directory and its subdirectories form the root module. You can create additional modules by creating directories and placing configuration files inside them. These modules can call other modules to connect them together and create a hierarchical structure.
For your Terraform project, you can create a directory to store all your configuration files. The main module typically contains the main set of configurations for your infrastructure. As your project grows, it’s a good idea to split the main module into smaller, more focused modules to make management and maintenance easier. You can do this by creating new directories and configuration files and updating the main module to import and use the new modules.
Here’s an example of a simple Terraform project structure:
├── main.tf
├── modules
│ ├── network.tf
│ ├── storage.tf
│ └── compute.tf
└── provider.tf
In this example, the main.tf file contains the main set of configurations for the infrastructure, and the modules directory contains three sub-modules: network.tf, storage.tf, and compute.tf. Each sub-module defines a set of resources and their dependencies, and can be imported and used in the main module. The provider.tf file defines the cloud provider and is imported by the main module.
When working with Terraform, the first step is to connect to an infrastructure platform such as AWS, GCP, or Azure. Terraform uses a provider to establish this connection and manage resources within the platform. Without a provider, Terraform cannot interact with any infrastructure.
Providers in Terraform are plugins that enable Terraform to interact with specific infrastructure platforms. Each provider has its own set of resources and functions that it manages. The Terraform website provides documentation for each provider, including information on what resources it manages and how to use it.
It’s important to note that the provider is a crucial component of Terraform, as it allows Terraform to communicate with the infrastructure platform and manage resources. Without a provider, Terraform would not be able to interact with the infrastructure, and therefore would not be able to provision or manage resources.
In summary, the main.tf file contains the main set of configurations for the infrastructure, and the modules directory contains sub-modules that define specific resources and their dependencies. The provider.tf file defines the cloud provider and is imported by the main module, allowing Terraform to interact with the infrastructure platform and manage resources.
https://registry.terraform.io/browse/providers
To use a provider, Terraform needs to download and install it. This is done by running the Terraform init command in the terminal, which initializes the working directory and downloads the required providers. Once the providers are installed, Terraform can be used to create and manage resources on the connected infrastructure platform.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
# Configure the AWS Provider
provider "aws" {
region = "us-east-1"
}In the above Terraform config file, each argument in the required_providers block enables one provider. Source is the global and unique source address for the provider, specifying the primary location where Terraform can download the provider. Version is the version constraint, specifying which subset of available provider versions the module is compatible with.
Terraform expects files with the native syntax (HCL) to be named with a .tf or files with the JSON syntax to be named with .tf.json suffix.
Block Type “lable” “label” {
Block body , contains multiple arguments
}Note : Using http and https proxy in terraform:
For Linux/Mac environment , set the proxy property like below in dos prompt or permanently in Linux/Mac environment variable.
export HTTPS_PROXY=”https://username:password@ram-tin.top:port”
export NO_PROXY=”grafana.local,10.10.10.1,etc.com
Steps in Terraform
In Terraform, we follow a three-step process to manage our infrastructure.
First, we write Terraform configuration files that describe the desired state of our infrastructure. This includes defining resources, their properties, and how they relate to each other.
Second, we use the terraform plan command to preview the changes that will be made to our infrastructure. This step is crucial in ensuring that we’re making the right changes, as it compares the current state of our infrastructure to the desired state defined in our Terraform configuration files. The plan command generates an execution plan that highlights the differences between the current state and the desired state, providing us with valuable information about what will be added, changed, or deleted. This step is similar to using Ansible’s dry-run mode, where we can see what changes will be made before they’re applied to the environment.
Finally, we use the terraform apply command to apply the changes to our infrastructure. This command takes the execution plan generated by the plan command and makes the necessary changes to our infrastructure. By following this three-step process, we can confidently manage our infrastructure using Terraform, knowing that we have a clear understanding of what changes will be made and how they will impact our environment.
In Terraform, we can use the -out option in the terraform plan command to save the plan to a file, which can then be used with the terraform apply command to apply the changes to our infrastructure. By using the -out option, we can specify a file name and path, and the plan will be saved to that file. This allows us to easily rerun the same plan later or share it with others for review or collaboration. For example, we can use the following command to save the plan to a file named plan.txt:
terraform plan -out plan.txt
This will generate a plan file that contains the desired state of our infrastructure, along with any changes that need to be made to achieve that state. We can then use the terraform apply command to apply the changes defined in the plan file.
terraform apply plan.txt
By using the -out option and saving the plan to a file, we can easily re-run the same plan later or share it with others for review or collaboration. This makes it easier to manage our infrastructure and collaborate with others on complex projects.
In Terraform, there are two useful subcommands that can help you make your configuration files more readable and validate their syntax.
The first subcommand is terraform fmt, which reformats your Terraform configuration files to make them more readable. You can run this command in the current directory to format all Terraform configuration files in the directory, or you can specify a directory path as an argument to format files in a specific directory. Note that the terraform fmt command does not check subdirectories by default, but you can use the -reverse option to include subdirectories in the formatting process.
The second subcommand is terraform validate, which checks the syntax of your Terraform configuration files and tells you if they are valid or not. If there are any errors in the configuration files, the terraform validate command will provide tips on how to fix them. This subcommand is particularly useful for catching syntax errors before you apply your Terraform configuration changes to your infrastructure. By using these two subcommands, you can ensure that your Terraform configuration files are well-formatted and error-free, making it easier to manage your infrastructure and collaborate with others on complex projects.
Deleting Resources with Terraform
To delete or destroy resources in a cloud provider using Terraform, you have two options:
- Run the terraform destroy command in your Terraform project directory. This will remove all resources created by the current Terraform configuration. If you want to delete a specific resource, you can use the -target option followed by the resource type and local name, for example: terraform destroy -target resource_type.resource_localname. Use the -auto-approve option to skip the approval prompt.
- Remove or comment out the resource from your Terraform configuration file and run terraform apply again. Terraform will read the updated configuration and remove the resource.
Note that Terraform keeps a history of changes for each project, so you can’t delete resources created by other Terraform projects or directly in the platform. Only resources created by the current Terraform project can be deleted.
To replace or recreate a resource with Terraform, you can use the -replace option in the terraform apply command. This will destroy the existing resource and create a new one with the same name and type. For example, imagine you have an EC2 machine that you created with Terraform, but you made a mistake inside the machine and want to recreate it.
If you run terraform apply without the -replace option, Terraform won’t create a new resource because it thinks the existing resource is still valid.
terraform apply -replace "resource_type.resource_localname"
This will destroy the existing resource with the specified name and type, and then create a new one with the same name and type. In older versions of Terraform, you could use the taint subcommand to taint a resource, and then use the apply command to replace the tainted resource. However, the taint subcommand is now deprecated, and the -replace option is the recommended way to replace or recreate a resource. (terraform taint <resource_type.resource_localname> then terraform apply )
How to Make a Terraform Project Dynamic with Variables
In Terraform, you can define variables to make your configuration more dynamic and reusable. Variables allow you to store values that can be used throughout your configuration files, making it easier to manage and maintain your infrastructure.
Defining Variables
To define a variable in Terraform, you can use the variable block in your configuration file. For example:
variable "vpc_cidr_block" {
default = "10.0.0.0/16"
description = "CIDR Block for the VPC"
type = string
}You can define variables in the main.tf file or in a separate variables.tf file in the project directory. It’s considered best practice to define variables in a separate file to keep your configuration organized.
Using Variables
Once you’ve defined a variable, you can use it in your configuration files by prefixing the variable name with var. , For example:
resource "aws_subnet" "web" {
vpc_id = aws_vpc.main.id
cidr_block = var.vpc_cidr_block
availability_zone = var.subnet_zone
tags = {
"Name" = "Web subnet"
}
}In this example, we are using the var.vpc_cidr_block and var.subnet_zone variables to dynamicize the CIDR block and availability zone of the web subnet.
Entering Variable Values
There are several ways to enter values for your variables:
- Interactive Input: When you run Terraform for the first time, it will prompt you to enter values for any variables that don’t have a default value set. If you define the default value , it won’t prompt for value.
- Terraform.tfvars: You can define values for your variables in a file named terraform.tfvars in the project directory. For example:
vpc_cidr_block = "10.0.0.0/16"
web_subnet = "10.0.100.0/24"
subnet_zone = "eu-central-1a"- –var-file Option: You can pass a file with variable values using the -var-file option when running Terraform. For example:
terraform apply -var-file=productionvars.tfvars
- –var Option: You can pass variable values using the -var option when running Terraform. For example:
terraform apply -var=”web_subnet=192.168.1.0/24”
- Shell Variables: You can define values for your variables in shell variables that start with TF_VAR_. For example:
export TF_VAR_web_subnet=”192.168.1.0/24”
The priority of these methods is as follows:
- –var or -var-file option: These methods take precedence over any values defined in the terraform.tfvars file or shell variables.
- terraform.tfvars: Values defined in this file take precedence over shell variables.
- Shell Variables: Values defined in shell variables take precedence over interactive input.
Using Variables in Strings
To use a variable in a string, you can concatenate the variable name with the string using the ${var.variable_name} syntax. For example:
“name” = “Production ${var.web_subnet}”This will output a string that includes the value of the web_subnet variable.
With terraform you can create new resources and you can delete the resources which are managed by the current terraform project. But what if we want to edit or add something to a resource which already has been created?
For example when you created a VPC, aws created a default route table and security group with that VPC; and if you want to add route to routeing table or rule to security group , we should use aws_default* type of resource and say add specific route or rule to default routing table or security group:
# Default Security Group
resource "aws_default_security_group" "default_sec_group" {
vpc_id = aws_vpc.main.id
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
# cidr_blocks = [var.my_public_ip]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "Default Security Group"
}
}Note: In a Security Group, there are two types of traffic: ingress and egress. Ingress refers to traffic that is incoming to an EC2 instance, while egress refers to traffic that is outgoing from an EC2 instance. A Security Group is a stateful firewall, which means that if you allow incoming or outgoing traffic, it will automatically allow the return packet (established or related) without the need to explicitly allow it. By default, all traffic is denied, so you only need to allow the specific traffic that you want.
On the other hand, a Network ACL is a stateless firewall that is placed in front of a Security Group and behind Routing table. By default, all traffic is allowed, and you need to explicitly deny or allow specific traffic.
Note: If you set tags for resources in your Terraform configuration file and then change a tag after initially applying the configuration, Terraform will not recreate the resource when you re-run the plan. This is because Terraform uses the tags to identify resources and avoid recreating them unnecessarily. If you want to update a resource with a new tag, you’ll need to remove the old tag from the resource before re-running Terraform.
Data Sources in Terraform
Each Provider may offer data sources alongside a set of resource types.
What are the data sources?
Data sources are a type of API that are used to fetch dynamic data from cloud providers. They are similar to queries and are used to retrieve data that changes frequently. Examples of data sources include a list of AMIs (Amazon Machine Images) that changes frequently or a list of availability zones.
For example, let’s say you want to create a Terraform configuration that retrieves a list of available AMIs from Amazon Web Services (AWS) and uses that list to create a new EC2 instance. You could use a data source to fetch the list of AMIs and then reference that data source in your Terraform configuration. Here’s an example of what that configuration might look like:
data "aws_ami" "available_amis" {
most_recent = true
owners = ["amazon"]
}
resource "aws_instance" "example" {
ami = data.aws_ami.available_amis.ids[0]
instance_type = "t2.micro"
vpc_security_group_ids = [aws_security_group.example.id]
subnet_id = aws_subnet.example.id
key_name = "my_key"
}In this example, the data "aws_ami" "available_amis" block fetches a list of available AMIs from AWS using the aws_ami data source. The most_recent = true argument tells Terraform to retrieve the most recent version of the AMI. The owners = ["amazon"] argument specifies that we want to retrieve AMIs that are owned by Amazon.
The resource "aws_instance" "example" block then references the data.aws_ami.available_amis.ids[0] data source to retrieve the ID of the first AMI in the list. It uses that ID to create a new EC2 instance with the specified instance type, security group, subnet, and key pair.
By using a data source to fetch the list of available AMIs, we can easily retrieve dynamic data from AWS without having to hardcode the list of AMIs in our Terraform configuration. This makes our configuration more flexible and easier to maintain over time.
OutPut in Terrafrom:
Terraform output values are a way to export structured data about your resources, similar to return values of functions in programming languages. These output values can be used to configure other parts of your infrastructure or by a child model. Additionally, the root module can use outputs to print values at the terminal after running terraform apply. Output declarations can appear anywhere in your Terraform configuration files, but it’s recommended to put them into a separate file called outputs.tf
This helps keep your configuration organized and makes it easier to manage your infrastructure. When defining an output, the label after the output keyword is the name of the value that will be exported. For example:
output "example_output" {
value = "This is an example output value"
}Terraform stores output values in its state file, and you can use the terraform output command to query all of them. If you add the -json option to this command, it will give you the result in JSON format. If you want to hide the value of an output, you can add sensitive = true to the output definition. This will prevent Terraform from showing the value of that output with the terraform output command. To view the value of a sensitive output, you can either put the name of the output in front of the terraform output command or use the -json option in the command. Here’s an example of a sensitive output:
output "sensitive_output" {
value = "This is a sensitive output value"
sensitive = true
}To view the value of the sensitive output, you can use the following command: terraform output -json Or, you can specify the name of the output in front of the command: terraform output <output_name> In summary, Terraform outputs allow you to export structured data about your resources, which can be used to configure other parts of your infrastructure or by a child model. Outputs can be declared anywhere in your configuration files, but it’s recommended to put them into a separate file. Sensitive outputs can be hidden from the terraform output command by adding sensitive = true to the output definition.
What is The Terraform State:
Terraform state is a critical component of Terraform that stores information about the managed infrastructure and configuration. The state is used by Terraform to map real-world resources to the configuration, keep track of metadata, and improve performance. By default, the state is stored in a local file called terraform.tfstate, but it can also be stored remotely, which is useful in a team environment.
The state file contains bindings between remote objects in the cloud and resources declared in the configuration file. When Terraform creates or destroys an object, it records the identity of the object in the state file.
The state file also contains all the attributes of each resource, including sensitive data like database passwords, user passwords, or private keys. Terraform provides several commands to interact with the state file:
- terraform show : This command generates a JSON output of all the resources contained in the state file.
- terraform state list : This command lists all the resource types and local names in the state file.
- terraform state show <resource type>.<resource name>: This command shows all the attributes of a single resource in the state file.
It’s important to note that direct file editing of the state file is not recommended, as it can lead to inconsistencies and errors. Instead, Terraform provides a mechanism to perform basic modifications of the state using the command line.When using remote state, the state file is only held in memory when used by Terraform, which provides better security than storing sensitive data in a local file. When applying changes, Terraform first refreshes the state file and then applies changes from that state file.
By default, Terraform will create a backup file of the state file whenever the state file is updated. The backup file is named terraform.tfstate.backup and is stored in the same directory as the state file. The backup file contains a copy of the state file as it was before the update, and it’s used to restore the state file in case of a failure or data loss.
This ensures that even if the state file becomes corrupted or is accidentally deleted, you can still restore the state file to its previous state using the backup file. It’s important to note that the backup file is not automatically updated when the state file is updated. Instead, Terraform will only create a new backup file when the state file is updated, and the previous backup file will be preserved. This means that you can have multiple backup files for different versions of the state file, which can be useful for auditing or debugging purposes.
In addition to the default backup file, you can also specify a custom backup file using the -backup flag when running the terraform command. This allows you to choose a different location or name for the backup file, or to disable the backup feature altogether.
Overall, the backup file of the state file is an important feature of Terraform’s state management, and it helps ensure that your infrastructure is protected against data loss or corruption.
In summary, Terraform state is a critical component that stores information about the managed infrastructure and configuration. It’s used to map real-world resources to the configuration, keep track of metadata, and improve performance. The state file can be stored locally or remotely, and Terraform provides several commands to interact with the state file. However, direct file editing is not recommended, and remote state provides better security for sensitive data.
Run Commands inside our machines
Let’s see how to run commands to customize the instance after it gets initialized :
You can do this in more than one way:
You can run command or scripts by giving them a value to an attribute called user_data
You could use cloud-init which is industry standard for cloud instance initialization
You can use packer which is an open source DevOps tools made by HashiCorp to create imaged from a single json file Or you could use Provisioners .
Let’s talk about the first method, which is used to run command on the EC2 instance using user_data The user_datat passes the command to the cloud provider and the cloud provider will run the command on the instance , so it’s idiomatic and native to the cloud provider Note that Terraform will pass control to the cloud so it will have no information about the running commands if they are successful or not
Cloud-init is the de facto standard for initializing cloud instances, and it’s available on most Linux distributions. It’s a versatile tool that supports both public and private cloud providers, as well as bare metal installations. Cloud-init can configure instances per-instance or per-boot, and it needs to determine whether the current boot is the first boot of a new instance or not. To do this, cloud-init stores a cache of its internal state to keep track of whether it has run on this instance before. On the first boot of a new instance, cloud-init runs all per-instance configuration. On subsequent boots, it runs per-boot configuration, using the cached state to avoid re-running unnecessary configuration steps.
Terraform provides another way to provision resources with software, known as provisioners. A provisioner is a block of code that runs on the resource after it’s created, allowing you to customize the resource without having to use user data. While user data is idiomatic and recommended for customizing resources, provisioners are specific to Terraform and can add complexity and uncertainty to your configuration. Provisioners are a last resort when there are no other alternatives, and they break the declarative model of Terraform. They also require Terraform to connect to the instance using SSH, which can be a security concern.
In Terraform, the “self” object represents the parent resource and has access to all of its attributes. This can be useful when writing provisioners that need to interact with the resource they’re provisioning.
Here’s an example of a provisioner that sets up a simple web server on an EC2 instance:
resource "aws_instance" "example" {
ami = "ami-12345678"
instance_type = "t2.micro"
vpc_security_group_ids = [aws_security_group.example.id]
subnet_id = aws_subnet.example.id
key_name = "my_key"
provisioner "remote-exec" {
connection {
type = "ssh"
host = self.public_ip
user = "root"
private_key = file("~/.ssh/my_key")
}
inline = [
"sudo yum install -y nginx",
"sudo systemctl start nginx",
"sudo systemctl enable nginx"
]
}
}In this example, the provisioner "remote-exec" block runs a series of shell commands on the EC2 instance after it’s created. The connection block specifies the connection details, including the host IP address, user name, and private key. The inline block specifies the commands to run, which in this case install and configure a simple web server using Nginx.
Note that this is just a simple example, and in a real-world scenario, you would likely want to handle errors and edge cases, and also ensure that the provisioner is idempotent (i.e., it can be run multiple times without causing issues). Additionally, you may want to consider using a configuration management tool like Ansible or Puppet to manage the software configuration of your instances, rather than relying on Terraform provisioners.
Variable in Terraform: A Comprehensive Guide
In Terraform, variables are used to store values that can be used throughout your configuration files. Variables can be simple or complex, and they can be used to store a wide range of values, from single values to entire lists or maps. In this guide, we’ll take a closer look at the different types of variables in Terraform and how they can be used.
Simple Variables
Simple variables in Terraform are used to store a single value. There are four simple types in Terraform: number, string, bool, and null. Here are some examples of how each type can be used:
- Number: You can use a number variable to store a numerical value, such as a port number or a version number. For example:
variable "port" {
type = number
default = 8080
}- String: You can use a string variable to store a text value, such as a server name or a path to a file. For example:
variable "server_name" {
type = string
default = "example-server"
}- Bool: You can use a bool variable to store a boolean value, such as a flag to indicate whether a feature is enabled or disabled. For example:
variable "enabled" {
type = bool
default = true
}- Null: You can use a null variable to store the absence of a value. For example:
variable "optional_value" {
type = null
default = null
}Complex Variables
Complex variables in Terraform are used to store multiple values. There are two categories of complex variables: collection and structural.
Collection Variables
Collection variables are used to store a group of values of the same type. There are three collection types in Terraform: list, map, and set. Here are some examples of how each type can be used:
- List: You can use a list variable to store a list of values, such as a list of server names or a list of IP addresses. For example:
variable "server_names" {
type = list(string)
default = ["server1", "server2", "server3"]
}- Map: You can use a map variable to store a map of key-value pairs, such as a map of environment variables or a map of configuration settings. For example:
variable "env_variables" {
type = map(string)
default = {
"DB_HOST" = "localhost"
"DB_USER" = "root"
"DB_PASSWORD" = "password"
}
}- Set: You can use a set variable to store a set of unique values, such as a set of IDs or a set of names. For example:
variable "ids" {
type = set(number)
default = [1, 2, 3, 4, 5]
}Structural Variables
Structural variables are used to store a group of values of different types. There are two structural types in Terraform: tuple and object. Here are some examples of how each type can be used:
- Tuple: You can use a tuple variable to store a group of values that are related, such as a group of database connection settings. For example:
variable "db_connection" {
type = tuple(string, string, string)
default = ("localhost", "root", "password")
}- Object: You can use an object variable to store a group of key-value pairs, such as a group of configuration settings. For example:
variable "config" {
type = object({
"server" = string
"port" = number
})
default = {
"server" = "example-server"
"port" = 8080
}
}we’ve learned about the different types of variables in Terraform and how they can be used to store a wide range of values. By using variables in your Terraform configuration files, you can make your configurations more flexible, reusable, and easier to manage.
Terraform Looping Techniques: count and for_each
In Terraform, there are times when you want to create multiple similar resources without having to write a separate block for each one. For example, you might want to create multiple virtual machines with similar configurations, or multiple databases with the same schema. To accomplish this, Terraform provides two looping techniques: count and for_each.
Count
The count technique is used to create a fixed number of resources. It allows you to specify a single block of configuration that will be repeated a specified number of times. Here’s an example of how you might use count to create three virtual machines with similar configurations:
variable "vm_count" {
type = number
default = 3
}
resource "virtualmachine" "example" {
count = var.vm_count
name = "example-vm-$count.index"
cpu = 2
memory = 4096
disk {
name = "example-vm-$count.index-disk"
size = 30000
}
}In this example, we’ve defined a variable vm_count with a default value of 3. We’ve then used the count argument in the resource block to specify that we want to create three virtual machines. The $count.index syntax in the name and disk blocks tells Terraform to use the current value of the count loop as a suffix for the resource names.
For Each
The for_each technique is used to create a dynamic number of resources based on a collection of values. It allows you to specify a block of configuration that will be repeated for each element in a list or map. Here’s an example of how you might use for_each to create a list of databases with a dynamic number of databases:
variable "db_names" {
type = list(string)
default = ["db1", "db2", "db3"]
}
resource "database" "example" {
for_each = var.db_names
name = each.value
host = "localhost"
port = 5432
}In this example, we’ve defined a variable db_names with a default value of a list of three database names. We’ve then used the for_each argument in the resource block to specify that we want to create a database for each element in the db_names list. The each.value syntax in the name block tells Terraform to use the current value of the for_each loop as the name of the database.
Key Differences between count and for_each
The main difference between count and for_each is that count is used to create a fixed number of resources, while for_each is used to create a dynamic number of resources based on a collection of values. Count is useful when you know the exact number of resources you want to create ahead of time. For example, if you want to create three virtual machines with similar configurations, you can use count to create them all at once. For_each is useful when you don’t know the exact number of resources you want to create ahead of time, or when you want to create resources based on a dynamic list of values. For example, if you want to create a list of databases with a dynamic number of databases, you can use for_each to create a database for each element in a list of database names.
so,by now we’ve learned about the two looping techniques in Terraform: count and for_each. Count is used to create a fixed number of resources, while for_each is used to create a dynamic number of resources based on a collection of values. By using these techniques, you can create similar resources in a more efficient and manageable way.
What is a dynamic nested block?
A dynamic nested block is a way to generate nested blocks of Terraform configuration code dynamically, using a loop. This technique is useful when you need to create a large number of resources or configurations that share similar attributes, but have some differences. Instead of writing out each resource or configuration individually, you can use a loop to generate the nested blocks of code automatically. This can save you a lot of time and effort, and make your Terraform configuration more maintainable and scalable. Here’s an example of a dynamic nested block in Terraform:
variable "security_groups" {
type = list(object({
name = string
description = string
vpc_id = string
subnet_ids = list(string)
ingress_rules = list(object({
protocol = string
from_port = number
to_port = number
cidr_blocks = list(string)
}))
}))
}
resource "aws_security_group" "example" {
for_each = var.security_groups
name = each.value.name
description = each.value.description
vpc_id = each.value.vpc_id
subnet_ids = each.value.subnet_ids
dynamic "ingress_rule" {
for_each = each.value.ingress_rules
content {
protocol = ingress_rule.value.protocol
from_port = ingress_rule.value.from_port
to_port = ingress_rule.value.to_port
cidr_blocks = ingress_rule.value.cidr_blocks
}
}
}
# Create a list of security groups with different names, descriptions, and ingress rules
variable "security_groups" {
[
{
name = "sg-01"
description = "Security group for subnet 1"
vpc_id = "vpc-01"
subnet_ids = ["subnet-01"]
ingress_rules = [
{
protocol = "tcp"
from_port = 22
to_port = 22
cidr_blocks = ["0.0.0.0/0"]
},
{
protocol = "udp"
from_port = 53
to_port = 53
cidr_blocks = ["0.0.0.0/0"]
}
]
},
{
name = "sg-02"
description = "Security group for subnet 2"
vpc_id = "vpc-01"
subnet_ids = ["subnet-02"]
ingress_rules = [
{
protocol = "tcp"
from_port = 80
to_port = 80
cidr_blocks = ["0.0.0.0/0"]
},
{
protocol = "udp"
from_port = 53
to_port = 53
cidr_blocks = ["0.0.0.0/0"]
}
]
},
{
name = "sg-03"
description = "Security group for subnet 3"
vpc_id = "vpc-01"
subnet_ids = ["subnet-03"]
ingress_rules = [
{
protocol = "tcp"
from_port = 443
to_port = 443
cidr_blocks = ["0.0.0.0/0"]
},
{
protocol = "udp"
from_port = 53
to_port = 53
cidr_blocks = ["0.0.0.0/0"]
}
]
}
]
}In this example, we define a list of security groups with different names, descriptions, and ingress rules. We use the for_each argument in the aws_security_group resource to loop over this list, and generate a nested block for each security group. The dynamic block inside the aws_security_group resource is used to generate the ingress_rule block dynamically. The for_each argument in the dynamic block loops over the ingress_rules list in each security group object, and generates a nested block for each ingress rule. This way, you can use dynamic nested blocks to configure security groups in AWS with different attributes, without having to write out each resource or configuration individually. This can save you a lot of time and effort, and make your Terraform configuration more maintainable and scalable.
Applications of dynamic nested blocks
Dynamic nested blocks are useful in a variety of situations, such as:
1. Creating multiple resources with similar attributes
Suppose you need to create multiple resources (e.g., AWS EC2 instances, AWS RDS instances, etc.) with similar attributes, but with some differences. You can use a dynamic nested block to generate the nested blocks of code automatically, and avoid duplicating code.
2. Generating configurations based on data
Dynamic nested blocks can also be used to generate configurations based on data. For example, you can use a loop to generate a list of objects, and then use the for_each argument to generate a nested block for each object in the list.
3. Creating modular and reusable code
Dynamic nested blocks can help you create modular and reusable code. You can define a module that uses a dynamic nested block to generate resources or configurations, and then reuse that module in different parts of your Terraform configuration.
4. Avoiding unnecessary repetition
Dynamic nested blocks can help you avoid unnecessary repetition in your Terraform configuration. Instead of writing out each resource or configuration individually, you can use a loop to generate the nested blocks of code automatically. This can save you a lot of time and effort, and make your Terraform configuration more maintainable and scalable.
Conditional Expression:
“If you have programmed in any other programming language, you may wonder if there is an if statement in Terraform. The answer is no, but you can accomplish the same thing using a conditional expression and the ternary operator. Let’s suppose we want to run different instances for test and production environments. You can make it conditional with a count and by defining a bool variable. If you give the resource configuration in Terraform a count statement with a value of 0, Terraform won’t do anything with it – it won’t remove, create, or replace anything. So, I can define a bool variable and say that if the value of that variable is true, create the resource, or if the value is not true, don’t do anything. Here’s an example of how to define the bool variable:
Variable "istest" {
Type = bool
Default = true
}And then I can add a count statement to my resource like this:
count = var.istest == true ? 1:0This says that if the value of istest is true, place 1 in front of the count, or else put a 0 in front of the count statement.”
Note : In terraform there are many built in functions which you can use but there is not any user defined function in terraform
More details about State File in Terraform
Terraform is a stateful application, meaning that it keeps track of everything it does in your cloud environment. This allows it to make changes for you if needed later. The state is stored in a file called a state file, which is in JSON format. The state file keeps track of everything Terraform does, and it will not exist until you have completed at least one Terraform apply.
Prior to any other operation, Terraform will do a refresh to update the state with the real infrastructure. The local state file, terraform.tfstate, is stored in the same directory as the configuration file, and there is also a backup copy in the same directory. This is suitable for testing, development, or solo work. However, in a production environment, working in a team presents some challenges when using a local state file. The state file must be shared somehow, and team members must ensure they always have the latest state before running Terraform.
Additionally, concurrency is a problem, as multiple team members running Terraform simultaneously can result in unseen current changes and a corrupted state file. To address these issues, Terraform provides a locking mechanism, and the solution is to store the state remotely.
Each Terraform configuration has an associated backend that defines how operations are executed and where the Terraform state is stored. The default backend is local, which stores the state as a plain file in the current working directory. However, Terraform supports storing state on many backends, such as AzureRM, Consul, COS, etcdv3, GCs, Kubernetes, and Amazon S3. For example, if you create an S3 bucket in AWS, you can add the following block inside your main Terraform configuration file or in a separate file like backend.tf:
terraform {
backend "s3" {
bucket = "nameof bucket"
key = "name of state file in bucket"
region = "region where your bucket is in"
acess_key = "access key of user"
secret_key = "secret key of user"
}
}Initializing the project will create the state file remotely in the S3 bucket. If you have a project with a local state file and want to migrate it to a remote state file, you can configure the backend and run terraform init –migrate-state. This will move the state file from local to remote. Note that if the state file is stored locally, locking is enabled by default, and you don’t have to worry about concurrent runs of Terraform or race conditions. However, in remote state, we need to handle concurrent problems. To solve this issue, you can create a DynamoDB table in AWS with a partition key LockID (you can leave other options with their default values). Then, in the Backend block of your configuration, add DynamoDB:
terraform {
backend "s3" {
bucket = "nameof bucket"
key = "name of state file in bucket"
dynamodb_table = "name of dynamodb table"
region = "region where your bucket is in"
acess_key = "access key of user"
secret_key = "secret key of user"
}
}Because the remote configuration has changed, you should run terraform init –reconfigure. This will restore the current configuration to the same remote state.
Using Terraform, we manage the entire infrastructure using code, so the security of the code is crucial. One of the most important rules of security regarding any application or system, not just Terraform, is “don’t store secrets in plain text.” No matter which technique you use to encrypt the secrets, they will still end up in plain text in the Terraform state. Therefore, you should store the Terraform state in a backend that supports encryption, instead of storing it in a local terraform.tfstate file. You can set Terraform to store the state file on a backend that supports encryption, such as Amazon S3. Tips for security:
- Do not store secrets in plain text.
- Use environment variables, encrypted files, or a secret store to securely store and retrieve secrets.
- Use a remote backend that supports encryption, such as Amazon S3 or Terraform Cloud.
Modules in Terraform
As you manage your infrastructure using Terraform, you may create increasingly complex configurations, which can lead to problems. Updating the configuration of a specific section can become risky because it may cause unintended consequences to other parts of your configuration. In this case, complexity can be an enemy. Moreover, you may find yourself repeating large parts of your configuration for different environments, such as development, staging, and production, which is not ideal. Furthermore, if you want to share some parts of the configuration between projects or with your team, you have to copy and paste them, which can lead to errors.
In general-purpose programming languages, if you had the same piece of code copied and pasted in several places, you would normally put that code inside a function and reuse it wherever needed. In Terraform, there are no user-defined functions, but there are modules.
Terraform Modules are a powerful way to use code and adhere to the DRY principle (do not repeat yourself). Modules will help you organize and encapsulate your configuration, reuse it, provide consistency, and ensure best practices. By using modules, you will also reduce errors because you’ll have the code in a single place and import it into different parts of your configuration.
A Terraform module is a set of different configuration files in a single directory. Even the simplest configuration, consisting of a single directory with one .tf file, is considered a module. When you run a Terraform command like terraform plan directly in such a directory, that directory will be considered the root module. The models that are imported from other directories into the root module are called child modules.
There are two types of modules: local and remote. Local modules are loaded from the local file system and are generally created by yourself or other members of your team to organize and encapsulate your code. Remote modules are loaded from a remote source, such as the Terraform Registry, and are created and maintained by HashiCorp, its partners, or third parties. Modules are essential in Terraform, as they are the key to writing reusable, maintainable, and testable Terraform code. It’s a good practice to start building everything as a module.
When working with modules, you should first use the terraform init command to download and configure the modules.
Example For Remote Module from terraform registry :
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
name = "my-ramtin-vpc"
cidr = "10.0.0.0/16"
azs = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
enable_nat_gateway = true
enable_vpn_gateway = true
tags = {
Terraform = "true"
Environment = "dev"
}
}After placing the above configuration in your Terraform configuration file inside the directory of your project, run the command terraform init to download the module from the registry and then apply it to create the resources. You can view the outputs and inputs that are set to this module in the module’s page, and use them in your configuration file.
https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest?tab=resources