Tom's Bits

I need to rename a terraform resource created with a for_each over a loop. I can just change the terraform and apply, which results in a destroy/create. Not terrible in this case, but I want something better just by changing state. Best of all, I want to script this.

Situation

My resource needs to go from:

resource "aws_ssm_parameter" "ecs_api_parameter" {
  for_each = var.ecs_api_parameters
  name     = each.key
  type     = "String"
  value    = each.value
}

To:

resource "aws_ssm_parameter" "ecs_parameter" {
  for_each = var.ecs_parameters
  name     = each.key
  type     = "String"
  value    = each.value
}

Terraform plan tells me it’ll be destroy/create –

$ terraform plan

# aws_ssm_parameter.ecs_api_parameter["one"] will be destroyed
# (because aws_ssm_parameter.ecs_api_parameter is not in configuration)
  - resource "aws_ssm_parameter" "ecs_api_parameter" {
<snip>
# aws_ssm_parameter.ecs_parameter["one"] will be created
  + resource "aws_ssm_parameter" "ecs_parameter" {
<snip>
Plan: 4 to add, 0 to change, 4 to destroy.

That would be okay in this case – these are just parameters, there’s no interesting history I want to keep, there’s no impact at all. There will be times you don’t want to do this – that DynamoDB table may well have a backup, but if you don’t have to restore it then life’s easier.

Instead we can just tell terraform the resources have changed their names, and we do this by working on the state file.

Changing resources in state

We cam work out what needs to happen just be looking at the code, but inspecting the plan output is rather helpful. We can see that we have these resources:

aws_ssm_parameter.ecs_api_parameter["one"]
aws_ssm_parameter.ecs_api_parameter["two"]
aws_ssm_parameter.ecs_api_parameter["ninety-nine"]
aws_ssm_parameter.ecs_api_parameter["5"]

Which need to become these:

aws_ssm_parameter.ecs_parameter["one"]
aws_ssm_parameter.ecs_parameter["two"]
aws_ssm_parameter.ecs_parameter["ninety-nine"]
aws_ssm_parameter.ecs_parameter["5"]

The terraform state mv command lets us do this easily enough. The basic solution is going to be along the lines of:

terraform state mv aws_ssm_parameter.ecs_api_parameter["one"] aws_ssm_parameter.ecs_parameter["one"] 
terraform state mv aws_ssm_parameter.ecs_api_parameter["two"] aws_ssm_parameter.ecs_parameter["two"] 
terraform state mv aws_ssm_parameter.ecs_api_parameter["ninety-nine"] aws_ssm_parameter.ecs_parameter["ninety-nine"] 
terraform state mv aws_ssm_parameter.ecs_api_parameter["5"] aws_ssm_parameter.ecs_parameter["5"] 

Four resources, I can do some copy and paste, it’s easy. The thing is there could be an awful lot more than four, and manual stuff can easily go wrong. Okay, scripting can too, but bear with me here.

Scripting this

I start with a dead basic one-liner to iterate over the output of terraform state list, grepping for the resource type, saying what I’ve found:

$ for param in terraform state list | grep aws_ssm_parameter.ecs_api_parameter; do echo $param ; done

aws_ssm_parameter.ecs_api_parameter["one"]
aws_ssm_parameter.ecs_api_parameter["two"]
aws_ssm_parameter.ecs_api_parameter["ninety-nine"]
aws_ssm_parameter.ecs_api_parameter["5"]

Next I want to get just that string so I can construct my terraform state mv command, I’ll use cut here to only give me what’s between the quotes:

$ for param in terraform state list | grep aws_ssm_parameter.ecs_api_parameter | cut -d '"' -f 2 ; do echo $param ; done

one
two
ninety-nine
5

Now to test I’ll try a terraform state show – something totally non-destructive. I need to escape the quotes in the terragrunt command or it doesn’t work. Try it and see…

$ for param in terraform state list | grep aws_ssm_parameter.ecs_api_parameter | cut -d '"' -f 2 ; do terraform state show aws_ssm_parameter.ecs_api_parameter[\"$param\"] ; done

# aws_ssm_parameter.ecs_api_parameter["one"]:
resource "aws_ssm_parameter" "ecs_api_parameter" {
arn = "arn:aws:ssm:eu-west-2:xxx:parameter/one"
data_type = "text"
id = "/one"
name = "/one"
<snip>

Then we can switch out the show with a mv to try the terraform state mv command – but I’m going to use the -dry-run option because I’m a coward:

$ for param in terraform state list | grep aws_ssm_parameter.ecs_api_parameter | cut -d '"' -f 2 ; do terraform state mv -dry-run aws_ssm_parameter.ecs_api_parameter[\"$param\"] aws_ssm_parameter.ecs_parameter[\"$param\"]; done

Would move "aws_ssm_parameter.ecs_api_parameter[\"one\"]" to "aws_ssm_parameter.ecs_parameter[\"one\"]"
Would move "aws_ssm_parameter.ecs_api_parameter[\"two\"]" to "aws_ssm_parameter.ecs_parameter[\"two\"]"
Would move "aws_ssm_parameter.ecs_api_parameter[\"ninety-nine\"]" to "aws_ssm_parameter.ecs_parameter[\"ninety-nine\"]"
Would move "aws_ssm_parameter.ecs_api_parameter[\"5\"]" to "aws_ssm_parameter.ecs_parameter[\"5\"]"

If I were being a real coward I’d change my loop more to pick a single resource before going for it, but I’m happy with this.

$ for param in `terraform state list | grep aws_ssm_parameter.ecs_api_parameter |  cut -d '"' -f 2` ; do terraform state mv aws_ssm_parameter.ecs_api_parameter[\"$param\"] aws_ssm_parameter.ecs_parameter[\"$param\"]; done

Move "aws_ssm_parameter.ecs_api_parameter[\"one\"]" to "aws_ssm_parameter.ecs_parameter[\"one\"]"
Successfully moved 1 object(s).
Move "aws_ssm_parameter.ecs_api_parameter[\"two\"]" to "aws_ssm_parameter.ecs_parameter[\"two\"]"
Successfully moved 1 object(s).
Move "aws_ssm_parameter.ecs_api_parameter[\"ninety-nine\"]" to "aws_ssm_parameter.ecs_parameter[\"ninety-nine\"]"
Successfully moved 1 object(s).
Move "aws_ssm_parameter.ecs_api_parameter[\"5\"]" to "aws_ssm_parameter.ecs_parameter[\"5\"]"
Successfully moved 1 object(s).

And now if I try a plan?

$ terraform plan

aws_ssm_parameter.ecs_parameter["one"]: Refreshing state... [id=/one]
aws_ssm_parameter.ecs_parameter["one"]: Refreshing state... [id=/two]
aws_ssm_parameter.ecs_parameter["ninety-nine"]: Refreshing state... [id=/ninety-nine]
aws_ssm_parameter.ecs_parameter["5"]: Refreshing state... [id=/5]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

So yes, it worked!

[WROTE THIS WITH A HEAVY COLD – I NEED TO CHECK IT]

I like for_each. I like modules that use for_each and create a bunch of resources based on an input map. But is this the best way?

What do I mean? A super basic module for parameter store might contain a single resource with a for_each iterating through the input map variable:

resource "aws_ssm_parameter" "store" {
  for_each = var.store

  name        = each.value.name
  description = each.value.description
  type        = each.value.type
  value       = each.value.value
}

And we’d call it as:

module "parameters" {
  source = "../..//modules/parameter-store"

  store = {
    beverley_hills_s3_bucket = {
      name        = "s3/buckets/beverley_hills_s3_bucket"
      description = "Bucket for 1990s TV show and more"
      type        = "String"
      value       = "s3a://bucket-90210/temp"
    }
    smokey_s3_bucket = {
      name        = "s3/buckets/smokey_s3_bucket"
      description = "An unusually mobile bucket"
      type        = "String"
      value       = "s3a://bucket-20252/temp"
    }
  }
}

And then we’d have a couple of parameters in SSM Parameter Store.

Why do I like this? I don’t have to call the module once per group of allied resources, which means my code is little DRYer. Why not a count through a list (instead of for_each through. map)? Because count really sucks.

What are the downsides though?

Simple modules are fine with this, but the more complex the set of resources, the larger and deeper the map has to get, and the muckier dynamic blocks can get. Instead, I’m looking at returning to the normal module method, assuming a single set of resources (which might include a map – for example, a map of ECS services and task definitions for the cluster that’s being managed).

The way forward will be to call the module once with a for_each inside that, so our single resource module goes back to a vanilla:

resource "aws_ssm_parameter" "store" {
  name        = var.name
  description = var.description
  type        = var.type
  value       = var.value
}

With the module call using a map and for_each like:

locals {
  store = {
    beverley_hills_s3_bucket = {
      name        = "s3/buckets/beverley_hills_s3_bucket"
      description = "Bucket for 1990s TV show and more"
      type        = "String"
      value       = "s3a://bucket-90210/temp"
    }
    smokey_s3_bucket = {
      name        = "s3/buckets/smokey_s3_bucket"
      description = "An unusually mobile bucket"
      type        = "String"
      value       = "s3a://bucket-20252/temp"
    }
  }
}

module "parameters" {
  source = "../..//modules/parameter-store"

  for_each = locals.store

  name        = each.value.name
  description = each.value.description
  type        = each.value.type
  value       = each.value.value
}

I’ve just recertified my Terraform Associate certification. Why? One set of reasons relates to my employer – I understand it’s useful for our relationship with Hashicorp, may look good to our customers, and it’s certainly good for my KPIs. So what?

Are there other reasons? Should other people bother with this? Is it really worth looking at as employer? Yes, yes and yes. And here I begin, followed by some exam tips and next steps…

Terraform Associate value

The syllabus covers a load of domains, but basically if you cover it sufficiently through a course (or following the exam guides) you’ll have knowledge of:

• Basic IaC and how Terraform works – what we want (our config), what terraform thinks we have (state) and what’s in place in AWS (etc) – and how he keep this all together.
• Terraform providers – how the core system does the basics, but plugins (providers) let you work with CSPs or… well, I’ve used it for GitHub repos and I found a not-quite-mature-enough plugin for LogicMonitor.
• Terraform language, variables, functions, and how to use modules to make life simpler.
• Terraform commands – so how to make it actually do something with the config you’ve written, also how to play games with the state file.
• Terraform workspaces and Terraform Cloud – neither of which I’ve used, but you need these for the exam, and I’ve no doubt they’re good to know.

Terraform Associate value as a developer

So the syllabus covers the essentials of Terraform. So what? Well, if you abide by the syllabus coupled with experience or a decent course then you’ll understand some key areas:

• Basic use
  Seems obvious, but you do need to get to grips with the language and commands. Sadly I reckon it’d be possible to pass the exam without playing with code, but I think it’d be faster to learn by doing. Follow the syllabus and any tutorials and you’ll end up with a good grasp of the basics – this’ll help you past the exam, but also give you confidence to use the tool in real life.
• State
  Okay, I was petrified of playing with state for ages, because I first encountered Terraform in a very large and complex environment where there were sometimes state issues – in particular resulting from removing a resource from the middle of a bunch provisioned through a list, also through only running Terraform through Jenkins, and when that went wrong we’d be left with a stale lock which I couldn’t undo. The Assoc cert doesn’t cover state in massive depth, but does teach the basics of reading state and manipulating resources. Just do make sure your state file is versioned 😉
• for_each
  This is a god-send. Count still has its place of course, but I mainly expect this to be 0 or 1 now – it’s just for conditional creation of a resource as far as I’m concerned. What’s wrong with count? Creating resources based on a count results in an ordered list with integer index. The integer index is only an issue in that it’s slightly harder to see what’s what in state (and plan/apply). The biggie is that ordered nature. If you remove an item from the middle of the list then all the items after will need to have a destroy/create as they shift up. The alternative could be some funky state games which could be scripted, but are probably best avoided.
  for_each instead gives us a nicely named bunch of resources, we can insert and remove as we wish.
• Dynamic blocks
  I don’t think they were in the Assoc cert I passed in 2021. They confused me when I first encountered them, but now they’re just normal. The Assoc syllabus might not make you an expert, but it’ll be a good introduction, and this is something you want to get your head around.
• Modules
  Love modules. They’re awesome. You can certainly use Terraform like you’d use Cloudformation (or how I’ve seen it in the one project that used it) – write everything out again and again and again, copying endless blocks of text, or you can embrace DRY and re-use code. Even better, you can re-use someone else’s code.

There’s lots more (workspaces, policies, Terraform Cloud), but not all places use these. Awareness of these will doubtless help get you started in a project using them. but I think you could probably skill-up quickly enough should it be needed.

Terraform Associate value as an employer/customer

I work in a consultancy, so I have two bunches of people to think about – my employer and my prospective customers/projects.

As an employer you want to know there’s a certain level of expertise in your staff. I feel this cert does actually give that. While I first passed and considered myself not very good, the cert had at least drilled into me the basic good practices, added understanding of state. It also taught me about Terraform Cloud and workspaces, neither of which are useful to me yet, but I get they *could* be.

Essentially it provides a base line – if someone’s passed this then they do know some stuff. If they’ve passed it through playing with code, all the better, so it’s worth providing a playground (AWS labs, whatever) so they can create and destroy stuff without much harm. Just make sure the cost implications are understood!

As a customer you probably want to know that prospective team members will work together effectively, quickly. If they’ve all relatively recently passed this then you know there’s a common understanding of the Hashicorp best practices, such as:

• That terraform fmt and validate are good things. These help enormously. Ideally these would just be “normal” in any team, built into workflows – this could be as an IDE plugin (vscode can fmt as you go), through precommit, through cicd. Basically though, if all devs do these in some way from the beginning, life is better.
• That secrets are stored in the right place. You really don’t want a rogue dev putting a secret somewhere silly. The cert covers this pretty well.
• That modules are the way to go. They reduce repeated code and help people deploy standardised resources. You don’t really want to have to change every terraform file that mentions a resource just because there’s now a requirement to add a feature – put this into the module and go.

But a dev needn’t have passed the cert to be good – there are some truly stellar devs I know who’ve not taken it recently (or at all) and I’m in awe of their Terraform-fu – it’s just that if they have recently passed then there’s a baseline you know you can expect. The really awesome devs will just be obvious anyway.

Learning Terraform

Approaching the exam

This depends on your experience.

• If you’re fairly new to Terraform then I think you’d be best served by working a course. If your workplace has one available, ask others what they think of it. Otherwise you could get something something on Udemy (other training providers are available) or schlep through the Hashicorp materials.
• If you’re been using this a lot then you know the language so you’ll probably need to swot up on the things you don’t use every day. I used a general summary of the syllabus to find my weaknesses and dipped into the documentation/tutorials. For what it’s worth, I knew little about Terraform Cloud, workspaces, policies.

Take practice tests! These are worth buying, I used the Bryan Krausen set on Udemy, but you may have others available or recommended. I took four of the five question sets available, passing each one fairly well. I’d intended to take the last on the morning of the exam, but didn’t wake up in time.

Taking the exam

I really hated this. It’s online only, and I far prefer exam centres. It wasn’t helped by me waking up only an hour before my 1pm exam due to my stinking cold.

Here’s what you need to sort out:

• Your laptop needs to have passed the basic checks beforehand. Work laptops can be an issue if they’re locked down, so do this way in advance so you can sort out a Plan B.
• You need to install the Secure Browser, but you’re only able to do this 30 mins before the exam, this threw me. My laptop passed the checks but somehow the secure browser had issues with the web cam. When I then installed it on my work laptop it got past that stage and told me the network connectivity in that back room wasn’t good enough.
• Your test environment needs to satisfy their requirements. The proctor will ask you to show all bits of the room to make sure there’s no funny business going on (people to help, recording devices). Having had to move to a different room this was a bit of a faff, I had to move things out of sight. The proctor asked me to show under the cushions next to me on the sofa, and round the side and back of the sofa. I really get why this is so, it was just unwelcome after the earlier palaver.

In the exam itself, do not speak. I uttered “Oh my gawd” (at feeling so rough with my cold) and my screen flashed up a warning about talking to myself.

The actual exam itself is multi choice (single or pick two/three), and there was one question expecting me to type a single “thing” (being careful with the NDA here!). Normal rules and methods apply – if not sure then eliminate the stupid and then work out the most likely, flag things you have a doubt about. Sometimes a bit of info in one question will give you think twice (or give a hint) about another.

Next steps

Use the language. Build stuff and destroy it.

Things to work on or read up on:

• Read and absorb Google’s Terraform best practices. I don’t care if you’re not using GCP, what it says is pretty good all round.
• Now read it again.
• Play with state – create resources manually and import them, get used to looking into the state file and breaking things.
• Create modules, shove them in git and version them. Play with using different module versions for different environments.
• Creating the remote state file using terraform. This feels chicken and egg, how would you approach it? Why? Are there tools that can help you?
• Play with Terragrunt (which might help with both of the above).