Fixing Pipeline Caching issues with Terraform and the AWS provider
I’ve been analyzing and optimizing the performance of our CI/CD pipeline in a current project and encountered some unexpected behavior with Terraform. Since my Googling didn’t lead to useful results, I’m writing this to share my experience. I’ll explain how I identified the reason why Terraform didn’t use the cached providers and how to avoid the underlying problem with platform specific hashes in the Terraform provider lock file. We’re using a private Gitlab instance as the platform to host our code and have a dedicated runner to execute our pipeline. The terraform part of the pipeline is responsible for rolling out code and infrastructure changes and consists of two stages with their own jobs - plan and apply. If you’ve used Terraform before, you probably already guessed, that the plan job creates a Terraform plan file, i.e., the diff between the current and the target state. The subsequent apply job consumes that and executes the changes (unless something else touched the state in the mean time). Depending on the configuration, the apply is sometimes automated and in other cases manual, which is one of the reasons for separating the two steps. ...