`terraform test`: support failure assertions for lifecycle errors

[edgetools]

Terraform Version

> terraform version
Terraform v1.14.0
on darwin_arm64

Use Cases

Currently, terraform test supports assertions for failure conditions using expect_failures.

However, as indicated by the documentation and other similar issues, this feature is only used to test custom conditions.

I have a use case where I am attempting to do opaque-box testing using terraform, meaning I am running a module and then using a "loader" module to verify the output (similar to how the docs describe).

Yet one of the core requirements of my implementation is that prevent_destroy is used to alleviate a deletion risk associated with a specific provider.

Currently, I cannot use expect_failures to assert on a test that attempts to destroy a resource being blocked by terraform.

This is a problem because my tests are designed to maintain the contracts of the terraform module. While I am not worried about terraform itself having a bug in the prevent_destroy feature, I am worried about other contributors to the module I'm using accidentally removing the prevent_destroy lifecycle block, leading to a deletion risk.

Attempted Solutions

Attempted to use expect_failures, but it only works with custom conditions.

Proposal

I propose that the expect_failures feature is either enhanced, or an additional failure assertion is added, which allows for asserting on lifecycle failures or errors generated during a terraform run.

References

• Allow expect_failures to target child module input variables in Terraform test run blocks #34951
• Terraform test execution doesn't handle prevent_destroy=true  #34960
• Terraform test: ability to expect overall failures of plan or apply command #35628
• Allow expect_failure to assert on an error message #37835
• Terraform test expect_failure doesn't work for permission errors #37799
• terraform test expected_failure not recognizing missing data resource as failure #35949
• [terraform test] Support expect_failures of resouces inside local module in run blocks #34700
• Have a force_delete true/false on repositories similar to AWS ECR. jfrog/terraform-provider-artifactory#1069

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[edgetools]

I may spin up an additional issue for this, but in general, looking at the issues I've encountered trying to use terraform test, as well as the numerous linked issues above, it seems that expect_failures does not meet the criteria needed to do proper failure assertions.

So far I'm encountering multiple different scenarios in which I want to use testing to ensure that required contracts are being met, and yet none of them are supported by the terraform test flow.

In addition to the lifecycle prevent_destroy issue above, I'm also encountering issues with unset or empty value validation.

If one goes to https://developer.hashicorp.com/terraform/language/validate there are multiple different validation features available, such as preconditions, postconditions, and checks. However, per the error message, expect_failures is limited to:

You cannot expect failures from each.value. You can only expect failures from checkable objects such as input variables, output values, check blocks, managed resources and data sources.

I have a resource that uses for_each based on a data structure object with dynamic contents.

resource "artifactory_local_debian_repository" "v1" {
  for_each = local.v1_repositories_to_create

  # name-type-maturity-locator
  key = each.key # required

  # resource-specific options
  # see: https://registry.terraform.io/providers/jfrog/artifactory/latest/docs/resources/local_debian_repository
  primary_keypair_ref       = try(each.value.primary_keypair_ref, null)
  secondary_keypair_ref     = try(each.value.secondary_keypair_ref, null)
  index_compression_formats = try(each.value.index_compression_formats, null)
  trivial_layout            = try(each.value.trivial_layout, null) # deprecated

  # generic local options
  # see: https://registry.terraform.io/providers/jfrog/artifactory/latest/docs/resources/local
  description              = each.value.description      # public description, required
  notes                    = try(each.value.notes, null) # private notes
  project_key              = try(each.value.project_key, null)
  project_environments     = try(each.value.project_environments, null)
  includes_pattern         = try(each.value.includes_pattern, null)
  excludes_pattern         = try(each.value.excludes_pattern, null)
  repo_layout_ref          = try(each.value.repo_layout_ref, null)
  blacked_out              = try(each.value.blacked_out, null)
  xray_index               = try(each.value.xray_index, null)
  priority_resolution      = try(each.value.priority_resolution, null)
  property_sets            = try(each.value.property_sets, null)
  archive_browsing_enabled = try(each.value.archive_browsing_enabled, null)
  download_direct          = try(each.value.download_direct, null)
  cdn_redirect             = try(each.value.cdn_redirect, null)

  lifecycle {
    # until the jfrog api or the terraform provider
    # supports preventing deletion of repositories that contain artifacts
    # prevent destroying ANY repository
    prevent_destroy = true
  
    precondition {
      condition = each.value.description != ""
      error_message = "description cannot be blank"
    }
  }
}

each.key will always be present, but each.value is a dynamic object that gets loaded from a yaml file and may contain any number of map keys. Users can set values and the try behavior allows most of them to be unset unless specified.

However, some values need to be required, for example each.value.description above.

I tried adding a precondition to see if I could test that with expect_failures, but I got the error message above.

The precondition only allows me to guard against an empty value, I can't actually guard against an unset value in a testable way with terraform test.

I would like to be able to do the following with the above:

• create repo with each.value.description unset, and validate that terraform throws error (unset value)
• create repo with each.value.description as empty string, and validate that terraform throws error
• create repo with each.value.description set to a non-empty string value, and validate that terraform proceeds

currently all I can do is the third validation, I can't validate the error cases

I understand that we aren't supposed to be testing "terraform internals", but this is not a terraform internals issue, this is enforcing a contract so that refactors and future modifications don't accidentally break the contracts. Good testing hygiene!