From 8d37a709877db9004f896db92ddc5e95837be378 Mon Sep 17 00:00:00 2001 From: Martin Atkins Date: Wed, 17 Feb 2021 17:12:20 -0800 Subject: [PATCH] website: Initial docs for the module integration testing experiment Since this is still at an early phase and likely to change significantly in future iterations, rather than attempting to guess on a suitable final location for documenting the testing feature I've instead taken the unusual approach of adding a new page that is explicitly about the experiment. My expectation is that once we conclude the experiment we'll replace this new page with a stub that just explains that there was once an experiment and then links to whatever final feature unfolded from the research. The URL for this page is hard-coded into the warning message in the "terraform test" command, so as we continue to evolve this feature in future releases we'll need to update the callout note on the page about which Terraform CLI version it's currently talking about, so users of older versions can clearly see when they'd need to upgrade in order to participate in a later incarnation of the experiment. --- website/docs/cli/commands/test.html.md | 16 + .../modules/testing-experiment.html.md | 323 ++++++++++++++++++ website/layouts/docs.erb | 4 + 3 files changed, 343 insertions(+) create mode 100644 website/docs/cli/commands/test.html.md create mode 100644 website/docs/language/modules/testing-experiment.html.md diff --git a/website/docs/cli/commands/test.html.md b/website/docs/cli/commands/test.html.md new file mode 100644 index 000000000..7edd1fb1b --- /dev/null +++ b/website/docs/cli/commands/test.html.md @@ -0,0 +1,16 @@ +--- +layout: "docs" +page_title: "Command: test" +sidebar_current: "docs-commands-test" +description: |- + Part of the ongoing design research for module integration testing. +--- + +# Command: test + +The `terraform test` command is currently serving as part of +[the module integration testing experiment](/docs/language/modules/testing-experiment.html). + +It's not ready for routine use, but if you'd be interested in trying the +prototype functionality then we'd love to hear your feedback. See the +experiment details page linked above for more information. diff --git a/website/docs/language/modules/testing-experiment.html.md b/website/docs/language/modules/testing-experiment.html.md new file mode 100644 index 000000000..0cbf31dd1 --- /dev/null +++ b/website/docs/language/modules/testing-experiment.html.md @@ -0,0 +1,323 @@ +--- +layout: "language" +page_title: "Module Testing Experiment - Configuration Language" +--- + +# Module Testing Experiment + +This page is about some experimental features available in recent versions of +Terraform CLI related to integration testing of shared modules. + +The Terraform team is aiming to use these features to gather feedback as part +of ongoing research into different strategies for testing Terraform modules. +These features are likely to change significantly in future releases based on +feedback. + +## Current Research Goals + +Our initial area of research is into the question of whether it's helpful and +productive to write module integration tests in the Terraform language itself, +or whether it's better to handle that as a separate concern orchestrated by +code written in other languages. + +Some existing efforts have piloted both approaches: + +* [Terratest](https://terratest.gruntwork.io/) and + [kitchen-terraform](https://github.com/newcontext-oss/kitchen-terraform) + both pioneered the idea of writing tests for Terraform modules with explicit + orchestration written in the Go programming language. + +* The Terraform provider + [`apparentlymart/testing`](https://registry.terraform.io/providers/apparentlymart/testing/latest) + introduced the idea of writing Terraform module tests in the Terraform + language itself, using a special provider that can evaluate assertions + and fail `terraform apply` if they don't pass. + +Both of these approaches have both advantages and disadvantages, and so it's +likely that both will coexist for different situations, but the community +efforts have already explored the external-language testing model quite deeply +while the Terraform-integrated testing model has not yet been widely trialled. +For that reason, the current iteration of the module testing experiment is +aimed at trying to make the Terraform-integrated approach more accessible so +that more module authors can hopefully try it and share their experiences. + +## Current Experimental Features + +-> This page describes the incarnation of the experimental features introduced +in **Terraform CLI v0.15.0**. If you are using an earlier version of Terraform +then you'll need to upgrade to v0.15.0 or later to use the experimental features +described here, though you only need to use v0.15.0 or later for running tests; +your module itself can remain compatible with earlier Terraform versions, if +needed. + +Our current area of interest is in what sorts of tests can and cannot be +written using features integrated into the Terraform language itself. As a +means to investigate that without invasive, cross-cutting changes to Terraform +Core we're using a special built-in Terraform provider as a placeholder for +potential new features. + +If this experiment is successful then we expect to run a second round of +research and design about exactly what syntax is most ergonomic for writing +tests, but for the moment we're interested less in the specific syntax and more +in the capabilities of this approach. + +The temporary extensions to Terraform for this experiment consist of the +following parts: + +* A temporary experimental provider `terraform.io/builtin/test`, which acts as + a placeholder for potential new language features related to test assertions. + +* A `terraform test` command for more conveniently running multiple tests in + a single action. + +* An experimental convention of placing test configurations in subdirectories + of a `tests` directory within your module, which `terraform test` will then + discover and run. + +We would like to invite adventurous module authors to try writing integration +tests for their modules using these mechanisms, and ideally also share the +tests you write (in a temporary VCS branch, if necessary) so we can see what +you were able to test, along with anything you felt unable to test in this way. + +If you're interested in giving this a try, see the following sections for +usage details. Because these features are temporary experimental extensions, +there's some boilerplate required to activate and make use of it which would +likely not be required in a final design. + +### Writing Tests for a Module + +For the purposes of the current experiment, module tests are arranged into +_test suites_, each of which is a root Terraform module which includes a +`module` block calling the module under test, and ideally also a number of +test assertions to verify that the module outputs match expectations. + +In the same directory where you keep your module's `.tf` and/or `.tf.json` +source files, create a subdirectory called `tests`. Under that directory, +make another directory which will serve as your first test suite, with a +directory name that concisely describes what the suite is aiming to test. + +Here's an example directory structure of a typical module directory layout +with the addition of a test suite called `defaults`: + +``` +main.tf +outputs.tf +providers.tf +variables.tf +versions.tf +tests/ + defaults/ + test_defaults.tf +``` + +The `tests/defaults/test_defaults.tf` file will contain a call to the +main module with a suitable set of arguments and hopefully also one or more +resources that will, for the sake of the experiment, serve as the temporary +syntax for defining test assertions. For example: + +```hcl +terraform { + required_providers { + # Because we're currently using a built-in provider as + # a substitute for dedicated Terraform language syntax + # for now, test suite modules must always declare a + # dependency on this provider. This provider is only + # available when running tests, so you shouldn't use it + # in non-test modules. + test = { + source = "terraform.io/builtin/test" + } + + # This example also uses the "http" data source to + # verify the behavior of the hypothetical running + # service, so we should declare that too. + http = { + source = "hashicorp/http" + } + } +} + +module "main" { + # source is always ../.. for test suite configurations, + # because they are placed two subdirectories deep under + # the main module directory. + source = "../.." + + # This test suite is aiming to test the "defaults" for + # this module, so it doesn't set any input variables + # and just lets their default values be selected instead. +} + +# As with all Terraform modules, we can use local values +# to do any necessary post-processing of the results from +# the module in preparation for writing test assertions. +locals { + # This expression also serves as an implicit assertion + # that the base URL uses URL syntax; the test suite + # will fail if this function fails. + api_url_parts = regex( + "^(?:(?P[^:/?#]+):)?(?://(?P[^/?#]*))?", + module.main.api_url, + ) +} + +# The special test_assertions resource type, which belongs +# to the test provider we required above, is a temporary +# syntax for writing out explicit test assertions. +resource "test_assertions" "api_url" { + # "component" serves as a unique identifier for this + # particular set of assertions in the test results. + component = "api_url" + + # equal and check blocks serve as the test assertions. + # the labels on these blocks are unique identifiers for + # the assertions, to allow more easily tracking changes + # in success between runs. + + equal "scheme" { + description = "default scheme is https" + got = local.api_url_parts.scheme + want = "https" + } + + check "port_number" { + description = "default port number is 8080" + condition = can(regex(":8080$", local.api_url_parts.authority)) + } +} + +# We can also use data resources to respond to the +# behavior of the real remote system, rather than +# just to values within the Terraform configuration. +data "http" "api_response" { + depends_on = [ + # make sure the syntax assertions run first, so + # we'll be sure to see if it was URL syntax errors + # that let to this data resource also failing. + test_assertions.api_url, + ] + + url = module.main.api_url +} + +resource "test_assertions" "api_response" { + component = "api_response" + + check "valid_json" { + description = "base URL responds with valid JSON" + condition = can(jsondecode(data.http.api_response.body)) + } +} +``` + +If you like, you can create additional directories alongside +the `default` directory to define additional test suites that +pass different variable values into the main module, and +then include assertions that verify that the result has changed +in the expected way. + +### Running Your Tests + +The `terraform test` command aims to make it easier to exercise all of your +defined test suites at once, and see only the output related to any test +failures or errors. + +The current experimental incarnation of this command expects to be run from +your main module directory. In our example directory structure above, +that was the directory containing `main.tf` etc, and _not_ the specific test +suite directory containing `test_defaults.tf`. + +Because these test suites are integration tests rather than unit tests, you'll +need to set up any credentials files or environment variables needed by the +providers your module uses before running `terraform test`. The test command +will, for each suite: + +* Install the providers and any external modules the test configuration depends + on. +* Create an execution plan to create the objects declared in the module. +* Apply that execution plan to create the objects in the real remote system. +* Collect all of the test results from the apply step, which would also have + "created" the `test_assertions` resources. +* Destroy all of the objects recorded in the temporary test state, as if running + `terraform destroy` against the test configuration. + +```shellsession +$ terraform test +─── Failed: defaults.api_url.scheme (default scheme is https) ─────────────── +wrong value + got: "http" + want: "https" +───────────────────────────────────────────────────────────────────────────── +``` + +In this case, it seems like the module returned an `http` rather than an +`https` URL in the default case, and so the `defaults.api_url.scheme` +assertion failed, and the `terraform test` command detected and reported it. + +The `test_assertions` resource captures any assertion failures but does not +return an error, because that can then potentially allow downstream +assertions to also run and thus capture as much context as possible. +However, if Terraform encounters any _errors_ while processing the test +configuration it will halt processing, which may cause some of the test +assertions to be skipped. + +## Known Limitations + +The design above is very much a prototype aimed at gathering more experience +with the possibilities of testing inside the Terraform language. We know it's +currently somewhat non-ergonomic, and hope to improve on that in later phases +of research and design, but the main focus of this iteration is on available +functionality and so with that in mind there are some specific possibilities +that we know the current prototype doesn't support well: + +* Testing of subsequent updates to an existing deployment of a module. + Currently tests written in this way can only exercise the create and destroy + behaviors. + +* Assertions about expected errors. For a module that includes variable + validation rules and data resources that function as assertion checks, + the current prototype doesn't have any way to express that a particular + set of inputs is _expected_ to produce an error, and thus report a test + failure if it doesn't. We'll hopefully be able to improve on this in a future + iteration with the test assertions better integrated into the language. + +* Capturing context about failures. Due to this prototype using a provider as + an approximation for new assertion syntax, the `terraform test` command is + limited in how much context it's able to gather about failures. A design + more integrated into the language could potentially capture the source + expressions and input values to give better feedback about what went wrong, + similar to what Terraform typically returns from expression evaluation errors + in the main language. + +* Unit testing without creating real objects. Although we do hope to spend more + time researching possibilities for unit testing against fake test doubles in + the future, we've decided to focus on integration testing to start because + it feels like the better-defined problem. + +## Sending Feedback + +The sort of feedback we'd most like to see at this stage of the experiment is +to see the source code of any tests you've written against real modules using +the features described above, along with notes about anything that you +attempted to test but were blocked from doing so by limitations of the above +features. The most ideal way to share that would be to share a link to a +version control branch where you've added such tests, if your module is open +source. + +If you've previously written or attempted to write tests in an external +language, using a system like Terratest or kitchen-terraform, we'd also be +interested to hear about comparative differences between the two: what worked +well in each and what didn't work so well. + +Our ultimate goal is to work towards an integration testing methodology which +strikes the best compromise between the capabilities of these different +approaches, ideally avoiding a hard requirement on any particular external +language and fitting well into the Terraform workflow. + +Since this is still early work and likely to lead to unstructured discussion, +we'd like to gather feedback primarily via new topics in +[the community forum](https://discuss.hashicorp.com/c/terraform-core/27). That +way we can have some more freedom to explore different ideas and approaches +without the structural requirements we typically impose on GitHub issues. + +Any feedback you'd like to share would be very welcome! diff --git a/website/layouts/docs.erb b/website/layouts/docs.erb index 7ebf352d6..ce0570f61 100644 --- a/website/layouts/docs.erb +++ b/website/layouts/docs.erb @@ -473,6 +473,10 @@ taint +
  • + test +
    • +
  • untaint