ResiLien
/
terraform
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

website: Initial docs for the module integration testing experiment 

Since this is still at an early phase and likely to change significantly
in future iterations, rather than attempting to guess on a suitable final
location for documenting the testing feature I've instead taken the
unusual approach of adding a new page that is explicitly about the
experiment. My expectation is that once we conclude the experiment we'll
replace this new page with a stub that just explains that there was once
an experiment and then links to whatever final feature unfolded from the
research.

The URL for this page is hard-coded into the warning message in the
"terraform test" command, so as we continue to evolve this feature in
future releases we'll need to update the callout note on the page about
which Terraform CLI version it's currently talking about, so users of
older versions can clearly see when they'd need to upgrade in order to
participate in a later incarnation of the experiment.
This commit is contained in:
Martin Atkins 2021-02-17 17:12:20 -08:00
parent 7f78342953
commit 8d37a70987
3 changed files with 343 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
website/docs/cli/commands/test.html.md Normal file
View File

@ -0,0 +1,16 @@
---
layout: "docs"
page_title: "Command: test"
sidebar_current: "docs-commands-test"
description: |-
Part of the ongoing design research for module integration testing.
---
# Command: test
The `terraform test` command is currently serving as part of
[the module integration testing experiment](/docs/language/modules/testing-experiment.html).
It's not ready for routine use, but if you'd be interested in trying the
prototype functionality then we'd love to hear your feedback. See the
experiment details page linked above for more information.

323
website/docs/language/modules/testing-experiment.html.md Normal file
View File

@ -0,0 +1,323 @@
---
layout: "language"
page_title: "Module Testing Experiment - Configuration Language"
---
# Module Testing Experiment
This page is about some experimental features available in recent versions of
Terraform CLI related to integration testing of shared modules.
The Terraform team is aiming to use these features to gather feedback as part
of ongoing research into different strategies for testing Terraform modules.
These features are likely to change significantly in future releases based on
feedback.
## Current Research Goals
Our initial area of research is into the question of whether it's helpful and
productive to write module integration tests in the Terraform language itself,
or whether it's better to handle that as a separate concern orchestrated by
code written in other languages.
Some existing efforts have piloted both approaches:
* [Terratest](https://terratest.gruntwork.io/) and
[kitchen-terraform](https://github.com/newcontext-oss/kitchen-terraform)
both pioneered the idea of writing tests for Terraform modules with explicit
orchestration written in the Go programming language.
* The Terraform provider
[`apparentlymart/testing`](https://registry.terraform.io/providers/apparentlymart/testing/latest)
introduced the idea of writing Terraform module tests in the Terraform
language itself, using a special provider that can evaluate assertions
and fail `terraform apply` if they don't pass.
Both of these approaches have both advantages and disadvantages, and so it's
likely that both will coexist for different situations, but the community
efforts have already explored the external-language testing model quite deeply
while the Terraform-integrated testing model has not yet been widely trialled.
For that reason, the current iteration of the module testing experiment is
aimed at trying to make the Terraform-integrated approach more accessible so
that more module authors can hopefully try it and share their experiences.
## Current Experimental Features
-> This page describes the incarnation of the experimental features introduced
in **Terraform CLI v0.15.0**. If you are using an earlier version of Terraform
then you'll need to upgrade to v0.15.0 or later to use the experimental features
described here, though you only need to use v0.15.0 or later for running tests;
your module itself can remain compatible with earlier Terraform versions, if
needed.
Our current area of interest is in what sorts of tests can and cannot be
written using features integrated into the Terraform language itself. As a
means to investigate that without invasive, cross-cutting changes to Terraform
Core we're using a special built-in Terraform provider as a placeholder for
potential new features.
If this experiment is successful then we expect to run a second round of
research and design about exactly what syntax is most ergonomic for writing
tests, but for the moment we're interested less in the specific syntax and more
in the capabilities of this approach.
The temporary extensions to Terraform for this experiment consist of the
following parts:
* A temporary experimental provider `terraform.io/builtin/test`, which acts as
a placeholder for potential new language features related to test assertions.
* A `terraform test` command for more conveniently running multiple tests in
a single action.
* An experimental convention of placing test configurations in subdirectories
of a `tests` directory within your module, which `terraform test` will then
discover and run.
We would like to invite adventurous module authors to try writing integration
tests for their modules using these mechanisms, and ideally also share the
tests you write (in a temporary VCS branch, if necessary) so we can see what
you were able to test, along with anything you felt unable to test in this way.
If you're interested in giving this a try, see the following sections for
usage details. Because these features are temporary experimental extensions,
there's some boilerplate required to activate and make use of it which would
likely not be required in a final design.
### Writing Tests for a Module
For the purposes of the current experiment, module tests are arranged into
_test suites_, each of which is a root Terraform module which includes a
`module` block calling the module under test, and ideally also a number of
test assertions to verify that the module outputs match expectations.
In the same directory where you keep your module's `.tf` and/or `.tf.json`
source files, create a subdirectory called `tests`. Under that directory,
make another directory which will serve as your first test suite, with a
directory name that concisely describes what the suite is aiming to test.
Here's an example directory structure of a typical module directory layout
with the addition of a test suite called `defaults`:
```
main.tf
outputs.tf
providers.tf
variables.tf
versions.tf
tests/
defaults/
test_defaults.tf
```
The `tests/defaults/test_defaults.tf` file will contain a call to the
main module with a suitable set of arguments and hopefully also one or more
resources that will, for the sake of the experiment, serve as the temporary
syntax for defining test assertions. For example:
```hcl
terraform {
required_providers {
# Because we're currently using a built-in provider as
# a substitute for dedicated Terraform language syntax
# for now, test suite modules must always declare a
# dependency on this provider. This provider is only
# available when running tests, so you shouldn't use it
# in non-test modules.
test = {
source = "terraform.io/builtin/test"
}
# This example also uses the "http" data source to
# verify the behavior of the hypothetical running
# service, so we should declare that too.
http = {
source = "hashicorp/http"
}
}
}
module "main" {
# source is always ../.. for test suite configurations,
# because they are placed two subdirectories deep under
# the main module directory.
source = "../.."
# This test suite is aiming to test the "defaults" for
# this module, so it doesn't set any input variables
# and just lets their default values be selected instead.
}
# As with all Terraform modules, we can use local values
# to do any necessary post-processing of the results from
# the module in preparation for writing test assertions.
locals {
# This expression also serves as an implicit assertion
# that the base URL uses URL syntax; the test suite
# will fail if this function fails.
api_url_parts = regex(
"^(?:(?P<scheme>[^:/?#]+):)?(?://(?P<authority>[^/?#]*))?",
module.main.api_url,
)
}
# The special test_assertions resource type, which belongs
# to the test provider we required above, is a temporary
# syntax for writing out explicit test assertions.
resource "test_assertions" "api_url" {
# "component" serves as a unique identifier for this
# particular set of assertions in the test results.
component = "api_url"
# equal and check blocks serve as the test assertions.
# the labels on these blocks are unique identifiers for
# the assertions, to allow more easily tracking changes
# in success between runs.
equal "scheme" {
description = "default scheme is https"
got = local.api_url_parts.scheme
want = "https"
}
check "port_number" {
description = "default port number is 8080"
condition = can(regex(":8080$", local.api_url_parts.authority))
}
}
# We can also use data resources to respond to the
# behavior of the real remote system, rather than
# just to values within the Terraform configuration.
data "http" "api_response" {
depends_on = [
# make sure the syntax assertions run first, so
# we'll be sure to see if it was URL syntax errors
# that let to this data resource also failing.
test_assertions.api_url,
]
url = module.main.api_url
}
resource "test_assertions" "api_response" {
component = "api_response"
check "valid_json" {
description = "base URL responds with valid JSON"
condition = can(jsondecode(data.http.api_response.body))
}
}
```
If you like, you can create additional directories alongside
the `default` directory to define additional test suites that
pass different variable values into the main module, and
then include assertions that verify that the result has changed
in the expected way.
### Running Your Tests
The `terraform test` command aims to make it easier to exercise all of your
defined test suites at once, and see only the output related to any test
failures or errors.
The current experimental incarnation of this command expects to be run from
your main module directory. In our example directory structure above,
that was the directory containing `main.tf` etc, and _not_ the specific test
suite directory containing `test_defaults.tf`.
Because these test suites are integration tests rather than unit tests, you'll
need to set up any credentials files or environment variables needed by the
providers your module uses before running `terraform test`. The test command
will, for each suite:
* Install the providers and any external modules the test configuration depends
on.
* Create an execution plan to create the objects declared in the module.
* Apply that execution plan to create the objects in the real remote system.
* Collect all of the test results from the apply step, which would also have
"created" the `test_assertions` resources.
* Destroy all of the objects recorded in the temporary test state, as if running
`terraform destroy` against the test configuration.
```shellsession
$ terraform test
─── Failed: defaults.api_url.scheme (default scheme is https) ───────────────
wrong value
got: "http"
want: "https"
─────────────────────────────────────────────────────────────────────────────
```
In this case, it seems like the module returned an `http` rather than an
`https` URL in the default case, and so the `defaults.api_url.scheme`
assertion failed, and the `terraform test` command detected and reported it.
The `test_assertions` resource captures any assertion failures but does not
return an error, because that can then potentially allow downstream
assertions to also run and thus capture as much context as possible.
However, if Terraform encounters any _errors_ while processing the test
configuration it will halt processing, which may cause some of the test
assertions to be skipped.
## Known Limitations
The design above is very much a prototype aimed at gathering more experience
with the possibilities of testing inside the Terraform language. We know it's
currently somewhat non-ergonomic, and hope to improve on that in later phases
of research and design, but the main focus of this iteration is on available
functionality and so with that in mind there are some specific possibilities
that we know the current prototype doesn't support well:
* Testing of subsequent updates to an existing deployment of a module.
Currently tests written in this way can only exercise the create and destroy
behaviors.
* Assertions about expected errors. For a module that includes variable
validation rules and data resources that function as assertion checks,
the current prototype doesn't have any way to express that a particular
set of inputs is _expected_ to produce an error, and thus report a test
failure if it doesn't. We'll hopefully be able to improve on this in a future
iteration with the test assertions better integrated into the language.
* Capturing context about failures. Due to this prototype using a provider as
an approximation for new assertion syntax, the `terraform test` command is
limited in how much context it's able to gather about failures. A design
more integrated into the language could potentially capture the source
expressions and input values to give better feedback about what went wrong,
similar to what Terraform typically returns from expression evaluation errors
in the main language.
* Unit testing without creating real objects. Although we do hope to spend more
time researching possibilities for unit testing against fake test doubles in
the future, we've decided to focus on integration testing to start because
it feels like the better-defined problem.
## Sending Feedback
The sort of feedback we'd most like to see at this stage of the experiment is
to see the source code of any tests you've written against real modules using
the features described above, along with notes about anything that you
attempted to test but were blocked from doing so by limitations of the above
features. The most ideal way to share that would be to share a link to a
version control branch where you've added such tests, if your module is open
source.
If you've previously written or attempted to write tests in an external
language, using a system like Terratest or kitchen-terraform, we'd also be
interested to hear about comparative differences between the two: what worked
well in each and what didn't work so well.
Our ultimate goal is to work towards an integration testing methodology which
strikes the best compromise between the capabilities of these different
approaches, ideally avoiding a hard requirement on any particular external
language and fitting well into the Terraform workflow.
Since this is still early work and likely to lead to unstructured discussion,
we'd like to gather feedback primarily via new topics in
[the community forum](https://discuss.hashicorp.com/c/terraform-core/27). That
way we can have some more freedom to explore different ideas and approaches
without the structural requirements we typically impose on GitHub issues.
Any feedback you'd like to share would be very welcome!

4
website/layouts/docs.erb
View File

@ -473,6 +473,10 @@
<a href="/docs/cli/commands/taint.html"><code>taint</code></a> <a href="/docs/cli/commands/taint.html"><code>taint</code></a>
</li> </li>
<li>
<a href="/docs/cli/commands/test.html"><code>test</code></a>
</li>
<li> <li>
<a href="/docs/cli/commands/untaint.html"><code>untaint</code></a> <a href="/docs/cli/commands/untaint.html"><code>untaint</code></a>
</li> </li>