Several top-level block types in the Terraform language have a body where
two different schemas are overlayed on top of one another: Terraform first
looks for "meta-arguments" that are built into the language, and then
evaluates all of the remaining arguments against some externally-defined
schema whose content is not fully controlled by Terraform.
So far we've been cautiously adding new meta-arguments in these namespaces
after research shows us that there are relatively few existing providers
or modules that would have functionality masked by those additions, but
that isn't really a viable path forward as we prepare to make stronger
compatibility promises.
In an earlier commit we've introduced the foundational parts of a new
language versioning mechanism called "editions" which should allow us to
make per-module-opt-in breaking changes in the future, but these shared
namespaces remain a liability because it would be annoying if adopting a
new edition made it impossible to use a feature of a third-party provider
or module that was already using a name that has now become reserved in
the new edition.
This commit introduces a new syntax intended to be a rarely-used escape
hatch for that situation. When we're designing new editions we will do our
best to choose names that don't conflict with commonly-used providers and
modules, but there are many providers and modules that we cannot see and
so there is a risk that any name we might choose could collide with at
least one existing provider or module. The automatic migration tool to
upgrade an existing module to a new edition should therefore detect that
situation and make use of this escaping block syntax in order to retain
the existing functionality until all the called providers or modules are
updated to no longer use conflicting names.
Although we can't put in technical constraints on using this feature for
other purposes (because we don't know yet what future editions will add),
this mechanism is intentionally not documented for now because it serves
no immediate purpose. In effect, this change is just squatting on the
syntax of a special block type named "_" so that later editions can make
use of it without it _also_ conflicting, creating a confusing nested
escaping situation. However, the first time a new edition actually makes
use of this syntax we should then document alongside the meta-arguments
so folks can understand the meaning of escaping blocks produced by
edition upgrade tools.
Passing a provider into a module requires that it be named within the
module. This would previously pass validation, however core would fail
to resolve the provider resulting in an unclear "provider not found"
error.
When rendering a stored plan file as JSON, we include a data structure
representing the sensitivity of the changed resource values. Prior to
this commit, this was a direct representation of the sensitivity marks
applied to values via mechanisms such as sensitive variables, sensitive
outputs, and the `sensitive` function.
This commit extends this to include sensitivity based on the provider
schema. This is in line with the UI rendering of the plan, which
considers these two different types of sensitivity to be equivalent.
Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
All the information is available to resolve provider types when building
the configuration, but some provider references still had no FQN. This
caused validation to assume a default type, and incorrectly reject valid
module calls with non-default namespaced providers.
Resolve as much provider type information as possible when loading the
config. Only use this internally for now, but this should be useful
outside of the package to avoid re-resolving the providers later on. We
can come back and find where this might be useful elsewhere, but for now
keep the change as small as possible to avoid any changes in behavior.
* checkpoint save: update InternalValidate tests to compare exact error
* configschema: extract and extend attribute validation
This commit adds an attribute-specific InternalValidate which was extracted directly from the block.InternalValidate logic and extended to verify any NestedTypes inside an Attribute. Only one error message changed, since it is now valid to have a cty.NilType for Attribute.Type as long as NestedType is set.
* terraform: validate provider schema's during NewContext
We haven't been able to guarantee that providers are validating their own schemas using (some version of) InternalValidate since providers were split out of the main codebase. This PR adds a call to InternalValidate when provider schemas are initially loaded by NewContext, which required a few other changes:
InternalValidate's handling of errors vs multierrors was a little weird - before this PR, it was occasionally returning a non-nil error which only stated "0 errors occurred" - so I addressed that in InternalValidate. I then tested this with a configuration that was using all of our most popular providers, and found that at least on provider had some invalid attribute names, so I commented that particular validation out. Adding that in would be a breaking change which we would have to coordinate with enablement and providers and (especially in this case) make sure it's well communicated to external provider developers.
I ran a few very unscientific tests comparing the timing with and without this validation, and it appeared to only cause a sub-second increase.
* refactor validate error message to closer match the sdk's message
* better error message
* tweak error message: move the instruction to run init to the end of the message, after the specific error.
We expect that in order to continue to evolve the language without
breaking existing modules we will at some point need to have a way to mark
when a particular module is expecting a newer interpretation of the
language.
Although it's too early to do any deep preparation for that, this commit
aims to proactively reserve an argument named "language" inside
"terraform" blocks, which currently only accepts the keyword TF2021 that
is intended to represent "the edition of the Terraform language as defined
in 2021".
That argument also defaults to TF2021 if not set, so in practice there's
no real reason to set this today, but this minimal validation today is
intended to give better feedback to users of older Terraform versions in
the event that we introduce a new language edition later and they try to
use an module incompatible with their Terraform version.
Our previous message conflated the requirement for a full sentence with
the suggestion to write in a style similar to Terraform's built-in error
messages, which created a sense that the system would actively reject an
error message written in another language.
There's no intent here to block writing error messages in other languages,
but there is a practical consideration that Terraform's UI output is
currently not localized and so consistency with Terraform's other output,
if that's important to a module author, will typically mean writing the
error message in English.
* command/format: check for sensitive NestedTypes
Eventually, the diff formatter will need to be updated to properly
handle NestedTypes, but for now we can let the existing function deal
with them as regular cty.Object-type attributes.
To avoid printing sensitive nested attributes, we will treat any
attribute with at least one sensitive nested attribute as an entirely
sensitive attribute.
* bugfix for Object ImpliedType()
ImpliedType() was returning too early when the given object had optional
attributes, therefore skipping the incredibly important step of
accounting for the nesting mode when returning said type.
A handful of bugs popped up while extending the testing in
plans/objchange. The main themes were failing to recurse through deeply
nested NestedType attributes and improperly building up the ImpliedType.
This commit fixes those issues and extends the test coverage to match.
Add validation which was removed from the configload package, along with
additional validation checks. The output is slightly different, as
instead of validating whether the modules are allowed to have provider
configurations, we validate the various combinations of provider
structures themselves.
The configload package should only be responsible for locating and
loading the configuration, and not be further inspecting the config
source itself. Moving the validating into the configs package.
Add support for parsing configuration_aliases in required_providers
entries. The decoder needed to be re-written here in order to support
the bare reference style usage of provider names so that they match the
usage in other location within configuration. The only change to
existing handling of the required_providers block is more precise error
locations in a couple cases.
The JSON plan output format includes a serialized, simplified version of
the configuration. One component of this config is a map of provider
configurations, which includes version constraints.
Until now, only version constraints specified in the provider config
blocks were exposed in the JSON plan output. This is a deprecated method
of specifying provider versions, and the recommended use of a
required_providers block resulted in the version constraints being
omitted.
This commit fixes this with two changes:
- When processing the provider configurations from a module, output the
fully-merged version constraints for the entire module, instead of any
constraints set in the provider configuration block itself;
- After all provider configurations are processed, iterate over the
required_providers entries to ensure that any configuration-less
providers are output to the JSON plan too.
No changes are necessary to the structure of the JSON plan output, so
this is effectively a semantic level bug fix.
This commit adds a new field, NestedType, to the Attribute schema, and
extends the current Attribute decoderSpec to account for the new type.
The codepaths are mostly unused and included in a separate commit to
verify that the included changes do not impact any other tests yet.
There are a few places where we want to perform some transformation on a
cty.Value, but require information from the schema. Rather than create
bespoke functions to walk the cty.Value and schema in concert, we can
provide Attribute information from a cty.Path allowing the use of
Value.Transform in these cases.
Terraform considers backend configurations only in the root module, so any
declarations in child modules are entirely ignored.
To avoid users mistakenly thinking that a root module backend
configuration has taken effect, we'll now emit a warning about it. This is
a warning rather than an error because it's reasonable to call a module
that would normally be a root module instead as a child module when
writing a wrapper module to handle integration testing.
We have an existing warning message to encourage moving away from the old
0.11-and-earlier style of redundantly wrapping standalone expressions in
templates, but due to the special rules for object keys the warning
message was giving misleading advice in that context: a user following the
advice as given would then encounter an error about the object key being
ambiguous.
To account for that, this introduces a special alternative version of the
warning just for that particular position, directing the user to replace
the template interpolation markers with parenthesis instead. That will
then get the same result as the former interpolation sequence, rather than
producing the ambiguity error.
Remove chef, habitat, puppet, and salt-masterless provsioners,
which follows their deprecation. Update the documentatin for these
provisioners to clarify that they have been removed from later versions
of Terraform. Adds the fmt Make target back and updates fmtcheck script
for correctness.
When a resource has no `provider` argument specified, its provider is
derived from the implied provider type based on the resource type. For
example, a `boop_instance` resource has an implied provider local name
of `boop`. Correspondingly, its provider configuration is specified with
a `provider "boop"` block.
However, users can use the `required_providers` configuration to give a
different local name to a given provider than its defined type. For
example, a provider may be published at `foobar/beep`, but provide
resources such as `boop_instance`. The most convenient way to use this
provider is with a `required_providers` map:
terraform {
required_providers {
boop = {
source = "foobar/beep"
}
}
}
Once that local name is defined, it is used for provider configuration
(a `provider "boop"` block, not `provider "beep"`). It should also be
used when looking up a resource's provider configuration or provider.
This commit fixes a bug with this edge case, where previously we were
looking up the local provider configuration block using the resource's
assigned provider type. Instead, if no provider argument is specified,
we should be using the implied provider type, as that is what binds the
resource to the local provider configuration.
DecoderSpec may be called many times, and deeply recursive calls are
expensive. Since we cannot synchronize the Blocks themselves due to them
being copied in parts of the code, we use a separate cache to store the
generated Specs.
This builds on an experimental feature in the underlying cty library which
allows marking specific attribtues of an object type constraint as
optional, which in turn modifies how the cty conversion package handles
missing attributes in a source value: it will silently substitute a null
value of the appropriate type rather than returning an error.
In order to implement the experiment this commit temporarily forks the
HCL typeexpr extension package into a local internal/typeexpr package,
where I've extended the type constraint syntax to allow annotating object
type attributes as being optional using the HCL function call syntax.
If the experiment is successful -- both at the Terraform layer and in
the underlying cty library -- we'll likely send these modifications to
upstream HCL so that other HCL-based languages can potentially benefit
from this new capability.
Because it's experimental, the optional attribute modifier is allowed only
with an explicit opt-in to the module_variable_optional_attrs experiment.
This new-ish package ended up under "helper" during the 0.12 cycle for
want of some other place to put it, but in retrospect that was an odd
choice because the "helper/" tree is otherwise a bunch of legacy code from
when the SDK lived in this repository.
Here we move it over into the "internal" directory just to distance it
from the guidance of not using "helper/" packages in new projects;
didyoumean is a package we actively use as part of error message hints.
Previous deprecations only included direct assignment of template-only
expressions to arguments. That is, this was not deprecated:
locals {
foo = ["${var.foo}"]
}
This commit uses hclsyntax.VisitAll to detect and show deprecations for
all template-only expressions, no matter how deep they are in a given
expression.
The providers schema command is using the Config.ProviderTypes method,
which had not been kept up to date with the changes to provider
requirements detection made in Config.ProviderRequirements. This
resulted in any currently-unused providers being omitted from the
output.
This commit changes the ProviderTypes method to use the same underlying
logic as ProviderRequirements, which ensures that `required_providers`
blocks are taken into account.
Includes an integration test case to verify that this fixes the provider
schemas command bug.
Martin Atkins