This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.
For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.
Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.
Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
Previously, resources without explicit provider configuration (i.e. a
`provider =` attribute) would be assigned a default provider based upon
the resource type. For example, a resource `foo_bar` would be assigned
provider `hashicorp/foo`.
This behaviour did not work well with community or partner providers,
with sources configured in `terraform.required_providers` blocks. With
the following configuration:
terraform {
required_providers {
foo = {
source = "acme/foo"
}
}
}
resource foo_bar "a" { }
the resource would be configured with the `hashicorp/foo` provider.
This commit fixes this implied provider behaviour. First we look for a
provider with local name matching the resource type in the module's
required providers map. If one is found, this provider is assigned to
the resource. Otherwise, we still fall back to a default provider.
We now permit at most one `required_providers` block per module (except
for overrides). This prevents users (and Terraform) from struggling to
understand how to merge multiple `required_providers` configurations,
with `version` and `source` attributes split across multiple blocks.
Because only one `required_providers` block is permitted, there is no
need to concatenate version constraints and resolve them. This allows us
to simplify the structs used to represent provider requirements,
aligning more closely with other structs in this package.
This commit also fixes a semantic use-before-initialize bug, where
resources defined before a `required_providers` block would be unable to
use its source attribute. We achieve this by processing the module's
`required_providers` configuration (and overrides) before resources.
Overrides for `required_providers` work as before, replacing the entire
block per provider.
The providers command has been refactored to use the modern provider types and
ProviderRequirements() functions. This resulted in a breaking change to
the output: it no longer outputs the providers by module and no longer
prints `(inherited)` or `(from state)` to show why a provider is
included. We decided that at this time it was best to stick with the
existing functions and make this change, but if we get feedback from the
community we will revisit.
Additional tests to exercise providers in modules and providers from
state have been included.
This PR adds iteration through any provider configuration blocks in the
config in addProviderRequirements().
A stale comment (of mine!) would leave one expecting the
module.ProviderRequirements to include any requirements from provider
configs. The comment was inaccurate and has been updated.
With provider dependencies now appearing inside a nested block, it seems
likely that configuration examples showing dependencies out of context
will sometimes mislead users into thinking that required_providers is
toplevel.
To give better feedback in that situation, we'll produce a specialized
error in that case hinting the correct structure to the user.
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".
This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.
To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.
The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.
This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
* terraform: large refactor to use Provider from configs.Resource
configs.Resource.ImpliedProvider() now returns a string; it is the
callers' responsibility to turn that into an addrs.Provider if needed.
GraphNodeProviderConsumer ProvidedBy() no longer returns nil (reverting
to earlier, pre-provider-fqn behavior): it will return either the
provider set in config, provider set in state, or the default provider.
This function can already produce suitable diagnostic messages which we'd
like to preserve, but it cannot produce source location information, and
so we'll amend the diagnostics to include that on the way out while
retaining all of the other values in the diagnostics.
* configs: parse provider source string during module merge
This was the smallest unit of work needed to start writing provider
source tests!
* Update configs/parser_test.go
Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Change ModuleInstance to Module in AbsProviderConfig, because providers
need to be handled before module expansion, and should not be used
defined inside an expanded module at all.
Renaming of the addrs type can happen later, when there's less work
in-flight around provider configuration.
* add Config to AttachSchemaTransformer for providerFqn lookup
* terraform: refactor ProvidedBy() to return nil when provider is not set
in config or state
Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries.
Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump.
Co-authored-by: Paul Tyng <paul@paultyng.net>
The provider FQN is becoming our primary identifier for a provider, so
it's important that we are clear about the equality rules for these
addresses and what characters are valid within them.
We previously had a basic regex permitting ASCII letters and digits for
validation and no normalization at all. We need to do at least case
folding and UTF-8 normalization because these names will appear in file
and directory names in case-insensitive filesystems and in repository
names such as on GitHub.
Since we're already using DNS-style normalization and validation rules
for the hostname part, rather than defining an entirely new set of rules
here we'll just treat the provider namespace and type as if they were
single labels in a DNS name. Aside from some internal consistency, that
also works out nicely because systems like GitHub use organization and
repository names as part of hostnames (e.g. with GitHub Pages) and so
tend to apply comparable constraints themselves.
This introduces the possibility of names containing letters from alphabets
other than the latin alphabet, and for latin letters with diacritics.
That's consistent with our introduction of similar support for identifiers
in the language in Terraform 0.12, and is intended to be more friendly to
Terraform users throughout the world that might prefer to name their
products using a different alphabet. This is also a further justification
for using the DNS normalization rules: modern companies tend to choose
product names that make good domain names, and now such names will be
usable as Terraform provider names too.
Added configs.Module.ProviderForLocalProviderConfig which allows
terraform.ProviderTransformer to get the provider FQN from the module,
instead of assuming NewLegacyProvider.
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).
The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
* configs: added map of ProviderLocalNames to configs.Module
We will need to lookup any user-supplied local names for a given FQN.
This PR adds a map of ProviderLocalNames to the Module, along with
adding tests for this and for decodeRequiredProvidersBlock.
This also introduces the appearance of support for a required_provider
"source" attribute, but ignores any user-supplied source and instead
continues to assume that addrs.NewLegacyProvider is the way to go.
This is a stepping-stone PR for the provider source project. In this PR
"legcay-stype" FQNs are created from the provider name string. Future
work involves encoding the FQN directly in the AbsProviderConfig and
removing the calls to addrs.NewLegacyProvider().
* Introduce "Local" terminology for non-absolute provider config addresses
In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.
This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.
In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
package, including removing addrs.NewLegacyProvider and
addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
work with addrs.Provider to identify providers, rather than legacy
strings.
In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.
* addrs: ProviderConfig interface
In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.
We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.
In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.
This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.
* rename LocalType to LocalName
Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
Renamed file.ProviderRequirements to file.RequiredProviders to match the
name of the block in the configuration. file.RequiredProviders contains
the contents of the file(s); module.ProviderRequirements contains the
parsed and merged provider requirements.
Extended decodeRequiredProvidersBlock to parse the new provider source
syntax (version only, it will ignore any other attributes).
Added some tests; swapped deep.Equal with cmp.Equal in the
terraform/module_dependencies_test.go because deep was not catching
incorrect constraints.
The existing "type" argument allows specifying a type constraint that
allows for some basic validation, but often there are more constraints on
a variable value than just its type.
This new feature (requiring an experiment opt-in for now, while we refine
it) allows specifying arbitrary validation rules for any variable which
can then cause custom error messages to be returned when a caller provides
an inappropriate value.
variable "example" {
validation {
condition = var.example != "nope"
error_message = "Example value must not be \"nope\"."
}
}
The core parts of this are designed to do as little new work as possible
when no validations are specified, and thus the main new checking codepath
here can therefore only run when the experiment is enabled in order to
permit having validations.
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block
With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
In an earlier change we switched to defining our own sets of detectors,
getters, etc for go-getter in order to insulate us from upstream changes
to those sets that might otherwise change the user-visible behavior of
Terraform's module installer.
However, we apparently neglected to actually refer to our local set of
detectors, and continued to refer to the upstream set. Here we catch up
with the latest detectors from upstream (taken from the version of
go-getter we currently have vendored) and start using that fixed set.
Currently we are maintaining these custom go-getter sets in two places
due to the configload vs. initwd distinction. That was already true for
goGetterGetters and goGetterDecompressors, and so I've preserved that for
now just to keep this change relatively simple; in later change it would
be nice to factor these "get with go getter" functions out into a shared
location which we can call from both configload and initwd.
* configs: move ProviderConfigCompact[Str] from addrs to configs
The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address. This is an intermediate step; the next step will change the returned types to something based in the configs package.
* command: rename choosePlugins to chooseProviders to clarify scope of function
* use `Provider.LegacyString()` (instead of `Provider.Type`) consistently
* explicitly create legacy-style provider (continuing from above change)
Traditionally we've preferred to release new language features in major
releases only, because we can then use the beta cycle to gather feedback
on the feature and learn about any usability challenges or other
situations we didn't consider during our design in time to make those
changes before inclusion in a stable release.
This "experiments" feature is intended to decouple the feedback cycle for
new features from the major release rhythm, and thus allow us to release
new features in minor releases by first releasing them as experimental for
a minor release or two, adjust for any feedback gathered during that
period, and then finally remove the experiment gate and enable the feature
for everyone.
The intended model here is that anything behind an experiment gate is
subject to breaking changes even in patch releases, and so any module
using these experimental features will be broken by a future Terraform
upgrade.
The behavior implemented here is:
- Recognize a new "experiments" setting in the "terraform" block which
allows module authors to explicitly opt in to experimental features.
terraform {
experiments = [resource_for_each]
}
- Generate a warning whenever loading a module that has experiments
enabled, to avoid accidentally depending on experimental features and
thus risking unexpected breakage on next Terraform upgrade.
- We check the enabled experiments against the configuration at module
load time, which means that experiments are scoped to a particular
module. Enabling an experiment in one module does not automatically
enable it in any other module.
This experiments mechanism is itself an experiment, and so I'd like to
use the resource for_each feature to trial it. Because any configuration
using experiments is subject to breaking changes, we are free to adjust
this experiments feature in future releases as we see fit, but once
for_each is shipped without an experiment gate we'll be blocked from
making significant changes to it until the next major release at least.
The configs package is aware of provider name and type (which are the
same thing today, but expected to be two different things in a future
release), and should be the source of truth for a provider config
address.
* huge change to weave new addrs.Provider into addrs.ProviderConfig
* terraform: do not include an empty string in the returned Providers /
Provisioners
- Fixed a minor bug where results included an extra empty string
* terraform/context: use new addrs.Provider as map key in provider factories
* added NewLegacyProviderType and LegacyString funcs to make it explicit that these are temporary placeholders
This PR introduces a new concept, provider fully-qualified name (FQN), encapsulated by the `addrs.Provider` struct.
Add deprecation warning for references from destroy provisioners or
their connections to external resources or values. In order to ensure
resource destruction can be completed correctly, destroy nodes must be
able to evaluate with only their instance state.
We have sufficient information to validate destroy-time provisioners
early on during the config loading process. Later on these can be
converted to hard errors, and only allow self, count.index, and each.key
in destroy provisioners. Limited the provisioner and block evaluation
scope later on is tricky, but if the references can never be loaded,
then they will never be encountered during evaluation.
`terraform 0.12upgrade` assumes that the configuration has passed 0.11
init, but did not explicitly check that the configuration was valid.
Certain issues would not get caught because the configuration was
syntactically valid. In this case, int or float values out of range
resulted in a panic from `Value()`.
Since running a 0.11 validate command is a breaking change, this PR
merely moves the `Value()` logic for ints and floats into `configupgrade` so
the error can be returned to the user, instead of causing a panic.
Following on from de652e22a26b, this introduces deprecation warnings for
when an attribute value expression is a template with only a single
interpolation sequence, and for variable type constraints given in quotes.
As with the previous commit, we allowed these deprecated forms with no
warning for a few releases after v0.12.0 to ensure that folks who need to
write cross-compatible modules for a while during upgrading would be able
to do so, but we're now marking these as explicitly deprecated to guide
users towards the new idiomatic forms.
The "terraform 0.12upgrade" tool would've already updated configurations
to not hit these warnings for those who had pre-existing configurations
written for Terraform 0.11.
The main target audience for these warnings are newcomers to Terraform who
are learning from existing examples already published in various spots on
the wider internet that may be showing older Terraform syntax, since those
folks will not be running their configurations through the upgrade tool.
These warnings will hopefully guide them towards modern Terraform usage
during their initial experimentation, and thus reduce the chances of
inadvertently adopting the less-readable legacy usage patterns in
greenfield projects.
Terraform 0.12.0 removed the need for putting references and keywords
in quotes, but we disabled the deprecation warnings for the initial
release in order to avoid creating noise for folks who were intentionally
attempting to maintain modules that were cross-compatible with both
Terraform 0.11 and Terraform 0.12.
However, with Terraform 0.12 now more widely used, the lack of these
warnings seems to be causing newcomers to copy the quoted versions from
existing examples on the internet, which is perpetuating the old and
confusing quoted form in newer configurations.
In preparation for phasing out these deprecated forms altogether in a
future major release, and for the shorter-term benefit of giving better
feedback to newcomers when they are learning from outdated examples, we'll
now re-enable those deprecation warnings, and be explicit that the old
forms are intended for removal in a future release.
In order to properly test this, we establish a new set of test
configurations that explicitly mark which warnings they are expecting and
verify that they do indeed produce those expected warnings. We also
verify that the "success" tests do _not_ produce warnings, while removing
the ones that were previously written to succeed but have their warnings
ignored.
Alisdair McDiarmid