This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
At the time of this commit we have a proposal #28700 which would, if
accepted, need to reserve a new reference prefix to represent template
arguments.
It seems unlikely that the proposal would be accepted and implemented
before Terraform v1.0 creates additional compatibility constraints, and so
this pre-emptively reserves a few candidate symbol names to allow
something like that proposal to potentially move forward later without
requiring a new opt-in language edition.
If we do move forward with the proposal then we'll select one of these
three reserved names depending on which form of the proposal we decide
to move forward with, and then un-reserve the other two. If we decide to
not pursue this proposal at all then we'll un-reserve all three once
that decision is finalized.
It's unlikely that there is any existing provider which has a resource
type named either "template", "lazy", or "arg", but in that unlikely event
users of that provider can keep using it by adding the "resource."
escaping prefix, such as changing "lazy.foo.bar" into
"resource.lazy.foo.bar".
The current way to refer to a managed resource is to use its resource type
name as a top-level symbol in the reference. This is convenient and makes
sense given that managed resources are the primary kind of object in
Terraform.
However, it does mean that an externally-extensible namespace (the set
of all possible resource type names) overlaps with a reserved word
namespace (the special prefixes like "path", "var", etc), and thus
introducing any new reserved prefix in future risks masking an existing
resource type so it can't be used anymore.
We only intend to introduce new reserved symbols as part of future
language editions that each module can opt into separately, and when doing
so we will always research to try to choose a name that doesn't overlap
with commonly-used providers, but not all providers are visible to us and
so there is always a small chance that the name we choose will already be
in use by a third-party provider.
In preparation for that event, this introduces an alternative way to refer
to managed resources that mimics the reference style used for data
resources: resource.type.name . When using this form, the second portion
is _always_ a resource type name and never a reserved word.
There is currently no need to use this because all of the already-reserved
symbol names are effectively blocked from use by existing Terraform
versions that lack this escape hatch. Therefore there's no explicit
documentation about it yet.
The intended use for this is that a module upgrade tool for a future
language edition would detect references to resource types that have now
become reserved words and add the "resource." prefix to keep that
functionality working. Existing modules that aren't opted in to the new
language edition would keep working without that prefix, thus keeping to
compatibility promises.
* Optimize (m ModuleInstance) String()
Optimize (m ModuleInstance) String() to preallocate the buffer and use strings.Builder instead of bytes.Buffer
This leads to a common case only doing a single allocation as opposed to a few allocations which the bytes.Buffer is doing.
* adding a benchmark test
Result:
```
$ go test -bench=String ./addrs -benchmem
BenchmarkStringShort-12 18271692 56.52 ns/op 16 B/op 1 allocs/op
BenchmarkStringLong-12 8057071 158.5 ns/op 96 B/op 1 allocs/op
PASS
$ git checkout main addrs/module_instance.go
$ go test -bench=String ./addrs -benchmem
BenchmarkStringShort-12 7690818 162.0 ns/op 80 B/op 2 allocs/op
BenchmarkStringLong-12 2922117 414.1 ns/op 288 B/op 3 allocs/op
```
* Update module_instance_test.go
switch spaces to tabs
Generating strings and comparing them to implement Equal is a quick and
easy solution. Unfortunately when this code is in the hot path, it
becomes very expensive, so this commit changes some of those instances
to compare the values directly.
Combined with using addr.Equal instead of checking for string equality,
this makes Terraform dramatically faster for some operations, such as
generating large JSON plans.
The Resource.Absolute function is there to conveniently construct an
AbsResource from a Resource by providing a module instance. Likewise, this
new InModule method allows conveniently constructing a ConfigResource from
a Resource by providing a module.
Builtin provider addrs (i.e. "terraform.io/builtin/terraform") should be
able to convert to legacy string form (i.e. "terraform"). This ensures
that we can safely round-trip through ParseLegacyAbsProviderConfig,
which can return either a legacy or a builtin provider addr.
The main motivation here is to produce a helpful error if a user
incorrectly uses the terraform-provider- prefix (which we see on provider
VCS repositories and plugin executables) as part of the source address.
However, this also more broadly blocks "terraform-" as a prefix in
anticipation of whatever instinct causes the phenomenon where e.g.
Python's PyPI has thousands of packages whose names start with "python-",
even though everything on PyPI is for Python by definition. This is
definitely not _necessary_, but it's better to be restrictive at first
and weaken later as needed.
If the last step in a module target is an unkeyed instance, and it's
being compared against keyed instances, we have to assume it was
intended to be used as a Module rather than a ModuleInstance.
* addrs: detect builtin provider when parsing legacy provider string
The ParseLegacyAbsProviderConfig was not detecting builtin providers
("terraform"), which caused issues for all users with 0.12 state and
the "terraform_remote_state" data source. Since "terraform" is the only
built-in provider this adds a very simple check to the parser so it
properly returns the builtin FQN.
* add tests to the addrs package
* addrs: replace NewLegacyProvider with NewDefaultProvider in ParseProviderSourceString
ParseProviderSourceString was still defaulting to NewLegacyProvider when
encountering single-part strings. This has been fixed.
This commit also adds a new function, IsProviderPartNormalized, which
returns a bool indicating if the string given is the same as a
normalized version (as normalized by ParseProviderPart) or an error.
This is intended for use by the configs package when decoding provider
configurations.
* terraform: fix provider local names in tests
* configs: validate that all provider names are normalized
The addrs package normalizes all source strings, but not the local
names. This caused very odd behavior if for e.g. a provider local name
was capitalized in one place and not another. We considered enabling
case-sensitivity for provider local names, but decided that since this
was not something that worked in previous versions of terraform (and we
have yet to encounter any use cases for this feature) we could generate
an error if the provider local name is not normalized. This error also
provides instructions on how to fix it.
* configs: refactor decodeProviderRequirements to consistently not set an FQN when there are errors
* internal/registry source: return error if requested provider version protocols are not supported
* getproviders: move responsibility for protocol compatibility checks into the registry client
The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.
This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
Module references, like resource references, need to always return the
and object containing all instances in order to handle modules as single
values, and to postpone index evaluation to when the expression as whole
is evaluated.
The provider fully-qualified name string used in configuration is very
long, and since most providers are hosted in the public registry, most
of that length is redundant. This commit adds and uses a `ForDisplay`
method, which simplifies the presentation of provider FQNs.
If the hostname is the default hostname, we now display only the
namespace and type. This is only used in UI, but should still be
unambiguous, as it matches the FQN string parsing behaviour.
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".
This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
The introduction of a heirarchical addressing scheme for providers gives
us an opportunity to make more explicit the special case of "built-in"
providers.
Thus far we've just had a special case in the "command" package that the
provider named "terraform" is handled differently than all others, though
there's nothing especially obvious about that in the UI.
Moving forward we'll put such "built-in" providers under the special
namespace terraform.io/builtin/terraform, which will be visible in the UI
as being different than the other providers and we can use the namespace
itself (rather than a particular name) as the trigger for our special-case
behaviors around built-in plugins.
We have no plans to introduce any built-in providers other than
"terraform" in the foreseeable future, so any others will produce an
error.
This commit just establishes the addressing convention, without making use
of it anywhere yet. Subsequent commits will make the provider installer
and resolver codepaths aware of it, replacing existing checks for the
provider just being called "terraform".
* terraform: large refactor to use Provider from configs.Resource
configs.Resource.ImpliedProvider() now returns a string; it is the
callers' responsibility to turn that into an addrs.Provider if needed.
GraphNodeProviderConsumer ProvidedBy() no longer returns nil (reverting
to earlier, pre-provider-fqn behavior): it will return either the
provider set in config, provider set in state, or the default provider.
Core needs a way to address resources through unexpanded modules, as
they are present in the configuration. There are already some cases of
paring `addrs.Module` with `addrs.Resource` for this purpose, but it is
going to be helpful to have a single type to describe that pair, as
well as have the ability to use TargetContains.
* configs: parse provider source string during module merge
This was the smallest unit of work needed to start writing provider
source tests!
* Update configs/parser_test.go
Co-Authored-By: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Change ModuleInstance to Module in AbsProviderConfig, because providers
need to be handled before module expansion, and should not be used
defined inside an expanded module at all.
Renaming of the addrs type can happen later, when there's less work
in-flight around provider configuration.
Since references are always within the scope of a single module, and we
can connect all module instance outputs for proper ordering, the
existing transformer works directly with only module paths as opposed to
module instances.
TODO: TransformApplyReferences for more precise module instance
targeting?
When making lists of providers (or lists that contain providers) it's
helpful to have a canonical ordering in order to produce deterministic
results.
This ordering has no semantic meaning and is just here for the sake of
having a predictable standard.
* WIP: dynamic expand
* WIP: add variable and local support
* WIP: outputs
* WIP: Add referencer
* String representation, fixing tests it impacts
* Fixes TestContext2Apply_outputOrphanModule
* Fix TestContext2Apply_plannedDestroyInterpolatedCount
* Update DestroyOutputTransformer and associated types to reflect PlannableOutputs
* Remove comment about locals
* Remove module count enablement
* Removes allowing count for modules, and reverts the test,
while adding a Skip()'d test that works when you re-enable
the config
* update TargetDownstream signature to match master
* remove unnecessary method
Co-authored-by: James Bardin <j.bardin@gmail.com>
The provider FQN is becoming our primary identifier for a provider, so
it's important that we are clear about the equality rules for these
addresses and what characters are valid within them.
We previously had a basic regex permitting ASCII letters and digits for
validation and no normalization at all. We need to do at least case
folding and UTF-8 normalization because these names will appear in file
and directory names in case-insensitive filesystems and in repository
names such as on GitHub.
Since we're already using DNS-style normalization and validation rules
for the hostname part, rather than defining an entirely new set of rules
here we'll just treat the provider namespace and type as if they were
single labels in a DNS name. Aside from some internal consistency, that
also works out nicely because systems like GitHub use organization and
repository names as part of hostnames (e.g. with GitHub Pages) and so
tend to apply comparable constraints themselves.
This introduces the possibility of names containing letters from alphabets
other than the latin alphabet, and for latin letters with diacritics.
That's consistent with our introduction of similar support for identifiers
in the language in Terraform 0.12, and is intended to be more friendly to
Terraform users throughout the world that might prefer to name their
products using a different alphabet. This is also a further justification
for using the DNS normalization rules: modern companies tend to choose
product names that make good domain names, and now such names will be
usable as Terraform provider names too.
When ModuleInstanceStep values appear alone in debug messages, it's easier
to read them in a compact, HCL-like form than as the default struct
printing style.
* fix outdated syntax in comments
* test for non-strings in ParseAbsProviderConfig
* ProviderConfigDefault and ProviderConfigAliased now take Providers
instead of strings
a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings).
The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.
* Introduce "Local" terminology for non-absolute provider config addresses
In a future change AbsProviderConfig and LocalProviderConfig are going to
become two entirely distinct types, rather than Abs embedding Local as
written here. This naming change is in preparation for that subsequent
work, which will also include introducing a new "ProviderConfig" type
that is an interface that AbsProviderConfig and LocalProviderConfig both
implement.
This is intended to be largely just a naming change to get started, so
we can deal with all of the messy renaming. However, this did also require
a slight change in modeling where the Resource.DefaultProviderConfig
method has become Resource.DefaultProvider returning a Provider address
directly, because this method doesn't have enough information to construct
a true and accurate LocalProviderConfig -- it would need to refer to the
configuration to know what this module is calling the provider it has
selected.
In order to leave a trail to follow for subsequent work, all of the
changes here are intended to ensure that remaining work will become
obvious via compile-time errors when all of the following changes happen:
- The concept of "legacy" provider addresses is removed from the addrs
package, including removing addrs.NewLegacyProvider and
addrs.Provider.LegacyString.
- addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded
in it and has an addrs.Provider and a string alias directly instead.
- The provider-schema-handling parts of Terraform core are updated to
work with addrs.Provider to identify providers, rather than legacy
strings.
In particular, there are still several codepaths here making legacy
provider address assumptions (in order to limit the scope of this change)
but I've made sure each one is doing something that relies on at least
one of the above changes not having been made yet.
* addrs: ProviderConfig interface
In a (very) few special situations in the main "terraform" package we need
to make runtime decisions about whether a provider config is absolute
or local.
We currently do that by exploiting the fact that AbsProviderConfig has
LocalProviderConfig nested inside of it and so in the local case we can
just ignore the wrapping AbsProviderConfig and use the embedded value.
In a future change we'll be moving away from that embedding and making
these two types distinct in order to represent that mapping between them
requires consulting a lookup table in the configuration, and so here we
introduce a new interface type ProviderConfig that can represent either
AbsProviderConfig or LocalProviderConfig decided dynamically at runtime.
This also includes the Config.ResolveAbsProviderAddr method that will
eventually be responsible for that local-to-absolute translation, so
that callers with access to the configuration can normalize to an
addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's
currently unused because existing callers are still relying on the
simplistic structural transform, but we'll switch them over in a later
commit.
* rename LocalType to LocalName
Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
Martin Atkins