The provider FQN is becoming our primary identifier for a provider, so
it's important that we are clear about the equality rules for these
addresses and what characters are valid within them.
We previously had a basic regex permitting ASCII letters and digits for
validation and no normalization at all. We need to do at least case
folding and UTF-8 normalization because these names will appear in file
and directory names in case-insensitive filesystems and in repository
names such as on GitHub.
Since we're already using DNS-style normalization and validation rules
for the hostname part, rather than defining an entirely new set of rules
here we'll just treat the provider namespace and type as if they were
single labels in a DNS name. Aside from some internal consistency, that
also works out nicely because systems like GitHub use organization and
repository names as part of hostnames (e.g. with GitHub Pages) and so
tend to apply comparable constraints themselves.
This introduces the possibility of names containing letters from alphabets
other than the latin alphabet, and for latin letters with diacritics.
That's consistent with our introduction of similar support for identifiers
in the language in Terraform 0.12, and is intended to be more friendly to
Terraform users throughout the world that might prefer to name their
products using a different alphabet. This is also a further justification
for using the DNS normalization rules: modern companies tend to choose
product names that make good domain names, and now such names will be
usable as Terraform provider names too.
This is a temporary helper so that we can potentially ship the new
provider installer without making a breaking change by relying on the
old default namespace lookup API on the default registry to find a proper
FQN for a legacy provider provider address during installation.
If it's given a non-legacy provider address then it just returns the given
address verbatim, so any codepath using it will also correctly handle
explicit full provider addresses. This also means it will automatically
self-disable once we stop using addrs.NewLegacyProvider in the config
loader, because there will therefore no longer be any legacy provider
addresses in the config to resolve. (They'll be "default" provider
addresses instead, assumed to be under registry.terraform.io/hashicorp/* )
It's not decided yet whether we will actually introduce the new provider
in a minor release, but even if we don't this API function will likely be
useful for a hypothetical automatic upgrade tool to introduce explicit
full provider addresses into existing modules that currently rely on
the equivalent to this lookup in the current provider installer.
This is dead code for now, but my intent is that it would either be called
as part of new provider installation to produce an address suitable to
pass to Source.AvailableVersions, or it would be called from the
aforementioned hypothetical upgrade tool.
Whatever happens, these functions can be removed no later than one whole
major release after the new provider installer is introduced, when
everyone's had the opportunity to update their legacy unqualified
addresses.
Our local filesystem mirror mechanism will allow provider packages to be
given either in packed form as an archive directly downloaded to disk or
in an unpacked form where the archive is extracted.
Distinguishing these two cases in the concrete Location types will allow
callers to reliably select the mode chosen by the selected installation
source and handle it appropriately, rather than resorting to out-of-band
heuristics like checking whether the object is a directory or a file.
In a future commit, these implementations of Source will allow finding
and retrieving provider packages via local mirrors, both in the local
filesystem and over the network using an HTTP-based protocol.
This is an API stub for a component that will be added in a future commit
to support considering a number of different installation sources for each
provider. These will eventually be configurable in the CLI configuration,
allowing users to e.g. mirror certain providers within their own
infrastructure while still being able to go upstream for those that aren't
mirrored, or permit locally-mirrored providers only, etc.
Some sources make network requests that are likely to be slow, so this
wrapper type can cache previous responses for its lifetime in order to
speed up repeated requests for the same information.
Registries backed by static files are likely to use relative paths to
their archives for simplicity's sake, but we'll normalize them to be
absolute before returning because the caller wouldn't otherwise know what
to resolve the URLs relative to.
We intend to support installation both directly from origin registries and
from mirrors in the local filesystem or over the network. This Source
interface will serve as our abstraction over those three options, allowing
calling code to treat them all the same.
Our existing provider installer was originally built to work with
releases.hashicorp.com and later retrofitted to talk to the official
Terraform Registry. It also assumes a flat namespace of providers.
We're starting a new one here, copying and adapting code from the old one
as necessary, so that we can build out this new API while retaining all
of the existing functionality and then cut over to this new implementation
in a later step.
Here we're creating a foundational component for the new installer, which
is a mechanism to query for the available versions and download locations
of a particular provider.
Subsequent commits in this package will introduce other Source
implementations for installing from network and filesystem mirrors.
* deps: bump terraform-config-inspect library
* configs: parse `version` in new required_providers block
With the latest version of `terraform-config-inspect`, the
required_providers attribute can now be a string or an object with
attributes "source" and "version". This change allows parsing the
version constraint from the new object while ignoring any given source attribute.
In an earlier change we switched to defining our own sets of detectors,
getters, etc for go-getter in order to insulate us from upstream changes
to those sets that might otherwise change the user-visible behavior of
Terraform's module installer.
However, we apparently neglected to actually refer to our local set of
detectors, and continued to refer to the upstream set. Here we catch up
with the latest detectors from upstream (taken from the version of
go-getter we currently have vendored) and start using that fixed set.
Currently we are maintaining these custom go-getter sets in two places
due to the configload vs. initwd distinction. That was already true for
goGetterGetters and goGetterDecompressors, and so I've preserved that for
now just to keep this change relatively simple; in later change it would
be nice to factor these "get with go getter" functions out into a shared
location which we can call from both configload and initwd.
faster
The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.
It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
Add versioned tfplugin proto files to the docs directory, for easier
reference. The latest version starts as a symlink to the current
file used for generated the tfplugin package in ./internal/tfplugin5.
When changing the protocol version, the old file must be copied to
./docs/plugin-protocol/, and a new symlink created for the latest
version.
Private data was previously created during Plan, and sent back to the
provider during Apply. This data also needs to be persisteded accross
Read calls, but rather than rely on core for that we can send the data
to the provider during Read to allow for more flexibilty.
This was already working, but since that codepath is separate from the
go-getter install codepath it's helpful to have a separate test for it,
in addition to the existing one for go-getter modules.
The "err" variable in the MaybeRelativePathErr condition was masking the
original err with nil in the "else" case of this branch, causing the
error message to be incomplete.
While here, also tweaked the wording to say "Could not download" rather
than "Error attempting to download", both to say the same thing in fewer
words and because the summary line above already starts with "Error:"
when we print out this message, so it looks weird to have both lines
start with the same word.
* internal/initwd: follow local module path symlink
Fixes#21060
While a previous commit fixed a problem when the local module directory
contained a symlink, it did not account for the possibility that the
entire directory was a symlink.
In study of existing providers we've found a pattern we werent previously
accounting for of using a nested block type to represent a group of
arguments that relate to a particular feature that is always enabled but
where it improves configuration readability to group all of its settings
together in a nested block.
The existing NestingSingle was not a good fit for this because it is
designed under the assumption that the presence or absence of the block
has some significance in enabling or disabling the relevant feature, and
so for these always-active cases we'd generate a misleading plan where
the settings for the feature appear totally absent, rather than showing
the default values that will be selected.
NestingGroup is, therefore, a slight variation of NestingSingle where
presence vs. absence of the block is not distinguishable (it's never null)
and instead its contents are treated as unset when the block is absent.
This then in turn causes any default values associated with the nested
arguments to be honored and displayed in the plan whenever the block is
not explicitly configured.
The current SDK cannot activate this mode, but that's okay because its
"legacy type system" opt-out flag allows it to force a block to be
processed in this way anyway. We're adding this now so that we can
introduce the feature in a future SDK without causing a breaking change
to the protocol, since the set of possible block nesting modes is not
extensible.
* configs/configupgrade: detect possible relative module sources
If a module source appears to be a relative local path but does not have
a preceding ./, print a #TODO message for the user.
* internal/initwd: limit go-getter detectors to those supported by terraform
* internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter
To avoid making two calls to getter.Detect, which potentially makes
non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to
a later step and a custom error type was added so user-friendly
diagnostics could be displayed in the event that a possible relative local
path was detected.
Identify module sources that look like relative paths ("child" instead
of "./child", for example) and surface a helpful error.
Previously, such module sources would be passed to go-getter, which
would fail because it was expecting an absolute, or properly relative,
path. This commit moves the check for improper relative paths sooner so
a user-friendly error can be displayed.
configs/configload and internal/initwd both had a copyDir function that
would fail if the source directory contained a symlinked directory,
because the os.FileMode.IsDir() returns false for symlinks.
This PR adds a check for a symlink and copies that symlink in the
target directory. It handles symlinks for both files and directories
(with included tests).
Fixes#20539
Due to the inprecision of our shimming from the legacy SDK type system to
the new Terraform Core type system, the legacy SDK produces a number of
inconsistencies that produce only minor quirky behavior or broken
edge-cases. To retain compatibility with those existing weird behaviors,
the legacy SDK opts out of our safety checks.
The intent here is to allow existing providers to continue to do their
previous unsafe behaviors for now, accepting that this will allow certain
quirky bugs from previous releases to persist, and then gradually migrate
away from the legacy SDK and remove this opt-out on a per-resource basis
over time.
As with the apply-time safety check opt-out, this is reserved only for
the legacy SDK and must not be used in any new SDK implementations. We
still include any inconsistencies as warnings in the logs as an aid to
anyone debugging weird behavior, so that they can see situations where
blame may be misplaced in the user-visible error messages.
The shim layer for the legacy SDK type system is not precise enough to
guarantee it will produce identical results between plan and apply. In
particular, values that are null during plan will often become zero-valued
during apply.
To avoid breaking those existing providers while still allowing us to
introduce this check in the future, we'll introduce a rather-hacky new
flag that allows the legacy SDK to signal that it is the legacy SDK and
thus disable the check.
Once we start phasing out the legacy SDK in favor of one that natively
understands our new type system, we can stop setting this flag and thus
get the additional safety of this check without breaking any
previously-released providers.
No other SDK is permitted to set this flag, and we will remove it if we
ever introduce protocol version 6 in future, assuming that any provider
supporting that protocol will always produce consistent results.
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.
This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.
In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.
Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.
The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
excludes Terraform versions prior to v0.12 then the configuration is
probably _already_ upgraded and so it's just a normal syntax error,
even if the early loader didn't detect it.
Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
This is an adaptation of the installation code from configs/configload,
now using the "earlyconfig" package instead of the "configs" package.
Module installation is an initialization-only process, with all other
commands assuming an already-initialized directory. Having it here can
therefore simplify the API of configs/configload, which can now focus only
on the problem of loading modules that have already been installed.
The old installer code in configs/configload is still in place for now
because the caller in "terraform init" isn't yet updated to use this.
"terraform init" is quite a complex beast in relation to other commands
since almost everything it does is unique to it and thus not factored out
into other packages.
To get some of that sprawl out of the "command" package, this new internal
package will give us somewhere to put this init functionality that is
also useful for test code that needs to mimic the initialization behavior
against fixture directories.
This is an alternative to the full config loader in the "configs" package
that is good for "early" use-cases like "terraform init", where we want
to find out what our dependencies are without getting tripped up on any
other errors that might be present.
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.
Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.
After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".
In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
Martin Atkins