This new command is intended to make it easy to create or update a mirror
directory containing suitable providers for the current configuration,
producing a layout that is appropriate both for a filesystem mirror or,
if copied into the document root of an HTTP server, a network mirror.
This initial version is not customizable aside from being able to select
multiple platforms to install packages for.
Future iterations of this could include commands to turn the JSON index
generation on and off, or to instruct it to produce the unpacked directory
layout instead of the packed directory layout as it currently does. Both
of those options would make the generated directory unsuitable to be
a network mirror, but it would still work as a filesystem mirror.
In the long run this will hopefully form part of a replacement workflow to
terraform-bundle as a way to put copies of providers somewhere so we don't
need to re-download them every time, but some other changes will be needed
outside of just this command before that'd be true, such as adding support
for network and/or filesystem mirrors in Terraform Enterprise.
This is a baby-step towards an intended future where all Terraform actions
which have side-effects in either remote objects or the Terraform state
can go through the plan+apply workflow.
This initial change is focused only on allowing plan+apply for changes to
root module output values, so that these can be written into a new state
snapshot (for consumption by terraform_remote_state elsewhere) without
having to go outside of the primary workflow by running
"terraform refresh".
This is also better than "terraform refresh" because it gives an
opportunity to review the proposed changes before applying them, as we're
accustomed to with resource changes.
The downside here is that Terraform Core was not designed to produce
accurate changesets for root module outputs. Although we added a place for
it in the plan model in Terraform 0.12, Terraform Core currently produces
inaccurate changesets there which don't properly track the prior values.
We're planning to rework Terraform Core's evaluation approach in a
forthcoming release so it would itself be able to distinguish between the
prior state and the planned new state to produce an accurate changeset,
but this commit introduces a temporary stop-gap solution of implementing
the logic up in the local backend code, where we can freeze a snapshot of
the prior state before we take any other actions and then use that to
produce an accurate output changeset to decide whether the plan has
externally-visible side-effects and render any changes to output values.
This temporary approach should be replaced by a more appropriately-placed
solution in Terraform Core in a release, which should then allow further
behaviors in similar vein, such as user-visible drift detection for
resource instances.
Fetching a default namespace provider from the public registry can
result in 404 Not Found error. This might be caused by a previously-
default provider moving to a new namespace, which means that the
configuration needs to be upgraded to use an explicit provider source.
This commit adds a more detailed diagnostic for this situation,
suggesting that the intended provider might be in a new namespace. The
recommended course of action is to run the 0.13upgrade command to
generate the correct required_providers configuration.
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.
To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.
As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.
This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.
This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.
Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
Relying on the early config for provider requirements was necessary in
Terraform 0.12, to allow the 0.12upgrade command to run after init
installs providers.
However in 0.13, the same restrictions do not apply, and the detection
of provider requirements has changed. As a result, the early config
loader gives incorrect provider requirements in some circumstances,
such as those in the new test in this commit.
Therefore we are changing the init command to use the requirements found
by the full configuration loader. This also means that we can remove the
internal initwd CheckCoreVersionRequirements function.
provider is not found.
Previously a user would see the following error even if terraform was
only searching the local filesystem:
"provider registry registry.terraform.io does not have a provider named
...."
This PR adds a registry-specific error type and modifies the MultiSource
installer to check for registry errors. It will return the
registry-specific error message if there is one, but if not the error
message will list all locations searched.
* providercache: add logging for errors from getproviders.SearchLocalDirectory
providercache.fillMetaCache() was silently swallowing errors when
searching the cache directory. This commit logs the error without
changing the behavior otherwise.
* command/cliconfig: validate plugin cache dir exists
The plugin cache directory must exist for terraform to use it, so we
will add a check at the begining.
Previously the diagnostics from the config loaders (earlyconfig and
regular) were only appended to the overall diags if an error was found.
This adds all diagnostics from the regular config loader so that any
generated warnings will be displayed, even if there are no errors.
I did not add the `earlyconfig` warnings since they will be displayed if
there is an error and are likely to be duplicated by the config loader.
* internal/registry source: return error if requested provider version protocols are not supported
* getproviders: move responsibility for protocol compatibility checks into the registry client
The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.
This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
When looking up the namespace for a legacy provider source, we need to
use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp
providers, the /v1/providers/-/{name} endpoint returns a 404.
This commit updates the LegacyProviderDefaultNamespace method and the
mock registry servers accordingly.
If a configuration had multiple blocks in the versions.tf file, it would
be added to the `rewritePaths` list multiple times. We would then remove
it from this slice, but only once, and so the output file would later be
rewritten to remove the required providers block.
This commit uses a set instead of a list to prevent this case, and adds
a regression test.
Instead of using providers.tf as the default output file for the
upgrader, we now default to versions.tf. This means that if the
configuration has no `required_providers` blocks at all, or has
multiple, the provider version requirements will be stored in the
versions.tf file.
We now also update the versions.tf file to set a `required_version`
attribute in the first `terraform` block, with value ">= 0.13". This
is similar to the behaviour of the 0.12upgrade command, and signals that
the configuration should not be used with older versions of Terraform.
This commit implements most of the intended functionality of the upgrade
command for rewriting configurations.
For a given module, it makes a list of all providers in use. Then it
attempts to detect the source address for providers without an explicit
source.
Once this step is complete, the tool rewrites the relevant configuration
files. This results in a single "required_providers" block for the
module, with a source for each provider.
Any providers for which the source cannot be detected (for example,
unofficial providers) will need a source to be defined by the user. The
tool writes an explanatory comment to the configuration to help with
this.
Providers like Okta and AWS Cognito expect that the PKCE challenge
uses base64 URL Encoding without any padding (base64.RawURLEncoding)
Additionally, Okta strictly adheres to section 4.2 of RFC 7636 and
requires that the unencoded key for the PKCE data is at least 43
characters in length.
The only situation where `state mv` needs to understand the each mode is
when with resource addresses that may reference a single instance, or a
group of for_each or count instances. In this case we can differentiate
the two by checking the existence of the NoKey instance key.
* Include eval in output walk
This allows outputs to be evaluated in the evalwalk,
impacting terraform console. Outputs are still not evaluated
for terraform console in the root module, so this has
no impact on writing to state (as child module outputs are not
written to state). Also adds test coverage to the console command,
including for evaluating locals (another use of the evalwalk)
These were being used in an earlier iteration of the provider installation
configuration but it was all collapsed down into a single
ProviderInstallationMethod type later, making these redundant.
This exercises the ability to customize the installation methods used by
the provider plugin installer, in this case forcing the use of a custom
local directory with a result essentially the same as what happens when
you pass -plugin-dir to "terraform init".
The CLI config can be written in both native HCL and HCL JSON syntaxes, so
the provider_installation block must be expressible using JSON too. Our
previous checks to approximate HCL 2-level strictness were too strict for
HCL JSON where things are more ambiguous even in HCL 2, so this includes
some additional relaxations if we detect that we're decoding an AST
produced from a JSON file.
This is still subject to the quirky ways HCL 1 handles JSON though, so
the JSON value must be structured in a way that doesn't trigger HCL's
heuristics that try to guess what is a block and what is an attribute.
(This is the issue that HCL 2 fixes by always decoding using a schema;
there's more context on this in:
https://log.martinatkins.me/2019/04/25/hcl-json/ )
Unfortunately in the user model the noun "source" is already used for the
argument in the required_providers block to specify which provider to use,
so it's confusing to use the same noun to also refer to the method used to
obtain that provider.
In the hope of mitigating that confusion, here we use the noun "method",
as in "installation method", to talk about the decision between getting
a provider directly from its origin registry or getting it from some
mirror. This is distinct from the provider's "source", which is the
location where a provider _originates_ (prior to mirroring).
This noun is also not super awesome, but better than overloading an
existing term in the same feature.
In the first pass of implementing this it was strict about what arguments
are allowed inside source blocks, but that was counter to our usual design
principles for CLI config where we tend to ignore unrecognized things to
allow for some limited kinds of future expansion without breaking
compatibility with older versions of Terraform that will be sharing the
same CLI configuration files with newer versions.
However, I'd removed the tracking of that prior to the initial commit. I
missed some leftover parts when doing that removal, so this cleans up the
rest of it.
An earlier commit added a redundant stub for a new network mirror source
that was already previously stubbed as HTTPMirrorSource.
This commit removes the unnecessary extra stub and changes the CLI config
handling to use it instead. Along the way this also switches to using a
full base URL rather than just a hostname for the mirror, because using
the usual "Terraform-native service discovery" protocol here doesn't isn't
as useful as in the places we normally use it (the mirror mechanism is
already serving as an indirection over the registry protocol) and using
a direct base URL will make it easier to deploy an HTTP mirror under
a path prefix on an existing static file server.
When we originally introduced this environment variable it was intended to
solve for the use-case where a particular invocation of Terraform needs
a different CLI configuration than usual, such as if Terraform is being
run as part of an automated test suite or other sort of automated
situation with different needs than normal use.
However, we accidentally had it only override the original singleton CLI
config file, while leaving the CLI configuration directory still enabled.
Now we'll take the CLI configuration out of the equation too, so that only
the single specified configuration file and any other environment-sourced
settings will be included.
* internal/providercache: verify that the provider protocol version is
compatible
The public registry includes a list of supported provider protocol
versions for each provider version. This change adds verification of
support and adds a specific error message pointing users to the closest
matching version.
This new CLI config block type allows explicitly specifying where
Terraform should look to find provider plugins for installation. This is
not used anywhere as of this commit, but in a future commit we'll change
package main to treat the presence of a block of this type as a request
to disable the default set of provider sources and use these explicitly-
specified ones instead.
A side effect of the various changes to the provider installer included losing the initialization required error message which would occur if a user removed or modified the .terraform directory.
Previously, plugin factories were created after the configuration was loaded, in terraform.NewContext. Terraform would compare the required providers (from config and state) to the available providers and return the aforementioned error if a provider was missing.
Provider factories are now loaded at the beginning of any terraform command, before terraform even loads the configuration, and therefore before terraform has a list of required providers.
This commit replaces the current error when a providers' schema cannot be found in the provider factories with the init error, and adds a command test (to plan tests, for no real reason other than that's what I thought of first).
This more closely replicates the 0.12-and-earlier behavior, where having
at least one version of a provider installed locally would totally disable
any attempt to look for newer versions remotely.
This is just for the implicit default behavior. Assumption is that later
we'll have an explicit configuration mechanism that will allow the user
to specify exactly where to look for what, and thus avoid tricky
heuristics like this.
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:
- First we must verify that the SHA256 hash of the package archive
matches the expected hash. This could be done for local installations
too, in the future.
- Next we ensure that the expected hash returned as part of the registry
API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
using the public keys provided by the registry.
Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.
The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.
There are three tiers of signature, each of which is presented
differently to the user:
- Signatures from the embedded HashiCorp public key indicate that the
provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
trust signature, which indicates that the provider is from one of
HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
Martin Atkins