This exercises the ability to customize the installation methods used by
the provider plugin installer, in this case forcing the use of a custom
local directory with a result essentially the same as what happens when
you pass -plugin-dir to "terraform init".
The CLI config can be written in both native HCL and HCL JSON syntaxes, so
the provider_installation block must be expressible using JSON too. Our
previous checks to approximate HCL 2-level strictness were too strict for
HCL JSON where things are more ambiguous even in HCL 2, so this includes
some additional relaxations if we detect that we're decoding an AST
produced from a JSON file.
This is still subject to the quirky ways HCL 1 handles JSON though, so
the JSON value must be structured in a way that doesn't trigger HCL's
heuristics that try to guess what is a block and what is an attribute.
(This is the issue that HCL 2 fixes by always decoding using a schema;
there's more context on this in:
https://log.martinatkins.me/2019/04/25/hcl-json/ )
Unfortunately in the user model the noun "source" is already used for the
argument in the required_providers block to specify which provider to use,
so it's confusing to use the same noun to also refer to the method used to
obtain that provider.
In the hope of mitigating that confusion, here we use the noun "method",
as in "installation method", to talk about the decision between getting
a provider directly from its origin registry or getting it from some
mirror. This is distinct from the provider's "source", which is the
location where a provider _originates_ (prior to mirroring).
This noun is also not super awesome, but better than overloading an
existing term in the same feature.
In the first pass of implementing this it was strict about what arguments
are allowed inside source blocks, but that was counter to our usual design
principles for CLI config where we tend to ignore unrecognized things to
allow for some limited kinds of future expansion without breaking
compatibility with older versions of Terraform that will be sharing the
same CLI configuration files with newer versions.
However, I'd removed the tracking of that prior to the initial commit. I
missed some leftover parts when doing that removal, so this cleans up the
rest of it.
An earlier commit added a redundant stub for a new network mirror source
that was already previously stubbed as HTTPMirrorSource.
This commit removes the unnecessary extra stub and changes the CLI config
handling to use it instead. Along the way this also switches to using a
full base URL rather than just a hostname for the mirror, because using
the usual "Terraform-native service discovery" protocol here doesn't isn't
as useful as in the places we normally use it (the mirror mechanism is
already serving as an indirection over the registry protocol) and using
a direct base URL will make it easier to deploy an HTTP mirror under
a path prefix on an existing static file server.
When we originally introduced this environment variable it was intended to
solve for the use-case where a particular invocation of Terraform needs
a different CLI configuration than usual, such as if Terraform is being
run as part of an automated test suite or other sort of automated
situation with different needs than normal use.
However, we accidentally had it only override the original singleton CLI
config file, while leaving the CLI configuration directory still enabled.
Now we'll take the CLI configuration out of the equation too, so that only
the single specified configuration file and any other environment-sourced
settings will be included.
This new CLI config block type allows explicitly specifying where
Terraform should look to find provider plugins for installation. This is
not used anywhere as of this commit, but in a future commit we'll change
package main to treat the presence of a block of this type as a request
to disable the default set of provider sources and use these explicitly-
specified ones instead.
A side effect of the various changes to the provider installer included losing the initialization required error message which would occur if a user removed or modified the .terraform directory.
Previously, plugin factories were created after the configuration was loaded, in terraform.NewContext. Terraform would compare the required providers (from config and state) to the available providers and return the aforementioned error if a provider was missing.
Provider factories are now loaded at the beginning of any terraform command, before terraform even loads the configuration, and therefore before terraform has a list of required providers.
This commit replaces the current error when a providers' schema cannot be found in the provider factories with the init error, and adds a command test (to plan tests, for no real reason other than that's what I thought of first).
This more closely replicates the 0.12-and-earlier behavior, where having
at least one version of a provider installed locally would totally disable
any attempt to look for newer versions remotely.
This is just for the implicit default behavior. Assumption is that later
we'll have an explicit configuration mechanism that will allow the user
to specify exactly where to look for what, and thus avoid tricky
heuristics like this.
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:
- First we must verify that the SHA256 hash of the package archive
matches the expected hash. This could be done for local installations
too, in the future.
- Next we ensure that the expected hash returned as part of the registry
API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
using the public keys provided by the registry.
Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.
The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.
There are three tiers of signature, each of which is presented
differently to the user:
- Signatures from the embedded HashiCorp public key indicate that the
provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
trust signature, which indicates that the provider is from one of
HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
The providers command has been refactored to use the modern provider types and
ProviderRequirements() functions. This resulted in a breaking change to
the output: it no longer outputs the providers by module and no longer
prints `(inherited)` or `(from state)` to show why a provider is
included. We decided that at this time it was best to stick with the
existing functions and make this change, but if we get feedback from the
community we will revisit.
Additional tests to exercise providers in modules and providers from
state have been included.
This PR adds iteration through any provider configuration blocks in the
config in addProviderRequirements().
A stale comment (of mine!) would leave one expecting the
module.ProviderRequirements to include any requirements from provider
configs. The comment was inaccurate and has been updated.
When a provider dependency is implicit rather than explicit, or otherwise
when version constraints are lacking, we produce a warning recommending
the addition of explicit version constraints in the configuration.
This restores the warning functionality from previous Terraform versions,
adapting it slightly to account for the new provider FQN syntax and to
recommend using a required_providers block rather than version constraints
in "provider" blocks, because the latter is no longer recommended in the
documentation.
The fake installable package meta used a ZIP archive which gave
different checksums between macOS and Linux targets. This commit removes
the target from the contents of this archive, and updates the golden
hash value in the test to match. This test should now pass on both
platforms.
The provider fully-qualified name string used in configuration is very
long, and since most providers are hosted in the public registry, most
of that length is redundant. This commit adds and uses a `ForDisplay`
method, which simplifies the presentation of provider FQNs.
If the hostname is the default hostname, we now display only the
namespace and type. This is only used in UI, but should still be
unambiguous, as it matches the FQN string parsing behaviour.
This restores some of the local search directories we used to include when
searching for provider plugins in Terraform 0.12 and earlier. The
directory structures we are expecting in these are different than before,
so existing directory contents will not be compatible without
restructuring, but we need to retain support for these local directories
so that users can continue to sideload third-party provider plugins until
the explicit, first-class provider mirrors configuration (in CLI config)
is implemented, at which point users will be able to override these to
whatever directories they want.
This also includes some new search directories that are specific to the
operating system where Terraform is running, following the documented
layout conventions of that platform. In particular, this follows the
XDG Base Directory specification on Unix systems, which has been a
somewhat-common request to better support "sideloading" of packages via
standard Linux distribution package managers and other similar mechanisms.
While it isn't strictly necessary to add that now, it seems ideal to do
all of the changes to our search directory layout at once so that our
documentation about this can cleanly distinguish "0.12 and earlier" vs.
"0.13 and later", rather than having to document a complex sequence of
smaller changes.
Because this behavior is a result of the integration of package main with
package command, this behavior is verified using an e2etest rather than
a unit test. That test, TestInitProvidersVendored, is also fixed here to
create a suitable directory structure for the platform where the test is
being run. This fixes TestInitProvidersVendored.
There was a remaining TODO in this package to find the true provider FQN
when looking up the schema for a resource type. We now have that data
available in the Provider field of configs.Resource, so we can now
complete that change.
The tests for this functionality actually live in the parent "command"
package as part of the tests for the "terraform show" command, so this
fix is verified by all of the TestShow... tests now passing except one,
and that remaining one is failing for some other reason which we'll
address in a later commit.
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".
Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.
For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
* command: refactor testBackendState to write states.State
testBackendState was using the older terraform.State format, which is no
longer sufficient for most tests since the state upgrader does not
encode provider FQNs automatically. Users will run `terraform
0.13upgrade` to update their state to include provider FQNs in
resources, but tests need to use the modern state format instead of
relying on the automatic upgrade.
* plan tests passing
* graph tests passing
* json packages test update
* command test updates
* update show test fixtures
* state show tests passing
In the new design the ProviderSource is decided by package main, not by
the "command" package, and so making sure the vendor directory is included
is the responsibility of that package instead. Therefore we can no longer
test this at the "command" package level, but we'll retain a test for it
in e2etests to record that it isn't currently working, so that we have
a prompt to fix it before releasing.
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.
This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
These tests make assertions against specific user-oriented output from the
"terraform init" command, but we've intentionally changed some of these
messages as part of introducing support for the decentralized provider
namespace.
Both of these are attempting to test -plugin-dir, which means we need some
additional help to populate some suitable directories for -plugin-dir to
refer to. The new installFakeProviderPackagesElsewhere helper generalizes
the earlier installFakeProviderPackages to allow installing fake provider
packages to an arbitrary other directory.
This test is focused on making sure that the required_providers syntax
is working, so the rewritten version does not include any special handling
of pre-installed packages or "vendored" packages. Pre-installed plugins
are tested in other tests such as TestInit_getUpgradePlugins.
This test now requires a bit of a different approach because it was
previously directly constructing a cache directory but we now use a
different directory layout.
Rather than manually constructing the new heirarchical directory layout
(which would've required a lot more inline code), this introduces a helper
function installFakeProviderPackages that installs a fake provider package
directly into the local cache directory associated with a Meta object,
with the correct directory layout.
This is a slightly different approach than we used to take for this
option: rather than disabling the installer and causing all future
commands to look elsewhere for plugins, we'll now leave the installer
enabled by constrain it to only look at the given directories.
This is overall simpler because it doesn't require any special tracking
of the plugin directories for subsequent commands. Instead, the selections
file generated by the installer will record the versions it selected from
the specified directories, and we'll link them in to the local cache just
as we would normally so that other commands don't need to do anything
special to select the right plugins in either case.
They still aren't passing, but this is just enough updating to make the
test program compile successfully after the refactoring related to
provider installation. They are now using the mock provider source offered
by the getproviders package, which is similar but not totally identical
to the idea of mocking the entire installer as these tests used to do, and
so many of them need further adjustment to still be testing what they
intended to test under this new architecture.
Subsequent commits will gradually repair the failing tests.
* terraform: add helper functions for creating test state
testSetResourceInstanceCurrent and testSetResourceInstanceTainted are
wrapper functions around states.Module.SetResourceInstanceCurrent()
used to set a resource in state. They work with current, non-deposed
resources with no dependencies.
testSetResourceInstanceDeposed can be used to set a desosed resource in state.
* terraform: update all tests to use modern providers and state
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.
However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.
The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.
This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
There's still a lot of work to do here around both the UX and the
follow-up steps that need to happen after installation completes, but this
is enough to faciliate some initial end-to-end testing of the new-style
install process.
Terraform 0.13 will allow the installation of providers from various
sources. If a user updates their configuration to change the source of
an in-use provider (for example, if the provider namespace changes),
they will also need to update the state file accordingly.
This commit introduces a new `state replace-provider` subcommand which
supports this. All resources using the `from` provider will be updated
to use the `to` provider.
Previously, if a diagnostic context spanned multiple lines, any lines
which did not overlap with the highlight range would be displayed as
blank. This commit fixes the bug.
The problem was caused by the unconditional use of `PartitionAround` to
split the line into before/highlighted/after ranges. When two ranges
don't overlap, this method returns empty ranges, which results in a
blank line. Instead, we first check if the ranges do overlap, and if not
we print the entire line from the context.
Previously, diagnostic errors would display the filename and line
number, along with "(source code not available)". This is because the
fmt command directly loads and parses the configuration, instead of
using the config loader.
This commit registers the manually parsed source as a synthetic
configuration file, so that the diagnostic formatter can look up the
source for the range with the error and display it.
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
missingPlugins was hard-coded to work only with provider plugins, so I
renamed it to clarify the usage.
Also renamed a test provider from greater_than to greater-than as the
underscore is an invalid provider name character and this will become a
hard error in the near future.
Martin Atkins