Earlier on in the stubbing of this package we realized that it wasn't
going to be possible to populate the authentication-related bits for all
packages because the relevant metadata just isn't available for packages
that are already local.
However, we just moved ahead with that awkward design at the time because
we needed to get other work done, and so we've been mostly producing
PackageMeta values with all-zeros hashes and just ignoring them entirely
as a temporary workaround.
This is a first step towards what is hopefully a more intuitive model:
authentication is an optional thing in a PackageMeta that is currently
populated only for packages coming from a registry.
So far this still just models checking a SHA256 hash, which is not a
sufficient set of checks for a real release but hopefully the "real"
implementation is a natural iteration of this starting point, and if not
then at least this interim step is a bit more honest about the fact that
Authentication will not be populated on every PackageMeta.
This merge introduces various work across the whole codebase to prepare
codepaths to deal with the new decentralized provider naming scheme and
to use the new provider installation codepaths that are aware of the
new scheme.
The incoming branch of commits here (the second commit in the merge)
contains a period where the tests were not passing as a tradeoff to keep
the individual changes separated while accepting that that makes that
part of the history unsuitable for "git bisect" usage. If you are using
git bisect on this portion of the history, exclude the commits from
the incoming branch.
These will now use "default" provider addresses, rather than "legacy"
ones, so that they can cooperate with the rest of Terraform that has been
updated to no longer use legacy provider addresses.
With provider dependencies now appearing inside a nested block, it seems
likely that configuration examples showing dependencies out of context
will sometimes mislead users into thinking that required_providers is
toplevel.
To give better feedback in that situation, we'll produce a specialized
error in that case hinting the correct structure to the user.
When a provider dependency is implicit rather than explicit, or otherwise
when version constraints are lacking, we produce a warning recommending
the addition of explicit version constraints in the configuration.
This restores the warning functionality from previous Terraform versions,
adapting it slightly to account for the new provider FQN syntax and to
recommend using a required_providers block rather than version constraints
in "provider" blocks, because the latter is no longer recommended in the
documentation.
* helper/resource: remove provider resolver test
* repl tests passing
* helper/resource: add some extra shimming to ShimLegacyState
Some of the tests in helper/resource have to shim between legacy and
modern states. Terraform expects modern states to have the Provider set
for each resource (and not be a legacy provider). This PR adds a wrapper
around terraform.ShimLegacyState which iterates through the resources
in the state (after the shim) and adds the provider FQN.
The fake installable package meta used a ZIP archive which gave
different checksums between macOS and Linux targets. This commit removes
the target from the contents of this archive, and updates the golden
hash value in the test to match. This test should now pass on both
platforms.
The provider fully-qualified name string used in configuration is very
long, and since most providers are hosted in the public registry, most
of that length is redundant. This commit adds and uses a `ForDisplay`
method, which simplifies the presentation of provider FQNs.
If the hostname is the default hostname, we now display only the
namespace and type. This is only used in UI, but should still be
unambiguous, as it matches the FQN string parsing behaviour.
This restores some of the local search directories we used to include when
searching for provider plugins in Terraform 0.12 and earlier. The
directory structures we are expecting in these are different than before,
so existing directory contents will not be compatible without
restructuring, but we need to retain support for these local directories
so that users can continue to sideload third-party provider plugins until
the explicit, first-class provider mirrors configuration (in CLI config)
is implemented, at which point users will be able to override these to
whatever directories they want.
This also includes some new search directories that are specific to the
operating system where Terraform is running, following the documented
layout conventions of that platform. In particular, this follows the
XDG Base Directory specification on Unix systems, which has been a
somewhat-common request to better support "sideloading" of packages via
standard Linux distribution package managers and other similar mechanisms.
While it isn't strictly necessary to add that now, it seems ideal to do
all of the changes to our search directory layout at once so that our
documentation about this can cleanly distinguish "0.12 and earlier" vs.
"0.13 and later", rather than having to document a complex sequence of
smaller changes.
Because this behavior is a result of the integration of package main with
package command, this behavior is verified using an e2etest rather than
a unit test. That test, TestInitProvidersVendored, is also fixed here to
create a suitable directory structure for the platform where the test is
being run. This fixes TestInitProvidersVendored.
This library implements the user-specific directory layout specifications
for various platforms (XDG on Unix, "Known Folders" on Windows, etc).
We'll use this in a subsequent commit to add additional system-specific
search directories for provider plugins, and perhaps later on also
CLI configuration directories.
There was a remaining TODO in this package to find the true provider FQN
when looking up the schema for a resource type. We now have that data
available in the Provider field of configs.Resource, so we can now
complete that change.
The tests for this functionality actually live in the parent "command"
package as part of the tests for the "terraform show" command, so this
fix is verified by all of the TestShow... tests now passing except one,
and that remaining one is failing for some other reason which we'll
address in a later commit.
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".
Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.
For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
This encapsulates the logic for selecting an implied FQN for an
unqualified type name, which could either come from a local name used in
a module without specifying an explicit source for it or from the prefix
of a resource type on a resource that doesn't explicitly set "provider".
This replaces the previous behavior of just directly calling
NewDefaultProvider everywhere so that we can use a different implication
for the local name "terraform", to refer to the built-in terraform
provider rather than the stale one that's on registry.terraform.io for
compatibility with other Terraform versions.
* command: refactor testBackendState to write states.State
testBackendState was using the older terraform.State format, which is no
longer sufficient for most tests since the state upgrader does not
encode provider FQNs automatically. Users will run `terraform
0.13upgrade` to update their state to include provider FQNs in
resources, but tests need to use the modern state format instead of
relying on the automatic upgrade.
* plan tests passing
* graph tests passing
* json packages test update
* command test updates
* update show test fixtures
* state show tests passing
In the new design the ProviderSource is decided by package main, not by
the "command" package, and so making sure the vendor directory is included
is the responsibility of that package instead. Therefore we can no longer
test this at the "command" package level, but we'll retain a test for it
in e2etests to record that it isn't currently working, so that we have
a prompt to fix it before releasing.
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.
This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
These tests make assertions against specific user-oriented output from the
"terraform init" command, but we've intentionally changed some of these
messages as part of introducing support for the decentralized provider
namespace.
Both of these are attempting to test -plugin-dir, which means we need some
additional help to populate some suitable directories for -plugin-dir to
refer to. The new installFakeProviderPackagesElsewhere helper generalizes
the earlier installFakeProviderPackages to allow installing fake provider
packages to an arbitrary other directory.
We previously skipped this one because it wasn't strictly necessary for
replicating the old "terraform init" behavior, but we do need it to work
so that things like the -plugin-dir option can behave correctly.
Linking packages from other cache directories and installing from unpacked
directories are fundamentally the same operation because a cache directory
is really just a collection of unpacked packages, so here we refactor
the LinkFromOtherCache functionality to actually be in
installFromLocalDir, and LinkFromOtherCache becomes a wrapper for
the installFromLocalDir function that just calculates the source and
target directories automatically and invalidates the metaCache.
This test is focused on making sure that the required_providers syntax
is working, so the rewritten version does not include any special handling
of pre-installed packages or "vendored" packages. Pre-installed plugins
are tested in other tests such as TestInit_getUpgradePlugins.
This test now requires a bit of a different approach because it was
previously directly constructing a cache directory but we now use a
different directory layout.
Rather than manually constructing the new heirarchical directory layout
(which would've required a lot more inline code), this introduces a helper
function installFakeProviderPackages that installs a fake provider package
directly into the local cache directory associated with a Meta object,
with the correct directory layout.
This is a slightly different approach than we used to take for this
option: rather than disabling the installer and causing all future
commands to look elsewhere for plugins, we'll now leave the installer
enabled by constrain it to only look at the given directories.
This is overall simpler because it doesn't require any special tracking
of the plugin directories for subsequent commands. Instead, the selections
file generated by the installer will record the versions it selected from
the specified directories, and we'll link them in to the local cache just
as we would normally so that other commands don't need to do anything
special to select the right plugins in either case.
We previously had only a stub implementation for a totally-empty
MultiSource. Here we have an initial implementation of the full
functionality, which we'll need to support "terraform init -plugin-dir=..."
in a subsequent commit.
On Unix-derived systems a directory must be marked as "executable" in
order to be accessible, so our previous mode of 0660 here was unsufficient
and would cause a failure if it happened to be the installer that was
creating the plugins directory for the first time here.
Now we'll make it executable and readable for all but only writable by
the same user/group. For consistency, we also make the selections file
itself readable by everyone. In both cases, the umask we are run with may
further constrain these modes.
They still aren't passing, but this is just enough updating to make the
test program compile successfully after the refactoring related to
provider installation. They are now using the mock provider source offered
by the getproviders package, which is similar but not totally identical
to the idea of mocking the entire installer as these tests used to do, and
so many of them need further adjustment to still be testing what they
intended to test under this new architecture.
Subsequent commits will gradually repair the failing tests.
These are some helpers to support unit testing in other packages, allowing
callers to exercise provider installation mechanisms without hitting any
real upstream source or having to prepare local package directories.
MockSource is a Source implementation that just scans over a provided
static list of packages and returns whatever matches.
FakePackageMeta is a shorthand for concisely constructing a
realistic-looking but uninstallable PackageMeta, probably for use with
MockSource.
FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to
the trouble of creating a real temporary archive on local disk so that
the resulting package meta is pointing to something real on disk. This
makes the result more useful to the caller, but in return they get the
responsibility to clean up the temporary file once the test is over.
Nothing is using these yet.
* terraform: add helper functions for creating test state
testSetResourceInstanceCurrent and testSetResourceInstanceTainted are
wrapper functions around states.Module.SetResourceInstanceCurrent()
used to set a resource in state. They work with current, non-deposed
resources with no dependencies.
testSetResourceInstanceDeposed can be used to set a desosed resource in state.
* terraform: update all tests to use modern providers and state
This commit reverts an earlier change which automatically converted
provider strings to legacy provider FQNs. It has become apparent that a
state upgrade step will be required before upgrading to v0.13.
These are cases where we were using the legacy string only to produce a
message to the user or to write to the log. It's enough to make some
basic Terraform commands like "terraform validate" not panic and get far
enough along to see that provider startup is working.
Back when we first introduced provider versioning in Terraform 0.10, we
did the provider version resolution in terraform.NewContext because we
weren't sure yet how exactly our versioning model was going to play out
(whether different versions could be selected per provider configuration,
for example) and because we were building around the limitations of our
existing filesystem-based plugin discovery model.
However, the new installer codepath is new able to do all of the
selections up front during installation, so we don't need such a heavy
inversion of control abstraction to get this done: the command package can
select the exact provider versions and pass their factories directly
to terraform.NewContext as a simple static map.
The result of this commit is that CLI commands other than "init" are now
able to consume the local cache directory and selections produced by the
installation process in "terraform init", passing all of the selected
providers down to the terraform.NewContext function for use in
implementing the main operations.
This commit is just enough to get the providers passing into the
terraform.Context. There's still plenty more to do here, including to
repair all of the tests this change has additionally broken.
The introduction of a heirarchical addressing scheme for providers gives
us an opportunity to make more explicit the special case of "built-in"
providers.
Thus far we've just had a special case in the "command" package that the
provider named "terraform" is handled differently than all others, though
there's nothing especially obvious about that in the UI.
Moving forward we'll put such "built-in" providers under the special
namespace terraform.io/builtin/terraform, which will be visible in the UI
as being different than the other providers and we can use the namespace
itself (rather than a particular name) as the trigger for our special-case
behaviors around built-in plugins.
We have no plans to introduce any built-in providers other than
"terraform" in the foreseeable future, so any others will produce an
error.
This commit just establishes the addressing convention, without making use
of it anywhere yet. Subsequent commits will make the provider installer
and resolver codepaths aware of it, replacing existing checks for the
provider just being called "terraform".
Just as with the old installer mechanism, our goal is that explicit
provider installation is the only way that new provider versions can be
selected.
To achieve that, we conclude each call to EnsureProviderVersions by
writing a selections lock file into the target directory. A later caller
can then recall the selections from that file by calling SelectedPackages,
which both ensures that it selects the same set of versions and also
verifies that the checksums recorded by the installer still match.
This new selections.json file has a different layout than our old
plugins.json lock file. Not only does it use a different hashing algorithm
than before, we also record explicitly which version of each provider
was selected. In the old model, we'd repeat normal discovery when
reloading the lock file and then fail with a confusing error message if
discovery happened to select a different version, but now we'll be able
to distinguish between a package that's gone missing since installation
(which could previously have then selected a different available version)
from a package that has been modified.
For the old-style provider cache directory model we hashed the individual
executable file for each provider. That's no longer appropriate because
we're giving each provider package a whole directory to itself where it
can potentially have many files.
This therefore introduces a new directory-oriented hashing algorithm, and
it's just using the Go Modules directory hashing algorithm directly
because that's already had its cross-platform quirks and other wrinkles
addressed during the Go Modules release process, and is now used
prolifically enough in Go codebases that breaking changes to the upstream
algorithm would be very expensive to the Go ecosystem.
This is also a bit of forward planning, anticipating that later we'll use
hashes in a top-level lock file intended to be checked in to user version
control, and then use those hashes also to verify packages _during_
installation, where we'd need to be able to hash unpacked zip files. The
Go Modules hashing algorithm is already implemented to consistently hash
both a zip file and an unpacked version of that zip file.
Chris Griggs