Commit Graph

35 Commits

Author SHA1 Message Date
Kristin Laemmert 47e657c611
internal/getproviders: decode and return any registry warnings (#25337)
* internal/getproviders: decode and return any registry warnings

The public registry may include a list of warnings in the "versions"
response for any given provider. This PR adds support for warnings from
the registry and an installer event to return those warnings to the
user.
2020-06-25 10:49:48 -04:00
Martin Atkins 9489672d54 internal/getproviders: Package hashing for local filesystem packages
We previously had this functionality available for cached packages in the
providercache package. This moves the main implementation of this over
to the getproviders package and then implements it also for PackageMeta,
allowing us to compute hashes in a consistent way across both of our
representations of a provider package.

The new methods on PackageMeta will only be effective for packages in the
local filesystem because we need direct access to the contents in order
to produce the hash. Hopefully in future the registry protocol will be
able to also provide hashes using this content-based (rather than
archive-based) algorithm and then we'll be able to make this work for
PackageMeta referring to a package obtained from a registry too, but
hashes for local packages only are still useful for some cases right now,
such as generating mirror directories in the "terraform providers mirror"
command.
2020-06-01 14:49:43 -07:00
Alisdair McDiarmid ef28671b34
Merge pull request #24932 from hashicorp/signing-language
Modify language for reporting signing state
2020-05-28 09:09:34 -04:00
Paddy 5127f1ef8b
command: Unmanaged providers
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.

To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.

As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.

This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.

This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.

Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
2020-05-26 17:48:57 -07:00
Paul Tyng 22ef5cc99c Modify language for reporting signing state
Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.
2020-05-26 13:14:05 -04:00
Kristin Laemmert 0d620018fe
provider cache: log errors and validate dir exists (#24993)
* providercache: add logging for errors from getproviders.SearchLocalDirectory

providercache.fillMetaCache() was silently swallowing errors when
searching the cache directory. This commit logs the error without
changing the behavior otherwise.

* command/cliconfig: validate plugin cache dir exists

The plugin cache directory must exist for terraform to use it, so we
will add a check at the begining.
2020-05-19 15:32:36 -04:00
Alisdair McDiarmid 070c3018f8 internal/providercache: Remove unused retry events 2020-05-13 09:48:41 -04:00
Kristin Laemmert 60321b41e8
getproviders: move protocol compatibility functions into registry client (#24846)
* internal/registry source: return error if requested provider version protocols are not supported

* getproviders: move responsibility for protocol compatibility checks into the registry client

The original implementation had the providercache checking the provider
metadata for protocol compatibility, but this is only relevant for the
registry source so it made more sense to move the logic into
getproviders.

This also addresses an issue where we were pulling the metadata for
every provider version until we found one that was supported. I've
extended the registry client to unmarshal the protocols in
`ProviderVersions` so we can filter through that list, instead of
pulling each version's metadata.
2020-05-11 13:49:12 -04:00
Kristin Laemmert cca0526705
providercache: actually break out of the loop when a matching version is found (#24823) 2020-05-01 08:49:47 -04:00
Kristin Laemmert ce03f1255f
internal/providercache: fix error message for protocol mismatch (#24818)
There was a bug in the installer trying to pass a nil error.
2020-04-30 11:12:04 -04:00
Kristin Laemmert 84e9d86c25
Update installer_test.go
update now-exported function (fix bad PR)
2020-04-23 08:33:28 -04:00
Kristin Laemmert 21b9da5a02
internal/providercache: verify that the provider protocol version is compatible (#24737)
* internal/providercache: verify that the provider protocol version is
compatible

The public registry includes a list of supported provider protocol
versions for each provider version. This change adds verification of
support and adds a specific error message pointing users to the closest
matching version.
2020-04-23 08:21:56 -04:00
Alisdair McDiarmid 54abb87fb4 Fix broken test due to function rename 2020-04-22 10:52:49 -04:00
Kristin Laemmert a43f141f9e
tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629)
* tools/terraform-bundle: refactor to use new provider installer and
provider directory layouts

terraform-bundle now supports a "source" attribute for providers,
uses the new provider installer, and the archive it creates preserves
the new (required) directory hierarchy for providers, under a "plugins"
directory.

This is a breaking change in many ways: source is required for any
non-HashiCorp provider, locally-installed providers must be given a
source (can be arbitrary, see docs) and placed in the expected directory
hierarchy, and the unzipped archive is no longer flat; there is a new
"plugins" directory created with providers in the new directory layout.

This PR also extends the existing test to check the contents of the zip
file.

TODO: Re-enable e2e tests (currently suppressed with a t.Skip)
This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.
2020-04-21 17:09:29 -04:00
Alisdair McDiarmid a5b3d497cc internal: Verify provider signatures on install
Providers installed from the registry are accompanied by a list of
checksums (the "SHA256SUMS" file), which is cryptographically signed to
allow package authentication. The process of verifying this has multiple
steps:

- First we must verify that the SHA256 hash of the package archive
  matches the expected hash. This could be done for local installations
  too, in the future.
- Next we ensure that the expected hash returned as part of the registry
  API response matches an entry in the checksum list.
- Finally we verify the cryptographic signature of the checksum list,
  using the public keys provided by the registry.

Each of these steps is implemented as a separate PackageAuthentication
type. The local archive installation mechanism uses only the archive
checksum authenticator, and the HTTP installation uses all three in the
order given.

The package authentication system now also returns a result value, which
is used by command/init to display the result of the authentication
process.

There are three tiers of signature, each of which is presented
differently to the user:

- Signatures from the embedded HashiCorp public key indicate that the
  provider is officially supported by HashiCorp;
- If the signing key is not from HashiCorp, it may have an associated
  trust signature, which indicates that the provider is from one of
  HashiCorp's trusted partners;
- Otherwise, if the signature is valid, this is a community provider.
2020-04-17 13:57:19 -04:00
Kristin Laemmert f09ae6f862
provider source tests: added test suite to exercise hyphenated providers (#24685) 2020-04-16 15:54:33 -04:00
Martin Atkins 0ad4c1be2f internal/getproviders: Tidy up some confusion about package hashes
Earlier on in the stubbing of this package we realized that it wasn't
going to be possible to populate the authentication-related bits for all
packages because the relevant metadata just isn't available for packages
that are already local.

However, we just moved ahead with that awkward design at the time because
we needed to get other work done, and so we've been mostly producing
PackageMeta values with all-zeros hashes and just ignoring them entirely
as a temporary workaround.

This is a first step towards what is hopefully a more intuitive model:
authentication is an optional thing in a PackageMeta that is currently
populated only for packages coming from a registry.

So far this still just models checking a SHA256 hash, which is not a
sufficient set of checks for a real release but hopefully the "real"
implementation is a natural iteration of this starting point, and if not
then at least this interim step is a bit more honest about the fact that
Authentication will not be populated on every PackageMeta.
2020-04-06 16:31:23 -07:00
Martin Atkins 958ea4f7d1 internal/providercache: Handle built-in providers
Built-in providers are special providers that are distributed as part of
Terraform CLI itself, rather than being installed separately. They always
live in the terraform.io/builtin/... namespace so it's easier to see that
they are special, and currently there is only one built-in provider named
"terraform".

Previous commits established the addressing scheme for built-in providers.
This commit makes the installer aware of them to the extent that it knows
not to try to install them the usual way and it's able to report an error
if the user requests a built-in provider that doesn't exist or tries to
impose a particular version constraint for a built-in provider.

For the moment the tests for this are the ones in the "command" package
because that's where the existing testing infrastructure for this
functionality lives. A later commit should add some more focused unit
tests here in the internal/providercache package, too.
2020-04-06 09:24:23 -07:00
Martin Atkins f35ebe2d65 internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir
Due to some incomplete rework of this function in an earlier commit, the
safety check for using the same directory as both the target and the
cache was inverted and was raising an error _unless_ they matched, rather
than _if_ they matched.

This change is verified by the e2etest TestInitProviders_pluginCache,
which is also updated to use the new-style cache directory layout as part
of this commit.
2020-04-06 09:24:23 -07:00
Martin Atkins ff55e1a1cd internal/providercache: installFromLocalDir
We previously skipped this one because it wasn't strictly necessary for
replicating the old "terraform init" behavior, but we do need it to work
so that things like the -plugin-dir option can behave correctly.

Linking packages from other cache directories and installing from unpacked
directories are fundamentally the same operation because a cache directory
is really just a collection of unpacked packages, so here we refactor
the LinkFromOtherCache functionality to actually be in
installFromLocalDir, and LinkFromOtherCache becomes a wrapper for
the installFromLocalDir function that just calculates the source and
target directories automatically and invalidates the metaCache.
2020-04-06 09:24:23 -07:00
Martin Atkins 2ff4582be2 internal/providercache: Fix positions on selections file/dir
On Unix-derived systems a directory must be marked as "executable" in
order to be accessible, so our previous mode of 0660 here was unsufficient
and would cause a failure if it happened to be the installer that was
creating the plugins directory for the first time here.

Now we'll make it executable and readable for all but only writable by
the same user/group. For consistency, we also make the selections file
itself readable by everyone. In both cases, the umask we are run with may
further constrain these modes.
2020-04-06 09:24:23 -07:00
Martin Atkins ae080481c0 internal/providercache: Installer records its selections in a file
Just as with the old installer mechanism, our goal is that explicit
provider installation is the only way that new provider versions can be
selected.

To achieve that, we conclude each call to EnsureProviderVersions by
writing a selections lock file into the target directory. A later caller
can then recall the selections from that file by calling SelectedPackages,
which both ensures that it selects the same set of versions and also
verifies that the checksums recorded by the installer still match.

This new selections.json file has a different layout than our old
plugins.json lock file. Not only does it use a different hashing algorithm
than before, we also record explicitly which version of each provider
was selected. In the old model, we'd repeat normal discovery when
reloading the lock file and then fail with a confusing error message if
discovery happened to select a different version, but now we'll be able
to distinguish between a package that's gone missing since installation
(which could previously have then selected a different available version)
from a package that has been modified.
2020-04-06 09:24:23 -07:00
Martin Atkins f6a7a4868b internal/providercache: Hashing of contents of cached packages
For the old-style provider cache directory model we hashed the individual
executable file for each provider. That's no longer appropriate because
we're giving each provider package a whole directory to itself where it
can potentially have many files.

This therefore introduces a new directory-oriented hashing algorithm, and
it's just using the Go Modules directory hashing algorithm directly
because that's already had its cross-platform quirks and other wrinkles
addressed during the Go Modules release process, and is now used
prolifically enough in Go codebases that breaking changes to the upstream
algorithm would be very expensive to the Go ecosystem.

This is also a bit of forward planning, anticipating that later we'll use
hashes in a top-level lock file intended to be checked in to user version
control, and then use those hashes also to verify packages _during_
installation, where we'd need to be able to hash unpacked zip files. The
Go Modules hashing algorithm is already implemented to consistently hash
both a zip file and an unpacked version of that zip file.
2020-04-06 09:24:23 -07:00
Martin Atkins 079b4cf7be internal/providercache: Clear the metadata cache during package install
This was previously happening during linking from another cache, but not
when installing an entirely new provider.
2020-04-06 09:24:23 -07:00
Martin Atkins 4061cbed38 internal/getproviders: A new shared model for provider requirements
We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.

To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.

The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.

This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
2020-03-27 09:01:32 -07:00
Martin Atkins 537c1bedcf internal/providercache: LinkFromOtherCache removes target, not source
This was incorrectly removing the _source_ entry prior to creating the
symlink, therefore ending up with a dangling symlink and no source file.

This wasn't obvious before because the test case for LinkFromOtherCache
was also incorrectly named and therefore wasn't running. Fixing the name
of that test made this problem apparent.

The TestLinkFromOtherCache test case now ends up seeing the final resolved
directory rather than the symlink target, because of upstream changes
to the internal/getproviders filesystem scanning logic to handle symlinks
properly.
2020-03-25 13:50:00 -07:00
Martin Atkins ad15459468 internal/{getproviders,providercache}: improved trace logging
There's a lot going on in these functions that can be hard to follow from
the outside, so we'll add some additional trace logging so that we can
more easily understand why things are behaving the way they are.
2020-03-25 13:50:00 -07:00
Martin Atkins 391ca0c991 internal/providercache: add windows test fixture
This was accidentally left out of an earlier commit due to our top-level
.gitignore file containing *.exe as an ignore pattern.
2020-03-25 13:50:00 -07:00
Martin Atkins 807267d1b5 internal/providercache: Installation from HTTP URLs and local archives
When a provider source produces an HTTP URL location we'll expect it to
resolve to a zip file, which we'll first download to a temporary
directory and then treat it like a local archive.

When a provider source produces a local archive path we'll expect it to
be a zip file and extract it into the target directory.

This does not yet include an implementation of installing from an
already-unpacked local directory. That will follow in a subsequent commit,
likely following a similar principle as in Dir.LinkFromOtherCache.
2020-03-25 11:29:48 -07:00
Martin Atkins 754b7ebb65 command: Expose providercache package objects for use elsewhere
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
2020-03-25 11:29:48 -07:00
Martin Atkins 18dd0a396d internal/providercache: First pass of the actual install process
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.

This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
2020-03-25 11:29:48 -07:00
Martin Atkins 03155daf98 internal/providercache: Start to stub Installer type
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.

So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
2020-03-25 11:29:48 -07:00
Martin Atkins e4d14324e0 internal/providercache: Stub of Dir.InstallPackage method
This will eventually be responsible for actually retrieving a package from
a source and then installing it into the cache directory, but for the
moment it's just a stub to complete the proposed API, which I intend to
test in a subsequent commit by writing the full "Installer" API that will
encapsulate the full installation logic.
2020-03-25 11:29:48 -07:00
Martin Atkins 67ca067910 internal/providercache: Linking from one cache to another
When a system-wide shared plugin cache is configured, we'll want to make
use of entries already in the shared cache when populating a local
(configuration-specific) cache.

This new method LinkFromOtherCache encapsulates the work of placing a link
from one cache to another. If possible it will create a symlink, therefore
retaining a key advantage of configuring a shared plugin cache, but
otherwise we'll do a deep copy of the package directory from one cache
to the other.

Our old provider installer would always skip trying to create symlinks on
Windows because Go standard library support for os.Symlink on Windows
was inconsistent in older versions. However, os.Symlink can now create
symlinks using a new API introduced in a Windows 10 update and cleanly
fail if symlink creation is impossible, so it's safe for us to just
try to create the symlink and react if that produces an error, just as we
used to do on non-Windows systems when possibly creating symlinks on
filesystems that cannot support them.
2020-03-25 11:29:48 -07:00
Martin Atkins d13001830b providercache: A package to encapsulate management of provider cache dirs
Historically our logic to handle discovering and installing providers has
been spread across several different packages. This package is intended
to become the home of all logic related to what is now called "provider
cache directories", which means directories on local disk where Terraform
caches providers in a form that is ready to run.

That includes both logic related to interrogating items already in a cache
(included in this commit) and logic related to inserting new items into
the cache from upstream provider sources (to follow in later commits).

These new codepaths are focused on providers and do not include other
plugin types (provisioners and credentials helpers), because providers are
the only plugin type that is represented by a heirarchical, decentralized
namespace and the only plugin type that has an auto-installation protocol
defined. The existing codepaths will remain to support the handling of
the other plugin types that require manual installation and that use only
a flat, locally-defined namespace.
2020-03-25 11:29:48 -07:00