We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.
To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.
The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.
This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
Previously this was failing to treat symlinks to directories as unpacked
layout, because our file info was only an Lstat result, not a full Stat.
Now we'll resolve the symlink first, allowing us to handle a symlink to
a directory. That's important because our internal/providercache behavior
is to symlink from one cache to another where possible.
There's a lot going on in these functions that can be hard to follow from
the outside, so we'll add some additional trace logging so that we can
more easily understand why things are behaving the way they are.
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
The MultiSource isn't actually properly implemented yet, but this is a
minimal implementation just for the case where there are no underlying
sources at all, because we use an empty MultiSource as a placeholder
when a test in the "command" package fails to explicitly populate a
ProviderSource.
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.
This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.
So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
The existing functionality in this package deals with finding packages
that are either available for installation or already installed. In order
to support installation we also need to determine the location where a
package should be installed.
This lives in the getproviders package because that way all of the logic
related to the filesystem layout for local provider directories lives
together here where they can be maintained together more easily in future.
Previously this was available by instantiating a throwaway
FilesystemMirrorSource, but that's pretty counter-intuitive for callers
that just want to do a one-off scan without retaining any ongoing state.
Now we expose SearchLocalDirectory as an exported function, and the
FilesystemMirrorSource then uses it as part of its implementation too.
Callers that just want to know what's available in a directory can call
SearchLocalDirectory directly.
This implies some notable changes that will have a visible impact to
end-users of official Terraform releases:
- Terraform is no longer compatible with MacOS 10.10 Yosemite, and
requires at least 10.11 El Capitan. (Relatedly, Go 1.14 is planned to be
the last release to support El Capitan, so while that remains supported
for now, it's notable that Terraform 0.13 is likely to be the last major
release of Terraform supporting it, with 0.14 likely to further require
MacOS 10.12 Sierra.)
- Terraform is no longer compatible with FreeBSD 10.x, which has reached
end-of-life. Terraform now requires FreeBSD 11.2 or later.
- Terraform now supports TLS 1.3 when it makes connections to remote
services such as backends and module registries. Although TLS 1.3 is
backward-compatible in principle, some legacy systems reportedly work
incorrectly when attempting to negotiate it. (This change does not
affect outgoing requests made by provider plugins, though they will see
a similar change in behavior once built with Go 1.13 or later.)
- Ed25519 certificates are now supported for TLS 1.2 and 1.3 connections.
- On UNIX systems where "use-vc" is set in resolv.conf, TCP will now be
used for DNS resolution. This is unlikely to cause issues in practice
because a system set up in this way can presumably already reach its
nameservers over TCP (or else other applications would misbehave), but
could potentially lead to lookup failures in unusual situations where a
system only runs Terraform, has historically had "use-vc" in its
configuration, but yet is blocked from reaching its configured
nameservers over TCP.
- Some parts of Terraform now support Unicode 12.0 when working with
strings. However, notably the Terraform Language itself continues to
use the text segmentation tables from Unicode 9.0, which means it lacks
up-to-date support for recognizing modern emoji combining forms as
single characters. (We may wish to upgrade the text segmentation tables
to Unicode 12.0 tables in a later commit, to restore consistency.)
This also includes some changes to the contents of "vendor", and
particularly to the format of vendor/modules.txt, per the changes to
vendoring in the Go 1.14 toolchain. This new syntax is activated by the
specification of "go 1.14" in the go.mod file.
Finally, the exact format of error messages from the net/http library has
changed since Go 1.12, and so a couple of our tests needed updates to
their expected error messages to match that.
This is a basic implementation of FilesystemMirrorSource for now aimed
only at the specific use-case of scanning the cache of provider plugins
Terraform will keep under the ".terraform" directory, as part of our
interim provider installer implementation for Terraform 0.13.
The full functionality of this will grow out in later work when we
implement explicit local filesystem mirrors, but for now the goal is to
use this just to inspect the work done by the automatic installer once
we switch it to the new provider-FQN-aware directory structure.
The various FIXME comments in this are justified by the limited intended
scope of this initial implementation, and they should be resolved by
later work to use FilesystemMirrorSource explicitly for user-specified
provider package mirrors.
These are utility functions to ease processing of lists of PackageMeta
elsewhere, once we have functionality that works with multiple packages
at once. The local filesystem mirror source will be the first example of
this, so these methods are motivated mainly by its needs.
This is just to have a centralized set of logic for converting from a
platform string (like "linux_amd64") to a Platform object, so we can do
normalization and validation consistently.
Although we tend to return these in contexts where at least one of these
values is implied, being explicit means that PackageMeta values are
self-contained and less reliant on such external context.
This is a temporary helper so that we can potentially ship the new
provider installer without making a breaking change by relying on the
old default namespace lookup API on the default registry to find a proper
FQN for a legacy provider provider address during installation.
If it's given a non-legacy provider address then it just returns the given
address verbatim, so any codepath using it will also correctly handle
explicit full provider addresses. This also means it will automatically
self-disable once we stop using addrs.NewLegacyProvider in the config
loader, because there will therefore no longer be any legacy provider
addresses in the config to resolve. (They'll be "default" provider
addresses instead, assumed to be under registry.terraform.io/hashicorp/* )
It's not decided yet whether we will actually introduce the new provider
in a minor release, but even if we don't this API function will likely be
useful for a hypothetical automatic upgrade tool to introduce explicit
full provider addresses into existing modules that currently rely on
the equivalent to this lookup in the current provider installer.
This is dead code for now, but my intent is that it would either be called
as part of new provider installation to produce an address suitable to
pass to Source.AvailableVersions, or it would be called from the
aforementioned hypothetical upgrade tool.
Whatever happens, these functions can be removed no later than one whole
major release after the new provider installer is introduced, when
everyone's had the opportunity to update their legacy unqualified
addresses.
Our local filesystem mirror mechanism will allow provider packages to be
given either in packed form as an archive directly downloaded to disk or
in an unpacked form where the archive is extracted.
Distinguishing these two cases in the concrete Location types will allow
callers to reliably select the mode chosen by the selected installation
source and handle it appropriately, rather than resorting to out-of-band
heuristics like checking whether the object is a directory or a file.
In a future commit, these implementations of Source will allow finding
and retrieving provider packages via local mirrors, both in the local
filesystem and over the network using an HTTP-based protocol.
This is an API stub for a component that will be added in a future commit
to support considering a number of different installation sources for each
provider. These will eventually be configurable in the CLI configuration,
allowing users to e.g. mirror certain providers within their own
infrastructure while still being able to go upstream for those that aren't
mirrored, or permit locally-mirrored providers only, etc.
Some sources make network requests that are likely to be slow, so this
wrapper type can cache previous responses for its lifetime in order to
speed up repeated requests for the same information.
Registries backed by static files are likely to use relative paths to
their archives for simplicity's sake, but we'll normalize them to be
absolute before returning because the caller wouldn't otherwise know what
to resolve the URLs relative to.
We intend to support installation both directly from origin registries and
from mirrors in the local filesystem or over the network. This Source
interface will serve as our abstraction over those three options, allowing
calling code to treat them all the same.
Our existing provider installer was originally built to work with
releases.hashicorp.com and later retrofitted to talk to the official
Terraform Registry. It also assumes a flat namespace of providers.
We're starting a new one here, copying and adapting code from the old one
as necessary, so that we can build out this new API while retaining all
of the existing functionality and then cut over to this new implementation
in a later step.
Here we're creating a foundational component for the new installer, which
is a mechanism to query for the available versions and download locations
of a particular provider.
Subsequent commits in this package will introduce other Source
implementations for installing from network and filesystem mirrors.
Martin Atkins