We've been using the models from the "moduledeps" package to represent our
provider dependencies everywhere since the idea of provider dependencies
was introduced in Terraform 0.10, but that model is not convenient to use
for any use-case other than the "terraform providers" command that needs
individual-module-level detail.
To make things easier for new codepaths working with the new-style
provider installer, here we introduce a new model type
getproviders.Requirements which is based on the type the new installer was
already taking as its input. We have new methods in the states, configs,
and earlyconfig packages to produce values of this type, and a helper
to merge Requirements together so we can combine config-derived and
state-derived requirements together during installation.
The advantage of this new model over the moduledeps one is that all of
recursive module walking is done up front and we produce a simple, flat
structure that is more convenient for the main use-cases of selecting
providers for installation and then finding providers in the local cache
to use them for other operations.
This new model is _not_ suitable for implementing "terraform providers"
because it does not retain module-specific requirement details. Therefore
we will likely keep using moduledeps for "terraform providers" for now,
and then possibly at a later time consider specializing the moduledeps
logic for only what "terraform providers" needs, because it seems to be
the only use-case that needs to retain that level of detail.
This was incorrectly removing the _source_ entry prior to creating the
symlink, therefore ending up with a dangling symlink and no source file.
This wasn't obvious before because the test case for LinkFromOtherCache
was also incorrectly named and therefore wasn't running. Fixing the name
of that test made this problem apparent.
The TestLinkFromOtherCache test case now ends up seeing the final resolved
directory rather than the symlink target, because of upstream changes
to the internal/getproviders filesystem scanning logic to handle symlinks
properly.
There's a lot going on in these functions that can be hard to follow from
the outside, so we'll add some additional trace logging so that we can
more easily understand why things are behaving the way they are.
When a provider source produces an HTTP URL location we'll expect it to
resolve to a zip file, which we'll first download to a temporary
directory and then treat it like a local archive.
When a provider source produces a local archive path we'll expect it to
be a zip file and extract it into the target directory.
This does not yet include an implementation of installing from an
already-unpacked local directory. That will follow in a subsequent commit,
likely following a similar principle as in Dir.LinkFromOtherCache.
These new functions allow command implementations to get hold of the
providercache objects and installation source object derived from the
current CLI configuration.
This is not tested yet, but it's a compilable strawman implementation of
the necessary sequence of events to coordinate all of the moving parts
of running a provider installation operation.
This will inevitably see more iteration in later commits as we complete
the surrounding parts and wire it up to be used by "terraform init". So
far, it's just dead code not called by any other package.
The Installer type will encapsulate the logic for running an entire
provider installation request: given a set of providers to install, it
will determine a method to obtain each of them (or detect that they are
already installed) and then take the necessary actions.
So far it doesn't do anything, but this stubs out an interface by which
the caller can request ongoing notifications during an installation
operation.
This will eventually be responsible for actually retrieving a package from
a source and then installing it into the cache directory, but for the
moment it's just a stub to complete the proposed API, which I intend to
test in a subsequent commit by writing the full "Installer" API that will
encapsulate the full installation logic.
When a system-wide shared plugin cache is configured, we'll want to make
use of entries already in the shared cache when populating a local
(configuration-specific) cache.
This new method LinkFromOtherCache encapsulates the work of placing a link
from one cache to another. If possible it will create a symlink, therefore
retaining a key advantage of configuring a shared plugin cache, but
otherwise we'll do a deep copy of the package directory from one cache
to the other.
Our old provider installer would always skip trying to create symlinks on
Windows because Go standard library support for os.Symlink on Windows
was inconsistent in older versions. However, os.Symlink can now create
symlinks using a new API introduced in a Windows 10 update and cleanly
fail if symlink creation is impossible, so it's safe for us to just
try to create the symlink and react if that produces an error, just as we
used to do on non-Windows systems when possibly creating symlinks on
filesystems that cannot support them.
Historically our logic to handle discovering and installing providers has
been spread across several different packages. This package is intended
to become the home of all logic related to what is now called "provider
cache directories", which means directories on local disk where Terraform
caches providers in a form that is ready to run.
That includes both logic related to interrogating items already in a cache
(included in this commit) and logic related to inserting new items into
the cache from upstream provider sources (to follow in later commits).
These new codepaths are focused on providers and do not include other
plugin types (provisioners and credentials helpers), because providers are
the only plugin type that is represented by a heirarchical, decentralized
namespace and the only plugin type that has an auto-installation protocol
defined. The existing codepaths will remain to support the handling of
the other plugin types that require manual installation and that use only
a flat, locally-defined namespace.
Martin Atkins