Any state modifying functions can only be run once during the plan-apply
cycle. When regenerating the Diff during ApplyResourceChange, strip out
all StateFunc and CustomizeDiff functions from the schema.
Thew NewExtra diff field was where config data that was modified by a
StateFunc was stored, and needs to be maintained between plan and apply.
During PlanResourceChange, store any NewExtra data from the Diff in the
PlannedPrivate data, and re-insert the NewExtra data into the Diff
generated during ApplyResourceChange.
The rest of Terraform is still using uint64 for this in various spots, but
we'll update that gradually later. We use int64 here because that matches
what's used in our protobuf definition, and unsigned integers are not
portable across all of the protobuf target languages anyway.
When verifying the signature of the SHA256SUMS file, we have been
hardcoding HashiCorp's public GPG key and using it as the keyring.
Going forward, Terraform will get a list of valid public keys for a
provider from the Terraform Registry (registry.terraform.io), and use
them as the keyring for the openpgp verification func.
The main significant change here is that the package name for the proto
definition is "tfplugin5", which is important because this name is part
of the wire protocol for references to types defined in our package.
Along with that, we also move the generated package into "internal" to
make it explicit that importing the generated Go package from elsewhere is
not the right approach for externally-implemented SDKs, which should
instead vendor the proto definition they are using and generate their
own stubs to ensure that the wire protocol is the only hard dependency
between Terraform Core and plugins.
After this is merged, any provider binaries built against our
helper/schema package will need to be rebuilt so that they use the new
"tfplugin5" package name instead of "proto".
In a future commit we will include more elaborate and organized
documentation on how an external codebase might make use of our RPC
interface definition to implement an SDK, but the primary concern here
is to ensure we have the right wire package name before release.
Since protoc is not go-gettable, and most development tasks in Terraform
won't involve recompiling protoc files anyway, we'll use a separate
mechanism for these.
This way "go generate" only depends on things we can "go get" in the
"make tools" target.
In a later commit we should also in some way specify a particular version
of protoc to use so that we don't get "flapping" regenerations as
developers work with different versions, but the priority here is just to
make "make generate" minimally usable again to restore the dev workflow
documented in the README.
This also includes some updates that resulted from running "make generate"
and "make protobuf" after those Makefile changes were in place.
Even if a provider doesn't indicate a specific attribute as the cause of
a resource operation error, we know the error relates to some aspect of
the resource, so we'll include that approximate information in the result
so that we don't produce user-hostile error messages with no context
whatsoever.
Later we can hopefully refine this to place the source range on the header
of the configuration block rather than on an empty part of the body, but
that'll require some more complex rework here and so for now we'll just
accept this as an interim state so that the user can at least figure out
which resource block the error is coming from.
An earlier change made the tests not compile here. We now need to use the
legacyPluginMap function, since pluginMap has now been replaced with
helpers to produce new-style plugin _sets_.
Previously we just left these out of the plan altogether, but in the new
plan types we intentionally include change information for every resource
instance, even if no changes are actually planned, to allow alternative
plan file viewers to show what isn't changing as well as what is.
Previously we would construct both provisioner and the provider objects if
either callback was set, but this is incorrect because a plugin should
actually set only one of these at a time, depending on what kind of plugin
it is.
Managing which function need to be shared between the terraform plugin
and the helper plugin without creating cycles was becoming difficult.
Move all functions related to converting between terraform and proto
type into plugin/convert.
Here we add the GRPCProvisioner and GRPCProvider which implement the
core provisioners.Interface and providers.Interface, and translate
betweeen the core types and the grpc protocol.
These will allow easier testing of the grpc endpoints in isolation.
Mocks are generated for ProviderClient, ProvisionerClient,
Provisioner_ProvisionResourceClient, and
Provisioner_ProvisionResourceServer using `go generate`
support the requested platform.
If the newest version of a provider which matches the version
constraints does not support the requested platform, filter the list of
available versions by platform support and try again.
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.
This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
While the TLS handshakes are a fairly small overhead compared to
downloading the providers, clients in some situation are failing to
complete the TLS handshake in a timely manner. It's unclear if this is
because of heavily constrained clients are stalling while doing the
major crpto operations, or the edge servers are throttling repeated
requests from the same IPs.
This should allow reusing the open TLS connection to the release edge
servers during init.
In order to parse provider, resource and data source configuration from
HCL2 config files, we need to know the relevant configuration schema.
This new method allows Terraform Core to request these from a provider.
This is a breaking change to this interface, so all of its implementers
in this package are updated too. This includes concrete implementations
of the new method in helper/schema that use the schema conversion code
added in an earlier commit to produce a configschema.Block automatically.
Plugins compiled against prior versions of helper/schema will not have
support for this method, and so calls to them will fail. Callers of
this new method will therefore need to sniff for support using the
SchemaAvailable field added to both ResourceType and DataSource.
This careful handling will need to persist until next time we increment
the plugin protocol version, at which point we can make the breaking
change of requiring this information to be available.
For users that have metered or slow internet connections it is annoying
to have Terraform constantly re-downloading the same files when they
initialize many separate directories.
To help such users, here we add an opt-in mechanism to use a local
directory as a read-through cache. When enabled, any plugin download will
be skipped if a suitable file already exists in the cache directory. If
the desired plugin isn't in the cache, it will be downloaded into the
cache for use next time.
This mechanism also serves to reduce total disk usage by allowing
plugin files to be shared between many configurations, as long as the
target system isn't Windows and supports either hardlinks or symlinks.
If we encounter something that isn't a file -- for example, a dangling
symlink whose referent has been deleted -- we'll ignore it so that we
can either later produce a "no such plugin" error or auto-install a plugin
that will actually work.
The go-plugin package now uses hclog. The default Logger has a level set
to Info, but all plugin output is relayed via Debug. Create a new named
logger for plugins with the level set to Trace so that all output comes
through.
On Windows systems the plugin binaries use a .exe suffix, which we were
misparsing as part of either the "v" or "x" parts of the filename.
This fixes#15578.
If the release site is missing the "x-terraform-protocol-version"
header, we should fetch the latest spec'ed release. Downloading the
wrong protocol version can't do any damage, and the version present is
more than likely compatible.
Previously we forced only installing for the current GOOS and GOARCH. Now
we allow this to be optionally overridden, which allows building tools
that can, for example, populate a directory with plugins to run on a Linux
server while working on a Mac.
Some errors from Get are essentially user error, so we want to be able to
recognize them and give the user good feedback on how to proceed.
Although sentinel values are not an ideal solution to this, it's something
reasonably simple we can do to get this done without lots of refactoring.
Fetch the SHA256SUMS file and verify it's signature before downloading
any plugins.
This embeds the hashicorp public key in the binary. If the publickey is
replaced, new releases will need to be cut anyway. A
--verify-plugin=false flag will be added to skip signature verification
in these cases.
Since the command package also needs to know about the specific OS_ARCH
directories, remove the logic fom the discovery package.
This doesn't completely remove the knowledge of the path from discovery,
in order to maintain the current behavior of skipping legacy plugin
names within a new-style path.
Previously we had a "getProvider" function type used to implement plugin
fetching. Here we replace that with an interface type, initially with
just a "Get" function.
For now this just simplifies the interface by allowing the target
directory and protocol version to be members of the struct rather than
passed as arguments.
A later change will extend this interface to also include a method to
purge unused plugins, so that upgrading frequently doesn't leave behind
a trail of unused executable files.
When running "terraform init" with providers that are unconstrained, we
will now produce information to help the user update configuration to
constrain for the particular providers that were chosen, to prevent
inadvertently drifting onto a newer major release that might contain
breaking changes.
A ~> constraint is used here because pinning to a single specific version
is expected to create dependency hell when using child modules. By using
this constraint mode, which allows minor version upgrades, we avoid the
need for users to constantly adjust version constraints across many
modules, but make major version upgrades still be opt-in.
Any constraint at all in the configuration will prevent the display of
these suggestions, so users are free to use stronger or weaker constraints
if desired, ignoring the recommendation.
We can filter the allowed versions and sort them before checking the
protocol version, that way we can just return the first one found
reducing network requests.
Extend the test reslease server to return the protocol version header
and a dummy zip file for the provider.
Test filtering the plugins by plugin protocol version and add a full
GetProvder test.
Get provider needs to be provided with the plugin protocol version,
because it can't be imported here directly.
The plugin url types and methods were confusing; replace them with a few
functions to format the urls.
This is used to mark the plugin protocol version. Currently we actually
just ignore this entirely, since only one protocol version exists anyway.
Later we will need to add checks here to ensure that we only pay attention
to plugins of the right version.
As well as constraining plugins by version number, we also want to be
able to pin plugins to use specific executables so that we can detect
drift in available plugins between commands.
This commit allows such requirements to be specified, but doesn't yet
specify any such requirements, nor validate them.
Add discovery.GetProviders to fetch plugins from the relases site.
This is an early version, with no tests, that only (probably) fetches
plugins from the default location. The URLs are still subject to change,
and since there are no plugin releases, it doesn't work at all yet.
Instead of providing the a path in BackendOpts, provide a loaded
*config.Config instead. This reduces the number of places where
configuration is loaded.
The semver library we were using doesn't have support for a "pessimistic
constraint" where e.g. the user wants to accept only minor or patch
version upgrades. This is important for providers since users will
generally want to pin their dependencies to not inadvertantly accept
breaking changes.
So here we switch to hashicorp's home-grown go-version library, which
has the ~> constraint operator for this sort of constraint.
Given how much the old version object was already intruding into the
interface and creating dependency noise in callers, this also now wraps
the "raw" go-version objects in package-local structs, thus keeping the
details encapsulated and allowing callers to deal just with this package's
own types.
Having this as a method of PluginMeta felt most natural, but unfortunately
that means that discovery must depend on plugin and plugin in turn
depends on core Terraform, thus making the discovery package hard to use
without creating dependency cycles.
To resolve this, we invert the dependency and make the plugin package be
responsible for instantiating clients given a meta, using a top-level
function.
The .terraformrc file allows the user to override the executable location
for certain plugins. This mechanism allows us to retain that behavior for
a deprecation period by treating such executables as an unversioned
plugin for the given name and excluding all other candidates for that
name, thus ensuring that the override will "win".
Users must eventually transition away from using this mechanism and use
vendor directories instead, because these unversioned overrides will never
match for a provider referenced with non-zero version constraints.
These new methods ClientConfig and Client provide the bridge into the
main plugin infrastructure by configuring and instantiating (respectively)
a client object for the referenced plugin.
This stops short of getting the proxy object from the client since that
then requires referencing the interface for the plugin kind, which would
then create a dependency on the main terraform package which we'd rather
avoid here. It'll be the responsibility of the caller in the command
package to do the final wiring to get a provider instance out of a
provider plugin client.
For now this supports both our old and new directory layouts, so we can
preserve compatibility with existing configurations until a future major
release where support for the old paths will be removed.
Currently this allows both styles in all given directories, which means we
support more variants than we actually intend to but this is accepted to
keep the interface simple such that we can easily remove the legacy
exception later. The documentation will reflect only the subset of
path layouts that we actually intend to support.
With forthcoming support for versioned plugins we need to be able to
answer questions like what versions of plugins are currently installed,
what's the newest version of a given plugin available, etc.
PluginMetaSet gives us a building block for this sort of plugin version
wrangling.
Martin Atkins