Commit Graph

274 Commits

Author SHA1 Message Date
Martin Atkins 51b0aee36c addrs: ModuleRegistryPackage for representing module registry packages
Previously we had a separation between ModuleSourceRemote and
ModulePackage as a way to represent within the type system that there's an
important difference between a module source address and a package address,
because module packages often contain multiple modules and so a
ModuleSourceRemote combines a ModulePackage with a subdirectory to
represent one specific module.

This commit applies that same strategy to ModuleSourceRegistry, creating
a new type ModuleRegistryPackage to represent the different sort of
package that we use for registry modules. Again, the main goal here is
to try to reflect the conceptual modelling more directly in the type
system so that we can more easily verify that uses of these different
address types are correct.

To make use of that, I've also lightly reworked initwd's module installer
to use addrs.ModuleRegistryPackage directly, instead of a string
representation thereof. This was in response to some earlier commits where
I found myself accidentally mixing up package addresses and source
addresses in the installRegistryModule method; with this new organization
those bugs would've been caught at compile time, rather than only at
unit and integration testing time.

While in the area anyway, I also took this opportunity to fix some
historical confusing names of fields in initwd.ModuleInstaller, to be
clearer that they are only for registry packages and not for all module
source address types.
2021-06-03 08:50:34 -07:00
Martin Atkins 7b2a0284e0 initwd: Fix registry acceptance tests for upstream registry changes
We have some tests in this package that install real modules from the real
registry at registry.terraform.io. Those tests were written at an earlier
time when the registry's behavior was to return the URL of a .tar.gz
archive generated automatically by GitHub, which included an extra level
of subdirectory that would then be reflected in the paths to the local
copies of these modules.

GitHub started rate limiting those tar archives in a way that Terraform's
module installer couldn't authenticate to, and so the registry switched
to returning direct git repository URLs instead, which don't have that
extra subdirectory and so the local paths on disk now end up being a
little different, because the actual module directories are at a different
subdirectory of the package.
2021-06-03 08:50:34 -07:00
Martin Atkins 17b766c3ea configs: EntersNewPackage methods for descendant modules
Now that we (in the previous commit) refactored how we deal with module
sources to do the parsing at config loading time rather than at module
installation time, we can expose a method to centralize the determination
for whether a particular module call (and its resulting Config object)
enters a new external package.

We don't use this for anything yet, but in later commits we will use this
for some cross-module features that are available only for modules
belonging to the same package, because we assume that modules grouped
together in a package can change together and thus it's okay to permit a
little more coupling of internal details in that case, which would not
be appropriate between modules that are versioned separately.
2021-06-03 08:50:34 -07:00
Martin Atkins 1a8da65314 Refactoring of module source addresses and module installation
It's been a long while since we gave close attention to the codepaths for
module source address parsing and external module package installation.
Due to their age, these codepaths often diverged from our modern practices
such as representing address types in the addrs package, and encapsulating
package installation details only in a particular location.

In particular, this refactor makes source address parsing a separate step
from module installation, which therefore makes the result of that parsing
available to other Terraform subsystems which work with the configuration
representation objects.

This also presented the opportunity to better encapsulate our use of
go-getter into a new package "getmodules" (echoing "getproviders"), which
is intended to be the only part of Terraform that directly interacts with
go-getter.

This is largely just a refactor of the existing functionality into a new
code organization, but there is one notable change in behavior here: the
source address parsing now happens during configuration loading rather
than module installation, which may cause errors about invalid addresses
to be returned in different situations than before. That counts as
backward compatible because we only promise to remain compatible with
configurations that are _valid_, which means that they can be initialized,
planned, and applied without any errors. This doesn't introduce any new
error cases, and instead just makes a pre-existing error case be detected
earlier.

Our module registry client is still using its own special module address
type from registry/regsrc for now, with a small shim from the new
addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also
rework the registry client to work with the new address type, but this
commit is already big enough as it is.
2021-06-03 08:50:34 -07:00
Martin Atkins 2e7db64968 addrs: Representation of module source addresses
We've previously had the syntax and representation of module source
addresses pretty sprawled around the codebase and intermingled with other
systems such as the module installer.

I've created a factored-out implementation here with the intention of
enabling some later refactoring to centralize the address parsing as part
of configuration decoding, and thus in turn allow the parsing result to
be seen by other parts of Terraform that interact with configuration
objects, so that they can more robustly handle differences between local
and remote modules, and can present module addresses consistently in the
UI.
2021-06-03 08:50:34 -07:00
Martin Atkins f25935649a getmodules: Beginnings of a new package about Terraform module packages
This new package aims to encapsulate all of our interactions with
go-getter to fetch remote module packages, to ensure that the rest of
Terraform will only use the small subset of go-getter functionality that
our modern module installer uses.

In older versions of Terraform, go-getter was the entire implementation
of module installation, but along the way we found that several aspects of
its design are poor fit for Terraform's needs, and so now we're using it
as just an implementation detail of Terraform's handling of remote module
packages only, hiding it behind this wrapper API which exposes only
the services that our module installer needs.

This new package isn't actually used yet, but in a later commit we will
change all of the other callers to go-getter to only work indirectly
through this package, so that this will be the only package that actually
imports the go-getter packages.
2021-06-03 08:50:34 -07:00
Martin Atkins e5f52e56f8 addrs: Rename DefaultRegistryHost to DefaultProviderRegistryHost
As the comment notes, this hostname is the default for provide source
addresses. We'll shortly be adding some address types to represent module
source addresses too, and so we'll also have DefaultModuleRegistryHost
for that situation.

(They'll actually both contain the the same hostname, but that's a
coincidence rather than a requirement.)
2021-06-03 08:50:34 -07:00
Martin Atkins 4e74a7a4f1 initwd: Error message for local paths escaping module packages
Our module installer has a somewhat-informal idea of a "module package",
which is some external thing we can go fetch in order to add one or more
modules to the current configuration. Our documentation doesn't talk much
about it because most users seem to have found the distinction between
external and local modules pretty intuitive without us throwing a lot of
funny terminology at them, but there are some situations where the
distinction between a module and a module package are material to the
end-user.

One such situation is when using an absolute rather than relative
filesystem path: we treat that as an external package in order to make the
resulting working directory theoretically "portable" (although users can
do various other things to defeat that), and so Terraform will copy the
directory into .terraform/modules in the same way as it would download and
extract a remote archive package or clone a git repository.

A consequence of this, though, is that any relative paths called from
inside a module loaded from an absolute path will fail if they try to
traverse upward into the parent directory, because at runtime we're
actually running from a copy of the directory that's been taking out of
its original context.

A similar sort of situation can occur in a truly remote module package if
the author accidentally writes a "../" source path that traverses up out
of the package root, and so this commit introduces a special error message
for both situations that tries to be a bit clearer about there being a
package boundary and use that to explain why installation failed.

We would ideally have made escaping local references like that illegal in
the first place, but sadly we did not and so when we rebuilt the module
installer for Terraform v0.12 we ended up keeping the previous behavior of
just trying it and letting it succeed if there happened to somehow be a
matching directory at the given path, in order to remain compatible with
situations that had worked by coincidence rather than intention. For that
same reason, I've implemented this as a replacement error message we will
return only if local module installation was going to fail anyway, and
thus it only modifies the error message for some existing error situations
rather than introducing new error situations.

This also includes some light updates to the documentation to say a little
more about how Terraform treats absolute paths, though aiming not to get
too much into the weeds about module packages since it's something that
most users can get away with never knowing.
2021-05-27 11:00:43 -07:00
CJ Horton 2f980f1dfe
Merge pull request #28820 from hashicorp/radditude/refresh-error-message
backend/remote: point refresh users towards -refresh-only
2021-05-27 07:21:37 -07:00
CJ Horton 314d322b59 backend/remote: encourage use of -refresh-only 2021-05-26 23:08:39 -07:00
Alisdair McDiarmid a3b1764fd7 backend/local: Fix lock when applying stale plan
When returning from the context method, a deferred function call checked
for error diagnostics in the `diags` variable, and unlocked the state if
any exist. This means that we need to be extra careful to mutate that
variable when returning errors, rather than returning a different set of
diags in the same position.

Previously this would result in an invalid plan file causing a lock to
become stuck.
2021-05-26 16:05:58 -04:00
Alisdair McDiarmid 52591ba5e9
Merge pull request #28806 from hashicorp/alisdair/remove-json-drift-summary
command/views: Remove unused drift summary message
2021-05-26 11:50:56 -04:00
James Bardin 99d0266585 return error for invalid resource import
Most legacy provider resources do not implement any import functionality
other than returning an empty object with the given ID, relying on core
to later read that resource and obtain the complete state. Because of
this, we need to check the response from ReadResource for a null value,
and use that as an indication the import id was invalid.
2021-05-25 17:13:49 -04:00
Alisdair McDiarmid 953738c128 command/views: Remove unused drift summary message
This was dead code, and there is no clear way to retrieve this
information, as we currently only derive the drift information as part
of the rendering process.
2021-05-25 15:54:57 -04:00
James Bardin 3bf498422c typo 2021-05-25 08:32:44 -04:00
James Bardin 45b5c289a0 no drift with only deposed changes
Changes to deposed instances should not triggers any drift to be
rendered, as they will have nothing to display.
2021-05-24 17:17:52 -04:00
James Bardin 57aa7c6025 skip drift rendering for deposed resources
Deposed instances have no current state and are only scheduled for
deletion, so there is no reason to try and render drift on these
instances.
2021-05-24 15:48:05 -04:00
James Bardin aae642fb07 fix schemas and add deposed test
The schemas for provider and the resources didn't match, so the changes
were not going to be rendered at all.

Add a test which contains a deposed resource.
2021-05-24 15:38:58 -04:00
James Bardin 95e788f13b deposed instances should not be counted as orphans
Do not add orphan nodes for resources that only contain deposed
instances.
2021-05-20 09:36:45 -04:00
James Bardin aaaeeffa81 a noop change with no state has no private data
There is no change here to use the private data, and currentState may be
nil.
2021-05-20 09:34:25 -04:00
CJ Horton 94ab6d7504
Merge pull request #28746 from hashicorp/remote-backend-refresh-replace 2021-05-19 10:33:19 -07:00
Kristin Laemmert 649095c602
providers subcommand tests (#28744)
* getproviders ParsePlatform: add check for invalid platform strings with too many parts

The existing logic would not catch things like a platform string containing multiple underscores. I've added an explicit check for exactly 2 parts and some basic tests to prove it.

* command/providers-lock: add tests

This commit adds some simple tests for the providers lock command. While adding this test I noticed that there was a mis-copied error message, so I replaced that with a more specific message. I also added .terraform.lock.hcl to our gitignore for hopefully obvious reasons.

getproviders.ParsePlatform: use parts in place of slice range, since it's available

* command: Providers mirror tests

The providers mirror command is already well tested in e2e tests, so this includes only the most absolutely basic test case.
2021-05-19 12:56:16 -04:00
James Bardin bafa44b958 return diagnostics from provisioners
Do not convert provisioner diagnostics to errors so that users can get
context from provisioner failures.

Return diagnostics from the builtin provisioners that can be annotated
with configuration context and instance addresses.
2021-05-19 11:24:54 -04:00
CJ Horton ab5fec9aec backend/remote: test new options and modes
Tests for -refresh=false, -refresh-only, and -replace,
as well as tests to ensure we error appropriately on
mismatched API versions.
2021-05-18 15:35:37 -07:00
CJ Horton eb1f113693 backend/remote: validate API version for refresh and replace 2021-05-18 15:35:37 -07:00
Kristin Laemmert 4928e1dd01
terraform: use ProtocolVersion from unmanaged providers' reattachConfig to chose the correct PluginClient (#28190)
* add/use ProtocolVersion with unmanaged providers reattach config
2021-05-18 10:59:14 -04:00
Alisdair McDiarmid 5fc3ba37a6 cli: Improve sensitivity change warning output
When an attribute value changes in sensitivity, we previously rendered
this in the diff with a `~` update action and a note about the
consequence of the sensitivity change. Since we also suppress the
attribute value, this made it impossible to know if the underlying value
was changing, too, which has significant consequences on the meaning of
the plan.

This commit adds an equality check of the old/new underlying values. If
these are unchanged, we add a note to the sensitivity warning to clarify
that only sensitivity is changing.
2021-05-18 10:33:25 -04:00
Chris Arcand 2171f8a1a6 backend/remote: Support replacements in plans 2021-05-17 16:31:34 -07:00
Chris Arcand a436c7fa2f backend/remote: Support refresh-only plans 2021-05-17 16:31:33 -07:00
Chris Arcand cb49a4c8d6 backend/remote: Remove the -refresh=false restriction 2021-05-17 16:31:32 -07:00
James Bardin 4d55e2604f save apply diagnostics on backend failure
Make sure to collect apply diagnostics before persisting the backend
state so runtime errors are not lost.
2021-05-17 17:22:04 -04:00
Martin Atkins d35bc05312 Update various non-code references to renamed Go packages
Previous commits have moved all of our Go packages under "internal", so
this is a retroactive update of non-Go references to those same packages.
2021-05-17 14:09:07 -07:00
Martin Atkins 36d0a50427 Move terraform/ to internal/terraform/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins f40800b3a4 Move states/ to internal/states/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins b40a4fb741 Move plugin/ and plugin6/ to internal/plugin{,6}/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 034e944070 Move plans/ to internal/plans/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins cdd9464f9a Move lang/ to internal/lang/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 31349a9c3a Move configs/ to internal/configs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins ffe056bacb Move command/ to internal/command/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins b9a93a0fe7 Move addrs/ to internal/addrs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 26e5320a78 Move experiments/ to internal/experiments/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins a6b8c8ba22 Move moduledeps/ to internal/moduledeps/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins e81162c4e1 Move provisioners/ to internal/provisioners/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 1409f30f9c Move providers/ to internal/providers/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins bc31dbc179 Move instances/ to internal/instances/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 05caff2ca3 Move tfdiags/ to internal/tfdiags/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins ec85fb1960 Move communicator/ to internal/communicator/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c254cc2be Move httpclient/ to internal/httpclient/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 415ab31db8 Move e2e/ to internal/e2e/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c5b866dea Move registry/ to internal/registry/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00