Commit Graph

831 Commits

Author SHA1 Message Date
Katy Moe df6dad5070
add failing test for IgnoreAllChanges override 2021-11-01 19:00:54 +00:00
Barrett Clark 9c13521d76 Allow cloud tests to be run in parallel
Add the `-parallel N` switch to tell the tests to run in N processes:

```
TFE_TOKEN=$TFE_TOKEN TFE_HOSTNAME=$TFE_HOSTNAME TF_ACC=1 go test -v \
  -tags=e2e ./internal/cloud/e2e/... -parallel 4
```
2021-11-01 13:58:29 -05:00
Nick Fagerlund 02e62c9851
Cloud: Init without erroring when no workspaces match the `tags` (#29835)
Previously, `terraform init` was throwing an error if you configured the cloud
block with `tags` and there weren't any tagged workspaces yet. Confusing and
alienating, since that that's a fairly normal situation! Basically TFC was
handling an empty list of workspaces worse than other backends, because it
doesn't support an unnamed default workspace.

This commit catches that condition during `Meta.selectBackend()` and asks the
user to pick a name for their first tagged workspace. If they cancel out, we
still error, but if we know what name they want, we can handle it the same way
as a nonexistent workspace specified in `name` -- just pass it to
`Meta.SetWorkspace()`, and let the workspace get implicitly created when
`InitCommand.Run()` eventually calls `StateMgr()`.
2021-11-01 10:20:15 -07:00
James Bardin b91d9435ea
Merge pull request #29832 from hashicorp/jbardin/nullable-variable
configs: explicitly nullable variable values
2021-11-01 12:46:31 -04:00
James Bardin 6d050c166e udpate diagnostic message 2021-11-01 12:28:55 -04:00
Martin Atkins 94cbc8fb5d experiments: config_driven_move has concluded
Based on feedback during earlier alpha releases, we've decided to move
forward with the current design for the first phase of config-driven
refactoring.

Therefore here we've marked the experiment as concluded with no changes
to the most recent incarnation of the functionality. The other changes
here are all just updating test fixtures to no longer declare that they
are using experimental features.
2021-11-01 08:46:15 -07:00
Alisdair McDiarmid a61bd8a96b cloud: Fix prerelease version constraint checks 2021-11-01 11:12:58 -04:00
xiaozhu36 5d19a34f88 backend/oss: Supports the new attribute sts_endpoint 2021-11-01 22:52:45 +08:00
Chris Arcand 3cd6c0be7a
Merge pull request #29836 from hashicorp/chrisarcand/fix-tfc-resource-counts
command: Adjust skipping of resource counts for any remote implementation
2021-11-01 09:50:13 -05:00
James Bardin 2ebdc099ac
Merge pull request #29307 from hayorov/main
Flatten `assume_role` block for OSS backend
2021-11-01 09:18:25 -04:00
James Bardin 1f873776e8 update null variable error text 2021-11-01 09:12:28 -04:00
James Bardin b71b393cf6 move nullable check to variable input evaluation 2021-11-01 09:02:32 -04:00
Chris Arcand 511afcd43a command: Adjust skipping of resource counts for any remote implementation
When using the Terraform Cloud integration - like the 'remote'
backend - resource count output should be suppressed if those counts are
being rendered remotely. This generalizes this to the shared
BackendWithRemoteTerraformVersion interface.
2021-10-29 21:23:28 -05:00
James Bardin 7b7972ac95 allow nullable override 2021-10-29 17:20:14 -04:00
James Bardin 81e709d185
Merge pull request #29825 from hashicorp/jbardin/no-panicwrap
Remove the use of panicwrap in the Terraform CLI
2021-10-29 14:41:05 -04:00
James Bardin f0a64eb456 configs: explicitly nullable variable values
The current behavior of module input variables is to allow users to
override a default by assigning `null`, which works contrary to the
behavior of resource attributes, and prevents explicitly accepting a
default when the input must be defined in the configuration.

Add a new variable attribute called `nullable` will allow explicitly
defining when a variable can be set to null or not. The current default
behavior is that of `nullable=true`.

Setting `nullable=false` in a variable block indicates that the variable
value can never be null. This either requires a non-null input value, or
a non-null default value. In the case of the latter, we also opt-in to
the new behavior of a `null` input value taking the default rather than
overriding it.

In a future language edition where we make `nullable=false` the default,
setting `nullable=true` will allow the legacy behavior of `null`
overriding a default value. The only future configuration in which this
would be required even if the legacy behavior were not desired is when
setting an optional+nullable value. In that case `default=null` would
also be needed and we could therefor imply `nullable=true` without
requiring it in the configuration.
2021-10-29 13:59:46 -04:00
Martin Atkins f266d1ee82
"Add cloud integration option"
A more native integration for Terraform Cloud and its CLI-driven run workflow.

Instead of a backend, users declare a special block in the top-level terraform settings
block to configure Terraform Cloud, then run terraform init.

Full documentation will follow in later commits.
2021-10-28 18:30:01 -07:00
Chris Arcand 369264865e internal/plugin[6]: Add generated mocks 2021-10-28 20:16:26 -05:00
Chris Arcand 6dc1fed6d5 Clarify legacy Ui comments 2021-10-28 19:29:21 -05:00
Chris Arcand 14260a5b4c Adjust default workspace naming prompt
The previous version is a little overdramatic and somewhat accusatory.
Just lay it out how it is: TFC needs workspaces to be named.
2021-10-28 19:29:21 -05:00
Brandon Croft 6d9c919f55 adds X-Terraform-Integration header
This can help differentiate cloud integration API requests from normal remote backend requests
2021-10-28 19:29:20 -05:00
Chris Arcand 3d23a8e062 cloud: Fix version check regression
When the 'select the exact version if possible' behavior was added, the
version check below it was never updated to take the newly updated
version in to account, resulting in a failed version check even as the
remote workspace updated to the correct version necessary.
2021-10-28 19:29:20 -05:00
Chris Arcand 652a68715f cloud/e2e: Misc test fixes 2021-10-28 19:29:20 -05:00
Chris Arcand f881c2d794 cloud: Skip E2E tests without an exact version 2021-10-28 19:29:20 -05:00
Chris Arcand e364ef2905 cloud/e2e: Disable cost estimation
E2E tests including cost estimation should indeed be added, but the
default case should be disabled; lots of cycles lost to pointless cost
estimates on null and random resources.
2021-10-28 19:29:19 -05:00
Omar Ismail fc5863844c Cloud migration: ignore backend version check when empty worksapces 2021-10-28 19:29:19 -05:00
Barrett Clark f5366468b4 Cloud Backend reference migrating away from TFC 2021-10-28 19:29:19 -05:00
Chris Arcand 2c0294c7e3 Tweak configuration copy for TFC
This aligns more with the existing copy for backends
2021-10-28 19:29:19 -05:00
Nick Fagerlund fabd8eb0b6 Fix broken rename behavior in tfe_client_mock.go
The delete + assign at the end of `Update` and `UpdateByID` are meant to handle
renaming a workspace — (remove old name), (insert new name).

However, `UpdateByID` was doing (remove new name), (insert new name) and leaving
the old name in place. This commit changes it to match `Update` by grabbing the
original name off the workspace object _before_ potentially renaming it.
2021-10-28 19:29:18 -05:00
Chris Arcand 50997d9a32 Restore API version checking, with internal usage accounted for 2021-10-28 19:29:18 -05:00
Nick Fagerlund cb26cbba22 Add tests for cloud.StateMgr() when TFC lacks local tf version
Alas, there's not a very good way to test the message we're supposed to print to
the console in this situation; we just don't appear to have a mock terminal that
the test can read from. But we can at least test that the function returns
without erroring under the exact conditions where it was erroring before.

Note that the behaviors of mc.Workspaces.Update and UpdateByID were already
starting to drift, so I consolidated their actual attribute update logic into a
helper function before they drifted much further.
2021-10-28 19:29:18 -05:00
Nick Fagerlund c99f5972de cloud.StateMgr(): Set terraform version AFTER creating workspace
Previously, if the remote TFC/TFE instance doesn't happen to have a tool_version
record whose name exactly matches the value of `tfversion.String()`, Terraform
would be completely blocked from using the `terraform workspace new` command
(when configured with the tags strategy) — the API would give a 422 to the
whole create request.

This commit changes the StateMgr() function to do the work in two passes; first
create the workspace (which should work fine regardless), THEN update the
Terraform version and print a warning to the terminal if it fails (which 99% of
the time is a benign failure with little impact on your future CLI usage).
2021-10-28 19:29:18 -05:00
Chris Arcand aace0015c2 cloud: Fix E2E variables test 2021-10-28 19:29:17 -05:00
Brandon Croft 64fbb0b613 remove reference to AccessInternalTools 2021-10-28 19:29:17 -05:00
Omar Ismail 1ff7827416 Multi-state migration prompt fix 2021-10-28 19:29:17 -05:00
Barrett Clark cc6de251d8 Update init reconfigure error message
If you move from the remote backend to the cloud block you will see this
error message.
2021-10-28 19:29:17 -05:00
Chris Arcand 12a4af3e72 Temporarily skip API version checking 2021-10-28 19:29:16 -05:00
Barrett Clark e16c53b561 Simplify/Consolidate the cloud conditionals 2021-10-28 19:29:16 -05:00
Barrett Clark ab304d831f Found another path to backend init error
There are actually a few different ways to get to this message.

1. Blank state — no previous terraform applied. Start with a cloud block.
1. Implicit local — start with no backend specified. This actually goes
   through the same code execution path as the first scenario.
1. Explicit local — start with a backend local block that has been
   applied, then change from the local backend to a cloud block. This
   will recognize the state, and is a different path through the code in
   the meta backend.

This commit handles the last case. The messaging has also been tweaked.

End to end test included as well.
2021-10-28 19:29:16 -05:00
Barrett Clark d29532cfeb tfc apply before init
https://app.asana.com/0/1199201948575144/1201019450474843/f
2021-10-28 19:29:16 -05:00
Barrett Clark e49f271eb5 Update copy for terraform init with cloud block 2021-10-28 19:29:16 -05:00
Omar Ismail 5ff2495ffa Update HasResource to HasManagedResourceInstanceObjects 2021-10-28 19:29:15 -05:00
Omar Ismail 86f93a2cbc Cloud UX test fixes 2021-10-28 19:29:15 -05:00
Chris Arcand 09d5b70b7a command/meta_backend_migrate:Tweak multi-to-single To TFC copy 2021-10-28 19:29:15 -05:00
Omar Ismail dfb4609be2 Backend State Migration from cloud to cloud.
* Add test for tfc to tfc mgiration
* Fix old tests, and remove unused code.
2021-10-28 19:29:15 -05:00
Barrett Clark 93bfcff61a More Terraform Cloud UX changes
* Update e2e tests to specify Terraform Cloud
* Update more of the terraform command flow to specify Terraform Cloud
2021-10-28 19:29:14 -05:00
Barrett Clark 261a2b49d3 UX for terraform init that copies state to TFC 2021-10-28 19:29:14 -05:00
Barrett Clark d04c01573c Fix iterator in a couple of e2e tests 2021-10-28 19:29:14 -05:00
Barrett Clark 0cd9490838 PR Feedback: cleanup variable setter 2021-10-28 19:29:14 -05:00
Chris Arcand f8256f6634 Update ux for "terraform init [-reconfigure]" 2021-10-28 19:29:14 -05:00
Omar Ismail dc76bbee73 Backend State Migration: Add remote backend test 2021-10-28 19:29:14 -05:00
Omar Ismail 3fedd6898c Backend State Migration to `cloud`: Multiple Workspaces
* Handle when there are multiple workspaces migrating to cloud,
using both the cloud name strategy and cloud tags strategy.
* Add e2e tests
2021-10-28 19:29:13 -05:00
Nick Fagerlund baa72ce235 Simplify logic flow: everything is a constraint
Explicit version strings are actually also version constraints! And the special
comparisons we were doing to allow a range of compatible versions can also be
expressed as version constraints.

Bonus: also simplify the way we handle version check errors, by composing the
messages inline and only extracting the repetitive parts into a function.
2021-10-28 19:29:13 -05:00
Barrett Clark 83538fdd6b Shift errors to error file, improve error text 2021-10-28 19:29:13 -05:00
Nick Fagerlund 9857097b34 Add tests for version constraints in workspace terraform-versions 2021-10-28 19:29:13 -05:00
Nick Fagerlund b43daeaa8d Cloud backend: accept version constraints from workspaces
The cloud backend (and remote before it) previously expected a TFC workspace's
`terraform-version` attribute to be either the magic string `"latest"` or an
explicit semver value. But a workspace might have a version constraint instead
(like `~> 1.1.0`), in which case the version check would blow up.

This commit checks whether `terraform-version` is a valid version constraint
before erroring out, and if so, returns success if the local version meets the
constraint.

Because it's not practical to deeply introspect the slice of version space
defined by a constraint, this check is slightly less robust than the version
comparisons below it:

- It can give a false OK on open-ended constraints like `>= 1.1.0`. Say you're
  running 1.3.0, it changed the state format, and the TFE instance admin has
  not yet added any 1.3.x Terraform versions; your workspace will now break.

- It will give a false not-OK when using different minor versions within a range
  that we know to be compatible, e.g. remote constraint of `~> 0.15.0` and local
  version of 1.1.0.

- This would be totally useless with the pre-0.14 versions of Terraform, where
  patch releases could change state format... but we're not going back in time
  to add this feature to them anyway.

Still, in the most common likely case (`~> x.y.z`), it'll complain at you (with
an error you can choose to override) if you're not using the same minor version,
and that seems proportionate, useful, and expected.
2021-10-28 19:29:13 -05:00
Barrett Clark 83337de654 Remove prefix from the cloud backend config
Now that we have tags we no longer need prefix.
2021-10-28 19:29:12 -05:00
Barrett Clark edbc84420c Run apply -refresh-state instead of refresh
When a user runs `terraform refresh` we give them an error message that
tells them to run `terraform apply -refresh-state`. We could just run
that command for them, though. That is what this PR does.
2021-10-28 19:29:12 -05:00
Brandon Croft bf02b5cb53 e2e test for cloud run variables 2021-10-28 19:29:12 -05:00
Brandon Croft 9963bf940a refactor: only fetch variables if unset variables are not allowed 2021-10-28 19:29:12 -05:00
Omar Ismail dd856f8a1b Add run variables to cloud plan operations from non-file sources
Run variables allow the run API to accept input variables not found
in the configuration slug (the file-based ones plus workspace vars)
2021-10-28 19:29:12 -05:00
Brandon Croft dbbfae5a1c refactor ParseVariableValues into separate operations
1. ParseDeclaredValues: parses unparsed variables into terraform.InputValues
2. ProbeUndeclaredVariableValues: compares variable declarations with unparsed values to warn/error about undeclared variables
2021-10-28 19:29:12 -05:00
Omar Ismail 55fc590904 Teraform Cloud Backend State Migration
* determining source or destination to cloud
* handling single to single state migrations to cloud,
using a name strategy or a tags strategy
* Add end-to-end tests for state migration.
2021-10-28 19:29:12 -05:00
Chris Arcand 7cc53fe163 cloud: Set minimum TFE version
These changes remove all of the preexisting version checking for
individual features, wiping the slate clean with an overall minimum
requirement of a future TFP-API-Version 2.5, which at the time of this
writing is expected to be TFE v202112-1.

It also actually provides that expected TFE version as an actionable
error message, rather than generically saying that it isn't supported or
using the somewhat opaque API version header.
2021-10-28 19:29:12 -05:00
Chris Arcand 46e47ed379 Use the 'tfe' service for everything.
The 'tfe' service was appended to with various versions to denote a new
'feature' implemented by a new 'service'. This quickly proved to not be
scalable, as adding an entry to the discovery document from every
feature is bad.

The new mechanism added was checking the TFP-API-Version header on
requests for a version, instead.

So we'll remove the separation here between different tfe service
'versions' and the separate 'state' service and Just Use TFE, as well as
the TFP-API-Version header for all feature versioning., as well as the
TFP-API-Version header for all feature versioning.
2021-10-28 19:29:12 -05:00
Chris Arcand 412679522d Remove all constraint checking
This is an outdated mechanism that isn't used anymore.
2021-10-28 19:29:11 -05:00
Chris Arcand a4e196d067 Port backend/local: Check dependency lock consistency before any operations
This is a port of the changes made in #29683
2021-10-28 19:29:11 -05:00
Omar Ismail d5567b479c Fix terraform version in cloud e2e tests to ignore prerelease. (#29) 2021-10-28 19:29:11 -05:00
Omar Ismail a387af6c61 Add auto-approve logic, e2e tests 2021-10-28 19:29:11 -05:00
Chris Arcand ee384e8716 Attempt to set current Terraform version on new workspaces, always 2021-10-28 19:29:11 -05:00
Chris Arcand aba7d96596 cloud: Support interop from 0.14 to 1.1
The previous conservative guarantee that we would not make backwards
incompatible changes to the state file format until at least Terraform
1.1 can now be extended. Terraform 0.14 through 1.1 will be able to
interoperably use state files, so we can update the remote backend
version compatibility check accordingly.

This is a port of https://github.com/hashicorp/terraform/pull/29645
2021-10-28 19:29:11 -05:00
Chris Arcand 74e087dc29 Rename cloud.testBackendDefault -> cloud.testBackendWithName 2021-10-28 19:29:11 -05:00
Chris Arcand d2b6b5f2b3 cloud: Align local and remote workspace name with 'name' strategy
This changes the 'name' strategy to always align the local configured
workspace name and the remote Terraform Cloud workspace, rather than the
implicit use of the 'default' unnamed workspace being used instead.

What this essentially means is that the Cloud integration does not fully
support workspaces when configured for a single TFC workspace (as was
the case with the 'remote' backend), but *does* use the
backend.Workspaces() interface to allow for normal local behaviors like
terraform.workspace to resolve to the correct name. It does this by
always setting the local workspace name when the 'name' strategy is
used, as a part of initialization.

Part of the diff here is exporting all the previously unexported types
for mapping workspaces. The command package (and init in particular)
needs to be able to handle setting the local workspace in this
particular scenario.
2021-10-28 19:29:11 -05:00
Omar Ismail a94b2405b1 static checker 2021-10-28 19:29:11 -05:00
Omar Ismail 594a390f84 Use context parallelism 2021-10-28 19:29:10 -05:00
Chris Arcand 1791b71196 cloud: TestCloud_backendWithTags
Implementing this test was quite a rabbithole, as in order to satisfy
backendTestBackendStates() the workspaces returned from
backend.Workspaces() must match exactly, and the shortcut taken to test
pagination in 3cc58813f0 created an
impossible circumstance that got plastered over with the fact that
prefix filtering is done clientside, not by the API as it should be.

Tagging does not rely on clientside filtering, and expects that the
request made to the TFC API returns exactly those workspaces with the
given tags.

These changes include a better way to test pagination, wherein we
actually create over a page worth of valid workspaces in the mock client
and implement a simplified pagination behavior to match how the TFC API
actually works.
2021-10-28 19:29:10 -05:00
Chris Arcand a97a1c8f66 cloud.MockClient: Use Contains() to implement Search
The TFC API doesn't behave this way; a search term just signals the
substring was found within the name.
2021-10-28 19:29:10 -05:00
Chris Arcand 9bdf1d2579 Rename testBackendNoDefault
"NoDefault" is now ambiguous with tags, and it does not imply using a
prefix.
2021-10-28 19:29:10 -05:00
Chris Arcand 6dcd0db265 Map CLI workspaces by TFC tags 2021-10-28 19:29:10 -05:00
Chris Arcand 7a243379fb cloud: Refactor workspaceMapping concerns into strategy() 2021-10-28 19:29:10 -05:00
Chris Arcand 922a8e4488 Refactor private workspace fields into workspaceMapping
A mostly cosemetic change; The fields 'workspace' and 'prefix' don't
really describe well what they are from a caller, so change these to use
a workspaceMapping struct to convey they are for implementing workspace
mapping strategies from CLI -> TFC
2021-10-28 19:29:10 -05:00
Omar Ismail 0a125c0400 Cloud: refactor setting attributes, add tests 2021-10-28 19:29:10 -05:00
Omar Ismail 8f49757f89 Add testing for Cloud PrepareConfig 2021-10-28 19:29:10 -05:00
Chris Arcand 471dd479e8 Update go-tfe to 26689e
These changes include additions to fulfill the interface for the client
mock, plus moving all that logic (which needn't be duplicated across
both the remote and cloud packages) over to the cloud package under a
dedicated mock client file.
2021-10-28 19:29:10 -05:00
Chris Arcand 34597c237f cloud: Don't ignore .log golden files 2021-10-28 19:29:10 -05:00
Chris Arcand 85f37f1aa5 Add missing log mocks
I missed adding these in the original porting commit from the remote
backend, because *.log is added to gitignore.
2021-10-28 19:29:10 -05:00
Martin Atkins f4758803fe cloud: Backend implements new version of the backend.Local interface
This makes the new backend compatible with the new terraform.Context API,
which has changed in main.
2021-10-28 19:29:09 -05:00
Chris Arcand 18d54c1129 Allow cloud block overrides
These changes allow cloud blocks to be overridden by backend blocks and
vice versa; the logic follows the current backend behavior of a block
overriding a preceding block in full, with no merges.
2021-10-28 19:29:09 -05:00
Chris Arcand 07b3d015d1 Defer cloud block overrides
This restriction is temporary. Overrides should be allowed, but have the
added complexity of needing to also override a 'backend' block, so this
work is being deferred for now.
2021-10-28 19:29:09 -05:00
Chris Arcand fccc873a3d Remove ability to declare a 'cloud' backend
With the alternative block introduced in 7bf9b2c7b, this removes the
ability to explicitly declare the 'cloud' backend. The literal backend
interface is an implementation detail and no longer a user-level
concept when using Terraform Cloud.
2021-10-28 19:29:09 -05:00
Chris Arcand a4c24e3147 Add cloud {} configuration block for Terraform Cloud
This is a replacement declaration for using Terraform Cloud as a remote
backend, leaving the literal backend as an implementation detail and not
a user-level concept.
2021-10-28 19:29:09 -05:00
Chris Arcand a5f063625a Replace generic host error messages for feature support 2021-10-28 19:29:09 -05:00
Chris Arcand ea8ad0b15a Initial commit of 'cloud' package
The cloud package intends to implement a new integration for
Terraform Cloud/Enterprise. The purpose of this integration is to better
support TFC users; it will shed some overly generic UX and architecture,
behavior changes that are otherwise backwards incompatible in the remote
backend, and technical debt - all of which are vestiges from before
Terraform Cloud existed.

This initial commit is largely a porting of the existing 'remote'
backend, which will serve as an underlying implementation detail and not
be a typical user-level backend. This is because to re-implement the
literal backend interface is orthogonal to the purpose of this
integration, and can always be migrated away from later.

As this backend is considered an implementation detail, it will not be
registered as a declarable backend. Within these changes it is, for easy
of initial development and a clean diff.
2021-10-28 19:29:09 -05:00
James Bardin 9b62dd6901
Merge pull request #29648 from hashicorp/jbardin/tfproto6.1
tfproto version 6.1
2021-10-28 16:00:02 -04:00
James Bardin 583e3a5f0b use 11 for the panic exit code 2021-10-28 15:34:02 -04:00
James Bardin bd37f43daa add note about calling PanicHandler first 2021-10-28 13:56:18 -04:00
James Bardin d03a037567 insert panic handlers 2021-10-28 11:51:39 -04:00
James Bardin 3f31533f86 logging.PanicHandler can now be for internal use
Repurpose logging.PanicHandler to recover internal panics and print a
message to users similar to panicwrap.
2021-10-28 11:51:39 -04:00
James Bardin 42742c173d remove wrapped streams and readline 2021-10-28 11:51:39 -04:00
James Bardin 622c4df14c remove the use of panicwrap
Stop using panicwrap, and execute terraform in the main process.
2021-10-28 11:51:39 -04:00
Alisdair McDiarmid ded4f1a0fd
Merge pull request #29805 from hashicorp/alisdair/fix-init-workspace-select-input-false
command/init: Fail if -input=false but required
2021-10-26 11:37:22 -04:00
Alisdair McDiarmid ecb98e1c43 command/init: Fail if -input=false but required
When running `terraform init` against a backend with multiple
workspaces, none of which are the currently indicated local workspace,
Terraform prompts the user to choose a workspace from the list. In
automation, using the `-input=false` argument should disable asking for
input, but previously would hang instead.
2021-10-25 15:08:10 -04:00
Alisdair McDiarmid 6d0521e6be
Merge pull request #29803 from hashicorp/alisdair/init-backend-provider-errors
cli: Fix backend init failure with deleted cache
2021-10-25 13:57:45 -04:00
Alisdair McDiarmid 22923f9873
Merge pull request #29793 from hashicorp/alisdair/fix-remote-backend-migrate-version-check
backend/remote: Fix version check when migrating
2021-10-25 12:50:27 -04:00
Alisdair McDiarmid d42d83572b cli: Fix backend init failure with deleted cache
When an explicit backend is configured with a configuration which has
not yet been initialized, running `terraform init` performs a state
migration to fetch the remotely stored state in order to operate on it.
Like the previous bug introduced by the recent provider diagnostics
change, this code path was not correctly configured to enable init mode
for the backend, which resulted in a fatal error during init when the
cache dir is deleted.

Setting the `Init` backend option allows this code path to continue
without error when first initializing the backend for state migration.
The new e2e test fails without this change.
2021-10-25 12:45:35 -04:00
Alisdair McDiarmid 1729431520 backend/remote: Fix version check when migrating
When migrating state to an existing Terraform Cloud workspace using the
remote backend, we check the remote version is compatible with the local
one by default.

This commit fixes two bugs in this code:

- If using the "name" strategy for the remote backend, the list of
  destination workspaces is empty. This resulted in no version checking
  of the remote workspace, and we fell back to the string equality
  check.
- The user-specified CLI flag `-ignore-remote-version` was not being
  applied for the state migration version checking.
2021-10-21 14:10:25 -04:00
Alex Khaerov 6f1eceb0e2
Revert autoupdate 2021-10-21 21:42:13 +08:00
Alisdair McDiarmid 39de3ebec7 cli: Fix init failure with deleted cache
The init command needs to initialize a backend, in order to access
state, in turn to derive provider requirements from state. The backend
initialization step requires building provider factories, which
previously would fail if a lockfile was present without a corresponding
local provider cache.

This commit ensures that in this situation only, errors with the
provider factories are temporarily ignored. This allows us to continue
to initialize the backend, fetch providers, and then report any errors
as necessary.
2021-10-21 08:44:26 -04:00
Alisdair McDiarmid 1190b95fe2 command/e2etest: Ensure init fixes missing cache
We test that a deleted provider cache results in an error when running
terraform plan, but previously did not test that running init (as
instructed) would resolve the issue. This (failing) e2e test adds that
step.
2021-10-21 08:41:01 -04:00
James Bardin 0fcb75020f
Merge pull request #29784 from xiaozhu36/xiaozhu
backend/oss: Fixes the nil pointer panic error when missing access key or secret key
2021-10-20 16:32:13 -04:00
Martin Atkins 5b266dd5ca command: Remove the experimental "terraform add" command
We introduced this experiment to gather feedback, and the feedback we saw
led to us deciding to do another round of design work before we move
forward with something to meet this use-case.

In addition to being experimental, this has only been included in alpha
releases so far, and so on both counts it is not protected by the
Terraform v1.0 Compatibility Promises.
2021-10-20 06:42:47 -07:00
xiaozhu36 7afaea4cf2 backend/oss: Fixes the nil pointer panic error when missing access key or secret key 2021-10-20 16:05:00 +08:00
Alisdair McDiarmid cdd5ee6fb3
Merge pull request #29773 from hashicorp/alisdair/init-lock-flags
cli: Restore -lock and -lock-timeout init flags
2021-10-19 09:45:15 -04:00
Alisdair McDiarmid c587384dff cli: Restore -lock and -lock-timeout init flags
The -lock and -lock-timeout flags were removed prior to the release of
1.0 as they were thought to have no effect. This is not true in the case
of state migrations when changing backends. This commit restores these
flags, and adds test coverage for locking during backend state
migration.

Also update the help output describing other boolean flags, showing the
argument as the user would type it rather than the default behavior.
2021-10-19 09:32:30 -04:00
Alisdair McDiarmid fb58f9e6d2 cli: Fix flaky init cancel test
There is a race between the MockSource and ShutdownCh which sometimes
causes this test to fail. Add a HangingSource implementation of Source
which hangs until the context is cancelled, so that there is always time
for a user-initiated shutdown to trigger the cancellation code path
under test.
2021-10-19 09:10:49 -04:00
Alex Khaerov 14f366dbf4
Update documentation 2021-10-18 12:54:40 +08:00
Alex Khaerov f32702c5c2
Support deprecated assume_role block 2021-10-18 12:32:57 +08:00
Alex Khaerov de8810cdd9
Merge branch 'hashicorp:main' into main 2021-10-18 11:42:37 +08:00
Martin Atkins 39779e7021 backend/remote-state/cos: Don't use github.com/likexian/gokit
We don't use this library anywhere else in Terraform, and this backend was
using it only for trivial helpers that are easy to express inline anyway.
The new direct code is also type-checkable, whereas these helper functions
seem to be written using reflection.

This gives us one fewer dependency to worry about and makes the test code
for this backend follow a similar assertions style as the rest of this
codebase.
2021-10-14 10:50:05 -07:00
James Bardin ef3c98466d
Merge pull request #29755 from hashicorp/jbardin/first-plan-lineage
Check for stale plan with no state metadata
2021-10-14 13:31:30 -04:00
James Bardin 9c80574417 test planfile may need to have a specific lineage
In order to test applying a plan from an existing state, we need to be
able to inject the state meta into the planfile.
2021-10-13 17:28:14 -04:00
James Bardin 5ffa0839f9 only check serial when applying the first plan
Ensure that we still check for a stale plan even when it was created
with no previous state.

Create separate errors for incorrect lineage vs incorrect serial.

To prevent confusion when applying a first plan multiple times, only
report it as a stale plan rather than different lineage.
2021-10-13 17:17:51 -04:00
Martin Atkins bee7403f3e command/workspace_delete: Allow deleting a workspace with empty husks
Previously we would reject attempts to delete a workspace if its state
contained any resources at all, even if none of the resources had any
resource instance objects associated with it.

Nowadays there isn't any situation where the normal Terraform workflow
will leave behind resource husks, and so this isn't as problematic as it
might've been in the v0.12 era, but nonetheless what we actually care
about for this check is whether there might be any remote objects that
this state is tracking, and for that it's more precise to look for
non-nil resource instance objects, rather than whole resources.

This also includes some adjustments to our error messaging to give more
information about the problem and to use terminology more consistent with
how we currently talk about this situation in our documentation and
elsewhere in the UI.

We were also using the old State.HasResources method as part of some of
our tests. I considered preserving it to avoid changing the behavior of
those tests, but the new check seemed close enough to the intent of those
tests that it wasn't worth maintaining this method that wouldn't be used
in any main code anymore. I've therefore updated those tests to use
the new HasResourceInstanceObjects method instead.
2021-10-13 13:54:11 -07:00
Megan Bang 9b9b26a3cd update error message for invalid json 2021-10-13 13:51:07 -05:00
James Bardin 87e852c832
Merge pull request #29741 from hashicorp/jbardin/test-fix
fix test fixture with multiple provider instances
2021-10-13 09:53:02 -04:00
James Bardin 73b1263a86 wrap multiple provider creations into a factory fn
When a test uses multiple instances of the same provider, we may need to
have separate objects to prevent overwriting of the MockProvider state.
Create a completely new MockProvider in each factory function call
rather than re-using the original provider value.
2021-10-12 17:47:50 -04:00
Martin Atkins 965c0f3f91 build: Run staticcheck with "go run"
Running the tool this way ensures that we'll always run the version
selected by our go.mod file, rather than whatever happened to be available
in $GOPATH/bin on the system where we're running this.

This change caused some contexts to now be using a newer version of
staticcheck with additional checks, and so this commit also includes some
changes to quiet the new warnings without any change in overall behavior.
2021-10-12 11:42:17 -07:00
Martin Atkins d76759a6a9 configs/configload: snapshotDir must be used via pointer
A snapshotDir tracks its current position as part of its state, so we need
to use it via pointer rather than value so that Readdirnames can actually
update that position, or else we'll just get stuck at position zero.

In practice this wasn't hurting anything because we only call Readdir once
on our snapshots, to read the whole directory at once. Still nice to fix
to avoid a gotcha for future maintenence, though.
2021-10-12 11:42:17 -07:00
James Bardin 903ae5edfd fix test fixtures with multiple providers
Allow these to share the same backing MockProvider.
2021-10-12 14:34:34 -04:00
James Bardin 656f03b250 fix test fixture had the instance in the wrong mod
Make the state match the fixture config. The old test was not
technically invalid, but because it caused multiple instances of the
provider to be created, they were backed by the same MockProvider value
resulting in the `*Called` fields interfering.
2021-10-12 13:53:02 -04:00
James Bardin ef8f1b3e38
Merge pull request #29734 from hashicorp/jbardin/missing-change
Check for nil change during apply
2021-10-12 12:22:51 -04:00
Martin Atkins 2e6b6e9a6b go.mod: go get google.golang.org/protobuf@v1.27.1 2021-10-11 16:37:39 -07:00
James Bardin 3f4b680f1c Check for nil change during apply
Because NodeAbstractResourceInstance.readDiff can return a nil change,
we must check for that in all callers.
2021-10-08 16:46:29 -04:00
James Bardin 3c942ee3ac
Merge pull request #29720 from hashicorp/jbardin/destroy-plan-deposed
don't refresh deposed instances during the destroy plan
2021-10-08 16:23:59 -04:00
James Bardin a036109bc1 add comment about when we call ConfigureProvider 2021-10-08 15:23:36 -04:00
megan07 7dda3366a6
Update internal/backend/remote-state/gcs/backend.go
Co-authored-by: appilon <apilon@hashicorp.com>
2021-10-08 10:02:05 -05:00
James Bardin 03f71c2f06 fixup tests for MockProvider changes
Resetting the *Called fields and enforcing configuration broke a few
tests.
2021-10-08 08:42:06 -04:00
Megan Bang 81201d69a3 check valid json 2021-10-07 16:33:21 -05:00
Megan Bang 348c761bea add better error if credentials are invalid 2021-10-07 16:28:47 -05:00
James Bardin 22b400b8de skip refreshing deposed during destroy plan
The destroy plan should not require a configured provider (the complete
configuration is not evaluated, so they cannot be configured).

Deposed instances were being refreshed during the destroy plan, because
this instance type is only ever destroyed and shares the same
implementation between plan and walkPlanDestroy. Skip refreshing during
walkPlanDestroy.
2021-10-07 16:51:48 -04:00
James Bardin ad9944e523 test that providers are configured for calls
Have the MockProvider ensure that Configure is always called before any
methods that may require a configured provider.

Ensure the MockProvider *Called fields are zeroed out when re-using the
provider instance.
2021-10-07 16:48:56 -04:00
Alisdair McDiarmid b9f3dab035 json-output: Release format version 1.0 2021-10-06 11:13:06 -04:00
Omar Ismail bea7e3ebce
Backend State Migration: change variable names from one/two to source/destination (#29699) 2021-10-05 16:36:50 -04:00
Martin Atkins 01b22f4b76 command/e2etest: TestProviderTampering
We have various mechanisms that aim to ensure that the installed provider
plugins are consistent with the lock file and that the lock file is
consistent with the provider requirements, and we do have existing unit
tests for them, but all of those cases mock our fake out at least part of
the process and in the past that's caused us to miss usability
regressions, where we still catch the error but do so at the wrong layer
and thus generate error message lacking useful additional context.

Here we'll add some new end-to-end tests to supplement the existing unit
tests, making sure things work as expected when we assemble the system
together as we would in a release. These tests cover a number of different
ways in which the plugin selections can grow inconsistent.

These new tests all run only when we're in a context where we're allowed
to access the network, because they exercise the real plugin installer
codepath. We could technically build this to use a local filesystem mirror
or other such override to avoid that, but the point here is to make sure
we see the expected behavior in the main case, and so it's worth the
small additional cost of downloading the null provider from the real
registry.
2021-10-05 10:59:59 -07:00
Martin Atkins d09510a8fb command: Early error message for missing cache entries of locked providers
In the original incarnation of Meta.providerFactories we were returning
into a Meta.contextOpts whose signature didn't allow it to return an
error directly, and so we had compromised by making the provider factory
functions themselves return errors once called.

Subsequent work made Meta.contextOpts need to return an error anyway, but
at the time we neglected to update our handling of the providerFactories
result, having it still defer the error handling until we finally
instantiate a provider.

Although that did ultimately get the expected result anyway, the error
ended up being reported from deep in the guts of a Terraform Core graph
walk, in whichever concurrently-visited graph node happened to try to
instantiate the plugin first. This meant that the exact phrasing of the
error message would vary between runs and the reporting codepath didn't
have enough context to given an actionable suggestion on how to proceed.

In this commit we make Meta.contextOpts pass through directly any error
that Meta.providerFactories produces, and then make Meta.providerFactories
produce a special error type so that Meta.Backend can ultimately return
a user-friendly diagnostic message containing a specific suggestion to
run "terraform init", along with a short explanation of what a provider
plugin is.

The reliance here on an implied contract between two functions that are
not directly connected in the callstack is non-ideal, and so hopefully
we'll revisit this further in future work on the overall architecture of
the CLI layer. To try to make this robust in the meantime though, I wrote
it to use the errors.As function to potentially unwrap a wrapped version
of our special error type, in case one of the intervening layers is
changed at some point to wrap the downstream error before returning it.
2021-10-05 10:59:59 -07:00
James Bardin aece887a85
Merge pull request #29701 from hashicorp/jbardin/proposed-new-null-objs
objchange: fix ProposedNew from null objects
2021-10-05 13:09:49 -04:00
James Bardin 58d85fcc2e add test for planing unknown data source values 2021-10-05 12:31:23 -04:00
James Bardin 18d354223e objchange: fix ProposedNew from null objects
The codepath for AllAttributesNull was not correct for any nested object
types with collections, and should create single null values for the
correct NestingMode rather than a single object with null attributes.

Since there is no reason to descend into nested object types to create
nullv alues, we can drop the AllAttributesNull function altogether and
create null values as needed during ProposedNew.

The corresponding AllBlockAttributesNull was only called internally in 1
location, and simply delegated to schema.EmptyValue. We can reduce the
package surface area by dropping that function too and calling
EmptyValue directly.
2021-10-04 15:03:25 -04:00
James Bardin c68422d125
Merge pull request #29682 from hashicorp/jbardin/less-data-depends_on
Refine data depends_on dependency checks
2021-10-04 11:39:21 -04:00
Martin Atkins df578afd7e backend/local: Check dependency lock consistency before any operations
In historical versions of Terraform the responsibility to check this was
inside the terraform.NewContext function, along with various other
assorted concerns that made that function particularly complicated.

More recently, we reduced the responsibility of the "terraform" package
only to instantiating particular named plugins, assuming that its caller
is responsible for selecting appropriate versions of any providers that
_are_ external. However, until this commit we were just assuming that
"terraform init" had correctly selected appropriate plugins and recorded
them in the lock file, and so nothing was dealing with the problem of
ensuring that there haven't been any changes to the lock file or config
since the most recent "terraform init" which would cause us to need to
re-evaluate those decisions.

Part of the game here is to slightly extend the role of the dependency
locks object to also carry information about a subset of provider
addresses whose lock entries we're intentionally disregarding as part of
the various little edge-case features we have for overridding providers:
dev_overrides, "unmanaged providers", and the testing overrides in our
own unit tests. This is an in-memory-only annotation, never included in
the serialized plan files on disk.

I had originally intended to create a new package to encapsulate all of
this plugin-selection logic, including both the version constraint
checking here and also the handling of the provider factory functions, but
as an interim step I've just made version constraint consistency checks
the responsibility of the backend/local package, which means that we'll
always catch problems as part of preparing for local operations, while
not imposing these additional checks on commands that _don't_ run local
operations, such as "terraform apply" when in remote operations mode.
2021-10-01 14:43:58 -07:00
Martin Atkins 702413702c plans/planfile: Include dependency locks in saved plan files
We recently removed the legacy way we used to track the SHA256 hashes of
individual provider executables as part of a plans.Plan, because these
days we want to track the checksums of entire provider packages rather
than just the executable.

In order to achieve that new goal, we can save a copy of the dependency
lock information inside the plan file. This follows our existing precedent
of using exactly the same serialization formats we'd normally use for
this information, and thus we can reuse the existing models and
serializers and be confident we won't lose any detail in the round-trip.

As of this commit there's not yet anything actually making use of this
mechanism. In a subsequent commit we'll teach the main callers that write
and read plan files to include and expect (respectively) dependency
information, verifying that the available providers still match by the
time we're applying the plan.
2021-10-01 14:43:58 -07:00
Martin Atkins 3f85591998 depsfile: SaveLocksToBytes function
In a future commit we'll use this to include dependency lock information
in full fidelity inside a saved plan file.
2021-10-01 14:43:58 -07:00
Martin Atkins 6a98e4720c plans/planfile: Create takes most arguments via a struct type
Previously the planfile.Create function had accumulated probably already
too many positional arguments, and I'm intending to add another one in
a subsequent commit and so this is preparation to make the callsites more
readable (subjectively) and make it clearer how we can extend this
function's arguments to include further components in a plan file.

There's no difference in observable functionality here. This is just
passing the same set of arguments in a slightly different way.
2021-10-01 14:43:58 -07:00
Martin Atkins 8d193ad268 core: Simplify and centralize plugin availability checks
Historically the responsibility for making sure that all of the available
providers are of suitable versions and match the appropriate checksums has
been split rather inexplicably over multiple different layers, with some
of the checks happening as late as creating a terraform.Context.

We're gradually iterating towards making that all be handled in one place,
but in this step we're just cleaning up some old remnants from the
main "terraform" package, which is now no longer responsible for any
version or checksum verification and instead just assumes it's been
provided with suitable factory functions by its caller.

We do still have a pre-check here to make sure that we at least have a
factory function for each plugin the configuration seems to depend on,
because if we don't do that up front then it ends up getting caught
instead deep inside the Terraform runtime, often inside a concurrent
graph walk and thus it's not deterministic which codepath will happen to
catch it on a particular run.

As of this commit, this actually does leave some holes in our checks: the
command package is using the dependency lock file to make sure we have
exactly the provider packages we expect (exact versions and checksums),
which is the most crucial part, but we don't yet have any spot where
we make sure that the lock file is consistent with the current
configuration, and we are no longer preserving the provider checksums as
part of a saved plan.

Both of those will come in subsequent commits. While it's unusual to have
a series of commits that briefly subtracts functionality and then adds
back in equivalent functionality later, the lock file checking is the only
part that's crucial for security reasons, with everything else mainly just
being to give better feedback when folks seem to be using Terraform
incorrectly. The other bits are therefore mostly cosmetic and okay to be
absent briefly as we work towards a better design that is clearer about
where that responsibility belongs.
2021-10-01 14:43:58 -07:00
James Bardin 618e9cf8ec test for unexpected data reads 2021-09-30 17:13:33 -04:00
James Bardin 016463ea9c don't check all ancestors for data depends_on
Only depends_on ancestors for transitive dependencies when we're not
pointed directly at a resource. We can't be much more precise here,
since in order to maintain our guarantee that data sources will wait for
explicit dependencies, if those dependencies happen to be a module,
output, or variable, we have to find some upstream managed resource in
order to check for a planned change.
2021-09-30 16:43:09 -04:00
James Bardin fe671206cc Add detail about the protocol deprecation
Make sure it's clear that the deprecated fields serve no purpose, and
should be ignored.
2021-09-29 16:45:29 -04:00
Zach Whaley c9a5fdb366
cliconfig: Fix error message about invalid credentials helper type 2021-09-29 13:36:59 -07:00
James Bardin f93f16824c
Merge pull request #29665 from hashicorp/jbardin/required_version
Check required_version as early as possible
2021-09-29 08:18:20 -04:00
James Bardin ab0322e406 remove debugging println 2021-09-28 17:58:40 -04:00
James Bardin c2e0d265cf LoadModule now always returns the module
We don't need to load the configuration twice, since configload can
return the module for us.
2021-09-28 17:58:40 -04:00
James Bardin a53faf43f6 return partial config from LoadConfig with errors
LoadConfig should return any parsed configuration in order for the
caller to verify `required_version`.
2021-09-28 13:30:03 -04:00
James Bardin 625e768678 make sure required_version is checked before diags
We must ensure that the terraform required_version is checked as early
as possible, so that new configuration constructs don't cause init to
fail without indicating the version is incompatible.

The loadConfig call before the earlyconfig parsing seems to be unneeded,
and we can delay that to de-tangle it from installing the modules which
may have their own constraints.

TODO: it seems that loadConfig should be able to handle returning the
version constraints in the same manner as loadSingleModule.
2021-09-28 13:30:03 -04:00
Martin Atkins f60d55d6ad core: Emit only one warning for move collisions in destroy-plan mode
Our current implementation of destroy planning includes secretly running a
normal plan first, in order to get its effect of refreshing the state.

Previously our warning about colliding moves would betray that
implementation detail because we'd return it from both of our planning
operations here and thus show the message twice. That would also have
happened in theory for any other warnings emitted by both plan operations,
but it's the move collision warning that made it immediately visible.

We'll now only return warnings from the initial plan if we're also
returning errors from that plan, and thus the warnings of both plans can
never mix together into the same diags and thus we'll avoid duplicating
any warnings.

This does mean that we'd lose any warnings which might hypothetically
emerge only from the hidden normal plan and not from the subsequent
destroy plan, but we'll accept that as an okay tradeoff here because those
warnings are likely to not be super relevant to the destroy case anyway,
or else we'd emit them from the destroy-plan walk too.
2021-09-27 15:46:36 -07:00
James Bardin 372814e49a
Merge pull request #29659 from hashicorp/jbardin/really-nested-within
refactoring: exhaustive NestedWithin checks
2021-09-27 12:55:04 -04:00
James Bardin cac1f5c264 refactoring: exhaustive NestedWithin checks
When checking dependencies between statements, we need to check all
combinations of To and From addresses.
2021-09-27 12:48:17 -04:00
Alisdair McDiarmid a742d7ee88
Merge pull request #29649 from hashicorp/alisdair/json-ui-exhaustiveness
json-output: Add change reasons to explain deletes
2021-09-24 14:02:19 -04:00
Alisdair McDiarmid b699391d04 json-output: Add change reasons to explain deletes
The extra feedback information for why resource instance deletion is
planned is now included in the streaming JSON UI output.

We also add an explicit case for no-op actions to switch statements in
this package to ensure exhaustiveness, for future linting.
2021-09-24 13:17:49 -04:00
James Bardin 8ce2254ad6
Merge pull request #29647 from hashicorp/jbardin/test-temp-cleanup
temp path clean for some backend tests
2021-09-24 12:27:53 -04:00
James Bardin 9847eaa9cf remove usage of MinItems/MaxItems
MinItems and MaxItems are not used on nested types in the protocol, so
remove their usage in Terraform to prevent future confusion.
2021-09-24 12:26:00 -04:00
James Bardin 24a2bd6301 tfproto version 6.1
Minor version increase to deprecate min_items and max_items in nested
types.

Nested types have MinItems and MaxItems fields that were inherited from
the block implementation, but were never validated by Terraform, and are
not supported by the HCL decoder validations. Mark these fields as
deprecated, indicating that the SDK should handle the required
validation.
2021-09-24 11:14:57 -04:00
James Bardin c8cd0b1e74
Merge pull request #29624 from hashicorp/jbardin/no-block-to-attr
skip FixUpBlockAttrs when we can detect types from a new SDK
2021-09-24 11:01:53 -04:00
Alisdair McDiarmid 57318ef561 backend/remote: Support interop from 0.14 to 1.1
The previous conservative guarantee that we would not make backwards
incompatible changes to the state file format until at least Terraform
1.1 can now be extended. Terraform 0.14 through 1.1 will be able to
interoperably use state files, so we can update the remote backend
version compatibility check accordingly.
2021-09-24 09:26:09 -04:00
Alisdair McDiarmid f1e9d88ddc
Merge pull request #29640 from hashicorp/alisdair/fix-refresh-only-with-orphans
core: Fix refresh-only interaction with orphans
2021-09-24 09:25:46 -04:00
Martin Atkins d97ef10bb8 core: Don't return other errors if move statements are invalid
Because our validation rules depend on some dynamic results produced by
actually running the plan, we deal with moves in a "backwards" order where
we try to apply them first -- ignoring anything strange we might find --
and then validate the original statements only after planning.

An unfortunate consequence of that approach is that when the move
statements are invalid it's likely that move execution will not fully
complete, and so the generated plan is likely to be incorrect and might
well include errors resulting from the unresolved moves.

To mitigate that, here we let any move validation errors supersede all
other diagnostics that the plan phase might've generated, in the hope that
it'll help the user focus on fixing the incorrect move statements without
creating confusing by reporting errors that only appeared as a quick of
how Terraform worked around the invalid move statements earlier.
2021-09-23 14:37:08 -07:00
Martin Atkins 1bff623fd9 core: Report a warning if any moves get blocked
In most cases Terraform will be able to automatically fully resolve all
of the pending move statements before creating a plan, but there are some
edge cases where we can end up wanting to move one object to a location
where another object is already declared.

One relatively-obvious example is if someone uses "terraform state mv" in
order to create a set of resource instance bindings that could never have
arising in normal Terraform use.

A less obvious example arises from the interactions between moves at
different levels of granularity. If we are both moving a module to a new
address and moving a resource into an instance of the new module at the
same time, the old module might well have already had a resource of the
same name and so the resource move will be unresolvable.

In these situations Terraform will move the objects as far as possible,
but because it's never valid for a move "from" address to still be
declared in the configuration Terraform will inevitably always plan to
destroy the objects that didn't find a final home. To give some additional
explanation for that result, here we'll add a warning which describes
what happened.

This is not a particularly actionable warning because we don't really
have enough information to guess what the user intended, but we do at
least prompt that they might be able to use the "terraform state" family
of subcommands to repair the ambiguous situation before planning, if they
want a different result than what Terraform proposed.
2021-09-23 14:37:08 -07:00
Martin Atkins 04f9e7148c command/format: Include deletion reasons in plan report
The core runtime is now able to specify a reason for some situations when
Terraform plans to delete a resource instance.

This commit makes that information visible in the human-oriented UI. A
previous commit already made the underlying data informing these new hints
visible as part of the machine-oriented (JSON) plan output.

This also removes the bold formatting from the existing "has moved to"
hints, because subjectively it seemed like the result was emphasizing too
many parts of the output and thus somewhat defeating the benefit of the
emphasis in trying to create additional visual hierarchy for sighted users
running Terraform in a terminal. Now only the first line containing the
main action statement will be in bold, and all of the parenthesized
follow-up notes will be unformatted.
2021-09-23 14:37:08 -07:00
Martin Atkins a1a713cf28 core: Report ActionReasons when we plan to delete "orphans"
There are a few different reasons why a resource instance tracked in the
prior state might be considered an "orphan", but previously we reported
them all identically in the planned changes.

In order to help users understand the reason for a surprising planned
delete, we'll now try to specify an additional reason for the planned
deletion, covering all of the main reasons why that could happen.

This commit only introduces the new detail to the plans.Changes result,
though it also incidentally exposes it as part of the JSON plan result
in order to keep that working without returning errors in these new
cases. We'll expose this information in the human-oriented UI output in
a subsequent commit.
2021-09-23 14:37:08 -07:00
Martin Atkins 7b99861b1c refactoring: Don't implicitly move for resources with for_each
Our previous rule for implicitly moving from IntKey(0) to NoKey would
apply that move even when the current resource configuration uses
for_each, because we were only considering whether "count" were set.

Previously this was relatively harmless because the resource instance in
question would end up planned for deletion anyway: neither an IntKey nor
a NoKey are valid keys for for_each.

Now that we're going to be announcing these moves explicitly in the UI,
it would be confusing to see Terraform report that IntKey moved to NoKey
in a situation where the config changed from count to for_each, so to
address that we'll only generate the implied statement if neither
repetition argument is set.
2021-09-23 14:37:08 -07:00
James Bardin 9c078c27cf temp path clean for some backend tests 2021-09-23 17:16:33 -04:00
Alisdair McDiarmid ceb580ec40 core: Fix refresh-only interaction with orphans
When planning in refresh-only mode, we must not remove orphaned
resources due to changed count or for_each values from the planned
state. This was previously happening because we failed to pass through
the plan's skip-plan-changes flag to the instance orphan node.
2021-09-23 16:38:08 -04:00
Chris Arcand 171cdbbf93 command: Clean up testInputResponseMap before failing on unused answers
If you don't, the unused answers will persist in the package-level var
and bleed in to other tests.
2021-09-22 16:03:11 -05:00
Chris Arcand 60bc7aa05d command: Auto-select single workspace if necessary
When initializing a backend, if the currently selected workspace does
not exist, the user is prompted to select from the list of workspaces
the backend provides.

Instead, we should automatically select the only workspace available
_if_ that's all that's there.

Although with being a nice bit of polish, this enables future
improvments with Terraform Cloud in potentially removing the implicit
depenency on always using the 'default' workspace when the current
configuration is mapped to a single TFC workspace.
2021-09-22 16:03:11 -05:00
James Bardin 8706a18c4b refine the skipFixup heuristic
We can also rule out some attribute types as indicating something other
than the legacy SDK.

- Tuple types were not generated at all.
- There were no single objects types, the convention was to use a block
  list or set of length 1.
- Maps of objects were not possible to generate, since named blocks were
  not implemented.
- Nested collections were not supported, but when they were generated they
  would have primitive types.
2021-09-22 16:29:50 -04:00
James Bardin 6b4e73af48 skip the blocktoattr fixup with nested types
If structural types are being used, we can be assured that the legacy
SDK SchemaConfigModeAttr is not being used, and the fixup is not needed.

This prevents inadvertent mapping of blocks to structural attributes,
and allows us to skip the fixup overhead when possible.
2021-09-22 12:17:20 -04:00
Martin Atkins 83f0376673 refactoring: ApplyMoves new return type
When we originally stubbed ApplyMoves we didn't know yet how exactly we'd
be using the result, so we made it a double-indexed map allowing looking
up moves in both directions.

However, in practice we only actually need to look up old addresses by new
addresses, and so this commit first removes the double indexing so that
each move is only represented by one element in the map.

We also need to describe situations where a move was blocked, because in
a future commit we'll generate some warnings in those cases. Therefore
ApplyMoves now returns a MoveResults object which contains both a map of
changes and a map of blocks. The map of blocks isn't used yet as of this
commit, but we'll use it in a later commit to produce warnings within
the "terraform" package.
2021-09-22 09:01:10 -07:00
Martin Atkins d054102d38 addrs: AbsResource.UniqueKey distinct from AbsResourceInstance.UniqueKey
The whole point of UniqueKey is to deal with the fact that we have some
distinct address types which have an identical string representation, but
unfortunately that fact caused us to not notice that we'd incorrectly
made AbsResource.UniqueKey return a no-key instance UniqueKey instead of
its own distinct unique key type.
2021-09-22 09:01:10 -07:00
Chris Arcand 08a86b6c13
Merge pull request #29621 from hashicorp/error-on-unused-test-answers
command: Ensure all answers were used in command.testInputResponseMap
2021-09-22 09:42:21 -05:00
Chris Arcand 8684a85e26 command: Ensure all answers were used in command.testInputResponseMap
Remove answers from testInputResponse as they are given, and raise an
error during cleanup if any answers remain unused.

This enables tests to ensure that the expected mock answers are actually
used in a test; previously, an entire branch of code including an input
sequence could be omitted and the test(s) would not fail.

The only test that had unused answers in this map is one leftover from
legacy state migrations, a prompt that was removed in
7c93b2e5e6
2021-09-21 22:26:16 -05:00
Alisdair McDiarmid b59b057591 json-output: Config-driven move support in JSON UI
Add previous address information to the `planned_change` and
`resource_drift` messages for the streaming JSON UI output of plan and
apply operations.

Here we also add a "move" action value to the `change` object of these
messages, to represent a move-only operation.

As part of this work we also simplify this code to use the plan's
DriftedResources values instead of recomputing the drift from state.
2021-09-20 15:25:23 -04:00
Alisdair McDiarmid 78c4a8c461 json-output: Previous address for resource changes
Configuration-driven moves are represented in the plan file by setting
the resource's `PrevRunAddr` to a different value than its `Addr`. For
JSON plan output, we here add a new field to resource changes,
`previous_address`, which is present and non-empty only if the resource
is planned to be moved.

Like the CLI UI, refresh-only plans will include move-only changes in
the resource drift JSON output. In normal plan mode, these are elided to
avoid redundancy with planned changes.
2021-09-20 15:25:23 -04:00
Martin Atkins f0034beb33 core: refactoring.ImpliedMoveStatements replaces NodeCountBoundary
Going back a long time we've had a special magic behavior which tries to
recognize a situation where a module author either added or removed the
"count" argument from a resource that already has instances, and to
silently rename the zeroth or no-key instance so that we don't plan to
destroy and recreate the associated object.

Now we have a more general idea of "move statements", and specifically
the idea of "implied" move statements which replicates the same heuristic
we used to use for this behavior, we can treat this magic renaming rule as
just another "move statement", special only in that Terraform generates it
automatically rather than it being written out explicitly in the
configuration.

In return for wiring that in, we can now remove altogether the
NodeCountBoundary graph node type and its associated graph transformer,
CountBoundaryTransformer. We handle moves as a preprocessing step before
building the plan graph, so we no longer need to include any special nodes
in the graph to deal with that situation.

The test updates here are mainly for the graph builders themselves, to
acknowledge that indeed we're no longer inserting the NodeCountBoundary
vertices. The vertices that NodeCountBoundary previously depended on now
become dependencies of the special "root" vertex, although in many cases
here we don't see that explicitly because of the transitive reduction
algorithm, which notices when there's already an equivalent indirect
dependency chain and removes the redundant edge.

We already have plenty of test coverage for these "count boundary" cases
in the context tests whose names start with TestContext2Plan_count and
TestContext2Apply_resourceCount, all of which continued to pass here
without any modification and so are not visible in the diff. The test
functions particularly relevant to this situation are:
 - TestContext2Plan_countIncreaseFromNotSet
 - TestContext2Plan_countDecreaseToOne
 - TestContext2Plan_countOneIndex
 - TestContext2Apply_countDecreaseToOneCorrupted

The last of those in particular deals with the situation where we have
both a no-key instance _and_ a zero-key instance in the prior state, which
is interesting here because to exercises an intentional interaction
between refactoring.ImpliedMoveStatements and refactoring.ApplyMoves,
where we intentionally generate an implied move statement that produces
a collision and then expect ApplyMoves to deal with it in the same way as
it would deal with all other collisions, and thus ensure we handle both
the explicit and implied collisions in the same way.

This does affect some UI-level tests, because a nice side-effect of this
new treatment of this old feature is that we can now report explicitly
in the UI that we're assigning new addresses to these objects, whereas
before we just said nothing and hoped the user would just guess what had
happened and why they therefore weren't seeing a diff.

The backend/local plan tests actually had a pre-existing bug where they
were using a state with a different instance key than the config called
for but getting away with it because we'd previously silently fix it up.
That's still fixed up, but now done with an explicit mention in the UI
and so I made the state consistent with the configuration here so that the
tests would be able to recognize _real_ differences where present, as
opposed to the errant difference caused by that inconsistency.
2021-09-20 09:06:22 -07:00
Martin Atkins ee9e346039 refactoring: ApplyMoves skips moving when destination address occupied
Per our rule that the content of the state can never make a move statement
invalid, our behavior for two objects trying to occupy the same address
will be to just ignore that and let the object already at the address
take priority.

For the moment this is silent from an end-user perspective and appears
only in our internal logs. However, I'm hoping that our future planned
adjustment to the interface of this function will include some way to
allow reporting these collisions in some end-user-visible way, either as
a separate warning per collision or as a single warning that collects
together all of the collisions into a single message somehow.

This situation can arise both because the previous run state already
contained an object at the target address of a move and because more than
one move ends up trying to target the same location. In the latter case,
which one "wins" is decided by our depth-first traversal order, which is
in turn derived from our chaining and nesting rules and is therefore
arbitrary but deterministic.
2021-09-20 09:06:22 -07:00
Martin Atkins ef5a1c9cfe refactoring: ImpliedMoveStatements function
This new function complements the existing function FindMoveStatements
by potentially generating additional "implied" move statements that aren't
written explicit in the configuration but that we'll infer by comparing
the configuration and te previous run state.

The goal here is to infer only enough to replicate the effect of the
"count boundary fixup" graph node (terraform.NodeCountBoundary) that we
currently use to deal with this concern of preserving the zero-instance
when switching between "count" and not "count".

This is just dead code for now. A subsequent commit will introduce this
into the "terraform" package while also removing
terraform.NodeCountBoundary, thus achieving the same effect as before but
in a way that'll get reported in the UI as a move, using the same language
that we'd use for an explicit move statement.
2021-09-20 09:06:22 -07:00
Martin Atkins 7f99a8802e addrs: MoveEndpointInModule.SelectsResource
This is similar to the existing SelectsModule method, returning true if
the reciever selects either a particular resource as a whole or any of the
instances of that resource.

We don't need this test in the normal case, but we will need it in a
subsequent commit when we'll be possibly generating _implied_ move
statements between instances of resources, but only if there aren't
explicit move statements mentioning those resources already.
2021-09-20 09:06:22 -07:00
Alisdair McDiarmid 638784b195 cli: Omit move-only drift, except for refresh-only
The set of drifted resources now includes move-only changes, where the
object value is identical but a move has been executed. In normal
operation, we previousl displayed these moves twice: once as part of
drift output, and once as part of planned changes.

As of this commit we omit move-only changes from drift display, except
for refresh-only plans. This fixes the redundant output.
2021-09-17 14:47:00 -04:00
Alisdair McDiarmid 61b2d8e3fe core: Plan drift includes move-only changes
Previously, drifted resources included only updates and deletes. To
correctly display the full changes which would result as part of a
refresh-only apply, the drifted resources must also include move-only
changes.
2021-09-17 14:47:00 -04:00
Alisdair McDiarmid 9a7bbdab6f Fix terraform add test failure due to bad merge 2021-09-17 14:46:44 -04:00
Alisdair McDiarmid d425c26d77
Merge pull request #29589 from hashicorp/alisdair/planfile-drifted-resources
core: Compute resource drift during plan phase, store in plan file
2021-09-17 14:23:04 -04:00
Alisdair McDiarmid 1f57b7a8bd
Merge pull request #29235 from magodo/terraform_add_output_append
`terraform add`: `-out` option append to existing config & optionally check resource existance
2021-09-17 11:19:55 -04:00
James Bardin 1bd5987a8c
Merge pull request #29573 from hashicorp/renamed-typo
Correct terraform.env deprecation message typo
2021-09-16 17:02:20 -04:00
Alisdair McDiarmid d5b5407ccc format: Fix incorrect nesting of Color/Sprintf
Colorizing the result of an interpolated string can result in
incorrect output, if the values used to generate the string happen to
include color codes such as `[red]` or `[bold]`. Instead we should
always colorize the format string before calling functions like
`Sprintf`. This commit fixes all instances in this file.
2021-09-16 15:22:37 -04:00
Alisdair McDiarmid f0cf4235f9 cli: Refactor resource drift rendering 2021-09-16 15:22:37 -04:00
Alisdair McDiarmid c4688345a1 plans: Add resource drift to the plan file format 2021-09-16 15:22:37 -04:00
Alisdair McDiarmid bebf1ad23a core: Compute resource drift after plan walk
Rather than delaying resource drift detection until it is ready to be
presented, here we perform that computation after the plan walk has
completed. The resulting drift is represented like planned resource
changes, using a slice of ResourceInstanceChangeSrc values.
2021-09-16 15:22:37 -04:00
Martin Atkins e6a76d8ba0 core: Fail if a moved resource instance is excluded by -target
Because "moved" blocks produce changes that span across more than one
resource instance address at the same time, we need to take extra care
with them during planning.

The -target option allows for restricting Terraform's attention only to
a subset of resources when planning, as an escape hatch to recover from
bugs and mistakes.

However, we need to avoid any situation where only one "side" of a move
would be considered in a particular plan, because that'd create a new
situation that would be otherwise unreachable and would be difficult to
recover from.

As a compromise then, we'll reject an attempt to create a targeted plan if
the plan involves resolving a pending move and if the source address of
that move is not included in the targets.

Our error message offers the user two possible resolutions: to create an
untargeted plan, thus allowing everything to resolve, or to add additional
-target options to include just the existing resource instances that have
pending moves to resolve.

This compromise recognizes that it is possible -- though hopefully rare --
that a user could potentially both be recovering from a bug or mistake at
the same time as processing a move, if e.g. the bug was fixed by upgrading
a module and the new version includes a new "moved" block. In that edge
case, it might be necessary to just add the one additional address to
the targets rather than removing the targets altogether, if creating a
normal untargeted plan is impossible due to whatever bug they're trying to
recover from.
2021-09-16 08:57:59 -07:00
Martin Atkins b4594551f7 refactoring: TestValidateMoves/cyclic_chain can now pass
When originally filling out these test cases we didn't yet have the logic
in place to detect chained moves and so this test couldn't succeed in
spite of being correct.

We now have chain-detection implemented and so consequently we can also
detect cyclic chains. This commit largely just enables the original test
unchanged, although it does include the text of the final error message
for reporting cyclic move chains which wasn't yet finalized when we were
stubbing out this test case originally.
2021-09-15 08:46:12 -07:00
James Bardin 331dc8b14c handle empty containers in ProposedNew NestedTypes
Empty containers of NestedTypes were not handled in ProposedNew, causing
plans to be submitted with null values where there was configuration
present.
2021-09-15 08:33:25 -04:00
James Bardin 9029870613 handle NestedTypes in Block.CoerceValue
The CoerceValue code was not updated to handle NestedTypes, and while
none of the new codepaths make use of this method, there are still some
internal uses.
2021-09-14 16:02:50 -04:00
Martin Atkins 332ea1f233 backend/local: TestLocal_plan_context_error to fail terraform.NewContext
The original intent of this test was to verify that we properly release
the state lock if terraform.NewContext fails. This was in response to a
bug in an earlier version of Terraform where that wasn't true.

In the recent refactoring that made terraform.NewContext no longer
responsible for provider constraint/checksum verification, this test began
testing a failed plan operation instead, which left the error return path
from terraform.NewContext untested.

An invalid parallelism value is the one remaining case where
terraform.NewContext can return an error, so as a localized fix for this
test I've switched it to just intentionally set an invalid parallelism
value. This is still not ideal because it's still testing an
implementation detail, but I've at least left a comment inline to try to
be clearer about what the goal is here so that we can respond in a more
appropriate way if future changes cause this test to fail again.

In the long run I'd like to move this last remaining check out to be the
responsibility of the CLI layer, with terraform.NewContext either just
assuming the value correct or panicking when it isn't, but the handling
of this CLI option is currently rather awkwardly spread across the
command and backend packages so we'll save that refactoring for a later
date.
2021-09-14 10:35:08 -07:00
Martin Atkins 718fa3895f backend: Remove Operation.Parallelism field
The presence of this field was confusing because in practice the local
backend doesn't use it for anything and the remote backend was using it
only to return an error if it's set to anything other than the default,
under the assumption that it would always match ContextOpts.Parallelism.

The "command" package is the one actually responsible for handling this
option, and it does so by placing it into the partial ContextOpts which it
passes into the backend when preparing for a local operation. To make that
clearer, here we remove Operation.Parallelism and change the few uses of
it to refer to ContextOpts.Parallelism instead, so that everyone is
reading and writing this value from the same place.
2021-09-14 10:35:08 -07:00
Chris Arcand 172200808b Correct terraform.env deprecation message typo 2021-09-13 14:21:26 -05:00
James Bardin d0993b0e80 fix temp directory handling in some tests
Cleanup some more test fixtures to use t.TempDir

Use EvalSymlinks with temp dir paths to help with MacOS errors from
various terraform components.
2021-09-13 13:45:04 -04:00
James Bardin 8ed9a270e5
Merge pull request #29559 from hashicorp/jbardin/optional-attrs
Prevent `ObjectWithOptionalAttrs` from escaping
2021-09-13 08:58:11 -04:00
James Bardin 7f26531d4f variable types should always be populated 2021-09-13 08:51:32 -04:00
James Bardin 53a73a8ab6 configs: add ConstraintType to config.Variable
In order to handle optional attributes, the Variable type needs to keep
track of the type constraint for decoding and conversion, as well as the
concrete type for creating values and type comparison.

Since the Type field is referenced throughout the codebase, and for
future refactoring if the handling of optional attributes changes
significantly, the constraint is now loaded into an entirely new field
called ConstraintType. This prevents types containing
ObjectWithOptionalAttrs from escaping the decode/conversion codepaths
into the rest of the codebase.
2021-09-13 08:51:32 -04:00
James Bardin 43123d284e optional attrs should not be in ProposedNew 2021-09-13 08:51:32 -04:00
James Bardin af4f4540a9 configschema: do not expose optional attributes
Objects with optional attributes are only used for the decoding of
HCL, and those types should never be exposed elsewhere within
terraform.

Separate the external ImpliedType method from the cty.Type generated
internally for the decoder spec. This unfortunately causes our
ImpliedType method to return a different type than the
hcldec.ImpliedType function, but the former is only used within
terraform for concrete values, while the latter is used to decode
HCL. Renaming the ImpliedType methods could be done to further
differentiate them, but that does cause fairly large diff in the
codebase that does not seem worth the effort at this time.
2021-09-13 08:51:32 -04:00
Martin Atkins 65e0c448a0 workdir: Start of a new package for working directory state management
Thus far our various interactions with the bits of state we keep
associated with a working directory have all been implemented directly
inside the "command" package -- often in the huge command.Meta type -- and
not managed collectively via a single component.

There's too many little codepaths reading and writing from the working
directory and data directory to refactor it all in one step, but this is
an attempt at a first step towards a future where everything that reads
and writes from the current working directory would do so via an object
that encapsulates the implementation details and offers a high-level API
to read and write all of these session-persistent settings.

The design here continues our gradual path towards using a dependency
injection style where "package main" is solely responsible for directly
interacting with the OS command line, the OS environment, the OS working
directory, the stdio streams, and the CLI configuration, and then
communicating the resulting information to the rest of Terraform by wiring
together objects. It seems likely that eventually we'll have enough wiring
code in package main to justify a more explicit organization of that code,
but for this commit the new "workdir.Dir" object is just wired directly in
place of its predecessors, without any significant change of code
organization at that top layer.

This first commit focuses on the main files and directories we use to
find provider plugins, because a subsequent commit will lightly reorganize
the separation of concerns for plugin launching with a similar goal of
collecting all of the relevant logic together into one spot.
2021-09-10 14:56:49 -07:00
Martin Atkins 343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
Martin Atkins 38ec730b0e core: Opportunistic schema loading during graph construction
Previously the graph builders all expected to be given a full manifest
of all of the plugin component schemas that they could need during their
analysis work. That made sense when terraform.NewContext would always
proactively load all of the schemas before doing any other work, but we
now have a load-as-needed strategy for schemas.

We'll now have the graph builders use the contextPlugins object they each
already hold to retrieve individual schemas when needed. This avoids the
need to prepare a redundant data structure to pass alongside the
contextPlugins object, and leans on the memoization behavior inside
contextPlugins to preserve the old behavior of loading each provider's
schema only once.
2021-09-10 14:56:49 -07:00
Martin Atkins a59c2fe1b9 core: EvalContextBuiltin no longer has a "Schemas"
By tolerating ProviderSchema and ProvisionerSchema potentially returning
errors, we can slightly simplify EvalContextBuiltin by having it retrieve
individual schemas when needed directly from the "Plugins" object.

EvalContextBuiltin already needs to be holding a contextPlugins instance
for other reasons anyway, so this allows us to get the same result with
fewer moving parts.
2021-09-10 14:56:49 -07:00
Martin Atkins 2bf1de1f5d core: Context.Schemas in terms of contextPlugins methods
The responsibility for actually instantiating a single plugin and reading
out its schema now belongs to the contextPlugins type, which memoizes the
results by each plugin's unique identifier so that we can avoid retrieving
the same schemas multiple times when working with the same context.

This doesn't change the API of Context.Schemas but it does restore the
spirit of an earlier version of terraform.Context which did all of the
schema loading proactively inside terraform.NewContext. In an earlier
commit we reduced the scope of terraform.NewContext, making schema loading
a separate step, but in the process of doing that removed the effective
memoization of the schema results that terraform.NewContext was providing.

The memoization here will play out in a different way than before, because
we'll be treating each plugin call as separate rather than proactively
loading them all up front, but this is effectively the same because all
of our operation methods on Context call Context.Schemas early in their
work and thus end up forcing all of the necessary schemas to load up
front nonetheless.
2021-09-10 14:56:49 -07:00
Martin Atkins 80b3fcf93e core: Replace contextComponentFactory with contextPlugins
In the v0.12 timeframe we made contextComponentFactory an interface with
the expectation that we'd write mocks of it for tests, but in practice we
ended up just always using the same "basicComponentFactory" implementation
throughout.

In the interests of simplification then, here we replace that interface
and its sole implementation with a new concrete struct type
contextPlugins.

Along with the general benefit that this removes an unneeded indirection,
this also means that we can add additional methods to the struct type
without the usual restriction that interface types prefer to be small.
In particular, in a future commit I'm planning to add methods for loading
provider and provisioner schemas, working with the currently-unused new
fields this commit has included in contextPlugins, as compared to its
predecessor basicComponentFactory.
2021-09-10 14:56:49 -07:00
Martin Atkins dcfa077adf core: contextComponentFactory doesn't need to enumerate components
In earlier Terraform versions we used the set of all available plugins of
each type to make graph-building decisions, but in modern Terraform we
make those decisions based entirely on the configuration.

Consequently, we no longer need the methods which can enumerate all of the
known plugin components of a given type. Instead, we just try to
instantiate each of the plugins that the configuration refers to and then
handle the error when that fails, which typically means that the user
needs to run "terraform init" to install some new plugins.
2021-09-10 14:56:49 -07:00
Martin Atkins d51921f085 core: Provider transformers don't use the set of all available providers
In earlier incarnations of these transformers we used the set of all
available providers for tasks such as generating implied provider
configuration nodes.

However, in modern Terraform we can extract all of the information we need
from the configuration itself, and so these transformers weren't actually
using this set of provider addresses.

These also ended up getting left behind as sets of string rather than sets
of addrs.Provider in our earlier refactoring work, which didn't really
matter because the result wasn't used anywhere anyway.

Rather than updating these to use addrs.Provider instead, I've just
removed the unused arguments entirely in the hope of making it easier to
see what inputs these transformers use to make their decisions.
2021-09-10 14:56:49 -07:00
Martin Atkins feb219a9c4 core: Un-export the LoadSchemas function
The public interface for loading schemas is Context.Schemas, which can
take into account the context's records of which plugin versions and
checksums we're expecting. loadSchemas is an implementation detail of
that, representing the part we run only after we've verified all of the
plugins.
2021-09-10 14:56:49 -07:00
James Bardin 782132da33 remove incorrect computed check
The config is already validated, and does not need to be checked in
AssertPlanValid. Add some more coverage for plan validation.
2021-09-10 14:44:07 -04:00
Alisdair McDiarmid cd29c3e5fd
Revert "json-output: Release format version 1.0" 2021-09-09 11:25:35 -04:00
Laura Pacilio a819d7db3a
Merge pull request #29502 from hashicorp/alisdair/json-format-version
json-output: Release format version 1.0
2021-09-09 11:06:58 -04:00
Alisdair McDiarmid 7f3a29b46e
Merge pull request #29523 from hashicorp/alisdair/moved-ui
command: Render "moved" annotations in plan UI
2021-09-07 14:28:11 -04:00
James Bardin 1ad0421e15
Merge pull request #29522 from hashicorp/jbardin/json-blocktoattr
allow json output to marshal ConfigModeAttr blocks
2021-09-07 11:32:21 -04:00
Alisdair McDiarmid 29999c1d6f command: Render "moved" annotations in plan UI
For resources which are planned to move, render the previous run address
as additional information in the plan UI. For the case of a move-only
resource (which otherwise is unchanged), we also render that as a
planned change, but without any corresponding action symbol.

If all changes in the plan are moves without changes, the plan is no
longer considered "empty". In this case, we skip rendering the action
symbols in the UI.
2021-09-03 17:44:07 -04:00
James Bardin ac2a870ea0 allow json output to marshal ConfigModeAttr blocks
In order to marshal config blocks using ConfigModeAttr, we need to
insert the fixup body to map hcl blocks to the attribute in the schema.
2021-09-03 13:53:52 -04:00
Alisdair McDiarmid 06956773dc configs: Disable experiment warnings at link time
This package level variable can be overridden at link time to allow
temporarily disabling the UI warning when experimental features are
enabled. This makes it easier to understand how UI will render when the
feature is no longer experimental.

This change is only for those developing Terraform.
2021-09-03 12:11:06 -04:00
Alisdair McDiarmid b60f201eed json-output: Release format version 1.0 2021-09-01 15:15:18 -04:00
James Bardin 863963e7a6 de-linting 2021-09-01 11:36:21 -04:00
James Bardin 7c02ef6220
Merge pull request #29482 from hashicorp/jbardin/computed-obj-attrs
Computed values within nested object types in `AssertPlanValid`
2021-08-31 11:26:52 -04:00
James Bardin f195ce7fd4 remove temp test 2021-08-31 11:17:32 -04:00
Martin Atkins 7803f69d42 core: Enable TestContext2Plan_movedResourceBasic
This is the first test exercising the basic functionality of config-driven
move. We previously had it skipped because Terraform's previous design
of treating all three of the state artifacts as mutable attributes of
terraform.Context meant that it was too late during planning to deal with
the move operations, and thus this test was failing.

Thanks to the previous commit, which changes the terraform.Context API
such that we can defer creating the three state artifacts until we're
already doing planning, this test now works and shows Terraform correctly
handling a resource that was formerly called "a" and is now called "b",
with a "moved" block recording that renaming.
2021-08-30 13:59:14 -07:00
Martin Atkins 89b05050ec core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.

This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.

Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.

However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.

My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
 - The "terraform graph" command is no longer hooked directly into the
   core graph builders, because that's no longer part of the public API.
   However, I did include a couple new Context functions whose contract
   is to produce a UI-oriented graph, and _for now_ those continue to
   return the physical graph we use for those operations. There's no
   exported API for generating the "validate" and "eval" graphs, because
   neither is particularly interesting in its own right, and so
   "terraform graph" no longer supports those graph types.
 - terraform.NewContext no longer has the responsibility for collecting
   all of the provider schemas up front. Instead, we wait until we need
   them. However, that means that some of our error messages now have a
   slightly different shape due to unwinding through a differently-shaped
   call stack. As of this commit we also end up reloading the schemas
   multiple times in some cases, which is functionally acceptable but
   likely represents a performance regression. I intend to rework this to
   use caching, but I'm saving that for a later commit because this one is
   big enough already.

The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.

However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-30 13:59:14 -07:00
Martin Atkins 4faac6ee43 core: Record move result information in the plan
Here we wire through the "move results" into the graph walk data
structures so that all of the the nodes which produce
plans.ResourceInstanceChange values can capture the "PrevRunAddr" for
each resource instance.

This doesn't actually quite work yet, because the logic in Context.Plan
isn't actually correct and so the updated state from
refactoring.ApplyMoves isn't actually visible as the "previous run state".
For that reason, the context test in this commit is currently skipped,
with the intent of re-enabling it once the updated state is properly
propagating into the plan graph walk and thus we can actually react to
the result of the move while choosing actions for those addresses.
2021-08-30 13:59:14 -07:00
Martin Atkins 22b36d1f4c Field for the previous address of each resource instance in the plan
In order to expose the effect of any relevant "moved" statements we dealt
with prior to creating the plan, we'll record with each
ResourceInstanceChange both is current address and the address it was
tracked at for the previous run.

To save consumers of these objects from having to special-case the
situation where there _was_ no previous run (e.g. because this is a Create
change), we'll just pretend the previous run address was the same as the
current address in that case, the same as for an update without any
renaming in effect.

This includes a breaking change to the plan file format, but one that
doesn't require a version number increment because there is no ambiguity
between the two formats and so mismatched parsers will already fail with
an error message.

As of this commit we've just added the new field but not yet populated it
with any useful information: it always just matches Addr. A future commit
will wire this up to the result of applying the moves so that we can
populate it correctly. We also don't yet expose this new information
anywhere in the UI layer.
2021-08-30 13:59:14 -07:00
James Bardin ea68d79ea2 nested object values can be computed
While blocks were not allowed to be computed by the provider, nested
objects can be. Remove the errors regarding blocks and verify unknown
values are valid.
2021-08-30 14:30:07 -04:00
James Bardin 903797084a objects vs maps in nested object types
When using NestedMap objects, unify the codepath for both maps and
objects as they may be interchangeable.
2021-08-30 14:30:02 -04:00
James Bardin 7ca6be8285 correctly verify planned nested object values
The validation for nested object types with computed attributes was
using the incorrect function call.
2021-08-30 14:29:15 -04:00
Martin Atkins ce96d82de0 build: Centralize our protobuf compilation steps
We have a few different .proto files in this repository that all need to
get recompiled into .pb.go files each time we change them, but we were
previously handling that with some scripts that just assumed that protoc
and the relevant plugins were already installed on the system somewhere,
at the right versions.

In practice we've been constantly flopping between different versions of
these tools due to folks having different versions installed in their
development environments. In particular, the state of the .pb.go files
in the prior commit wasn't reproducible by any single version of the tools
because they've all slightly diverged from one another.

In the interests of being more consistent here and avoiding accidental
inconsistencies, we'll now centralize the protocol buffer compile steps
all into a single tool that knows how to fetch and install the expected
versions of the various tools we need and then run those tools with the
right options to get a stable result.

If we want to upgrade to either a newer protoc or a newer protoc-gen-go
in future then we'll do that in a central location and update all of the
.pb.go files at the same time, so that we're always consistently tracking
the same version of protocol buffers everywhere.

While doing this I attempted to keep as close as possible to the toolchain
we'd most recently used, but since they were not consistent with each
other they've now all changed which version numbers they record at minimum,
and the planproto stub in particular now also has a slightly different
descriptor serialization but is otherwise offering the same API.
2021-08-20 16:18:48 -07:00
James Bardin 249e05f827
Merge pull request #29437 from hashicorp/jbardin/move
refactoring: complete CanChainFrom and NestedWithin
2021-08-20 17:54:27 -04:00
Alisdair McDiarmid 40dd4292b8
Merge pull request #29438 from hashicorp/alisdair/init-force-copy-multiple-workspaces
command: Suppress prompt for init -force-copy
2021-08-20 15:21:43 -04:00
James Bardin 30afb492ab fix ApplyMove test with nested modules working
The addrs package can now correctly handle the combinations of nested
module endpoints, which fixes these 2 tests.
2021-08-20 15:17:06 -04:00