Commit Graph

82 Commits

Author SHA1 Message Date
Martin Atkins 04f9e7148c command/format: Include deletion reasons in plan report
The core runtime is now able to specify a reason for some situations when
Terraform plans to delete a resource instance.

This commit makes that information visible in the human-oriented UI. A
previous commit already made the underlying data informing these new hints
visible as part of the machine-oriented (JSON) plan output.

This also removes the bold formatting from the existing "has moved to"
hints, because subjectively it seemed like the result was emphasizing too
many parts of the output and thus somewhat defeating the benefit of the
emphasis in trying to create additional visual hierarchy for sighted users
running Terraform in a terminal. Now only the first line containing the
main action statement will be in bold, and all of the parenthesized
follow-up notes will be unformatted.
2021-09-23 14:37:08 -07:00
Martin Atkins a1a713cf28 core: Report ActionReasons when we plan to delete "orphans"
There are a few different reasons why a resource instance tracked in the
prior state might be considered an "orphan", but previously we reported
them all identically in the planned changes.

In order to help users understand the reason for a surprising planned
delete, we'll now try to specify an additional reason for the planned
deletion, covering all of the main reasons why that could happen.

This commit only introduces the new detail to the plans.Changes result,
though it also incidentally exposes it as part of the JSON plan result
in order to keep that working without returning errors in these new
cases. We'll expose this information in the human-oriented UI output in
a subsequent commit.
2021-09-23 14:37:08 -07:00
Chris Arcand 171cdbbf93 command: Clean up testInputResponseMap before failing on unused answers
If you don't, the unused answers will persist in the package-level var
and bleed in to other tests.
2021-09-22 16:03:11 -05:00
Chris Arcand 60bc7aa05d command: Auto-select single workspace if necessary
When initializing a backend, if the currently selected workspace does
not exist, the user is prompted to select from the list of workspaces
the backend provides.

Instead, we should automatically select the only workspace available
_if_ that's all that's there.

Although with being a nice bit of polish, this enables future
improvments with Terraform Cloud in potentially removing the implicit
depenency on always using the 'default' workspace when the current
configuration is mapped to a single TFC workspace.
2021-09-22 16:03:11 -05:00
Chris Arcand 08a86b6c13
Merge pull request #29621 from hashicorp/error-on-unused-test-answers
command: Ensure all answers were used in command.testInputResponseMap
2021-09-22 09:42:21 -05:00
Chris Arcand 8684a85e26 command: Ensure all answers were used in command.testInputResponseMap
Remove answers from testInputResponse as they are given, and raise an
error during cleanup if any answers remain unused.

This enables tests to ensure that the expected mock answers are actually
used in a test; previously, an entire branch of code including an input
sequence could be omitted and the test(s) would not fail.

The only test that had unused answers in this map is one leftover from
legacy state migrations, a prompt that was removed in
7c93b2e5e6
2021-09-21 22:26:16 -05:00
Alisdair McDiarmid b59b057591 json-output: Config-driven move support in JSON UI
Add previous address information to the `planned_change` and
`resource_drift` messages for the streaming JSON UI output of plan and
apply operations.

Here we also add a "move" action value to the `change` object of these
messages, to represent a move-only operation.

As part of this work we also simplify this code to use the plan's
DriftedResources values instead of recomputing the drift from state.
2021-09-20 15:25:23 -04:00
Alisdair McDiarmid 78c4a8c461 json-output: Previous address for resource changes
Configuration-driven moves are represented in the plan file by setting
the resource's `PrevRunAddr` to a different value than its `Addr`. For
JSON plan output, we here add a new field to resource changes,
`previous_address`, which is present and non-empty only if the resource
is planned to be moved.

Like the CLI UI, refresh-only plans will include move-only changes in
the resource drift JSON output. In normal plan mode, these are elided to
avoid redundancy with planned changes.
2021-09-20 15:25:23 -04:00
Martin Atkins f0034beb33 core: refactoring.ImpliedMoveStatements replaces NodeCountBoundary
Going back a long time we've had a special magic behavior which tries to
recognize a situation where a module author either added or removed the
"count" argument from a resource that already has instances, and to
silently rename the zeroth or no-key instance so that we don't plan to
destroy and recreate the associated object.

Now we have a more general idea of "move statements", and specifically
the idea of "implied" move statements which replicates the same heuristic
we used to use for this behavior, we can treat this magic renaming rule as
just another "move statement", special only in that Terraform generates it
automatically rather than it being written out explicitly in the
configuration.

In return for wiring that in, we can now remove altogether the
NodeCountBoundary graph node type and its associated graph transformer,
CountBoundaryTransformer. We handle moves as a preprocessing step before
building the plan graph, so we no longer need to include any special nodes
in the graph to deal with that situation.

The test updates here are mainly for the graph builders themselves, to
acknowledge that indeed we're no longer inserting the NodeCountBoundary
vertices. The vertices that NodeCountBoundary previously depended on now
become dependencies of the special "root" vertex, although in many cases
here we don't see that explicitly because of the transitive reduction
algorithm, which notices when there's already an equivalent indirect
dependency chain and removes the redundant edge.

We already have plenty of test coverage for these "count boundary" cases
in the context tests whose names start with TestContext2Plan_count and
TestContext2Apply_resourceCount, all of which continued to pass here
without any modification and so are not visible in the diff. The test
functions particularly relevant to this situation are:
 - TestContext2Plan_countIncreaseFromNotSet
 - TestContext2Plan_countDecreaseToOne
 - TestContext2Plan_countOneIndex
 - TestContext2Apply_countDecreaseToOneCorrupted

The last of those in particular deals with the situation where we have
both a no-key instance _and_ a zero-key instance in the prior state, which
is interesting here because to exercises an intentional interaction
between refactoring.ImpliedMoveStatements and refactoring.ApplyMoves,
where we intentionally generate an implied move statement that produces
a collision and then expect ApplyMoves to deal with it in the same way as
it would deal with all other collisions, and thus ensure we handle both
the explicit and implied collisions in the same way.

This does affect some UI-level tests, because a nice side-effect of this
new treatment of this old feature is that we can now report explicitly
in the UI that we're assigning new addresses to these objects, whereas
before we just said nothing and hoped the user would just guess what had
happened and why they therefore weren't seeing a diff.

The backend/local plan tests actually had a pre-existing bug where they
were using a state with a different instance key than the config called
for but getting away with it because we'd previously silently fix it up.
That's still fixed up, but now done with an explicit mention in the UI
and so I made the state consistent with the configuration here so that the
tests would be able to recognize _real_ differences where present, as
opposed to the errant difference caused by that inconsistency.
2021-09-20 09:06:22 -07:00
Alisdair McDiarmid 638784b195 cli: Omit move-only drift, except for refresh-only
The set of drifted resources now includes move-only changes, where the
object value is identical but a move has been executed. In normal
operation, we previousl displayed these moves twice: once as part of
drift output, and once as part of planned changes.

As of this commit we omit move-only changes from drift display, except
for refresh-only plans. This fixes the redundant output.
2021-09-17 14:47:00 -04:00
Alisdair McDiarmid 9a7bbdab6f Fix terraform add test failure due to bad merge 2021-09-17 14:46:44 -04:00
Alisdair McDiarmid d425c26d77
Merge pull request #29589 from hashicorp/alisdair/planfile-drifted-resources
core: Compute resource drift during plan phase, store in plan file
2021-09-17 14:23:04 -04:00
Alisdair McDiarmid 1f57b7a8bd
Merge pull request #29235 from magodo/terraform_add_output_append
`terraform add`: `-out` option append to existing config & optionally check resource existance
2021-09-17 11:19:55 -04:00
Alisdair McDiarmid d5b5407ccc format: Fix incorrect nesting of Color/Sprintf
Colorizing the result of an interpolated string can result in
incorrect output, if the values used to generate the string happen to
include color codes such as `[red]` or `[bold]`. Instead we should
always colorize the format string before calling functions like
`Sprintf`. This commit fixes all instances in this file.
2021-09-16 15:22:37 -04:00
Alisdair McDiarmid f0cf4235f9 cli: Refactor resource drift rendering 2021-09-16 15:22:37 -04:00
Martin Atkins 718fa3895f backend: Remove Operation.Parallelism field
The presence of this field was confusing because in practice the local
backend doesn't use it for anything and the remote backend was using it
only to return an error if it's set to anything other than the default,
under the assumption that it would always match ContextOpts.Parallelism.

The "command" package is the one actually responsible for handling this
option, and it does so by placing it into the partial ContextOpts which it
passes into the backend when preparing for a local operation. To make that
clearer, here we remove Operation.Parallelism and change the few uses of
it to refer to ContextOpts.Parallelism instead, so that everyone is
reading and writing this value from the same place.
2021-09-14 10:35:08 -07:00
James Bardin d0993b0e80 fix temp directory handling in some tests
Cleanup some more test fixtures to use t.TempDir

Use EvalSymlinks with temp dir paths to help with MacOS errors from
various terraform components.
2021-09-13 13:45:04 -04:00
Martin Atkins 65e0c448a0 workdir: Start of a new package for working directory state management
Thus far our various interactions with the bits of state we keep
associated with a working directory have all been implemented directly
inside the "command" package -- often in the huge command.Meta type -- and
not managed collectively via a single component.

There's too many little codepaths reading and writing from the working
directory and data directory to refactor it all in one step, but this is
an attempt at a first step towards a future where everything that reads
and writes from the current working directory would do so via an object
that encapsulates the implementation details and offers a high-level API
to read and write all of these session-persistent settings.

The design here continues our gradual path towards using a dependency
injection style where "package main" is solely responsible for directly
interacting with the OS command line, the OS environment, the OS working
directory, the stdio streams, and the CLI configuration, and then
communicating the resulting information to the rest of Terraform by wiring
together objects. It seems likely that eventually we'll have enough wiring
code in package main to justify a more explicit organization of that code,
but for this commit the new "workdir.Dir" object is just wired directly in
place of its predecessors, without any significant change of code
organization at that top layer.

This first commit focuses on the main files and directories we use to
find provider plugins, because a subsequent commit will lightly reorganize
the separation of concerns for plugin launching with a similar goal of
collecting all of the relevant logic together into one spot.
2021-09-10 14:56:49 -07:00
Martin Atkins 343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
Martin Atkins 80b3fcf93e core: Replace contextComponentFactory with contextPlugins
In the v0.12 timeframe we made contextComponentFactory an interface with
the expectation that we'd write mocks of it for tests, but in practice we
ended up just always using the same "basicComponentFactory" implementation
throughout.

In the interests of simplification then, here we replace that interface
and its sole implementation with a new concrete struct type
contextPlugins.

Along with the general benefit that this removes an unneeded indirection,
this also means that we can add additional methods to the struct type
without the usual restriction that interface types prefer to be small.
In particular, in a future commit I'm planning to add methods for loading
provider and provisioner schemas, working with the currently-unused new
fields this commit has included in contextPlugins, as compared to its
predecessor basicComponentFactory.
2021-09-10 14:56:49 -07:00
Alisdair McDiarmid cd29c3e5fd
Revert "json-output: Release format version 1.0" 2021-09-09 11:25:35 -04:00
Laura Pacilio a819d7db3a
Merge pull request #29502 from hashicorp/alisdair/json-format-version
json-output: Release format version 1.0
2021-09-09 11:06:58 -04:00
Alisdair McDiarmid 7f3a29b46e
Merge pull request #29523 from hashicorp/alisdair/moved-ui
command: Render "moved" annotations in plan UI
2021-09-07 14:28:11 -04:00
James Bardin 1ad0421e15
Merge pull request #29522 from hashicorp/jbardin/json-blocktoattr
allow json output to marshal ConfigModeAttr blocks
2021-09-07 11:32:21 -04:00
Alisdair McDiarmid 29999c1d6f command: Render "moved" annotations in plan UI
For resources which are planned to move, render the previous run address
as additional information in the plan UI. For the case of a move-only
resource (which otherwise is unchanged), we also render that as a
planned change, but without any corresponding action symbol.

If all changes in the plan are moves without changes, the plan is no
longer considered "empty". In this case, we skip rendering the action
symbols in the UI.
2021-09-03 17:44:07 -04:00
James Bardin ac2a870ea0 allow json output to marshal ConfigModeAttr blocks
In order to marshal config blocks using ConfigModeAttr, we need to
insert the fixup body to map hcl blocks to the attribute in the schema.
2021-09-03 13:53:52 -04:00
Alisdair McDiarmid b60f201eed json-output: Release format version 1.0 2021-09-01 15:15:18 -04:00
James Bardin 863963e7a6 de-linting 2021-09-01 11:36:21 -04:00
Martin Atkins 89b05050ec core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.

This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.

Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.

However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.

My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
 - The "terraform graph" command is no longer hooked directly into the
   core graph builders, because that's no longer part of the public API.
   However, I did include a couple new Context functions whose contract
   is to produce a UI-oriented graph, and _for now_ those continue to
   return the physical graph we use for those operations. There's no
   exported API for generating the "validate" and "eval" graphs, because
   neither is particularly interesting in its own right, and so
   "terraform graph" no longer supports those graph types.
 - terraform.NewContext no longer has the responsibility for collecting
   all of the provider schemas up front. Instead, we wait until we need
   them. However, that means that some of our error messages now have a
   slightly different shape due to unwinding through a differently-shaped
   call stack. As of this commit we also end up reloading the schemas
   multiple times in some cases, which is functionally acceptable but
   likely represents a performance regression. I intend to rework this to
   use caching, but I'm saving that for a later commit because this one is
   big enough already.

The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.

However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-30 13:59:14 -07:00
Alisdair McDiarmid 40dd4292b8
Merge pull request #29438 from hashicorp/alisdair/init-force-copy-multiple-workspaces
command: Suppress prompt for init -force-copy
2021-08-20 15:21:43 -04:00
Alisdair McDiarmid 2762a940c0 command: Suppress prompt for init -force-copy
The -force-copy flag to init should automatically migrate state.
Previously this was not applied to one case: when migrating from
a backend with multiple workspaces to another backend supporting
multiple workspaces. I believe this was an oversight so this commit
fixes that.
2021-08-20 14:46:09 -04:00
Alisdair McDiarmid 70a4f7a6b6 command: Fix stale lock after running add 2021-08-20 13:22:11 -04:00
Martin Atkins 43d753e727 command: "terraform add" is experimental
We're aware of several quirks of this command's current design, which
result from some existing architectural limitations that we can't address
immediately.

However, we do still want to make this command available in its current
capacity as an incremental improvement, so as a compromise we'll document
it as experimental. Our intent here is to exclude it from the
Terraform 1.0 Compatibility Promises so that we can have the space to
continue to improve the design as other parts of the overall Terraform
system gain new capabilities.

We don't currently have any concrete plan for this command to be
stabilized and subject to compatibility promises. That decision will
follow from ongoing discussions with other teams whose systems may need to
change in order to support the final design of "terraform add".
2021-08-19 09:27:30 -07:00
James Bardin 68ed50616e handle null and unknown values in attr diffs
The code adopted from block diffs was not set to handle null and unknown
values, as those are not allowed for blocks.

We also revert the change to formatting nested object types as single
attributes, because the attribute formatter cannot handle sensitive
values from the schema. This presents some awkward syntax for diffs for
now, but should suffice until the entire formatter can be refactored to
better handle these new nested types.
2021-08-18 14:12:01 -04:00
Martin Atkins 383bbdeebc Upgrade to Go 1.17
This includes the addition of the new "//go:build" comment form in addition
to the legacy "// +build" notation, as produced by gofmt to ensure
consistent behavior between Go versions. The new directives are all
equivalent to what was present before, so there's no change in behavior.

Go 1.17 continues to use the Unicode 13 tables as in Go 1.16, so this
upgrade does not require also upgrading our Unicode-related dependencies.

This upgrade includes the following breaking changes which will also
appear as breaking changes for Terraform users, but that are consistent
with the Terraform v1.0 compatibility promises.

- On MacOS, Terraform now requires macOS 10.13 High Sierra or later.

This upgrade also includes the following breaking changes which will
appear as breaking changes for Terraform users that are inconsistent with
our compatibility promises, but have justified exceptions as follows:

- cidrsubnet, cidrhost, and cidrnetmask will now reject IPv4 CIDR
  addresses whose decimal components have leading zeros, where previously
  they would just silently ignore those leading zeros.

  This is a security-motivated exception to our compatibility promises,
  because some external systems interpret zero-prefixed octets as octal
  numbers rather than decimal, and thus the previous lenient parsing could
  lead to a different interpretation of the address between systems, and
  thus potentially allow bypassing policy when configuring firewall rules
  etc.

This upgrade also includes the following breaking changes which could
_potentially_ appear as breaking changes for Terraform users, but that do
not in practice for the reasons given:

- The Go net/url package no longer allows query strings with pairs
  separated by semicolons instead of ampersands. This primarily affects
  HTTP servers written in Go, and Terraform includes a special temporary
  HTTP server as part of its implementation of OAuth for "terraform login",
  but that server only needs to accept URLs created by Terraform itself
  and Terraform does not generate any URLs that would be rejected.
2021-08-17 15:20:05 -07:00
James Bardin a94155d0ca
Merge pull request #29397 from hashicorp/jbardin/format-id-name-marks
unmark object ID or Name for formatting
2021-08-17 14:58:56 -04:00
James Bardin a48b024c0a unmark object ID or Name for formatting 2021-08-17 12:24:43 -04:00
James Bardin 296a757ab4 check for null sets in diff rendering 2021-08-16 18:25:16 -04:00
James Bardin fbfb14142e render empty nested containers as attributes
Don't try to break down containers that are empty to render the diff, so
we can avoid having to check for empty vs null in all cases.
2021-08-16 18:13:55 -04:00
Alisdair McDiarmid 3b33dc1105 json-output: Add output changes to plan logs
Extend the outputs JSON log message to support an `action` field (and
make the `type` and `value` fields optional). This allows us to emit a
useful output change summary as part of the plan, bringing the JSON log
output into parity with the text output.

While we do have access to the before/after values in the output
changes, attempting to wedge those into a structured log message is not
appropriate. That level of detail can be extracted from the JSON plan
output from `terraform show -json`.
2021-08-05 15:32:26 -04:00
magodo 90e6a3dffb
modify test case 2021-07-26 11:25:47 +08:00
magodo 67afee693b
target resource in module check only when `-out` points to a module 2021-07-24 12:03:32 +08:00
magodo dea645797a
`terraform add -out` append to existing config 2021-07-24 11:35:15 +08:00
Kristin Laemmert 0b827ab6b6
format/diff: fix panic with null map in NestedType attrs (#29206) 2021-07-21 08:51:35 -04:00
Alisdair McDiarmid a456d030db Fix flapping JSON output test properly
This commit makes the output order of the resource drift messages
stable, by building a slice of changes and sorting it by address.
2021-07-15 13:30:11 -04:00
Alisdair McDiarmid c51112a30c Fix flapping JSON output test
This test would previously fail randomly due to the use of multiple
resource instances. Instance keys are iterated over as a map for
presentation, which has intentionally inconsistent ordering.

To fix this, I changed the test to use different resource addresses for
the three drift cases. I also extracted them to a separate test, and
tweaked the test helper functions to reduce the number of fatal exit
points, to make failed test debugging easier.
2021-07-15 12:03:01 -04:00
Alisdair McDiarmid 5a34825fc1
Merge pull request #29131 from hashicorp/alisdair/sequence-diff-commentary
Add comments explaining how ctySequenceDiff works
2021-07-13 16:05:05 -04:00
Alisdair McDiarmid 72a7c95353
Merge pull request #29072 from hashicorp/alisdair/json-ui-resource-drift
json-output: Add resource drift to machine readable UI
2021-07-12 09:54:42 -04:00
Alisdair McDiarmid ef0181cfbd Add comments explaining how ctySequenceDiff works
The logic behind this code took me a while to understand, so I wrote
down what I understand to be the reasoning behind how it works. The
trickiest part is rendering changing objects as updates. I think the
other pieces are fairly common to LCS sequence diff rendering, so I
didn't explain those in detail.
2021-07-09 13:14:20 -04:00
Martin Atkins ab350289ab addrs: Rename AbsModuleCallOutput to ModuleCallInstanceOutput
The previous name didn't fit with the naming scheme for addrs types:
The "Abs" prefix typically means that it's an addrs.ModuleInstance
combined with whatever type name appears after "Abs", but this is instead
a ModuleCallOutput combined with an InstanceKey, albeit structured the
other way around for convenience, and so the expected name for this would
be the suffix "Instance".

We don't have an "Abs" type corresponding with this one because it would
represent no additional information than AbsOutputValue.
2021-07-01 08:28:02 -07:00