Commit Graph

417 Commits

Author SHA1 Message Date
Martin Atkins c23a7fce4e lang/funcs: Preserve IP address leading zero behavior from Go 1.16
Go 1.17 includes a breaking change to both net.ParseIP and net.ParseCIDR
functions to reject IPv4 address octets written with leading zeros.

Our use of these functions as part of the various CIDR functions in the
Terraform language doesn't have the same security concerns that the Go
team had in evaluating this change to the standard library, and so we
can't justify an exception to our v1.0 compatibility promises on the same
sort of security grounds that the Go team used to justify their
compatibility exception.

For that reason, we'll now use our own fork of the Go library functions
which has the new check disabled in order to preserve the prior behavior.
We're taking this path, rather than pre-normalizing the IP address before
calling into the standard library, because an additional normalization
layer would be entirely new code and additional complexity, whereas this
fork is relatively minor in terms of code size and avoids any significant
changes to our own calls to these functions.

Thanks to the Kubernetes team for their prior work on carving out a subset
of the "net" package for their similar backward-compatibility concern.
Our "ipaddr" package here is a lightly-modified fork of their fork, with
only the comments changed to talk about Terraform instead of Kubernetes.

This fork is not intended for use in any other future feature
implementations, because they wouldn't be subject to the same
compatibility constraints as our existing functions. We will use these
forked implementations for new callers only if consistency with the
behavior of the existing functions is a key requirement.
2021-08-17 15:20:05 -07:00
Martin Atkins 383bbdeebc Upgrade to Go 1.17
This includes the addition of the new "//go:build" comment form in addition
to the legacy "// +build" notation, as produced by gofmt to ensure
consistent behavior between Go versions. The new directives are all
equivalent to what was present before, so there's no change in behavior.

Go 1.17 continues to use the Unicode 13 tables as in Go 1.16, so this
upgrade does not require also upgrading our Unicode-related dependencies.

This upgrade includes the following breaking changes which will also
appear as breaking changes for Terraform users, but that are consistent
with the Terraform v1.0 compatibility promises.

- On MacOS, Terraform now requires macOS 10.13 High Sierra or later.

This upgrade also includes the following breaking changes which will
appear as breaking changes for Terraform users that are inconsistent with
our compatibility promises, but have justified exceptions as follows:

- cidrsubnet, cidrhost, and cidrnetmask will now reject IPv4 CIDR
  addresses whose decimal components have leading zeros, where previously
  they would just silently ignore those leading zeros.

  This is a security-motivated exception to our compatibility promises,
  because some external systems interpret zero-prefixed octets as octal
  numbers rather than decimal, and thus the previous lenient parsing could
  lead to a different interpretation of the address between systems, and
  thus potentially allow bypassing policy when configuring firewall rules
  etc.

This upgrade also includes the following breaking changes which could
_potentially_ appear as breaking changes for Terraform users, but that do
not in practice for the reasons given:

- The Go net/url package no longer allows query strings with pairs
  separated by semicolons instead of ampersands. This primarily affects
  HTTP servers written in Go, and Terraform includes a special temporary
  HTTP server as part of its implementation of OAuth for "terraform login",
  but that server only needs to accept URLs created by Terraform itself
  and Terraform does not generate any URLs that would be rejected.
2021-08-17 15:20:05 -07:00
James Bardin a94155d0ca
Merge pull request #29397 from hashicorp/jbardin/format-id-name-marks
unmark object ID or Name for formatting
2021-08-17 14:58:56 -04:00
James Bardin a48b024c0a unmark object ID or Name for formatting 2021-08-17 12:24:43 -04:00
James Bardin 296a757ab4 check for null sets in diff rendering 2021-08-16 18:25:16 -04:00
James Bardin fbfb14142e render empty nested containers as attributes
Don't try to break down containers that are empty to render the diff, so
we can avoid having to check for empty vs null in all cases.
2021-08-16 18:13:55 -04:00
Radek Simko 2496bc2b1e
internal/registry: Add URL to error message for clarity (#29298) 2021-08-10 15:20:40 +01:00
James Bardin 6087b1bdb9 CanChainFrom and NestedWithin
Add implementations of CanChainFrom and NestedWithin for
MoveEndpointInModule.

CanChainFrom allows the linking of move statements of the same address,
which means the prior destination address must equal the following
source address. If the destination and source addresses are of different
types, they must be covered by NestedWithin rather than CanChainFrom.

NestedWithin checks if the destination contains the source address. Any
matching types would be covered by CanChainFrom.
2021-08-10 10:13:21 -04:00
James Bardin 493ec4e6c5 correct the direction and walk order of the graph 2021-08-10 10:12:39 -04:00
James Bardin 88ad938cc6 Equal methods for move AbsMoveable
Make sure all the types are comparable
2021-08-10 10:12:17 -04:00
James Bardin 08edb02270 MoveStatement.Name()
makes the graph printable for debugging
2021-08-10 10:11:57 -04:00
James Bardin 789317dc05 additional test 2021-08-10 10:11:57 -04:00
James Bardin 6401022bc8 don't take the address of a range variable 2021-08-10 10:11:57 -04:00
Alisdair McDiarmid 3b33dc1105 json-output: Add output changes to plan logs
Extend the outputs JSON log message to support an `action` field (and
make the `type` and `value` fields optional). This allows us to emit a
useful output change summary as part of the plan, bringing the JSON log
output into parity with the text output.

While we do have access to the before/after values in the output
changes, attempting to wedge those into a structured log message is not
appropriate. That level of detail can be extracted from the JSON plan
output from `terraform show -json`.
2021-08-05 15:32:26 -04:00
James Bardin 09ab952683 fix ApplyMoves tests
Add empty result value, since ApplyMoves does not return nil.
Fix the desired addresses for moves.
2021-08-02 17:23:35 -04:00
James Bardin 97a2694528
Merge pull request #28838 from remilapeyre/consul-size-limit
Fix handling large states in the Consul backend
2021-07-30 14:18:34 -04:00
Martin Atkins aa414f3ab3 refactoring: First round of ValidateMoves rules
This is a first pass at implementing refactoring.ValidateMoves, covering
the main validation rules.

This is not yet complete. A couple situations not yet covered are
represented by commented test cases in TestValidateMoves, although that
isn't necessarily comprehensive. We'll do a further pass of filling this
out with any other subtleties before we ship this feature.
2021-07-29 12:29:36 -07:00
Martin Atkins ae2c93f255 addrs: All AbsMovable implementations implement UniqueKeyer 2021-07-29 12:29:36 -07:00
Martin Atkins f76a3467dc core: Call into refactoring.ValidateMoves after creating a plan
As of this commit, refactoring.ValidateMoves doesn't actually do anything
yet (always returns nil) but the goal here is to wire in the set of all
declared instances so that refactoring.ValidateMoves will then have all
of the information it needs to encapsulate our validation rules.

The actual implementation of refactoring.ValidateMoves will follow in
subsequent commits.
2021-07-28 13:54:10 -07:00
Martin Atkins 51346f0d87 instances: Expander.AllInstances
In order to precisely implement the validation rules for "moved"
statements we need to be able to test whether particular instances were
declared in the configuration.

The instance expander is the source of record for which instances we
decided while creating a plan, but it's API is far more involved than what
our validation rules need, so this new AllInstances method returns a
wrapper object with a more straightforward API that provides read-only
access to just the question of whether particular instances got registered
in the expander already.

This API covers all three of the kinds of objects that move statements can
refer to. It includes module calls and resources, even though they aren't
_themselves_ "instances" in the sense we usually mean, because the module
instance addresses they are contained within _are_ instances and so we
need to take their dynamic instance keys into account when answering these
queries.
2021-07-28 13:54:10 -07:00
Martin Atkins 57d36c1d9d addrs: ModuleInstance.ChildCall method
This allows us to concisely construct AbsModuleCall address values by
method chaining from module instance addresses.
2021-07-28 13:54:10 -07:00
Martin Atkins 5a6d11e375 addrs: Factor out MoveEndpointInModule module prefix matching
All of our MoveDestination methods have the common problem of deciding
whether the receiver is even potentially in the scope of a particular
MoveEndpointInModule, which requires that the receiver belong to an
instance of the module where the move statement was found.

Previously we had this logic inline in all three cases, but now we'll
factor it out into a shared helper function.

At first it seemed like there ought to be more factoring possible for
the AbsResource vs. AbsResourceInstance implementations, since textually
they look very similar, but in practice they only look similar because
those two types have a lot of method names in common, but the Go compiler
sees them as completely distinct and thus we must write the same logic
out twice. I did try some further refactoring to address that but it
made the resulting code significantly more complicated and, by my
judgement, harder to follow. Consequently I decided that a little
duplication was okay and warranted here because this logic is already
quite fiddly to read through and isn't likely to change significantly once
released (due to backward-compatibility promises).
2021-07-28 13:33:26 -07:00
Martin Atkins 45d16b4a2b addrs: MoveDestination for AbsResourceInstance-based move endpoints
Previously our MoveDestination methods only honored move statements whose
endpoints were module calls, module instances, or resources.

Now we'll additionally handle when the endpoints are individual resource
instances. This situation only applies to
AbsResourceInstance.MoveDestination because no other objects can be
contained inside of a resource instance.

This completes all of the MoveDestination cases for all supported move
statement types and moveable object types.
2021-07-28 13:33:26 -07:00
Martin Atkins 5e86bab159 addrs: MoveDestination for AbsResource-based move endpoints
Previously our MoveDestination methods only honored move statements whose
endpoints were module calls or module instances.

Now we'll additionally handle when the endpoints are whole resource
addresses. This includes both renaming resource blocks and moving resource
blocks into or out of child modules.

This doesn't yet include endpoints that are specific resource _instances_,
which will follow in a subsequent commit. For the moment that situation
will always indicate a non-match.
2021-07-28 13:33:26 -07:00
Rémi Lapeyre da6717761e Merge remote-tracking branch 'origin/main' into update-consul 2021-07-28 12:18:01 +02:00
Martin Atkins 994ee23c06 addrs: Module move support for AbsResource and AbsResourceInstance
This is a subset of the MoveDestination behavior for AbsResource and
AbsResourceInstance which deals with source and destination addresses that
refer to module calls or module instances.

They both work by delegating to ModuleInstance.MoveDestination and then
applying the same resource or resource instance address to the
newly-chosen module instance address, thus ensuring that when we move
a module we also move all of the resources inside that module in the same
way.

This doesn't yet include support for moving between specific resource or
resource instance addresses; that'll follow later. This commit should have
enough logic to support moving between module names and module instance
keys, including any module calls or resources nested within.
2021-07-27 09:13:01 -07:00
Martin Atkins 4d733b4d2d addrs: Implement ModuleInstance.MoveDestination
This method encapsulates the move-processing rules for applying move
statements to ModuleInstance addresses. It honors both module call moves
and module instance moves by trying to find a subsequence of the input
that matches the "from" endpoint and then, if found, replacing it with
the "to" endpoint while preserving the prefix and suffix around the match,
if any.
2021-07-27 09:13:01 -07:00
Kristin Laemmert 0729e9fdd7
configs/configschema: extend block.AttributeByPath to descend into Objects (#29222)
* configs/configschema: extend block.AttributeByPath to descend into Objects

This commit adds a recursive Object.AttributeByPath function which will step through Attributes with NestedTypes. If an Attribute without a NestedType is encountered while there is still more to the path, it will return nil.
2021-07-22 09:45:33 -04:00
James Bardin b12502679f
Merge pull request #29208 from hashicorp/jbardin/update-hcl
update hcl v2.10.1
2021-07-21 13:51:02 -04:00
James Bardin aaf03d3251 update test error for hclv2.10.1 2021-07-21 09:09:30 -04:00
James Bardin 6fa04091f5
Merge pull request #29193 from hashicorp/jbardin/ignore-changes-marks
Handle marks on ignore_changes values
2021-07-21 09:05:50 -04:00
Kristin Laemmert 0b827ab6b6
format/diff: fix panic with null map in NestedType attrs (#29206) 2021-07-21 08:51:35 -04:00
James Bardin 07ee689eff update comment and extend test 2021-07-20 16:09:46 -04:00
James Bardin 570b70b02f
Merge pull request #28078 from jasons42/configure-etcdv3-client-max-request-size
Expose etcd client MaxCallSendMsgSize config
2021-07-20 15:49:14 -04:00
James Bardin 8dd722ece0
Merge pull request #29167 from xiaozhu36/xiaozhu
backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug
2021-07-20 15:12:31 -04:00
James Bardin 431aa0280e
Merge pull request #29157 from remilapeyre/unique-constraint
Add uniqueness constraint on workspaces name for the pg backend
2021-07-20 15:11:35 -04:00
Jason Smith d1608d7a7f Expose etcd client MaxCallSendMsgSize config
The etcdv3 client has a default request send limit of 2.0 MiB. This change
exposes the configuration option to increase that limit enabling larger
state using the etcdv3 backend.

This also requires that the corresponding --max-request-bytes flag be
increased on the server side. The default there is 1.5 MiB.

Fixes https://github.com/hashicorp/terraform/issues/25745
2021-07-20 14:04:45 -05:00
James Bardin dd330e5194
Merge pull request #29200 from hashicorp/jbardin/rebase-25554
manual rebase of #25554
2021-07-20 14:29:56 -04:00
Kevin Burke c047958b57 go.mod,backend: update coreos/etcd dependency to release-3.4 branch
etcd rewrote its import path from coreos/etcd to go.etcd.io/etcd.
Changed the imports path in this commit, which also updates the code
version.

This lets us remove the github.com/ugorji/go/codec dependency, which
was pinned to a fairly old version. The net change is a loss of 30,000
lines of code in the vendor directory. (I first noticed this problem
because the outdated go/codec dependency was causing a dependency
failure when I tried to put Terraform and another project in the same
vendor directory.)

Note the version shows up funkily in go.mod, but I verified
visually it's the same commit as the "release-3.4" tag in
github.com/coreos/etcd. The etcd team plans to fix the release version
tagging in v3.5, which should be released soon.
2021-07-20 12:27:22 -04:00
James Bardin 9c20ed6185 StateMgr must be able to return with locked state
The current usage of internal remote state backends requires that
`StateMgr` be able to return an instance of `statemgr.Full` even if the
state is currently locked.
2021-07-20 12:10:34 -04:00
James Bardin 66a5950acb ignored value in this test should not be marked 2021-07-19 16:42:26 -04:00
James Bardin e574f73f61 test with marked, unknown, planned values 2021-07-19 16:42:26 -04:00
James Bardin ea077cbd90 handle marks within ignore_changes
Up until now marks were not considered by `ignore_changes`, that however
means changes to sensitivity within a configuration cannot ignored, even
though they are planned as changes.

Rather than separating the marks and tracking their paths, we can easily
update the processIgnoreChanges routine to handle the marked values
directly. Moving the `processIgnoreChanges` call also cleans up some of
the variable naming, making it more consistent through the body of the
function.
2021-07-19 16:42:26 -04:00
Alisdair McDiarmid a456d030db Fix flapping JSON output test properly
This commit makes the output order of the resource drift messages
stable, by building a slice of changes and sorting it by address.
2021-07-15 13:30:11 -04:00
Kristin Laemmert 0d80a74539
configs/configschema: fix missing "computed" attributes from NestedObject's ImpliedType (#29172)
* configs/configschema: fix missing "computed" attributes from NestedObject's ImpliedType

listOptionalAttrsFromObject was not including "computed" attributes in the list of optional object attributes. This is now fixed. I've also added some tests and fixed some panics and otherwise bad behavior when bad input is given. One natable change is in ImpliedType, which was panicking on an invalid nesting mode. The comment expressly states that it will return a result even when the schema is inconsistent, so I removed the panic and instead return an empty object.
2021-07-15 13:00:07 -04:00
Paddy 8bdea502ab
Add support for protocol 6 providers during init. (#29153)
Update the version constraints for what providers will be downloaded
from the registry, allowing protocol 6 providers to be downloaded from
the registry.
2021-07-15 09:39:40 -07:00
Alisdair McDiarmid c51112a30c Fix flapping JSON output test
This test would previously fail randomly due to the use of multiple
resource instances. Instance keys are iterated over as a map for
presentation, which has intentionally inconsistent ordering.

To fix this, I changed the test to use different resource addresses for
the three drift cases. I also extracted them to a separate test, and
tweaked the test helper functions to reduce the number of fatal exit
points, to make failed test debugging easier.
2021-07-15 12:03:01 -04:00
xiaozhu36 c495caafeb backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug 2021-07-15 16:07:18 +08:00
Martin Atkins 3e5bfa7364 refactoring: Stubbing of the logic for handling moves
This is a whole lot of nothing right now, just stubbing out some control
flow that ultimately just leads to TODOs that cause it to do nothing at
all.

My intent here is to get this cross-cutting skeleton in place and thus
make it easier for us to collaborate on adding the meat to it, so that
it's more likely we can work on different parts separately and still get
a result that tessellates.
2021-07-14 17:37:48 -07:00
Martin Atkins 22eee529e3 addrs: MoveEndpointInModule
We previously built out addrs.UnifyMoveEndpoints with a different
implementation strategy in mind, but that design turns out to not be
viable because it forces us to move to AbsMoveable addresses too soon,
before we've done the analysis required to identify chained and nested
moves.

Instead, UnifyMoveEndpoints will return a new type MoveEndpointInModule
which conceptually represents a matching pattern which either matches or
doesn't match a particular AbsMoveable. It does this by just binding the
unified relative address from the MoveEndpoint to the module where it
was declared, and thus allows us to distinguish between the part of the
module path which applies to any instances of the given modules vs. the
user-specified part which must identify particular module instances.
2021-07-14 17:37:48 -07:00
Martin Atkins cd06572c4f addrs: ModuleInstance and AbsResourceInstance are UniqueKeyers
Since these address types are not directly comparable themselves, we use
an unexported named type around the string representation, whereby the
special type can avoid any ambiguity between string representations of
different types and thus each type only needs to worry about possible
ambiguity of its _own_ string representation.
2021-07-14 17:37:48 -07:00
Martin Atkins f3a57db293 addrs: UniqueKey and UniqueKeyer
Many times now we've seen situations where we need to use addresses
as map keys, but not all of our address types are comparable and thus
we tend to end up using string representations as keys instead.

That's problematic because conversion to string uses type information
and some of the address types have string representations that are
ambiguous with one another.

UniqueKey therefore represents an opaque key that is unique for each
functionally-distinct address across all types that implement
UniqueKeyer.

For this initial commit I've implemented UniqueKeyer only for the
Referenceable family of types. These are an easy case because they
were all already comparable (intentionally) anyway. Later commits
can implement UniqueKeyer for other types that are not naturally
comparable, such as any which include a ModuleInstance.

This also includes a new type addrs.Set which wraps a map as a set
of addresses, using the unique keys to ensure that there can be only
one element for each distinct address.
2021-07-14 17:37:48 -07:00
Rémi Lapeyre 3c34e15d40 Update the Consul API client
Some users would want to use Consul namespaces when using the Consul
backend but the version of the Consul API client we use is too old and
don't support them. In preparation for this change this patch just update
it the client and replace testutil.NewTestServerConfig() by
testutil.NewTestServerConfigT() in the tests.
2021-07-14 14:49:13 +02:00
Rémi Lapeyre b8e0d6f418 Add UNIQUE constraint in the states table for the pg backend 2021-07-14 11:59:14 +02:00
Rémi Lapeyre 3bf053a5fc Add a test to ensure that workspaces must have a unique name in pg backend 2021-07-14 11:57:28 +02:00
Kristin Laemmert 43f698dc9d
states: new `Move` funcs for Resource, ResourceInstance, and ModuleInstances (#29068)
* states: add MoveAbsResource and MoveAbsResourceInstance state functions and corresponding syncState wrapper functions.

* states: add MoveModuleInstance and MaybeMoveModuleInstance

* addrs: adding a new function, ModuleInstance.IsDeclaredByCall, which returns true if the receiver is an instance of the given AbsModuleCall.
2021-07-13 16:10:46 -04:00
Alisdair McDiarmid 5a34825fc1
Merge pull request #29131 from hashicorp/alisdair/sequence-diff-commentary
Add comments explaining how ctySequenceDiff works
2021-07-13 16:05:05 -04:00
Alisdair McDiarmid 72a7c95353
Merge pull request #29072 from hashicorp/alisdair/json-ui-resource-drift
json-output: Add resource drift to machine readable UI
2021-07-12 09:54:42 -04:00
Alisdair McDiarmid ef0181cfbd Add comments explaining how ctySequenceDiff works
The logic behind this code took me a while to understand, so I wrote
down what I understand to be the reasoning behind how it works. The
trickiest part is rendering changing objects as updates. I think the
other pieces are fairly common to LCS sequence diff rendering, so I
didn't explain those in detail.
2021-07-09 13:14:20 -04:00
Martin Atkins 6b8e103d6a configs: Include "moved" blocks when merging multiple files into a module
An earlier commit added logic to decode "moved" blocks and do static
validation of them. Here we now include that result also in modules
produced from those files, which we can then use in Terraform Core to
actually implement the moves.

This also places the feature behind an active experiment keyword called
config_driven_move. For now activating this doesn't actually achieve
anything except let you include moved blocks that Terraform will summarily
ignore, but we'll expand the scope of this in later commits to eventually
reach the point where it's really usable.
2021-07-01 08:28:02 -07:00
Martin Atkins d92b5e5f5e configs: valid-modules test ignores experimental features warning
A common source of churn when we're running experiments is that a module
that would otherwise be valid ends up generating a warning merely because
the experiment is active. That means we end up needing to shuffle the
test files around if the feature ultimately graduates to stable.

To reduce that churn in simple cases, we'll make an exception to disregard
the "Experiment is active" warning for any experiment that a module has
intentionally opted into, because those warnings are always expected and
not a cause for concern.

It's still possible to test those warnings explicitly using the
testdata/warning-files directory, if needed.
2021-07-01 08:28:02 -07:00
Martin Atkins 708003b035 configs: For Moved blocks, use addrs.MoveEndpoint instead of addrs.Target
Although addrs.Target can in principle capture the information we need to
represent move endpoints, it's semantically confusing because
addrs.Targetable uses addrs.Abs... types which are typically for absolute
addresses, but we were using them for relative addresses here.

We now have specialized address types for representing moves and probably
other things which have similar requirements later on. These types
largely communicate the same information in the end, but aim to do so in
a way that's explicit about which addresses are relative and which are
absolute, to make it less likely that we'd inadvertently misuse these
addresses.
2021-07-01 08:28:02 -07:00
Martin Atkins 4cbe6cabfc addrs: AbsMoveable, ConfigMoveable, and MoveableEndpoint
These three types represent the three different address representations we
need to represent different stages of analysis for "moved" blocks in the
configuration.

The goal here is to encapsulate all of the static address wrangling inside
these types so that users of these types elsewhere would have to work
pretty hard to use them incorrectly.

In particular, the MovableEndpoint type intentionally fully encapsulates
the weird relative addresses we use in configuration so that code
elsewhere in Terraform can never end up holding an address of a type that
suggests absolute when it's actually relative. That situation only occurs
in the internals of MoveableEndpoint where we use not-really-absolute
AbsMoveable address types to represent the not-yet-resolved relative
addresses.

This only takes care of the static address wrangling. There's lots of
other rules for what makes a "moved" block valid which will need to be
checked elsewhere because they require more context than just the content
of the address itself.
2021-07-01 08:28:02 -07:00
Martin Atkins 3212f6f367 addrs: AbsModuleCall type
Our documentation for ModuleCall originally asserted that we didn't need
AbsModuleCall because ModuleInstance captured the same information, but
when we added count and for_each for modules we introduced
ModuleCallInstance to represent a reference to an instance of a local
module call, and now _that_ is the type whose absolute equivalent is
ModuleInstance.

We previously had no absolute representation of the call itself, without
any particular instance. That's what AbsModuleCall now represents,
allowing us to be explicit about when we're talking about the module block
vs. instances it declares, which is the same distinction represented by
AbsResource vs. AbsResourceInstance.

Just like with AbsResource and AbsResourceInstance though, there is
syntactic ambiguity between a no-key call instance and a whole module call,
and so some codepaths might accept both to start and then use other
context to dynamically choose a particular interpretation, in which case
this distinction becomes meaningful in representing the result of that
decision.
2021-07-01 08:28:02 -07:00
Martin Atkins ab350289ab addrs: Rename AbsModuleCallOutput to ModuleCallInstanceOutput
The previous name didn't fit with the naming scheme for addrs types:
The "Abs" prefix typically means that it's an addrs.ModuleInstance
combined with whatever type name appears after "Abs", but this is instead
a ModuleCallOutput combined with an InstanceKey, albeit structured the
other way around for convenience, and so the expected name for this would
be the suffix "Instance".

We don't have an "Abs" type corresponding with this one because it would
represent no additional information than AbsOutputValue.
2021-07-01 08:28:02 -07:00
Alisdair McDiarmid 71a067242d json-output: Add resource drift to machine readable UI 2021-06-30 14:57:55 -04:00
Kristin Laemmert 35c19d7c9f
command/jsonstate: remove redundant remarking of resource instance (#29049)
* command/jsonstate: remove redundant remarking of resource instance

ResourceInstanceObjectSrc.Decode already handles marking values with any marks stored in ri.Current.AttrSensitivePaths, so re-applying those marks is not necessary.

We've gotten reports of panics coming from this line of code, though I have yet to reproduce the panic in a test.

* Implement test to reproduce panic on #29042

Co-authored-by: David Alger <davidmalger@gmail.com>
2021-06-29 10:59:20 -04:00
Martin Atkins 70bc432f85 command/views/json: Never generate invalid diagnostic snippet offsets
Because our snippet generator is trying to select whole lines to include
in the snippet, it has some edge cases for odd situations where the
relevant source range starts or ends directly at a newline, which were
previously causing this logic to return out-of-bounds offsets into the
code snippet string.

Although arguably it'd be better for the original diagnostics to report
more reasonable source ranges, it's better for us to report a
slightly-inaccurate snippet than to crash altogether, and so we'll extend
our existing range checks to check both bounds of the string and thus
avoid downstreams having to deal with out-of-bounds indices.

For completeness here I also added some similar logic to the
human-oriented diagnostic formatter, which consumes the result of the
JSON diagnostic builder. That's not really needed with the additional
checks in the JSON diagnostic builder, but it's nice to reinforce that
this code can't panic (in this way, at least) even if its input isn't
valid.
2021-06-28 13:42:28 -07:00
James Bardin c687ebeaf1
Merge pull request #29039 from hashicorp/jbardin/sensitive
New marks.Sensitive type, and audit of sensitive marks usage
2021-06-25 17:11:59 -04:00
James Bardin 80ef795cbf add marks.Raw 2021-06-25 14:27:43 -04:00
James Bardin 55ebb2708c remove IsMarked and ContainsMarked calls
Make sure sensitivity checks are looking for specific marks rather than
any marks at all.
2021-06-25 14:17:06 -04:00
James Bardin cdf7469efd marks.Has and marks.Contains 2021-06-25 14:17:03 -04:00
James Bardin d9dfd451ea update to use typed sensitive marks 2021-06-25 12:49:07 -04:00
James Bardin 2c493e38c7 marks package
marks.Sensitive
2021-06-25 12:35:51 -04:00
Kristin Laemmert 096010600d
terraform: use hcl.MergeBodies instead of configs.MergeBodies for pro… (#29000)
* terraform: use hcl.MergeBodies instead of configs.MergeBodies for provider configuration

Previously, Terraform would return an error if the user supplied provider configuration via interactive input iff the configuration provided on the command line was missing any required attributes - even if those attributes were already set in config.

That error came from configs.MergeBody, which was designed for overriding valid configuration. It expects that the first ("base") body has all required attributes. However in the case of interactive input for provider configuration, it is perfectly valid if either or both bodies are missing required attributes, as long as the final body has all required attributes. hcl.MergeBodies works very similarly to configs.MergeBodies, with a key difference being that it only checks that all required attributes are present after the two bodies are merged.

I've updated the existing test to use interactive input vars and a schema with all required attributes. This test failed before switching from configs.MergeBodies to hcl.MergeBodies.

* add a command package test that shows that we can still have providers with dynamic configuration + required + interactive input merging

This test failed when buildProviderConfig still used configs.MergeBodies instead of hcl.MergeBodies
2021-06-25 08:48:47 -04:00
Kristin Laemmert 3acb5e2841
configs: add decodeMovedBlock behind a locked gate. (#28973)
This PR adds decoding for the upcoming "moved" blocks in configuration. This code is gated behind an experiment called EverythingIsAPlan, but the experiment is not registered as an active experiment, so it will never run (there is a test in place which will fail if the experiment is ever registered).

This also adds a new function to the Targetable interface, AddrType, to simplifying comparing two addrs.Targetable.

There is some validation missing still: this does not (yet) descend into resources to see if the actual resource types are the same (I've put this off in part because we will eventually need the provider schema to verify aliased resources, so I suspect this validation will have to happen later on).
2021-06-21 10:53:16 -04:00
Alisdair McDiarmid 5611da8eb5
Merge pull request #28975 from hashicorp/alisdair/jsonplan-drift
json-output: Omit unchanged resource_drift entries
2021-06-18 11:37:49 -04:00
James Bardin 6a495c8d42 fixupBody missing MissingItemRange
When blocktoattr.fixupBody returned its content, the value for
`MissingItemRange` was omitted, losing the diagnostic Subject.
2021-06-18 09:05:56 -04:00
Alisdair McDiarmid 3326ab7dae json-output: Omit unchanged resource_drift entries
Previously, if any resources were found to have drifted, the JSON plan
output would include a drift entry for every resource in state. This
commit aligns the JSON plan output with the CLI UI, and only includes
those resources where the old value does not equal the new value---i.e.
drift has been detected.

Also fixes a bug where the "address" field was missing from the drift
output, and adds some test coverage.
2021-06-17 15:09:16 -04:00
Kristin Laemmert 583859e510
commands: `terraform add` (#28874)
* command: new command, terraform add, generates resource templates

terraform add ADDRESS generates a resource configuration template with all required (and optionally optional) attributes set to null. This can optionally also pre-populate nonsesitive attributes with values from an existing resource of the same type in state (sensitive vals will be populated with null and a comment indicating sensitivity)

* website: terraform add documentation
2021-06-17 12:08:37 -04:00
Bryan Eastes 714632206c
Quoting filesystem path in scp command argument (#28626)
* Quoting filesystem path in scp command argument

* Adding proper shell quoting for scp commands

* Running go fmt

* Using a library for quoting shell commands

* Don't export quoteShell function
2021-06-15 10:04:01 -07:00
James Bardin 2ecdf44918
Merge pull request #28941 from hashicorp/jbardin/objchange-unknown-blocks
handle unexpected changes to unknown block
2021-06-14 10:31:31 -04:00
Kristin Laemmert 329585d07d
jsonconfig: properly unwind and enumerate references (#28884)
The "references" included in the expression representation now properly unwrap for each traversal step, to match what was documented.
2021-06-14 09:22:22 -04:00
Kristin Laemmert ac03d35997
jsonplan and jsonstate: include sensitive_values in state representations (#28889)
* jsonplan and jsonstate: include sensitive_values in state representations

A sensitive_values field has been added to the resource in state and planned values which is a map of all sensitive attributes with the values set to true.

It wasn't entirely clear to me if the values in state would suffice, or if we also need to consult the schema - I believe that this is sufficient for state files written since v0.15, and if that's incorrect or insufficient, I'll add in the provider schema check as well.

I also updated the documentation, and, since we've considered this before, bumped the FormatVersions for both jsonstate and jsonplan.
2021-06-14 09:19:13 -04:00
James Bardin b7f8ef4dc6 handle unexpected changes to unknown block
An unknown block represents a dynamic configuration block with an
unknown for_each value. We were not catching the case where a provider
modified this value unexpectedly, which would crash with block of type
NestingList blocks where the config value has no length for comparison.
2021-06-11 13:13:40 -04:00
James Bardin 09c33fa449 account for noop deposed instances in json plan
When rendering a json plan, we need to account for deposed instances
that have become a noop rather than a destroy.
2021-06-09 17:57:14 -04:00
Nick Fagerlund ab70879b9b Remote backend: Stop setting message when creating runs
Historically, we've used TFC's default run messages as a sort of dumping
ground for metadata about the run. We've recently decided to mostly stop
doing that, in favor of:

- Only specifying the run's source in the default message.
- Letting TFC itself handle the default messages.

Today, the remote backend explicitly sets a run message, overriding
any default that TFC might set. This commit removes that explicit message
so we can allow TFC to sort it out.

This shouldn't have any bad effect on TFE out in the wild, because it's
known how to set a default message for remote backend runs since late 2018.
2021-06-09 11:07:39 -07:00
Alisdair McDiarmid 24ace6ae7d
Merge pull request #28864 from hashicorp/alisdair/fix-remote-backend-multi-workspace-state-migration
Fix remote backend multi workspace state migration
2021-06-08 10:10:58 -04:00
Kristin Laemmert 5a48530f47
tools: remove terraform-bundle. (#28876)
* tools: remove terraform-bundle.

terraform-bundle is no longer supported in the main branch of terraform. Users can build terraform-bundle from terraform tagged v0.15 and older.

* add a README pointing users to the v0.15 branch
2021-06-03 14:08:04 -04:00
Martin Atkins 51b0aee36c addrs: ModuleRegistryPackage for representing module registry packages
Previously we had a separation between ModuleSourceRemote and
ModulePackage as a way to represent within the type system that there's an
important difference between a module source address and a package address,
because module packages often contain multiple modules and so a
ModuleSourceRemote combines a ModulePackage with a subdirectory to
represent one specific module.

This commit applies that same strategy to ModuleSourceRegistry, creating
a new type ModuleRegistryPackage to represent the different sort of
package that we use for registry modules. Again, the main goal here is
to try to reflect the conceptual modelling more directly in the type
system so that we can more easily verify that uses of these different
address types are correct.

To make use of that, I've also lightly reworked initwd's module installer
to use addrs.ModuleRegistryPackage directly, instead of a string
representation thereof. This was in response to some earlier commits where
I found myself accidentally mixing up package addresses and source
addresses in the installRegistryModule method; with this new organization
those bugs would've been caught at compile time, rather than only at
unit and integration testing time.

While in the area anyway, I also took this opportunity to fix some
historical confusing names of fields in initwd.ModuleInstaller, to be
clearer that they are only for registry packages and not for all module
source address types.
2021-06-03 08:50:34 -07:00
Martin Atkins 7b2a0284e0 initwd: Fix registry acceptance tests for upstream registry changes
We have some tests in this package that install real modules from the real
registry at registry.terraform.io. Those tests were written at an earlier
time when the registry's behavior was to return the URL of a .tar.gz
archive generated automatically by GitHub, which included an extra level
of subdirectory that would then be reflected in the paths to the local
copies of these modules.

GitHub started rate limiting those tar archives in a way that Terraform's
module installer couldn't authenticate to, and so the registry switched
to returning direct git repository URLs instead, which don't have that
extra subdirectory and so the local paths on disk now end up being a
little different, because the actual module directories are at a different
subdirectory of the package.
2021-06-03 08:50:34 -07:00
Martin Atkins 17b766c3ea configs: EntersNewPackage methods for descendant modules
Now that we (in the previous commit) refactored how we deal with module
sources to do the parsing at config loading time rather than at module
installation time, we can expose a method to centralize the determination
for whether a particular module call (and its resulting Config object)
enters a new external package.

We don't use this for anything yet, but in later commits we will use this
for some cross-module features that are available only for modules
belonging to the same package, because we assume that modules grouped
together in a package can change together and thus it's okay to permit a
little more coupling of internal details in that case, which would not
be appropriate between modules that are versioned separately.
2021-06-03 08:50:34 -07:00
Martin Atkins 1a8da65314 Refactoring of module source addresses and module installation
It's been a long while since we gave close attention to the codepaths for
module source address parsing and external module package installation.
Due to their age, these codepaths often diverged from our modern practices
such as representing address types in the addrs package, and encapsulating
package installation details only in a particular location.

In particular, this refactor makes source address parsing a separate step
from module installation, which therefore makes the result of that parsing
available to other Terraform subsystems which work with the configuration
representation objects.

This also presented the opportunity to better encapsulate our use of
go-getter into a new package "getmodules" (echoing "getproviders"), which
is intended to be the only part of Terraform that directly interacts with
go-getter.

This is largely just a refactor of the existing functionality into a new
code organization, but there is one notable change in behavior here: the
source address parsing now happens during configuration loading rather
than module installation, which may cause errors about invalid addresses
to be returned in different situations than before. That counts as
backward compatible because we only promise to remain compatible with
configurations that are _valid_, which means that they can be initialized,
planned, and applied without any errors. This doesn't introduce any new
error cases, and instead just makes a pre-existing error case be detected
earlier.

Our module registry client is still using its own special module address
type from registry/regsrc for now, with a small shim from the new
addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also
rework the registry client to work with the new address type, but this
commit is already big enough as it is.
2021-06-03 08:50:34 -07:00
Martin Atkins 2e7db64968 addrs: Representation of module source addresses
We've previously had the syntax and representation of module source
addresses pretty sprawled around the codebase and intermingled with other
systems such as the module installer.

I've created a factored-out implementation here with the intention of
enabling some later refactoring to centralize the address parsing as part
of configuration decoding, and thus in turn allow the parsing result to
be seen by other parts of Terraform that interact with configuration
objects, so that they can more robustly handle differences between local
and remote modules, and can present module addresses consistently in the
UI.
2021-06-03 08:50:34 -07:00
Martin Atkins f25935649a getmodules: Beginnings of a new package about Terraform module packages
This new package aims to encapsulate all of our interactions with
go-getter to fetch remote module packages, to ensure that the rest of
Terraform will only use the small subset of go-getter functionality that
our modern module installer uses.

In older versions of Terraform, go-getter was the entire implementation
of module installation, but along the way we found that several aspects of
its design are poor fit for Terraform's needs, and so now we're using it
as just an implementation detail of Terraform's handling of remote module
packages only, hiding it behind this wrapper API which exposes only
the services that our module installer needs.

This new package isn't actually used yet, but in a later commit we will
change all of the other callers to go-getter to only work indirectly
through this package, so that this will be the only package that actually
imports the go-getter packages.
2021-06-03 08:50:34 -07:00
Martin Atkins e5f52e56f8 addrs: Rename DefaultRegistryHost to DefaultProviderRegistryHost
As the comment notes, this hostname is the default for provide source
addresses. We'll shortly be adding some address types to represent module
source addresses too, and so we'll also have DefaultModuleRegistryHost
for that situation.

(They'll actually both contain the the same hostname, but that's a
coincidence rather than a requirement.)
2021-06-03 08:50:34 -07:00
Alisdair McDiarmid 3f0c6a2217 cli: Add -ignore-remote-version flag for init
When performing state migration to a remote backend target, Terraform
may fail due to mismatched remote and local Terraform versions. Here we
add the `-ignore-remote-version` flag to allow users to ignore this
version check when necessary.
2021-06-02 15:30:05 -04:00
Alisdair McDiarmid 6692336541 cli: Fix state migration version check
When migrating multiple local workspaces to a remote backend target
using the `prefix` argument, we need to perform the version check
against all existing workspaces returned by the `Workspaces` method.
Failing to do so will result in a version check error.
2021-06-02 15:23:56 -04:00
Rémi Lapeyre 5f444a6862 Fix handling large states in the Consul backend 2021-05-28 21:31:07 +02:00
Rémi Lapeyre e5fb84a01a Add test for storing state just under the size limit of the Consul backend
Storing a state just a bit smaller than the limit of the KV API fails
because we are using the Transaction API which has an overhead.
2021-05-28 21:30:56 +02:00