Commit Graph

89 Commits

Author SHA1 Message Date
Martin Atkins 343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
James Bardin 863963e7a6 de-linting 2021-09-01 11:36:21 -04:00
Martin Atkins 89b05050ec core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.

This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.

Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.

However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.

My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
 - The "terraform graph" command is no longer hooked directly into the
   core graph builders, because that's no longer part of the public API.
   However, I did include a couple new Context functions whose contract
   is to produce a UI-oriented graph, and _for now_ those continue to
   return the physical graph we use for those operations. There's no
   exported API for generating the "validate" and "eval" graphs, because
   neither is particularly interesting in its own right, and so
   "terraform graph" no longer supports those graph types.
 - terraform.NewContext no longer has the responsibility for collecting
   all of the provider schemas up front. Instead, we wait until we need
   them. However, that means that some of our error messages now have a
   slightly different shape due to unwinding through a differently-shaped
   call stack. As of this commit we also end up reloading the schemas
   multiple times in some cases, which is functionally acceptable but
   likely represents a performance regression. I intend to rework this to
   use caching, but I'm saving that for a later commit because this one is
   big enough already.

The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.

However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-30 13:59:14 -07:00
Alex Khaerov 9d5f1752c8
oss backend: flattern assume_role block 2021-08-03 14:26:15 +08:00
James Bardin 97a2694528
Merge pull request #28838 from remilapeyre/consul-size-limit
Fix handling large states in the Consul backend
2021-07-30 14:18:34 -04:00
Rémi Lapeyre da6717761e Merge remote-tracking branch 'origin/main' into update-consul 2021-07-28 12:18:01 +02:00
James Bardin 570b70b02f
Merge pull request #28078 from jasons42/configure-etcdv3-client-max-request-size
Expose etcd client MaxCallSendMsgSize config
2021-07-20 15:49:14 -04:00
James Bardin 8dd722ece0
Merge pull request #29167 from xiaozhu36/xiaozhu
backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug
2021-07-20 15:12:31 -04:00
James Bardin 431aa0280e
Merge pull request #29157 from remilapeyre/unique-constraint
Add uniqueness constraint on workspaces name for the pg backend
2021-07-20 15:11:35 -04:00
Jason Smith d1608d7a7f Expose etcd client MaxCallSendMsgSize config
The etcdv3 client has a default request send limit of 2.0 MiB. This change
exposes the configuration option to increase that limit enabling larger
state using the etcdv3 backend.

This also requires that the corresponding --max-request-bytes flag be
increased on the server side. The default there is 1.5 MiB.

Fixes https://github.com/hashicorp/terraform/issues/25745
2021-07-20 14:04:45 -05:00
James Bardin dd330e5194
Merge pull request #29200 from hashicorp/jbardin/rebase-25554
manual rebase of #25554
2021-07-20 14:29:56 -04:00
Kevin Burke c047958b57 go.mod,backend: update coreos/etcd dependency to release-3.4 branch
etcd rewrote its import path from coreos/etcd to go.etcd.io/etcd.
Changed the imports path in this commit, which also updates the code
version.

This lets us remove the github.com/ugorji/go/codec dependency, which
was pinned to a fairly old version. The net change is a loss of 30,000
lines of code in the vendor directory. (I first noticed this problem
because the outdated go/codec dependency was causing a dependency
failure when I tried to put Terraform and another project in the same
vendor directory.)

Note the version shows up funkily in go.mod, but I verified
visually it's the same commit as the "release-3.4" tag in
github.com/coreos/etcd. The etcd team plans to fix the release version
tagging in v3.5, which should be released soon.
2021-07-20 12:27:22 -04:00
James Bardin 9c20ed6185 StateMgr must be able to return with locked state
The current usage of internal remote state backends requires that
`StateMgr` be able to return an instance of `statemgr.Full` even if the
state is currently locked.
2021-07-20 12:10:34 -04:00
xiaozhu36 c495caafeb backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug 2021-07-15 16:07:18 +08:00
Rémi Lapeyre 3c34e15d40 Update the Consul API client
Some users would want to use Consul namespaces when using the Consul
backend but the version of the Consul API client we use is too old and
don't support them. In preparation for this change this patch just update
it the client and replace testutil.NewTestServerConfig() by
testutil.NewTestServerConfigT() in the tests.
2021-07-14 14:49:13 +02:00
Rémi Lapeyre b8e0d6f418 Add UNIQUE constraint in the states table for the pg backend 2021-07-14 11:59:14 +02:00
Rémi Lapeyre 3bf053a5fc Add a test to ensure that workspaces must have a unique name in pg backend 2021-07-14 11:57:28 +02:00
Nick Fagerlund ab70879b9b Remote backend: Stop setting message when creating runs
Historically, we've used TFC's default run messages as a sort of dumping
ground for metadata about the run. We've recently decided to mostly stop
doing that, in favor of:

- Only specifying the run's source in the default message.
- Letting TFC itself handle the default messages.

Today, the remote backend explicitly sets a run message, overriding
any default that TFC might set. This commit removes that explicit message
so we can allow TFC to sort it out.

This shouldn't have any bad effect on TFE out in the wild, because it's
known how to set a default message for remote backend runs since late 2018.
2021-06-09 11:07:39 -07:00
Rémi Lapeyre 5f444a6862 Fix handling large states in the Consul backend 2021-05-28 21:31:07 +02:00
Rémi Lapeyre e5fb84a01a Add test for storing state just under the size limit of the Consul backend
Storing a state just a bit smaller than the limit of the KV API fails
because we are using the Transaction API which has an overhead.
2021-05-28 21:30:56 +02:00
CJ Horton 2f980f1dfe
Merge pull request #28820 from hashicorp/radditude/refresh-error-message
backend/remote: point refresh users towards -refresh-only
2021-05-27 07:21:37 -07:00
CJ Horton 314d322b59 backend/remote: encourage use of -refresh-only 2021-05-26 23:08:39 -07:00
Alisdair McDiarmid a3b1764fd7 backend/local: Fix lock when applying stale plan
When returning from the context method, a deferred function call checked
for error diagnostics in the `diags` variable, and unlocked the state if
any exist. This means that we need to be extra careful to mutate that
variable when returning errors, rather than returning a different set of
diags in the same position.

Previously this would result in an invalid plan file causing a lock to
become stuck.
2021-05-26 16:05:58 -04:00
CJ Horton ab5fec9aec backend/remote: test new options and modes
Tests for -refresh=false, -refresh-only, and -replace,
as well as tests to ensure we error appropriately on
mismatched API versions.
2021-05-18 15:35:37 -07:00
CJ Horton eb1f113693 backend/remote: validate API version for refresh and replace 2021-05-18 15:35:37 -07:00
Chris Arcand 2171f8a1a6 backend/remote: Support replacements in plans 2021-05-17 16:31:34 -07:00
Chris Arcand a436c7fa2f backend/remote: Support refresh-only plans 2021-05-17 16:31:33 -07:00
Chris Arcand cb49a4c8d6 backend/remote: Remove the -refresh=false restriction 2021-05-17 16:31:32 -07:00
James Bardin 4d55e2604f save apply diagnostics on backend failure
Make sure to collect apply diagnostics before persisting the backend
state so runtime errors are not lost.
2021-05-17 17:22:04 -04:00
Martin Atkins 36d0a50427 Move terraform/ to internal/terraform/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins f40800b3a4 Move states/ to internal/states/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 034e944070 Move plans/ to internal/plans/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 31349a9c3a Move configs/ to internal/configs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins ffe056bacb Move command/ to internal/command/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins b9a93a0fe7 Move addrs/ to internal/addrs/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 1409f30f9c Move providers/ to internal/providers/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 05caff2ca3 Move tfdiags/ to internal/tfdiags/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 4c254cc2be Move httpclient/ to internal/httpclient/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00
Martin Atkins 73dda868cc Move backend/ to internal/backend/
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.

If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
2021-05-17 14:09:07 -07:00