Commit Graph

647 Commits

Author SHA1 Message Date
Kristin Laemmert d2e999ba1f
remove unused code (#26503)
* remove unused code

I've removed the provider-specific code under registry, and unused nil
backend, and replaced a call to helper from backend/oss (the other
callers of that func are provisioners scheduled to be deprecated).

I also removed the Dockerfile, as our build process uses a different
file.

Finally I removed the examples directory, which had outdated examples
and links. There are better, actively maintained examples available.

* command: remove various unused bits

* test wasn't running

* backend: remove unused err
2020-10-07 11:00:06 -04:00
Pam Selle f84a7c1d57
Merge pull request #26476 from remilapeyre/postgres-backend-escape-schema_name
Properly quote schema_name in the pg backend configuration
2020-10-05 14:57:53 -04:00
James Bardin 70652814e3 add missing Context argument 2020-10-05 14:43:46 -04:00
James Bardin 5e13d57f4e
Merge pull request #26463 from hashicorp/kt/azurerm-secret-backend-fix
AzureRM backend: correctly lookup environment from metadata host
2020-10-05 14:26:32 -04:00
Alisdair McDiarmid 35d4926bf7
Merge pull request #25837 from remilapeyre/consul-force-unlock
Fix `terraform force-unlock <lock_id>` for Consul backend
2020-10-05 13:11:08 -04:00
Kristin Laemmert 3933cbd491
remove LegacyProvider (#26433) 2020-10-05 08:33:49 -04:00
Rémi Lapeyre 42be5854a2 Properly quote schema_name in the pg backend configuration 2020-10-03 18:02:13 +02:00
kt b66364f7aa
AzureRM backend: correctly lookup environment from metadata host 2020-10-01 16:34:51 -07:00
Kristin Laemmert d76cfc8c0c
Merge pull request #26440 from hashicorp/mildwonkey/remove-older-state-vs
refactor tests to use modern states.State in favor of terraform.State where possible
2020-10-01 08:11:27 -04:00
Pam Selle 3a99405b8b
Merge pull request #26420 from remilapeyre/postgres-default-backend
Always have the default workspace in the pg backend
2020-09-30 17:54:24 -04:00
Kristin Laemmert 479655ad47 refactor tests to use modern states.State in favor of terraform.State where possible 2020-09-30 16:07:54 -04:00
James Bardin ab6d6f99ae fix races in remote backend mock 2020-09-30 13:36:04 -04:00
Rémi Lapeyre 9acfe30ca6 Always have the default workspace in the pg backend
Closes https://github.com/hashicorp/terraform/issues/23121
2020-09-29 23:18:34 +02:00
Rémi Lapeyre 5c7008bd89 Merge remote-tracking branch 'origin/master' into consul-force-unlock 2020-09-29 10:59:36 +02:00
James Bardin 37569f5cc3 insert PlanRefresh into the context 2020-09-24 09:34:49 -04:00
James Bardin bc82347a04 fix tests
Update tests to match the new behavior. Some were incorrect, some no
longer make sense, and some just weren't setup to handle th plan api
calls.
2020-09-21 16:17:46 -04:00
James Bardin 312317abd0 wrong instance key in test state
This was never picked up by the tests until now
2020-09-17 09:55:00 -04:00
James Bardin 8658424059 skip plan with no refresh test
We still need to determine if `-refresh=false` is even useful with the
new planning strategy.
2020-09-17 09:55:00 -04:00
James Bardin f52d836e0a fix local backend tests to match new behavior
Leaving plan with -refresh=false tests failing for now.
2020-09-17 09:55:00 -04:00
James Bardin be757bd416 Refresh instances during plan
This change refreshes the instance state during plan, so a complete
Refresh no longer needs to happen before Plan.
2020-09-17 09:54:59 -04:00
Alisdair McDiarmid e183163739
Merge pull request #25856 from remilapeyre/consul-split-state
Split the state in chunks when they outgrow the limit of Consul KV store
2020-09-15 10:00:25 -04:00
Alisdair McDiarmid 92abaadc02
Merge pull request #25842 from remilapeyre/consul-path-slash
Sanitize lock path for the Consul backend when it ends with a /
2020-09-11 11:14:49 -04:00
Alisdair McDiarmid 09d8355f43 command: Add experimental concise diff renderer
When rendering a diff between current state and projected state, we only
show resources and outputs which have changes. However, we show a full
structural diff for these values, which includes all attributes and
blocks for a changed resource or output. The result can be a very long
diff, which makes it difficult to verify what the changed fields are.

This commit adds an experimental concise diff renderer, which suppresses
most unchanged fields, only displaying the most relevant changes and
some identifying context. This means:

- Always show all identifying attributes, initially defined as `id`,
  `name`, and `tags`, even if unchanged;
- Only show changed, added, or removed primitive values: `string`,
  `number`, or `bool`;
- Only show added or removed elements in unordered collections and
  structural types: `map`, `set`, and `object`;
- Show added or removed elements with any surrounding unchanged elements
  for sequence types: `list` and `tuple`;
- Only show added or removed nested blocks, or blocks with changed
  attributes.

If any attributes, collection elements, or blocks are hidden, a count
is kept and displayed at the end of the parent scope. This ensures that
it is clear that the diff is only displaying a subset of the resource.

The experiment is currently enabled by default, but can be disabled by
setting the TF_X_CONCISE_DIFF environment variable to 0.
2020-09-10 10:35:55 -04:00
Kirill Zaborsky fbd3f191bd
Minor typo 2020-09-03 11:25:55 +03:00
Matt Kasa c2e35ae3b5
backend/http: add support for configuration by environment variable (#25439) 2020-09-02 08:35:08 -04:00
Kristin Laemmert 196c183dda
terraform: remove state from `validate` graph walk (#26063)
This pull reverts a recent change to backend/local which created two context, one with and one without state. Instead I have removed the state entirely from the validate graph (by explicitly passing a states.NewState() to the validate graph builder).

This changed caused a test failure, which (ty so much for the help) @jbardin discovered was inaccurate all along: the test's call to `Validate()` was actually what was removing the output from state. The new expected test output matches terraform's actual behavior on the command line: if you use -target to destroy a resource, an output that references only that resource is *not* removed from state even though that test would lead you to believe it did.

This includes two tests to cover the expected behavior:

TestPlan_varsUnset has been updated so it will panic if it gets more than one request to input a variable
TestPlan_providerArgumentUnset covers #26035

Fixes #26035, #26027
2020-08-31 15:45:39 -04:00
James Bardin 0c5430f88d
Merge pull request #25585 from chushenmeshile/master
fix ossbackend only returned 100 workspaces
2020-08-19 16:55:31 -04:00
James Bardin bd5ff98ca7
Merge pull request #25830 from hashicorp/kt/secret-metadata-host
backend/azurerm: support loading environments from a metadata host
2020-08-19 13:57:10 -04:00
James Bardin d46e9a4198
Merge pull request #25770 from hashicorp/f/azure-backend-spcert
backend/azurerm: support for authenticating using a Client Certificate
2020-08-19 13:56:30 -04:00
Angie Pinilla 582fb73a7c deps: Update aws-sdk-go-based to 0.6.0 2020-08-18 12:49:31 -04:00
Rémi Lapeyre e680211bc0 Split the state in chunks when they outgrow the limit of Consul KV store
The Consul KV store limits the size of the values in the KV store to 524288
bytes. Once the state reaches this limit Consul will refuse to save it. It is
currently possible to try to bypass this limitation by enable Gzip but the issue
will manifest itself later. This is particularly inconvenient as it is possible
for the state to reach this limit without changing the Terraform configuration
as datasources or computed attributes can suddenly return more data than they
used to. Several users already had issues with this.

To fix the problem once and for all we now split the payload in chunks of 524288
bytes when they are to large and store them separatly in the KV store. A small
JSON payload that references all the chunks so we can retrieve them later and
concatenate them to reconstruct the payload.

While this has the caveat of requiring multiple calls to Consul that cannot be
done as a single transaction as those have the same size limit, we use unique
paths for the chunks and CAS when setting the last payload so possible issues
during calls to Put() should not result in unreadable states.

Closes https://github.com/hashicorp/terraform/issues/19182
2020-08-14 17:38:18 +02:00
Rémi Lapeyre 032d339915 Sanitize lock path for the Consul backend when it ends with a /
When the path ends with / (e.g. `path = "tfstate/"), the lock
path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which
Consul does not accept.

This change the lock path so it is sanitized to `tfstate/.lock`.

If the user has two different Terraform project, one with `path = "tfstate"` and
the other with `path = "tfstate/"`, the paths for the locks will be the same
which will be confusing as locking one project will lock both. I wish it were
possible to forbid ending slashes altogether but doing so would require all
users currently having an ending slash in the path to manually move their
Terraform state and would be a poor user experience.

Closes https://github.com/hashicorp/terraform/issues/15747
2020-08-13 16:29:43 +02:00
Rémi Lapeyre 11eb88753d Fix `terraform force-unlock <lock_id>` for Consul backend
When locking was enabled with the Consul backend and the lock not properly
released, the `terraform force-unlock <lock_id>` command would do nothing as
its implementation would exit early in that case.

It now destroys the session that created the lock and clean both the lock and
the lock-info keys.

A regression test is added to TestConsul_destroyLock() to catch the issue if it
happends again.

Closes https://github.com/hashicorp/terraform/issues/22174
2020-08-13 15:20:24 +02:00
kt 23b4c2db42
azurerm backend: support loading metadata from a given host 2020-08-12 15:04:40 -07:00
James Bardin 1c09df1a66
Merge pull request #25779 from hashicorp/jbardin/remove-state-attrs
Remove resource state attributes that are no longer in the schema
2020-08-12 10:49:44 -04:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
Kristin Laemmert 86e9ba3d65
* backend/local: push responsibility for unlocking state into individual operations
* unlock the state if Context() has an error, exactly as backend/remote does today
* terraform console and terraform import will exit before unlocking state in case of error in Context()
* responsibility for unlocking state in the local backend is pushed down the stack, out of backend.go and into each individual state operation
* add tests confirming that state is not locked after apply and plan

* backend/local: add checks that the state is unlocked after operations

This adds tests to plan, apply and refresh which validate that the state
is unlocked after all operations, regardless of exit status. I've also
added specific tests that force Context() to fail during each operation
to verify that locking behavior specifically.
2020-08-11 11:23:42 -04:00
James Bardin 3cf84bb3f9 don't add state to the validate context
The validate command should work with the configuration, but when
validate was run at the start of a plan or apply command the state was
inserted in preparation for the next walk. This could lead to errors
when the resource schemas had changes and the state could not be
upgraded or decoded.
2020-08-07 14:13:57 -04:00
tombuildsstuff d215453495 backend/azurerm: adding a missing docs string. fixes #25765 2020-08-07 12:01:16 +02:00
tombuildsstuff 0d34e5d97c backend/azurerm: support for authenticating using a Client Certificate
fixes #24179
supersedes #19606
dependent on #25769
2020-08-07 11:58:33 +02:00
zjd 25de713214 fix ossbackend only returned 100 workspaces 2020-07-21 11:18:00 +08:00
Kristin Laemmert 604e65bb62 Revert "backend/local: release lock if there is an error in Context() (#25427)"
This reverts commit 1ba0d615e7.
2020-06-30 14:12:32 -04:00
Kristin Laemmert 1ba0d615e7
backend/local: release lock if there is an error in Context() (#25427)
* command/console: return in case of errors before trying to unlock remote
state

The remote backend `Context` would exit without an active lock if there
was an error, while the local backend `Context` exited *with* a lock. This
caused a problem in `terraform console`, which would call unlock
regardless of error status.

This commit makes the local and remote backend consistently unlock the
state incase of error, and updates terraform console to check for errors
before trying to unlock the state.

* adding tests for remote and local backends
2020-06-29 14:57:42 -04:00
Even Holthe 0f85b283b9
Azure backend: support snapshots/versioning (#24069)
* Azure backend: support snapshots/versioning

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>

* Azure backend: Versioning -> Snapshot

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>
2020-06-25 11:50:16 +02:00
James Bardin 35d232481f prevent panic in remote backend retry
Ensure that the *http.Response is not nil before checking the status.
2020-06-22 15:02:23 -04:00
Petros Kolyvas 4a3d0118c6
Merge pull request #25139 from xiaozhu36/master
backend/oss: supports setting HTTPS_PROXY
2020-06-16 14:49:34 -04:00
James Bardin 7800ef63d4
Merge pull request #19525 from dramich/k8sback
backend/kubernetes: Add Kubernetes as a backend
2020-06-08 15:59:32 -04:00
John Houston 01ebdd27f2 Rework soak test to error on unlock failure 2020-06-05 18:27:30 -04:00
John Houston b0f7e273e8 Add errors for lock disappearing 2020-06-05 18:10:24 -04:00
Brian Flad ba081aa10a
backend/s3: Updates for Terraform v0.13.0 (#25134)
* deps: Update github.com/hashicorp/aws-sdk-go-base@v0.5.0

Updated via:

```
$ go get github.com/hashicorp/aws-sdk-go-base@v0.5.0
$ go mod tidy
$ go mod vendor
```

* backend/s3: Updates for Terraform v0.13.0

Reference: https://github.com/hashicorp/terraform/issues/13410
Reference: https://github.com/hashicorp/terraform/issues/18774
Reference: https://github.com/hashicorp/terraform/issues/19482
Reference: https://github.com/hashicorp/terraform/issues/20062
Reference: https://github.com/hashicorp/terraform/issues/20599
Reference: https://github.com/hashicorp/terraform/issues/22103
Reference: https://github.com/hashicorp/terraform/issues/22161
Reference: https://github.com/hashicorp/terraform/issues/22601
Reference: https://github.com/hashicorp/terraform/issues/22992
Reference: https://github.com/hashicorp/terraform/issues/24252
Reference: https://github.com/hashicorp/terraform/issues/24253
Reference: https://github.com/hashicorp/terraform/issues/24480
Reference: https://github.com/hashicorp/terraform/issues/25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
2020-06-05 16:41:32 -04:00