Commit Graph

298 Commits

Author SHA1 Message Date
Rémi Lapeyre 9acfe30ca6 Always have the default workspace in the pg backend
Closes https://github.com/hashicorp/terraform/issues/23121
2020-09-29 23:18:34 +02:00
Alisdair McDiarmid e183163739
Merge pull request #25856 from remilapeyre/consul-split-state
Split the state in chunks when they outgrow the limit of Consul KV store
2020-09-15 10:00:25 -04:00
Alisdair McDiarmid 92abaadc02
Merge pull request #25842 from remilapeyre/consul-path-slash
Sanitize lock path for the Consul backend when it ends with a /
2020-09-11 11:14:49 -04:00
Matt Kasa c2e35ae3b5
backend/http: add support for configuration by environment variable (#25439) 2020-09-02 08:35:08 -04:00
James Bardin 0c5430f88d
Merge pull request #25585 from chushenmeshile/master
fix ossbackend only returned 100 workspaces
2020-08-19 16:55:31 -04:00
James Bardin bd5ff98ca7
Merge pull request #25830 from hashicorp/kt/secret-metadata-host
backend/azurerm: support loading environments from a metadata host
2020-08-19 13:57:10 -04:00
James Bardin d46e9a4198
Merge pull request #25770 from hashicorp/f/azure-backend-spcert
backend/azurerm: support for authenticating using a Client Certificate
2020-08-19 13:56:30 -04:00
Angie Pinilla 582fb73a7c deps: Update aws-sdk-go-based to 0.6.0 2020-08-18 12:49:31 -04:00
Rémi Lapeyre e680211bc0 Split the state in chunks when they outgrow the limit of Consul KV store
The Consul KV store limits the size of the values in the KV store to 524288
bytes. Once the state reaches this limit Consul will refuse to save it. It is
currently possible to try to bypass this limitation by enable Gzip but the issue
will manifest itself later. This is particularly inconvenient as it is possible
for the state to reach this limit without changing the Terraform configuration
as datasources or computed attributes can suddenly return more data than they
used to. Several users already had issues with this.

To fix the problem once and for all we now split the payload in chunks of 524288
bytes when they are to large and store them separatly in the KV store. A small
JSON payload that references all the chunks so we can retrieve them later and
concatenate them to reconstruct the payload.

While this has the caveat of requiring multiple calls to Consul that cannot be
done as a single transaction as those have the same size limit, we use unique
paths for the chunks and CAS when setting the last payload so possible issues
during calls to Put() should not result in unreadable states.

Closes https://github.com/hashicorp/terraform/issues/19182
2020-08-14 17:38:18 +02:00
Rémi Lapeyre 032d339915 Sanitize lock path for the Consul backend when it ends with a /
When the path ends with / (e.g. `path = "tfstate/"), the lock
path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which
Consul does not accept.

This change the lock path so it is sanitized to `tfstate/.lock`.

If the user has two different Terraform project, one with `path = "tfstate"` and
the other with `path = "tfstate/"`, the paths for the locks will be the same
which will be confusing as locking one project will lock both. I wish it were
possible to forbid ending slashes altogether but doing so would require all
users currently having an ending slash in the path to manually move their
Terraform state and would be a poor user experience.

Closes https://github.com/hashicorp/terraform/issues/15747
2020-08-13 16:29:43 +02:00
kt 23b4c2db42
azurerm backend: support loading metadata from a given host 2020-08-12 15:04:40 -07:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
tombuildsstuff d215453495 backend/azurerm: adding a missing docs string. fixes #25765 2020-08-07 12:01:16 +02:00
tombuildsstuff 0d34e5d97c backend/azurerm: support for authenticating using a Client Certificate
fixes #24179
supersedes #19606
dependent on #25769
2020-08-07 11:58:33 +02:00
zjd 25de713214 fix ossbackend only returned 100 workspaces 2020-07-21 11:18:00 +08:00
Even Holthe 0f85b283b9
Azure backend: support snapshots/versioning (#24069)
* Azure backend: support snapshots/versioning

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>

* Azure backend: Versioning -> Snapshot

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>
2020-06-25 11:50:16 +02:00
Petros Kolyvas 4a3d0118c6
Merge pull request #25139 from xiaozhu36/master
backend/oss: supports setting HTTPS_PROXY
2020-06-16 14:49:34 -04:00
James Bardin 7800ef63d4
Merge pull request #19525 from dramich/k8sback
backend/kubernetes: Add Kubernetes as a backend
2020-06-08 15:59:32 -04:00
John Houston 01ebdd27f2 Rework soak test to error on unlock failure 2020-06-05 18:27:30 -04:00
John Houston b0f7e273e8 Add errors for lock disappearing 2020-06-05 18:10:24 -04:00
Brian Flad ba081aa10a
backend/s3: Updates for Terraform v0.13.0 (#25134)
* deps: Update github.com/hashicorp/aws-sdk-go-base@v0.5.0

Updated via:

```
$ go get github.com/hashicorp/aws-sdk-go-base@v0.5.0
$ go mod tidy
$ go mod vendor
```

* backend/s3: Updates for Terraform v0.13.0

Reference: https://github.com/hashicorp/terraform/issues/13410
Reference: https://github.com/hashicorp/terraform/issues/18774
Reference: https://github.com/hashicorp/terraform/issues/19482
Reference: https://github.com/hashicorp/terraform/issues/20062
Reference: https://github.com/hashicorp/terraform/issues/20599
Reference: https://github.com/hashicorp/terraform/issues/22103
Reference: https://github.com/hashicorp/terraform/issues/22161
Reference: https://github.com/hashicorp/terraform/issues/22601
Reference: https://github.com/hashicorp/terraform/issues/22992
Reference: https://github.com/hashicorp/terraform/issues/24252
Reference: https://github.com/hashicorp/terraform/issues/24253
Reference: https://github.com/hashicorp/terraform/issues/24480
Reference: https://github.com/hashicorp/terraform/issues/25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
2020-06-05 16:41:32 -04:00
John Houston e5f7237582 Soak test for kubernetes backend 2020-06-05 13:32:59 -04:00
He Guimin 5b4950e2cb backend(oss): supports setting HTTPS_PROXY 2020-06-05 14:48:58 +08:00
John Houston 23fb8f6d21 Add kubernetes backend
Co-authored-by: Dan Ramich <danold215@gmail.com>
2020-06-04 02:29:57 -04:00
Matthew Frahry 481b03c34a
Update azure backend storage sdk (#24669)
* update vendored azure sdk

* vendor giovanni storage sdk

* Add giovanni clients

* go mod vendor

* Swap to new storage sdk

* workable tests

* update .go-version to 1.14.2

* Tests working minus SAS

* Add SAS Token support

* Update vendor

* Passing tests

* Add date randomizer

* Captalize RG

* Remove random bits

* Update client var name

Co-authored-by: kt <kt@katbyte.me>
2020-05-20 17:29:02 +02:00
kayrus bd344f9d73 Sync auth options with upstream openstack provider 2020-04-25 08:39:33 +02:00
kayrus 50084f5b17 gofmt -w -s 2020-04-25 08:39:30 +02:00
Mathias Lafeldt 1f3a2c0e02
backend/remote-state/oss: Add test revealing bug in state locking 2020-02-19 11:41:56 +01:00
Mathias Lafeldt 07139e453a
backend/remote-state/oss: extract pkName constant 2020-02-18 18:47:15 +01:00
Mathias Lafeldt 6bb22907a1
backend/remote-state/oss: Fix state locking by using LockID as PK 2020-02-18 16:38:41 +01:00
Mathias Lafeldt b4a735779c
backend/remote-state/oss: Prepend bucket name to LockID
To allow using the same Tablestore table with multiple OSS buckets.

e.g. instead of env:/some/path/terraform.tfstate

the LockID now becomes some-bucket/env:/some/path/terraform.tfstate
2020-02-18 16:38:41 +01:00
Mathias Lafeldt 3b3739b0ca
backend/remote-state/oss: Add missing lock path to lock info
So that is shows up in lock errors, etc.
2020-02-18 16:38:41 +01:00
Mathias Lafeldt bbd2d46ac8
backend/remote-state/oss: Format code using goimports 2020-02-18 16:38:40 +01:00
Li Kexian 76e5b446ba
backend/cos: Add TencentCloud backend cos with lock (#22540)
* add TencentCloud COS backend for remote state

* add vendor of dependence

* fixed error not handle and remove default value for prefix argument

* get appid from TF_COS_APPID environment variables
2020-02-13 11:37:11 -05:00
Pam Selle a93298bd14
Merge pull request #23656 from hashicorp/paddy_gcs_backend_env_var
Add a backend-specific env var for the GCS backend.
2019-12-17 07:30:41 -05:00
Pam Selle 41675543c8
Merge pull request #23627 from alrs/fix-backend-remote-state-dropped-test-errs
Fix backend remote state dropped test errors
2019-12-12 15:21:29 -05:00
Paddy Carver b8752c7610 Add a backend-specific env var for the GCS backend.
Right now, the only environment variable available is the same
environment variable that will be picked up by the GCP provider. Users
would like to be able to store state in separate projects or accounts or
otherwise authenticate to the provider with a service account that
doesn't have access to the state. This seems like a reasonable enough
practice to me, and the solution seems straightforward--offer an
environment variable that doesn't mean anything to the provider to
configure the backend credentials. I've added GOOGLE_BACKEND_CREDENTIALS
to manage just the backend credentials, and documented it appropriately.
2019-12-12 03:35:39 -08:00
Pam Selle d45a4d33bb
Merge pull request #22963 from rekahsoft/s3-backend-pagination
Allow s3 backends to contain more then 1000 workspaces
2019-12-10 13:11:45 -05:00
Pam Selle 45fa6a3145 Fix S3 test 2019-12-10 12:48:08 -05:00
Lars Lehtonen 77ffb4a0eb
backend/remote-state/http: fix dropped test errors 2019-12-10 07:36:54 -08:00
Lars Lehtonen 2845896fc9
backend/remote-state/http: drop unused assertError() 2019-12-10 07:33:22 -08:00
Martin Atkins 30bf83cdeb helper/logging: Bring the LevelFilter into our own codebase
In order to make this work reasonably we can't avoid using some funny
heuristics, which are somewhat reasonable to apply within the context of
Terraform itself but would not be good to add to the general "logutils".

Specifically, this is adding the additional heuristic that lines starting
with spaces are continuation lines and so should inherit the log level
of the most recent non-continuation line.
2019-12-05 15:22:03 -08:00
Martin Atkins 6db3cf8e5b vendor: upgrade Azure SDK and Azure/go-autorest
This involves some minor changes to the "azure" backend code to account
for upstream breaking changes.
2019-12-02 10:04:49 -08:00
Joaquín Fernández Campo d2bc7c25b9 backend/artifactory: Honor HTTP_PROXY and HTTPS_PROXY environment variables 2019-11-06 14:18:33 -08:00
He Guimin bfae627112 add a new field ecs_role_name to support more scenario 2019-11-02 00:09:46 +08:00
Martin Atkins 39e609d5fd vendor: switch to HCL 2.0 in the HCL repository
Previously we were using the experimental HCL 2 repository, but now we'll
shift over to the v2 import path within the main HCL repository as part of
actually releasing HCL 2.0 as stable.

This is a mechanical search/replace to the new import paths. It also
switches to the v2.0.0 release of HCL, which includes some new code that
Terraform didn't previously have but should not change any behavior that
matters for Terraform's purposes.

For the moment the experimental HCL2 repository is still an indirect
dependency via terraform-config-inspect, so it remains in our go.sum and
vendor directories for the moment. Because terraform-config-inspect uses
a much smaller subset of the HCL2 functionality, this does still manage
to prune the vendor directory a little. A subsequent release of
terraform-config-inspect should allow us to completely remove that old
repository in a future commit.
2019-10-02 15:10:21 -07:00
Collin J. Doering 6d838ed76c
Allow s3 backends to contain more then 1000 workspaces
* backend/remote-state/s3/backend_state.go: Prior to this commit, the terraform s3 backend did
  not paginate calls to s3 when finding workspaces, which resulted in workspaces 'disappearing'
  once they are switched away from, even though the state file still exists. This is due to the
  ListBucket operation defaulting MaxItems to 1000, so terraform s3 backends that contained
  more then 1000 workspaces did not function as expected. This rectifies this situation by
  paginating calls to s3 when finding workspaces.

Signed-off-by: Collin J. Doering <collin@rekahsoft.ca>
2019-10-01 11:46:34 -04:00
Kristin Laemmert 006ef022e6
Merge pull request #22918 from hashicorp/mildwonkey/to-e2e-or-not-e2e
Sundry fixes to e2e tests
2019-09-27 08:46:04 -04:00
Kristin Laemmert 68dfc3046d backend/remote-state: etcdv3, oss, and manta acc tests should fail
faster

The acceptance tests for etcdv3, oss and manta were not validating
required env variablea, chosing to assume that if one was running
acceptance tests they had already configured the credentials.

It was not always clear if this was a bug in the tests or the provider,
so I opted to make the tests fail faster when required attributes were
unset (or "").
2019-09-27 08:45:12 -04:00
Kristin Laemmert 5300f85a79 backend/manta: fix panic when insecure_skip_tls_verify was not set
The DefaultFunc for insecure_skip_tls_verify was sending an empty string
instead of a bool. Fixes to default to `false`.
2019-09-26 10:08:29 -04:00