Commit Graph

313 Commits

Author SHA1 Message Date
Upo 4ccc63d79d
backend: Add service account impersonation to GCS Backend and update the docs (#26700) 2020-11-06 09:06:07 -05:00
James Bardin 0b31ffa587 use a single log writer
Use a single log writer instance for all std library logging.

Setup the std log writer in the logging package, and remove boilerplate
from test packages.
2020-10-19 14:29:54 -04:00
James Bardin 6ca477f042 move helper/logging to internal
remove a dead code file too
2020-10-19 14:27:53 -04:00
James Bardin b8df47c9ac add struct field names 2020-10-14 14:05:41 -04:00
Rémi Lapeyre 12a0a21c0b Add skip_table_creation and skip_index_creation options to the pg backend
Closes https://github.com/hashicorp/terraform/issues/25708
2020-10-12 22:47:19 +02:00
Kristin Laemmert 6f4f6fbc72 helper: deprecate pathorcontents
pathorcontents was solely used by the gcs backend. I moved the function
into the backend package so it could still be used by other backends for
good measure.
2020-10-08 08:42:16 -04:00
Kristin Laemmert d2e999ba1f
remove unused code (#26503)
* remove unused code

I've removed the provider-specific code under registry, and unused nil
backend, and replaced a call to helper from backend/oss (the other
callers of that func are provisioners scheduled to be deprecated).

I also removed the Dockerfile, as our build process uses a different
file.

Finally I removed the examples directory, which had outdated examples
and links. There are better, actively maintained examples available.

* command: remove various unused bits

* test wasn't running

* backend: remove unused err
2020-10-07 11:00:06 -04:00
Pam Selle f84a7c1d57
Merge pull request #26476 from remilapeyre/postgres-backend-escape-schema_name
Properly quote schema_name in the pg backend configuration
2020-10-05 14:57:53 -04:00
James Bardin 70652814e3 add missing Context argument 2020-10-05 14:43:46 -04:00
James Bardin 5e13d57f4e
Merge pull request #26463 from hashicorp/kt/azurerm-secret-backend-fix
AzureRM backend: correctly lookup environment from metadata host
2020-10-05 14:26:32 -04:00
Alisdair McDiarmid 35d4926bf7
Merge pull request #25837 from remilapeyre/consul-force-unlock
Fix `terraform force-unlock <lock_id>` for Consul backend
2020-10-05 13:11:08 -04:00
Rémi Lapeyre 42be5854a2 Properly quote schema_name in the pg backend configuration 2020-10-03 18:02:13 +02:00
kt b66364f7aa
AzureRM backend: correctly lookup environment from metadata host 2020-10-01 16:34:51 -07:00
Rémi Lapeyre 9acfe30ca6 Always have the default workspace in the pg backend
Closes https://github.com/hashicorp/terraform/issues/23121
2020-09-29 23:18:34 +02:00
Rémi Lapeyre 5c7008bd89 Merge remote-tracking branch 'origin/master' into consul-force-unlock 2020-09-29 10:59:36 +02:00
Alisdair McDiarmid e183163739
Merge pull request #25856 from remilapeyre/consul-split-state
Split the state in chunks when they outgrow the limit of Consul KV store
2020-09-15 10:00:25 -04:00
Alisdair McDiarmid 92abaadc02
Merge pull request #25842 from remilapeyre/consul-path-slash
Sanitize lock path for the Consul backend when it ends with a /
2020-09-11 11:14:49 -04:00
Matt Kasa c2e35ae3b5
backend/http: add support for configuration by environment variable (#25439) 2020-09-02 08:35:08 -04:00
James Bardin 0c5430f88d
Merge pull request #25585 from chushenmeshile/master
fix ossbackend only returned 100 workspaces
2020-08-19 16:55:31 -04:00
James Bardin bd5ff98ca7
Merge pull request #25830 from hashicorp/kt/secret-metadata-host
backend/azurerm: support loading environments from a metadata host
2020-08-19 13:57:10 -04:00
James Bardin d46e9a4198
Merge pull request #25770 from hashicorp/f/azure-backend-spcert
backend/azurerm: support for authenticating using a Client Certificate
2020-08-19 13:56:30 -04:00
Angie Pinilla 582fb73a7c deps: Update aws-sdk-go-based to 0.6.0 2020-08-18 12:49:31 -04:00
Rémi Lapeyre e680211bc0 Split the state in chunks when they outgrow the limit of Consul KV store
The Consul KV store limits the size of the values in the KV store to 524288
bytes. Once the state reaches this limit Consul will refuse to save it. It is
currently possible to try to bypass this limitation by enable Gzip but the issue
will manifest itself later. This is particularly inconvenient as it is possible
for the state to reach this limit without changing the Terraform configuration
as datasources or computed attributes can suddenly return more data than they
used to. Several users already had issues with this.

To fix the problem once and for all we now split the payload in chunks of 524288
bytes when they are to large and store them separatly in the KV store. A small
JSON payload that references all the chunks so we can retrieve them later and
concatenate them to reconstruct the payload.

While this has the caveat of requiring multiple calls to Consul that cannot be
done as a single transaction as those have the same size limit, we use unique
paths for the chunks and CAS when setting the last payload so possible issues
during calls to Put() should not result in unreadable states.

Closes https://github.com/hashicorp/terraform/issues/19182
2020-08-14 17:38:18 +02:00
Rémi Lapeyre 032d339915 Sanitize lock path for the Consul backend when it ends with a /
When the path ends with / (e.g. `path = "tfstate/"), the lock
path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which
Consul does not accept.

This change the lock path so it is sanitized to `tfstate/.lock`.

If the user has two different Terraform project, one with `path = "tfstate"` and
the other with `path = "tfstate/"`, the paths for the locks will be the same
which will be confusing as locking one project will lock both. I wish it were
possible to forbid ending slashes altogether but doing so would require all
users currently having an ending slash in the path to manually move their
Terraform state and would be a poor user experience.

Closes https://github.com/hashicorp/terraform/issues/15747
2020-08-13 16:29:43 +02:00
Rémi Lapeyre 11eb88753d Fix `terraform force-unlock <lock_id>` for Consul backend
When locking was enabled with the Consul backend and the lock not properly
released, the `terraform force-unlock <lock_id>` command would do nothing as
its implementation would exit early in that case.

It now destroys the session that created the lock and clean both the lock and
the lock-info keys.

A regression test is added to TestConsul_destroyLock() to catch the issue if it
happends again.

Closes https://github.com/hashicorp/terraform/issues/22174
2020-08-13 15:20:24 +02:00
kt 23b4c2db42
azurerm backend: support loading metadata from a given host 2020-08-12 15:04:40 -07:00
Kristin Laemmert 6621501ae3
state: remove deprecated state package (#25490)
Most of the state package has been deprecated by the states package.
This PR replaces all the references to the old state package that
can be done simply - the low-hanging fruit.

* states: move state.Locker to statemgr

The state.Locker interface was a wrapper around a statemgr.Full, so
moving this was relatively straightforward.

* command: remove unnecessary use of state package for writing local terraform state files

* move state.LocalState into terraform package

state.LocalState is responsible for managing terraform.States, so it
made sense (to me) to move it into the terraform package.

* slight change of heart: move state.LocalState into clistate instead of
terraform
2020-08-11 11:43:01 -04:00
tombuildsstuff d215453495 backend/azurerm: adding a missing docs string. fixes #25765 2020-08-07 12:01:16 +02:00
tombuildsstuff 0d34e5d97c backend/azurerm: support for authenticating using a Client Certificate
fixes #24179
supersedes #19606
dependent on #25769
2020-08-07 11:58:33 +02:00
zjd 25de713214 fix ossbackend only returned 100 workspaces 2020-07-21 11:18:00 +08:00
Even Holthe 0f85b283b9
Azure backend: support snapshots/versioning (#24069)
* Azure backend: support snapshots/versioning

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>

* Azure backend: Versioning -> Snapshot

Co-authored-by: Reda Ahdjoudj <reda.ahdjoudj@gmail.com>
Co-authored-by: Patrick F. Marques <patrickfmarques@gmail.com>
2020-06-25 11:50:16 +02:00
Petros Kolyvas 4a3d0118c6
Merge pull request #25139 from xiaozhu36/master
backend/oss: supports setting HTTPS_PROXY
2020-06-16 14:49:34 -04:00
James Bardin 7800ef63d4
Merge pull request #19525 from dramich/k8sback
backend/kubernetes: Add Kubernetes as a backend
2020-06-08 15:59:32 -04:00
John Houston 01ebdd27f2 Rework soak test to error on unlock failure 2020-06-05 18:27:30 -04:00
John Houston b0f7e273e8 Add errors for lock disappearing 2020-06-05 18:10:24 -04:00
Brian Flad ba081aa10a
backend/s3: Updates for Terraform v0.13.0 (#25134)
* deps: Update github.com/hashicorp/aws-sdk-go-base@v0.5.0

Updated via:

```
$ go get github.com/hashicorp/aws-sdk-go-base@v0.5.0
$ go mod tidy
$ go mod vendor
```

* backend/s3: Updates for Terraform v0.13.0

Reference: https://github.com/hashicorp/terraform/issues/13410
Reference: https://github.com/hashicorp/terraform/issues/18774
Reference: https://github.com/hashicorp/terraform/issues/19482
Reference: https://github.com/hashicorp/terraform/issues/20062
Reference: https://github.com/hashicorp/terraform/issues/20599
Reference: https://github.com/hashicorp/terraform/issues/22103
Reference: https://github.com/hashicorp/terraform/issues/22161
Reference: https://github.com/hashicorp/terraform/issues/22601
Reference: https://github.com/hashicorp/terraform/issues/22992
Reference: https://github.com/hashicorp/terraform/issues/24252
Reference: https://github.com/hashicorp/terraform/issues/24253
Reference: https://github.com/hashicorp/terraform/issues/24480
Reference: https://github.com/hashicorp/terraform/issues/25056

Changes:

```
NOTES

* backend/s3: Deprecated `lock_table`, `skip_get_ec2_platforms`, `skip_requesting_account_id` arguments have been removed
* backend/s3: Credential ordering has changed from static, environment, shared credentials, EC2 metadata, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata) to static, environment, shared credentials, default AWS Go SDK (shared configuration, web identity, ECS, EC2 Metadata)
* The `AWS_METADATA_TIMEOUT` environment variable no longer has any effect as we now depend on the default AWS Go SDK EC2 Metadata client timeout of one second with two retries

ENHANCEMENTS

* backend/s3: Always enable shared configuration file support (no longer require `AWS_SDK_LOAD_CONFIG` environment variable)
* backend/s3: Automatically expand `~` prefix for home directories in `shared_credentials_file` argument
* backend/s3: Add `assume_role_duration_seconds`, `assume_role_policy_arns`, `assume_role_tags`, and `assume_role_transitive_tag_keys` arguments

BUG FIXES

* backend/s3: Ensure configured profile is used
* backend/s3: Ensure configured STS endpoint is used during AssumeRole API calls
* backend/s3: Prefer AWS shared configuration over EC2 metadata credentials
* backend/s3: Prefer ECS credentials over EC2 metadata credentials
* backend/s3: Remove hardcoded AWS Provider messaging
```

Output from acceptance testing:

```
--- PASS: TestBackend (16.32s)
--- PASS: TestBackendConfig (0.58s)
--- PASS: TestBackendConfig_AssumeRole (0.02s)
--- PASS: TestBackendConfig_conflictingEncryptionSchema (0.00s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyEncoding (0.00s)
--- PASS: TestBackendConfig_invalidSSECustomerKeyLength (0.00s)
--- PASS: TestBackendExtraPaths (13.21s)
--- PASS: TestBackendLocked (28.98s)
--- PASS: TestBackendPrefixInWorkspace (5.65s)
--- PASS: TestBackendSSECustomerKey (17.60s)
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestForceUnlock (17.50s)
--- PASS: TestKeyEnv (50.25s)
--- PASS: TestRemoteClient (4.78s)
--- PASS: TestRemoteClientLocks (16.85s)
--- PASS: TestRemoteClient_clientMD5 (12.08s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient_stateChecksum (17.92s)
```
2020-06-05 16:41:32 -04:00
John Houston e5f7237582 Soak test for kubernetes backend 2020-06-05 13:32:59 -04:00
He Guimin 5b4950e2cb backend(oss): supports setting HTTPS_PROXY 2020-06-05 14:48:58 +08:00
John Houston 23fb8f6d21 Add kubernetes backend
Co-authored-by: Dan Ramich <danold215@gmail.com>
2020-06-04 02:29:57 -04:00
Matthew Frahry 481b03c34a
Update azure backend storage sdk (#24669)
* update vendored azure sdk

* vendor giovanni storage sdk

* Add giovanni clients

* go mod vendor

* Swap to new storage sdk

* workable tests

* update .go-version to 1.14.2

* Tests working minus SAS

* Add SAS Token support

* Update vendor

* Passing tests

* Add date randomizer

* Captalize RG

* Remove random bits

* Update client var name

Co-authored-by: kt <kt@katbyte.me>
2020-05-20 17:29:02 +02:00
kayrus bd344f9d73 Sync auth options with upstream openstack provider 2020-04-25 08:39:33 +02:00
kayrus 50084f5b17 gofmt -w -s 2020-04-25 08:39:30 +02:00
Mathias Lafeldt 1f3a2c0e02
backend/remote-state/oss: Add test revealing bug in state locking 2020-02-19 11:41:56 +01:00
Mathias Lafeldt 07139e453a
backend/remote-state/oss: extract pkName constant 2020-02-18 18:47:15 +01:00
Mathias Lafeldt 6bb22907a1
backend/remote-state/oss: Fix state locking by using LockID as PK 2020-02-18 16:38:41 +01:00
Mathias Lafeldt b4a735779c
backend/remote-state/oss: Prepend bucket name to LockID
To allow using the same Tablestore table with multiple OSS buckets.

e.g. instead of env:/some/path/terraform.tfstate

the LockID now becomes some-bucket/env:/some/path/terraform.tfstate
2020-02-18 16:38:41 +01:00
Mathias Lafeldt 3b3739b0ca
backend/remote-state/oss: Add missing lock path to lock info
So that is shows up in lock errors, etc.
2020-02-18 16:38:41 +01:00
Mathias Lafeldt bbd2d46ac8
backend/remote-state/oss: Format code using goimports 2020-02-18 16:38:40 +01:00
Li Kexian 76e5b446ba
backend/cos: Add TencentCloud backend cos with lock (#22540)
* add TencentCloud COS backend for remote state

* add vendor of dependence

* fixed error not handle and remove default value for prefix argument

* get appid from TF_COS_APPID environment variables
2020-02-13 11:37:11 -05:00
Pam Selle a93298bd14
Merge pull request #23656 from hashicorp/paddy_gcs_backend_env_var
Add a backend-specific env var for the GCS backend.
2019-12-17 07:30:41 -05:00