Commit Graph

606 Commits

Author SHA1 Message Date
Alex Pilon 4bf43efcfd
move hcl2shim package to configs 2019-08-06 19:58:58 -04:00
Chris Marchesi d43fc71135
modules: update go-azure-helpers to v0.5.0
To help address the issues posed on #22087 and #22085.
2019-08-01 11:57:34 -07:00
He Guimin a490dfa495 backend/oss: Support for assume role config 2019-07-30 23:27:17 +08:00
Kristin Laemmert 412d459292
backend/remote: remove milseading contents from error message (#22148)
Previously, terraform was returning a potentially-misleading error
message in response to anything other than a 404 from the
b.client.Workspaces.Read operation. This PR simplifies Terraform's error
message with the intent of encouraging those who encounter it to focus
on the error message returned from the tfe client.

The added test is odd, and a bit hacky, and possibly overkill.
2019-07-22 09:06:39 -04:00
Kristin Laemmert 190ef537ec backend/remote: notify users when uploading something other than cwd
When a TFC workspace is configured without a VCS root, and with a
working directory, and a user is running `terraform init` from that same
directory, TFC uploads the entire configuration directory, not only the
user's cwd. This is not obvious to the user, so we are adding a descriptive
message explaining what is being uploaded, and why.
2019-07-18 09:50:17 -07:00
Kristin Laemmert 89eeaed0a0
[WIP] backend/enhanced: start with absolute configuration path (#22096)
* backend/enhanced: start with absolute config path

We recently started normalizing the config path before all "command"
operations, which was necessary for consistency but had unexpected
consequences for remote backend operations, specifically when a vcs root
with a working directory are configured.

This PR de-normalizes the path back to an absolute path.

* Check the error and add a test

It turned out all required logic was already present, so I just needed to add a test for this specific use case.
2019-07-17 08:39:37 -04:00
Pam Selle 23a187d85d
Merge pull request #21739 from hasheddan/s3-backend-comment-typo
s3 backend: minor typo in putMD5 comment
2019-07-12 14:01:26 -04:00
Colin Fowler cd7bfba141 rebased to terraform master branch 2019-07-10 18:05:10 +01:00
Brian Williams 5e3c3bafb8 Add support for SSE-C to S3 backend
These changes add support for encrypting terraform remote-state in S3 using customer-supplied encryption keys (SSE-C).
2019-07-06 10:10:54 -05:00
Luis Silva f6c90c1d96 Add OAuth2 token support for GCS backend 2019-07-05 10:06:44 +01:00
Radek Simko 5b9f2fafc8 Standardise directory name for test data 2019-06-30 10:16:15 +02:00
Daniel Mangum 3bc14620c0 backend/remote-state: remove dead code
This remote-state adapter is no longer used, because the old remote state
mechanism was removed in Terraform v0.12.
2019-06-18 14:18:05 -07:00
hasheddan 961d43142c s3 backend: minor typo in putMD5 comment
Signed-off-by: hasheddan <georgedanielmangum@gmail.com>
2019-06-15 08:20:37 -05:00
Mary Cutrali e44ca40702 update remote apply language to use proper its/it's 2019-06-14 12:22:21 -05:00
Pam Selle cb593c25aa
Merge pull request #21381 from davidcelis/remove-cost-estimation
Remove pre-alpha cost estimation logic from remote backend
2019-06-13 10:59:56 -04:00
kayrus d06609dd23 Swift backend: add application credential support 2019-06-12 17:23:13 +02:00
Joe Topjian d8343aa95f backend/swift: Authentication updates
Support for cross-domain authentication has been added and mapping
environment variables to the correct domain settings has been
fixed.

In addition, support for clouds.yaml files has been added.
2019-06-12 08:05:54 -07:00
James Bardin 0c91d227fa
Merge branch 'master' into master 2019-06-10 15:50:59 -04:00
Ivan Kalita 5b6b1663ef backend/http: implement retries for the http backend (#19702)
Fixes #19619
2019-06-05 16:12:07 -04:00
Martin Atkins b1213f7f6c backend/local: don't panic when an instance has only a deposed object
This unusual situation isn't supposed to arise in normal use, but it can
come up in practice in some edge-case scenarios where Terraform fails in
a severe way during a create_before_destroy.

Some earlier versions of Terraform also had bugs in their handling of
deposed objects, so this may also arise if upgrading from one of those
older versions with some leftover deposed objects in the state.
2019-06-04 09:23:29 -07:00
Mateusz Gozdek b6de825723 backend/gcs: make Unlock error message more clear 2019-05-27 20:07:14 -07:00
David Celis 5ccccfb91f
Remove pre-alpha cost estimation logic 2019-05-20 16:48:38 -07:00
Radek Simko 8a6d1d62b6
stringer: Regenerate files with latest version 2019-05-13 15:34:27 +01:00
Stephen Buergler fc5b186e8d Don't leak so many connections in the pg backend
This change fixes an error I get:
Error: pq: too many connections for role "asdf"
because I can only have so many connections.
2019-04-30 23:34:51 -05:00
Sander van Harmelen 394f20f59c backend/remote: do not unlock after a failed upload
When changes are made and we failed to upload the state, we should not
try to unlock the workspace. Leaving the workspace locked is a good
indication something went wrong and also prevents other changes from
being applied before the newest state is properly uploaded.

Additionally we now output the lock ID when a lock or force-unlock
action failed.
2019-04-29 21:23:33 +02:00
Paul Thrasher 151c91ffda
use scanner for reading logs
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 11:17:08 -07:00
Paul Thrasher 0e27a8862f
remove duplicate remote output line
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-25 10:52:19 -07:00
Sander van Harmelen bb12206bca Fixup the tests 2019-04-25 10:32:00 +02:00
Sander van Harmelen 7cf744241a Do not use a scanner to read the logs
Using a scanner can cause issues when reading long lines. Also make sure we return the error correctly while planning.
2019-04-25 09:51:52 +02:00
Paul Thrasher c7a023a95c
update test for new go-tfe version
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:46:17 -07:00
Paul Thrasher 53f977bee2
TFCE api correct pluralization
Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Paul Thrasher e479bd5dc3
update to latest go-tfe
same version number but pointing to a new sha

Signed-off-by: Paul Thrasher <pthrasher@hashicorp.com>
2019-04-24 16:44:00 -07:00
Sander van Harmelen 90bc237b7b Prevent a panic caused by writing to a nil map 2019-04-11 14:34:14 +02:00
He Guimin 3f44dd9dec Add tablestore config to store state lock 2019-04-11 07:11:10 +08:00
He Guimin b887d44712 Add Alibaba Cloud backend OSS with lock 2019-04-11 07:06:43 +08:00
Sander van Harmelen 39a95e4222 backend/remote: correctly load remote variables
When using `terraform console` in combination with the remote backend, variables defined in Terraform Enterprise were load loaded correctly.
2019-03-28 17:23:48 +01:00
Sander van Harmelen 57f6e01830 backend/local: preserve serial and lineage on failure
When failing to write the state, the local backend writes the state to a local file called `errrored.tfstate`. Previously it would do so by creating a new state file which would use a new serial and lineage. By exorting the existing state file and directly assigning the new state, the serial and lineage are preserved.
2019-03-27 16:15:16 +01:00
Justin Downing 1e32ae243c grammatical updates to comments and docs (#20195) 2019-03-21 14:05:41 -07:00
Stefan Schmidt 3ca1253796 backend/gcs: Mark the unused attributes project and region as removed.
These two attributes are not used since aec45e6967.
2019-03-19 17:29:04 -07:00
Sander van Harmelen 9f6a126293 backend/remote: check for external updates 2019-03-08 19:18:07 +01:00
Sander van Harmelen 4628fbcc65
Merge pull request #20615 from hashicorp/svh/f-input-context
core: add a context to the UIInput interface
2019-03-08 18:44:28 +01:00
Sander van Harmelen e75e845804
Merge pull request #20588 from hashicorp/svh/f-retry
backend/remote: also retry on server errors
2019-03-08 18:17:29 +01:00
Sander van Harmelen 973e2a7cf9 core: add a context to the UIInput interface 2019-03-08 10:24:40 +01:00
Sander van Harmelen 0232d84a0d backend/remote: also retry on server errors
Enably retrying on server errors in the updated `go-tfe` client and add a retry log hook for writing retry messages to the CLI.
2019-03-06 13:36:06 +01:00
Kristin Laemmert b9d8e96e0c
command/plan: plan output should indicate if a resource is being (#20580)
replaced because the instance was tainted.
2019-03-05 16:18:55 -08:00
Martin Atkins 709487b4f1 backend: Cap number of "undeclared variable" warnings at four
For users who in previous versions have relied on our lack of checking for
whether variables are declared, they may previously have seen an
overwhelming number of warnings when running Terraform v0.12.

Here we cap that number at three specific warnings and then one general
warning, so we can still give a specific source location for the first
couple (for users who have genuinely made a typo) but summarize away a
large number for those who are seeing this because they've not yet
migrated to using environment variables.
2019-03-05 15:42:08 -08:00
Mars Hall 34fa67c6b0 Correct pg backend error info 2019-03-05 11:37:08 -08:00
Mars Hall ccf549a62d Stricter pg backend locking during workspace creation 2019-03-05 11:19:56 -08:00
Sergey Kondrashov 43e7a7b552 fix misspelling 2019-03-05 16:12:52 +03:00
Mars Hall 31c9776d55 Switch pg backend to session-level advisory locking, to avoid rollback of partial state updates 2019-03-04 17:01:08 -08:00
Brian Flad e34a97abca
backend/s3: Add debug logging and user agent
Porting over previous behavior from terraform-provider-aws.
2019-02-27 12:59:08 -05:00
Sander van Harmelen 63e2dcef8a
Merge pull request #20481 from hashicorp/svh/b-exit-code
backend/remote: exit with 1 when a run is canceled
2019-02-27 07:52:20 +01:00
James Bardin eada955721
Merge pull request #19070 from mars/postgres-backend
Postgres backend
2019-02-26 20:20:01 -05:00
Mars Hall 6d9e2048f4 Skip more pg backend acceptance tests 2019-02-26 16:50:36 -08:00
Mars Hall 920e7a7acc Mutex pg backend‘s database transaction 2019-02-26 13:45:16 -08:00
Mars Hall c6bf3442fc Lock() in pg backend now returns LockError for retryable errors 2019-02-26 13:20:40 -08:00
Mars Hall a075a68c58 Correct name of pg backend‘s default schema 2019-02-26 12:59:31 -08:00
Sander van Harmelen 01f17fa0ca backend/remote: exit with 1 when a run is canceled 2019-02-26 21:00:07 +01:00
James Bardin dd16455a79 fix s3 default region to match provider 2019-02-25 20:27:21 -05:00
Mars Hall 2621f95bd2 📚 doc corrections 2019-02-25 16:30:30 -08:00
Mars Hall b9a91b7c1e Switch pg backend to use native Postgres locks 2019-02-25 16:05:53 -08:00
James Bardin c814f2da37 Change backend.ValidateConfig to PrepareConfig
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
2019-02-25 18:37:20 -05:00
Sander van Harmelen 435b82d6d0
Merge pull request #20460 from hashicorp/svh/f-permissions
backend/remote: use the can-queue-apply permission
2019-02-25 19:32:20 +01:00
James Bardin aebfecf871
Merge pull request #20211 from yanndegat/swift_remote_backend
Swift remote backend support for workspaces & locking
2019-02-25 12:58:59 -05:00
Sander van Harmelen 58961026a2 backend/remote: use the can-queue-apply permission
Previously we checked can-update in order to determine if a user had the
required permissions to apply a run, but that wasn't sufficient. So we
added a new permission, can-queue-apply, that we now use instead.
2019-02-25 14:45:40 +01:00
Mars Hall 8cb2943b6b pg backend tests run when TF_ACC is set (no more TF_PG_TEST) 2019-02-22 14:42:24 -08:00
Mars Hall c8b8227967 Compatibility fixes for 0.12.0 2019-02-22 13:54:41 -08:00
Mars Hall d4d798c25e Fix to use existing pg indexes 2019-02-22 13:51:57 -08:00
Mars Hall 9feda7aab3 Correct code comment for pg 2019-02-22 13:51:57 -08:00
Mars Hall 6d7e8268e7 Include pg backend in main init 2019-02-22 13:51:57 -08:00
Mars Hall 9043005cfb Postgres database backend 2019-02-22 13:48:49 -08:00
James Bardin 31ca293777 fix slash handling around workspace_key_prefix
The handling of slashes was broken around listing workspaces in
workspace_key_prefix. While it worked in most places by splitting an
extra time around the spurious slashes, it failed in the case that the
prefix ended with a slash of its own.

A test was temporarily added to verify that the backend works with the
unusual keys, but rather than risking silent breakage around prefixes
with trailing slashes, we also add validation to prevent users from
entering keys with trailing slashes at all.
2019-02-22 10:29:10 -05:00
James Bardin f6af61f990 fix test that never worked
The 0.12 update left this test broken. Make sure we're working with the
correct objects here.
2019-02-22 10:14:53 -05:00
James Bardin 6cc3e1d0bd move init error to where it is generated
The init error was output deep in the backend by detecting a
special ResourceProviderError and formatted directly to the CLI.

Create some Diagnostics closer to where the problem is detected, and
passed that back through the normal diagnostic flow. While the output
isn't as nice yet, this restores the helpful error message and makes the
code easier to maintain. Better formatting can be handled later.
2019-02-20 14:18:37 -05:00
Brian Flad 185a3304e5
Merge pull request #20374 from hashicorp/td-backend-s3-dependencies
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
2019-02-20 08:42:07 -08:00
Brian Flad a41e545198
backend/s3: Reinstate region validation and update copy for skip_requesting_account_id deprecation message 2019-02-20 08:29:09 -08:00
Sander van Harmelen 43eb7c08e0
Merge pull request #20379 from hashicorp/svh/f-state-service
backend/remote: use `state.v2` for remote state only
2019-02-20 10:48:05 +01:00
Brian Flad 43f12bbfe0
backend/s3: Update copy for skip_get_ec2_platforms and skip_requesting_account_id argument deprecations 2019-02-19 10:54:53 -08:00
Sander van Harmelen 54736b068b backend/remote: use `state.v2` for remote state only
The API surface area is much smaller when we use the remote backend for remote state only.

So in order to try and prevent any backwards incompatibilities when TF runs inside of TFE, we’ve split up the discovery services into `state.v2` (which can be used for remote state only configurations, so when running in TFE) and `tfe.v2.1` (which can be used for all remote configurations).
2019-02-19 10:59:51 +01:00
Brian Flad 1aaac172b0
backend/s3: Switch from github.com/terraform-providers/terraform-provider-aws to github.com/hashicorp/aws-sdk-go-base
Output from acceptance testing (no new failures):

```
--- PASS: TestBackend_impl (0.00s)
--- PASS: TestBackendConfig (0.37s)
--- PASS: TestBackendConfig_invalidKey (0.00s)
--- PASS: TestBackend (3.26s)
--- PASS: TestBackendLocked (6.80s)
--- FAIL: TestBackendExtraPaths (2.32s)
--- PASS: TestBackendPrefixInWorkspace (2.06s)
--- PASS: TestKeyEnv (8.20s)
--- PASS: TestRemoteClient_impl (0.00s)
--- PASS: TestRemoteClient (2.42s)
--- PASS: TestRemoteClientLocks (6.33s)
--- PASS: TestForceUnlock (13.31s)
--- PASS: TestRemoteClient_clientMD5 (11.75s)
--- PASS: TestRemoteClient_stateChecksum (10.07s)
```
2019-02-18 02:30:30 -05:00
Yann DEGAT e70b8928e9 remote/backend/swift: Add support for workspaces & locking 2019-02-11 11:13:35 +01:00
Martin Atkins 6eb7bfbdfb
Merge #20265: Don't presume unknown for values unset in config
This changes the contract for `PlanResourceChange` so that the provider is now responsible
for populating all default values during plan, including inserting any unknown values for
defaults it will fill in at apply time.
2019-02-08 13:54:55 -08:00
Martin Atkins e3618f915b backend/local: Fix mock provider in tests
We've changed the contract for PlanResourceChange to now require the
provider to populate any default values (including unknowns) it wants to
set for computed arguments, so our mock provider here now needs to be a
little more complex to deal with that.

This fixes several of the tests in this package. A minor change to
TestLocal_applyEmptyDirDestroy was required to make it properly configure
the mock provider so PlanResourceChange can access the schema.
2019-02-08 12:48:32 -08:00
Radek Simko e6777105b7
Merge pull request #20275 from hashicorp/vendor-openstack-upgrade
vendor: Bump openstack provider to v1.15.0
2019-02-08 16:22:35 +00:00
Sander van Harmelen aefbec63b1 backend/remote: update the test logic 2019-02-08 16:56:37 +01:00
Sander van Harmelen 3b80f69eec
Merge pull request #20242 from hashicorp/svh/b-scanner-buffer-v0.12
backend/remote: fix bufio.Scanner: token too long
2019-02-08 16:52:22 +01:00
Radek Simko a7f0722729
backend/swift: Fix interface after upgrade 2019-02-08 11:03:03 +00:00
Sander van Harmelen 47a00ea34b backend/remote: cleanup test connections
Cleanup test connection to prevent file descriptor issues when running the tests on a Mac.
2019-02-07 09:55:19 +01:00
Sander van Harmelen 5249d0fe83 backend/remote: fix bufio.Scanner: token too long 2019-02-07 09:54:43 +01:00
Radek Simko 6365fffe4f
Fix interface after upgrade 2019-02-01 10:48:56 +00:00
Sander van Harmelen 6dd9f22780
Merge pull request #20086 from hashicorp/svh/f-service-constraints
backend/remote: make sure we show the correct error
2019-01-24 20:22:20 +01:00
Sander van Harmelen 1e4c20686e backend/remote: make sure we show the correct error
Previously we would show two errors when there was a version constraint
error. But of course one is enough.
2019-01-23 15:09:42 +01:00
Martin Atkins 10bf4c763b backend: Undeclared variables in -var-file is a warning, not an error
In Terraform 0.11 and earlier we just silently ignored undeclared
variables in -var-file and the automatically-loaded .tfvars files. This
was a bad user experience for anyone who made a typo in a variable name
and got no feedback about it, so we made this an error for 0.12.

However, several users are now relying on the silent-ignore behavior for
automation scenarios where they pass the same .tfvars file to all
configurations in their organization and expect Terraform to ignore any
settings that are not relevant to a specific configuration. We never
intentionally supported that, but we don't want to immediately break that
workflow during 0.12 upgrade.

As a compromise, then, we'll make this a warning for v0.12.0 that contains
a deprecation notice suggesting to move to using environment variables
for this "cross-configuration variables" use-case. We don't produce errors
for undeclared variables in environment variables, even though that
potentially causes the same UX annoyance as ignoring them in vars files,
because environment variables are assumed to live in the user's session
and this it would be very inconvenient to have to unset such variables
when moving between directories. Their "ambientness" makes them a better
fit for these automatically-assigned general variable values that may or
may not be used by a particular configuration.

This can revert to being an error in a future major release, after users
have had the opportunity to migrate their automation solutions over to
use environment variables.

We don't seem to have any tests covering this specific situation right
now. That isn't ideal, but this change is so straightforward that it would
be relatively expensive to build new targeted test cases for it and so
I instead just hand-tested that it is indeed now producing a warning where
we were previously producing an error. Hopefully if there is any more
substantial work done on this codepath in future that will be our prompt
to add some unit tests for this.
2019-01-22 07:46:51 -08:00
Sander van Harmelen 8937fedb76 backend/remote: fix a small typo 2019-01-18 19:40:57 +01:00
Sander van Harmelen e08a7e979e backend/remote: use the correct test operation 2019-01-15 16:13:16 +01:00
Martin Atkins 0c0a437bcb Move module install functionality over to internal/initwd 2019-01-14 11:33:21 -08:00
Brian Flad ed37d07632
backend/s3: Configure AWS Client MaxRetries and provide enhanced S3 NoSuchBucket error message
The AWS Go SDK automatically provides a default request retryer with exponential backoff that is invoked via setting `MaxRetries` or leaving it `nil` will default to 3. The terraform-aws-provider `config.Client()` sets `MaxRetries` to 0 unless explicitly configured above 0. Previously, we were not overriding this behavior by setting the configuration and therefore not invoking the default request retryer.

The default retryer already handles HTTP error codes above 500, including S3's InternalError response, so the extraneous handling can be removed. This will also start automatically retrying many additional cases, such as temporary networking issues or other retryable AWS service responses.

Changes:
* s3/backend: Add `max_retries` argument
* s3/backend: Enhance S3 NoSuchBucket error to include additional information
2019-01-09 13:01:37 -05:00
Sander van Harmelen bba03384d5 backend/remote: log early to indicate execution started 2019-01-08 17:06:24 +01:00
Sander van Harmelen 7b51af72b2 backend/remote: compare versions without the prerelease 2018-12-19 19:06:22 +01:00
Sander van Harmelen 55b6153b04 backend/remote: fix an error that prevents checking constraints 2018-12-15 21:36:47 +01:00
Sander van Harmelen 2ef8315885
Merge pull request #19659 from hashicorp/svh/f-check-constraints
backend/remote: return detailed incompatibility info
2018-12-14 21:26:02 +01:00
Sander van Harmelen 8f04e93739 backend/remote: return detailed incompatibility info 2018-12-14 21:11:41 +01:00
Radek Simko f64978b64c
backend/local: Render CBD replacement (+/-) correctly (#19642)
* backend/local: Render CBD replacement (+/-) correctly

* command/format: Use IsReplace helper function
2018-12-14 13:45:47 +00:00
Sander van Harmelen da8e02eb2e
Merge pull request #19647 from hashicorp/svh/f-versions
core:  add a method to the disco package retrieve version constraints
2018-12-14 12:38:52 +01:00
Sander van Harmelen 268c0f85ce Add a method to retrieve version contraints 2018-12-14 12:17:31 +01:00
Radek Simko 3ab4739ba4
backend/local: Avoid rendering data sources on destroy 2018-12-12 18:21:49 +00:00
Radek Simko e55079abc9
backend/local: Fix incorrect destroy/update count on apply 2018-12-11 18:06:53 +00:00
Brian Flad 058434d28b
Merge pull request #19571 from hashicorp/f-backend-s3-other-endpoints
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
2018-12-10 19:28:01 -05:00
Tom Harvey 383bc98f5c
backend/azurerm: Support for authenticating using the Azure CLI (#19465)
* Upgrading to 2.0.0 of github.com/hashicorp/go-azure-helpers

* Support for authenticating using Azure CLI

* backend/azurerm: support for authenticating using the Azure CLI
2018-12-10 21:23:30 +00:00
Sander van Harmelen a5a2156584 core: enhance service discovery
This PR improves the error handling so we can provide better feedback about any service discovery errors that occured.

Additionally it adds logic to test for specific versions when discovering a service using `service.vN`. This will enable more informational errors which can indicate any version incompatibilities.
2018-12-10 20:52:05 +01:00
Brian Flad 9a3b02cd6c
backend/s3: Support DynamoDB, IAM, and STS endpoint configurations
This change enables a few related use cases:
* AWS has partitions outside Commercial, GovCloud (US), and China, which are the only endpoints automatically handled by the AWS Go SDK. DynamoDB locking and credential verification can not currently be enabled in those regions.
* Allows usage of any DynamoDB-compatible API for state locking
* Allows usage of any IAM/STS-compatible API for credential verification
2018-12-07 03:10:51 -05:00
Sander van Harmelen 9062d887b8 backend/remote: use entitlements to select backends
Use the entitlements to a) determine if the organization exists, and b) as a means to select which backend to use (the local backend with remote state, or the remote backend).
2018-12-05 12:29:08 +01:00
Kristin Laemmert 67aa944166 backend/local: decode variables with cty.DynamicPseudoType
Variables values are marshalled with an explicit type of
cty.DynamicPseudoType, but were being decoded using `Implied Type` to
try and guess the type. This was causing errors because `Implied Type`
does not expect to find a late-bound value.
2018-11-30 15:15:28 -08:00
Martin Atkins 12572e97bc core: Automatically upgrade resource instance states on read
If an instance object in state has an earlier schema version number then
it is likely that the schema we're holding won't be able to decode the
raw data that is stored. Instead, we must ask the provider to upgrade it
for us first, which might also include translating it from flatmap form
if it was last updated with a Terraform version earlier than v0.12.

This ends up being a "seam" between our use of int64 for schema versions
in the providers package and uint64 everywhere else. We intend to
standardize on int64 everywhere eventually, but for now this remains
consistent with existing usage in each layer to keep the type conversion
noise contained here and avoid mass-updates to other Terraform components
at this time.

This also includes a minor change to the test helpers for the
backend/local package, which were inexplicably setting a SchemaVersion of
1 on the basic test state but setting the mock schema version to zero,
creating an invalid situation where the state would need to be downgraded.
2018-11-30 11:22:39 -08:00
Sander van Harmelen fe05609c5e backend/remote: support the new force-unlock API
Add support for the new `force-unlock` API and at the same time improve
performance a bit by reducing the amount of API calls made when using
the remote backend for state storage only.
2018-11-30 19:39:18 +01:00
Sander van Harmelen 4c878db8fd
Merge pull request #19464 from hashicorp/svh/f-context
backend/remote: implement the Local interface
2018-11-28 20:16:58 +01:00
Sander van Harmelen a061725132
Merge pull request #19403 from hashicorp/svh/f-policies
backend/remote: also show policies when there are no changes
2018-11-28 16:30:31 +01:00
Martin Atkins 168d84b3c4 core: Make resource type schema versions visible to callers
Previously we were fetching these from the provider but then immediately
discarding the version numbers because the schema API had nowhere to put
them.

To avoid a late-breaking change to the internal structure of
terraform.ProviderSchema (which is constructed directly all over the
tests) we're retaining the resource type schemas in a new map alongside
the existing one with the same keys, rather than just switching to
using the providers.Schema struct directly there.

The methods that return resource type schemas now return two arguments,
intentionally creating a little API friction here so each new caller can
be reminded to think about whether they need to do something with the
schema version, though it can be ignored by many callers.

Since this was a breaking change to the Schemas API anyway, this also
fixes another API wart where there was a separate method for fetching
managed vs. data resource types and thus every caller ended up having a
switch statement on "mode". Now we just accept mode as an argument and
do the switch statement within the single SchemaForResourceType method.
2018-11-27 15:53:54 -08:00
Sander van Harmelen 35d9ce3f92 backend/remote: implement the Local interface 2018-11-26 20:50:25 +01:00
Tom Harvey 6d4f702467
backend/azurerm: support for custom resource manager endpoints (#19460)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail

* authentication: support for custom resource manager endpoints

* Adding debug prefixes to the log statements
2018-11-26 14:42:16 +01:00
Tom Harvey d580f30e03
backend/azurerm: removing the `arm_` prefix from keys (#19448)
* backend/azurerm: removing the `arm_` prefix from keys

* removing the deprecated fields test because the deprecation makes it fail
2018-11-26 11:19:43 +01:00
Tom Harvey 96b1c951fa
backend/azurerm: support for authenticating via SAS Tokens (#19440)
* adding acceptance tests for msi auth

* including the resource group name in the tests

* backend/azurerm: support for authenticating using a SAS Token

* resolving merge conflicts

* moving the defer to prior to the error
2018-11-22 18:02:33 +01:00
Tom Harvey e437feadc2
backend/azurerm: fixing a bug where locks couldn't be unlocked (#19441)
Fixes #17046
2018-11-22 17:47:35 +01:00
Tom Harvey c928962f44
backend/azurerm: support for authenticating via msi (#19433)
* backend/azurerm: support for authenticating via msi

* adding acceptance tests for msi auth

* including the resource group name in the tests

* support for using the test client via msi
2018-11-22 16:52:27 +01:00
Tom Harvey 0ec109bdc0
backend/azurerm: upgrading the SDK / support for proxies (#19414)
* vendor updates

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go
- updating to v10.15.4 of github.com/Azure/go-autorest
- vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

* backend/azurerm: refactoring to use the new auth package

- refactoring the backend to use a shared client via the new auth package
- adding tests covering both Service Principal and Access Key auth
- support for authenticating using a proxy
- rewriting the backend documentation to include examples of both authentication types

* switching to use the build-in logging function

* documenting it's also possible to retrieve the access key from an env var
2018-11-21 22:06:03 +01:00
Sander van Harmelen 4561c80c1d Also show policies when there are no changes
This behavior was recently updated in the TFE UI, so lets follow that behavior here as well.
2018-11-21 11:34:59 +01:00
Sander van Harmelen a17f317025 Change how to fall back from remote to local backend
In order to support free organizations, we need a way to load the `remote` backend and then, depending on the used offering/plan, enable or disable remote operations.

In other words, we should be able to dynamically fall back to the `local` backend if needed, after first configuring the `remote` backend.

To make this works we need to change the way this was done previously when the env var `TF_FORCE_LOCAL_BACKEND` was set. The clear difference of course being that the env var would be available on startup, while the used offering/plan is only known after being able to connect to TFE.
2018-11-20 22:25:52 +01:00
Martin Atkins 27abd9c6b8 command: Fix TestMetaBackend_localDoesNotDeleteLocal
The changes to how we handle setting the state path on the local backend
broke the heuristic we were using here for detecting migration from one
local backend to another with the same state path, which would by default
end up deleting the state altogether after migration.

We now use the StatePaths method to do this, which takes into account
both the default values and any settings that have been set.

Additionally this addresses a flaw in the old method which could
potentially have deleted all non-default workspace state files if the
"path" setting were changed without also changing the "workspace_dir"
setting. This new approach is conservative because it will preserve all
of the files if any one overlaps.
2018-11-19 09:02:35 -08:00
Martin Atkins ec27526cc3 command: Fix TestMetaBackend_configuredChangeCopy_multiToMulti
This was failing because we now handle the settings for the local backend
a little differently as a result of decoding it with the HCL2 machinery.

Specifically, the backend.State* fields are now assumed to be what is
given in configuration, and any CLI overrides are maintained separately
in OverrideState* fields so that they can be imposed "just in time" in
StatePaths.

This is particularly important because OverrideStatePath (when set) is
used regardless of workspace name, while StatePath is a suitable value
only for the "default" workspace, with others needing to be constructed
from StateWorkspaceDir instead.
2018-11-19 09:02:35 -08:00
Sander van Harmelen 04439595ab Make the Atlas backend work after updating depencies
Newer versions of the retryablehttp package use a context, so we need to
add that in our custom `CheckRetry` function.

In addition I removed the `return true, nil` to continue retrying in
case of an error, and instead directly call the `DefaultRetryPolicy`.
This is because the `DefaultRetryPolicy` will now also take the context
into consideration.
2018-11-15 20:34:31 +01:00
Sander van Harmelen 52a1b22f7a Implement the remote enhanced backend
This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.
2018-11-06 16:29:46 +01:00
Sander van Harmelen b62a22ab62 Add a VariableSourceType for names .tfvars files
This new source type should be used for variables loaded from .tfvars files that were explicitly passed as command line arguments (e.g. -var-file=foo.tfvars)
2018-11-05 19:29:34 +01:00
Sander van Harmelen b846c5f653 Make the test compile again
Not sure if these checks still make sense, but without this change the test don’t compile for the related packages.
2018-10-19 15:29:14 +02:00
Sander van Harmelen 48ef7ecfa6 Updates after running `make fmt` with Go v1.11.1 2018-10-17 14:11:08 -07:00
James Bardin 0a0eece15c add backend cli options after configuration
The cli should override the config
2018-10-17 09:41:58 -04:00
Martin Atkins 541952bb8f Revert some work that happened since v0.12-dev branched
This work was done against APIs that were already changed in the branch
before work began, and so it doesn't apply to the v0.12 development work.

To allow v0.12 to merge down to master, we'll revert this work out for now
and then re-introduce equivalent functionality in later commits that works
against the new APIs.
2018-10-16 19:48:28 -07:00
Martin Atkins 98bbd560b5 command: Fix most (but not all) "terraform plan" tests 2018-10-16 19:14:11 -07:00
Martin Atkins 741d334ee4 command: Even more fixes for "apply" command tests 2018-10-16 19:14:11 -07:00
Martin Atkins c5940f2438 backend/local: Increase log verbosity for backend context construction
There are several steps here and a number of them can include reaching out
to remote servers or executing local processes, so it's helpful to have
some trace logs to better narrow down causes of errors and hangs during
this step.
2018-10-16 19:14:11 -07:00
Martin Atkins a6f399517b backend/local: reinstate additional steps in plan file processing
In earlier refactoring we skipped implementing prior state safety checks,
propagating the target addresses from plan, and verifying that all of
the providers are exactly the same from the plan being created.

This change reinstates those checks, including a new error message for
the "stale plan" situation.
2018-10-16 19:14:11 -07:00
Martin Atkins 91d2de6a25 backend/local: Stub out remaining planfile todos with errors
This is just to make sure they show up later when we are working on the
tests, so we can be sure not to forget to address them.
2018-10-16 19:14:11 -07:00
Martin Atkins fa3b4fc9de backend/local: Populate changes into a context built from planfile 2018-10-16 19:14:11 -07:00
Martin Atkins cbc548eb36 command: Do CLI init of backend loaded from plan
If we don't do this, we can't produce any output when applying a saved
plan file.

Here we also introduce a check to the local backend's ReportResult
function so that it won't panic if CLI init is skipped, although that
will no longer happen in the apply-from-file case due to the change
described in the previous paragraph.
2018-10-16 19:14:11 -07:00
Martin Atkins 2b80df0163 backend/local: Require caller to set PlanOutBackend with PlanOutPath
We can't generate a valid plan file without a backend configuration to
write into it, but it's the responsibility of the caller (the command
package) to manage the backend configuration mechanism, so we require it
to tell us what to write here.

This feels a little strange because the backend in principle knows its
own config, but in practice the backend only knows the _processed_ version
of the config, not the raw configuration value that was used to configure
it.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 2d3cb87789 backend/local tests tests tests
converted the existing testPlanState() from terraform.State to
states.State to fix various plan tests.

reverted the "bandaid" in plans/planfile/tfplan.go - at this moment the
backend tests do not include backend configuration, and so the planfile
package can write the plan file but not read it back in. That will be
revisted in a separate track of work.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 6a37ee9277 backend/local: more tests passing
I have no confidence in the change to plans/planfile/tfplan.go. The
tests were passing an empty backend config, which planfile  was able to
write to a file but not read from the same file. This change let me move
past that and it did not break any tests in the planfile package, but I
am concerned that it introduces undesired behavior.
2018-10-16 19:14:11 -07:00
Kristin Laemmert 56b879d0c0 backend/local: updated DiffFn adn ReadResourceFn for new models 2018-10-16 19:14:11 -07:00
Kristin Laemmert 64f696d9b3 backend/local refresh tests refactor 2018-10-16 19:14:11 -07:00
Kristin Laemmert e84def1e29 backend/local: test fixes
for TestLocal_applyBackendFail, config is relative to the tests' wd
2018-10-16 19:14:11 -07:00
Kristin Laemmert d865df7338 revert to explicitly declare schema for TestLocal_applyEmptyDirDestroy 2018-10-16 19:14:11 -07:00