Commit Graph

651 Commits

Author SHA1 Message Date
Sander van Harmelen 62ab84a319 backend/remote: compat changes for latest `go-tfe` 2018-09-08 11:47:18 +02:00
Sander van Harmelen 3cfc83e335 backend/remote: take working directories into account
In TFE you can configure a workspace to use a custom working directory. When determining which directory that needs to be uploaded to TFE, this working directory should be taken into account to make sure we are uploading the correct root directory for the workspace.
2018-08-31 20:35:16 +02:00
Sander van Harmelen ce2869dced
Merge pull request #18760 from hashicorp/f-remote-backend
backend/migrations: migrate the default state
2018-08-30 11:16:25 +02:00
Sander van Harmelen f410a5bb26 backend/migrations: migrate the default state
Certain backends (currently only the `remote` backend) do not support using both the default and named workspaces at the same time.

To make the migration easier for users that currently use both types of workspaces, this commit adds logic to ask the user for a new workspace name during the migration process.
2018-08-29 21:37:39 +02:00
Sander van Harmelen 77b9fad7f0 backend/manta: deprecate camelcase attribute name
This change is needed (next to making it consistent as its the only attribute name that uses camelcase) because passing that attribute objectName in the update `terraform_remote_state` resource, ended up giving me this error: https://github.com/hashicorp/terraform/blob/master/helper/schema/schema.go#L736
2018-08-29 20:22:04 +02:00
Sander van Harmelen 7049d973a9 Do not use the TFE_TOKEN env variable
Instead promote the use of shared credentials using the CLI Config File
2018-08-10 20:20:16 +02:00
Sander van Harmelen 83d5f4147b backend/remote: use schema max/min items options 2018-08-05 15:30:27 +02:00
Sander van Harmelen 7fb2d1b8de Implement the Enterprise enhanced remote backend 2018-08-03 22:22:55 +02:00
Sander van Harmelen 495d1ea350 Use New() instead of `once.Do(b.init)` 2018-08-03 11:29:11 +02:00
Sander van Harmelen 97d1c46602 Update the backend import names
It’s a purely cosmetic change, but I find it easier to read them like this.
2018-08-03 11:29:11 +02:00
Kristin Laemmert 85be12d783
cli: show workspace name in destroy confirmation (#18253)
* cli: show workspace name in destroy confirmation

If the workspace name is not "default", include it in the confirmation
message for `terraform destroy`.

Fixes #15480
2018-06-19 13:35:28 -07:00
stack72 df9b2913ae backend/manta: Add support for manta backend tiered path structure
This was already added to triton-go and is now making its way to
the manta backend

```
% acctests backend/remote-state/manta
=== RUN   TestBackend_impl
--- PASS: TestBackend_impl (0.00s)
=== RUN   TestBackend
--- PASS: TestBackend (27.36s)
=== RUN   TestBackendLocked
--- PASS: TestBackendLocked (16.24s)
=== RUN   TestRemoteClient_impl
--- PASS: TestRemoteClient_impl (0.00s)
=== RUN   TestRemoteClient
--- PASS: TestRemoteClient (3.40s)
=== RUN   TestRemoteClientLocks
--- PASS: TestRemoteClientLocks (7.17s)
PASS
ok  	github.com/hashicorp/terraform/backend/remote-state/manta
```
2018-04-03 17:32:10 +03:00
James Bardin 28c46d1a90 cleanup temp files from backend tests 2018-03-28 11:00:23 -04:00
stack72 a6f76aa0af backend/manta: Manta Backend was not dealing with a ResourceNotFound
Fixes: #17314

We now deal correctly with the creation of the state file - we were
not dealing well with a ResourceNotFound error

Now that this has been changed around, we try and create the statefile
and if there is an error, we look for an existing statefile - previously
this was not the order of operations
2018-03-12 17:27:57 +02:00
Martin Atkins 6aefa5835c Merge #17218: Add -auto-approve to "terraform destroy" for consistency 2018-03-08 17:42:15 -08:00
Paul Tyng c868092d2d
Standardize http.Client creation with User-Agent 2018-02-28 12:09:50 -05:00
James Bardin e9a76808df create clistate.Locker interface
Simplify the use of clistate.Lock by creating a clistate.Locker
instance, which stores the context of locking a state, to allow unlock
to be called without knowledge of how the state was locked.

This alows the backend code to bring the needed UI methods to the point
where the state is locked, and still unlock the state from an outer
scope.

Provide a NoopLocker as well, so that callers can always call Unlock
without verifying the status of the lock.

Add the StateLocker field to the backend.Operation, so that the state
lock can be carried between the different function scopes of the backend
code. This will allow the backend context to lock the state before it's
read, while allowing the different operations to unlock the state when
they complete.
2018-02-23 16:48:15 -05:00
James Bardin 8242c773b8 missed the local state backend 2018-02-20 22:09:54 -05:00
James Bardin 57cbcbe99d update remote state tests to use new test helpers 2018-02-20 21:05:58 -05:00
James Bardin 112c37458f report the proper lock id from a state lock error
Fix the now failing state unlock test by reporting the correct ID.

The ID used by GCS is the generation number of the info object, which
isn't known until the info is already written out. While we can't get
the correct ID from the info data for the error rmessage, we can update
it with the generation number after it's read.
2018-02-20 20:54:01 -05:00
James Bardin 1fee5ae172 update gcs to use the new backend tests 2018-02-20 20:36:08 -05:00
James Bardin 85d6b1d9cc add test for lock error and force-unlock
This adds a general test to verify that a remote state backend returns
the expected error type when it cannot lock a state. It then extracts
the ID reported in the error, and attempts to unlock the state using
that ID, which simulated the force-unlock scenario. This is a separate
test, since not all backends have persistent locks that can be unlocked
later.

We also split out the backend test to be called individually as needed.
2018-02-20 20:32:07 -05:00
James Bardin ef8ed1e275 coalesce the backened interrupt code
Moves the nested select statements for backend operations into a single
function. The only difference in this part was that apply called
PersistState, which should be harmless regardless of the type of
operation being run.
2018-02-12 11:56:54 -05:00
James Bardin 7cba68326a always wait for a RunningOperation to return
If the user wishes to interrupt the running operation, only the first
interrupt was communicated to the operation by canceling the provided
context. A second interrupt would start the shutdown process, but not
communicate this to the running operation. This order of event could
cause partial writes of state.

What would happen is that once the command returns, the plugin system
would stop the provider processes. Once the provider processes dies, all
pending Eval operations would return return with an error, and quickly
cause the operation to complete. Since the backend code didn't know that
the process was shutting down imminently, it would continue by
attempting to write out the last known state. Under the right
conditions, the process would exit part way through the writing of the
state file.

Add Stop and Cancel CancelFuncs to the RunningOperation, to allow it to
easily differentiate between the two signals. The backend will then be
able to detect a shutdown and abort more gracefully.

In order to ensure that the backend is not in the process of writing the
state out, the command will always attempt to wait for the process to
complete after cancellation.
2018-02-12 11:56:03 -05:00
Laura Martin 6e1e614a56 Change -force to -auto-approve when destroying
Since an early version of Terraform, the `destroy` command has always
had the `-force` flag to allow an auto approval of the interactive
prompt. 0.11 introduced `-auto-approve` as default to `false` when using
the `apply` command.

The `-auto-approve` flag was introduced to reduce ambiguity of it's
function, but the `-force` flag was never updated for a destroy.

People often use wrappers when automating commands in Terraform, and the
inconsistency between `apply` and `destroy` means that additional logic
must be added to the wrappers to do similar functions. Both commands are
more or less able to run with similar syntax, and also heavily share
their code.

This commit updates the command in `destroy` to use the `-auto-approve` flag
making working with the Terraform CLI a more consistent experience.

We leave in `-force` in `destroy` for the time-being and flag it as
deprecated to ensure a safe switchover period.
2018-02-01 00:14:42 +00:00
James Bardin a5ed7d0ae4
Merge pull request #17086 from hashicorp/jbardin/GH-17083
only split the prefix string once from the s3 key
2018-01-17 18:51:43 -05:00
stack72 e9476c6765 backend/manta: Update manta dependencies
Internally, triton-go has changed how it handles errors. We can now get rid of
checking strings for errors, and we have introduced an errors library that
wraps some of the major errors we encounter and test for
2018-01-17 11:34:02 -08:00
Masayuki Morita f440dba137 Standardize on log level "WARN" rather than "WARNING" 2018-01-16 18:05:26 -08:00
James Bardin 43d000625d only split the prefix string once from the s3 key
Ensure that the prefix is only split off a single time when the
workspace_key_prefix is a substring of the workspace or key name.
2018-01-11 10:08:02 -05:00
Stefan Schmidt c200c170ad Handle refresh errors. 2018-01-10 16:40:20 +01:00
Paddy e4cdbd6c9f
Merge pull request #16936 from negz/gcskeys
Support 'customer supplied encryption keys' in the GCS backend
2018-01-09 01:17:35 -08:00
James Bardin 7c703f2ab2
Merge pull request #16932 from rv-jmaggio/master
Fixing issues with workspace_key_prefix for s3 backend
2018-01-05 13:24:40 -05:00
James Bardin dd5882e5b2
Merge pull request #16992 from hashicorp/jbardin/go1.10-prep
go1.10 prep
2018-01-03 15:41:41 -05:00
goraxe fb6b349e58 backend/s3: Allow forcing use of the S3 path-style API
This is useful when the underlying API is an S3-workalike service, rather than official S3.
2018-01-03 12:15:47 -08:00
Paul Stack 191cf283d5 backend/manta: Support Triton RBAC
Triton Manta allows an account other than the main triton account to be used via RBAC.

Here we expose the SDC_USER / TRITON_USER options to the backend so that a user can be specified.
2018-01-03 12:12:46 -08:00
James Bardin 4b49a323c3 go fmt
slight change to go fmt coming in 0.10
2017-12-26 13:26:38 -05:00
rv-jmaggio b313ce80c4 Changing prefix for empty workspace prefix 2017-12-19 13:14:31 -05:00
Henry Bell 56357c0ab7 Fix typo ('depracted' -> 'deprecated') 2017-12-19 16:43:06 +00:00
rv-jmaggio bef64cfe91 Fixing implementation for empty string and making acceptance test work 2017-12-19 09:31:53 -05:00
rv-jmaggio b02a1c8a46 clarifying tests and using SplitN in implementation 2017-12-18 16:24:34 -05:00
Nic Cope 011841124b Support 'customer supplied encryption keys' in the GCS backend
https://cloud.google.com/storage/docs/encryption#customer-supplied

GCS state created using customer supplied encryption keys can only be read or
modified using the same key.
2017-12-17 19:27:52 -08:00
rv-jmaggio 7f8d686074 refactor and add a test 2017-12-15 21:04:15 -05:00
rv-jmaggio 0889c118a8 Fixing issues with workspace_key_prefix 2017-12-15 17:50:36 -05:00
Rob Campbell 5daeee5f6d Update various files for new version of "stringer"
The latest version of stringer now uses strconv instead of fmt.
2017-12-11 13:26:29 -08:00
James Bardin 52eced589f accept a path or contents for credentials
Match the operation of the google provider, by accepting either a file
path or contents for both `credentials` and `GOOGLE_CREDENTIALS`
2017-12-06 18:33:59 -05:00
James Bardin 2932203492 verify that a state can be read even when locked
This should only happen when a state is loaded via the backend, as well
as a remote state.
2017-12-06 18:33:59 -05:00
James Bardin 9dea2f78d4 create unique buckets for each test, and clean up
This creates a unique bucket name for each test, so that the tests in
parallel don't collide, and buckets left over from interrupted tests
don't cause future failures.

Also make sure that buckets are removed, regardless of content.
2017-12-06 18:33:59 -05:00
James Bardin aec45e6967 backends should never create unmanaged resources
The backend was creating bucket named in the configuration if it didn't
exist. We don't allow other backends to do this, because these are not
managed resources that terraform can control.
2017-12-06 18:32:41 -05:00
Kaveh Mousavi Zamani 7507e3cd21 backend/gcs: fix locking issue when used with terraform_remote_state
Previously there was a problem with double-locking when using the GCS backend with the terraform_remote_state data source.

Here we adjust the locking methodology to avoid that problem.
2017-12-06 09:36:16 -08:00
James Bardin 12b7dac124
Merge pull request #16833 from hashicorp/jbardin/plan-shutdown
Fully enable shutdown for plan and refresh in the local backend
2017-12-05 16:48:34 -05:00
Sander van Harmelen aaedf255c3 backend/s3: allow skipping the region check (#16757)
Without the possibility to skip this check, it’s not possible to use a custom region with a third-party service that mimicks the S3 API.
2017-12-05 11:19:36 -08:00
James Bardin 85295e5c23 watch for cancellation in plan and refresh
Cancellation in the local backend was only implemented for apply.
2017-12-05 10:17:20 -05:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
James Bardin 00b7715710
Merge pull request #16586 from hashicorp/jbardin/providers
Store resolved providers in state
2017-11-08 14:27:48 -05:00
Nic Cope 27ba7de77d backend/gcs: automatically add trailing slash to prefix
The backend doesn't function correctly if a trailing slash is not present, due to how workspaces are enumerated.
2017-11-08 11:24:36 -08:00
James Bardin 09180a10ff cannot destroy without a config 2017-11-07 21:23:37 -05:00
James Bardin d62e9217ae update test state strings for backend/local 2017-11-07 21:23:10 -05:00
Malik Bougacha ed0d57d1b7 Get the enviroment from the environment variable (#16326)
* Get the enviroment from the enviroment variable

* typo
2017-11-06 16:59:48 +00:00
stack72 1fd0f803e4 Migrate Manta Remote state to be a backend
This PR changes manta from being a legacy remote state client to a new backend type. This also includes creating a simple lock within manta

This PR also unifies the way the triton client is configured (the schema) and also uses the same env vars to set the backend up

It is important to note that if the remote state path does not exist, then the backend will create that path. This means the user doesn't need to fall into a chicken and egg situation of creating the directory in advance before interacting with it
2017-10-30 18:36:50 +02:00
James Bardin cf54ca3b0f update tests for new consul packages
Reuse the running consul server for all tests.

Update the lostLockConnection package, since the api client should no
longer lose a lock immediately on network errors.
2017-10-29 12:40:44 -04:00
James Bardin 3a03d3683e update consul api packages
This is from a commit just after the v1.0.0 release, because it removes
the Porter service dependency for tests. The client api package was not
changed.
2017-10-28 19:44:10 -04:00
Martin Atkins 671aace8ec backend/local: disable local backup of remote state
Previously we forced all remote state backends to be wrapped in a
BackupState wrapper that generates a local "terraform.tfstate.backup"
file before updating the remote state.

This backup mechanism was motivated by allowing users to recover a
previous state if user error caused an undesirable change such as loss
of the record of one or more resources. However, it also has the downside
of flushing a possibly-sensitive state to local disk in a location where
users may not realize its purpose and accidentally check it into version
control. Those using remote state would generally prefer that state never
be flushed to local disk at all.

The use-case of recovering older states can be dealt with for remote
backends by selecting a backend that has preservation of older versions
as a first-class feature, such as S3 versioning or Terraform Enterprise's
first-class historical state versioning mechanism.

There remains still one case where state can be flushed to local disk: if
a write to the remote backend fails during "terraform apply" then we will
still create the "errored.tfstate" file to allow the user to recover. This
seems like a reasonable compromise because this is done only in an
_exceptional_ case, and the console output makes it very clear that this
file has been created.

Fixes #15339.
2017-10-27 17:06:33 -07:00
James Bardin 55089e472d
Merge pull request #16484 from hashicorp/f-gcloud-backend
Convert gcloud backend
2017-10-27 17:23:22 -04:00
Florian Forster afa13a3d8e backend/remote-state/gcs: Move toBucketName to the tests. 2017-10-27 16:52:21 -04:00
Florian Forster 3023ae2813 backend/remote-state/gcs: Include project ID in bucket names when testing.
Since bucket names must be *globally* unique. By including the project
ID in the bucket name we ensure that people don't step on each other's
feet when testing.
2017-10-27 16:52:21 -04:00
Florian Forster dcb84ee2c2 backend/remote-state/gcs: Sanitize bucket names. 2017-10-27 16:52:21 -04:00
Florian Forster 454d6bbe2a backend/remote-state/gcs: Delete test buckets after tests complete.
This way tests clean up after themselves and don't leak buckets.
2017-10-27 16:52:21 -04:00
Florian Forster a6669c9e99 backend/remote-state/gcs: Don't enable versioning on new buckets.
Enabling versioning without setting up lifecycle management leads to
every lock file being archived, slowly accruing useless data.
2017-10-27 16:52:21 -04:00
Florian Forster a84823f597 backend/remote-state/gcs: Require TF_ACC for tests using the network. 2017-10-27 16:52:21 -04:00
Florian Forster 72ccf22a92 backend/remote-state/gcs: Implement additional tests.
This calls backend.TestBackend() and remote.TestRemoteLocks() for
standardized acceptance tests. It removes custom listing tests since
those are performed by backend.TestBackend(), too.

Since each tests uses its own bucket, all tests can be run in parallel.
2017-10-27 16:52:21 -04:00
Florian Forster 927085289d backend/remote-state/gcs: Implement the "region" config option.
This allows to select the region in which a bucket is created.
This copies behavior from the Google Cloud provider.
2017-10-27 16:52:21 -04:00
Florian Forster 9583d0945c backend/remote-state/gcs: Add support for the GOOGLE_PROJECT environment variable.
This copies behavior from the Google Cloud provider.
2017-10-27 16:52:21 -04:00
Florian Forster 37dc95158d backend/remote-state/gcs: Enable versioning on automatically created buckets. 2017-10-27 16:52:21 -04:00
Florian Forster df386d3133 backend/remote-state/gcs: Automatically create the bucket if needed.
This resurrects the previously documented but unused "project" option.
This option is required to create buckets (so they are associated with the
right cloud project) but not to access the buckets later on (because their
names are globally unique).
2017-10-27 16:52:21 -04:00
Florian Forster 14263223e7 backend/remote-state/gcs: Simplify initialization of the GCS client.
This also implements the (already documented) behavior of checking the
GOOGLE_CREDENTIALS environment variable.
2017-10-27 16:52:21 -04:00
Florian Forster 816c98f387 backend/remote-state/gcs: Read credentials with ioutil.ReadFile().
We never expect the raw JSON to appear in the config, so pathorcontents is
not the right package here.
2017-10-27 16:52:21 -04:00
Florian Forster c00e929ee5 backend/remote-state/gcs: Mark the "path" option as deprecated. 2017-10-27 16:52:21 -04:00
Florian Forster 91b1a39a40 backend/remote-state/gcs: Implement an end-to-end test.
The code is loosely based on state/remote/gcs_test.go. If the
GOOGLE_PROJECT environment variable is set, this test will

1) create a new bucket; error out if the bucket already exists.
2) create a new state
3) list states and ensure that the newly created state is listed
4) ensure that an object with the expected name exists
5) rum "state/remote".TestClient()
6) delete the state

The bucket is deleted when the test exits, though this may fail if the
bucket it not empty.
2017-10-27 16:51:21 -04:00
Florian Forster 14bfbf0617 backend/remote-state/gcs: Document the "prefix" option.
"state_dir" has been renamed to "prefix" to better fix the GCS
terminology.
2017-10-27 16:51:21 -04:00
Florian Forster c054bd0939 backend/remote-state/gcs: Rename "gcloud" to "gcs" for backwards compatibility. 2017-10-27 16:51:21 -04:00
Florian Forster 93a55f15e9 backend/remote-state/gcloud: Add test for Backend.{state,lock}File(). 2017-10-27 16:51:21 -04:00
Florian Forster 5a4e2076e9 backend/remote-state/gcloud: Add the "path" config option.
This config option was used by the legacy "gcs" client. If set, we're
using it for the default state -- all other states still use the
"state_dir" setting.
2017-10-27 16:51:21 -04:00
Florian Forster f80b872bc3 backend/remote-state/gcloud: Unify on the "context" package.
We don't need to use the legacy package here.
2017-10-27 16:51:21 -04:00
Florian Forster 5d4e25ada4 backend/remote-state/gcloud: Make gcsBackend private.
This class is only used via the "backend".Backend interface, so there is
no need to export this type beyond the gcloud package.
2017-10-27 16:51:21 -04:00
Florian Forster 9ec39573ee backend/remote-state/gcloud: Make remoteClient private.
This class is only used via the "state/remote".State interface, so there
is no need to export this type beyond the gcloud package.
2017-10-27 16:51:21 -04:00
Florian Forster 42e8441a2b backend/remote-state/gcloud: Refactor Backend.State().
Fixes:
* https://github.com/golang/go/wiki/CodeReviewComments#doc-comments
* https://github.com/golang/go/wiki/CodeReviewComments#error-strings
* https://github.com/golang/go/wiki/CodeReviewComments#initialisms
2017-10-27 16:51:21 -04:00
Florian Forster 5cb574035a backend/remote-state/gcloud: Refactor Backend.remoteClient().
This replaces stateFileName() and lockFileName() with path.Join().

Fixes:
* https://github.com/golang/go/wiki/CodeReviewComments#doc-comments
* https://github.com/golang/go/wiki/CodeReviewComments#doc-comments
2017-10-27 16:51:21 -04:00
Florian Forster 9ae45e320f backend/remote-state/gcloud: Refactor Backend.DeleteState().
Fixes:
* https://github.com/golang/go/wiki/CodeReviewComments#doc-comments
* https://github.com/golang/go/wiki/CodeReviewComments#error-strings
2017-10-27 16:51:21 -04:00
Florian Forster fabba5c0c8 backend/remote-state/gcloud: Refactor Backend.States().
The previous code listed all objects in the bucket and used local filtering
(using regular expressions) to find .tfstate objects. This new code sets
the delimiter to "/", which causes GCS to only return objects directly in
the given prefix, but not any sub"directories".

Fixes:
* https://github.com/golang/go/wiki/CodeReviewComments#doc-comments
* https://github.com/golang/go/wiki/CodeReviewComments#error-strings
2017-10-27 16:51:21 -04:00
Florian Forster 97e1aa7ce9 backend/remote-state/gcloud: Use the lock file's generation as lock ID.
This allows Unlock() to call Delete() without reading the lock file's
content first.
2017-10-27 16:51:21 -04:00
Florian Forster edf2096e28 backend/remote-state/gcloud: Use the context provided to configure().
Calling context.Background() from outside the main() function is
discouraged. The configure functions are only called from
"…/helper/schema".Backend.Configure which provides the Background context,
i.e. a long-living context we can use for backend communication.
2017-10-27 16:51:21 -04:00
Florian Forster 2e5fca78c2 backend/remote-state/gcloud: Coding style changes.
Refactorings to make the code more idiomatic.
2017-10-27 16:51:21 -04:00
Florian Forster f33005faba backend/remote-state/gcloud: Handle errors returned by Write(), too.
Many GCS errors are returned by Close(), but not all.
2017-10-27 16:51:21 -04:00
Florian Forster 1ee194986d backend/remote-state/gcloud: Add the RemoteClient.{state,lock}File() methods. 2017-10-27 16:51:20 -04:00
Florian Forster e54d36b489 backend/remote-state/gcloud: Sort standard library imports before other imports. 2017-10-27 16:51:20 -04:00
Florian Forster 52ac764036 backend/remote-state/gcloud: Rename Url -> URL 2017-10-27 16:51:20 -04:00
Florian Forster 5313e15e07 backend/remote-state/gcloud: Move the definition of the Backend struct. 2017-10-27 16:51:20 -04:00
Florian Forster def3279ec7 backend/remote-state/gcloud: Use package provided OAuth scope. 2017-10-27 16:51:20 -04:00
Piotrek Bzdyl 5854373018 Implemented GCloud backend supporting remote locking and multiple workspaces. 2017-10-27 16:51:20 -04:00
James Bardin b040cd0837 dynamoDB reads are not fully consisten by default
Use fully consistent reads for backend operations.
2017-10-27 16:43:30 -04:00
James Bardin 31912956ce Merge pull request #16290 from hashicorp/jbardin/s3-errors
retry on s3 state upload errors
2017-10-17 16:04:14 -04:00
James Bardin 611b1ced59 retry on s3 state upload errors
While #16243 added the ability to retry getting a state from S3, Put can
return the same InternalError status. Use the same retry logic when
uploading state to S3.
2017-10-08 16:46:33 -04:00
James Bardin f5e9a20c66 reset testLockHook 2017-10-08 16:24:45 -04:00
James Bardin 25a8227291 add broken test for lock lost on connection error
Add a way to inject network errors by setting an immediate deadline on
open consul connections. The consul client currently doesn't retry on
some errors, and will force us to lose our lock.

Once the consul api client is fixed, this test will fail.
2017-10-08 16:16:57 -04:00
James Bardin fd9adcdb36 only init one consul client, and lower keepalive
The consul Client is analogous to an http.Client, and we really don't
need more than 1. Configure a single client and store it in the backend.

Replace the default Transport's Dialer to reduce the KeepAlive setting
from 30s to 17s. This avoids racing with the common network timeout
value of 30s, and is also coprime to other common intervals.
2017-10-08 11:51:35 -04:00
James Bardin d0ecb232ae record consul session ID in lock info
This can help correlate TF and consul logs
2017-10-08 11:24:43 -04:00
James Bardin b20ab9f0bb Merge pull request #16243 from hashicorp/jbardin/s3-errors
retry on s3 backend internal errors
2017-10-04 09:26:26 -04:00
bclodius fba02f0bea retry on s3 backend internal errors
Internal errors from S3 are usually transient, and can be immediately retried.
Make 2 attempts at retreiving the state object before returning an error.
2017-10-04 09:11:30 -04:00
James Bardin 91442b7146 Merge pull request #15680 from brunomcustodio/etcdv3-backend
[WIP] etcd v3 backend with lock support.
2017-10-03 14:15:58 -04:00
James Bardin d477d1f6d4 Merge pull request #15553 from bonifaido/custom_s3_backend
Allow non-AWS S3 backends
2017-10-02 19:39:48 -04:00
James Bardin e0ee1e4d6e Merge pull request #16070 from octo/defaultdatadir
backend/local: Remove unused const DefaultDataDir.
2017-10-02 16:27:23 -04:00
Martin Atkins 0fe43c8977 cli: allow disabling "next steps" message in terraform plan
In #15884 we adjusted the plan output to give an explicit command to run
to apply a plan, whereas before this command was just alluded to in the
prose.

Since releasing that, we've got good feedback that it's confusing to
include such instructions when Terraform is running in a workflow
automation tool, because such tools usually abstract away exactly what
commands are run and require users to take different actions to
proceed through the workflow.

To accommodate such environments while retaining helpful messages for
normal CLI usage, here we introduce a new environment variable
TF_IN_AUTOMATION which, when set to a non-empty value, is a hint to
Terraform that it isn't being run in an interactive command shell and
it should thus tone down the "next steps" messaging.

The documentation for this setting is included as part of the "...in
automation" guide since it's not generally useful in other cases. We also
intentionally disclaim comprehensive support for this since we want to
avoid creating an extreme number of "if running in automation..."
codepaths that would increase the testing matrix and hurt maintainability.

The focus is specifically on the output of the three commands we give in
the automation guide, which at present means the following two situations:

* "terraform init" does not include the final paragraphs that suggest
  running "terraform plan" and tell you in what situations you might need
  to re-run "terraform init".
* "terraform plan" does not include the final paragraphs that either
  warn about not specifying "-out=..." or instruct to run
  "terraform apply" with the generated plan file.
2017-09-14 10:51:41 -07:00
Florian Forster ce85600cd4 backend/local: Remove unused const DefaultDataDir.
Not to be confused with the const of the same name in the "command"
package.
2017-09-12 08:20:12 +02:00
Bruno Miguel Custodio 524c3934c6
Add TLS support. 2017-09-09 08:03:59 +01:00
Bruno Miguel Custodio c8ff10f603
Reuse 'etcd' client. 2017-09-09 00:21:23 +01:00
Bruno Miguel Custodio 6daf1d9d84
Allow for username and password to be read from envvars. 2017-09-08 23:41:27 +01:00
Bruno Miguel Custodio 8f7b315037
Add documentation. 2017-09-08 23:40:05 +01:00
Bruno Miguel Custodio 54dc50ec5c
Cleanup 'etcd' after every test. 2017-09-08 22:49:23 +01:00
Bruno Miguel Custodio 3c21b9c56f
Get rid of 'keyEnvPrefix'. 2017-09-08 22:34:15 +01:00
Bruno Miguel Custodio 6570948bea
Fixing the test suite. 2017-09-08 17:46:39 +01:00
Bruno Miguel Custodio 038f5eb638
Remove 'lockSuffix' as not needed in 'etcd' v3. 2017-09-08 16:59:39 +01:00
Bruno Miguel Custodio bb4dec6032
Make sure we don't relock. 2017-09-08 16:22:14 +01:00
Bruno Miguel Custodio b8f4f6d3e2
Delete lock info when unlocking. 2017-09-08 15:21:06 +01:00
Bruno Miguel Custodio b896348230
Sort the result of 'Backend.States()'. 2017-09-08 12:25:15 +01:00
Bruno Miguel Custodio 70aad79b6e
Make 'endpoints' a schema.TypeList. 2017-09-08 12:16:00 +01:00
Bruno Miguel Custodio fee5cabeda
Make 'gofmt' happy. 2017-09-08 10:22:58 +01:00
Bruno Miguel Custodio 52c97e9fc9
First steps towards an 'etcdv3' backend. 2017-09-08 09:43:33 +01:00
tombuildsstuff d074b0da29 Obtaining the current metadata before setting it 2017-09-06 12:41:05 -07:00
tombuildsstuff a10d23dd95 Removing dead code 2017-09-06 12:41:05 -07:00
Minkyu Kim f12a237747 Fix crash when using consul backend 2017-09-02 20:05:30 +09:00
Martin Atkins 83414beb8f command: various adjustments to the diff presentation
The previous diff presentation was rather "wordy", and not very friendly
to those who can't see color either because they have color-blindness or
because they don't have a color-supporting terminal.

This new presentation uses the actual symbols used in the plan output
and tries to be more concise. It also uses some framing characters to
try to separate the different stages of "terraform plan" to make it
easier to visually navigate.

The apply command also adopts this new plan presentation, in preparation
for "terraform apply" (with interactive plan confirmation) becoming the
primary, safe workflow in the next major release.

Finally, we standardize on the terminology "perform" and "actions" rather
than "execute" and "changes" to reflect the fact that reading is now an
action and that isn't actually a _change_.
2017-09-01 17:55:05 -07:00
Martin Atkins 3ea159297c command/format: improve consistency of plan results
Previously the rendered plan output was constructed directly from the
core plan and then annotated with counts derived from the count hook.
At various places we applied little adjustments to deal with the fact that
the user-facing diff model is not identical to the internal diff model,
including the special handling of data source reads and destroys. Since
this logic was just muddled into the rendering code, it behaved
inconsistently with the tally of adds, updates and deletes.

This change reworks the plan formatter so that it happens in two stages:
- First, we produce a specialized Plan object that is tailored for use
  in the UI. This applies all the relevant logic to transform the
  physical model into the user model.
- Second, we do a straightforward visual rendering of the display-oriented
  plan object.

For the moment this is slightly overkill since there's only one rendering
path, but it does give us the benefit of letting the counts be derived
from the same data as the full detailed diff, ensuring that they'll stay
consistent.

Later we may choose to have other UIs for plans, such as a
machine-readable output intended to drive a web UI. In that case, we'd
want the web UI to consume a serialization of the _display-oriented_ plan
so that it doesn't need to re-implement all of these UI special cases.

This introduces to core a new diff action type for "refresh". Currently
this is used _only_ in the UI layer, to represent data source reads.
Later it would be good to use this type for the core diff as well, to
improve consistency, but that is left for another day to keep this change
focused on the UI.
2017-09-01 17:55:05 -07:00
James Bardin 76eb65c911 Fix crash in consul backend
A TLS config was being assigned to a Transport in a nil http.Client. The
Transport is built in the consul config by default, but the http.Client
is not built until later in NewClient.
2017-08-31 09:29:43 -04:00
Martin Atkins c12d64f340 Use t.Helper() in our test helpers
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.

This covers many -- but probaly not all -- of our test helpers across
various packages.
2017-08-28 09:59:30 -07:00
Loïc Albertin 21b287e919 backend/consul: allow specifying various Consul client security options
These new options are equivalent to the consul CLI arguments -ca-file, -client-cert and -client-key.
2017-08-23 09:47:21 -07:00
James Bardin 5d50e764ea deprecate the "azure" backend name
Alert users to use the current "azurerm" naming.
2017-08-17 12:49:23 -04:00
James Bardin 0e873d70ed Merge pull request #15840 from hashicorp/jbardin/deprecate-backend
add the ability to mark a backend as deprecated
2017-08-17 12:39:05 -04:00
James Bardin cc454fcc56 add the ability to mark a backend as deprecated
This is a simple wrapper to allow a backend to be marked as deprecated
during initialization.
2017-08-17 12:29:19 -04:00
Peter McAtominey b74b26bcfa backend: alias azurerm to azure 2017-08-17 16:37:51 +01:00
Peter McAtominey 6b9b5ec297 backend/azure: update to latest SDK 2017-08-17 16:32:18 +01:00
Peter McAtominey f9e8e54835 backend: convert Azure remote state to a backend
Added locking support via blob leasing (requires that an empty state is
created before any lock can be acquired.

Added support for "environments" in much the same way as the S3 backend.
2017-08-17 16:32:17 +01:00
James Bardin 2bb5007690 Merge pull request #15683 from hashicorp/jbardin/remote-state-lineage
Remove strict lineage check in remote.State
2017-08-09 17:49:19 -04:00
James Bardin 4c7cd549cd don't allow leading slashes in s3 remote state key
S3 accepts objects with a leading slash and strips them off. This works
fine except in our workspace hierarchy, which then can no longer find
suffixes matching the full key name.
2017-08-08 09:47:44 -04:00
James Bardin dc30761497 Merge pull request #15567 from hashicorp/jbardin/consul-update
Update consul vendor and remote state
2017-08-03 13:20:31 -04:00
James Bardin c3e943bed2 add another failing test for remote.State lineage
Want to make sure we don't hit this again.
2017-08-01 19:34:21 -04:00
James Bardin 18d71f273e make inmem behave more like remote backends
When remote backend imeplemtations create a new named state, they may
need to acquire a lock and/or save an actual empty state to the backend.
Copy this behavior in the inmem backend for testing.
2017-08-01 19:34:21 -04:00
James Bardin 5deef9621d export Reset()
This package is used for testing, so there needs to be an easy method
for reinitializing the stored data between tests.
2017-08-01 19:34:21 -04:00
James Bardin ac60ddcd40 Support named states in inmeme backend
Used to expand test coverage
2017-08-01 19:34:21 -04:00
James Bardin 1070d04b07 Remove consul context adapter for RenewPeriodic
Updated the vendored consul which no longer requires the channel adapter
to convert a `chan stuct{}` to a `<-chan struct{}`.

Call testutil.NewTestServerConfigT with the new signature.
2017-07-20 11:09:45 -04:00
James Bardin 9a7ffbfb1b Merge pull request #15566 from hashicorp/jbardin/state-serial
Add warning to mismatched plan state
2017-07-18 18:09:00 -04:00
James Bardin dd9a54c661 Merge pull request #15435 from hashicorp/jbardin/state-path-message
only show state path help if state is local
2017-07-18 18:08:18 -04:00
James Bardin 50f412bff4 Merge pull request #15557 from hashicorp/jbardin/consul-lock-sessions
have the consul client manage the lock session
2017-07-17 10:56:30 -04:00
James Bardin a1727ec4c2 Add warning to mismatched plan state
Forward-port the plan state check from the 0.9 series.
0.10 has improved the serial handling for the state, so this adds
relevant comments and some more test coverage for the case of an
incrementing serial during apply.
2017-07-17 10:41:29 -04:00
James Bardin bcb11f6d89 have the consul client manage the lock session
When a consul lock is lost, there is a possibility that the associated
session is still active. Most commonly, the long request to watch the
lock key may error out, while the session is continually refreshed at a
rate of TTL/2.

First have the lock monitor retry the lock internally for at least 10
seconds (5 attempts with the default 2 second wait time). In most cases
this will reconnect on the first try, keeping the lock channel open.

If the consul lock can't recover itself, then cancel the session as soon
as possible (terminating the PreiodicRenew will call Session.Destroy),
and start over. In the worse case, the consul agents were split, and the
session still exists on the leader so we may need to wait for the old
session TTL, plus the LockWait time to renew the lock.

We use a Context for the cancellation channels here, because that
removes the need to worry about double-closes and nil channels. It
requires an awkward adapter goroutine for now to convert the Done()
`<-chan` to a `chan` for PeriodicRenew, but makes the rest of the code
safer in the long run.
2017-07-14 14:42:42 -04:00
James Bardin 193d4b868c backend state tests must honor lineage
Remote state implementations may initialize a lineage when creating a
new named state (i.e. "workspace"). The tests were ignoring that initial
lineage to write a new state to the backend.
2017-07-14 13:50:26 -04:00
Nandor Kracser f6c77339f2 Allow non-AWS S3 backends
This commit makes sts, metadata and other AWS related API calls optional, thus the backend initialization will not send non-AWS API tokens to AWS APIs
2017-07-14 13:08:47 +02:00
James Bardin 3fb76f3ebb only show state path help if state is local 2017-06-29 15:30:44 -04:00
James Bardin 5066fa2151 Merge pull request #13871 from fatmcgav/swift_move_to_backend
Move Swift remote state to backend
2017-06-27 16:23:46 -04:00
Martin Atkins 6afa72f6ca command: minor adjustments to the -auto-approve=false UX
Some tweaks to the messaging and presentation.
2017-06-27 11:22:35 -07:00
David Glasser 14af879fe0 command: also print plan for destroy 2017-06-27 11:22:31 -07:00
David Glasser 039d36bf91 command: add "apply -auto-approve=false" flag
A common reason to want to use `terraform plan` is to have a chance to
review and confirm a plan before running it.  If in fact that is the
only reason you are running plan, this new `terraform apply -auto-approve=false`
flag provides an easier alternative to

    P=$(mktemp -t plan)
    terraform refresh
    terraform plan -refresh=false -out=$P
    terraform apply $P
    rm $P

The flag defaults to true for now, but in a future version of Terraform it will
default to false.
2017-06-27 11:22:26 -07:00
Martin Atkins 45a4ba1ea7 Merge #15344: Avoid double-counting resources to create 2017-06-27 10:48:45 -07:00
James Bardin 42cbb716b7 Merge pull request #15370 from rowleyaj/s3_backend_workspaces
Add configurable workspace prefix for S3 Backend
2017-06-27 11:44:12 -04:00
Alex Rowley 2dd464cf8f
Clarify field description 2017-06-27 16:32:00 +01:00
Chris Marchesi 5654a676d9 core: Skip diff hooks for stubs on eval altogether
Rather than overloading InstanceDiff with a "Stub" attribute that is
going to be largely meaningless, we are just going to skip
pre/post-diff hooks altogether. This is under the notion that we will
eventually not need to "stub" a diff for scale-out, stateless nodes on
refresh at all, so diff behaviour won't be necessary at that point, so
we should not assume that hooks will run at this stage anyway.

Also as part of this removed the CountHook test that is now failing
because CountHook is out of scope of the new behaviour.
2017-06-24 08:01:17 -07:00
Chris Marchesi 50cd33f781 core: Move Refreh/Plan diff count to general operation test
We are changing the behaviour of the "stub" diff operation to just have
the pre/post-diff hooks skipped on eval, meaning that the test against
CountHook will ultimately be meaningless and fail, hence we need a
different test here that tests it on a more general level.
2017-06-24 07:54:40 -07:00
James Bardin 1fa7667ad4 s3 backend should use Client.Delete to DeleteState
The s3.Backend was using it's own code for DeleteState, but the dynamo
entries are only handled through the RemoteClient. Have DeleteState use
a RemoteClient for delete.
2017-06-23 10:19:50 -04:00
James Bardin ebf4413e95 add named named state delete+create-delete test
This ensures that we don't leave any conflicting state artifacts
preventing the recreation of a named state.
2017-06-23 10:16:34 -04:00
James Bardin ac937a890d improve plugin reinit error text 2017-06-22 15:11:37 -04:00
James Bardin 7a955f990c make display plugin checksum error to user
The error follows a generic message, so can be ignored by users who may
not understand the implications.
2017-06-22 13:38:55 -04:00
Alex Rowley db40dc06ab
Add configurable workspace prefix for S3 Backend
Fixes #13184
2017-06-22 18:24:10 +01:00
James Bardin 5be15ed77c have the local backend provide a plugin init msg
During plan and apply, because the provider constraints need to be built
from a plan, they are not checked until the terraform.Context is
created. Since the context is always requested by the backend during the
Operation, the backend needs to be responsible for generating contextual
error messages for the user.

Instead of formatting the ResolveProviders errors during NewContext,
return a special error type, ResourceProviderError to signal that
init will be required. The backend can then extract and format the
errors.
2017-06-22 13:15:30 -04:00
Chris Marchesi f249386c8a core: Test to ensure PostDiff is ignoring stubs
Added a test that shows that PostDiff is ignoring diffs where the Stub
attribute is set.
2017-06-21 09:39:52 -07:00
Chris Marchesi 45528b2217 core: Instance/EvalDiff.Quiet -> Stub
Changed the language of this field to indicate that this diff is not a
"real" diff, in that it should not be acted on, versus a "quiet" mode,
which would indicate just simply to act silently.
2017-06-21 09:15:08 -07:00
Chris Marchesi eef933f2a7 core: Don't count scaled-out resources twice in the UI
This fixes a bug with the new refresh graph behaviour where a resource
was being counted twice in the UI on part of being scaled out:

 * We are no longer transforming refresh nodes without state to
   plannable resources (the transformer will be removed shortly)
 * A Quiet flag has been added to EvalDiff and InstanceDiff - this
   allows for the flagging of a diff that should not be treated as real
   diff for purposes of planning
 * When there is no state for a refresh node now, a new path is taken
   that is similar to plan, but flags Quiet, and does nothing with the
   diff afterwards.

Tests pending - light testing has confirmed this should fix the double
count issue, but we should have some tests to actually confirm the bug.
2017-06-20 07:37:32 -07:00
Gavin Williams c2550e78cf Rename openstack provider for swift remote backend 2017-06-13 22:11:16 +01:00
Gavin Williams ee2e390f85 remote/swift: Migrate Swift remote state to a backend
Move the Swift State from a legacy remote state to an official backend.
Add `container` and `archive_container` configuration variables, and deprecate `path` and `archive_path` variables.

Future improvements: Add support for locking and environments.
2017-06-13 22:04:01 +01:00
James Bardin ac959af6b4 rename aws provider import in s3 backend 2017-06-12 13:43:51 -04:00
Martin Atkins f7ce6a15f8 backend: Operation.Environment renamed to "Workspace"
This is part of an effort to switch this terminology across all of
Terraform.
2017-06-09 16:26:26 -07:00
Martin Atkins 418a8a8bc9 command + backend: rename various API objects to "Workspace" terminology
We're shifting terminology from "environment" to "workspace". This takes
care of some of the main internal API surface that was using the old
terminology, though is not intended to be entirely comprehensive and is
mainly just to minimize the amount of confusion for maintainers as we
continue moving towards eliminating the old terminology.
2017-06-09 16:26:25 -07:00
Martin Atkins 5026e1d313 backend/local: "environment_dir" renamed to "workspace_dir"
As part of switching our terminology, we begin a deprecation cycle for
"environment_dir" and advise users to switch to "workspace_dir" instead.
2017-06-09 15:01:39 -07:00
David Glasser c25d848ffb core: allow overriding environment name via env var
This allows you to run multiple concurrent terraform operations against
different environments from the same source directory.

Fixes #14447.

Also removes some dead code which appears to do the same thing as the function I
modified.
2017-06-09 15:01:39 -07:00
Martin Atkins c835ef8ff3 Update tests for the new ProviderResolver interface
Rather than providing an already-resolved map of plugins to core, we now
provide a "provider resolver" which knows how to resolve a set of provider
dependencies, to be determined later, and produce that map.

This requires the context to be instantiated in a different way, so this
very noisy diff is a mostly-mechanical update of all of the existing
places where contexts get created for testing, using some adapted versions
of the pre-existing utilities for passing in mock providers.
2017-06-09 14:03:59 -07:00
Martin Atkins a42ebe389c Revert "have StateHook periodically PersistState"
This reverts commit b73d037761.

This commit seems to have introduced a race condition where we can
concurrently keep updating state after we've checked if we need to
increase the serial, and thus end up writing partial changes
to the state backend.

In the case of Terraform Enterprise, this fails altogether because
of the state hash consistency check it does.
2017-06-07 16:25:19 -07:00
Radek Simko 1244309579 Fix stringer comments (#15069) 2017-06-05 10:17:35 +01:00
He Guimin 87562be855 provider/alicloud: Add the function of replacing ecs instance's system disk (#15048)
* add replacing system disk function for ecs

* remove ForceNew of system_disk_size
2017-06-05 11:27:49 +03:00
Jake Champlin ac177492fb
core: Revert stringer changes from earlier commits 2017-06-01 11:37:12 -04:00
Thomas Schaaf 79c91e11c8 provider/aws: Add aws elastic beanstalk solution stack (#14944)
* Add aws elastic beanstalk solution stack

Signed-off-by: Thomas Schaaf <thomaschaaf@Thomass-MBP.fritz.box>

* Fix incorrect naming

Signed-off-by: Thomas Schaaf <thomaschaaf@Thomass-MBP.fritz.box>

* Use unique go variable/function names

Signed-off-by: Thomas Schaaf <thomaschaaf@Thomass-MacBook-Pro.local>

* Add docs to sidebar

* Sort provider by alphabet

* Fix indent

* Add required statement

* Fix acceptance test
2017-06-01 02:23:06 +03:00
James Bardin 6b700ff1fb replace lock_table with dynamodb_table in s3 cfg
Since the DynamoDB table used by the S3 backend is no longer only used
for locks, rename it in the config to remove any confusion about it
being lock-specific.
2017-05-30 16:27:04 -04:00
James Bardin ecc5bfb801 Merge pull request #14937 from hashicorp/jbardin/GH-14927
check for nil state in s3 client Get
2017-05-30 15:12:55 -04:00
James Bardin 82eba5801d Test losing and reacquiring a consul lock 2017-05-30 14:38:33 -04:00
James Bardin 3df48bfc27 relock consul when lock is lost
Consul locks are based on liveness, and may be lost due timeouts,
network issued, etc. If the client determines the lock was lost, attempt
to reacquire the lock immediately.

The client was also not using the `lock` config option. Disable locks if
that is not set.
2017-05-30 14:38:33 -04:00
James Bardin e8330b6f53 use CAS for consul state Put 2017-05-30 14:38:32 -04:00
James Bardin e7502454b4 check for nil state in s3 client Get
The S3 client can return (nil, nil) when the remote state doesn't exist.
The caused a nil pointer dereference when checking the payload.MD5
against the expected value.

This can happen if the remote state was manually removed, but the digest
entry was left in the DynamoDB table.
2017-05-30 14:36:13 -04:00
James Bardin f8bfc0a80d check for empty diff in CountHook.PreApply
Make sure we don't try to count anything from a nil diff.
2017-05-26 15:04:56 -04:00
James Bardin 1d585762dd Merge pull request #14834 from hashicorp/jbardin/state-hook
Persist state more frequently
2017-05-25 18:47:12 -04:00
James Bardin 9e4c0ff2ad call PersistState immediately when cancelling
When the backend operation is cancelled, immediately call PersistState.
The is a high likelihood that the user is going to terminate the process
early if the provider doesn't return in a timely manner, so persist as
much state as possible.
2017-05-25 11:20:51 -04:00
James Bardin b73d037761 have StateHook periodically PersistState
Have StateHook periodically call PersistState to flush any cached state
to permanent storage. This uses a minimal 10 second interval between
calls to PersistState.
2017-05-25 11:20:51 -04:00
James Bardin ef1d53934c Merge pull request #14746 from hashicorp/jbardin/s3-consistency
store and verify s3 remote state checksum to avoid consistency issues.
2017-05-24 16:47:57 -04:00
James Bardin 91be40a577 add detailed error message s3 checksum mismatch
Have the s3 RemoteClient return a detailed error message to the user in
the case of a mismatch state checksum.
2017-05-24 13:51:33 -04:00
James Bardin 0022d224e8 store and verify s3 remote state checksum
Updates to objects in S3 are only eventually consistent. If the
RemoteClient has a DynamoDB table available, use that to store a
checksum of the last written state, so the object can be verified by the
next client to call Get.

Terraform currently doesn't have any sort of user feedback around
RefreshState/Get, so we poll only for a short time before returning an
error.
2017-05-24 13:39:30 -04:00
James Bardin e1d9ad40d3 Merge pull request #14680 from hashicorp/jbardin/unlock-s3
Check that a named state in s3 doesn't exist before acquiring a lock
2017-05-24 12:56:00 -04:00
Martin Atkins 9cda37205d backend/local: create local state file if backend write fails
In the old remote state system we had the idea of a local backup, which
is actually still present for the legacy backends but no longer applies
for the new-style backends like the s3 backend.

It's problematic when an apply runs for long enough that someone's
time-limited AWS STS credentials expire and then Terraform fails and can't
persist state to S3.

To reduce the risk of lost state, here we add some extra fallback code
for the local apply operation in particular. If either state writing
or state persisting fail then we attempt to write the state to a special
backup file errored.tfstate, and produce an error message that guides the
user on how to retry uploading this state.

In the unlikely event that we can't write to local disk either (e.g.
permissions problems) we take a last-ditch attempt to dump the JSON onto
stdout and advise the user to manually copy it into a file for import.
If even that doesn't work for some reason, we assume a critical Terraform
bug (JSON-serialization problem with states?) and bail out with an
apologetic error message.

This is implemented for the apply command in particular because this is
the one command where new objects are created in real APIs that we don't
want to lose track of. For other operations it's less bad to just generate
a simple error message and have the user retry.

This fixes #14298.
2017-05-23 11:18:01 -07:00
James Bardin b279b1abb5 check for named s3 states before acquiring a lock
In order to force-unlock a named state, we have to fetch that state
first. Don't attempt to acquire a lock if we know the state already
exists in s3.
2017-05-19 14:40:59 -04:00
James Bardin a2d452acec failing test to force-unlock a named state in s3
The State call attempts to get a lock before determining if a named
state exists. This prevents force-unlock, since we need the state to
call Unlock.
2017-05-19 14:39:42 -04:00
James Bardin 58759b1167 Merge pull request #13941 from hashicorp/jbardin/sigint-message
improve SIGINT output
2017-04-25 20:16:22 -04:00
James Bardin 7dad3f4d48 remove redundant output when interrupting apply
The backend apply operation doesn't need to output the same text as the
cli itself. Instead notify the user that we are in the process of
stopping the operation.
2017-04-25 11:44:51 -04:00
James Bardin 563cfd00df always wrap remote state in a BackupState
Use a local backup for remote state operations. This allows for manual
recovery in the case of a put failure.
2017-04-24 22:15:19 -04:00
Martin Atkins b1763e262a Restore stringer-generated files back to new version
stringer has changed the boilerplate it generates in a recent version.
We'd previously updated to the new format but accientally rolled back
to the old while merging a long-running feature branch.

This restores us back to the new format again.
2017-04-21 14:49:18 -07:00
Jasmin Gacic 61499cfcf0 Provider Oneandone (#13633)
* Terraform Provider 1&1

* Addressing pull request remarks

* Fixed imports

* Fixing remarks

* Test optimiziation
2017-04-21 17:19:10 +03:00
James Bardin 928e60672f context Refresh and Apply sometimes return nil
The documentation for Refresh indicates that it will always return a
valid state, but that wasn't true in the case of a graph builder error.
While this same concept wasn't documented for Apply, it was still
assumed in the terraform apply code.

Since the helper testing framework relies on the absence of a state to
determine if it can call Destroy, the Context can't can't start
returning a state in all cases. Document this, and use the State method
to fetch the correct state value after Apply.

Add a nil check to the WriteState function, so that writing a nil state
is a noop.

Make sure to init before sorting the state, to make sure we're not
attempting to sort nil values. This isn't technically needed with the
current code, but it's just safer in general.
2017-04-14 14:56:10 -04:00
James Bardin 9c431aee1b only list environments when the keyName matches
Prevent extra keys in the s3 envPrefix path from showing up as
listed environments.

Better handle keys containing slashes

Add tests for unexpected keys in s3.
2017-04-12 13:57:22 -04:00
James Bardin 5621d97925 cleanup consul lock entries
This matches the consul cli behavior, where locks are cleaned up after
use.

Return an error from re-locking the state. This isn't required by the
Locker interface, but it's an added sanity check for state operations.
What was incorrect here was returning an empty ID and error, which would
indicate that Lock/Unlock isn't supported.
2017-04-06 14:19:55 -04:00
James Bardin 6e136c848a use the aws provider client initialization
Use the aws provider code to create the clients for the s3 backend, so
that all the behavior matches that of the provider.

Remove the fake creds from the test, as the aws provider will attempt to
validate them.
2017-04-05 12:39:50 -04:00
James Bardin 54aa466b74 initialize the s3 lock path in one place 2017-04-04 14:44:58 -04:00
James Bardin d059939f88 Merge pull request #13262 from hashicorp/jbardin/lock-timeouts
lock timeouts
2017-04-04 14:30:20 -04:00
James Bardin 305ef43aa6 provide contexts to clistate.Lock calls
Add fields required to create an appropriate context for all calls to
clistate.Lock.

Add missing checks for Meta.stateLock, where we would attempt to lock,
even if locking should be skipped.
2017-04-01 17:09:20 -04:00
James Bardin 3f0dcd1308 Have the clistate Lock use LockWithContext
- Have the ui Lock helper use state.LockWithContext.
- Rename the message package to clistate, since that's how it's imported
  everywhere.
- Use a more idiomatic placement of the Context in the LockWithContext
  args.
2017-04-01 17:09:20 -04:00
James Bardin 75458a182d remove extra state.Locker assertions
All states are lockers, so get rid of extra asertions.
2017-04-01 17:01:45 -04:00
Paul Stack 4501be7e5c backend/remote-state: Add support for assume role extensions to s3 backend (#13236)
Fixes: #13234

This now matches the AWS provider for the Assume Role support
2017-03-31 19:59:29 +03:00
Martin Atkins 21cd5595e2 Update stringer-generated files to new boilerplate
golang/tools commit 23ca8a263 changed the format of the leading comment
to comply with some new standards discussed here:
https://golang.org/issue/13560

This is the result of running generate with the latest version of
stringer. Everyone working on Terraform will need to update stringer
after this is merged, to avoid reverting this:
    go get -u golang.org/x/tools/cmd/stringer
2017-03-29 08:07:06 -07:00
James Bardin 7b8e1aff3d fix local backend test
The local backend can't define a StateOut path if we want to test
writing multiple named state files. Use a default local backend.
2017-03-23 11:15:46 -04:00
James Bardin 49b9a6ad92 test for proper state persistence
The backend state tests weren't properly checking for persistence.
Update the test to persist states and fetch them again from the backend,
checking that lineage is preserved.
2017-03-23 10:06:07 -04:00
James Bardin 575e7f1811 Properly create a new named state in s3
If the state doesn't exist, we need to initialize one so that it can be
listed be States.
2017-03-23 10:03:22 -04:00
James Bardin fa4dc01cf4 add named state support to the s3 backend
This adds named state (environment) support to the S3 backend.

A state NAME will prepend the configured s3 key with `env:/NAME/`.
The default state will remain rooted in the bucket for backwards
compatibility.

Locks in DynamoDB use the S3 key as the as the primary key value, so
locking will work as expected for multiple states.
2017-03-22 16:59:13 -04:00
James Bardin 4980fa20e7 move s3 config from client to backend
The RemoteClient needs to be configured for the named state, so move the
general config to the backend.

Rename some fields for consistency.
2017-03-22 15:52:55 -04:00
James Bardin 9f5cf2b105 convert S3 remote state to a backend
Move the S3 State from a legacy remote state to an official backend.

This increases test coverage, uses a set schema for configuration, and
will allow new backend features to be implemented for the S3 state, e.g.
"environments".
2017-03-22 10:59:37 -04:00
Mitchell Hashimoto d01886a644
command: remove legacy remote state on migration
Fixes #12871

We were forgetting to remove the legacy remote state from the actual
state value when migrating. This only causes an issue when saving a plan
since the plan contains the state itself and causes an error where both
a backend + legacy state exist.

If saved plans aren't used this causes no noticable issue.

Due to buggy upgrades already existing in the wild, I also added code to
clear the remote section if it exists in a standard unchanged backend
2017-03-20 10:14:59 -07:00
Mitchell Hashimoto 23dd2a0ee5
backend/atlas: accept ATLAS_ADDRESS env var for address
This is required for on-premise TFE.
2017-03-16 23:04:07 -07:00
Mitchell Hashimoto 2be1f55cbb
backend/local: allow refresh on empty/non-existent state
This allows a refresh on a non-existent or empty state file. We changed
this in 0.9.0 to error which seemed reasonable but it turns out this
complicates automation that runs refresh since it now needed to
determine if the state file was empty before running.

Its easier to just revert this into a warning with exit code zero.

The reason this changed is because in 0.8.x and earlier, the output
would be simply empty with exit code zero which seemed odd.
2017-03-16 12:11:31 -07:00
Mitchell Hashimoto 2969b29d9b
backend/local: call new test API correctly 2017-03-15 08:46:58 -07:00
Mitchell Hashimoto 1ca0352e5f Merge pull request #12558 from hashicorp/f-backend-testing
backend/local: run backend.TestBackend
2017-03-15 08:45:20 -07:00
Mitchell Hashimoto 0cd15176ff
backend/consul: rebase and modify gzip test 2017-03-14 17:59:54 -07:00
Mitchell Hashimoto 1daff7a826
backend/consul: support "lock" option to disable locking
This adds a "lock" config (default true) to allow users to optionally
disable state locking with Consul. This is necessary if the token given
doesn't have session permission and is necessary for backwards
compatibility.
2017-03-14 17:59:10 -07:00
James Bardin 08d2b44ada add gzip to consul backend tests 2017-03-14 15:41:00 -04:00
James Bardin 1527eae6b7 Merge pull request #8748 from christoe/christoe/8491-gzip-consul-remote-state
state/remote: Add gzip support to consul remote state (#8491)
2017-03-14 15:27:46 -04:00
James Bardin 86f711f6fc Make consul backend tests opt-in
This way we don't require contributers to have consul installed to run
make test.
2017-03-14 14:44:03 -04:00
James Bardin fa7743b627 quiet the consul server during backend tests
Don't display logs unless using `-v`
2017-03-14 10:49:38 -04:00
James Bardin 90055c6ae2 convert the consul backend to use consul/testutil
Start up our own consul server for unit tests.
2017-03-13 18:25:58 -04:00
Christoffer Kylvåg abfa35db7c backend/remote-state/consul: Make gzip compression configurable (#8491) 2017-03-13 08:17:33 +01:00
Mitchell Hashimoto f63e8b3398
backend/local: run backend.TestBackend
This verifies that local behaves in an expected way by the backend
package.
2017-03-09 16:17:21 +05:30
Christoffer Kylvåg e098c7c24a backend/remote-state/consul: Add gzip support to consul backend client (#8491) 2017-03-09 08:00:19 +01:00
Mitchell Hashimoto 35afb7bc86
backend/consul: some comments 2017-03-01 23:01:28 -08:00
Mitchell Hashimoto b842fd0c27
backend/consul: support named states 2017-03-01 22:58:51 -08:00
Mitchell Hashimoto 3db55cf747
backend/consul: build your own backend 2017-03-01 22:19:36 -08:00
Mitchell Hashimoto 08b47cf73b
backend/atlas: adhere to new multi-state interface 2017-03-01 13:31:52 -08:00
Mitchell Hashimoto e6a87cf8de
backend/atlas: adhere to backend.CLI 2017-03-01 13:29:16 -08:00
Mitchell Hashimoto 942572b574
backend/init: add atlas, remove legacy atlas remote state 2017-03-01 13:29:16 -08:00
Mitchell Hashimoto d00d6bc7fe
backend/atlas: initial state working 2017-03-01 13:29:16 -08:00
Mitchell Hashimoto bdde7d845a Merge pull request #12347 from hashicorp/b-env-migrate
command: migrating envs when changing backends
2017-03-01 12:58:40 -08:00
Mitchell Hashimoto 549d525487 Merge pull request #12320 from hashicorp/b-legacy-state
command: fix loading legacy remote state + migration guide
2017-03-01 12:39:17 -08:00
Mitchell Hashimoto e75b666591
command: test multi-state to single state 2017-03-01 11:34:45 -08:00
Mitchell Hashimoto 1d8b76c89d
command: initial work on migrating envs, basic cases first 2017-03-01 10:59:17 -08:00
Mitchell Hashimoto 6ea0dc0964
backend/consul: make address have a default to prevent input 2017-02-28 19:29:05 -08:00
James Bardin 4dac986a91 Local.StatePaths doesn't need to reutrn an error
add a test to ensure we have consistent output
2017-02-28 19:18:16 -05:00
James Bardin b53704ed87 Thread the environment through all commands
Add Env and SetEnv methods to command.Meta to retrieve the current
environment name inside any command.

Make sure all calls to Backend.State contain an environment name, and
make the package compile against the update backend package.
2017-02-28 16:35:46 -05:00
James Bardin f866bb545c update remote-state/consul 2017-02-28 16:35:46 -05:00
James Bardin 597eb6c918 update remote-state/inmem client 2017-02-28 16:35:45 -05:00
James Bardin 8fdf3a42b8 update remote-state.Backend 2017-02-28 16:35:45 -05:00
James Bardin 5762878eba Make backcend/legacy match new Backend iface
move the unsupported error value to backend.ErrNamedStatesNotSupported
to be used by any backend implementation.
2017-02-28 16:35:45 -05:00
James Bardin 65527f35a4 update local.Local to match the latest Backend
Update the methods, remove the handling of "current", and make tests
pass.
2017-02-28 16:07:31 -05:00
James Bardin 96194fbc0d Update Backend interface to latest iteration
What will hopfully be the final version of the Backend interface. This
combines the MultiState interface into Backend since it will be required
to implement, and simplifies the interface because the Backend is no
longer responsible for tracking the current state.
2017-02-28 16:07:07 -05:00
James Bardin 7f453f3341 remove some leftover methods in the legacy backend
These were left from the initial implementation, but are not used.
2017-02-28 16:07:07 -05:00
James Bardin fbc11c7961 fix incorrect current state in local backend
Forgot to remove the currentState field, which was not always set. The
current state should always just be read from the environment file.

Always return the default state name when we can't determine the state.
2017-02-28 16:07:07 -05:00
James Bardin e6eb71dde5 Add tests to check Backend delegation
Ensure that when MultiState methods are properly delegated when there is
a defined Local.Backend.
2017-02-28 16:07:06 -05:00
James Bardin 0933541a8c Split out the backend environment interface
Split the interface to change environments out from the minimal Backend
interface, to make it optional for backend implementations. If
backend.MultiState isn't implemented, return a "not implemented" from
environment related methods.

Have the Local backend delegate the MultiState methods to the proper
backend.
2017-02-28 16:06:14 -05:00
James Bardin dbc45b907c Make the Local backend handle its own named states
Add the functionality required for terraform environments
2017-02-28 16:03:36 -05:00
James Bardin 761c63d14a Update Backend to incorporate environments
Add the missing methods/arguments to handle Terraform environments in
Backends. Extra functionality simply returns defaults for now.
2017-02-28 16:03:36 -05:00
Mitchell Hashimoto 3cedfa00f4
command: use backend.CLIIinit
I made this interface way back with the original backend work and I
guess I forgot to hook it up! This is becoming an issue as I'm working
on our 2nd enhanced backend that requires this information and I
realized it was hardcoded before.

This propertly uses the CLIInit interface allowing any backend to gain
access to this data.
2017-02-28 10:58:29 -08:00
Mitchell Hashimoto 8f11068ab2 Merge pull request #12173 from hashicorp/b-remote-state-ds
providers/terraform: remote state data source supports backends
2017-02-22 18:43:06 -08:00
Mitchell Hashimoto d2d87bccf0 Merge pull request #12155 from hashicorp/b-state-backend
command: refresh state in old commands for backend
2017-02-22 18:40:55 -08:00
Mitchell Hashimoto 9574f16f92
backend/local: refresh with no config should not crash on input
Fixes #12174

You're allowed to refresh with a nil module (no configs) as long as you
have state. However, if `-input=true` (default) then this would crash
since the input attempts to read the configs.

The API contract with `terraform.Context` says that the module tree must
be non-nil and loaded. To do this for other commands we create an empty
module tree. We do that here now.
2017-02-22 13:10:08 -08:00
Mitchell Hashimoto ebb22d3ecd
backend/local: don't RefreshState on State API 2017-02-22 13:01:16 -08:00
Mitchell Hashimoto 52720ce880
providers/terraform: data source uses backends for state loading 2017-02-22 11:37:56 -08:00
Mitchell Hashimoto f79e04500f
backend/init: a package for storing the factories for backends 2017-02-22 11:17:06 -08:00
James Bardin 2392455a67 Add inmem remote backend
Add an Inmem remote state backend.

The only config option right now is a lock_id, which will instantiate
the backend in a locked state for testing.
2017-02-20 14:49:46 -05:00
Mitchell Hashimoto d443bf1b56
backend/local: allow nil modules (no config) if executing a plan 2017-02-16 10:56:39 -08:00
Mitchell Hashimoto 1480d0c5b8
backend/local: check for empty config on apply
This prevents Terraform from crashing on apply/destroy with a directory
with no Terraform configuration files. We allow a destroy with no files
but not an apply.
2017-02-15 16:00:59 -08:00
Mitchell Hashimoto efe754183b
state/remote: export ClientLocker, test for implementation
This adds unit tests (that will fail at compile time) if various structs
don't implement the right interfaces for locking
2017-02-15 14:20:59 -08:00
James Bardin ec00564be6 Clean up LockInfo and LockError and use them
Gove LockInfo a Marshal method for easy serialization, and a String
method for more readable output.

Have the state.Locker implementations use LockError when possible to
return LockInfo and an error.
2017-02-15 14:44:43 -05:00
James Bardin 67bbebce08 Have consul state reutrn the lock ID
The lock ID isn't used because the lock is tied to the client, but
return the lock ID to match the behavior of other locks.
2017-02-15 14:44:43 -05:00
James Bardin f2e496a14c Have backend operations properly unlock state
Make sure unlock is called with the correct LockID during operations
2017-02-15 14:41:55 -05:00
James Bardin f5ed8cd288 Use NewLockInfo to get a pre-populated value
Using NewLockInfo ensure we start with all required fields filled.
2017-02-15 14:41:55 -05:00
James Bardin cd233fef6a make consul client pass state.Locker tests 2017-02-15 14:41:55 -05:00
James Bardin 67dc16c9ca Make backend/local test pass 2017-02-15 14:41:55 -05:00
Mitchell Hashimoto 235b7eb38e Merge pull request #11944 from hashicorp/f-state-slow
show message if state lock acquisition/release is slow
2017-02-14 14:00:23 -08:00
Mitchell Hashimoto 5e4f6cf2b1
backend/local: fix could not to did not to prevent error look 2017-02-14 12:09:45 -08:00
Mitchell Hashimoto 65982bd412
backend/local: use new command/state package for better UX 2017-02-14 11:17:28 -08:00
James Bardin 80fab23e04 Don't test consul using demo.consul.io
We shoudn't require an external service for unit test.

TODO: create some proper acceptance tests for consul
2017-02-08 11:34:31 -05:00
James Bardin 14d965722e Use single state.LockInfo struct
Remove redundant structures
2017-02-08 11:34:31 -05:00
James Bardin 9b76f6e138 Move TestRemoteLocks to state/remote
This was legacy remote state client and backends can use this test
function without an import cycle.
2017-02-08 11:25:52 -05:00
James Bardin 54cac349a3 Add state locking to consul backend
Use consul locks to implement state locking. The lock path is state path
+ "/.lock" which matches the consul cli default for locks. Lockinfo is
stored at path + "/.lockinfo".
2017-02-08 11:25:52 -05:00
Mitchell Hashimoto bdca9bffe4
backend/local: output warnings
Fixes #11628

This is a simple fix to output warnings. I originally forgot to do this
since the local backend didn't have a CLI UI at the time. It does now so
this is an easy fix.
2017-02-07 13:22:28 -08:00
James Bardin 0d7752b0f5 Update runningOp.Err with State.Unlock error
Have the defer'ed State.Unlock call append any error to the
RunningOperation.Err field. Local error would be rare and
self-correcting, but when the backend.Local is using a remote state the
error may require user intervention.
2017-02-06 09:54:15 -05:00
James Bardin 9cdba1f199 enable local state locking for apply
Have the LocalBackend lock the state during operations, and enble this
for the apply comand.
2017-02-02 18:08:28 -05:00
Mitchell Hashimoto a424203ea3
backend/local: validate module exists for plan
Fixes #11504

The local backend should error if `terraform plan` is called in a
directory with no Terraform config files (same behavior as 0.8.x).
**New behavior:** We now allow `terraform plan -destroy` with no
configuration files since that seems reasonable.
2017-01-29 20:02:12 -08:00
Mitchell Hashimoto 31f7cca77f
backend/local: fix crash (in tests) due to not guarding nil CLI 2017-01-26 14:33:50 -08:00
Mitchell Hashimoto 1f5d425428
backend/remote-state
This allows migration of the remote state implementations to a richer
experience including input asking.
2017-01-26 14:33:49 -08:00
Mitchell Hashimoto 13c34b16e8
backend/legacy
This allows using legacy remote state backends with the new backend
interface.
2017-01-26 14:33:49 -08:00
Mitchell Hashimoto 397e1b3132
backend/local
The local backend implementation is an implementation of
backend.Enhanced that recreates all the behavior of the CLI but through
the backend interface.
2017-01-26 14:33:49 -08:00
Mitchell Hashimoto 8a070ddef0
backend: introduce the backend set of interfaces
Backends are a mechanism that allow abstracting the behavior of
Terraform CLI from the actual core. This allows us to slip in special
behavior such as state loading, remote operations, etc.
2017-01-26 14:33:49 -08:00