Commit Graph

167 Commits

Author SHA1 Message Date
James Bardin 8f7f1917f2 Remove state file data when writing a nil state
The old behavior in this situation was to simply delete the file. Since
we now have a lock on this file we don't want to close or delete it, so
instead truncate the file at offset 0.

Fix a number of related tests
2017-01-30 17:16:57 -05:00
James Bardin da0c325e5c Silence state package logs during tests
Output log output when testing is verbose
2017-01-30 17:16:57 -05:00
James Bardin 10f6d7f30f Add locking for s3 state
Use a DynamoDB table to coodinate state locking in S3.

We use a simple strategy here, defining a key containing the value of
the bucket/key of the state file as the lock. If the keys exists, the
locks fails.

TODO: decide if locks should automatically be expired, or require manual
intervention.
2017-01-30 17:16:57 -05:00
James Bardin 35307d5a60 Add remote state locking
In order to provide lockign for remote states, the Cache state,
remote.State need to expose Lock and Unlock methods. The actual locking
will be done by the remote.Client, which can implement the same
state.Locker methods.
2017-01-30 17:16:57 -05:00
James Bardin 6162cde6ff Add basic local state locking
Add the LockUnlock methods to LocalState and BackupState.

The implementation for LocalState will be platform specific. We will use
OS-native locking on the state files, speficially locking whichever
state file we intend to write to.
2017-01-30 17:16:57 -05:00
James Bardin cc0712edab add state.Locker interface
Changed from state.StateLocker to remove the stutter.

State implementations can provide Lock/Unlock methods to lock the state
file. Remote clients can also provide these same methods, which will be
called through remote.State.
2017-01-30 17:16:57 -05:00
Mitchell Hashimoto d3633ab368
state/remote: remove consul since we converted that 2017-01-26 14:33:50 -08:00
Ben Slusky e3b20c3508 Basic authentication for HTTP remote state backend 2017-01-19 21:01:50 -05:00
Aleksander Modzelewski 2627e4bd8d Add token authentication to Swift remote 2016-12-20 18:22:48 +01:00
Joe Topjian 8ecc6fda0a Merge pull request #10055 from fatmcgav/swift_add_version_expire
state/remote/swift: Add support for versioning state file in swift, and expiring objects
2016-11-22 09:57:49 -07:00
Gavin Williams c77ed69a4e state/remote/swift: Add support for versioning state file in swift, and expiring object after period of time.
Update website docs with changes
2016-11-22 16:46:10 +00:00
James Bardin 91378d0499 Merge pull request #10233 from hashicorp/jbardin/GH-10229
An empty module in state can panic
2016-11-21 17:35:33 -05:00
James Bardin 9616618de1 Make sure test has a valid ResourceState
Empty resources are now pruned more aggressively, so make sure there is
a valid ResourceState in the test ModuleState.
2016-11-20 13:13:43 -05:00
James Bardin 7715bc8423 change failing test to use subtests 2016-11-20 13:02:48 -05:00
Peter McAtominey 507efcb180 state/azure: support passing of lease ID when writing storage blob (#10115)
Also fixed tests failing auth caused by getStorageAccountAccessKey returning the
key name rather than the value

TF_ACC= go test ./state/remote -v -run=TestAz -timeout=10m -parallel=4
=== RUN   TestAzureClient_impl
--- PASS: TestAzureClient_impl (0.00s)
=== RUN   TestAzureClient
2016/11/18 13:57:34 [DEBUG] New state was assigned lineage "96037426-f95e-45c3-9183-6c39b49f590b"
2016/11/18 13:57:34 [TRACE] Preserving existing state lineage "96037426-f95e-45c3-9183-6c39b49f590b"
--- PASS: TestAzureClient (130.60s)
=== RUN   TestAzureClientEmptyLease
2016/11/18 13:59:44 [DEBUG] New state was assigned lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
2016/11/18 13:59:44 [TRACE] Preserving existing state lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
--- PASS: TestAzureClientEmptyLease (128.15s)
=== RUN   TestAzureClientLease
2016/11/18 14:01:55 [DEBUG] New state was assigned lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
2016/11/18 14:01:55 [TRACE] Preserving existing state lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
--- PASS: TestAzureClientLease (138.09s)
PASS
ok  	github.com/hashicorp/terraform/state/remote	397.111s
2016-11-18 17:26:25 +02:00
Gavin Williams bf8612b9b7 state/remote/swift: Enhancements to support full set of Openstack configuration options, plus SSL certs. Documentation updated to support 2016-11-03 07:36:18 +00:00
Gavin Williams ac07430124 state/remote/swift: Update to use Gophercloud/gophercloud, restructure to support insecure TLS, added Keystone v3 auth params
Update swift remote documentation
2016-11-01 16:55:05 +00:00
James Bardin f175497dd7 Fix vet issues
None were critical, but these will fail with the next version of vet.
2016-10-18 11:11:12 -04:00
James Nugent afe2d7b65b Merge pull request #7320 from dtolnay/conflict
core: Allow refresh of local state with no resources
2016-10-14 11:00:46 -05:00
Cameron Watters b4eb63d710 state/remote: New provider - manta
- add remote state provider backed by Joyent's Manta
- add documentation of Manta remote state provider
- explicitly check for passphrase-protected SSH keys, which are currently
  unsupported, and generate a more helpful error (borrowed from Packer's
  solution to the same problem):
  https://github.com/mitchellh/packer/blob/master/common/ssh/key.go#L27
2016-09-14 20:44:52 +01:00
David Tolnay bb7dfaec81 Fix TestCacheState_RefreshState 2016-09-07 19:34:13 -07:00
David Tolnay 61185f083c Merge branch hashicorp/master into dtolnay/conflict 2016-09-07 19:17:22 -07:00
James Nugent 8d0a68e1d4 state/remote: Officially Support local backend
This is a rework of pull request #6213 submitted by @joshuaspence,
adjusted to work with the remote state data source. We also add
a deprecation warning for people using the unsupported API, and retain
the ability to refer to "_local" as well as "local" for users in a mixed
version environment.
2016-09-03 15:42:40 -07:00
Ian Duffy 767914bbdc [GH-1275] Support for AWS access via IAMs AssumeRole functionality
This commit enables terraform to utilise the assume role functionality
of sts to execute commands with different privileges than the API
keys specified.

Signed-off-by: Ian Duffy <ian@ianduffy.ie>
2016-09-02 10:22:57 -07:00
James Bardin cdb80f68a8 Ensure better state normalization
Fix checksum issue with remote state

If we read a state file with "null" objects in a module and they become
initialized to an empty map the state file may be written out with empty
objects rather than "null", changing the checksum. If we can detect
this, increment the serial number to prevent a conflict in atlas.

Our fakeAtlas test server now needs to decode the state directly rather
than using the ReadState function, so as to be able to read the state
unaltered.

The terraform.State data structures have initialization spread out
throughout the package. More thoroughly initialize State during
ReadState, and add a call to init() during WriteState as another
normalization safeguard.

Expose State.init through an exported Init() method, so that a new State
can be completely realized outside of the terraform package.
Additionally, the internal init now completely walks all internal state
structures ensuring that all maps and slices are initialized.  While it
was mentioned before that the `init()` methods are problematic with too
many call sites, expanding this out better exposes the entry points that
will need to be refactored later for improved concurrency handling.

The State structures had a mix of `omitempty` fields. Remove omitempty
for all maps and slices as part of this normalization process. Make
Lineage mandatory, which is now explicitly set in some tests.
2016-08-12 11:09:50 -04:00
Guido Bakker ee9ebe849f Support setting datacenter when using consul remote state (#8102)
* Support setting datacenter when using consul remote state

Change-Id: I8c03f4058e9373f0de8fde7ce291ec552321cc60

* Add documentation for setting datacenter when using consul remote state

Change-Id: Ia62feea7a910a76308f0a5e7f9505c9a210e0339
2016-08-11 22:58:20 +12:00
James Bardin a1a501a26a Merge pull request #8053 from hashicorp/jbardin/atlas-retry-policy
don't retry the atlas requests with the wrong cert
2016-08-10 17:09:00 -04:00
James Bardin df0c795b39 Don't retry the atlas requests with the wrong cert
This probably won't recover, so abort immediately.
Requires retryablehttp CheckRetry patch.
2016-08-10 13:45:30 -04:00
Renier Morales c2bcb5fbe5 Skip IAM/STS validation and metadata check (#7874)
* Skip IAM/STS validation and metadata check

* Skip IAM/STS identity validation - For environments or other api
  implementations where there are no IAM/STS endpoints available, this
  option lets you opt out from that provider initialization step.
* Skip metdata api check - For environments in which you know ahead of
  time there isn't going to be a metadta api endpoint, this option lets
  you opt out from that check to save time.

* Allow iam/sts initialization even if skipping account/cred validation

(#7874)

* Split out skip of IAM validation into credentials and account id

(#7874)
2016-08-10 15:10:34 +01:00
James Bardin 0a3714eaac Don't send access_token in request params
Always send the access_token in the X-Atlas-Token header.
2016-08-05 11:44:14 -04:00
James Bardin 74813821ec Add remote state init test
Verify that a remote state file is correctly initialized in the same
manner as used by the `terraform remote config`
2016-07-07 16:24:38 -04:00
James Bardin 3622bfddd6 Revert #7464 and allow an empty state
Revert back to using a nil state. The external usage of the state shoudl
always check the Empty() method.
2016-07-07 16:19:58 -04:00
James Bardin 24f6d3fe98 Return an error when there's no remote state
When refreshing remote state, indicate when no state file was found with
an ErrRemoteStateNotFound error. This prevents us from inadvertantly
getting a nil state into a terraform.State where we assume there's
always a root module.
2016-07-01 18:44:23 -04:00
Paul Stack 079e1f9a56 provider/azurerm: Bump azure-sdk-for-go to 3.0.0-beta (#7420)
provider/azurerm: Bump azure-sdk-for-go to 3.0.0-beta
2016-06-30 15:36:08 +01:00
David Tolnay ff80e7b245
Allow refresh of local state with no resources 2016-06-24 21:54:43 -07:00
James Nugent d60365af02 core: Correctly ensure that State() is a copy
The previous mechanism for testing state threw away the mutation made on
the state by calling State() twice - this commit corrects the test to
match the comment.

In addition, we replace the custom copying logic with the copystructure
library to simplify the code.
2016-06-22 17:21:27 +03:00
Paul Stack 32e1a32476 remote: Rename mas to azure (#7114)
MAS wasn't obvious it was Azure so renamed it to Azure
2016-06-10 20:04:40 +02:00
Maxime Bury c98f391bee Add basic implementation for remote state on azure (#7064)
* Add basic implementation for remote state on azure

* Don't auto-provision the container

* Fix compilation errors

* Add factory to the remote map

* Add documentation

* Add acceptance tests
2016-06-10 19:27:57 +02:00
James Nugent 706ccb7dfe core: Introduce state v3 and upgrade process
This commit makes the current Terraform state version 3 (previously 2),
and a migration process as part of reading v2 state. For the most part
this is unnecessary: helper/schema will deal with upgrading state for
providers written with that framework. However, for providers which
implemented the resource model directly, this gives a best-efforts
attempt at lossless upgrade.

The heuristics used to change the count of a map from the .# key to the
.% key are as follows:

    - if the flat map contains any non-numeric keys, we treat it as a
      map
    - if the map is empty it must be computed or optional, so we remove
      it from state

There is a known edge condition: maps with all-numeric keys are
indistinguishable from sets without access to the schema. They will need
manual conversion or may result in spurious diffs.
2016-06-09 10:49:49 +01:00
Matt Morrison cbfb4d8b86 remote state: Add GCS provider for remote state 2016-05-31 13:42:57 -05:00
James Nugent 3ea3c657b5 core: Use OutputState in JSON instead of map
This commit forward ports the changes made for 0.6.17, in order to store
the type and sensitive flag against outputs.

It also refactors the logic of the import for V0 to V1 state, and
fixes up the call sites of the new format for outputs in V2 state.

Finally we fix up tests which did not previously set a state version
where one is required.
2016-05-18 13:25:20 -05:00
Joseph Anthony Pasquale Holsten 546fb94265 atlas: update test err msg to reflect real timeout 2016-05-12 15:30:17 -07:00
James Nugent 6a20e8927d core: Fix issues from rebasing dev-0.7 onto master
- Fix sensitive outputs for lists and maps
- Fix test prelude which was missed during conflict resolution
- Fix `terraform output` to match old behaviour and not have outputs
  header and colouring
- Bump timeout on TestAtlasClient_UnresolvableConflict
2016-05-10 15:43:50 -04:00
James Nugent 6aac79e194 state: Add support for outputs of multiple types
This commit adds the groundwork for supporting module outputs of types
other than string. In order to do so, the state version is increased
from 1 to 2 (though the "public-facing" state version is actually as the
first state file was binary).

Tests are added to ensure that V2 (1) state is upgraded to V3 (2) state,
though no separate read path is required since the V2 JSON will
unmarshal correctly into the V3 structure.

Outputs in a ModuleState are now of type map[string]interface{}, and a
test covers round-tripping string, []string and map[string]string, which
should cover all of the types in question.

Type switches have been added where necessary to deal with the
interface{} value, but they currently default to panicking when the input
is not a string.
2016-05-10 14:40:12 -04:00
John Engelman 14f6f90621 Support standard AWS config in the S3 remote backend. (#5270) 2016-05-06 17:52:18 +01:00
Paul Hinze 4ac6dda633
state/remote/atlas: Use go-rootcerts for certificate loading
Allows CA certs to be configured via `ATLAS_CAFILE` and `ATLAS_CAPATH`
env vars, and works around https://github.com/golang/go/issues/14514 on
OS X.
2016-05-03 09:52:36 -05:00
Anubhav Mishra f58290c83f Moving octet-stream to json for remote http backend 2016-03-07 18:39:41 -08:00
Trevor Pounds bac909fdbf Fix `go vet -unreachable` warnings. 2016-02-17 11:59:50 -08:00
Trevor Pounds 79742fc367 Enable `go vet -composites` check and fix warnings. 2016-02-17 11:59:50 -08:00
Paul Hinze 6bafa74011 tests: allow opt-out of remote tests via env var
Adds the `TF_SKIP_REMOTE_TESTS` env var to be used in cases where the
`http.Get()` smoke test passes but the network is not able to service
the needs of the tests.

Fixes #4421
2016-01-21 15:44:18 -06:00
Paul Hinze ba21769083 Merge pull request #2903 from kjmkznr/remote-s3-sse-kms
state/remote/s3: Allows KMS Key Encryption setting when using S3 backend with encrypt
2016-01-19 18:54:03 -06:00
Sander van Harmelen 9b27db6fea Add the option to specify a custom (AWS compatible) S3 endpoint
Same fix/option as I added in Vault not too long ago:
https://github.com/hashicorp/vault/pull/750
2016-01-11 14:28:52 +01:00
James Nugent ff9345287b Merge branch 'artifactory-remote-state' of https://github.com/lusis/terraform into lusis-artifactory-remote-state 2015-12-19 13:36:55 -05:00
Paul Hinze 1100243536 state/remote/atlas: switch to retryablehttp
The retryablehttp package implements basic retries w/ exponential
backoff, which helps the remote state push recover in cases of
connectivity blips or transient errors.
2015-12-18 09:48:54 -06:00
Paul Hinze 1a19f43ee1 core: support HTTP basic auth in consul remote state
Closes #1663
2015-12-04 07:15:18 -06:00
Rafal Jeczalik 9e66e18334 provider/aws: fix for https://github.com/aws/aws-sdk-go/issues/452 2015-11-24 09:30:21 +01:00
Paul Hinze cadbbbae08 aws: fix build after upstream breaking change
see
1a69d06935
2015-10-29 18:52:10 -05:00
John E. Vincent c3f863f4c5 add artifactory remote state storage 2015-10-29 09:33:09 -04:00
Paul Hinze 6aa5fdc938 state/remote/atlas: handle conflicts on equivalent states
Atlas returns an HTTP 409 - Conflict if the pushed state reports the same
Serial number but the checksum of the raw content differs. This can
sometimes happen when Terraform changes state representation internally
between versions in a way that's semantically neutral but affects the JSON
output and therefore the checksum.

Here we detect and handle this situation by ticking the serial and retrying
iff for the previous state and the proposed state:

  * the serials match
  * the parsed states are Equal (semantically equivalent)

In other words, in this situation Terraform can override Atlas's detected
conflict by asserting that the state it is pushing is indeed correct.
2015-10-22 15:40:45 -05:00
Jeff Mitchell 1373a6086b Use cleanhttp for new http clients 2015-10-22 14:23:59 -04:00
Paul Hinze 15a36d06cf Merge pull request #3532 from hashicorp/remove-default-client
Remove usage of http.DefaultClient
2015-10-20 10:42:31 -05:00
Jeff Mitchell b0ceffc322 Remove usage from dependencies as well. Other dependencies need upstream merging to completely solve this. 2015-10-19 12:06:34 -04:00
Jeff Mitchell 3c0ed11922 Remove usage of http.DefaultClient 2015-10-19 12:06:33 -04:00
Nathan Zadoks bb51882f33 Etcd remote state backend 2015-10-18 23:24:55 +02:00
Kazunori Kojima 9186c29dd8 Fix typo 2015-10-07 23:39:08 +09:00
Kazunori Kojima 576b2d1109 Change KMS Key ID configuration name to used in other 2015-10-07 23:09:03 +09:00
Kazunori Kojima ba8f1fa1f0 Add support S3 server side encryption with KMS.
* Example

```
terraform remote config \
  -backend=s3
  -backend-config="bucket=bucket-tfstate"
  -backend-config="key=terraform.tfstate"
  -backend-config="region=ap-northeast-1"
  -backend-config="encrypt=1"
  -backend-config="kmsKeyID=arn:aws:kms:ap-northeast-1:123456789:key/ac54dbd2-f301-42c1-bab9-88e6a84292a9"
```
2015-10-07 23:03:55 +09:00
Martin Atkins ccc8f0d0fb S3 remote state use application/json Content-Type.
The state is always JSON, in spite of the fact that this interface
presents it as an opaque byte array. It's more helpful to those interacting
with the state object outside of Terraform for it to have a more specific
content-type.
2015-10-03 18:02:36 -07:00
Martin Atkins 859c6c5e68 Merge #3233: Allow canned ACLs on S3 remote state. 2015-10-03 17:40:17 -07:00
Radek Simko 3d77d158f7 remote/s3: Add support for ACL 2015-09-14 10:40:46 +01:00
Jack Pearkes eba638d044 remote/atlas: if `ATLAS_RUN_ID` is set, send it with remote state save
This detects the presence of ATLAS_RUN_ID in the environment and sends
it if exists with remote state PUT requests with query params.
2015-08-20 13:54:26 -07:00
Jack Pearkes d19ec9d57a remote/atlas: remove old force param commented out
It appears this was left in through development, but as force
is not a parameter likely doesn't have any relevance anymore.
2015-08-20 13:21:47 -07:00
Paul Hinze 1a44b427a7 state/remote/s3: match with upstream changes 2015-08-10 15:46:22 -05:00
Clint Shryock 579ccbefea provider/aws: Update source to comply with upstream breaking change 2015-07-28 15:29:46 -05:00
Mitchell Hashimoto e135ff546a state/remote: clean up the encryption flag stuff
/cc @hobbeswalsh
2015-06-25 09:23:12 -07:00
Mitchell Hashimoto a5af429457 Merge pull request #2405 from hobbeswalsh/master
remote: allowing at-rest encryption when using S3
2015-06-25 09:20:13 -07:00
Mitchell Hashimoto cddd54c3de fmt 2015-06-23 22:31:24 -07:00
Robin Walsh ad17cf55a0 Allowing at-rest encryption when using S3
This change allows the user to specify `-backend-config="encrypt=1"`
to tell S3 to encrypt the data that's in the bucket when using S3
for remote config storage.

The encryption uses "Amazon S3-managed encryption keys" so it should
not require any further user intervention.

A line was added to the unit test just for coverage.
The acceptance test was modified to:
  a) Use encryption
  b) Push some test data up to the bucket created to ensure
     that Amazon accepts the header.
2015-06-19 11:33:03 -07:00
Mitchell Hashimoto 76d920f504 state/remote: more canonical Go for skip TLS verify
/cc @LeftyBC - Hope this helps! Some basic point:

* Idiomatic Go is to use thisCasingStyle and not this_casing_style
* Less repetitive to just create an http.Client once and re-use, also
  more flexible for later.
* The empty `if ok {}` was kind of awkward, replace with proper check
2015-06-07 22:24:31 -07:00
Mitchell Hashimoto 6649658d62 Merge pull request #2220 from LeftyBC/master
state/remote: Add a boolean flag to http remote that disables cert validity checking (for e.g. self-signed certs)
2015-06-07 22:19:31 -07:00
George Hartzell 4fc6dd0141 Only run Swift tests when Swift is available
Only run the Swift remote tests when OpenStack seems
to have been set up and when the autho host is reachable.
2015-06-06 10:19:25 -07:00
George Hartzell 258b1a4263 Implement OpenStack/Swift remote
Rework devcamcar's OpenStack Swift remote [pull
request](https://github.com/hashicorp/terraform/pull/942) to work with
Terraform's new `state/remote` and Gophercloud's current implementation.

`Get()` changed up a bit from devcamcar's version (using different
Gopercloud functionality resulted in less fussing around to figure out
the error case).

Otherwise this is a transliteration/remix of his ideas.
2015-06-06 10:04:31 -07:00
Colin Moller 67e56671ec Merge remote branch 'upstream/master' 2015-06-03 17:47:32 -07:00
Colin Moller e7130e9c85 go fmt 2015-06-03 17:23:45 -07:00
Colin Moller fc2f97ca89 Add a boolean flag to http remote that disables cert validity checking (for e.g. self-signed certs) 2015-06-03 17:09:02 -07:00
Paul Hinze b71fa3d0ae provider/aws: handle upstream aws-sdk-go repo move
`awslabs/aws-sdk-go => aws/aws-sdk-go`

Congrats to upstream on the promotion. :)
2015-06-03 13:36:57 -05:00
Clint Shryock 883e2848c6 update remote state s3 file to fix error 2015-05-20 15:20:30 -05:00
Paul Hinze 31258e06c6 provider/aws: fix breakages from awserr refactor
This landed in aws-sdk-go yesterday, breaking the AWS provider in many places:

3c259c9586

Here, with much sedding, grepping, and manual massaging, we attempt to
catch Terraform up to the new `awserr.Error` interface world.
2015-05-20 06:21:23 -05:00
Sander van Harmelen 897bf5e53d Added `SharedCredentialsProvider` and `EC2RoleProvider` as well... 2015-05-05 20:38:35 +02:00
Sander van Harmelen 5435815524 Fixing PR #1804 2015-05-05 20:30:35 +02:00
Sander van Harmelen 7274de3c1d Small fix needed to be inline with the updated AWS SDK again...
Executed a couple of AWS ACC test as well to make sure everything works
properly again after this fix…
2015-05-05 12:26:26 +02:00
Martin Atkins 53e93d6f63 S3 remote stage storage backend, against the new remote state API.
Stores state in a particular key in a given S3 bucket.
2015-04-30 09:21:49 -07:00
Mitchell Hashimoto c58e09abaa state/remote: make malformed name message clearer [GH-1332] 2015-04-22 08:59:30 +02:00
Mitchell Hashimoto 1fac7b6488 state: cache should use State.Equal to check equality
reflect.DeepEqual was returning false, sometimes.
2015-04-07 15:03:04 -07:00
Paul Hinze 0a24e72c3b state/remote: allow https consul addresses
Sending state over a cleartext protocol is bad in untrusted networks.
Expose `-backend-config="scheme=https"` and wire it through to the
Consul client.
2015-03-17 17:11:35 -04:00
Mitchell Hashimoto aee27314eb state/remote: add undocumented file backend for remote state 2015-03-05 13:15:14 -08:00
Mitchell Hashimoto cc8e6b6331 state: deep copies are required 2015-02-23 21:36:58 -08:00
Mitchell Hashimoto ed6128aa6e state/remote: increment serial properly 2015-02-23 21:30:59 -08:00
Mitchell Hashimoto c2bf600603 state: only change serial if changed 2015-02-23 21:26:33 -08:00
Mitchell Hashimoto bfe0edef51 state/remote: passing Atlas state test 2015-02-23 17:56:29 -08:00