Use a DynamoDB table to coodinate state locking in S3.
We use a simple strategy here, defining a key containing the value of
the bucket/key of the state file as the lock. If the keys exists, the
locks fails.
TODO: decide if locks should automatically be expired, or require manual
intervention.
In order to provide lockign for remote states, the Cache state,
remote.State need to expose Lock and Unlock methods. The actual locking
will be done by the remote.Client, which can implement the same
state.Locker methods.
Add the LockUnlock methods to LocalState and BackupState.
The implementation for LocalState will be platform specific. We will use
OS-native locking on the state files, speficially locking whichever
state file we intend to write to.
Changed from state.StateLocker to remove the stutter.
State implementations can provide Lock/Unlock methods to lock the state
file. Remote clients can also provide these same methods, which will be
called through remote.State.
Also fixed tests failing auth caused by getStorageAccountAccessKey returning the
key name rather than the value
TF_ACC= go test ./state/remote -v -run=TestAz -timeout=10m -parallel=4
=== RUN TestAzureClient_impl
--- PASS: TestAzureClient_impl (0.00s)
=== RUN TestAzureClient
2016/11/18 13:57:34 [DEBUG] New state was assigned lineage "96037426-f95e-45c3-9183-6c39b49f590b"
2016/11/18 13:57:34 [TRACE] Preserving existing state lineage "96037426-f95e-45c3-9183-6c39b49f590b"
--- PASS: TestAzureClient (130.60s)
=== RUN TestAzureClientEmptyLease
2016/11/18 13:59:44 [DEBUG] New state was assigned lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
2016/11/18 13:59:44 [TRACE] Preserving existing state lineage "d9997445-1ebf-4b2c-b4df-15ae152f6417"
--- PASS: TestAzureClientEmptyLease (128.15s)
=== RUN TestAzureClientLease
2016/11/18 14:01:55 [DEBUG] New state was assigned lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
2016/11/18 14:01:55 [TRACE] Preserving existing state lineage "85912a12-2e0e-464c-9886-8add39ea3a87"
--- PASS: TestAzureClientLease (138.09s)
PASS
ok github.com/hashicorp/terraform/state/remote 397.111s
- add remote state provider backed by Joyent's Manta
- add documentation of Manta remote state provider
- explicitly check for passphrase-protected SSH keys, which are currently
unsupported, and generate a more helpful error (borrowed from Packer's
solution to the same problem):
https://github.com/mitchellh/packer/blob/master/common/ssh/key.go#L27
This is a rework of pull request #6213 submitted by @joshuaspence,
adjusted to work with the remote state data source. We also add
a deprecation warning for people using the unsupported API, and retain
the ability to refer to "_local" as well as "local" for users in a mixed
version environment.
This commit enables terraform to utilise the assume role functionality
of sts to execute commands with different privileges than the API
keys specified.
Signed-off-by: Ian Duffy <ian@ianduffy.ie>
Fix checksum issue with remote state
If we read a state file with "null" objects in a module and they become
initialized to an empty map the state file may be written out with empty
objects rather than "null", changing the checksum. If we can detect
this, increment the serial number to prevent a conflict in atlas.
Our fakeAtlas test server now needs to decode the state directly rather
than using the ReadState function, so as to be able to read the state
unaltered.
The terraform.State data structures have initialization spread out
throughout the package. More thoroughly initialize State during
ReadState, and add a call to init() during WriteState as another
normalization safeguard.
Expose State.init through an exported Init() method, so that a new State
can be completely realized outside of the terraform package.
Additionally, the internal init now completely walks all internal state
structures ensuring that all maps and slices are initialized. While it
was mentioned before that the `init()` methods are problematic with too
many call sites, expanding this out better exposes the entry points that
will need to be refactored later for improved concurrency handling.
The State structures had a mix of `omitempty` fields. Remove omitempty
for all maps and slices as part of this normalization process. Make
Lineage mandatory, which is now explicitly set in some tests.
* Support setting datacenter when using consul remote state
Change-Id: I8c03f4058e9373f0de8fde7ce291ec552321cc60
* Add documentation for setting datacenter when using consul remote state
Change-Id: Ia62feea7a910a76308f0a5e7f9505c9a210e0339
* Skip IAM/STS validation and metadata check
* Skip IAM/STS identity validation - For environments or other api
implementations where there are no IAM/STS endpoints available, this
option lets you opt out from that provider initialization step.
* Skip metdata api check - For environments in which you know ahead of
time there isn't going to be a metadta api endpoint, this option lets
you opt out from that check to save time.
* Allow iam/sts initialization even if skipping account/cred validation
(#7874)
* Split out skip of IAM validation into credentials and account id
(#7874)
When refreshing remote state, indicate when no state file was found with
an ErrRemoteStateNotFound error. This prevents us from inadvertantly
getting a nil state into a terraform.State where we assume there's
always a root module.
The previous mechanism for testing state threw away the mutation made on
the state by calling State() twice - this commit corrects the test to
match the comment.
In addition, we replace the custom copying logic with the copystructure
library to simplify the code.
This commit makes the current Terraform state version 3 (previously 2),
and a migration process as part of reading v2 state. For the most part
this is unnecessary: helper/schema will deal with upgrading state for
providers written with that framework. However, for providers which
implemented the resource model directly, this gives a best-efforts
attempt at lossless upgrade.
The heuristics used to change the count of a map from the .# key to the
.% key are as follows:
- if the flat map contains any non-numeric keys, we treat it as a
map
- if the map is empty it must be computed or optional, so we remove
it from state
There is a known edge condition: maps with all-numeric keys are
indistinguishable from sets without access to the schema. They will need
manual conversion or may result in spurious diffs.
This commit forward ports the changes made for 0.6.17, in order to store
the type and sensitive flag against outputs.
It also refactors the logic of the import for V0 to V1 state, and
fixes up the call sites of the new format for outputs in V2 state.
Finally we fix up tests which did not previously set a state version
where one is required.
- Fix sensitive outputs for lists and maps
- Fix test prelude which was missed during conflict resolution
- Fix `terraform output` to match old behaviour and not have outputs
header and colouring
- Bump timeout on TestAtlasClient_UnresolvableConflict
This commit adds the groundwork for supporting module outputs of types
other than string. In order to do so, the state version is increased
from 1 to 2 (though the "public-facing" state version is actually as the
first state file was binary).
Tests are added to ensure that V2 (1) state is upgraded to V3 (2) state,
though no separate read path is required since the V2 JSON will
unmarshal correctly into the V3 structure.
Outputs in a ModuleState are now of type map[string]interface{}, and a
test covers round-tripping string, []string and map[string]string, which
should cover all of the types in question.
Type switches have been added where necessary to deal with the
interface{} value, but they currently default to panicking when the input
is not a string.
Adds the `TF_SKIP_REMOTE_TESTS` env var to be used in cases where the
`http.Get()` smoke test passes but the network is not able to service
the needs of the tests.
Fixes#4421
The retryablehttp package implements basic retries w/ exponential
backoff, which helps the remote state push recover in cases of
connectivity blips or transient errors.
Atlas returns an HTTP 409 - Conflict if the pushed state reports the same
Serial number but the checksum of the raw content differs. This can
sometimes happen when Terraform changes state representation internally
between versions in a way that's semantically neutral but affects the JSON
output and therefore the checksum.
Here we detect and handle this situation by ticking the serial and retrying
iff for the previous state and the proposed state:
* the serials match
* the parsed states are Equal (semantically equivalent)
In other words, in this situation Terraform can override Atlas's detected
conflict by asserting that the state it is pushing is indeed correct.
The state is always JSON, in spite of the fact that this interface
presents it as an opaque byte array. It's more helpful to those interacting
with the state object outside of Terraform for it to have a more specific
content-type.
This change allows the user to specify `-backend-config="encrypt=1"`
to tell S3 to encrypt the data that's in the bucket when using S3
for remote config storage.
The encryption uses "Amazon S3-managed encryption keys" so it should
not require any further user intervention.
A line was added to the unit test just for coverage.
The acceptance test was modified to:
a) Use encryption
b) Push some test data up to the bucket created to ensure
that Amazon accepts the header.
/cc @LeftyBC - Hope this helps! Some basic point:
* Idiomatic Go is to use thisCasingStyle and not this_casing_style
* Less repetitive to just create an http.Client once and re-use, also
more flexible for later.
* The empty `if ok {}` was kind of awkward, replace with proper check
Rework devcamcar's OpenStack Swift remote [pull
request](https://github.com/hashicorp/terraform/pull/942) to work with
Terraform's new `state/remote` and Gophercloud's current implementation.
`Get()` changed up a bit from devcamcar's version (using different
Gopercloud functionality resulted in less fussing around to figure out
the error case).
Otherwise this is a transliteration/remix of his ideas.
This landed in aws-sdk-go yesterday, breaking the AWS provider in many places:
3c259c9586
Here, with much sedding, grepping, and manual massaging, we attempt to
catch Terraform up to the new `awserr.Error` interface world.
Sending state over a cleartext protocol is bad in untrusted networks.
Expose `-backend-config="scheme=https"` and wire it through to the
Consul client.
James Bardin