ResiLien
/
terraform
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
terraform/states/remote/state_test.go

598 lines
18 KiB
Go
Raw Normal View History

state: a bunch of state stuff
2015-02-21 20:52:55 +01:00
package remote
import (
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
"log"
add mutexes to remote.State
2017-05-25 17:01:25 +02:00
"sync"
state: a bunch of state stuff
2015-02-21 20:52:55 +01:00
"testing"
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
"github.com/google/go-cmp/cmp"
"github.com/zclconf/go-cty/cty"
"github.com/hashicorp/terraform/states"
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
"github.com/hashicorp/terraform/states/statefile"
state/remote: Switch to statemgr interfaces in test These statemgr interfaces are the new names for the older interfaces in the "state" package. These types alias each other so it doesn't really matter which we use, but the "state" package is deprecated and we intend to eventually remove it, so this is a further step in that direction.
2019-06-20 02:30:34 +02:00
"github.com/hashicorp/terraform/states/statemgr"
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
"github.com/hashicorp/terraform/version"
state: a bunch of state stuff
2015-02-21 20:52:55 +01:00
)
func TestState_impl(t *testing.T) {
state/remote: Switch to statemgr interfaces in test These statemgr interfaces are the new names for the older interfaces in the "state" package. These types alias each other so it doesn't really matter which we use, but the "state" package is deprecated and we intend to eventually remove it, so this is a further step in that direction.
2019-06-20 02:30:34 +02:00
var _ statemgr.Reader = new(State)
var _ statemgr.Writer = new(State)
var _ statemgr.Persister = new(State)
var _ statemgr.Refresher = new(State)
var _ statemgr.Locker = new(State)
state: a bunch of state stuff
2015-02-21 20:52:55 +01:00
}
add mutexes to remote.State
2017-05-25 17:01:25 +02:00
func TestStateRace(t *testing.T) {
s := &State{
Client: nilClient{},
}
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
current := states.NewState()
add mutexes to remote.State
2017-05-25 17:01:25 +02:00
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
defer wg.Done()
s.WriteState(current)
s.PersistState()
s.RefreshState()
}()
}
wg.Wait()
}
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// testCase encapsulates a test state test
type testCase struct {
name string
// A function to mutate state and return a cleanup function
mutationFunc func(*State) (*states.State, func())
// The expected request to have taken place
expectedRequest mockClientRequest
// Mark this case as not having a request
noRequest bool
}
// isRequested ensures a test that is specified as not having
// a request doesn't have one by checking if a method exists
// on the expectedRequest.
func (tc testCase) isRequested(t *testing.T) bool {
hasMethod := tc.expectedRequest.Method != ""
if tc.noRequest && hasMethod {
t.Fatalf("expected no content for %q but got: %v", tc.name, tc.expectedRequest)
}
return !tc.noRequest
}
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
func TestStatePersist(t *testing.T) {
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
testCases := []testCase{
// Refreshing state before we run the test loop causes a GET
{
name: "refresh state",
mutationFunc: func(mgr *State) (*states.State, func()) {
return mgr.State(), func() {}
},
expectedRequest: mockClientRequest{
Method: "Get",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "mock-lineage",
"serial": 1.0, // encoding/json decodes this as float64 by default
"terraform_version": "0.0.0",
"outputs": map[string]interface{}{},
"resources": []interface{}{},
},
},
},
{
name: "change lineage",
mutationFunc: func(mgr *State) (*states.State, func()) {
originalLineage := mgr.lineage
mgr.lineage = "some-new-lineage"
return mgr.State(), func() {
mgr.lineage = originalLineage
}
},
expectedRequest: mockClientRequest{
Method: "Put",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "some-new-lineage",
"serial": 2.0, // encoding/json decodes this as float64 by default
"terraform_version": version.Version,
"outputs": map[string]interface{}{},
"resources": []interface{}{},
},
},
},
{
name: "change serial",
mutationFunc: func(mgr *State) (*states.State, func()) {
originalSerial := mgr.serial
mgr.serial++
return mgr.State(), func() {
mgr.serial = originalSerial
}
},
expectedRequest: mockClientRequest{
Method: "Put",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "mock-lineage",
"serial": 4.0, // encoding/json decodes this as float64 by default
"terraform_version": version.Version,
"outputs": map[string]interface{}{},
"resources": []interface{}{},
},
},
},
{
name: "add output to state",
mutationFunc: func(mgr *State) (*states.State, func()) {
s := mgr.State()
s.RootModule().SetOutputValue("foo", cty.StringVal("bar"), false)
return s, func() {}
},
expectedRequest: mockClientRequest{
Method: "Put",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "mock-lineage",
"serial": 3.0, // encoding/json decodes this as float64 by default
"terraform_version": version.Version,
"outputs": map[string]interface{}{
"foo": map[string]interface{}{
"type": "string",
"value": "bar",
},
},
"resources": []interface{}{},
},
},
},
{
name: "mutate state bar -> baz",
mutationFunc: func(mgr *State) (*states.State, func()) {
s := mgr.State()
s.RootModule().SetOutputValue("foo", cty.StringVal("baz"), false)
return s, func() {}
},
expectedRequest: mockClientRequest{
Method: "Put",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "mock-lineage",
"serial": 4.0, // encoding/json decodes this as float64 by default
"terraform_version": version.Version,
"outputs": map[string]interface{}{
"foo": map[string]interface{}{
"type": "string",
"value": "baz",
},
},
"resources": []interface{}{},
},
},
},
{
name: "nothing changed",
mutationFunc: func(mgr *State) (*states.State, func()) {
s := mgr.State()
return s, func() {}
},
noRequest: true,
},
{
name: "reset serial (force push style)",
mutationFunc: func(mgr *State) (*states.State, func()) {
mgr.serial = 2
return mgr.State(), func() {}
},
expectedRequest: mockClientRequest{
Method: "Put",
Content: map[string]interface{}{
"version": 4.0, // encoding/json decodes this as float64 by default
"lineage": "mock-lineage",
"serial": 3.0, // encoding/json decodes this as float64 by default
"terraform_version": version.Version,
"outputs": map[string]interface{}{
"foo": map[string]interface{}{
"type": "string",
"value": "baz",
},
},
"resources": []interface{}{},
},
},
},
}
// Initial setup of state just to give us a fixed starting point for our
// test assertions below, or else we'd need to deal with
// random lineage.
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
mgr := &State{
Client: &mockClient{
current: []byte(`
{
"version": 4,
"lineage": "mock-lineage",
"serial": 1,
"terraform_version":"0.0.0",
"outputs": {},
"resources": []
}
`),
},
}
// In normal use (during a Terraform operation) we always refresh and read
// before any writes would happen, so we'll mimic that here for realism.
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// NB This causes a GET to be logged so the first item in the test cases
// must account for this
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
if err := mgr.RefreshState(); err != nil {
t.Fatalf("failed to RefreshState: %s", err)
}
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// Our client is a mockClient which has a log we
// use to check that operations generate expected requests
mockClient := mgr.Client.(*mockClient)
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// logIdx tracks the current index of the log separate from
// the loop iteration so we can check operations that don't
// cause any requests to be generated
logIdx := 0
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// Run tests in order.
for _, tc := range testCases {
s, cleanup := tc.mutationFunc(mgr)
Add remote state test for serial and lineage changes We only persist a new state if the actual state contents have changed. This test demonstrates that behavior by calling write and persist methods when either the lineage or serial have changed.
2020-05-04 17:45:36 +02:00
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
if err := mgr.WriteState(s); err != nil {
t.Fatalf("failed to WriteState for %q: %s", tc.name, err)
}
if err := mgr.PersistState(); err != nil {
t.Fatalf("failed to PersistState for %q: %s", tc.name, err)
}
if tc.isRequested(t) {
// Get captured request from the mock client log
// based on the index of the current test
if logIdx >= len(mockClient.log) {
t.Fatalf("request lock and index are out of sync on %q: idx=%d len=%d", tc.name, logIdx, len(mockClient.log))
}
loggedRequest := mockClient.log[logIdx]
logIdx++
if diff := cmp.Diff(tc.expectedRequest, loggedRequest); len(diff) > 0 {
t.Fatalf("incorrect client requests for %q:\n%s", tc.name, diff)
}
}
cleanup()
Add remote state test for serial and lineage changes We only persist a new state if the actual state contents have changed. This test demonstrates that behavior by calling write and persist methods when either the lineage or serial have changed.
2020-05-04 17:45:36 +02:00
}
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
logCnt := len(mockClient.log)
if logIdx != logCnt {
log.Fatalf("not all requests were read. Expected logIdx to be %d but got %d", logCnt, logIdx)
Add remote state test for serial and lineage changes We only persist a new state if the actual state contents have changed. This test demonstrates that behavior by calling write and persist methods when either the lineage or serial have changed.
2020-05-04 17:45:36 +02:00
}
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
}
Add remote state test for serial and lineage changes We only persist a new state if the actual state contents have changed. This test demonstrates that behavior by calling write and persist methods when either the lineage or serial have changed.
2020-05-04 17:45:36 +02:00
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
type migrationTestCase struct {
name string
// A function to generate a statefile
stateFile func(*State) *statefile.File
// The expected request to have taken place
expectedRequest mockClientRequest
// Mark this case as not having a request
expectedError string
// force flag passed to client
force bool
}
func TestWriteStateForMigration(t *testing.T) {
mgr := &State{
Client: &mockClient{
current: []byte(`
{
"version": 4,
"lineage": "mock-lineage",
"serial": 3,
"terraform_version":"0.0.0",
"outputs": {"foo": {"value":"bar", "type": "string"}},
"resources": []
}
`),
},
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
}
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
testCases := []migrationTestCase{
// Refreshing state before we run the test loop causes a GET
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
{
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
name: "refresh state",
stateFile: func(mgr *State) *statefile.File {
return mgr.StateForMigration()
},
expectedRequest: mockClientRequest{
Method: "Get",
Content: map[string]interface{}{
"version": 4.0,
"lineage": "mock-lineage",
"serial": 3.0,
"terraform_version": "0.0.0",
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
},
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
},
},
{
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
name: "cannot import lesser serial without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, mgr.lineage, 1)
},
expectedError: "cannot import state with serial 1 over newer state with serial 3",
},
{
name: "cannot import differing lineage without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, "different-lineage", mgr.serial)
},
expectedError: `cannot import state with lineage "different-lineage" over unrelated state with lineage "mock-lineage"`,
},
{
name: "can import lesser serial with force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, mgr.lineage, 1)
},
expectedRequest: mockClientRequest{
Fix bug for force push for backends besides the remote backend In refactoring the force push code when implementing force push for the Terraform remote backend, a bug was introduced that meant that backends that don't implement the EnableForcePush method would still have their state validated. This commit fixes that, and adds test coverage such that there is a separate mockRemoteClient that has this method implemented.
2020-09-09 21:25:23 +02:00
Method: "Put",
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
Content: map[string]interface{}{
"version": 4.0,
"lineage": "mock-lineage",
"serial": 2.0,
"terraform_version": version.Version,
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
},
},
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
force: true,
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
},
{
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
name: "cannot import differing lineage without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, "different-lineage", mgr.serial)
},
expectedRequest: mockClientRequest{
Fix bug for force push for backends besides the remote backend In refactoring the force push code when implementing force push for the Terraform remote backend, a bug was introduced that meant that backends that don't implement the EnableForcePush method would still have their state validated. This commit fixes that, and adds test coverage such that there is a separate mockRemoteClient that has this method implemented.
2020-09-09 21:25:23 +02:00
Method: "Put",
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
Content: map[string]interface{}{
"version": 4.0,
"lineage": "different-lineage",
"serial": 3.0,
"terraform_version": version.Version,
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
},
},
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
force: true,
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
},
}
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
// In normal use (during a Terraform operation) we always refresh and read
// before any writes would happen, so we'll mimic that here for realism.
// NB This causes a GET to be logged so the first item in the test cases
// must account for this
if err := mgr.RefreshState(); err != nil {
t.Fatalf("failed to RefreshState: %s", err)
}
if err := mgr.WriteState(mgr.State()); err != nil {
t.Fatalf("failed to write initial state: %s", err)
}
// Our client is a mockClient which has a log we
// use to check that operations generate expected requests
mockClient := mgr.Client.(*mockClient)
Fix bug for force push for backends besides the remote backend In refactoring the force push code when implementing force push for the Terraform remote backend, a bug was introduced that meant that backends that don't implement the EnableForcePush method would still have their state validated. This commit fixes that, and adds test coverage such that there is a separate mockRemoteClient that has this method implemented.
2020-09-09 21:25:23 +02:00
// logIdx tracks the current index of the log separate from
// the loop iteration so we can check operations that don't
// cause any requests to be generated
logIdx := 0
for _, tc := range testCases {
sf := tc.stateFile(mgr)
err := mgr.WriteStateForMigration(sf, tc.force)
shouldError := tc.expectedError != ""
// If we are expecting and error check it and move on
if shouldError {
if err == nil {
t.Fatalf("test case %q should have failed with error %q", tc.name, tc.expectedError)
} else if err.Error() != tc.expectedError {
t.Fatalf("test case %q expected error %q but got %q", tc.name, tc.expectedError, err)
}
continue
}
if err != nil {
t.Fatalf("test case %q failed: %v", tc.name, err)
}
// At this point we should just do a normal write and persist
// as would happen from the CLI
mgr.WriteState(mgr.State())
mgr.PersistState()
if logIdx >= len(mockClient.log) {
t.Fatalf("request lock and index are out of sync on %q: idx=%d len=%d", tc.name, logIdx, len(mockClient.log))
}
loggedRequest := mockClient.log[logIdx]
logIdx++
if diff := cmp.Diff(tc.expectedRequest, loggedRequest); len(diff) > 0 {
t.Fatalf("incorrect client requests for %q:\n%s", tc.name, diff)
}
}
logCnt := len(mockClient.log)
if logIdx != logCnt {
log.Fatalf("not all requests were read. Expected logIdx to be %d but got %d", logCnt, logIdx)
}
}
// This test runs the same test cases as above, but with
// a client that implements EnableForcePush -- this allows
// us to test that -force continues to work for backends without
// this interface, but that this interface works for those that do.
func TestWriteStateForMigrationWithForcePushClient(t *testing.T) {
mgr := &State{
Client: &mockClientForcePusher{
current: []byte(`
{
"version": 4,
"lineage": "mock-lineage",
"serial": 3,
"terraform_version":"0.0.0",
"outputs": {"foo": {"value":"bar", "type": "string"}},
"resources": []
}
`),
},
}
testCases := []migrationTestCase{
// Refreshing state before we run the test loop causes a GET
{
name: "refresh state",
stateFile: func(mgr *State) *statefile.File {
return mgr.StateForMigration()
},
expectedRequest: mockClientRequest{
Method: "Get",
Content: map[string]interface{}{
"version": 4.0,
"lineage": "mock-lineage",
"serial": 3.0,
"terraform_version": "0.0.0",
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
},
},
},
{
name: "cannot import lesser serial without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, mgr.lineage, 1)
},
expectedError: "cannot import state with serial 1 over newer state with serial 3",
},
{
name: "cannot import differing lineage without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, "different-lineage", mgr.serial)
},
expectedError: `cannot import state with lineage "different-lineage" over unrelated state with lineage "mock-lineage"`,
},
{
name: "can import lesser serial with force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, mgr.lineage, 1)
},
expectedRequest: mockClientRequest{
Method: "Force Put",
Content: map[string]interface{}{
"version": 4.0,
"lineage": "mock-lineage",
"serial": 2.0,
"terraform_version": version.Version,
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
},
},
force: true,
},
{
name: "cannot import differing lineage without force",
stateFile: func(mgr *State) *statefile.File {
return statefile.New(mgr.state, "different-lineage", mgr.serial)
},
expectedRequest: mockClientRequest{
Method: "Force Put",
Content: map[string]interface{}{
"version": 4.0,
"lineage": "different-lineage",
"serial": 3.0,
"terraform_version": version.Version,
"outputs": map[string]interface{}{"foo": map[string]interface{}{"type": string("string"), "value": string("bar")}},
"resources": []interface{}{},
},
},
force: true,
},
}
// In normal use (during a Terraform operation) we always refresh and read
// before any writes would happen, so we'll mimic that here for realism.
// NB This causes a GET to be logged so the first item in the test cases
// must account for this
if err := mgr.RefreshState(); err != nil {
t.Fatalf("failed to RefreshState: %s", err)
}
if err := mgr.WriteState(mgr.State()); err != nil {
t.Fatalf("failed to write initial state: %s", err)
}
// Our client is a mockClientForcePusher which has a log we
// use to check that operations generate expected requests
mockClient := mgr.Client.(*mockClientForcePusher)
Add support for force pushing with the remote backend Both differing serials and lineage protections should be bypassed with the -force flag (in addition to resources). Compared to other backends we aren’t just shipping over the state bytes in a simple payload during the persistence phase of the push command and the force flag added to the Go TFE client needs to be specified at that time. To prevent changing every method signature of PersistState of the remote client I added an optional interface that provides a hook to flag the Client as operating in a force push context. Changing the method signature would be more explicit at the cost of not being used anywhere else currently or the optional interface pattern could be applied to the state itself so it could be upgraded to support PersistState(force bool) only when needed. Prior to this only the resources of the state were checked for changes not the lineage or the serial. To bring this in line with documented behavior noted above those attributes also have a “read” counterpart just like state has. These are now checked along with state to determine if the state as a whole is unchanged. Tests were altered to table driven test format and testing was expanded to include WriteStateForMigration and its interaction with a ClientForcePusher type.
2020-04-15 22:54:03 +02:00
if mockClient.force {
t.Fatalf("client should not default to force")
}
// logIdx tracks the current index of the log separate from
// the loop iteration so we can check operations that don't
// cause any requests to be generated
logIdx := 0
for _, tc := range testCases {
// Always reset client to not be force pushing
mockClient.force = false
sf := tc.stateFile(mgr)
err := mgr.WriteStateForMigration(sf, tc.force)
shouldError := tc.expectedError != ""
// If we are expecting and error check it and move on
if shouldError {
if err == nil {
t.Fatalf("test case %q should have failed with error %q", tc.name, tc.expectedError)
} else if err.Error() != tc.expectedError {
t.Fatalf("test case %q expected error %q but got %q", tc.name, tc.expectedError, err)
}
continue
}
if err != nil {
t.Fatalf("test case %q failed: %v", tc.name, err)
}
if tc.force && !mockClient.force {
t.Fatalf("test case %q should have enabled force push", tc.name)
}
// At this point we should just do a normal write and persist
// as would happen from the CLI
mgr.WriteState(mgr.State())
mgr.PersistState()
if logIdx >= len(mockClient.log) {
t.Fatalf("request lock and index are out of sync on %q: idx=%d len=%d", tc.name, logIdx, len(mockClient.log))
}
loggedRequest := mockClient.log[logIdx]
logIdx++
if diff := cmp.Diff(tc.expectedRequest, loggedRequest); len(diff) > 0 {
t.Fatalf("incorrect client requests for %q:\n%s", tc.name, diff)
}
}
logCnt := len(mockClient.log)
if logIdx != logCnt {
log.Fatalf("not all requests were read. Expected logIdx to be %d but got %d", logCnt, logIdx)
state/remote: Don't persist snapshot for unchanged state Previously we would write to the backend for every call to PersistState, even if nothing changed since the last write, but update the serial only if the state had changed. The Terraform Cloud & Enterprise state storage have a simple safety check that any future write with an already-used lineage and serial must be byte-for-byte identical. StatesMarshalEqual is intended to detect that, but it only actually detects changes the state itself, and not changes to the snapshot metadata. Because we write the current Terraform version into the snapshot metadata during serialization, we'd previously have an issue where if the first state write after upgrading Terraform to a new version happened to change nothing about the state content then we'd write a new snapshot that differed only by Terraform version, and Terraform Cloud/Enterprise would then reject it. The snapshot header is discarded immediately after decoding, so we can't use information from it when deciding whether to increment the serial. The next best thing is to skip sending no-op snapshot updates to the state client in the first place. These writes are unnecessary anyway, and state storage owners have asked us in the past to elide these to avoid generating noise in their version logs, so we'll also finally meet those requests as a nice side-effect of this change. We didn't previously have tests for the full flow of retrieving and then successively updating persisted state snapshots, so this includes a test which covers that logic and includes an assertion that a no-op update does not get written to the state client.
2019-06-20 02:42:09 +02:00
}
}