Implementation notes:
* The hash implementation was not considering key value, causing "diffs
did not match" errors when a value was updated. Switching to default
HashResource implementation fixes this
* Using HashResource as a default exposed a bug in helper/schema that
needed to be fixed so the Set function is picked up properly during
Read
* Stop writing back values into the `key` attribute; it triggers extra
diffs when `default` is used. Computed values all just go into `var`.
* Includes a state migration to prevent unnecessary diffs based on
"key" attribute hashcodes changing.
In the tests:
* Switch from leaning on the public demo Consul instance to requiring a
CONSUL_HTTP_ADDR variable be set pointing to a `consul agent -dev`
instance to be used only for testing.
* Add a test that exposes the updating issues and covers the fixes
Fixes#774Fixes#1866Fixes#3023
Changing the Set internals makes a lot of sense as it saves doing
conversions in multiple places and gives a central place to alter
the key when a item is computed.
This will have no side effects other then that the ordering is now
based on strings instead on integers, so the order will be different.
This will however have no effect on existing configs as these will
use the individual codes/keys and not the ordering to determine if
there is a diff or not.
Lastly (but I think also most importantly) there is a fix in this PR
that makes diffing sets extremely more performand. Before a full diff
required reading the complete Set for every single parameter/attribute
you wanted to diff, while now it only gets that specific parameter.
We have a use case where we have a Set that has 18 parameters and the
set consist of about 600 items (don't ask 😉). So when doing a diff
it would take 100% CPU of all cores and stay that way for almost an
hour before being able to complete the diff.
Debugging this we learned that for retrieving every single parameter
it made over 52.000 calls to `func (c *ResourceConfig) get(..)`. In
this function a slice is created and used only for the duration of the
call, so the time needed to create all needed slices and on the other
hand the time the garbage collector needed to clean them up again caused
the system to cripple itself. Next to that there are also some expensive
reflect calls in this function which also claimed a fair amount of CPU
time.
After this fix the number of calls needed to get a single parameter
dropped from 52.000+ to only 2! 😃
I promised myself that next time I jumped in this file I'd fix this up.
Now we don't have to manually index the file with comments, we can just
add descriptive names to the test cases!
Previous this would return the following sort of error:
expected type 'string', got unconvertible type '[]interface {}'
This is the raw error returned by the underlying mapstructure library.
This is not a helpful error message for anyone who doesn't know Go's
type system, and it exposes Terraform's internals to the UI.
Instead we'll catch these cases before we try to use mapstructure and
return a more straightforward message.
By checking the type before the IsComputed exception this also avoids
a crash caused when the assigned value is a computed list. Otherwise
the list of interpolations is allowed through here and then crashes later
during Diff when the value is not a primitive as expected.
A common issue with new resource implementations is not considering parts
of a complex structure that's used inside a set, which causes quirky
behavior.
The schema helper has enough information to provide a default reasonable
implementation of a set function that includes all non-computed attributes
in a deterministic way. Here we implement such a function and use it
when no explicit hashing function is provided.
In order to achieve this we encapsulate the construction of the zero
value for a schema in a new method schema.ZeroValue, which allows us to
put the fallback logic to the new default function in a single spot.
It is no longer valid to use &Set{F: schema.Set} and all uses of that
construct should be replaced with schema.ZeroValue().(*Set) .
We need to set the value to an empty value so the state file does
indeed change the value. Otherwise the obsolote value is still
intact and doesn't get changed at all. This means `terraform show`
still shows the obsolote value when the particular value is not
existing anymore. This is due the AWS API which is returning a null
instead of an empty string.
This was just a missed exit from the resource.Apply function -
subsequent refreshes would add the SchemaVersion back into the state,
but having the state recorded once without the meta information can
cause problems with Atlas's remote state checksumming.
By prefixing them with `cmd /c` it will work with both `winner` and
`ssh` connection types.
This PR also reverts some bad stringer changes made in PR #2673
In `helper/schema` we already makes a distinction between `Default`
which is always applied and `InputDefault` which is displayed to the
user for an empty field.
But for variables we just have `Default` which is treated like
`InputDefault`. This changes it to _not_ prompt the user for a value
when the variable declaration includes a default.
Treating this as a UX bugfix and the "don't prompt for variables w/
defaults set" behavior as the originally expected behavior we were
failing to honor.
Added an already-passing test to verify and cover the `helper/schema`
behavior.
Perhaps down the road we can add a `input_default` attribute to
variables to allow similar behavior to `helper/schema` in variables, but
for now just sticking with the fix.
Fixes#2592
This is the initial pure "all tests passing without a diff" stage. The
plan is to change the internal representation of StringList to include a
suffix delimiter, which will allow us to recognize empty and
single-element lists.
I couldn't see a simple path get this working for Maps, Sets,
and Lists, so lets land it as a primitive-only schema feature.
I think validation on primitives comprises 80% of the use cases anyways.
Guarantees that the `interface{}` arg to ValidateFunc is the proper
type, allowing implementations to be simpler.
Finish the docstring on `ValidateFunc` to call this out.
/cc @mitchellh
The runtime impl of ConfictsWith uses Resource.Get(), which makes it
work with any other attribute of the resource - the InternalValidate was
only checking against the local schemaMap though, preventing subResource
from using ConflictsWith properly.
It's a lot of wiring and it's a bit ugly, but it's not runtime code, so
I'm a bit less concerned about that aspect.
This should take care of the problem mentioned in #1909
Other than the fact that "The the" doesn't really make any sense anywhere
that it's used in Terraform, they're a post-punk band from the UK.
Fixes "The The" so that they can get back to playing songs.
This fixes some perpetual diffs I saw in Atlas AccTests where an empty
map (`map[string]interface{}{}`) was being `d.Set` for "metadata_full".
Because the MapFieldWriter was not distinguishing between empty and nil,
this trigger the "map delete" logic and no count was written to the
state. This caused subsequent plans to improperly report a diff.
Here we redefine the map delete functionality to explicitly trigger only
on `nil`, so we catch the `.#` field for empty maps.
- Users
- Groups
- Roles
- Inline policies for the above three
- Instance profiles
- Managed policies
- Access keys
This is most of the data types provided by IAM. There are a few things
missing, but the functionality here is probably sufficient for 95% of
the cases. Makes a dent in #28.
If a given resource does not define an `Update` function, then all of
its attributes must be specified as `ForceNew`, lest Applys fail with
"doesn't support update" like #1367.
This is something we can detect automatically, so this adds a check for
it when we validate provider implementations.
We were previously only recording the schema version on refresh. This
caused the state to be incorrectly written after a `terraform apply`
causing subsequent commands to run the state through an unnecessary
migration.
Providers get a per-resource SchemaVersion integer that they can bump
when a resource's schema changes format. Each InstanceState with an
older recorded SchemaVersion than the cureent one is yielded to a
`MigrateSchema` function to be transformed such that it can be addressed
by the current version of the resource's Schema.
Removed fields show a customizable error message to the user when they
are used in a Terraform config. This is a tool that provider authors can
use for user feedback as they evolve their Schemas.
refs #957
Deprecated fields show a customizable warning message to the user when
they are used in a Terraform config. This is a tool that provider
authors can use for user feedback as they evolve their Schemas.
fixes#957
Now that readMap filters out '#' fields, when maps are nested in sets,
we exposed a related bug where a set was iterating over nested maps and
expected the '#' key to be present in those nested maps.
By skipping _all_ count fields when iterating over set keys, all is
right with the world again.
An `InstanceDiff` will include `ResourceAttrDiff` entries for the
"length" / `#` field of maps. This makes sense, since for something like
`terraform plan` it's useful to see when counts are changing.
The `DiffFieldReader` was not taking these entries into account when
reading maps out, and was therefore incorrectly returning maps that
included an extra `'#'` field, which was causing all sorts of havoc
for providers (extra tags on AWS instances, broken google compute
instance launch, possibly others).
* fixes#914 - extra tags on AWS instances
* fixes#883 - general core issue sprouted from #757
* removes the hack+TODO from #757
We were waiting until the higher-level (m schemaMap) diffString method
to apply defaults, which was messing with set hashcode evaluation for
cases when a field with a default is included in the hash function.
fixes#824
This was actually quite nasty as the first bug covered the second one…
The first bug is with HasChange. This function uses reflect.DeepEqual
to check if two instances are the same/have the same content. This
works fine for all types except for Set’s as they contain a function.
And reflect.DeepEqual will only say the functions are equal if they are
both nil (which they aren’t in a Set). So in effect it means that
currently HasChange will always say true for Set’s, even when they are
actually being equal.
As soon as you fix this problem, you will notice the second one (which
the added test is written for). Without saying you want the exact diff,
you will end up with a merged value which will (in most cases) be the
same.
Run all unit tests and a good part of the acc tests to verify this
works as expected and all look good.
Currently the `sync.Once` call is only used to init a Set in the add()
func. So when you add a value to a Set that is the result of one of the
Set operations (i.e. union, difference, intersect) the Set will be
reinitialised and the exiting values will be lost.
I don’t have a clue why this is showing up in my ACC tests just now, as
this code is in there for quite some time already. Somehow it seems to
have something to do with the refactoring of the helper/schema done
last week, as I cannot reproduce this with
47f02f80bc
/cc @svanharmelen - I think some logic changed after my refactor. I now
return Exists: true when Computed: true but the value might be blank to
note that the FieldReader FOUND a value, its just unknown. I think
before it didn't do that so the logic for GetOk has to be "does it exist
and is it _not_ computed"
Seems weird because I just realized there is no way to get the OLD value
of something if it is being computed now, but I looked and there are
tests that verify this and they're like... test #5 of Get. So, they're
not new meaning that must've been expected behavior? Hm. Let me know if
you find any other issues from acceptance tests
/cc @phinze - This is pretty straightforward, almost magically so. The
reason this works is because in `diffString` we use mapstructure[1] with
"weak decode mode" to just be responisble for turning anything into a
string.
[1]: https://github.com/mitchellh/mapstructure
Don't check if the root key is being computed for composite types.
Instead, continue recursing the composite type in order to check if
the sub-key, key.N, for each individual element is being computed.
Fixes a panic which occurs when validating a composite type where
the value is an unknown kind for the schema.
Paul Hinze