Commit Graph

298 Commits

Author SHA1 Message Date
Martin Atkins 91def7b6f6 core: Better failure output for TestContext2Apply_moduleDestroyOrder 2018-10-16 18:49:20 -07:00
Martin Atkins 38f37af7e9 core: remove TestContext2Apply_outputInvalid
This was testing that returning a number as an output would be reported as
an error, but that is intentionally allowed now.
2018-10-16 18:49:20 -07:00
Martin Atkins b99b31ebea core: Correct schema for TestContext2Apply_issue5254
This test is now failing due to the fact that WritePlan is currently
disabled pending a rewrite. This will be addressed in a subsequent commit.
2018-10-16 18:49:20 -07:00
James Bardin df2907abb9 core: TestContext2Apply_dataDependsOn 2018-10-16 18:49:20 -07:00
Martin Atkins d961b1de1b core: Stop loading provider schema during graph walk
We now fetch all of the necessary schemas during context creation, so we
can just thread that repository of schemas through into EvalContext and
Evaluator and access the schemas as needed without any further fetching.

This requires updating a few tests to have a valid Provider address in
their state objects, because we need that in order to trigger the loading
of the relevant schema.
2018-10-16 18:49:20 -07:00
Martin Atkins aa9d88ad3c core: Schema for TestContext2Apply_multiProviderDestroy
This test depends on having a correct schema, so we'll specify the minimum
schema for its fixture inline here rather than using the superset schema
returned by testProvider.
2018-10-16 18:49:20 -07:00
James Bardin aa07f34dbe core: TestContext2Apply_createBeforeDestroy_hook
The instnace info ID has changed, and no longer reflects the type of
node. Record the state to determine apply order for this test.
2018-10-16 18:49:20 -07:00
Martin Atkins 26f76dd222 core: When planning to destroy an orphan instance, nil it in state
While we're planning we must always update the state with the proposed new
data resulting from the plan. In this case, we must record that the
orphan instance doesn't exist at all in the proposed new state by storing
its state as nil.

This in turn allows references to the containing resource to evaluate
properly, using the new updated resource count. This fixes
TestContext2Apply_multiVarCountDec.

This also includes a number of changes to the test output of
TestContext2Apply_multiVarCountDec that make it easier to debug failures.
2018-10-16 18:49:20 -07:00
James Bardin 65cd163365 core: provider alias inheritance
An aliased provider should not be automatically inherited, nor
implicitly instantiated in a module. This test should not have
previously passed.

Add a proxy provider block to the module and update the provider to
match the schema.
2018-10-16 18:49:20 -07:00
James Bardin 46ba98233a core: change more "count" variables to "num" 2018-10-16 18:49:20 -07:00
Martin Atkins bba9660dd4 core: context apply test output distinguish plan vs. apply errors 2018-10-16 18:49:20 -07:00
Martin Atkins b9ed280182 core: Fix TestContext2Apply_destroyComputed
The Provider field in ResourceState is now required, whereas before it
could be omitted and have Terraform try to discover a fitting provider
configuration automatically.

The automatic behavior was a compatibility shim added in v0.11 to support
states from prior versions without an explicit migration, but for v0.12
we will have a migration to our new state format anyway and so we will
fix this up during that migration pass.
2018-10-16 18:49:20 -07:00
Martin Atkins 3e64311dc2 core: Update TestContext2Apply_multiVarComprehensive for new assumptions
This comprehensive test was covering a few different behaviors that are
intentionally different for v0.12:

- Applying the splat operator to a list of resource instances that haven't
  been created yet produces a list of unknown values rather than a single
  unknown list as before. This is important because it allows that list
  to be passed into length().

- Wrapping a splat expression in another round of brackets now produces
  a list of lists, whereas before we had a special case (for compatibility
  with prior to v0.10) that would flatten this away in the schema layer.
2018-10-16 18:49:20 -07:00
James Bardin 90893eb3f9 core: update test count vars to num 2018-10-16 18:49:20 -07:00
James Bardin 7edff454f2 core: update test fixtures 2018-10-16 18:49:20 -07:00
Martin Atkins c5bd659a0f core: Make context apply tests produce more useful failure output 2018-10-16 18:49:20 -07:00
James Bardin 2f6787c9b2 core: update provisioner test fixtures
They need to conform to the test schema
2018-10-16 18:49:20 -07:00
James Bardin 1350ed8e3a core: update fixture to match schema 2018-10-16 18:49:20 -07:00
Martin Atkins aedbbc6207 core: Fix TestContext2Apply_outputOrphanModule
The adaptation of ModuleState.RemovedOutputs for the new config types
was incorrect because it took the absence of any output map as "nothing to
do", rather than "everything has been removed" as expected.

Now it treats a nil map like an empty map, detecting _all_ of the outputs
as having been removed if the output map is nil.
2018-10-16 18:49:20 -07:00
James Bardin d9239e1f48 use provisioner attr defined in schema 2018-10-16 18:49:20 -07:00
James Bardin 41dde5fafc missing attribute now caught after apply 2018-10-16 18:49:20 -07:00
James Bardin 36aca84572 update ConnInfo test to work with new schema
We can't handle dynamic attributes in the tests any longer
2018-10-16 18:49:20 -07:00
Martin Atkins 05bf08b2ec core: Remove context apply test for map variable merging
We no longer have this merge behavior, because it is inconsistent with how
variables behave in all other contexts and similar behavior can now be
achieved by merging the user's input with a predefined map in a local
value expression.
2018-10-16 18:49:20 -07:00
Martin Atkins 687830a5d5 core: Mock schema for TestContext2Apply_multiProviderDestroy
This test seems to have been buggy before our current work, with the test
fixture containing a reference to a resource that doesn't exist.

This both fixes the fixture and adds a mock schema for it, though this
just revealed another error which isn't fixed here, where the a_ids value
seems to come through as unknown after apply. That will be fixed in a
subsequent commit.
2018-10-16 18:49:20 -07:00
Martin Atkins cdb32390b9 core: Add schema mocks to more of the context apply tests 2018-10-16 18:49:20 -07:00
Martin Atkins 5cf06e86c6 core: Remove TestContext2Apply_provisionerMultiSelfRefSingle
We no longer support using "self.count" in a provisioner to access the
resolved count meta-argument value of the associated resource.

This was only possible before because of a special exception in how
Terraform resolved variables, and in new HCL that exception isn't possible
because resource instances are real values in the scope and we don't want
to add this implied "count" attribute to all of them.

"count" is a property of the resource config rather than of the resource
instances, and since "self" is a resource _instance_ it doesn't make sense
to expose it there.

There is no replacement for this feature. In the rare case where it is
needed, the user must factor the count out into a named local value and
refer to that both in the count meta-argument and in the provisioner.
2018-10-16 18:49:20 -07:00
Martin Atkins 1caffba0e6 core: update various context apply tests for schema/fixtures
Most of these changes are just adding schema to describe the expectations
of the existing test fixtures. However, some of them require the fixtures
themselves to be changed due to changing assumptions in the language.
2018-10-16 18:49:20 -07:00
Martin Atkins 8774c857f6 core: "computed ref type mismatch" test is now a plan test
This was previously an apply-time failure due to our inability to
type-check unknowns in 0.11, but we now retain type information for
unknown values and so this check now fails during plan instead.
2018-10-16 18:49:20 -07:00
Martin Atkins 02932b5a50 core: Schema for TestContext2Apply_outputDiffVars
The fixtures for this test assume some atypical arguments to the resources
and also need a provisioner schema.

This doesn't actually fix the test, but by fixing the schema/fixture this
exposes a problem that seems to exist in the main code, which will be
fixed in a subsequent commit.
2018-10-16 18:49:20 -07:00
James Bardin 8d062fc577 update apply test fixtures 2018-10-16 18:49:20 -07:00
James Bardin c6787d0266 update test fixtures
Update test fixtures to work in our new world.
This is mostly changing out attribute names for those in the schema,
adding Providers to states, and updating the test-fixture
configurations.
2018-10-16 18:49:20 -07:00
James Bardin cae5c2feaa update apply tests 2018-10-16 18:48:28 -07:00
James Bardin 07042a95fa fixing context apply tests 2018-10-16 18:48:28 -07:00
Martin Atkins fc7453c073 core: Update tests that use the apply-vars fixture
These now need to include a suitable schema in order to properly process
the configuration.
2018-10-16 18:48:28 -07:00
Martin Atkins 4a21b763aa core: Get tests compiling again
After the refactoring to integrate HCL2 many of the tests were no longer
using correct types, attribute names, etc.

This is a bulk update of all of the tests to make them compile again, with
minimal changes otherwise. Although the tests now compile, many of them
do not yet pass. The tests will be gradually repaired in subsequent
commits, as we continue to complete the refactoring and retrofit work.
2018-10-16 18:46:46 -07:00
James Bardin cfefeec926 walkDestroy is a form of "apply"
When computing the count value, make sure to include walkDestroy with
walkApply, as the former is only a special case of the latter.

When applying a saved plan, the computed count values are lost and we
can no longer query the state for those values. The apply walk was
already considered in the `resourceCountMax` function, but the destroy
walk was not.  This worked when destroying in a single operation
("terraform destroy"), since the state would still be updated with the
latest counts from the plan.
2018-04-10 11:46:29 -04:00
James Bardin 2f8532407a
Merge pull request #17071 from MarkFreebairn/Issue-16360-empty-target-points-to-all-resources-in-state
command: Fix #16360 - Fail fast if -target is specified but empty
2018-04-05 14:19:32 -04:00
James Bardin f25718b065
Merge pull request #17768 from hashicorp/jbardin/unused-destroy-outputs
unused outputs in a destroy should be pruned
2018-04-04 15:20:12 -04:00
James Bardin 620f1985a1 unused outputs in a destroy should be pruned
During a full destroy when outputs are removed, the
NodeDestroyableOutput was preventing it's sibling output from being
destroyed. Prune the output node if it only has its destroy node as a
dependent.

The destroy output test is simply run a second time with no state, which
would cause the output interpolation to fail if it remained in the
graph.
2018-04-03 13:19:04 -04:00
James Bardin 79b948c9cc detect scaled in resources when evaluating *s
If an existing resources is scaled back to 0, locals and outputs will
still have a multi-variable reference to evaluate, which should return
an empty list. Due to how the resource is removed, the resource will
still exist in the state but with no primary instance, which needs to be
ignored in the instance count.
2018-04-03 10:00:45 -04:00
James Bardin 88e911af45 fix a provisionerFail test which was incorrect
The provisionerFail_createBeforeDestroy test was verifying the incorrect
output. The create_before_destroy instance in the state has an ID of
"bar" with require_new="abc", and a new instance would get an ID of
"foo" with require_new="xyz". The existing test was expecting the
following state:

aws_instance.bar: (1 deposed)
  ID = bar
  provider = provider.aws
  require_new = abc
  Deposed ID 1 = foo (tainted)

Which showed "bar" still the primary instance in the state, with the new
instance "foo" as being the deposed instance, though properly tainted.

The new output is:

aws_instance.bar: (tainted) (1 deposed)
  ID = foo
  provider = provider.aws
  require_new = xyz
  type = aws_instance
  Deposed ID 1 = bar

Showing the new "foo instance as being the primary instance in the
state, with "bar" as the deposed instance.
2018-03-15 16:04:05 -04:00
James Bardin f3d1fb3aff failing test for interpolated count from plan
An interpolated count value that is determined during plan, is lost
during plan serialization, causing apply to fail when the interpolation
string can't be evaluated.
2018-03-09 19:04:39 -05:00
James Bardin 7fbc35a36c Make sure outputs are removed when targeting
Similar to NodeApplyableOuptut, NodeDestroyableOutputs also need to stay
in the graph if any ancestor nodes

Use the same GraphNodeTargetDownstream method to keep them from being
pruned, since they are dependent on the output node and all its
descendants.
2018-01-31 13:51:40 -05:00
James Bardin 99867f0082 add PruneUnusedValuesTransformer
Since outputs and local nodes are always evaluated, if the reference a
resource form the configuration that isn't in the state, the
interpolation could fail.

Prune any local or output values that have no references in the graph.
2018-01-30 10:47:17 -05:00
James Bardin 2d138d9917 add a more complex locals test
Using destroy provisioners again for edge cases during destroy.
2018-01-30 10:47:17 -05:00
James Bardin d31fe5ab9d delete outputs during destroy
Now that outputs are always evaluated, we still need a way to remove
them from state when they are destroyed.

Previously, outputs were removed during destroy from the same
"Applyable" node type that evaluates them. Now that we need to possibly
both evaluate and remove output during an apply, we add a new node -
NodeDestroyableOutput.

This new node is added to the graph by the DestroyOutputTransformer,
which make the new destroy node depend on all descendants of the output
node.  This ensures that the output remains in the state as long as
everything which may interpolate the output still exists.
2018-01-29 19:30:04 -05:00
James Bardin 7ac0a46981 add destroy provisioner test with locals, outputs
Add a complex destroy provisioner testcase using locals, outputs and
variables.

Add that pesky "id" attribute to the instance states for interpolation.
2018-01-29 18:01:58 -05:00
James Bardin 7da1a39480 always evaluate locals, even during destroy
Destroy-time provisioners require us to re-evaluate during destroy.

Rather than destroying local values, which doesn't do much since they
aren't persisted to state, we always evaluate them regardless of the
type of apply. Since the destroy-time local node is no longer a
"destroy" operation, the order of evaluation need to be reversed. Take
the existing DestroyValueReferenceTransformer and change it to reverse
the outgoing edges, rather than in incoming edges. This makes it so that
any dependencies of a local or output node are destroyed after
evaluation.

Having locals evaluated during destroy failed one other test, but that
was the odd case where we need `id` to exist as an attribute as well as
a field.
2018-01-29 16:16:41 -05:00
Martin Atkins b511caf049 core: interpolate the count config during the apply walk
Previously we would interpolate the count config (ResourceConfig.RawCount)
only while preparing to dynamic-expand aggregate resource nodes. This is
problematic because we do not dynamic-expand any resource nodes during the
apply walk, and so previously the count value was not available for
interpolation during apply and would result in an error.

Now we interpolate RawCount once for each resource we visit during the
apply walk -- even though that redundantly interpolates the same config
multiple times when count > 1 -- to ensure that it's available by the
time we interpolate any remaining expressions in the config and any
expressions within "connection" and "provisioner" blocks.

This error was masked by us sharing a single RawConfig instance between
the plan and apply walks when "terraform apply" is run with no explicit
plan file argument, but was exposed by the workflow where the plan is
written first to disk since in that case the interpolation result from
during the plan phase is not present in the deflated plan object. For
this reason, the new context test serializes the plan into an in-memory
buffer and reloads it in order to simulate the effect of the two-step
workflow.
2018-01-19 13:06:00 -08:00
Mark Freebairn 1384cf6861 Issue 16360 - Empty -target= points to all resources in state 2018-01-09 21:00:55 +00:00
James Bardin 4b49a323c3 go fmt
slight change to go fmt coming in 0.10
2017-12-26 13:26:38 -05:00
Martin Atkins ba0514106a return tfdiags.Diagnostics from validation methods
Validation is the best time to return detailed diagnostics
to the user since we're much more likely to have source
location information, etc than we are in later operations.

This change doesn't actually add any detail to the messages
yet, but it changes the interface so that we can gradually
introduce more detailed diagnostics over time.

While here there are some minor adjustments to some of the
messages to improve their consistency with terminology we
use elsewhere.
2017-11-28 11:15:29 -08:00
James Bardin 29264df7c0 normalize missing provider names
The provider name coming from ProvidedBy may be resolved if it only
exists in the state. Make sure to strip the module and provider
prefixes for the provider name when adding missing providers.
2017-11-14 15:53:37 -05:00
James Bardin 15ea04af8a remove modules from state
Remove the module entry from the state if a module is no longer in the
configuration. Modules are not removed if there are any existing
resources with the module path as a prefix. The only time this should be
the case is if a module was removed in the config, but the apply didn't
target that module.

Create a NodeModuleRemoved and an associated EvalDeleteModule to track
the module in the graph then remove it from the state. The
NodeModuleRemoved dependencies are simply any other node which contains
the module path as a prefix in its path.

This could have probably been done much easier as a step in pruning the
state, but modules are going to have to be promoted to full graph nodes
anyway in order to support count.
2017-11-08 19:11:53 -05:00
James Bardin 68b07a766a add failing test for orphaned modules outputs
When an entire module is removed from the config, that module's outputs
are not removed from the state.
2017-11-08 14:28:02 -05:00
James Bardin b8b7548614 remove module referencing existing provider 2017-11-07 22:05:52 -05:00
James Bardin 3801ca5bbf test that Refresh updates Provider fields in state 2017-11-07 21:42:30 -05:00
James Bardin c2f3522f7d write providers to state for data resources
And update the test state strings

Destroying with no config is no longer allowed, run an exlpicit destroy
for the destroyOrder test.
2017-11-07 21:05:21 -05:00
James Bardin 8dfaae1f23 rewrite all of the test state string for providers
Now that the resolved provider is always stored in state, we need to
udpate all the test data to match. There will probably be some more
breakage once the provider field is properly diffed.
2017-11-07 13:49:43 -05:00
Martin Atkins 37e276e043 core: test correct behavior of plan+apply with unstable values
We have a few pesky functions that don't act like proper functions and
instead return different values on each call. These are tricky because
we need to make sure we don't trip over ourselves by re-generating these
between plan and apply.

Here we add a context test to verify correct behavior in the presence
of such functions.

There's actually a pre-existing bug which this test caught as originally
written: we re-evaluate the interpolation expressions during apply,
causing these unstable functions to produce new values, and so the
applied value ends up not exactly matching the plan. This is a bug that
needs fixing, but it's been around at least since v0.7.6 (random old
version I tried this with to see) so we'll put it on the list and address
it separately. For now, this part of the test is commented out with a
TODO attached.
2017-11-03 16:11:13 -07:00
James Bardin 241dae7ead resintate disabled tests
Reinstate the disabled tests that required some sort of inheritance
during graph evaluation.
2017-11-02 15:29:51 -04:00
James Bardin a14fd0344c WIP reference providers by full name
This turned out to be a big messy commit, since the way providers are
referenced is tightly coupled throughout the code. That starts to unify
how providers are referenced, using the format output node Name method.

Add a new field to the internal resource data types called
ResolvedProvider. This is set by a new setter method SetProvider when a
resource is connected to a provider during graph creation. This allows
us to later lookup the provider instance a resource is connected to,
without requiring it to have the same module path.

The InitProvider context method now takes 2 arguments, one if the
provider type and the second is the full name of the provider. While the
provider type could still be parsed from the full name, this makes it
more explicit and, and changes to the name format won't effect this
code.
2017-11-02 15:00:06 -04:00
James Bardin db7596c045 use the inherited provider configs in the graph
Use the configured providers directly, rather than looking for inherited
provider configuration during graph evaluation.

First remove the provider config cache, and the associated
SetProviderConfig and ParentProviderConfig methods on the eval context.
Every provider must be configured, so there's no need to look for
configuration from other provider instances.

The config.ProviderConfig struct now has a Scope field which stores the
proper path for the interpolation scope. To get this metadata to the
interpolator, we add an EvalInterpolatProvider node which can carry the
ProviderConfig, and an InterpolateProvider context method to carry the
ProviderConfig.Scope into the InterplationScope.

Some of the tests could be adjusted to account for the new inheritance
behavior, and some were simply no longer valid and will be removed.

The remaining tests have questions on how they should work in practice.
This mostly concerns orphaned modules where there is no longer a way to
obtain a provider. In some cases we may require that a minimal provider
config be present to handle the destroy process, but we need further
testing.

All disabled code was commented out in this commit to record any
additional comments. The following commit will be a cleanup pass.
2017-10-27 09:08:15 -04:00
James Bardin 3ed7b1abec fix the providerWithLocals test
This test was set to fail once this issue was fixed, and now it's fixed.
2017-10-02 16:21:37 -04:00
James Bardin da0e74276a fix provider with local value test and docs
Make sure this fails during destroy

Add a note in the documentation that local values may fail during
destroy.
2017-09-29 17:14:07 -04:00
James Bardin 061597304c add test for destroying with locals in a provider
Verify that locals aren't removed from the state before providers are
evaluated during destroy.
2017-09-28 15:31:41 -04:00
James Bardin 77396107c4 don't evaluate locals during destroy
Locals don't need to be evaluated during destroy.  Rather than simply
skipping them, remove them from the state as they are encountered. Even
though they are not persisted in the state, it keeps the state up to
date as the destroy happens, and we reduce the chance of other
inconstancies later on.
2017-09-28 12:56:25 -04:00
James Bardin 9d8ab55658 Add failing test for destroy with locals 2017-09-28 11:06:37 -04:00
Martin Atkins 892f60efe0 core: test that we skip hooks for data source destroy
Data source destroy is an implementation detail and not something that
external callers should see or expect.
2017-09-01 17:55:05 -07:00
Martin Atkins 3a30bfe845 core: evaluate locals and return them for interpolation
We stash the locals in the module state in a map that is ignored for JSON
serialization. We don't include locals in the persisted state because they
can be trivially recomputed and this allows us to assume that they will
pass through verbatim, without any normalization or other transforms
caused by the JSON serialization.

From a user standpoint a local is just a named alias for an expression,
so it's desirable that the result passes through here in as raw a form
as possible, so it behaves as closely as possible to simply using the
given expression directly.
2017-08-21 15:15:25 -07:00
Martin Atkins a8c58b081c core: -target option to also select resources in descendant modules
Previously the behavior for -target when given a module address was to
target only resources directly within that module, ignoring any resources
defined in child modules.

This behavior turned out to be counter-intuitive, since users expected
the -target address to be interpreted hierarchically.

We'll now use the new "Contains" function for addresses, which provides
a hierarchical "containment" concept that is more consistent with user
expectations. In particular, it allows module.foo to match
module.foo.module.bar.aws_instance.baz, where before that would not have
been true.

Since Contains isn't commutative (unlike Equals) this requires some
special handling for targeting specific indices. When given an argument
like -target=aws_instance.foo[0], the initial graph construction (for
both plan and refresh) is for the resource nodes from configuration, which
have not yet been expanded to separate indexed instances. Thus we need
to do the first pass of TargetsTransformer in mode where indices are
ignored, with the work then completed by the DynamicExpand method which
re-applies the TargetsTransformer in index-sensitive mode.

This is a breaking change for anyone depending on the previous behavior
of -target, since it will now select more resources than before. There is
no way provided to obtain the previous behavior. Eventually we may support
negative targeting, which could then combine with positive targets to
regain the previous behavior as an explicit choice.
2017-06-16 16:36:08 -07:00
Martin Atkins c835ef8ff3 Update tests for the new ProviderResolver interface
Rather than providing an already-resolved map of plugins to core, we now
provide a "provider resolver" which knows how to resolve a set of provider
dependencies, to be determined later, and produce that map.

This requires the context to be instantiated in a different way, so this
very noisy diff is a mostly-mechanical update of all of the existing
places where contexts get created for testing, using some adapted versions
of the pre-existing utilities for passing in mock providers.
2017-06-09 14:03:59 -07:00
Martin Atkins 8bfc6e7b1c core: add missing ResourceState types in context tests
Previously the Type of a ResourceState was generally ignored, but we're
now starting to use it to figure out which providers are needed to
support the resources in state so our tests need to set it accurately
in order to get the expected result.
2017-06-09 14:03:59 -07:00
Martin Atkins 410b60cb7f Stop requiring multi-vars (splats) to be in array brackets
Prior to Terraform 0.7, lists in Terraform were just a shallow abstraction
on top of strings with a magic delimiter between items. Wrapping a single
string in brackets in the configuration was Terraform's prompt that it
needed to split the string on that delimiter during interpolation.

In 0.7, when first-class lists were added, this convention was preserved
by flattening lists-of-lists by one level when they were encountered in
configuration. However, there was an oversight in that change where it
did not correctly handle the case where the inner list was unknown.

In #14135 we removed some code that was flattening partially-unknown lists
into fully-unknown (untyped) values. This inadvertently exposed the missed
case from the previous paragraph, causing issues for list-wrapped splat
expressions with unknown members. While this worked fine for resources,
due to some fixup done inside helper/schema, this did not work for other
interpolation contexts such as module blocks.

Various attempts to fix this up and restore the flattening behavior
selectively were unsuccessful, due to a proliferation of assumptions all
over the core code that would be too risky to change just to fix this bug.

This change, then, takes the different approach of removing the
requirement that splats be presented inside list brackets. This
requirement didn't make much sense anymore anyway, since no other
list-returning expression had this constraint and so the rest of Terraform
was already successfully dealing with both cases.

This leaves us with two different scenarios:

- For resource arguments, existing normalization code in helper/schema
  does its own flattening that preserves compatibility with the common
  practice of using bracketed splats. This change proves this with a test
  within the "test" provider that exercises the whole Terraform core and
  helper/schema stack that assigns bracketed splats to list and set
  attributes.

- For arguments in other blocks, such as in module callsites, the
  interpolator's own flattening behavior applies to known lists,
  preserving compatibility with configurations from before
  partially-computed splats were possible, but those wishing to use
  partially-computed splats are required to drop the surrounding brackets.
  This is less concerning because this scenario was introduced only in
  0.9.5, so the scope for breakage is limited to those who adopted this
  new feature quickly after upgrading.

As of this commit, the recommendation is to stop using brackets around
splats but the old form continues to be supported for backward
compatibility. In a future _major_ version of Terraform we will probably
phase out this legacy form to improve consistency, but for now both
forms are acceptable at the expense of some (pre-existing) weird behavior
when _actual_ lists-of-lists are used.

This addresses #14521 by officially adopting the suggested workaround of
dropping the brackets around the splat. However, it doesn't yet allow
passing of a partially-unknown list between modules: that still violates
assumptions in Terraform's core, so for the moment partially-unknown lists
work only within a _single_ interpolation expression, and cannot be
passed around between expressions. Until more holistic work is done to
improve Terraform's type handling, passing a partially-unknown splat
through to a module will result in a fully-unknown list emerging on
the other side, just as was the case before #14135; this change just
addresses the fact that this was failing with an error in 0.9.5.
2017-05-23 11:22:37 -07:00
Martin Atkins 45b04c826a core: don't crash if no module state exists for multi var
For child modules, a ModuleState isn't allocated until the first time a
module instance is inserted into the state under the module's path.
Normally interpolations of resource attributes are delayed until at least
one resource has been created due to the nature of the dependency graph,
but if the interpolation value is a multi-var (splat) then it is possible
that the referenced resource has count=0 and thus created _no_ resource
states when it was visited.

Previously we would crash when trying to access the resource map for the
nil module in order to count how many instances are present. Since we know
there can't be any instances present in a nil module, we now preempt
this crash by returning zero early.

This edge-case does not apply to the root module because its ModuleState
is allocated as part of initializing the main State instance.

This fixes #14438.
2017-05-16 09:54:33 -07:00
Martin Atkins 7bdf4a925d core: Allow downstream targeting of certain node types
The previous behavior of targets was that targeting a particular node
would implicitly target everything it depends on. This makes sense when
the dependencies in question are between resources, since we need to
make sure all of a resource's dependencies are in place before we can
create or update it.

However, it had the undesirable side-effect that targeting a resource
would _exclude_ any outputs referring to it, since the dependency edge
goes from output to resource. This then causes the output to be "stale",
which is problematic when outputs are being consumed by downstream
configs using terraform_remote_state.

GraphNodeTargetDownstream allows nodes to opt-in to a new behavior where
they can be targeted by _inverted_ dependency edges. That is, it allows
outputs to be considered targeted if anything they directly depend on
is targeted.

This is different than the implied targeting behavior in the other
direction because transitive dependencies are not considered unless the
intermediate nodes themselves have TargetDownstream. This means that
an output1→output2→resource chain can implicitly target both outputs, but
an output→resource1→resource2 chain _won't_ target the output if only
resource2 is targeted.

This behavior creates a scenario where an output can be visited before
all of its dependencies are ready, since it may have a mixture of both
targeted and untargeted dependencies. This is fine for outputs because
they silently ignore any errors encountered during interpolation anyway,
but other hypothetical future implementers of this interface may need to
be more careful.

This fixes #14186.
2017-05-11 11:57:46 -07:00
James Bardin ad1cef4227 fix a few races in the context tests
these were so infrequent that the race detector couldn't catch them
either, but on a heavily loaded CI host they occasionally fail.
2017-04-21 12:30:53 -04:00
James Bardin 3f49227b72 add state an context tests
Make sure duplicate depends_on entries are pruned from existing states
on read.

Make sure new state built from configs with multiple references to the
same resource only add it once to the Dependencies.
2017-04-08 15:37:15 -04:00
James Bardin 7c9863ccb0 fix another hot lop in tests
Found another test spinlock.
Slow it down to prevent it from blocking the runtime scheduler.
2017-03-15 12:02:28 -04:00
Mitchell Hashimoto 9900bd752a
terraform: string through the context meta 2017-03-13 16:21:09 -07:00
Mitchell Hashimoto 19a594ba3d
terraform: run Gosched in test so other goroutines run
Starting with Go 1.8 betas, we've periodically received SIGQUITs on our
tests in Travis. The stack trace looks like this:

https://gist.github.com/mitchellh/abf09b0980f8ea01269f8d9d6133884d

The tests are timing out! This is a test that hasn't been touched really
in a very long time and has always passed. I've **reproduced this
locally** by setting `GOMAXPROCS=1` and running the test. By yielding
the scheduler in the hot loop, it now passes almost instantly every
time.

Perhaps the test can be written in a different way, but this gets tests
passing and I think will fix our periodic errors.
2017-03-01 23:12:41 -08:00
Mitchell Hashimoto f6ab0bc5c4
terraform: interpolation failures on a destroy provisioner should error 2017-02-17 14:32:32 -08:00
Mitchell Hashimoto a1ec81964b
terraform: destroy ordering needs to handle destroy provisioner edges
This ensures that things aren't destroyed before their values are used.
2017-02-17 14:29:22 -08:00
Mitchell Hashimoto 757217b91f
terraform: destroy resource should depend on destroy-time prov deps 2017-02-17 13:13:44 -08:00
Mitchell Hashimoto 4d6085b46a
terraform: outputs should not be included if not targeted
Fixes #10911

Outputs that aren't targeted shouldn't be included in the graph.

This requires passing targets to the apply graph. This is unfortunate
but long term should be removable since I'd like to move output changes
to the diff as well.
2017-02-13 12:52:45 -08:00
Mitchell Hashimoto 8ed9bdfedc
terraform: node referenceable name from state shuldn't contain path
Fixes #11749

I'm **really** surprised this didn't come up earlier.

When only the state is available for a node, the advertised
referenceable name (the name used for dependency connections) included
the module path. This module path is automatically prepended to the
name. This means that probably every non-root resource for state-only
operations (destroys) didn't order properly.

This fixes that by omitting the path properly.

Multiple tests added to verify both graph correctness as well as a
higher level context test.

Will backport to 0.8.x
2017-02-07 20:14:38 -08:00
Mitchell Hashimoto 61881d2795 Merge pull request #10934 from hashicorp/f-provisioner-stop
core: stoppable provisioners, helper/schema for provisioners
2017-01-30 12:53:15 -08:00
Mitchell Hashimoto d59725e9fd
terraform: convert StateDeps to use new structs 2017-01-26 20:47:20 -08:00
Mitchell Hashimoto a8f64cbcee
terraform: make sure Stop blocks until full completion 2017-01-26 15:10:30 -08:00
Mitchell Hashimoto f8c7b639c9
terraform: switch to Context for stop, Stoppable provisioners
This switches to the Go "context" package for cancellation and threads
the context through all the way to evaluation to allow behavior based on
stopping deep within graph execution.

This also adds the Stop API to provisioners so they can quickly exit
when stop is called.
2017-01-26 15:03:27 -08:00
Mitchell Hashimoto f40fdde708
terraform: PostProvision hook gets the error from the provision step 2017-01-20 20:21:12 -08:00
Mitchell Hashimoto b56ee1a169
terraform: test on_failure with non-destroy provisioners 2017-01-20 20:05:28 -08:00
Mitchell Hashimoto 4a8c2d0958
terraform: on_failure for provisioners 2017-01-20 19:55:32 -08:00
Mitchell Hashimoto 85cb3a16b0
terraform: on destroy prov failure, don't taint and preserve state 2017-01-20 18:26:41 -08:00
Mitchell Hashimoto e9f6c9c429
terraform: run destroy provisioners on destroy 2017-01-20 18:07:51 -08:00
Mitchell Hashimoto 5f1e6ad020
terraform: TargetsTransformer should preserve module variables
Fixes #10680

This moves TargetsTransformer to run after the transforms that add
module variables is run. This makes targeting work across modules (test
added).

This is a bug that only exists in the new graph, but was caught by a
shadow error in #10680. Tests were added to protect against regressions.
2016-12-12 20:59:14 -08:00
James Bardin ed3517858f Merge pull request #10670 from hashicorp/jbardin/GH-10603
Prevent data sources from being aplied early
2016-12-12 14:53:03 -05:00
James Bardin d2c6f1b57f Prevent data sources from being aplied early
If a data source has explicit dependencies in `depends_on`, we can
assume the user has added those because of a dependency not tracked
directly in the config. If there are any entries in `depends_on`, don't
apply the data source early during Refresh.
2016-12-12 10:35:10 -05:00
Mitchell Hashimoto b346ba32d1
terraform: dependent provider resources are destroyed first in modules
This extends the prior commit to also verify (and fix) that resources of
dependent providers are destroyed first even when they're within
modules.
2016-12-10 20:22:12 -05:00
Mitchell Hashimoto 14d079f914
terraform: destroy resources in dependent providers first
Fixes #4645

This is something that never worked (even in legacy graphs), but as we
push forward towards encouraging multi-provider usage especially with
things like the Vault data source, I want to make sure we have this
right for 0.8.

When you have a config like this:

```
resource "foo_type" "name" {}
provider "bar" { attr = "${foo_type.name.value}" }
resource "bar_type" "name" {}
```

Then the destruction ordering MUST be:

  1. `bar_type`
  2. `foo_type`

Since configuring the client for `bar_type` requires accessing data from
`foo_type`. Prior to this PR, these two would be done in parallel. This
properly pushes forward the dependency.

There are more cases I want to test but this is a basic case that is
fixed.
2016-12-10 20:11:24 -05:00
Mitchell Hashimoto 0e4a6e3e89
terraform: apply resource must depend on destroy deps
Fixes #10440

This updates the behavior of "apply" resources to depend on the
destroy versions of their dependencies.

We make an exception to this behavior when the "apply" resource is CBD.
This is odd and not 100% correct, but it mimics the behavior of the
legacy graphs and avoids us having to do major core work to support the
100% correct solution.

I'll explain this in examples...

Given the following configuration:

    resource "null_resource" "a" {
       count = "${var.count}"
    }

    resource "null_resource" "b" {
      triggers { key = "${join(",", null_resource.a.*.id)}" }
    }

Assume we've successfully created this configuration with count = 2.
When going from count = 2 to count = 1, `null_resource.b` should wait
for `null_resource.a.1` to destroy.

If it doesn't, then it is a race: depending when we interpolate the
`triggers.key` attribute of `null_resource.b`, we may get 1 value or 2.
If `null_resource.a.1` is destroyed, we'll get 1. Otherwise, we'll get
2. This was the root cause of #10440

In the legacy graphs, `null_resource.b` would depend on the destruction
of any `null_resource.a` (orphans, tainted, anything!). This would
ensure proper ordering. We mimic that behavior here.

The difference is CBD. If `null_resource.b` has CBD enabled, then the
ordering **in the legacy graph** becomes:

  1. null_resource.b (create)
  2. null_resource.b (destroy)
  3. null_resource.a (destroy)

In this case, the update would always have 2 values for `triggers.key`,
even though we were destroying a resource later! This scenario required
two `terraform apply` operations.

This is what the CBD check is for in this PR. We do this to mimic the
behavior of the legacy graph.

The correct solution to do one day is to allow splat references
(`null_resource.a.*.id`) to happen in parallel and only read up to to
the `count` amount in the state. This requires some fairly significant
work close to the 0.8 release date, so we can defer this to later and
adopt the 0.7.x behavior for now.
2016-12-03 23:54:29 -08:00
Mitchell Hashimoto 08a56304bb Merge pull request #10455 from hashicorp/b-non-cbd-promote
terraform: when promoting non-CBD to CBD, mark the config as such
2016-12-02 09:51:27 -05:00
Mitchell Hashimoto f3a62c694d
terraform: when promoting non-CBD to CBD, mark the config as such
Fixes #10439

When a CBD resource depends on a non-CBD resource, the non-CBD resource
is auto-promoted to CBD. This was done in
cf3a259. This PR makes it so that we
also set the config CBD to true. This causes the proper runtime
execution behavior to occur where we depose state and so on.

So in addition to simple graph edge tricks we also treat the non-CBD
resources as CBD resources.
2016-12-02 09:46:04 -05:00
Mitchell Hashimoto 41c56ad002
terraform: new apply graph understands destroying deposed only 2016-11-28 14:34:24 -08:00
Mitchell Hashimoto 3bf93501a1
terraform: test applying tainted + deposed (passes)
This is added just trying to reproduce a crash I saw. It passes so
adding it to the master tests.
2016-11-28 14:29:38 -08:00
Mitchell Hashimoto f50c7acf95
terraform: record dependency to self (other index)
Fixes #10313

The new graph wasn't properly recording resource dependencies to a
specific index of itself. For example: `foo.bar.2` depending on
`foo.bar.0` wasn't shown in the state when it should've been.

This adds a test to verify this and fixes it.
2016-11-23 09:25:20 -08:00
Mitchell Hashimoto f35d02cbee
terraform: test for alias inheritance [GH-4789] 2016-11-21 21:51:07 -08:00
Mitchell Hashimoto e7d59ab245
terraform: test for interpolation escapes 2016-11-20 21:14:16 -08:00
Mitchell Hashimoto df34fa88ce Merge pull request #10076 from hashicorp/f-depend-module
terraform: depends_on can reference entire modules
2016-11-14 11:53:12 -08:00
Mitchell Hashimoto 538302f143
terraform: resources nested within a module must also be depended on
For example: A => B => C (modules). If A depends on module B, then it
also must depend on everything in module C.
2016-11-12 15:38:28 -08:00
Mitchell Hashimoto bcfec4e24e
terraform: test that dependencies in the state are enough to maintain
order
2016-11-12 15:22:48 -08:00
Mitchell Hashimoto 0b87ef82c3
terraform: depends_on can reference entire modules 2016-11-12 08:07:45 -08:00
Mitchell Hashimoto 70a41c5e15
terraform: output nodes reference `depends_on` values 2016-11-11 18:16:04 -08:00
Mitchell Hashimoto b68b95dad0
terraform: destroy graph must connect edges for proper target ordering
This connects the destroy edges so that when a `-target` is specified on
a destroy, the proper dependencies get destroyed as well.
2016-11-10 21:14:44 -08:00
Mitchell Hashimoto 66ccc19d94 Merge pull request #9973 from hashicorp/f-new-plan
terraform: new plan graph
2016-11-09 08:15:40 -08:00
Mitchell Hashimoto 9e0af96afa Merge pull request #9971 from hashicorp/b-orphan-output
terraform: prune orphan outputs in old graph
2016-11-09 07:53:15 -08:00
Mitchell Hashimoto 2b7177cfe7 Merge pull request #9607 from hashicorp/f-provider-stop-redo
terraform: ResourceProvider.Stop (redo)
2016-11-08 15:58:48 -08:00
Mitchell Hashimoto 19350d617d
terraform: references can have backups
terraform: more specific resource references

terraform: outputs need to know about the new reference format

terraform: resources w/o a config still have a referencable name
2016-11-08 13:59:30 -08:00
Mitchell Hashimoto 1efdba9b30
terraform: target at the right moment to get the right values 2016-11-08 13:59:30 -08:00
Mitchell Hashimoto 9890a2ee91
terraform: prune orphan outputs in old graph
This makes the old graph also prune orphan outputs in modules.
This will fix shadow graph errors such as #9905 since the old graph will
also behave correctly in these scenarios.

Luckily, because orphan outputs don't rely on anything, we were able to
simply use the same transformer!
2016-11-08 13:24:09 -08:00
Mitchell Hashimoto fb29b6a2dc
terraform: destroy edges should never point to self
Fixes #9920

This was an issue caught with the shadow graph. Self references in
provisioners were causing a self-edge on destroy apply graphs.

We need to explicitly check that we're not creating an edge to ourself.
This is also how the reference transformer works.
2016-11-08 12:27:33 -08:00
Mitchell Hashimoto f580a8ed7b Merge pull request #9894 from hashicorp/b-provider-alias
terraform: configure provider aliases in the new apply graph
2016-11-07 07:59:33 -08:00
Mitchell Hashimoto 1beda4cd07 Merge pull request #9898 from hashicorp/b-new-ref-existing
terraform: remove pruning of module vars if they ref non-existing nodes
2016-11-07 07:59:25 -08:00
Mitchell Hashimoto b7954a42fe
terraform: remove pruning of module vars if they ref non-existing nodes
Fixes a shadow graph error found during usage.

The new apply graph was only adding module variables that referenced
data that existed _in the graph_. This isn't a valid optimization since
the data it is referencing may be in the state with no diff, and
therefore available but not in the graph.

This just removes that optimization logic, which causes no failing
tests. It also adds a test that exposes the bug if we had the pruning
logic.
2016-11-04 17:47:20 -07:00
Mitchell Hashimoto d7ed6637c1
terraform: configure provider aliases in the new apply graph
Found via a shadow graph failure:

Provider aliases weren't being configured by the new apply graph.

This was caused by the transform that attaches configs to provider nodes
not being able to handle aliases and therefore not attaching a config.
Added a test to this and fixed it.
2016-11-04 16:51:52 -07:00
Mitchell Hashimoto 944ffdb95b
terraform: multi-var ordering is by count
Fixes #9444

This appears to be a regression from 0.7.0, but there were no tests
covering it so we missed it and changed the behavior at some point! Oh
no.

This PR make the ordering of multi-var access: `resource.name.*.attr`
consistent: it is the ordering of the count, not the lexical ordering of
the value. This allows behavior where two lists are indexed by count
index and can be assumed to be related (for example user data for an aws
instance, as shown in the above referenced issue).

Two new context tests added to cover this case.
2016-11-04 11:07:01 -07:00
Mitchell Hashimoto d2e9c35007
terraform: new apply graph creates provisioners in modules
Fixes #9840

The new apply graph wasn't properly nesting provisioners. This resulted
in reading the provisioners being nil on apply in the shadow graph which
caused the crash in the above issue.

The actual cause of this is that the new graphs we're moving towards do
not have any "flattening" (they are flat to begin with): all modules are
in the root graph from the beginning of construction versus building a
number of different graphs and flattening them. The transform that adds
the provisioners wasn't modified to handle already-flat graphs and so
was only adding provisioners to the root module, not children.

The change modifies the `MissingProvisionerTransformer` (primarily) to
support already-flat graphs and add provisioners for all module levels.
Tests are there to cover this as well.

**NOTE:** This PR focuses on fixing that specific issue. I'm going to follow up
this PR with another PR that is more focused on being robust against
crashing (more nil checks, recover() for shadow graph, etc.). In the
interest of focus and keeping a PR reviewable this focuses only on the
issue itself.
2016-11-03 10:25:11 -07:00
Mitchell Hashimoto 9e5d1f10b0
terraform: add test to verify orphan outputs in modules are removed
For #7598

This doesn't work with the old graph, we guard it as such.
2016-11-01 22:42:41 -07:00
Mitchell Hashimoto b005a83143
terraform: deposed should trigger PostApply hook
Fixes #6327

Deposed instances weren't calling PostApply which was causing the counts
for what happened during `apply` to be wrong. This was a simple fix to
ensure we call that hook.
2016-10-30 15:24:20 -07:00
Mitchell Hashimoto aed23a0a31 Merge pull request #9527 from hashicorp/f-destroy-builder2
terraform: destroy graph builder based on state
2016-10-26 12:53:20 -04:00
Mitchell Hashimoto a61b7227f5
terraform: Context.Stop() calls Stop on providers if running 2016-10-25 12:08:35 -07:00
Mitchell Hashimoto 60140b28f4
Revert "Merge pull request #9536 from hashicorp/f-provider-stop"
This reverts commit c3a4cff133, reversing
changes made to 791a02e6e4.

This change requires plugin recompilation and we should hold off until a
minor release for that.
2016-10-25 12:00:36 -07:00
Mitchell Hashimoto 0cca4fc093
terraform: Context.Stop() calls Stop on providers if running 2016-10-25 11:31:55 -07:00
Mitchell Hashimoto 791a02e6e4
terraform: test that depends_on is used for destroy ordering 2016-10-25 11:05:48 -07:00
Mitchell Hashimoto 1523504645
terraform: enable shadow graph and destroy resource mode with addr
This enables the shadow graph since all tests pass!

We also change the destroy node to check the resource type using the
addr since that is always available and reliable. The configuration can
be nil for orphans.
2016-10-22 12:12:30 -07:00
Mitchell Hashimoto 1d27e554a5
terraform: test to ensure data sources work on Apply operation
It appears data sources have always been coded to work during apply, as
can be verified with this test (no impl. changes were necessary to make
it pass).

This test should be added to ensure our apply graph always works with
data sources as well.
2016-10-20 21:53:54 -07:00
Mitchell Hashimoto e4ef1fe553
terraform: disable providers in new apply graph
This adds the proper logic for "disabling" providers to the new apply
graph: interolating and storing the config for inheritance but not
actually initializing and configuring the provider.

This is important since parent modules will often contain incomplete
provider configurations for the purpose of inheritance that would error
if they were actually attempted to be configured (since they're
incomplete). If the provider is not used, it should be "disabled".
2016-10-19 14:54:00 -07:00
Mitchell Hashimoto eb9ecea863
terraform: don't set Provider on destroy nodes
This doesn't explicitly set `rs.Provider` on destroy nodes.

To be honest, I'm not sure why this was done in the first place (git
blame points to 6fda7bb5483a155b8ae1e1e4e4b7b7c4073bc1d9). Tests always
passed without it, and by adding it it causes other tests to fail. I
should've never changed those other tests.

Removing it now to get tests passing, this also reverts the test changes
made in 8213824962f085279810f04b60b95d1176a3a3f2.
2016-10-19 13:39:13 -07:00
Mitchell Hashimoto 7c2c9b82a3
terraform: interpolation for multi-var checks both ".0" and "" suffix 2016-10-19 13:39:13 -07:00
Mitchell Hashimoto 924f7a49e0
terraform: module variable transform must do children later (tested) 2016-10-19 13:38:53 -07:00
Mitchell Hashimoto c1664d2eaa
terraform: cbd works! 2016-10-19 13:38:53 -07:00
Mitchell Hashimoto bd5d97f9f5
terraform: transform to attach resource configs 2016-10-19 13:38:52 -07:00
Mitchell Hashimoto cd04ccfa62
terraform: update a test to be easier to target 2016-10-19 13:38:51 -07:00
Mitchell Hashimoto dfa02e4412
terraform: rename attach config to only attach provider config 2016-10-19 13:38:51 -07:00
Mitchell Hashimoto 0e666aa575
terraform: get tests to not panic on failures 2016-10-19 13:38:51 -07:00
Mitchell Hashimoto 7dd4813730
terraform: rename test to be more easily targetable 2016-10-19 13:38:50 -07:00
Mitchell Hashimoto 87bff933ef
terraform: ParentProviderTransform to connect parent providers 2016-10-19 13:38:49 -07:00
Mitchell Hashimoto e784e4a434
terraform: remove more nil panics (doesn't change test logic) 2016-10-19 13:38:49 -07:00
Mitchell Hashimoto 77b9177bd5
terraform: an incredible number of failing tests! 2016-10-19 13:38:49 -07:00
Mitchell Hashimoto 3edb8599b1
terraform: Shadow interface, properly string through errors at the right
time
2016-10-11 22:17:29 +08:00