Commit Graph

22249 Commits

Author SHA1 Message Date
James Bardin 6a1b57a1e4 split PriorStateRaw into JSON and Flatmap
This way the provider doesn't need to be concerned with determining the
state encoding.
2018-10-16 18:50:29 -07:00
James Bardin 91b5bbbde0 add versions to the provider schemas
We need to know the schema version for all resources. This is stored in
order to allow providers to upgrade the state from a known previous
version.
2018-10-16 18:50:29 -07:00
James Bardin 1473d09c50 rename provider and provisioner packages
Using plural names to avoid collisions in existing code.
2018-10-16 18:50:29 -07:00
Martin Atkins 0742e756e5 tfdiags: Sort order for diagnostics
Because we gather together diagnostics from many different parts of the
codebase, the list often ends up being in a non-ideal order. Here we
define a partial ordering for diagnostics that should hopefully make them
easier to scan when many are present, by grouping together diagnostics
that are of the same severity and belong to the same file.

We use sort.Stable here because we have a partial order and so we need
to make sure that diagnostics that do not have a relative ordering will
remain in their original order.

This sorting is applied just in time before rendering the diagnostics
in command.Meta.showDiagnostics.
2018-10-16 18:50:29 -07:00
James Bardin 27d90c7550 Add provider plugin interface and types
This adds the new interface for the resource providers. The methods here
closely follow the grpc service methods, using the types expected in
terraform core
2018-10-16 18:50:29 -07:00
James Bardin fdfbc5c1a7 Add provisioner interface and request types
This adds the interface for the new resource provisioners.
The methods closely follow the grpc service methods, using the type
expected within terraform core.
2018-10-16 18:50:29 -07:00
Martin Atkins a2eb462f5d plans/planfile: Reading and writing the new plan format
The new format is radically different in than the old in physical
structure, but still has the same logical parts: the plan itself, a
snapshot of the input configuration, and a snapshot of the state as it
existed when the plan was created.

Rather than creating plan-specific serializations of state and config, the
new format instead leans on the existing file formats implemented
elsewhere, wrapping the result up in a zip archive with some internal file
naming conventions.

The plan portion of the file is serialized with protobuf, consistent with
our general strategy of replacing all use of encoding/gob with protobuf
moving forward.
2018-10-16 18:50:29 -07:00
Martin Atkins 7357e7f734 plans: New package for in-memory plan models
The types in this package are intended to replace plan- and diff-related
types from the "terraform" package, although those older types must remain
for now so that they can be used to implement shims for older codepaths.

type "Changes" is approximately equivalent to terraform.Diff, but renamed
since it now describes whole objects before and after rather than an
attribute-level diff as before. The term "diff" is now reserved for the
visual rendition of the changes we'll display to the user, although
rendering of this new Changes model is not yet implemented.
2018-10-16 18:50:29 -07:00
Martin Atkins eb7aaf2414 configload: Configuration snapshots
Here we introduce a new idea of a "configuration snapshot", which is an
in-memory copy of the source code of each of the files that make up
the configuration. The primary intended purpose for this is as an
intermediate step before writing the configuration files into a plan file,
and then reading them out when that plan file is later applied.

During earlier configs package development we expected to use an afero vfs
implementation to read directly from the zip file, but that doesn't work
in practice because we need to preserve module paths from the source file
system that might include parent directory traversals (../) while
retaining the original path for use in error messages.

The result, for now, is a bit of an abstraction inversion: we implement
a specialized afero vfs implementation that makes the sparse filesystem
representation from a snapshot appear like a normal filesystem just well
enough that the config loader and parser can work with it.

In future we may wish to rework the internals here so that the main
abstraction is at a similar level to the snapshot and then that API is
mapped to the native filesystem in the normal case, removing afero. For
now though, this approach avoids the need for a significant redesign
of the parser/loader internals, at the expense of some trickiness in the
case where we're reading from a snapshot.

This commit does not yet include the reading and writing of snapshots into
plan files. That will follow in a subsequent commit.
2018-10-16 18:50:29 -07:00
Martin Atkins 93cda6dbd2 govendor fetch github.com/zclconf/go-cty/cty/...
This allows automatic conversions between different object types as long
as the target type is a subset of the given type.
2018-10-16 18:50:29 -07:00
Martin Atkins b1247bf7af govendor fetch github.com/zclconf/go-cty/cty/...
The existing cty packages were already at the latest version, but we were
not yet vendoring the msgpack package.

This also imports some dependencies from:
    github.com/vmihailenco/msgpack
2018-10-16 18:50:29 -07:00
Martin Atkins da17afaa85 govendor fetch github.com/golang/protobuf/proto/... 2018-10-16 18:49:20 -07:00
Martin Atkins a33f941778 states: New SyncState type
This is a wrapper around State that is able to perform higher-level
manipulations (at the granularity of the entire state) in a
concurrency-safe manner, using the lower-level APIs exposed by State and
all of the types it contains.

The granularity of a SyncState operation roughly matches the granularity
off a state-related EvalNode in the "terraform" package, performing a
sequence of more primitive operations while guaranteeing atomicity of the
entire change.

As a compromise for convenience of usage, it's still possible to access
the individual state data objects via this API, but they are always copied
before returning to ensure that two distinct callers cannot have data
races. Callers should access the most granular object possible for their
operation.
2018-10-16 18:49:20 -07:00
Martin Atkins 53cafc542b statemgr: New package for state managers
This idea of a "state manager" was previously modelled via the
confusingly-named state.State interface, which we've been calling a "state
manager" only in some local variable names in situations where there were
also *terraform.State variables.

As part of reworking our state models to make room for the new type
system, we also need to change what was previously the state.StateReader
interface. Since we've found the previous organization confusing anyway,
here we just copy all of those interfaces over into statemgr where we can
make the relationship to states.State hopefully a little clearer.

This is not yet a complete move of the functionality from "state", since
we're not yet ready to break existing callers. In a future commit we'll
turn the interfaces in the old "state" package into aliases of the
interfaces in this package, and update all the implementers of what will
by then be statemgr.Reader to use *states.State instead of
*terraform.State.

This also includes an adaptation of what was previously state.LocalState
into statemgr.FileSystem, using the new state serialization functionality
from package statefile instead of the old terraform.ReadState and
terraform.WriteState.
2018-10-16 18:49:20 -07:00
Martin Atkins 5c1c6e9d9c statefile: New package for loading and saving state files
Whereas the parent directory "states" contains the models that represent
state in memory, this package's responsibility is in serializing a subset
of that data to a JSON-based file format and then reloading that data
back into memory later.

For reading, this package supports state file formats going back to
version 1, using lightly-adapted versions of the migration code previously
used in the "terraform" package. State data is upgraded to the latest
version step by step and then transformed into the in-memory state
representation, which is distinct from any of the file format structs in
this package to enable these to evolve separately.

For writing, only the latest version (4) is supported, which is a new
format that is a slightly-flattened version of the new in-memory state
models introduced in the prior commit. This format retains the outputs
from only the root module and it flattens out the module and instance
parts of the hierarchy by including the identifiers for these inside
the child object. The loader then reconstructs the multi-layer structure
we use for more convenient access in memory.

For now, the only testing in this package is of round-tripping different
versions of state through a read and a write, ensuring the output is
as desired. This exercises all of the reading, upgrading, and writing
functions but should be augmented in later commits to improve coverage
and introduce more focused tests for specific parts of the functionality.
2018-10-16 18:49:20 -07:00
Martin Atkins b975ada8db states: New package with modern models for Terraform state
Our previous state models in the "terraform" package had a few limitations
that are addressed here:

- Instance attributes were stored as map[string]string with dot-separated
  keys representing traversals through a data structure. Now that we have
  a full type system, it's preferable to store it as a real data
  structure.

- The existing state structures skipped over the "resource" concept and
  went straight to resource instance, requiring heuristics to decide
  whether a particular resource should appear as a single object or as
  a list of objects when used in configuration expressions.

- Related to the previous point, the state models also used incorrect
  terminology where "ResourceState" was really a resource instance state
  and "InstanceState" was really the state of a particular remote object
  associated with an instance. These new models use the correct names for
  each of these, introducing the idea of a "ResourceInstanceObject" as
  the local record of a remote object associated with an instance.

This is a first pass at fleshing out a new model for state. Undoubtedly
there will be further iterations of this as we work on integrating these
new models into the "terraform" package.

These new model types no longer serve double-duty as a description of the
JSON state file format, since they are for in-memory use only. A
subsequent commit will introduce a separate package that deals with
persisting state to files and reloading those files later.
2018-10-16 18:49:20 -07:00
Martin Atkins ba894ee05c addrs: More string parsing helpers for addresses
Our main "parse" methods in this package work with hcl.Traversals, but
we're gradually adding helpers to parse these directly froms strings since
the visual noise of doing the traversal parse first is inconvenient in
situations where addresses are coming from non-config locations where
no source information is available anyway.
2018-10-16 18:49:20 -07:00
Martin Atkins 00199cd2ed addrs: "Less" comparison method for resource and module instances
This can be used to sort lists of resource instance and module instance
addresses, such as in a rendered plan.
2018-10-16 18:49:20 -07:00
Kristin Laemmert c23a971ed1 minor fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert 0dbecc54c0 functions: ValuesFunc - cleanup return type 2018-10-16 18:49:20 -07:00
Kristin Laemmert a213c4a648 functions: add tests and support for unknown values 2018-10-16 18:49:20 -07:00
Kristin Laemmert d802d5c624 functions: pr feedback fixes 2018-10-16 18:49:20 -07:00
Kristin Laemmert ff4b3d763b functions: fix lookup()'s handling of numberical defaults 2018-10-16 18:49:20 -07:00
Kristin Laemmert 4f5c03339a functions: ZipmapFunc 2018-10-16 18:49:20 -07:00
Martin Atkins 2ef56b3e05 Fix up some missed "go fmt"
Because of the size of some of these files, automatic format-on-save was
implicitly disabled in my editor, which I didn't notice before committing.
2018-10-16 18:49:20 -07:00
James Bardin 82d2c5b484 core: use absolute address in CloserProvider
Otherwsie relative addresses may collide and close the incorrect
provider.
2018-10-16 18:49:20 -07:00
James Bardin a2c76a28f5 core: run ProvisionerTransformer during plan
This makes sure the graph is complete for validation
2018-10-16 18:49:20 -07:00
Martin Atkins 0b710792a8 website: "functions" layout "Terraform Language" is back
In the heirarchy, both "Terraform Language" and "Functions" are "up" from
the individual function reference pages, so we'll class them as such to
use the back-facing arrow instead of the forward-facing arrow.
2018-10-16 18:49:20 -07:00
Martin Atkins 1360948a41 website: document the functions "keys", "lookup", and "values"
I missed these on the first pass because in the legacy function table they
are, for some reason, added in a different place than the others.
2018-10-16 18:49:20 -07:00
Martin Atkins 3c10a3b213 helper/schema: Tolerate incorrectly-specified collection elems
We historically tolerated this, so we need to tolerate it here too in
order to work correctly with existing provider code.
2018-10-16 18:49:20 -07:00
Martin Atkins 91def7b6f6 core: Better failure output for TestContext2Apply_moduleDestroyOrder 2018-10-16 18:49:20 -07:00
Martin Atkins bab9f7ce09 core: update Stringer implementations for GraphType and walkOperation
These are no longer correct due to the input walk being removed in an
earlier commit.
2018-10-16 18:49:20 -07:00
Martin Atkins 38f37af7e9 core: remove TestContext2Apply_outputInvalid
This was testing that returning a number as an output would be reported as
an error, but that is intentionally allowed now.
2018-10-16 18:49:20 -07:00
Martin Atkins b4d7882e2f core: Fix TestContextImport_providerVarConfig
I updated the "Variables" map incorrectly in earlier commit 10fe50bbdb
while making bulk updates to get the tests compiling again with the
changed underlying APIs.

The original value here was "bar", incorrectly changed to "foo" in that
commit. Here we return it back to "bar".
2018-10-16 18:49:20 -07:00
James Bardin 6eb897e293 core: attach provisioner schemas in subgraphs
DynamicExpand also needs to add the provisioner schemas to make sure all
references are found in the subgraph.
2018-10-16 18:49:20 -07:00
Martin Atkins 059de66fcc core: Don't save provider input for non-root module
We only support provider input for the root module. This is already
checked in ProviderInput, but was not checked in SetProviderInput. We
can't actually do anything particularly clever with an invalid call here,
but we will at least generate a WARN log to help with debugging.

Also need to update TestBuiltinEvalContextProviderInput to expect this
new behavior of ignoring input for non-root modules.
2018-10-16 18:49:20 -07:00
Martin Atkins b99b31ebea core: Correct schema for TestContext2Apply_issue5254
This test is now failing due to the fact that WritePlan is currently
disabled pending a rewrite. This will be addressed in a subsequent commit.
2018-10-16 18:49:20 -07:00
Martin Atkins 432331e484 core: Fix TestContext2Refresh_dataState
Now that we fetch schemas during NewContext, we need to configure the
mock GetSchema method before constructing the context.
2018-10-16 18:49:20 -07:00
James Bardin df2907abb9 core: TestContext2Apply_dataDependsOn 2018-10-16 18:49:20 -07:00
James Bardin 711f305dce core: TestContext2Validate_interpolateMap 2018-10-16 18:49:20 -07:00
Martin Atkins 71cedf19a4 core: Don't create indirect provider dependencies for references
The prior commit changed the schema-access model so that all schemas are
fetched up front during context creation and are then readily available
for use throughout graph building and evaluation.

As a result, we no longer need to create dependency edges to a provider
when one of its resources is referenced by another node, and so the
ProviderTransformer needs only to worry about direct ownership
dependencies.

This also avoids the need for us to run AttachSchemaTransformer twice,
since ProviderTransformer no longer needs schema and we can therefore
defer attaching until just before ReferenceTransformer, when all of the
referencable and referencing nodes are already present in the graph.
2018-10-16 18:49:20 -07:00
Martin Atkins d3e4565681 core: LoadSchemas must detect provisioners in non-root modules 2018-10-16 18:49:20 -07:00
Martin Atkins d961b1de1b core: Stop loading provider schema during graph walk
We now fetch all of the necessary schemas during context creation, so we
can just thread that repository of schemas through into EvalContext and
Evaluator and access the schemas as needed without any further fetching.

This requires updating a few tests to have a valid Provider address in
their state objects, because we need that in order to trigger the loading
of the relevant schema.
2018-10-16 18:49:20 -07:00
James Bardin 555cd977f8 core: TestContext2Validate_interpolateMap 2018-10-16 18:49:20 -07:00
Martin Atkins aa9d88ad3c core: Schema for TestContext2Apply_multiProviderDestroy
This test depends on having a correct schema, so we'll specify the minimum
schema for its fixture inline here rather than using the superset schema
returned by testProvider.
2018-10-16 18:49:20 -07:00
Martin Atkins f14369e7fb core: Remove machinery for the "input" walk
Provider input is now longer handled with a graph walk, so the code
related to the input graph and walk are no longer needed.

For now the Input method is retained on the ResourceProvider interface,
but it will never be called. Subsequent work to revamp the provider API
will remove this method.
2018-10-16 18:49:20 -07:00
James Bardin aa07f34dbe core: TestContext2Apply_createBeforeDestroy_hook
The instnace info ID has changed, and no longer reflects the type of
node. Record the state to determine apply order for this test.
2018-10-16 18:49:20 -07:00
James Bardin 403abb52b9 core: TestContext2Apply_createBeforeDestroyUpdate
The test fixture no longer changes the ID on updates.
2018-10-16 18:49:20 -07:00
James Bardin fa87397f50 core: GraphNodeAttachDestroyer
Add a graphNodeAttachDestroy interface, so destroy nodes can be attached
to their companion create node. The creator can then reference the
CreateBeforeDestroy status of the destroyer, determining  if the current
state needs to be replaced or deposed.

This is needed when a node is forced to become CreateBeforeDestroy by a
dependency rather than the config, since because the config is
immutable, only the destroyer is aware that it has been forced
CreateBeforeDestroy.
2018-10-16 18:49:20 -07:00
Martin Atkins fb70eaa7d1 core: EvalDiffDestroy only update state if requested
The earlier change 5f07201a made it so that the state is always rewritten
by EvalDiffDestroy, but that was too disruptive to other users of
EvalDiffDestroy.

Now we follow the lead of EvalDiff and have a separate pointer for the
_output_ state, which allows the caller to opt in to having its state
pointer updated to reflect the new (nil) state.

NodePlannableResourceInstanceOrphan is the only caller that currently opts
in to this, since that was the focus of 5f07201a. We may need to make a
similar change to other plannable resource destroy nodes, but we'll wait
to see if that needs to be done in a subsequent commit.
2018-10-16 18:49:20 -07:00