In order for providers to determine if a computed value was unset in the
configuration, the configuration must be passed along with the prior and
proposed states. Terraform core will still handle the creation of the
ProposedNewState to ensure correctness, but the raw configuration value
will also be supplied for comparison.
* builtin/providers: implement terraform remote state datasource as providers.Interface
* append and return diags separately (to match the idiomatic usage
elsewhere in Terraform)
* diagnostic summary style improvements
* update tests to pass config to schema.CoerceValue
* trust that the schema will be enforced and there is no need to check
that a given attribute exists
* added dataSourceRemoteStateGetSchema() (effectively replacing a
function that was inappropriately removed) for consistency with other
terraform providers
* builtin/provider terraform test: added InternalValidate() test for dataSourceRemoteStateGetSchema
In the old state package we had this as a separate manager
state.BackupState, but that doesn't work with our new interfaces because
we handle lineage and serial within the state managers themselves and
don't expose them to callers anymore.
In practice it being built in to the filesystem manager is not a problem
because we only use the backup functionality for local state anyway.
This also slightly adjusts the behavior to be more intuitive. The old
BackupState relied on the implementation detail that Terraform re-persists
the original state early in an apply operation, which meant that by
coincidence it would back up the right snapshot. In this new approach,
we instead take an in-memory copy during State and then write _that_ to
disk in WriteState if the new state seems different, so we're guaranteed
that we'll always write out what we read before any changes were made.
In future we may improve this further, such as keeping multiple
generations of backups, etc. But for now this is intended to preserve the
goals of the original implementation while making its behavior
self-contained and not dependent on coincidences.
The update protocol shims will also check for this this, but eventually
"id" will only be a normal attribute, and we shouldn't have to special
case this.
When converting a legacy schemaMap to a configschema, we need to add
"id" as a required attribute to top-level resources if it's not
declared.
The "id" field will be required to interoperate with the legacy helper
schema, since the presence of an id was used to indicate the existence
of a resource.
We shouldn't really need these because the plan is already filtered to
include diffs only for targeted resources, but we currently rely on this
to filter out non-resource items from the diff, and so we'll retain it
for now to avoid reworking how the apply-time graph builder works.
On the first pass here we erroneously assumed that this was redundant with
the backend settings embedded in the configuration itself. In practice,
users can override backend configuration when running "terraform init"
and so we need to record the _effective_ backend configuration.
Along with this, we also return the selected workspace name at the time
the plan was created so we'll later be able to produce a specialized error
for the situation of having the wrong workspace selected. This isn't
strictly required because we'll also check the lineage of the state, but
the error message that would result from that failure would be relatively
opaque and thus less helpful to the user.
A nil flatmap will be encoded as a NullVal of the correct type. When
Converting a NullVal back to a flatmap, return nil immediately rather
than attempting to build the values.
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
Since the "References" function on graph nodes can't return errors, we
need to catch invalid depends_on references during the validation pass.
In this case, we're checking that the address is exact, rather than being
part of a traversal into an attribute of the object. In other words,
aws_instance.example is valid but aws_instance.example.id is not.
The only contextual information returned by the plugins in a diagnostic
is an attribute path. In order to test the conversions, we need to be
able to extract the cty.Path from a diagnostic if it exists.
In order to properly migrate the contents of resource, data, provider and
provisioner blocks we will need the provider's schema in order to
understand what is expected, so we can resolve some ambiguities inherent
in the legacy HCL AST.
This includes an initial prototype of migrating the content of resource
blocks just to verify that the information is being gathered correctly.
As with the rest of the upgrade_native.go file, this will be reorganized
significantly once the basic end-to-end flow is established and we can
see how to organize this code better.
This is based on c811440188 made against the
old "config" package implementations, but also catches a few other cases
where we would previously have printed the private key into the error
messages.
Previously we would attempt to DynamicExpand during the validate walk and
then validate each expanded instance separately. However, this meant that
we would not be able to validate the contents of a block where count = 0
or if count is not yet known.
Here we instead do a more static validation pass against the resource
configuration itself, setting count.index to cty.UnknownVal(cty.Number) so
we can type-check everything inside the block as being correct regardless
of the final count.
This is another step towards repairing the "validate" command for our
changed assumptions in a world where we have a more sophisticated type
checker.
This doesn't yet address the remaining problem that the expression
evaluator can't, with the current state structures, distinguish between
a completed resource with count = 0 and a resource that doesn't exist
at all (during validate), and so we'll still get errors if an expression
elsewhere in configuration refers to a dynamic index of a resource with
"count" set. That's a pre-existing condition that's no longer being masked
by _this_ problem, but can't be addressed until we've introduced the new
state types (states.State, etc) and thus we _can_ distinguish these two
situations. That will therefore be addressed in a later commit.
Previously we had the evaluate methods accept directly an
addrs.InstanceKey and had our evaluator infer a suitable value for
count.index for it, but that prevents us from setting the index to be
unknown in the validation scenario where we may not be able to predict
the number of instances yet but we still want to be able to check that
the configuration block is type-safe for all possible count values.
To achieve this, we separate the concern of deciding on a value for
count.index from the concern of evaluating it, which then allows for
other implementations of this in future. For the purpose of this commit
there is no change in behavior, with the count.index value being populated
whenever the instance key is a number.
This commit does a little more groundwork for the future implementation
of the for_each feature (which'll support each.key and each.value) but
still doesn't yet implement it, leaving it just stubbed out for the
moment.
Since the intent of the validate command is to check config validity
regardless of context (input variables, state, etc), we use unknown values
of the requested type here, which will then allow us to complete type
checking against the specified types of the variables without assuming
any particular values.
Since schemas are required to interpret provider, resource, and
provisioner attributes in configs, states, and plans, these helpers intend
to make it easier to gather up the the necessary provider types in order
to preload all of the needed schemas before beginning further processing.
Config.ProviderTypes returns directly the list of provider types, since
at this level further detail is not useful: we've not yet run the
provider allocation algorithm, and so the only thing we can reliably
extract here is provider types themselves.
State.ProviderAddrs and Plan.ProviderAddrs each return a list of
absolute provider addresses, which can then be turned into a list of
provider types using the new helper providers.AddressedTypesAbs.
Since we're already using configs.Config throughout core, this also
updates the terraform.LoadSchemas helper to use Config.ProviderTypes
to find the necessary providers, rather than implementing its own
discovery logic. states.State is not yet plumbed in, so we cannot yet
use State.ProviderAddrs to deal with the state but there's a TODO comment
to remind us to update that in a later commit when we swap out
terraform.State for states.State.
A later commit will probably refactor this further so that we can easily
obtain schema for the providers needed to interpret a plan too, but that
is deferred here because further work is required to make core work with
the new plan types first. At that point, terraform.LoadSchemas may become
providers.LoadSchemas with a different interface that just accepts lists
of provider and provisioner names that have been gathered by the caller
using these new helpers.
This allows us to record the resource-to-provider associations we made
during the plan phase and ensure that we use the same config during
apply.
This isn't technically necessary, since we can in principle just repeat
the same matching algorithm against the config, but that algorithm is
relatively complicated and so if we execute it just once during plan we
remove the risk of bugs causing different decisions to be made during
the apply phase.
This also includes updates to the plan file format to include the provider
addresses. Since we've not actually shipped any version of Terraform
using our protobuf schema yet, we renumber here all of the fields after
the new one to keep them incrementing consecutively. This example should
not be followed after we have a released version of Terraform using this,
as a courtesy to anyone trying to parse these files (even though we're
not yet guaranteeing compatibility between releases).
This is the frontend to the work-in-progress codepath for upgrading the
source code for a module written for Terraform v0.11 or earlier to use
the new syntax and idiom of v0.12.
The underlying upgrade code is not yet complete as of this commit, and
so the command is not yet very useful. We will continue to iterate on
the upgrade code in subsequent commits.
This covers all of the expression node types in HIL's AST, and also
includes initial support for some of our top-level blocks so that we can
easily test that.
The initial implementations of the "variable" and "output" blocks are
pretty redundant and messy, so we can hopefully improve on these in a
later pass.
This function is the main functionality of this package. So far it just
deals with detecting and renaming JSON files that are mislabeled as
native syntax files. Other functionality will follow in later commits.
This package will do all of its work in-memory so that it can avoid making
partial updates and then failing, so we need to be able to load the
sources files from a particular directory into memory.
The upgrade process isn't idempotent, so we also attempt to detect
heuristically whether an upgrade has already been performed (can parse
with the new parser and has a version constraint that prevents versions
earlier than 0.12) so that the CLI tool that will eventually wrap this
will be able to produce a warning and prompt for confirmation in that
case.
Although the new HCL implementation and configuration loader is broadly
compatible with the prior implementation, it has a number of new idiomatic
forms and it also cannot parse some more extreme non-idiomatic usages
that were possible under the old parser.
To help users migrate to the new implementation, this package will rewrite
configuration to comply with the new idiom and fix as many cases as
possible where the legacy parser was too liberal or exposed implementation
details.
It is common for the same module source package to be referenced multiple
times in the same configuration, either because there are literally
multiple instances of the same module source or because a single package
(or repository) contains multiple modules in sub-directories and many
of them are referenced.
To optimize this, here we introduce a simple caching behavior where the
module installer will detect if it's asked to install multiple times from
the same source and produce the second and subsequent directories by
copying the first, rather than by downloading again over the network.
This optimization is applied once all of the go-getter detection has
completed and sub-directory portions have been trimmed, so it is also
able to normalize differently-specified source addresses that all
ultimately detect to the same resolved address. When installing, we
always extract the entire specified package (or repository) and then
reference the specified sub-directory, so we can safely re-use existing
directories when the base package is the same, even if the sub-directory
is different.
However, as a result we do not yet address the fact that the same package
will be stored multiple times _on disk_, which may still be problematic
when referencing large repositories multiple times in
disk-storage-constrained environments. We could address this in a
subsequent change by investigating the use of symlinks where possible.
Since the Registry installer is implemented just as an extra resolution
step in front of go-getter, this optimization applies to registry
modules too. This does not apply to local relative references, which will
continue to just resolve into the already-prepared directory of their
parent module.
The cache of previously installed paths lives only for the duration of
one call to InstallModules, so we will never re-use directories that
were created by previous runs of "terraform init" and there is no risk
that older versions will pollute the cache when attempting an upgrade
from a source address that doesn't explicitly specify a version.
No additional tests are added here because the existing module installer
tests (when TF_ACC=1) already cover the case of installing multiple
modules from the same source.
The ImportResourceStateResponse command can return mutliple types.
ImportedResource maps the state to the appropriate type name, and also
allows providers to return Private data for each resource.
Because we gather together diagnostics from many different parts of the
codebase, the list often ends up being in a non-ideal order. Here we
define a partial ordering for diagnostics that should hopefully make them
easier to scan when many are present, by grouping together diagnostics
that are of the same severity and belong to the same file.
We use sort.Stable here because we have a partial order and so we need
to make sure that diagnostics that do not have a relative ordering will
remain in their original order.
This sorting is applied just in time before rendering the diagnostics
in command.Meta.showDiagnostics.
This adds the new interface for the resource providers. The methods here
closely follow the grpc service methods, using the types expected in
terraform core
This adds the interface for the new resource provisioners.
The methods closely follow the grpc service methods, using the type
expected within terraform core.
The new format is radically different in than the old in physical
structure, but still has the same logical parts: the plan itself, a
snapshot of the input configuration, and a snapshot of the state as it
existed when the plan was created.
Rather than creating plan-specific serializations of state and config, the
new format instead leans on the existing file formats implemented
elsewhere, wrapping the result up in a zip archive with some internal file
naming conventions.
The plan portion of the file is serialized with protobuf, consistent with
our general strategy of replacing all use of encoding/gob with protobuf
moving forward.
The types in this package are intended to replace plan- and diff-related
types from the "terraform" package, although those older types must remain
for now so that they can be used to implement shims for older codepaths.
type "Changes" is approximately equivalent to terraform.Diff, but renamed
since it now describes whole objects before and after rather than an
attribute-level diff as before. The term "diff" is now reserved for the
visual rendition of the changes we'll display to the user, although
rendering of this new Changes model is not yet implemented.
Here we introduce a new idea of a "configuration snapshot", which is an
in-memory copy of the source code of each of the files that make up
the configuration. The primary intended purpose for this is as an
intermediate step before writing the configuration files into a plan file,
and then reading them out when that plan file is later applied.
During earlier configs package development we expected to use an afero vfs
implementation to read directly from the zip file, but that doesn't work
in practice because we need to preserve module paths from the source file
system that might include parent directory traversals (../) while
retaining the original path for use in error messages.
The result, for now, is a bit of an abstraction inversion: we implement
a specialized afero vfs implementation that makes the sparse filesystem
representation from a snapshot appear like a normal filesystem just well
enough that the config loader and parser can work with it.
In future we may wish to rework the internals here so that the main
abstraction is at a similar level to the snapshot and then that API is
mapped to the native filesystem in the normal case, removing afero. For
now though, this approach avoids the need for a significant redesign
of the parser/loader internals, at the expense of some trickiness in the
case where we're reading from a snapshot.
This commit does not yet include the reading and writing of snapshots into
plan files. That will follow in a subsequent commit.
The existing cty packages were already at the latest version, but we were
not yet vendoring the msgpack package.
This also imports some dependencies from:
github.com/vmihailenco/msgpack
This is a wrapper around State that is able to perform higher-level
manipulations (at the granularity of the entire state) in a
concurrency-safe manner, using the lower-level APIs exposed by State and
all of the types it contains.
The granularity of a SyncState operation roughly matches the granularity
off a state-related EvalNode in the "terraform" package, performing a
sequence of more primitive operations while guaranteeing atomicity of the
entire change.
As a compromise for convenience of usage, it's still possible to access
the individual state data objects via this API, but they are always copied
before returning to ensure that two distinct callers cannot have data
races. Callers should access the most granular object possible for their
operation.
This idea of a "state manager" was previously modelled via the
confusingly-named state.State interface, which we've been calling a "state
manager" only in some local variable names in situations where there were
also *terraform.State variables.
As part of reworking our state models to make room for the new type
system, we also need to change what was previously the state.StateReader
interface. Since we've found the previous organization confusing anyway,
here we just copy all of those interfaces over into statemgr where we can
make the relationship to states.State hopefully a little clearer.
This is not yet a complete move of the functionality from "state", since
we're not yet ready to break existing callers. In a future commit we'll
turn the interfaces in the old "state" package into aliases of the
interfaces in this package, and update all the implementers of what will
by then be statemgr.Reader to use *states.State instead of
*terraform.State.
This also includes an adaptation of what was previously state.LocalState
into statemgr.FileSystem, using the new state serialization functionality
from package statefile instead of the old terraform.ReadState and
terraform.WriteState.
Whereas the parent directory "states" contains the models that represent
state in memory, this package's responsibility is in serializing a subset
of that data to a JSON-based file format and then reloading that data
back into memory later.
For reading, this package supports state file formats going back to
version 1, using lightly-adapted versions of the migration code previously
used in the "terraform" package. State data is upgraded to the latest
version step by step and then transformed into the in-memory state
representation, which is distinct from any of the file format structs in
this package to enable these to evolve separately.
For writing, only the latest version (4) is supported, which is a new
format that is a slightly-flattened version of the new in-memory state
models introduced in the prior commit. This format retains the outputs
from only the root module and it flattens out the module and instance
parts of the hierarchy by including the identifiers for these inside
the child object. The loader then reconstructs the multi-layer structure
we use for more convenient access in memory.
For now, the only testing in this package is of round-tripping different
versions of state through a read and a write, ensuring the output is
as desired. This exercises all of the reading, upgrading, and writing
functions but should be augmented in later commits to improve coverage
and introduce more focused tests for specific parts of the functionality.
Our previous state models in the "terraform" package had a few limitations
that are addressed here:
- Instance attributes were stored as map[string]string with dot-separated
keys representing traversals through a data structure. Now that we have
a full type system, it's preferable to store it as a real data
structure.
- The existing state structures skipped over the "resource" concept and
went straight to resource instance, requiring heuristics to decide
whether a particular resource should appear as a single object or as
a list of objects when used in configuration expressions.
- Related to the previous point, the state models also used incorrect
terminology where "ResourceState" was really a resource instance state
and "InstanceState" was really the state of a particular remote object
associated with an instance. These new models use the correct names for
each of these, introducing the idea of a "ResourceInstanceObject" as
the local record of a remote object associated with an instance.
This is a first pass at fleshing out a new model for state. Undoubtedly
there will be further iterations of this as we work on integrating these
new models into the "terraform" package.
These new model types no longer serve double-duty as a description of the
JSON state file format, since they are for in-memory use only. A
subsequent commit will introduce a separate package that deals with
persisting state to files and reloading those files later.
Martin Atkins