When using the enhanced remote backend, a subset of all Terraform
operations are supported. Of these, only plan and apply can be executed
on the remote infrastructure (e.g. Terraform Cloud). Other operations
run locally and use the remote backend for state storage.
This causes problems when the local version of Terraform does not match
the configured version from the remote workspace. If the two versions
are incompatible, an `import` or `state mv` operation can cause the
remote workspace to be unusable until a manual fix is applied.
To prevent this from happening accidentally, this commit introduces a
check that the local Terraform version and the configured remote
workspace Terraform version are compatible. This check is skipped for
commands which do not write state, and can also be disabled by the use
of a new command-line flag, `-ignore-remote-version`.
Terraform version compatibility is defined as:
- For all releases before 0.14.0, local must exactly equal remote, as
two different versions cannot share state;
- 0.14.0 to 1.0.x are compatible, as we will not change the state
version number until at least Terraform 1.1.0;
- Versions after 1.1.0 must have the same major and minor versions, as
we will not change the state version number in a patch release.
If the two versions are incompatible, a diagnostic is displayed,
advising that the error can be suppressed with `-ignore-remote-version`.
When this flag is used, the diagnostic is still displayed, but as a
warning instead of an error.
Commands which will not write state can assert this fact by calling the
helper `meta.ignoreRemoteBackendVersionConflict`, which will disable the
checks. Those which can write state should instead call the helper
`meta.remoteBackendVersionCheck`, which will return diagnostics for
display.
In addition to these explicit paths for managing the version check, we
have an implicit check in the remote backend's state manager
initialization method. Both of the above helpers will disable this
check. This fallback is in place to ensure that future code paths which
access state cannot accidentally skip the remote version check.
The short description of our commands (as shown in the main help output
from "terraform") was previously very inconsistent, using different
tense/mood for different commands. Some of the commands were also using
some terminology choices inconsistent with how we currently talk about
the related ideas in our documentation.
Here I've tried to add some consistency by first rewriting them all in
the imperative mood (except the ones that just are just subcommand
groupings), and tweaking some of the terminology to hopefully gel better
with how we present similar ideas in our recently-updated docs.
While working on this I inevitably spotted some similar inconsistencies
in the longer-form help output of some of the commands. I've not reviewed
all of these for consistency, but I did update some where the wording
was either left inconsstent with the short form changes I'd made or
where the prose stood out to me as particularly inconsistent with our
current usual documentation language style.
All of this is subjective, so I expect we'll continue to tweak these over
time as we continue to develop our documentation writing style based on
user questions and feedback.
Terraform Core expects all variables to be set, but for some ancillary
commands it's fine for them to just be set to placeholders because the
variable values themselves are not key to the command's functionality
as long as the terraform.Context is still self-consistent.
For such commands, rather than prompting for interactive input for
required variables we'll just stub them out as unknowns to reflect that
they are placeholders for values that a user would normally need to
provide.
This achieves a similar effect to how these commands behaved before, but
without the tendency to produce a slightly invalid terraform.Context that
would fail in strange ways when asked to run certain operations.
Any command using meta.defaultFlagSet *might* occasionally exit before
the flag package's output got written. This caused flag error messages
to get lost. This PR discards the flag package output in favor of
directly returning the error to the end user.
A lot of commands used `c.Meta.flagSet()` to create the initial flagset for the command, while quite a few of them didn’t actually use or support the flags that are then added.
So I updated a few commands to use `flag.NewFlagSet()` instead to only add the flags that are actually needed/supported.
Additionally this prevents a few commands from using locking while they actually don’t need locking (as locking is enabled as a default in `c.Meta.flagSet()`.
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.
The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.
The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.
Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.
The three main goals here are:
- Use the configuration models from the "configs" package instead of the
older models in the "config" package, which is now deprecated and
preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
new "lang" package, instead of the Interpolator type and related
functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
rather than hand-constructed strings. This is not critical to support
the above, but was a big help during the implementation of these other
points since it made it much more explicit what kind of address is
expected in each context.
Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.
I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
This is a rather-messy, complex change to get the "command" package
building again against the new backend API that was updated for
the new configuration loader.
A lot of this is mechanical rewriting to the new API, but
meta_config.go and meta_backend.go in particular saw some major
changes to interface with the new loader APIs and to deal with
the change in order of steps in the backend API.
The state locking improvements for the regular command had the side
effect of locking the state in the console, import, graph and push
commands. Those commands had been updated to get a state via the
Backend.Context method, which locks the state whenever possible, and now
need to call Unlock directly.
Add Unlock calls to all commands that call Context directly.
Previously we required callers to separately call .Validate on the root
module to determine if there were any value errors, but we did that
inconsistently and would thus see crashes in some cases where later code
would try to use invalid configuration as if it were valid.
Now we run .Validate automatically after config loading, returning the
resulting diagnostics. Since we return a diagnostics here, it's possible
to return both warnings and errors.
We return the loaded module even if it's invalid, so callers are free to
ignore returned errors and try to work with the config anyway, though they
will need to be defensive against invalid configuration themselves in
that case.
As a result of this, all of the commands that load configuration now need
to use diagnostic printing to signal errors. For the moment this just
allows us to return potentially-multiple config errors/warnings in full
fidelity, but also sets us up for later when more subsystems are able
to produce rich diagnostics so we can show them all together.
Finally, this commit also removes some stale, commented-out code for the
"legacy" (pre-0.8) graph implementation, which has not been available
for some time.
Instead of providing the a path in BackendOpts, provide a loaded
*config.Config instead. This reduces the number of places where
configuration is loaded.
The dot format generation was done with a mix of code from the terraform
package and the dot package. Unify the dot generation code, and it into
the dag package.
Use an intermediate structure to allow a dag.Graph to marshal itself
directly. This structure will be ablt to marshal directly to JSON, or be
translated to dot format. This was we can record more information about
the graph in the debug logs, and provide a way to translate those logged
structures to dot, which is convenient for viewing the graphs.
This means that terraform commands like `plan`, `apply`, `show`, and
`graph` will expand all modules by default.
While modules-as-black-boxes is still very true in the conceptual design
of modules, feedback on this behavior has consistently suggested that
users would prefer to see more verbose output by default.
The `-module-depth` flag and env var are retained to allow output to be
optionally limited / summarized by these commands.
When you specify `-verbose` you'll get the whole graph of operations,
which gives a better idea of the operations terraform performs and in
what order.
The DOT graph is now generated with a small internal library instead of
simple string building. This allows us to ensure the graph generation is
as consistent as possible, among other benefits.
We set `newrank = true` in the graph, which I've found does just as good
a job organizing things visually as manually attempting to rank the nodes
based on depth.
This also fixes `-module-depth`, which was broken post-AST refector.
Modules are now expanded into subgraphs with labels and borders. We
have yet to regain the plan graphing functionality, so I removed that
from the docs for now.
Finally, if `-draw-cycles` is added, extra colored edges will be drawn
to indicate the path of any cycles detected in the graph.
A notable implementation change included here is that
{Reverse,}DepthFirstWalk has been made deterministic. (Before it was
dependent on `map` ordering.) This turned out to be unnecessary to gain
determinism in the final DOT-level implementation, but it seemed
a desirable enough of a property that I left it in.
Most CBD-related cycles include destroy nodes, and destroy nodes were
all being pruned from the graph before staring the Validate walk.
In practice this meant that we had scenarios that would error out with
graph cycles on Apply that _seemed_ fine during Plan.
This introduces a Verbose option to the GraphBuilder that tells it to
generate a "worst-case" graph. Validate sets this to true so that cycle
errors will always trigger at this step if they're going to happen.
(This Verbose option will be exposed as a CLI flag to `terraform graph`
in a second incoming PR.)
refs #1651
Alisdair McDiarmid