As part of integrating the new "remote" backend we relaxed the requirement
that a "default" workspace must exist in all backends and now skip
migrating empty workspace states to avoid creating unnecessary "default"
workspaces when switching between backends that require it and backends
that don't, such as when switching from the local backend (which always
has a "default" workspace) to Terraform Enterprise.
This was failing because we now handle the settings for the local backend
a little differently as a result of decoding it with the HCL2 machinery.
Specifically, the backend.State* fields are now assumed to be what is
given in configuration, and any CLI overrides are maintained separately
in OverrideState* fields so that they can be imposed "just in time" in
StatePaths.
This is particularly important because OverrideStatePath (when set) is
used regardless of workspace name, while StatePath is a suitable value
only for the "default" workspace, with others needing to be constructed
from StateWorkspaceDir instead.
Our new state model has a different implementation of "empty" that doesn't
consider lineage/serial, so we need to have some actual content in these
state fixtures to avoid them being skipped during state migrations.
We previously hacked around the import/export functionality being missing
in the statemgr layer after refactoring, but now it's been reintroduced
to fix functionality elsewhere we should use the centralized Import and
Export functions to ensure consistent behavior.
In particular, this pushes the logic for checking lineage and serial
during push down into the state manager itself, which is better because
all other details about lineage and serial are managed within the state
managers.
This test was initially failing because its fixture had a state which our
new state models consider to be "empty", and thus it was not migrated.
After fixing that (by adding an output to the fixture), this revealed a
bug that the lineage was not being persisted through the migration. This
is fixed by using the statemgr.Migrate method instead of writing via the
normal Writer interface, which allows two cooperating state managers to
properly transfer the lineage and serial along with the state snapshot.
This test was incorrectly updated in a previous iteration, with it
creating a modified state to write but then not actually writing it,
writing an empty test state instead.
This made the test fail because a backup state file is created only if
the new state snapshot is different to the old when written.
Some other test is leaving behind a terraform.tfstate after it concludes,
which can cause this test to fail in a strange way due to picking up
extra provider requirements from that state.
This check doesn't fix that problem, but it at least makes the test fail
in a more helpful way to avoid time wasted trying to debug this test when
it's some other test that actually has the bug.
This test is currently failing due to the command completing successfully,
which would previously cause a panic because we didn't properly initialize
the MockUi and so its error buffer is nil unless written to.
(The failure this was masking will be fixed in a subsequent commit.)
In prior refactoring we lost the required core version check from
"terraform init", which we restore here.
Additionally, this test used to have an incorrect name that suggested it
was testing something in the "getProvider" codepath, but version checking
happens regardless of what other options are selected.
After all of the refactoring we were no longer checking the Terraform
version field in a state file, causing this test to fail.
This restores that check, though with a slightly different error message.
This test was using old-style state files as its input, differing only by
lineage. Since lineages are now managed within the state manager itself,
the test can't use that to distinguish the two files and so we put a
different output in each one instead.
This also introduces some TRACE logging to the migration codepaths.
There's some hard-to-follow control flow here and so this extra logging
helps to understand the reason for a particular outcome, and since this
codepath is visited only in "terraform init" anyway it doesn't hurt to
be a bit more verbose here.
In the refactoring for new HCL this codepath stopped taking into account
changes to the CLI -backend-config options when deciding if a backend
migration is required.
This restores that behavior in a different way than it used to be: rather
than re-hashing the merged config and comparing the hashes, we instead
just compare directly the configuration values, which must be exactly
equal in order to skip migration.
This change is covered by the test TestInit_inputFalse, although as of
this commit it is still not passing due a downstream problem within the
migration code itself.
This test was re-using the same InitCommand value to run multiple times,
which is not realistic. Since we now cache configuration source code
inside command.Meta on load, it's important that we use a fresh
InitCommand instance here so it'll see the modified configuration file
we've left on disk.
Here we were going to the trouble of copying the body so we could mutate
it, but then ended up mutating the original anyway and then returning the
unmodified copy. Whoops!
This fix is verified by a number of "init" command tests that exercise the
-backend-config option, including TestInit_backendConfigFile and several
others whose names have the prefix TestInit_backendConfig .
When we originally wrote this message we struggled a bit for how to refer
to the releases server without writing an awkwardly-ungrammatical
sentence, and so "the official repository" became a placeholder name for
it.
Now that we'll be looking in Terraform Registry this gives us a nice
proper noun to use. This message will need to evolve more as our
integration with the registry gets more sophisticated, but for now this
works.
Some over-zealous bulk updating of this test file caused this test to be
producing a remote state config cache file on disk when it doesn't
actually need one: the backend config comes from the plan file when
applying a saved plan.
Some merging conflict shenanigans here led to this usage not lining up
with the imported symbol name, meaning that the tests couldn't compile any
more.
We missed fixing this up during the big updates for the new plan/state
models since the failures were being masked by testBackendState being
broken.
This is the same sort of update made to many other tests: add schema to
the mock provider, adjust for the new plan/state types, and make
allowances for the new built-in diffing behavior in core.
The hashing function for cached backend configuration is different now, so
our hard-coded hash of the HTTP backend address wasn't working anymore.
Here we update the hash so that tests using this test backend will work
again. Rather than leaving it hard-coded, we'll instead compute it the
same way as "terraform init" would.
In practice only one test is actually using this function right now, so
we also update the test fixture for that test (TestPlan_outBackend) to
match the new expectations, though as of this commit it's still failing
with an unrelated error.
The mission of this process method used to include dealing with
auto-loaded tfvars files, but it doesn't do that anymore.
It does still deal with the -no-color option, but the test wasn't
exercising that part before.
Now the test here focuses on the -no-color behavior.
The process method still has a "vars" flag argument which is no longer
used. Since this is an unexported method we could potentially address this
but this commit is intentionally limited only to fixing the test.
Comments here indicate that this was erroneously returning an error but
we accepted it anyway to get the tests passing again after other work.
The tests over in the "terraform" package agree that cancelling should be
a successful outcome rather than an error.
I think that cancelling _should_ actually be an error, since Terraform did
not complete the operation it set out to complete, but that's a change
we'd need to make cautiously since automation wrapper scripts may be
depending on the success-on-cancel behavior.
Therefore this just fixes the command package test to agree with the
Terraform package tests and adds some FIXME notes to capture the potential
that we might want to update this later.
The State.Equal function is now more precise than this test needs. It's
only trying to distinguish between an empty state and a non-empty state,
so the string representation of state is good enough to get that done
while disregarding other subtle differences.
This new source type should be used for variables loaded from .tfvars files that were explicitly passed as command line arguments (e.g. -var-file=foo.tfvars)
Without using absolute paths any module info is lost in the output. And the attributes were randomly ordered and so changed between different executions of the command.
When HCL encounters an error during expression evaluation, it annotates
its diagnostics with information about the expression that was being
evaluated and the EvalContext it was evaluated in.
This gives us enough information to show helpful hints to the user about
the final values of any reference expressions that are present in the
expression, which is very useful extra context for expressions that get
evaluated multiple times, such as:
- Any expression in a block with "count" or "for_each" set
- The sub-expressions within a "for" expression
This work was done against APIs that were already changed in the branch
before work began, and so it doesn't apply to the v0.12 development work.
To allow v0.12 to merge down to master, we'll revert this work out for now
and then re-introduce equivalent functionality in later commits that works
against the new APIs.
This reinstates an old behavior that was lost in the reorganization of how
we deal with the -var and -var-file options.
This fix is verified by TestApply_planVars now passing.
In the new implementation of collecting variables I initially forgot the
JSON variant of terraform.tfvars.
This fix is verified by TestApply_varFileDefaultJSON now passing.
This was previously targeting the old state manager and state types, so it
needed some considerable rework to get it working again with the new state
types.
Since our new resource address syntax lacks the weird extra .deposed
special case we had before, we instead interpret addresses as
whole-instance addresses here and remove the deposed objects along with
the current one (if present), since this is more likely to match with
user expectations because we don't consider deposed objects to be
independently addressable in any other situation.
With that said, to be more explicit about what is going on we do now have
a -dry-run mode and maintain separate counts of current and deposed
instances so that we can expose that in the UI where relevant.
We temporarily disabled this because it needed some further work to update
it for the new state models, which has now been done.
We no longer need the configuration objects for the outputs because the
state itself contains all of the information needed for displaying these.
We used to treat the "id" attribute of a resource as special and elevate
it into its own struct field "ID" in the state, but the new state format
and provider protocol treats it just as any other attribute.
However, it's still useful to show the value of a single identifying
attribute when there isn't room in the UI for showing all of the
attributes, and so here we take a new strategy of considering "id" along
with some other conventional names as special only in the UI layer.
This new heuristic approach can be adjusted over time as new provider
patterns emerge, but for now it covers some common conventions we've seen
in real providers.
With that said, since all existing providers made for Terraform versions
prior to v0.12 were forced to set "id", we won't see any use of other
attributes here until providers are updated to remove the placeholder
ids they were generating in cases where an id was not actually relevant
but was forced by the old protocol. At that point the UX should be
improved by showing a more relevant attribute instead.
We now also allow for the possibility of no id at all, since that is valid
for resources that exist only within the Terraform state, like the ones
from the "random" and "tls" providers.
This command isn't yet updated for the new state types, but since we were
not returning a non-successful error status here the tests were just
failing in a weird way instead. Now we'll fail with a message that makes
it clear there is still work to do in the real implementation here.
We previously stubbed most of this out because it hadn't yet been updated
to support the new state types, etc.
This restores all of the previous behavior as covered by the tests.
We intentionally remove one behavior that was not covered by the tests:
we used to allow retrieval of outputs from non-root modules using the
-module option, but since we no longer persist non-root outputs in the
state we can no longer support this without a full expression evaluation
walk, and that'd be overkill for this otherwise-simple command. Descendant
module outputs are not part of the public interface of a configuration
anyway, so accessing them from outside in this way is an anti-pattern.
(For debugging scenarios it is still possible to access these from
"terraform console", which _does_ do a full evaluation graph walk to
prepare its evaluation scope.)
The plan file writer requires a backend config to be present, but we don't
really need one for the sake of _this_ test, since we don't activate the
backend to render a plan graph, and so we just write in a placeholder.
This connects a missing link left by earlier refactoring: the command
package is responsible for gathering up variable values provided by the
user and passing them through to the backend to use in operations.
Our serialization of the backend configuration has changed slightly for
Terraform 0.12 due to reimplementing it in terms of the HCL2 types, so
the base case that should be unchanged during the test needs to be
changed.
In all real cases the schemas should be populated here, but we don't want
to panic in UI rendering code if there's a bug here.
This can also be tripped up by tests with incomplete mocks. It's
unfortunate that this can therefore mask some problems in tests, but tests
can protect against it by asserting on specific output text rather than
just assuming that a zero exit status is a pass.
If we fail to parse the resource address given to "terraform import" then
it's helpful to produce a "source code" snippet of what the user provided
so they might see more precisely which part of the address was invalid.
Most of this is just updates to allow for the fact that we now always save
the provider address as part of resource state, whereas before it was only
saved conditionally.
This also updates TestTaint_module for the intentional change that it now
expects a child module to be specified using normal resource address
syntax, rather than as a separate -module option.
Added a very simple test with state and schema.
TODO: if tests are added we should test using golden files (and example
state files, instead of strings). This seemed unnecessary with the
simple test cases.
In previous work we didn't quite connect these dots. The connection here
is sub-awesome since the existing interfaces here had some unfortunate
assumptions that we'd like to move away from (like the idea of a "nil
backend" implying the local backend) but we're accepting this for now to
avoid another big round of refactoring.
The main implication of this is that we will now always include a backend
configuration in the plan, though it might just be a placeholder config
for the local backend in the remaining cases where that's still implicitly
set. Later we will change this so that there is no implicit local backend
at all (terraform init is always required, _it_ will deal with setting
implicitly setting the local backend when appropriate), which will allow
us to rework this to be more straightforward and less "spooky".
If we don't do this, we can't produce any output when applying a saved
plan file.
Here we also introduce a check to the local backend's ReportResult
function so that it won't panic if CLI init is skipped, although that
will no longer happen in the apply-from-file case due to the change
described in the previous paragraph.
It must now provide a basic implementation of plan and apply for its
mock provider, which in this case can just pass through the proposed value
generated by core because there are no computed attributes in this schema.
Previously we used a single plan action "Replace" to represent both the
destroy-before-create and the create-before-destroy variants of replacing.
However, this forces the apply graph builder to jump through a lot of
hoops to figure out which nodes need it forced on and rebuild parts of
the graph to represent that.
If we instead decide between these two cases at plan time, the actual
determination of it is more straightforward because each resource is
represented by only one node in the plan graph, and then we can ensure
we put the right nodes in the graph during DiffTransformer and thus avoid
the logic for dealing with deposed instances being spread across various
different transformers and node types.
As a nice side-effect, this also allows us to show the difference between
destroy-then-create and create-then-destroy in the rendered diff in the
CLI, although this change doesn't fully implement that yet.
For PreApply hook purposes we only actually use the Delete, Create, and
Update actions, because other actions are handled in different ways than
a direct call to ApplyResourceChange.
However, if there's a bug in core that causes it to pass a different
action, it's better for us to mark it as being explicitly unknown in the
UI rather than simply defaulting to "Modifying...", which can thus obscure
the problem and make for a confusing result.
We'll now show an "update" symbol prior to the argument to this synthetic
jsonencode(...) call, for consistency with how we show nested values in
other cases and to attach a verb to any "# forces replacement".
We'll also show a special form in the case where the value seems to differ
only in whitespace, so users can understand what's going on in that
hopefully-rare situation, particularly if those whitespace-only changes
end up forcing us to replace a remote object.
Since our own syntax for primitive values is similar to that of JSON, and
since we permit automatic conversions from number and bool to string, we
must do this special JSON value diff formatting only if the value is a
JSON array or object to avoid confusing results.
Because so far we've not supported dynamically-typed complex data
structures, several providers have used strings containing JSON to stand
in for these.
In order to get a readable diff in those cases, we'll recognize situations
where old and new are both JSON and present a diff of the effective value
of the JSON, using a faux call to the jsonencode(...) function to indicate
when we've done so.
This is a bit of a "cute" heuristic, but is important at least for now
until we can migrate away from that practice of passing large JSON strings
to providers and use dynamically-typed attributes instead.
This extra comment line gives us a place to show the full resource address
(since the block header line only includes type and name) and also allows
us to explain in long form the meaning of the change icon on the following
line.
This is a light adaptation of our earlier prototype of structural diff
rendering, as a starting point for what we'll actually ship. This is not
consistent with the latest mocks, so will need some additional work before
it is ready, but integrating this allows us to at least see the plan
contents while fixing up remaining issues elsewhere.
Previously we just left these out of the plan altogether, but in the new
plan types we intentionally include change information for every resource
instance, even if no changes are actually planned, to allow alternative
plan file viewers to show what isn't changing as well as what is.
This codepath is going to be significantly changed before release to make
it support structural diff of the new data types, but this lets us lean on
the old renderer to produce partial output in the mean time while we
continue to work on getting things working end-to-end after the
considerable refactoring that's been going on.
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.
The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.
The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.
Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
The "config" package is no longer used and will be removed as part
of the 0.12 release cleanup. Since configschema is part of the
"new world" of configuration modelling, it makes more sense for
it to live as a subdirectory of the newer "configs" package.
In order to properly migrate the contents of resource, data, provider and
provisioner blocks we will need the provider's schema in order to
understand what is expected, so we can resolve some ambiguities inherent
in the legacy HCL AST.
This includes an initial prototype of migrating the content of resource
blocks just to verify that the information is being gathered correctly.
As with the rest of the upgrade_native.go file, this will be reorganized
significantly once the basic end-to-end flow is established and we can
see how to organize this code better.
Since the intent of the validate command is to check config validity
regardless of context (input variables, state, etc), we use unknown values
of the requested type here, which will then allow us to complete type
checking against the specified types of the variables without assuming
any particular values.
This is the frontend to the work-in-progress codepath for upgrading the
source code for a module written for Terraform v0.11 or earlier to use
the new syntax and idiom of v0.12.
The underlying upgrade code is not yet complete as of this commit, and
so the command is not yet very useful. We will continue to iterate on
the upgrade code in subsequent commits.
Because we gather together diagnostics from many different parts of the
codebase, the list often ends up being in a non-ideal order. Here we
define a partial ordering for diagnostics that should hopefully make them
easier to scan when many are present, by grouping together diagnostics
that are of the same severity and belong to the same file.
We use sort.Stable here because we have a partial order and so we need
to make sure that diagnostics that do not have a relative ordering will
remain in their original order.
This sorting is applied just in time before rendering the diagnostics
in command.Meta.showDiagnostics.
This doesn't yet include test updates, since there are problems in core
currently blocking these tests from running. The tests will therefore be
updated in a subsequent commit.
Previously we were defaulting the provider configuration selection to a
provider in the root module inferred from the resource type name.
This is close, but not quite right: we need to _start_ with a provider
configuration in the same module as we're importing into, and then our
provider resolution steps during import graph construction will use that
as a starting point for a walk up the tree to find the nearest matching
configuration (which might eventually still be in the root, but not
necessarily).
This now uses the HCL2 parser and evaluator APIs and evaluates in terms
of a new-style *lang.Scope, rather than the old terraform.Interpolator
type that is no longer functional.
The Context.Eval method used here behaves differently than the
Context.Interpolater method used previously: it performs a graph walk
to populate transient values such as input variables, local values, and
output values, and produces its scope in terms of the result of that
graph walk. Because of this, it is a lot more robust than the prior method
when asked to resolve references other than those that are persisted
in the state.
Previously an empty diagnostics would appear as "null" in the JSON output,
since that is how encoding/json serializes a nil slice. It's more
convenient for users of dynamic languages to keep the type consistent
in all cases, since they can then just iterate the list without needing a
special case for when it is null.
Due to how deeply the configuration types go into Terraform Core, there
isn't a great way to switch out to HCL2 gradually. As a consequence, this
huge commit gets us from the old state to a _compilable_ new state, but
does not yet attempt to fix any tests and has a number of known missing
parts and bugs. We will continue to iterate on this in forthcoming
commits, heading back towards passing tests and making Terraform
fully-functional again.
The three main goals here are:
- Use the configuration models from the "configs" package instead of the
older models in the "config" package, which is now deprecated and
preserved only to help us write our migration tool.
- Do expression inspection and evaluation using the functionality of the
new "lang" package, instead of the Interpolator type and related
functionality in the main "terraform" package.
- Represent addresses of various objects using types in the addrs package,
rather than hand-constructed strings. This is not critical to support
the above, but was a big help during the implementation of these other
points since it made it much more explicit what kind of address is
expected in each context.
Since our new packages are built to accommodate some future planned
features that are not yet implemented (e.g. the "for_each" argument on
resources, "count"/"for_each" on modules), and since there's still a fair
amount of functionality still using old-style APIs, there is a moderate
amount of shimming here to connect new assumptions with old, hopefully in
a way that makes it easier to find and eliminate these shims later.
I apologize in advance to the person who inevitably just found this huge
commit while spelunking through the commit history.
For the moment this is just a lightly-adapted copy of
ModuleTreeDependencies named ConfigTreeDependencies, with the goal that
the two can live concurrently for the moment while not all callers are yet
updated and then we can drop ModuleTreeDependencies and its helper
functions altogether in a later commit.
This can then be used to make "terraform init" and "terraform providers"
work properly with the HCL2-powered configuration loader.
This is a rather-messy, complex change to get the "command" package
building again against the new backend API that was updated for
the new configuration loader.
A lot of this is mechanical rewriting to the new API, but
meta_config.go and meta_backend.go in particular saw some major
changes to interface with the new loader APIs and to deal with
the change in order of steps in the backend API.
The new config loader requires some steps to happen in a different
order, particularly in regard to knowing the schema in order to
decode the configuration.
Here we lean directly on the configschema package, rather than
on helper/schema.Backend as before, because it's generally
sufficient for our needs here and this prepares us for the
helper/schema package later moving out into its own repository
to seed a "plugin SDK".
The remote API this talks to will be going away very soon, before our next
major release, and so we'll remove the command altogether in that release.
This also removes the "encodeHCL" function, which was used only for
adding a .tfvars-formatted file to the uploaded archive.
In the long run we'd like to offer machine-readable output for more
commands, but for now we'll just start with a tactical feature in
"terraform validate" since this is useful for automated testing scenarios,
editor integrations, etc, and doesn't include any representations of types
that are expected to have breaking changes in the near future.
As part of some light reorganization of our commands, this new
implementation no longer does validation of variables and will thus avoid
the need to spin up a fully-valid context. Instead, its focus is on
validating the configuration itself, regardless of any variables, state,
etc.
This change anticipates us later adding a -validate-only flag to
"terraform plan" which will then take over the related use-case of
checking if a particular execution of Terraform is valid, _including_ the
state, variables, etc.
Although leaving variables out of validate feels pretty arbitrary today
while all of the variable sources are local anyway, we have plans to
allow per-workspace variables to be stored in the backend in future and
at that point it will no longer be possible to fully validate variables
without accessing the backend. The "terraform plan" command explicitly
requires access to the backend, while "terraform validate" is now
explicitly for local-only validation of a single module.
In a future commit this will be extended to do basic type checking of
the configuration based on provider schemas, etc.
We need to share a single config loader across all callers because that
allows us to maintain the source code cache we'll use for snippets in
error messages.
Nothing calls this yet. Callers will be gradually updated away from Module
and Config in subsequent commits.
If we get a diagnostic message that references a source range, and if the
source code for the referenced file is available, we'll show a snippet of
the source code with the source range highlighted.
At the moment we have no cache of source code, so in practice this
codepath can never be visited. Callers to format.Diagnostic will be
gradually updated in subsequent commits.
In some cases this is needed to keep the UX clean and to make sure any remote exit codes are passed through to the local process.
The most obvious example for this is when using the "remote" backend. This backend runs Terraform remotely and stream the output back to the local terminal.
When an error occurs during the remote execution, all the needed error information will already be in the streamed output. So if we then return an error ourselves, users will get the same errors twice.
By allowing the backend to specify the correct exit code, the UX remains the same while preserving the correct exit codes.
Certain backends (currently only the `remote` backend) do not support using both the default and named workspaces at the same time.
To make the migration easier for users that currently use both types of workspaces, this commit adds logic to ask the user for a new workspace name during the migration process.
Martin Atkins