* configs/configupgrade: detect possible relative module sources
If a module source appears to be a relative local path but does not have
a preceding ./, print a #TODO message for the user.
* internal/initwd: limit go-getter detectors to those supported by terraform
* internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter
To avoid making two calls to getter.Detect, which potentially makes
non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to
a later step and a custom error type was added so user-friendly
diagnostics could be displayed in the event that a possible relative local
path was detected.
Our initial prototype of new-style diff rendering excluded this because
the old SDK has no support for this construct. However, we want to be able
to introduce this construct in the new SDK without breaking compatibility
with existing versions of Terraform Core, so we need to implement it now
so it's ready to be used once the SDK implements it.
The key associated with each block allows us to properly correlate the
items to recognize the difference between an in-place update of an
existing block and the addition/deletion of a block.
Our null-to-empty normalization was previously assuming these would always
be collection types, but that isn't true when a block contains something
dynamic since we must then use tuple or object types instead to properly
represent all of the individual element types.
We use cty a little differently when a nested list block contains a
dynamically-typed attribute: it appears as a tuple value instead of a
list value so that we can retain the individual types of each element.
Here we introduce a test for that case, but doing so required also making
the runTestCases function handle types in a stricter way so that it will
produce planned values that match how Terraform Core would do it,
including the necessary late-bound type information for the
dynamically-typed attribute.
Previously, these commands were not checking if the user specified a
`-plugin-dir` flag during `terraform init` and would therefor fail if
providers were not in one of the standard directories.
Fixes#20547
When the user aborts input, it may end up as an unknown value, which
needs to be converted to null for PrepareConfig.
Allow PrepareConfig to accept null config values in order to fill in
missing defaults.
When a planfile is supplied to the `terraform show -json` command, the
context that loads only included schemas for resources in the plan. We
found an edge case where removing a data source from the configuration
(though only if there are no managed resources from the same provider)
would cause jsonstate.Marshal to fail because the provider schema wasn't
in the plan context.
jsonplan.Marshal now takes two schemas, one for plan and one for state.
If the state schema is nil it will simply use the plan schemas.
* command/show: fixing bugs in modulecalls
jsonconfig and jsonplan both had subtle bugs with the logic for
marshaling module calls that only showed up when multiple modules were
referenced. This PR fixes those bugs and extends the existing tests to
include multiple modules.
* sort all the things, mostly for tests
* docs: update plan command documentation. Fixes#19235
* docs: added a missing reserved variable name. Fixes#19159.
* website: add note that resource names cannot start with a number
* website: add some notes to the 0.12 upgrade guide
We are now allowing the legacy SDK to opt out of the safety checks we try
to do after plan and apply, and so in such cases the before/after values
in planned changes may be inconsistent with our usual rules.
To avoid adding lots of extra complexity to the diff renderer to deal with
these situations, instead we'll normalize the handling of nested blocks
prior to using these values.
In the long run it'd be better to do this normalization at the source,
immediately after we receive an object from a provider using the opt-out,
but we're doing this at the outermost layer for now to avoid risking
unintended impacts on other Terraform Core components when we're just
about to enter the beta phase of the v0.12.0 release cycle.
This mirrors the change made for providers, so that default values can
be inserted into the config by the backend implementation. This is only
the interface and method name changes, it does not yet add any default
values.
We brought forward a new implementation of "terraform validate" that was
originally scheduled for a later release after finding that it would be
simpler than reworking the old implementation for new v0.12 assumptions,
but we didn't yet implement "terraform plan -validate-only" in spite of
it being mentioned in the updated docs for "terraform validate".
For now then, the documentation will make the weaker suggestion of running
"terraform plan" to validate a particular _run_ rather than a particular
_module_, which is the closest thing we have for now. At some point after
v0.12.0 we will evaluate whether a validate-only mode for "terraform plan"
(which could then run without configuring the providers at all) is needed.
A common new-user mistake is to place variable _declarations_ into .tfvars
files instead of variable _values_. To guide towards the correct approach
here, we add a specialized error message for that situation that includes
guidance on the distinction between declaring and setting values for
variables, and an example of what setting a value should look like.
* command/jsonconfig: provider config marshaling enhancements
This PR fixes a bug wherein the keys in "provider_config" were the
"addrs.ProviderConfig", and therefore being overwritten for each module,
instead of the intended "addrs.AbsProviderConfig".
We realized that there was still opportunity for ambiguity, for example
if a user made a provider alias that was the same name as a module, so
we opted to use the syntax `modulename:providername(.provideralias)`
* command/json*: fixed a bug where we were attempting to lookup schemas
with the provider name, instead of provider type.
* command/show: add "module_version" to "module_calls" in config portion
of `terraform show`.
Also extended the `terraform show -json` test to run `init` so we could
add examples with modules. This does _not_ test the "module_version"
yet, but it _did_ help expose a bug in jsonplan where modules were
duplicated. This is also fixed in this PR.
* command/jsonconfig: rename version to version_constraint and
resolved_source to source.
* command/jsonconfig: display module variables in config output
The tests have been updated to reflect this change.
* command/jsonconfig: properly handle variables with nil defaults
Now that we're actually verifying correct behavior of providers during
plan and apply, our mock providers need to behave like real providers,
properly propagating any configured values through the plan and into the
final state.
For most of these it was simpler to just switch over to using the newer
PlanResourceChangeFn mock interface, away from the legacy DiffFn approach,
because then we can just return the ProposedNewState verbatim because our
schema for these tests does not require any default values to be
populated.
* command/jsonplan:
- add variables to plan output
- print known planned values for resources
Previously, resource attribute values were only displayed if the values
were wholly known. Now we will filter the unknown values out of the
change and print the known values.
* command/jsonstate: added depends_on and tainted fields
* command/show: update tests to reflect added fields
We now require a provider to populate all of its defaults -- including
unknown value placeholders -- during PlanResourceChange. That means the
mock provider for testing "terraform show -json" must now manage the
population of the computed "id" attribute during plan.
To make this logic a little easier, we also change the ApplyResourceChange
implementation to fill in a non-null id, since that makes it easier for
the mock PlanResourceChange to recognize when it needs to populate that
default value during an update.
* command/jsonstate: do not hide SchemaVersion of '0'
* command/jsonconfig: module_calls should be a map
* command/jsonplan: include current terraform version in output
* command/jsonconfig: properly marshal expressions from a module call
Previously this was looking at the root module's variables, instead of
the child module variables, to build the module schema. This fixes that
bug.
* command/show: add support for -json output for state
* command/jsonconfig: do not marshal empty count/for each expressions
* command/jsonstate: continue gracefully if the terraform version is somehow missing from state
* command/jsonplan: sort resources by address
* command/show: extend test case to include resources with count
* command/json*: document resource ordering as consistent but undefined
* command/show: properly marshal attribute values to json
marshalAttributeValues in jsonstate and jsonplan packages was returning
a cty.Value, which json/encoding could not marshal. These functions now
convert those cty.Values into json.RawMessages.
* command/jsonplan: planned values should include resources that are not changing
* command/jsonplan: return a filtered list of proposed 'after' attributes
Previously, proposed 'after' attributes were not being shown if the
attributes were not WhollyKnown. jsonplan now iterates through all the
`after` attributes, omitting those which are not wholly known.
The same was roughly true for after_unknown, and that structure is now
correctly populated. In the future we may choose to filter the
after_unknown structure to _only_ display unknown attributes, instead of
all attributes.
* command/jsonconfig: use a unique key for providers so that aliased
providers don't get munged together
This now uses the same "provider" key from configs.Module, e.g.
`providername.provideralias`.
* command/jsonplan: unknownAsBool needs to iterate through objects that are not wholly known
* command/jsonplan: properly display actions as strings according to the RFC,
instead of a plans.Action string.
For example:
a plans.Action string DeleteThenCreate should be displayed as ["delete",
"create"]
Tests have been updated to reflect this.
* command/jsonplan: return "null" for unknown list items.
The length of a list could be meaningful on its own, so we will turn
unknowns into "null". The same is less likely true for maps and objects,
so we will continue to omit unknown values from those.
We missed this on the initial update pass because this was calling
directly into the module package API rather than going through the Meta
methods that we updated for the new config loader.
m.installModules here is the same method that "terraform init" is using
for this purpose, ensuring the two will behave the same way. This changes
the output a little compared to the old installer, but it still includes
the important information about where each module is coming from.
This possibility was lost in the rewrite to use HCL2, but it's used by
a number of external utilities and text editor integrations, so we'll
restore it here.
Using the stdin/stdout mode is generally preferable for text editor use
since it allows formatting of the in-memory buffer rather than directly
the file on disk, but for editors that don't have support for that sort of
tooling it can be convenient to just launch a single command and directly
modify the on-disk file.
Since the HCL formatter only works with tokens, it can in principle be
called with any input and produce some output. However, when given invalid
syntax it will tend to produce nonsensical results that may drastically
change the input file and be hard for the user to undo.
Since there's no strong reason to try to format an invalid or incomplete
file, we'll instead try parsing first and fail if parsing does not
complete successfully.
Since we talk directly to the HCL API here this is only a _syntax_ check,
and so it can be applied to files that are invalid in other ways as far
as Terraform is concerned, such as using unsupported top-level block types,
resource types that don't exist, etc.
There are a few constructs from 0.11 and prior that cause 0.12 parsing to
fail altogether, which previously created a chicken/egg problem because
we need to install the providers in order to run "terraform 0.12upgrade"
and thus fix the problem.
This changes "terraform init" to use the new "early configuration" loader
for module and provider installation. This is built on the more permissive
parser in the terraform-config-inspect package, and so it allows us to
read out the top-level blocks from the configuration while accepting
legacy HCL syntax.
In the long run this will let us do version compatibility detection before
attempting a "real" config load, giving us better error messages for any
future syntax additions, but in the short term the key thing is that it
allows us to install the dependencies even if the configuration isn't
fully valid.
Because backend init still requires full configuration, this introduces a
new mode of terraform init where it detects heuristically if it seems like
we need to do a configuration upgrade and does a partial init if so,
before finally directing the user to run "terraform 0.12upgrade" before
running any other commands.
The heuristic here is based on two assumptions:
- If the "early" loader finds no errors but the normal loader does, the
configuration is likely to be valid for Terraform 0.11 but not 0.12.
- If there's already a version constraint in the configuration that
excludes Terraform versions prior to v0.12 then the configuration is
probably _already_ upgraded and so it's just a normal syntax error,
even if the early loader didn't detect it.
Once the upgrade process is removed in 0.13.0 (users will be required to
go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can
be simplified to remove that special mode, but the idea of doing the
dependency version checks against the liberal parser will remain valuable
to increase our chances of reporting version-based incompatibilities
rather than syntax errors as we add new features in future.
* command/show: added test scaffold for json output
More test cases will be added once the basic shape of the tests is
validated.
- command/json* packages now sort resources by address, matching
behavior elsewhere
- using cmp in tests instead of reflect.DeepEqual for the diffs
- updating expected output in tests to match sorting
Previously we were doing this rather inconsistently: some commands would
do it and others would not. By doing it here we ensure we always apply the
same normalization, regardless of which operation we're running.
This normalization is mostly for cosmetic purposes in error messages, but
it also ends up being used to populate path.module and path.root and so
it's important that we always produce consistent results here so that
we don't produce flappy changes as users work with different commands.
The fact that thus mutates a data structure as a side-effect is not ideal
but this is the best place to ensure it always gets applied without doing
any significant refactoring, since everything after this point happens in
the backend package where the normalizePath method is not available.
* command/show: adding functions to aid refactoring
The planfile -> statefile -> state logic path was getting hard to follow
with blurry human eyes. The getPlan... and getState... functions were
added to help streamline the logic flow. Continued refactoring may follow.
* command/show: use ctx.Config() instead of a config snapshot
As originally written, the jsonconfig marshaller was getting an error
when loading configs that included one or more modules. It's not clear
if that was an error in the function call or in the configloader itself,
but as a simpler solution existed I did not dig too far.
* command/jsonplan: implement jsonplan.Marshal
Split the `config` portion into a discrete package to aid in naming
sanity (so we could have for example jsonconfig.Resource instead of
jsonplan.ConfigResource) and to enable marshaling the config on it's
own.
Older versions of terraform could save the backend hash number in a
value larger than an int.
While we could conditionally decode the state into an intermediary data
structure for upgrade, or detect the specific decode error and modify
the json, it seems simpler to just decode into the most flexible value
for now, which is a uint64.
Fixes#18822
The `tuncatedId` function had been introduced in #12261 and increased the
`maxIdLen` to 80 in #13317. Since the number of bytes itself seems to be
unimportant, the ID should be truncated to 80 characters, not 80 bytes.
A lot of commands used `c.Meta.flagSet()` to create the initial flagset for the command, while quite a few of them didn’t actually use or support the flags that are then added.
So I updated a few commands to use `flag.NewFlagSet()` instead to only add the flags that are actually needed/supported.
Additionally this prevents a few commands from using locking while they actually don’t need locking (as locking is enabled as a default in `c.Meta.flagSet()`.
Next to adding the locking for the `state push` command, this commit also fixes a small bug where the lock would not be propertly released when running the `state show` command.
And finally it renames some variables in the `[un]taint` code in order to try to standardize the var names of a few frequently used variables (e.g. statemgr.Full, states.State, states.SyncState).
In a couple places in tests we execute a child "go build" to make a helper
program. Now that we're running in module mode, "go build" will normally
default to downloading and caching dependencies, which we don't want
because we're still using vendoring for the moment.
Therefore we need to instruct these child builds to use vendoring too,
avoiding the need to download all of the dependencies and ensuring that
we'll be building with the same dependencies that we'd use for a normal
build.
Several of these tests rely on external services (e.g. Terraform Registry)
that have not yet been updated to support the needs of Terraform v0.12.0,
so for now we'll skip all of these tests and wait until those systems have
been updated.
This should be removed before Terraform v0.12.0 final to enable these
tests to be used as part of pre-release smoke testing.
The local filesystem state manager no longer creates backup files eagerly,
instead creating them only if on first write there is already a snapshot
present in the target file.
Therefore for this test to exercise the codepaths it intends to we must
create an initial state snapshot for it to overwrite, creating the backup
in the process.
There are several other tests for this behavior elsewhere, so this test
is primarily to verify that the refresh command is configuring the backend
appropriately to get the backups written in the desired location.
We now only create a backup state file if the given output file already
exists, which it does not in this test.
(The behavior of creating the backup files is already covered by other
tests, so no need for this one go out of its way to do it.)
We now don't create a local state backup until the first snapshot write,
so we don't expect there to be a backup file until the end of the test.
(There is already a check at the end there, unmodified by this change.)
The filesystem backend has the option of using a different file for its
initial read.
Previously we were incorrectly writing the contents of that file out into
the backup file, rather than the prior contents of the output file. Now
we will always read the output file in RefreshState in order to decide
what we will back up but then we will optionally additionally read the
input file and prefer its content as the "current" state snapshot.
This is verified by command.TestMetaBackend_planLocalStatePath and
TestMetaBackend_configureNew, which are both now passing.
The changes to how we handle setting the state path on the local backend
broke the heuristic we were using here for detecting migration from one
local backend to another with the same state path, which would by default
end up deleting the state altogether after migration.
We now use the StatePaths method to do this, which takes into account
both the default values and any settings that have been set.
Additionally this addresses a flaw in the old method which could
potentially have deleted all non-default workspace state files if the
"path" setting were changed without also changing the "workspace_dir"
setting. This new approach is conservative because it will preserve all
of the files if any one overlaps.
In an earlier change we fixed the "backendFromConfig" codepath to be
able to properly detect changes to the -backend-config arguments during
"terraform init", but this detection is too strict for the normal case
of running an operation in a previously-initialized directory.
Before any of the recent changes, the logic here was to selectively update
the hash to include -backend-config settings in the init case. Since
that late hash recalculation was confusing, here we take the alternative
path of using the hash only in the normal case and full value comparison
in the init case. Treating both of these cases separately makes things
marginally easier to follow here.
The import command was imposing the default state path at the CLI level,
rather than leaving that to be handled by the backend. As a result, the
output state was always forced to be terraform.tfstate, regardless of
the backend settings.
This test is testing some strange implementation details of the old
local backend which do not hold with the new filesystem state manager.
Specifically, it was expecting state to be read from the stateOutPath
rather than the statePath, which makes no sense here because the backend
is configured to read from the default terraform.tfstate file (which does
not exist.)
There is another problem with this test which will be addressed in a
subsequent commit.
As part of integrating the new "remote" backend we relaxed the requirement
that a "default" workspace must exist in all backends and now skip
migrating empty workspace states to avoid creating unnecessary "default"
workspaces when switching between backends that require it and backends
that don't, such as when switching from the local backend (which always
has a "default" workspace) to Terraform Enterprise.
This was failing because we now handle the settings for the local backend
a little differently as a result of decoding it with the HCL2 machinery.
Specifically, the backend.State* fields are now assumed to be what is
given in configuration, and any CLI overrides are maintained separately
in OverrideState* fields so that they can be imposed "just in time" in
StatePaths.
This is particularly important because OverrideStatePath (when set) is
used regardless of workspace name, while StatePath is a suitable value
only for the "default" workspace, with others needing to be constructed
from StateWorkspaceDir instead.
Our new state model has a different implementation of "empty" that doesn't
consider lineage/serial, so we need to have some actual content in these
state fixtures to avoid them being skipped during state migrations.
We previously hacked around the import/export functionality being missing
in the statemgr layer after refactoring, but now it's been reintroduced
to fix functionality elsewhere we should use the centralized Import and
Export functions to ensure consistent behavior.
In particular, this pushes the logic for checking lineage and serial
during push down into the state manager itself, which is better because
all other details about lineage and serial are managed within the state
managers.
This test was initially failing because its fixture had a state which our
new state models consider to be "empty", and thus it was not migrated.
After fixing that (by adding an output to the fixture), this revealed a
bug that the lineage was not being persisted through the migration. This
is fixed by using the statemgr.Migrate method instead of writing via the
normal Writer interface, which allows two cooperating state managers to
properly transfer the lineage and serial along with the state snapshot.
This test was incorrectly updated in a previous iteration, with it
creating a modified state to write but then not actually writing it,
writing an empty test state instead.
This made the test fail because a backup state file is created only if
the new state snapshot is different to the old when written.
Some other test is leaving behind a terraform.tfstate after it concludes,
which can cause this test to fail in a strange way due to picking up
extra provider requirements from that state.
This check doesn't fix that problem, but it at least makes the test fail
in a more helpful way to avoid time wasted trying to debug this test when
it's some other test that actually has the bug.
This test is currently failing due to the command completing successfully,
which would previously cause a panic because we didn't properly initialize
the MockUi and so its error buffer is nil unless written to.
(The failure this was masking will be fixed in a subsequent commit.)
In prior refactoring we lost the required core version check from
"terraform init", which we restore here.
Additionally, this test used to have an incorrect name that suggested it
was testing something in the "getProvider" codepath, but version checking
happens regardless of what other options are selected.
After all of the refactoring we were no longer checking the Terraform
version field in a state file, causing this test to fail.
This restores that check, though with a slightly different error message.
This test was using old-style state files as its input, differing only by
lineage. Since lineages are now managed within the state manager itself,
the test can't use that to distinguish the two files and so we put a
different output in each one instead.
This also introduces some TRACE logging to the migration codepaths.
There's some hard-to-follow control flow here and so this extra logging
helps to understand the reason for a particular outcome, and since this
codepath is visited only in "terraform init" anyway it doesn't hurt to
be a bit more verbose here.
In the refactoring for new HCL this codepath stopped taking into account
changes to the CLI -backend-config options when deciding if a backend
migration is required.
This restores that behavior in a different way than it used to be: rather
than re-hashing the merged config and comparing the hashes, we instead
just compare directly the configuration values, which must be exactly
equal in order to skip migration.
This change is covered by the test TestInit_inputFalse, although as of
this commit it is still not passing due a downstream problem within the
migration code itself.
This test was re-using the same InitCommand value to run multiple times,
which is not realistic. Since we now cache configuration source code
inside command.Meta on load, it's important that we use a fresh
InitCommand instance here so it'll see the modified configuration file
we've left on disk.
Here we were going to the trouble of copying the body so we could mutate
it, but then ended up mutating the original anyway and then returning the
unmodified copy. Whoops!
This fix is verified by a number of "init" command tests that exercise the
-backend-config option, including TestInit_backendConfigFile and several
others whose names have the prefix TestInit_backendConfig .
When we originally wrote this message we struggled a bit for how to refer
to the releases server without writing an awkwardly-ungrammatical
sentence, and so "the official repository" became a placeholder name for
it.
Now that we'll be looking in Terraform Registry this gives us a nice
proper noun to use. This message will need to evolve more as our
integration with the registry gets more sophisticated, but for now this
works.
Some over-zealous bulk updating of this test file caused this test to be
producing a remote state config cache file on disk when it doesn't
actually need one: the backend config comes from the plan file when
applying a saved plan.
Some merging conflict shenanigans here led to this usage not lining up
with the imported symbol name, meaning that the tests couldn't compile any
more.
We missed fixing this up during the big updates for the new plan/state
models since the failures were being masked by testBackendState being
broken.
This is the same sort of update made to many other tests: add schema to
the mock provider, adjust for the new plan/state types, and make
allowances for the new built-in diffing behavior in core.
The hashing function for cached backend configuration is different now, so
our hard-coded hash of the HTTP backend address wasn't working anymore.
Here we update the hash so that tests using this test backend will work
again. Rather than leaving it hard-coded, we'll instead compute it the
same way as "terraform init" would.
In practice only one test is actually using this function right now, so
we also update the test fixture for that test (TestPlan_outBackend) to
match the new expectations, though as of this commit it's still failing
with an unrelated error.
The mission of this process method used to include dealing with
auto-loaded tfvars files, but it doesn't do that anymore.
It does still deal with the -no-color option, but the test wasn't
exercising that part before.
Now the test here focuses on the -no-color behavior.
The process method still has a "vars" flag argument which is no longer
used. Since this is an unexported method we could potentially address this
but this commit is intentionally limited only to fixing the test.
Comments here indicate that this was erroneously returning an error but
we accepted it anyway to get the tests passing again after other work.
The tests over in the "terraform" package agree that cancelling should be
a successful outcome rather than an error.
I think that cancelling _should_ actually be an error, since Terraform did
not complete the operation it set out to complete, but that's a change
we'd need to make cautiously since automation wrapper scripts may be
depending on the success-on-cancel behavior.
Therefore this just fixes the command package test to agree with the
Terraform package tests and adds some FIXME notes to capture the potential
that we might want to update this later.
The State.Equal function is now more precise than this test needs. It's
only trying to distinguish between an empty state and a non-empty state,
so the string representation of state is good enough to get that done
while disregarding other subtle differences.
This new source type should be used for variables loaded from .tfvars files that were explicitly passed as command line arguments (e.g. -var-file=foo.tfvars)
Without using absolute paths any module info is lost in the output. And the attributes were randomly ordered and so changed between different executions of the command.
Kristin Laemmert