Along with all of the other information we previously reported in the
"terraform version" output, we'll now include the name of the current
platform as our provider mechanisms represent it.
This is addressing a long-standing minor annoyance where we often can't
tell from an incomplete bug report which platform Terraform was running
on, and incomplete bug reporters do tend to at least include the
"terraform version" output even if they don't also include the requested
full trace log.
However, what motivated doing it _now_ is that anyone building a provider
registry or mirror needs to have some awareness of these platform
identifiers which have been, until v0.13, mostly an implementation detail.
This additional information is a small thing we can do to help registry
builders find out what the platform identifier ought to be for each of
the platforms they aim to support, even if some of them are platforms
which the Go compiler allows but which HashiCorp doesn't officially
support.
The new information is on a line of its own in the output as a pragmatic
way to avoid breaking anyone who might be using something like
$(terraform version | head -n1) to print a brief Terraform version
identifier into some logs. That's not an interface we officially support
for machine consumption, but it's easy to avoid breaking it here and so we
won't do so.
When using the enhanced remote backend, a subset of all Terraform
operations are supported. Of these, only plan and apply can be executed
on the remote infrastructure (e.g. Terraform Cloud). Other operations
run locally and use the remote backend for state storage.
This causes problems when the local version of Terraform does not match
the configured version from the remote workspace. If the two versions
are incompatible, an `import` or `state mv` operation can cause the
remote workspace to be unusable until a manual fix is applied.
To prevent this from happening accidentally, this commit introduces a
check that the local Terraform version and the configured remote
workspace Terraform version are compatible. This check is skipped for
commands which do not write state, and can also be disabled by the use
of a new command-line flag, `-ignore-remote-version`.
Terraform version compatibility is defined as:
- For all releases before 0.14.0, local must exactly equal remote, as
two different versions cannot share state;
- 0.14.0 to 1.0.x are compatible, as we will not change the state
version number until at least Terraform 1.1.0;
- Versions after 1.1.0 must have the same major and minor versions, as
we will not change the state version number in a patch release.
If the two versions are incompatible, a diagnostic is displayed,
advising that the error can be suppressed with `-ignore-remote-version`.
When this flag is used, the diagnostic is still displayed, but as a
warning instead of an error.
Commands which will not write state can assert this fact by calling the
helper `meta.ignoreRemoteBackendVersionConflict`, which will disable the
checks. Those which can write state should instead call the helper
`meta.remoteBackendVersionCheck`, which will return diagnostics for
display.
In addition to these explicit paths for managing the version check, we
have an implicit check in the remote backend's state manager
initialization method. Both of the above helpers will disable this
check. This fallback is in place to ensure that future code paths which
access state cannot accidentally skip the remote version check.
Remove chef, habitat, puppet, and salt-masterless provsioners,
which follows their deprecation. Update the documentatin for these
provisioners to clarify that they have been removed from later versions
of Terraform. Adds the fmt Make target back and updates fmtcheck script
for correctness.
When building a context, we read the dependency locks and ensure that
the provider requirements from the configuration can be satisfied.
If the configured requirements change such that the locks need to be
updated, we explain this and recommend running "terraform init".
This check is ignored for any providers which are locally marked as in
development. This includes unmanaged providers and those listed in the
provider installation `dev_overrides` block.
Core is only using the PrepareProviderConfig call for the validation
part of the method, but we should be re-validating the final config
immediately before Configure.
This change elects to not start using the PreparedConfig here, since
there is no useful reason for it at this point, and it would
introduce a functional difference between terraform releases that can be
avoided.
Previously when printing the relevant variables involved in a failed
expression evaluation we would just skip over unknown values entirely.
There are some errors, though, which are _caused by_ a value being
unknown, in which case it's helpful to show which of the inputs to that
expression were known vs. unknown so that the user can limit their further
investigation only to the unknown ones.
While here I also added a special case for sensitive values that overrides
all other display, because we don't know what about a value is sensitive
and so better to give nothing away at the expense of a slightly less
helpful error message.
For this version of Terraform and forward, we no longer refuse to read
compatible state files written by future versions of Terraform. This is
a commitment that any changes to the semantics or format of the state
file after this commit will require a new state file version 5.
The result of this is that users of this Terraform version will be able
to share remote state with users of future versions, and all users will
be able to read and write state. This will be true until the next major
state file version is required.
This does not affect users of previous versions of Terraform, which will
continue to refuse to read state written by later versions.
The short description of our commands (as shown in the main help output
from "terraform") was previously very inconsistent, using different
tense/mood for different commands. Some of the commands were also using
some terminology choices inconsistent with how we currently talk about
the related ideas in our documentation.
Here I've tried to add some consistency by first rewriting them all in
the imperative mood (except the ones that just are just subcommand
groupings), and tweaking some of the terminology to hopefully gel better
with how we present similar ideas in our recently-updated docs.
While working on this I inevitably spotted some similar inconsistencies
in the longer-form help output of some of the commands. I've not reviewed
all of these for consistency, but I did update some where the wording
was either left inconsstent with the short form changes I'd made or
where the prose stood out to me as particularly inconsistent with our
current usual documentation language style.
All of this is subjective, so I expect we'll continue to tweak these over
time as we continue to develop our documentation writing style based on
user questions and feedback.
Now that hclog can independently set levels on related loggers, we can
separate the log levels for different subsystems in terraform.
This adds the new environment variables, `TF_LOG_CORE` and
`TF_LOG_PROVIDER`, which each take the same set of log level arguments,
and only applies to logs from that subsystem. This means that setting
`TF_LOG_CORE=level` will not show logs from providers, and
`TF_LOG_PROVIDER=level` will not show logs from core. The behavior of
`TF_LOG` alone does not change.
While it is not necessarily needed since the default is to disable logs,
there is also a new level argument of `off`, which reflects the
associated level in hclog.
Use a separate log sink to always capture trace logs for the panicwrap
handler to write out in a crash log.
This requires creating a log file in the outer process and passing that
path to the child process to log to.
If you run the e2etests locally and use a configured plugin_cache_dir,
the test will leave a bad directory behind in your cache dir that causes
later `init`s to fail. To circumvent this, pass an explicity-empty CLI
config file.
This is a nicety for local developers and not necessarily required, but
it happens to me often enough that I'd like to fix it. It's probably not
a *bad* idea to pass an explicit cli config to all e2etests, honestly,
but this is the only one that causes active problems so I limited this
PR to that one test.
Here's the error which occurs on subsequent `init` if this test is run on a
machine that uses a plugin cache dir:
2020/10/13 10:41:05 [TRACE] providercache.fillMetaCache: error while scanning directory /Users/mildwonkey/.terraform.d/plugin-cache: failed to read metadata about /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: stat /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: no such file or directory
If a configuration requires a partial provider version (with some parts
unspecified), Terraform considers this as a constrained-to-zero version.
For example, a version constraint of 1.2 will result in an attempt to
install version 1.2.0, even if 1.2.1 is available.
When writing the dependency locks file, we previously would write 1.2.*,
as this is the in-memory representation of 1.2. This would then cause an
error on re-reading the locks file, as this is not a valid constraint
format.
Instead, we now explicitly convert the constraint to its zero-filled
representation before writing the locks file. This ensures that it
correctly round-trips.
Because this change is made in getproviders.VersionConstraintsString, it
also affects the output of the providers sub-command.
Use a single log writer instance for all std library logging.
Setup the std log writer in the logging package, and remove boilerplate
from test packages.
For normal provider installation we want to associate each provider with
a selected version number and find a suitable package for that version
that conforms to the official hashes for that release.
Those requirements are very onerous for a provider developer currently
testing a not-yet-released build, though. To allow for that case this new
CLI configuration feature allows overriding specific providers to refer
to give local filesystem directories.
Any provider overridden in this way is not subject to the usual
restrictions about selected versions or checksum conformance, and
activating an override won't cause any changes to the selections recorded
in the lock file because it's intended to be a temporary setting for one
developer only.
This is, in a sense, a spiritual successor of an old capability we had to
override specific plugins in the CLI configuration file. There were
some vestiges of that left in the main package and CLI config package
but nothing has actually been honoring them for several versions now and
so this commit removes them to avoid confusion with the new mechanism.
This originated in the cliconfig code to write out credentials files. The
Windows implementation of this in particular was quite onerous to get
right because it needs a very specific sequence of operations to avoid
running into exclusive file locks, and so by factoring this out with
only cosmetic modification we can avoid repeating all of that engineering
effort for other atomic file writing use-cases.
Terraform v0.10 introduced .terraform/plugins as a cache directory for
automatically-installed plugins, Terraform v0.13 later reorganized the
directory structure inside but retained its purpose as a cache.
The local cache used to also serve as a record of specifically which
packages were selected in a particular working directory, with the intent
that a second run of "terraform init" would always select the same
packages again. That meant that in some sense it behaved a bit like a
local filesystem mirror directory, even though that wasn't its intended
purpose.
Due to some unfortunate miscommunications, somewhere a long the line we
published some documentation that _recommended_ using the cache directory
as if it were a filesystem mirror directory when working with Terraform
Cloud. That was really only working as an accident of implementation
details, and Terraform v0.14 is now going to break that because the source
of record for the currently-selected provider versions is now the
public-facing dependency lock file rather than the contents of an existing
local cache directory on disk.
After some consideration of how to move forward here, this commit
implements a compromise that tries to avoid silently doing anything
surprising while still giving useful guidance to folks who were previously
using the unsupported strategy. Specifically:
- The local cache directory will now be .terraform/providers rather than
.terraform/plugins, because .terraform/plugins is effectively "poisoned"
by the incorrect usage that we can't reliably distinguish from prior
version correct usage.
- The .terraform/plugins directory is now the "legacy cache directory". It
is intentionally _not_ now a filesystem mirror directory, because that
would risk incorrectly interpreting providers automatically installed
by Terraform v0.13 as if they were a local mirror, and thus upgrades
and checksum fetches from the origin registry would be blocked.
- Because of the previous two points, someone who _was_ trying to use the
legacy cache directory as a filesystem mirror would see installation
fail for any providers they manually added to the legacy directory.
To avoid leaving that user stumped as to what went wrong, there's a
heuristic for the case where a non-official provider fails installation
and yet we can see it in the legacy cache directory. If that heuristic
matches then we'll produce a warning message hinting to move the
provider under the terraform.d/plugins directory, which is a _correct_
location for "bundled" provider plugins that belong only to a single
configuration (as opposed to being installed globally on a system).
This does unfortunately mean that anyone who was following the
incorrectly-documented pattern will now encounter an error (and the
aforementioned warning hint) after upgrading to Terraform v0.14. This
seems like the safest compromise because Terraform can't automatically
infer the intent of files it finds in .terraform/plugins in order to
decide automatically how best to handle them.
The internals of the .terraform directory are always considered
implementation detail for a particular Terraform version and so switching
to a new directory for the _actual_ cache directory fits within our usual
set of guarantees, though it's definitely non-ideal in isolation but okay
when taken in the broader context of this problem, where the alternative
would be silent misbehavior when upgrading.
When an attribute's sensitivity changes, but its value remains the same,
we consider this an update operation for the plan. This commit updates
the diff renderer to match this, detecting and displaying the change in
sensitivity.
Previously, the renderer would detect no changes to the value of the
attribute, and consider it a no-op action. This resulted in suppression
of the attribute when the plan is in concise mode.
This is achieved with a new helper function, ctyEqualValueAndMarks. We
call this function whenever we want to check that two values are equal
in order to determine whether the action is update or no-op.
Previously we were just letting hclwrite do its default formatting
behavior here. The current behavior there isn't ideal anyway -- it puts
big data structures all on one line -- but even ignoring that our goal
for this file format is to keep things in a highly-normalized shape so
that diffs against the file are clear and easy to read.
With that in mind, here we directly control how we write that value into
the file, which means that later changes to hclwrite's list/set
presentation won't affect it, regardless of what form they take.
This command is intended to help support situations where Terraform is
configured to use only local mirrors for provider installation and so the
normal "terraform init" flow would not have direct access to the official
package checksums published in the origin registry.
The intended workflow here is to use this command only when adding a new
provider or changing an existing provider's version in the configuration,
to augment the lock file with all of the checksums required to verify
the provider across a variety of different platforms. Once this command
has recorded all of the official checksums, future runs of
"terraform init" will verify that provider packages obtained from a local
mirror match with those upstream checksums.
This probably isn't the best UI we could do here, but it's a placeholder
for now just to avoid making it seem like we're ignoring the lock file
and checking for new versions anyway.
This changes the approach used by the provider installer to remember
between runs which selections it has previously made, using the lock file
format implemented in internal/depsfile.
This means that version constraints in the configuration are considered
only for providers we've not seen before or when -upgrade mode is active.
helper/copy CopyDir was used heavily in tests. It differes from
internal/copydir in a few ways, the main one being that it creates the
dst directory while the internal version expected the dst to exist
(there are other differences, which is why I did not just switch tests
to using internal's CopyDir).
I moved the CopyDir func from helper/copy into command_test.go; I could
also have moved it into internal/copy and named it something like
CreateDirAndCopy so if that seems like a better option please let me
know.
helper/copy/CopyFile was used in a couple of spots so I moved it into
internal, at which point I thought it made more sense to rename the
package copy (instead of copydir).
There's also a `go mod tidy` included.
* remove unused code
I've removed the provider-specific code under registry, and unused nil
backend, and replaced a call to helper from backend/oss (the other
callers of that func are provisioners scheduled to be deprecated).
I also removed the Dockerfile, as our build process uses a different
file.
Finally I removed the examples directory, which had outdated examples
and links. There are better, actively maintained examples available.
* command: remove various unused bits
* test wasn't running
* backend: remove unused err
If an entire block is marked sensitive (possibly because it is of type
NestedSet) and results in replacement of the resource, we should render
the standard "forces replacement" text after the opening line of the
block.
If a value rendered for the diff is sensitive and results in replacement
of the resource, we should render the standard "forces replacement" text
after the "(sensitive)" value display.
When a value in a nested block is marked as sensitive,
it will result in the behavior of every member of
that block being sensitive. As such, show a
specialized diff that reduces noise of (sensitive)
for many attributes that we won't show. Also update
the warning to differentiate between showing a warning
for an attribute or warning for blocks.
We no longer need to support 0.12-and-earlier-style provider addresses
because users should've upgraded their existing configurations and states
on Terraform 0.13 already.
For now this is only checked in the "init" command, because various test
shims are still relying on the idea of legacy providers the core layer.
However, rejecting these during init is sufficient grounds to avoid
supporting legacy provider addresses in the new dependency lock file
format, and thus sets the stage for a more severe removal of legacy
provider support in a later commit.
The tmp directory path is longer on Macs than other systems and was wrapping
across lines when printed, breaking the string match in the test.
Fix suggested by @apparentlymart is to add two spaces before the leading
'-' when printing to prevent the diagnostic renderer wrapping the line.
As we add and remove support for different target platforms over time,
there will be transition periods where the available platforms for
each provider will be different than the available platforms for Terraform
CLI itself.
In recognition of that possibility, here we add a more specialized error
message for that situation which tries to explain the problem a little
more clearly than the generic error message that came before it.
In an ideal world we'd be able to detect situations where a newer or older
version has support in a similar vein to what we do with provider protocol
incompatibilities, but we don't currently have logic to fetch the data
necessary to implement that, so this is instead a presentation-only change
aimed at allowing some possible near-future changes to the supported
target platforms.
In earlier commits we started to make the installation codepath
context-aware so that it could be canceled in the event of a SIGINT, but
we didn't complete wiring that through the API of the getproviders
package.
Here we make the getproviders.Source interface methods, along with some
other functions that can make network requests, take a context.Context
argument and act appropriately if that context is cancelled.
The main providercache.Installer.EnsureProviderVersions method now also
has some context-awareness so that it can abort its work early if its
context reports any sort of error. That avoids waiting for the process
to wind through all of the remaining iterations of the various loops,
logging each request failure separately, and instead returns just
a single aggregate "canceled" error.
We can then set things up in the "terraform init" and
"terraform providers mirror" commands so that the context will be
cancelled if we get an interrupt signal, allowing provider installation
to abort early while still atomically completing any local-side effects
that may have started.
We only preserve these major upgrade versions for one major version after
they are added, because our upgrade path assumes moving forward only one
major version at a time. Now that our main branch is tracking towards
Terraform 0.14, we no longer need the 0.13upgrade subcommand.
This also includes some minor adjustments to the 0.12upgrade command to
align the terminology used in the output of both commands. We usually
use the word "deprecated" to mean that something is still available but
no longer recommended, but neither of these commands is actually available
so "removed" is clearer.
We could in principle have removed even the removal notice for 0.12upgrade
here, but it's relatively little code and not a big deal to keep around
to help prompt those who might try to upgrade directly from 0.11 to 0.14.
We may still remove the historical configuration upgrade commands prior to
releasing Terraform 1.0, though.
* Fix taint and untaint commands when in a workspace
Fixes#22157. Removes DefaultStateFilepath as the default for the
-state flag, allowing workspaces to be used properly.
* update test with modern state types
Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>
In Terraform 0.11 and earlier, the "terraform fmt" command was very
opinionated in the interests of consistency. While that remains its goal,
for pragmatic reasons Terraform 0.12 significantly reduced the number
of formatting behaviors in the fmt command. We've held off on introducing
0.12-and-later-flavored cleanups out of concern it would make it harder
to maintain modules that are cross-compatible with both Terraform 0.11
and 0.12, but with this aimed to land in 0.14 -- two major releases
later -- our new goal is to help those who find older Terraform language
examples learn about the more modern idiom.
More rules may follow later, now that the implementation is set up to
allow modifications to tokens as well as modifications to whitespace, but
for this initial pass the command will now apply the following formatting
conventions:
- 0.11-style quoted variable type constraints will be replaced with their
0.12 syntax equivalents. For example, "string" becomes just string.
(This change quiets a deprecation warning.)
- Collection type constraints that don't specify an element type will
be rewritten to specify the "any" element type explicitly, so
list becomes list(any).
- Arguments whose expressions consist of a quoted string template with
only a single interpolation sequence inside will be "unwrapped" to be
the naked expression instead, which is functionally equivalent.
(This change quiets a deprecation warning.)
- Block labels are given in quotes.
Two of the rules above are coming from a secondary motivation of
continuing down the deprecation path for two existing warnings, so authors
can have two active deprecation warnings quieted automatically by
"terraform fmt", without the need to run any third-party tools.
All of these rules match with current documented idiom as shown in the
Terraform documentation, so anyone who follows the documented style should
see no changes as a result of this. Those who have adopted other local
style will see their configuration files rewritten to the standard
Terraform style, but it should not make any changes that affect the
functionality of the configuration.
There are some further similar rewriting rules that could be added in
future, such as removing 0.11-style quotes around various keyword or
static reference arguments, but this initial pass focused only on some
rules that have been proven out in the third-party tool
terraform-clean-syntax, from which much of this commit is a direct port.
For now this doesn't attempt to re-introduce any rules about vertical
whitespace, even though the 0.11 "terraform fmt" would previously apply
such changes. We'll be more cautious about those because the results of
those rules in Terraform 0.11 were often sub-optimal and so we'd prefer
to re-introduce those with some care to the implications for those who
may be using vertical formatting differences for some semantic purpose,
like grouping together related arguments.
Previously we were just using hclwrite.Format, a token-only formatting
pass. Now we'll do that via the full hclwrite parser, getting the
formatting as a side-effect of the parsing and re-serialization.
This should have no change in observable behavior as-is, but in a future
commit we'll add some additional processing rules that modify the syntax
tree before re-serializing it.
Previously formatting was just a simple wrapper around hclwrite.Format.
That remains true here, but the call is factored out into a separate
method in preparation for making it also do some Terraform-specific
cleanups in a future commit.
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.
Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.
Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
* Add creation test and simplify in-place test
* Add deletion test
* Start adding marking from state
Start storing paths that should be marked
when pulled out of state. Implements deep
copy for attr paths. This commit also includes some
comment noise from investigations, and fixing the diff test
* Fix apply stripping marks
* Expand diff tests
* Basic apply test
* Update comments on equality checks to clarify current understanding
* Add JSON serialization for sensitive paths
We need to serialize a slice of cty.Path values to be used to re-mark
the sensitive values of a resource instance when loading the state file.
Paths consist of a list of steps, each of which may be either getting an
attribute value by name, or indexing into a collection by string or
number.
To serialize these without building a complex parser for a compact
string form, we render a nested array of small objects, like so:
[
[
{ type: "get_attr", value: "foo" },
{ type: "index", value: { "type": "number", "value": 2 } }
]
]
The above example is equivalent to a path `foo[2]`.
* Format diffs with map types
Comparisons need unmarked values to operate on,
so create unmarked values for those operations. Additionally,
change diff to cover map types
* Remove debugging printing
* Fix bug with marking non-sensitive values
When pulling a sensitive value from state,
we were previously using those marks to remark
the planned new value, but that new value
might *not* be sensitive, so let's not do that
* Fix apply test
Apply was not passing the second state
through to the third pass at apply
* Consistency in checking for length of paths vs inspecting into value
* In apply, don't mark with before paths
* AttrPaths test coverage for DeepCopy
* Revert format changes
Reverts format changes in format/diff for this
branch so those changes can be discussed on a separate PR
* Refactor name of AttrPaths to AttrSensitivePaths
* Rename AttributePaths/attributePaths for naming consistency
Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
Despite not requiring the configuration for any other reason, the taint
subcommand should not execute if the required_version constraints cannot
be met. Doing so can result in an undesirable state file upgrade.
The providers schema command is using the Config.ProviderTypes method,
which had not been kept up to date with the changes to provider
requirements detection made in Config.ProviderRequirements. This
resulted in any currently-unused providers being omitted from the
output.
This commit changes the ProviderTypes method to use the same underlying
logic as ProviderRequirements, which ensures that `required_providers`
blocks are taken into account.
Includes an integration test case to verify that this fixes the provider
schemas command bug.
James Bardin