This "running in automation" idea is a best effort thing where we try to
avoid printing out specific suggestions of commands to run in the main
workflow when the user is running Terraform inside a wrapper script or
other automation, because they probably don't want to bypass that
automation.
This just makes that information available to the main views.View type,
so we can then make use of it in the implementation of more specialized
view types that embed views.View.
However, nothing is using it as of this commit. We'll use it in later
commits.
This is just a prototype to gather some feedback in our ongoing research
on integration testing of Terraform modules. The hope is that by having a
command integrated into Terraform itself it'll be easier for interested
module authors to give it a try, and also easier for us to iterate quickly
based on feedback without having to coordinate across multiple codebases.
Everything about this is subject to change even in future patch releases.
Since it's a CLI command rather than a configuration language feature it's
not using the language experiments mechanism, but generates a warning
similar to the one language experiments generate in order to be clear that
backward compatibility is not guaranteed.
Terraform supports multiple output formats for several sub-commands.
The default format is user-readable text, but many sub-commands support
a `-json` flag to output a machine-readable format for the result. The
output command also supports a `-raw` flag for a simpler, scripting-
focused machine readable format.
This commit adds a "views" abstraction, intended to help ensure
consistency between the various output formats. This extracts the render
specific code from the command package, and moves it into a views
package. Each command is expected to create an interface for its view,
and one or more implementations of that interface.
By doing so, we separate the concerns of generating the sub-command
result from rendering the result in the specified output format. This
should make it easier to ensure that all output formats will be updated
together when changes occur in the result-generating phase.
There are some other consequences of this restructuring:
- Views now directly access the terminal streams, rather than the
now-redundant cli.Ui instance;
- With the reorganization of commands, parsing CLI arguments is now the
responsibility of a separate "arguments" package.
For now, views are added only for the output sub-command, as an example.
Because this command uses code which is shared with the apply and
refresh commands, those are also partially updated.
Here we propagate in the initialized terminal.Streams from package main,
and then onwards to backends running in CLI mode.
This also replaces our use of helper/wrappedstreams to determine whether
stdin is a terminal or a pipe. helper/wrappedstreams returns incorrect
file descriptors on Windows, causing StdinPiped to always return false on
that platform and thus causing one of the odd behaviors discussed in
Finally, this includes some wrappers around the ability to look up the
number of columns in the terminal in preparation for use elsewhere. These
wrappers deal with the fact that our unit tests typically won't populate
meta.Streams.
We need to call into terminal.Init in early startup to make sure that we
either have a suitable Terminal or that we disable attempts to use virtual
terminal escape sequences.
This commit gets the terminal initialized but doesn't do much with it
after that. Subsequent commits will make more use of this.
The revision field is only populated on dev builds so this means
most releases of Terraform have an empty "terraform_revision" field
in the JSON output. Since we recommend developers use go tooling
to `go build` this tool when developing, the revision is not useful
data and so it is removed.
Along with all of the other information we previously reported in the
"terraform version" output, we'll now include the name of the current
platform as our provider mechanisms represent it.
This is addressing a long-standing minor annoyance where we often can't
tell from an incomplete bug report which platform Terraform was running
on, and incomplete bug reporters do tend to at least include the
"terraform version" output even if they don't also include the requested
full trace log.
However, what motivated doing it _now_ is that anyone building a provider
registry or mirror needs to have some awareness of these platform
identifiers which have been, until v0.13, mostly an implementation detail.
This additional information is a small thing we can do to help registry
builders find out what the platform identifier ought to be for each of
the platforms they aim to support, even if some of them are platforms
which the Go compiler allows but which HashiCorp doesn't officially
support.
The new information is on a line of its own in the output as a pragmatic
way to avoid breaking anyone who might be using something like
$(terraform version | head -n1) to print a brief Terraform version
identifier into some logs. That's not an interface we officially support
for machine consumption, but it's easy to avoid breaking it here and so we
won't do so.
My initial motivation here was to update the example output from
Terraform's top-level help list to match recent updates in the layout
and language used.
However, while here I took the opportunity to update some dated language
that was not consistent with our modern documentation writing style,
in particular including a totally unnecessary and potentially-alienating
claim that Terraform is "very easy to use". Our modern writing style
discourages this sort of "boastful" language and encourages us to focus on
the facts at hand.
A long time ago we introduced this separation between "common commands"
and "all other commands", but over the intervening years we've not really
done a good job of classifying new commands we've added and so by default
most of them ended up being classified as "common".
In the interests of making this output more useful for those getting
started, this switches the two categories so that "Main commands" is now
the curated list, and "all other commands" is the bucket for everything
else.
The intent here is that the "main commands" are the ones users are likely
to try as part of their initial learning of Terraform, while the other
commands are for less common situations where the user is more likely to
learn about a specific command in some other context, like a tutorial
about a special situation.
The "main commands" are also now ordered by the sequence users will
typically run them in, rather than alphabetical order. That's a subjective
readability tradeoff, but I think as long as the list stays relatively
short (which it should) it's still relatively easy to scan and find a
particular command in the shortlist.
These are commands that either no longer do anything aside from emitting
an error message or are just backward-compatibility aliases for other
commands.
This generalizes our previous situation where we were specifically
hiding "internal-plugin", and does so in a way that fixes the
long-standing cosmetic bug that the column width in the help output was
chosen based on the hidden command "internal-plugin", which is
unfortunately also the longest command in our command set.
For normal provider installation we want to associate each provider with
a selected version number and find a suitable package for that version
that conforms to the official hashes for that release.
Those requirements are very onerous for a provider developer currently
testing a not-yet-released build, though. To allow for that case this new
CLI configuration feature allows overriding specific providers to refer
to give local filesystem directories.
Any provider overridden in this way is not subject to the usual
restrictions about selected versions or checksum conformance, and
activating an override won't cause any changes to the selections recorded
in the lock file because it's intended to be a temporary setting for one
developer only.
This is, in a sense, a spiritual successor of an old capability we had to
override specific plugins in the CLI configuration file. There were
some vestiges of that left in the main package and CLI config package
but nothing has actually been honoring them for several versions now and
so this commit removes them to avoid confusion with the new mechanism.
This command is intended to help support situations where Terraform is
configured to use only local mirrors for provider installation and so the
normal "terraform init" flow would not have direct access to the official
package checksums published in the origin registry.
The intended workflow here is to use this command only when adding a new
provider or changing an existing provider's version in the configuration,
to augment the lock file with all of the checksums required to verify
the provider across a variety of different platforms. Once this command
has recorded all of the official checksums, future runs of
"terraform init" will verify that provider packages obtained from a local
mirror match with those upstream checksums.
This new option is intended to address the previous inconsistencies where
some older subcommands supported partially changing the target directory
(where Terraform would use the new directory inconsistently) where newer
commands did not support that override at all.
Instead, now Terraform will accept a -chdir command at the start of the
command line (before the subcommand) and will interpret it as a request
to direct all actions that would normally be taken in the current working
directory into the target directory instead. This is similar to options
offered by some other similar tools, such as the -C option in "make".
The new option is only accepted at the start of the command line (before
the subcommand) as a way to reflect that it is a global command (not
specific to a particular subcommand) and that it takes effect _before_
executing the subcommand. This also means it'll be forced to appear before
any other command-specific arguments that take file paths, which hopefully
communicates that those other arguments are interpreted relative to the
overridden path.
As a measure of pragmatism for existing uses, the path.cwd object in
the Terraform language will continue to return the _original_ working
directory (ignoring -chdir), in case that is important in some exceptional
workflows. The path.root object gives the root module directory, which
will always match the overriden working directory unless the user
simultaneously uses one of the legacy directory override arguments, which
is not a pattern we intend to support in the long run.
As a first step down the deprecation path, this commit adjusts the
documentation to de-emphasize the inconsistent old command line arguments,
including specific guidance on what to use instead for the main three
workflow commands, but all of those options remain supported in the same
way as they were before. In a later commit we'll make those arguments
produce a visible deprecation warning in Terraform's output, and then
in an even later commit we'll remove them entirely so that -chdir is the
single supported way to run Terraform from a directory other than the
one containing the root module configuration.
This new command is intended to make it easy to create or update a mirror
directory containing suitable providers for the current configuration,
producing a layout that is appropriate both for a filesystem mirror or,
if copied into the document root of an HTTP server, a network mirror.
This initial version is not customizable aside from being able to select
multiple platforms to install packages for.
Future iterations of this could include commands to turn the JSON index
generation on and off, or to instruct it to produce the unpacked directory
layout instead of the packed directory layout as it currently does. Both
of those options would make the generated directory unsuitable to be
a network mirror, but it would still work as a filesystem mirror.
In the long run this will hopefully form part of a replacement workflow to
terraform-bundle as a way to put copies of providers somewhere so we don't
need to re-download them every time, but some other changes will be needed
outside of just this command before that'd be true, such as adding support
for network and/or filesystem mirrors in Terraform Enterprise.
This adds supports for "unmanaged" providers, or providers with process
lifecycles not controlled by Terraform. These providers are assumed to
be started before Terraform is launched, and are assumed to shut
themselves down after Terraform has finished running.
To do this, we must update the go-plugin dependency to v1.3.0, which
added support for the "test mode" plugin serving that powers all this.
As a side-effect of not needing to manage the process lifecycle anymore,
Terraform also no longer needs to worry about the provider's binary, as
it won't be used for anything anymore. Because of this, we can disable
the init behavior that concerns itself with downloading that provider's
binary, checking its version, and otherwise managing the binary.
This is all managed on a per-provider basis, so managed providers that
Terraform downloads, starts, and stops can be used in the same commands
as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable
is added, and is a JSON encoding of the provider's address to the
information we need to connect to it.
This change enables two benefits: first, delve and other debuggers can
now be attached to provider server processes, and Terraform can connect.
This allows for attaching debuggers to provider processes, which before
was difficult to impossible. Second, it allows the SDK test framework to
host the provider in the same process as the test driver, while running
a production Terraform binary against the provider. This allows for Go's
built-in race detector and test coverage tooling to work as expected in
provider tests.
Unmanaged providers are expected to work in the exact same way as
managed providers, with one caveat: Terraform kills provider processes
and restarts them once per graph walk, meaning multiple times during
most Terraform CLI commands. As unmanaged providers can't be killed by
Terraform, and have no visibility into graph walks, unmanaged providers
are likely to have differences in how their global mutable state behaves
when compared to managed providers. Namely, unmanaged providers are
likely to retain global state when managed providers would have reset
it. Developers relying on global state should be aware of this.
Terraform 0.13 will allow the installation of providers from various
sources. If a user updates their configuration to change the source of
an in-use provider (for example, if the provider namespace changes),
they will also need to update the state file accordingly.
This commit introduces a new `state replace-provider` subcommand which
supports this. All resources using the `from` provider will be updated
to use the `to` provider.
Following the same approach we use for other CLI-Config-able objects like
the service discovery system, the main package is responsible for
producing a suitable implementation of this interface which the command
package can then use.
When unit testing in the command package we can then substitute mocks as
necessary, following the dependency inversion principle.
main.ConfigDir is just a wrapper around cliconfig.ConfigDir to allow us to
gradually clean up the old calls here, but since this is new code we might
as well do it right from the start.
We're not ready to ship this in a release yet because there's still some
remaining work to do on the Terraform Cloud side, but we want to get the
implementation work behind this into the master branch so it's easier to
maintain it in the mean time, rather than letting this long-lived branch
live even longer.
We'll continue to iterate on UX polish and other details in subsequent
commits, and eventually enable this.
For unit testing in particular we can't launch a real browser for testing,
so this indirection is primarily to allow us to substitute a mock when
testing a command that can launch a browser.
This includes a simple mock implementation that expects to interact with
a running web server directly.
This should not cause any change in behavior yet, but using this new
implementation will allow the "terraform login" and "terraform logout"
commands to store and forget credentials when they are implemented in
subsequent commits.
This is just a wholesale move of the CLI configuration types and functions
from the main package into its own package, leaving behind some type
aliases and wrappers for now to keep existing callers working.
This commit alone doesn't really achieve anything, but in future commits
we'll expand the functionality in this package.
This is the frontend to the work-in-progress codepath for upgrading the
source code for a module written for Terraform v0.11 or earlier to use
the new syntax and idiom of v0.12.
The underlying upgrade code is not yet complete as of this commit, and
so the command is not yet very useful. We will continue to iterate on
the upgrade code in subsequent commits.
The remote API this talks to will be going away very soon, before our next
major release, and so we'll remove the command altogether in that release.
This also removes the "encodeHCL" function, which was used only for
adding a .tfvars-formatted file to the uploaded archive.
By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts.
Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!
This allows the user to customize the location where Terraform stores
the files normally placed in the ".terraform" subdirectory, if e.g. the
current working directory is not writable.
For situations where the default network-based discovery is inappropriate
or inconvenient, this allows users to provide a hard-coded discovery
document for a particular hostname in the CLI config.
This is a new config block, rather than combined with the existing
"credentials" block, because credentials should ideally live in separate
files from other config so that they can be managed more carefully.
However, this new "host" block _is_ designed to have room for additional
host-specific configuration _other than_ credentials in future, which
might include TLS certificate overrides or other such things used during
the discovery step.
Now that we are looking into the CLI config file for service host
credentials, it's important to support multiple separate files so that
users can keep credentials separate from other settings and credentials
for different hosts separate from one another.
There is no restriction on which settings can appear in which locations.
This is up to the user to decide, depending on their security needs and
e.g. on whether certain files are generated vs. manually-edited.
The command package is the main place we need access to these, so that
we can use them during init (to install packages, for example) and so that
we can use them to configure remote backends.
For the moment we're just providing an empty credentials object, which
will start to include both statically-configured and
helper-program-provided credentials sources in subsequent commits.
Either the environment variable TF_PLUGIN_CACHE_DIR or a setting in the
CLI config file (~/.terraformrc on Unix) allow opting in to the plugin
caching behavior.
This is opt-in because for new users we don't want to pollute their system
with extra directories they don't know about. By opting in to caching, the
user is assuming the responsibility to occasionally prune the cache over
time as older plugins become stale and unused.
This, in principle, allows us to make use of configuration information
when we populate the Meta structure, though we won't actually make use
of that until a subsequent commit.
In #15884 we adjusted the plan output to give an explicit command to run
to apply a plan, whereas before this command was just alluded to in the
prose.
Since releasing that, we've got good feedback that it's confusing to
include such instructions when Terraform is running in a workflow
automation tool, because such tools usually abstract away exactly what
commands are run and require users to take different actions to
proceed through the workflow.
To accommodate such environments while retaining helpful messages for
normal CLI usage, here we introduce a new environment variable
TF_IN_AUTOMATION which, when set to a non-empty value, is a hint to
Terraform that it isn't being run in an interactive command shell and
it should thus tone down the "next steps" messaging.
The documentation for this setting is included as part of the "...in
automation" guide since it's not generally useful in other cases. We also
intentionally disclaim comprehensive support for this since we want to
avoid creating an extreme number of "if running in automation..."
codepaths that would increase the testing matrix and hurt maintainability.
The focus is specifically on the output of the three commands we give in
the automation guide, which at present means the following two situations:
* "terraform init" does not include the final paragraphs that suggest
running "terraform plan" and tell you in what situations you might need
to re-run "terraform init".
* "terraform plan" does not include the final paragraphs that either
warn about not specifying "-out=..." or instruct to run
"terraform apply" with the generated plan file.
Martin Atkins