Commit Graph

29343 Commits

Author SHA1 Message Date
James Bardin af4f4540a9 configschema: do not expose optional attributes
Objects with optional attributes are only used for the decoding of
HCL, and those types should never be exposed elsewhere within
terraform.

Separate the external ImpliedType method from the cty.Type generated
internally for the decoder spec. This unfortunately causes our
ImpliedType method to return a different type than the
hcldec.ImpliedType function, but the former is only used within
terraform for concrete values, while the latter is used to decode
HCL. Renaming the ImpliedType methods could be done to further
differentiate them, but that does cause fairly large diff in the
codebase that does not seem worth the effort at this time.
2021-09-13 08:51:32 -04:00
Draško Radovanović c8e2be76d2
Fix a documentation typo 2021-09-11 15:33:14 +02:00
Martin Atkins 65e0c448a0 workdir: Start of a new package for working directory state management
Thus far our various interactions with the bits of state we keep
associated with a working directory have all been implemented directly
inside the "command" package -- often in the huge command.Meta type -- and
not managed collectively via a single component.

There's too many little codepaths reading and writing from the working
directory and data directory to refactor it all in one step, but this is
an attempt at a first step towards a future where everything that reads
and writes from the current working directory would do so via an object
that encapsulates the implementation details and offers a high-level API
to read and write all of these session-persistent settings.

The design here continues our gradual path towards using a dependency
injection style where "package main" is solely responsible for directly
interacting with the OS command line, the OS environment, the OS working
directory, the stdio streams, and the CLI configuration, and then
communicating the resulting information to the rest of Terraform by wiring
together objects. It seems likely that eventually we'll have enough wiring
code in package main to justify a more explicit organization of that code,
but for this commit the new "workdir.Dir" object is just wired directly in
place of its predecessors, without any significant change of code
organization at that top layer.

This first commit focuses on the main files and directories we use to
find provider plugins, because a subsequent commit will lightly reorganize
the separation of concerns for plugin launching with a similar goal of
collecting all of the relevant logic together into one spot.
2021-09-10 14:56:49 -07:00
Martin Atkins 343279110a core: Graph walk loads plugin schemas opportunistically
Previously our graph walker expected to recieve a data structure
containing schemas for all of the provider and provisioner plugins used in
the configuration and state. That made sense back when
terraform.NewContext was responsible for loading all of the schemas before
taking any other action, but it no longer has that responsiblity.

Instead, we'll now make sure that the "contextPlugins" object reaches all
of the locations where we need schema -- many of which already had access
to that object anyway -- and then load the needed schemas just in time.

The contextPlugins object memoizes schema lookups, so we can safely call
it many times with the same provider address or provisioner type name and
know that it'll still only load each distinct plugin once per Context
object.

As of this commit, the Context.Schemas method is now a public interface
only and not used by logic in the "terraform" package at all. However,
that does leave us in a rather tenuous situation of relying on the fact
that all practical users of terraform.Context end up calling "Schemas" at
some point in order to verify that we have all of the expected versions
of plugins. That's a non-obvious implicit dependency, and so in subsequent
commits we'll gradually move all responsibility for verifying plugin
versions into the caller of terraform.NewContext, which'll heal a
long-standing architectural wart whereby the caller is responsible for
installing and locating the plugin executables but not for verifying that
what's installed is conforming to the current configuration and dependency
lock file.
2021-09-10 14:56:49 -07:00
Martin Atkins 38ec730b0e core: Opportunistic schema loading during graph construction
Previously the graph builders all expected to be given a full manifest
of all of the plugin component schemas that they could need during their
analysis work. That made sense when terraform.NewContext would always
proactively load all of the schemas before doing any other work, but we
now have a load-as-needed strategy for schemas.

We'll now have the graph builders use the contextPlugins object they each
already hold to retrieve individual schemas when needed. This avoids the
need to prepare a redundant data structure to pass alongside the
contextPlugins object, and leans on the memoization behavior inside
contextPlugins to preserve the old behavior of loading each provider's
schema only once.
2021-09-10 14:56:49 -07:00
Martin Atkins a59c2fe1b9 core: EvalContextBuiltin no longer has a "Schemas"
By tolerating ProviderSchema and ProvisionerSchema potentially returning
errors, we can slightly simplify EvalContextBuiltin by having it retrieve
individual schemas when needed directly from the "Plugins" object.

EvalContextBuiltin already needs to be holding a contextPlugins instance
for other reasons anyway, so this allows us to get the same result with
fewer moving parts.
2021-09-10 14:56:49 -07:00
Martin Atkins 2bf1de1f5d core: Context.Schemas in terms of contextPlugins methods
The responsibility for actually instantiating a single plugin and reading
out its schema now belongs to the contextPlugins type, which memoizes the
results by each plugin's unique identifier so that we can avoid retrieving
the same schemas multiple times when working with the same context.

This doesn't change the API of Context.Schemas but it does restore the
spirit of an earlier version of terraform.Context which did all of the
schema loading proactively inside terraform.NewContext. In an earlier
commit we reduced the scope of terraform.NewContext, making schema loading
a separate step, but in the process of doing that removed the effective
memoization of the schema results that terraform.NewContext was providing.

The memoization here will play out in a different way than before, because
we'll be treating each plugin call as separate rather than proactively
loading them all up front, but this is effectively the same because all
of our operation methods on Context call Context.Schemas early in their
work and thus end up forcing all of the necessary schemas to load up
front nonetheless.
2021-09-10 14:56:49 -07:00
Martin Atkins 80b3fcf93e core: Replace contextComponentFactory with contextPlugins
In the v0.12 timeframe we made contextComponentFactory an interface with
the expectation that we'd write mocks of it for tests, but in practice we
ended up just always using the same "basicComponentFactory" implementation
throughout.

In the interests of simplification then, here we replace that interface
and its sole implementation with a new concrete struct type
contextPlugins.

Along with the general benefit that this removes an unneeded indirection,
this also means that we can add additional methods to the struct type
without the usual restriction that interface types prefer to be small.
In particular, in a future commit I'm planning to add methods for loading
provider and provisioner schemas, working with the currently-unused new
fields this commit has included in contextPlugins, as compared to its
predecessor basicComponentFactory.
2021-09-10 14:56:49 -07:00
Martin Atkins dcfa077adf core: contextComponentFactory doesn't need to enumerate components
In earlier Terraform versions we used the set of all available plugins of
each type to make graph-building decisions, but in modern Terraform we
make those decisions based entirely on the configuration.

Consequently, we no longer need the methods which can enumerate all of the
known plugin components of a given type. Instead, we just try to
instantiate each of the plugins that the configuration refers to and then
handle the error when that fails, which typically means that the user
needs to run "terraform init" to install some new plugins.
2021-09-10 14:56:49 -07:00
Martin Atkins d51921f085 core: Provider transformers don't use the set of all available providers
In earlier incarnations of these transformers we used the set of all
available providers for tasks such as generating implied provider
configuration nodes.

However, in modern Terraform we can extract all of the information we need
from the configuration itself, and so these transformers weren't actually
using this set of provider addresses.

These also ended up getting left behind as sets of string rather than sets
of addrs.Provider in our earlier refactoring work, which didn't really
matter because the result wasn't used anywhere anyway.

Rather than updating these to use addrs.Provider instead, I've just
removed the unused arguments entirely in the hope of making it easier to
see what inputs these transformers use to make their decisions.
2021-09-10 14:56:49 -07:00
Martin Atkins feb219a9c4 core: Un-export the LoadSchemas function
The public interface for loading schemas is Context.Schemas, which can
take into account the context's records of which plugin versions and
checksums we're expecting. loadSchemas is an implementation detail of
that, representing the part we run only after we've verified all of the
plugins.
2021-09-10 14:56:49 -07:00
James Bardin 16e8bf028a
Merge pull request #29563 from hashicorp/jbardin/optional-computed-nested-objects
remove incorrect computed check
2021-09-10 15:47:03 -04:00
James Bardin 782132da33 remove incorrect computed check
The config is already validated, and does not need to be checked in
AssertPlanValid. Add some more coverage for plan validation.
2021-09-10 14:44:07 -04:00
Laura Pacilio 43f960b197
Merge pull request #28579 from ChadBailey/patch-2
Added clarity: remote-exec connection requirement
2021-09-09 15:09:31 -04:00
Laura Pacilio 48768a0037
Merge pull request #29451 from kmadof/patch-1
Added required paramter `resource_group_name` for MSI
2021-09-09 14:54:17 -04:00
Paddy 64a95fc29f
Add docs on how to release a new major protocol version. (#29552)
Releasing a new major protocol version requires coordination between a
few different projects and codebases, and it's easy to forget a step.
This commit introduces a doc to keep track of these steps, making it
less likely one will be omitted or forgotten.
2021-09-09 11:33:07 -07:00
Alisdair McDiarmid 540217635d
Merge pull request #29549 from hashicorp/revert-29502-alisdair/json-format-version
Revert "json-output: Release format version 1.0"
2021-09-09 11:45:46 -04:00
Alisdair McDiarmid cd29c3e5fd
Revert "json-output: Release format version 1.0" 2021-09-09 11:25:35 -04:00
Laura Pacilio a819d7db3a
Merge pull request #29502 from hashicorp/alisdair/json-format-version
json-output: Release format version 1.0
2021-09-09 11:06:58 -04:00
hc-github-team-tf-core 1cd6c9fae2 Cleanup after v1.1.0-alpha20210908 release 2021-09-08 19:21:43 +00:00
hc-github-team-tf-core 4a0b8d1e09
Release v1.1.0-alpha20210908 2021-09-08 19:05:57 +00:00
Omar Ismail cb5b159228
Update terraform_remote_state data source docs: (#29534)
* Add docs for tfe_outputs data source
* Add docs for Terraform Cloud usage
2021-09-08 10:44:46 -04:00
Paul Hinze ad634f60a5
Add clarification to message about community PR review 2021-09-07 21:04:13 -05:00
Alisdair McDiarmid 7f3a29b46e
Merge pull request #29523 from hashicorp/alisdair/moved-ui
command: Render "moved" annotations in plan UI
2021-09-07 14:28:11 -04:00
James Bardin 5e8806f397 update CHANGELOG.md 2021-09-07 11:34:25 -04:00
James Bardin 1ad0421e15
Merge pull request #29522 from hashicorp/jbardin/json-blocktoattr
allow json output to marshal ConfigModeAttr blocks
2021-09-07 11:32:21 -04:00
Alisdair McDiarmid 29999c1d6f command: Render "moved" annotations in plan UI
For resources which are planned to move, render the previous run address
as additional information in the plan UI. For the case of a move-only
resource (which otherwise is unchanged), we also render that as a
planned change, but without any corresponding action symbol.

If all changes in the plan are moves without changes, the plan is no
longer considered "empty". In this case, we skip rendering the action
symbols in the UI.
2021-09-03 17:44:07 -04:00
Alisdair McDiarmid a9dfe7ba7b
Merge pull request #29521 from hashicorp/alisdair/disable-experiment-warnings
configs: Disable experiment warnings at link time
2021-09-03 17:37:47 -04:00
James Bardin ac2a870ea0 allow json output to marshal ConfigModeAttr blocks
In order to marshal config blocks using ConfigModeAttr, we need to
insert the fixup body to map hcl blocks to the attribute in the schema.
2021-09-03 13:53:52 -04:00
Laura Pacilio 3c518880d3
Merge pull request #29509 from drewmullen/d-module-source-revision-option
documentation: commit sha can be passed to ref
2021-09-03 12:11:34 -04:00
Alisdair McDiarmid 06956773dc configs: Disable experiment warnings at link time
This package level variable can be overridden at link time to allow
temporarily disabling the UI warning when experimental features are
enabled. This makes it easier to understand how UI will render when the
feature is no longer experimental.

This change is only for those developing Terraform.
2021-09-03 12:11:06 -04:00
drewmullen e3b6c6403c include sha example and git docs 2021-09-03 10:32:10 -04:00
Laura Pacilio 1e1d47d16d
Merge pull request #28345 from yvespp/destroy_provisioners_doc
document that destroy provisioners don't run with create_before_destroy
2021-09-03 10:19:30 -04:00
Yves Peter fa4c590f51
Apply suggestions from code review
Co-authored-by: Laura Pacilio <83350965+laurapacilio@users.noreply.github.com>
2021-09-03 08:23:17 +02:00
Chris Arcand 4f10de2ac6
Merge pull request #29490 from hashicorp/dont-ignore-log-goldenfiles
Clean up ignored files
2021-09-02 15:38:28 -05:00
Laura Pacilio 73a3bb2702
Merge pull request #28334 from paultyng/patch-1
Add null to type conversion docs
2021-09-02 14:47:38 -04:00
Chris Arcand 9e71da61eb Remove several ignore rules
The main purpose of this change is to avoid a problem where new golden
files added to certain directories for test purposes (like .log)
shouldn't be ignored.

Cleaning up a bit more and broadening the definition, this removes
ignore rules for artifacts of Terraform itself (state files, plans).
It's generally not recommended to be using this codebase as your
Terraform working directory anyway; build here, test elsewhere.
2021-09-01 14:37:26 -05:00
Alisdair McDiarmid b60f201eed json-output: Release format version 1.0 2021-09-01 15:15:18 -04:00
James Bardin 05f21cdff8
Merge pull request #29492 from hashicorp/jbardin/staticcheck
Add staticcheck to CI
2021-09-01 11:44:47 -04:00
James Bardin 863963e7a6 de-linting 2021-09-01 11:36:21 -04:00
James Bardin 34325d5b66 add staticcheck to circleci 2021-09-01 11:36:21 -04:00
James Bardin a3fb07d008 add staticcheck make target
cleanup the old fmtcheck script while we're in here
2021-09-01 11:36:21 -04:00
Peter Mescalchin f238e9395a Quality of life: updated all AWS document links to https:// 2021-09-01 14:02:28 +10:00
Peter Mescalchin 110d482033 Update S3 backend documentation - DynamoDB uses Partition keys, not primary keys 2021-09-01 13:59:08 +10:00
James Bardin 5eb7170f70 add staticcheck to tools 2021-08-31 17:58:40 -04:00
James Bardin 7c02ef6220
Merge pull request #29482 from hashicorp/jbardin/computed-obj-attrs
Computed values within nested object types in `AssertPlanValid`
2021-08-31 11:26:52 -04:00
James Bardin f195ce7fd4 remove temp test 2021-08-31 11:17:32 -04:00
Martin Atkins 48859417fa
Update CHANGELOG.md 2021-08-30 14:12:17 -07:00
Martin Atkins 7803f69d42 core: Enable TestContext2Plan_movedResourceBasic
This is the first test exercising the basic functionality of config-driven
move. We previously had it skipped because Terraform's previous design
of treating all three of the state artifacts as mutable attributes of
terraform.Context meant that it was too late during planning to deal with
the move operations, and thus this test was failing.

Thanks to the previous commit, which changes the terraform.Context API
such that we can defer creating the three state artifacts until we're
already doing planning, this test now works and shows Terraform correctly
handling a resource that was formerly called "a" and is now called "b",
with a "moved" block recording that renaming.
2021-08-30 13:59:14 -07:00
Martin Atkins 89b05050ec core: Functional-style API for terraform.Context
Previously terraform.Context was built in an unfortunate way where all of
the data was provided up front in terraform.NewContext and then mutated
directly by subsequent operations. That made the data flow hard to follow,
commonly leading to bugs, and also meant that we were forced to take
various actions too early in terraform.NewContext, rather than waiting
until a more appropriate time during an operation.

This (enormous) commit changes terraform.Context so that its fields are
broadly just unchanging data about the execution context (current
workspace name, available plugins, etc) whereas the main data Terraform
works with arrives via individual method arguments and is returned in
return values.

Specifically, this means that terraform.Context no longer "has-a" config,
state, and "planned changes", instead holding on to those only temporarily
during an operation. The caller is responsible for propagating the outcome
of one step into the next step so that the data flow between operations is
actually visible.

However, since that's a change to the main entry points in the "terraform"
package, this commit also touches every file in the codebase which
interacted with those APIs. Most of the noise here is in updating tests
to take the same actions using the new API style, but this also affects
the main-code callers in the backends and in the command package.

My goal here was to refactor without changing observable behavior, but in
practice there are a couple externally-visible behavior variations here
that seemed okay in service of the broader goal:
 - The "terraform graph" command is no longer hooked directly into the
   core graph builders, because that's no longer part of the public API.
   However, I did include a couple new Context functions whose contract
   is to produce a UI-oriented graph, and _for now_ those continue to
   return the physical graph we use for those operations. There's no
   exported API for generating the "validate" and "eval" graphs, because
   neither is particularly interesting in its own right, and so
   "terraform graph" no longer supports those graph types.
 - terraform.NewContext no longer has the responsibility for collecting
   all of the provider schemas up front. Instead, we wait until we need
   them. However, that means that some of our error messages now have a
   slightly different shape due to unwinding through a differently-shaped
   call stack. As of this commit we also end up reloading the schemas
   multiple times in some cases, which is functionally acceptable but
   likely represents a performance regression. I intend to rework this to
   use caching, but I'm saving that for a later commit because this one is
   big enough already.

The proximal reason for this change is to resolve the chicken/egg problem
whereby there was previously no single point where we could apply "moved"
statements to the previous run state before creating a plan. With this
change in place, we can now do that as part of Context.Plan, prior to
forking the input state into the three separate state artifacts we use
during planning.

However, this is at least the third project in a row where the previous
API design led to piling more functionality into terraform.NewContext and
then working around the incorrect order of operations that produces, so
I intend that by paying the cost/risk of this large diff now we can in
turn reduce the cost/risk of future projects that relate to our main
workflow actions.
2021-08-30 13:59:14 -07:00