In earlier Terraform versions we had an extra validation step prior to
the graph walk which tried to partially validate root module input
variable values (just checking their type constraints) and then return
error messages which specified as accurately as possible where the value
had originally come from.
We're now handling that sort of validation exclusively during the graph
walk so that we can share the main logic between both root module and
child module variable values, but previously that shared code wasn't
able to generate such specific information about where the values had
originated, because it was adapted from code originally written to only
deal with child module variables.
Here then we restore a similar level of detail as before, when we're
processing root module variables. For child module variables, we use
synthetic InputValue objects which state that the value was declared
in the configuration, thus causing us to produce a similar sort of error
message as we would've before which includes a source range covering
the argument expression in the calling module block.
Previously we had three different layers all thinking they were
responsible for substituting a default value for an unset root module
variable:
- the local backend, via logic in backend.ParseVariableValues
- the context.Plan function (and other similar functions) trying to
preprocess the input variables using
terraform.mergeDefaultInputVariableValues .
- the newer prepareFinalInputVariableValue, which aims to centralize all
of the variable preparation logic so it can be common to both root and
child module variables.
The second of these was also trying to handle type constraint checking,
which is also the responsibility of the central function and not something
we need to handle so early.
Only the last of these consistently handles both root and child module
variables, and so is the one we ought to keep. The others are now
redundant and are causing prepareFinalInputVariableValue to get a slightly
corrupted view of the caller's chosen variable values.
To rectify that, here we remove the two redundant layers altogether and
have unset root variables pass through as cty.NilVal all the way to the
central prepareFinalInputVariableValue function, which will then handle
them in a suitable way which properly respects the "nullable" setting.
This commit includes some test changes in the terraform package to make
those tests no longer rely on the mergeDefaultInputVariableValues logic
we've removed, and to instead explicitly set cty.NilVal for all unset
variables to comply with our intended contract for PlanOpts.SetVariables,
and similar. (This is so that we can more easily catch bugs in callers
where they _don't_ correctly handle input variables; it allows us to
distinguish between the caller explicitly marking a variable as unset vs.
not describing it at all, where the latter is a bug in the caller.)
Previously we had a significant discrepancy between these two situations:
we wrote the raw root module variables directly into the EvalContext and
then applied type conversions only at expression evaluation time, while
for child modules we converted and validated the values while visiting
the variable graph node and wrote only the _final_ value into the
EvalContext.
This confusion seems to have been the root cause for #29899, where
validation rules for root module variables were being applied at the wrong
point in the process, prior to type conversion.
To fix that bug and also make similar mistakes less likely in the future,
I've made the root module variable handling more like the child module
variable handling in the following ways:
- The "raw value" (exactly as given by the user) lives only in the graph
node representing the variable, which mirrors how the _expression_
for a child module variable lives in its graph node. This means that
the flow for the two is the same except that there's no expression
evaluation step for root module variables, because they arrive as
constant values from the caller.
- The set of variable values in the EvalContext is always only "final"
values, after type conversion is complete. That in turn means we no
longer need to do "just in time" conversion in
evaluationStateData.GetInputVariable, and can just return the value
exactly as stored, which is consistent with how we handle all other
references between objects.
This diff is noisier than I'd like because of how much it takes to wire
a new argument (the raw variable values) through to the plan graph builder,
but those changes are pretty mechanical and the interesting logic lives
inside the plan graph builder itself, in NodeRootVariable, and
the shared helper functions in eval_variable.go.
While here I also took the opportunity to fix a historical API wart in
EvalContext, where SetModuleCallArguments was built to take a set of
variable values all at once but our current caller always calls with only
one at a time. That is now just SetModuleCallArgument singular, to match
with the new SetRootModuleArgument to deal with root module variables.
This test seems to be a holdover from the many-moons-ago switch from one
graph for all operations to separate graphs for plan and apply. It is
effectively just a copy of a subset of the content of the Context.Validate
function and is a maintainability hazard because it tends to lag behind
updates to that function unless changes there happen to make it fail.
This test doesn't cover anything that the other validate context tests
don't exercise as an implementation detail of calling Context.Validate,
so I've just removed it with no replacement.
Our original messaging here was largely just lifted from the equivalent
message for unknown values in "count", and it didn't really include any
specific advice on how to update a configuration to make for_each valid,
instead focusing only on the workaround of using the -target planning
option.
It's tough to pack in a fully-actionable suggestion here since unknown
values in for_each keys tends to be a gnarly architectural problem rather
than a local quirk -- when data flows between modules it can sometimes be
unclear whether it'll end up being used in a context which allows unknown
values.
I did my best to summarize the advice we've been giving in community forum
though, in the hope that more people will be able to address this for
themselves without asking for help, until we're one day able to smooth
this out better with a mechanism such as "partial apply".
Thank you. Your suggestion to use "instead of" makes the sentence even more easy to understand.
Co-authored-by: Laura Pacilio <83350965+laurapacilio@users.noreply.github.com>
Normally when we cross-compile we do so without CGo, because we don't have
suitable C headers available for systems other than the host.
However, building for macOS on macOS is special because there are
sufficient headers available on darwin_amd64 to build for both
darwin_amd64 _and_ darwin_arm64. Also, we _must_ use CGo on macOS because
the system resolver is only available via darwin's libc, and so building
without CGo produces executables that don't resolve hostnames correctly.
This is a conditional in bash to avoid having to duplicate the entire
step. Perhaps later we'll find a more general version of this which can
avoid the special case, but this is sufficient for the moment.
We recently made a change to how provisioners upload files in order to
address an unintended remote code execution vector when using SSH, which
revealed that we had not previously documented well enough the expected
contract for how provisioners upload files to remote systems, and so some
users were depending on unintended consequences of the the bug now fixed.
We are retaining the fix on security-related grounds, but this is a good
prompt to be clearer in the docs about what exactly Terraform is doing
when asked to upload files over SSH and WinRM, so users can understand
what is supported and write their configurations accordingly.
This also includes an additional section to the v1.1 upgrade guide, since
we apparently neglected to document this intentional breaking change in
the first draft of that page.
Of course, provisioners as a whole remain a last resort, and so we're
documenting this as hopefully a helpful aid to those who have no other
option, and not meaning in any way to recommend their use for any new
use-cases.
The specific output order is meaningless, but it should always be the same after
two Encode() calls with identical (ignoring in-memory order) dependency sets.
In our build workflow we'll treat Linux distribution packaging (currently
.deb and .rpm packages) as a separate job, instead of embedding it into
the "build" job, so that this step can happen concurrently with the other
derived actions like the docker image build, and the e2etest runs.
We can use an extra matrix dimension to select which execution environment
we'll use for each GOOS/GOARCH pair, and thus avoid duplicating the
job definition for darwin just to set runs-on: macos-latest for it.
This is not really an intended use of a matrix dimension because it's
directly related to the existing "goos" one, rather than being an
independent third dimension, but it doesn't matter in practice because
we're using the "include" option to specify exact combinations, and thus
we're not relying on the built-in functionality to generate all possible
matrix combinations.
This should eventually grow to be a step that actually verifies the
validity of the docs source prior to publishing the artifact that a
downstream publishing pipeline can consume, but for the moment it's really
just a placeholder since we have no such validation step and no downstream
pipeline consuming this artifact.
The general idea here is that the artifacts from this workflow should be
sufficient for all downstream release steps to occur without any direct
access to the Terraform CLI repository, and so this is intended to
eventually meet that ideal but as of this commit the website docs
publishing step _does_ still depend on direct access to this repository.
This uses the decoupled build and run strategy to run the e2etests so that
we can arrange to run the tests against the real release packages produced
elsewhere in this workflow, rather than ones generated just in time by
the test harness.
The modifications to make-archive.sh here make it more consistent with its
originally-intended purpose of producing a harness for testing "real"
release executables. Our earlier compromise of making it include its own
terraform executable came from a desire to use that script as part of
manual cross-platform testing when we weren't yet set up to support
automation of those tests as we're doing here. That does mean, however,
that the terraform-e2etest package content must be combined with content
from a terraform release package in order to produce a valid contest for
running the tests.
We use a single job to cross-compile the test harness for all of the
supported platforms, because that build is relatively fast and so not
worth the overhead of matrix build, but then use a matrix build to
actually run the tests so that we can run them in a worker matching the
target platform.
We currently have access only to amd64 (x64) runners in GitHub Actions
and so for the moment this process is limited only to the subset of our
supported platforms which use that architecture.
For the moment this is just an experimental additional sidecar package
build process, separate from the one we really use for releases, so that
we can get some experience building in the GitHub Actions environment
before hopefully eventually switching to using the artifacts from this
process as the packages we'll release through the official release
channels.
It will react to any push to one of our release branches or to a release
tag by building official-release-like .zip, .deb, and .rpm packages, along
with Docker images, based on the content of the corresponding commit.
For the moment this doesn't actually produce _shippable_ packages because
in particular it doesn't know how to update our version/version.go file
to hard-code the correct version number. Once Go 1.18 is release and we've
upgraded to it we'll switch to using debug.ReadBuildInfo to determine
our version number at runtime and so no longer need to directly update
a source file for each release, but that functionality isn't yet available
in our current Go 1.17 release.
When creating a Set of BasicEdges, the Hashcode function is used to determine
map keys for the underlying set data structure.
The string hex representation of the two vertices' pointers is unsafe to use
as a map key, since these addresses may change between the time they are added
to the set and the time the set is operated on.
Instead we modify the Hashcode function to maintain the references to the
underlying vertices so they cannot be garbage collected during the lifetime
of the Set.
TransitiveReduction does not rely on having a single root, and only
must be free of cycles.
DepthFirstWalk and ReverseDepthFirstWalk do not do a topological sort,
so if order matters TransitiveReduction must be run first.
These two functions were left during a refactor to ensure the old
behavior of a sorted walk was still accessible in some manner. The
package has since been removed from any public API, and the sorted
versions are no longer called, so we can remove them.
Create a separate `validateMoveStatementGraph` function so that
`ValidateMoves` and `ApplyMoves` both check the same conditions. Since
we're not using the builtin `graph.Validate` method, because we may have
multiple roots and want better cycle diagnostics, we need to add checks
for self references too. While multiple roots are an error enforced by
`Validate` for the concurrent walk, they are OK when using
`TransitiveReduction` and `ReverseDepthFirstWalk`, so we can skip that
check.
Apply moves must first use `TransitiveReduction` to reduce the graph,
otherwise nodes may be skipped if they are passed over by a transitive
edge.
Changing only the index on a nested module will cause all nested moves
to create cycles, since their full addresses will match both the From
and To addresses. When building the dependency graph, check if the
parent is only changing the index of the containing module, and prevent
the backwards edge for the move.
Add a method for checking if the From and To addresses in a move
statement are only changing the indexes of modules relative to the
statement module.
This is needed because move statement nested within the module will be
able to match against both the From and To addresses, causing cycles in
the order of move operations.
There was an unintended regression in go-getter v1.5.9's GitGetter which
caused us to temporarily fork that particular getter into Terraform to
expedite a fix. However, upstream v1.5.10 now includes a
functionally-equivalent fix and so we can heal that fork by upgrading.
We'd also neglected to update the Module Sources docs when upgrading to
go-getter v1.5.9 originally and so we were missing documentation about the
new "depth" argument to enable shadow cloning, which I've added
retroactively here along with documenting its restriction of only
supporting named refs.
This new go-getter release also introduces a new credentials-passing
method for the Google Cloud Storage getter, and so we must incorporate
that into the Terraform-level documentation about module sources.
Changing only the index on a nested module will cause all nested moves
to create cycles, since their full addresses will match both the From
and To addresses. When building the dependency graph, check if the
parent is only changing the index of the containing module, and prevent
the backwards edge for the move.
This paragraph is trying to say that try only works for dynamic errors and
not for errors that are _not_ based on dynamic decision-making in
expressions.
I'm not sure if this typo was always here or if it was mistakenly "corrected"
at some point, but either way the word "probably" changes the meaning
of this sentence entirely, making it seem like Terraform is hedging
the likelihood of a problem rather than checking exactly for one.
Martin Atkins