In order to parse provider, resource and data source configuration from
HCL2 config files, we need to know the relevant configuration schema.
This new method allows Terraform Core to request these from a provider.
This is a breaking change to this interface, so all of its implementers
in this package are updated too. This includes concrete implementations
of the new method in helper/schema that use the schema conversion code
added in an earlier commit to produce a configschema.Block automatically.
Plugins compiled against prior versions of helper/schema will not have
support for this method, and so calls to them will fail. Callers of
this new method will therefore need to sniff for support using the
SchemaAvailable field added to both ResourceType and DataSource.
This careful handling will need to persist until next time we increment
the plugin protocol version, at which point we can make the breaking
change of requiring this information to be available.
Having a reference to the originating source range will allow us to
generate decent error messages if certain references can't be resolved
at interpolation time.
This is not yet populated or used. It will never be populated nor used by
the current HCL/HIL-based interpolation path, but will be used by the
experimental HCL2-based interpolation path to give it the necessary info
to produce diagnostics.
The value-conversion machinery is also needed in the main "terraform"
package to help us populate our HCL2 evaluation scope, so a subset of the
shim functions move here into a new package where they can be public.
Some of them remain private within the config package since they depend
on some other symbols in the config package, and they are not needed
by outside callers anyway.
Previously we were showing only the summaries when converting to a string
error, but HCL generates summaries that indicate only the _type_ of error,
expecting that the detail will give the details, and so we need to show
both in order to produce a useful error message.
This launches Terraform inside a headless dlv configured to accept a
remote debugging process. It's configured this way so it can be easily
used from a debugger GUI integrated into an IDE/editor, but it can also
be used from the CLI by running the command it prints.
Using a remote debugger here is useful even when debugging with the CLI,
since it keeps Terraform's verbose and colorful output from interfering
with the debugger UI.
This new version supports a conversion from object types to map types,
which is important for Terraform because HCL2 { ... } syntax produces
objects but lots of Terraform attributes require maps.
HCL will be the most frequent origin of a source range, so this helper
should make it easier to translate these between the two worlds so we
can more easily use them in native Terraform diagnostics and other
messaging.
Uses Levenshtein distance to decide if the input is similar enough to one
of the given suggestions, and returns that suggestion if so.
The distance threshold of three was arrived at experimentally, and has
no objective basis.
This uses the new diagnostics printer for config-related errors in the
main five commands that deal with config.
The immediate motivation for this is to allow HCL2-produced diagnostics
to be printed out in their full fidelity, though it also slightly changes
the presentation of other errors so that they are not presented in all
red text, which can be hard to read on some terminals.
This new method showDiagnostics takes any value that would be accepted by
tfdiags.Append and renders it to the UI.
This is intended to encourage consistent handling of the different kinds
of errors and diagnostics that can be produced, and allow richer error
objects like the HCL2 diagnostics to be easily unwrapped and shown in
their full-fidelity.
Due to the use of interfaces, Diagnostics is not super-friendly to the gob
encoding we currently use for plugin RPC. To mitigate this, we provide
a helper that converts all of the wrapped objects into a predictable flat
structure that we can pre-emptively register with gob.
This means that the decoded Diagnostics still has the same meaning as
the original, though the original wrapped errors (if any) are lost and
thus our errwrap integration won't be effective any longer.
Previously we were using fmt.Sprintf and thus forcing the stringification
of the wrapped error.
Using errwrap allows us to unpack the original error at the top of the
stack, which is useful when the wrapped error is really a hcl.Diagnostics
containing potentially-multiple errors and possibly warnings.
Currently we lean heavily on the Go error type as our primary means of
describing errors, and along with that use several more specialized
implementations of it in different spots for additional capabilities such
as multiple errors in one object, source code range references, etc.
We also have a rather ad-hoc approach of returning an array of warnings
from certain functions along with one or multiple errors.
This rather-disorganized approach makes it hard for us to present
user-facing error messages consistently. As a step towards mitigating
this, package tfdiags provides a model for user-facing error and warning
messages and helper functions for creating them from various other
error and warning types used elsewhere in Terraform.
This mechanism is intended to be used to report errors and warnings where
the audience is the Terraform user, and so it may go a few layers deep
down the call stack into codepaths like config parsing, interpolation, etc
but is primarily a UX concern. The deepest reaches of Terraform core will
continue using "error" as normal, with higher layers preparing error
messages for presentation to the user.
To avoid needing to change the interface of every function that might
generate error diagnostics, the Diagnostics type can be "smuggled" via
an error value through other APIs and then unwrapped at the other end,
though it will lose any naked warnings (without at least one error) along
the way, and so codepaths that are expected to generate warnings
(validation, primarily) should use the concrete Diagnostics type
throughout the call chain.
We have a generated cookie for googlesource.com so that we don't get so rate-limited when cloning Google-hosted Go libraries.
The previous credential was invalidated, so this is a newly-generated one. This credential does nothing except allow us to fetch git repositories from go.googlesource.com with a slightly-higher rate limit.
We don't currently have any need for this information, but we're
propagating it out of helper/schema here pre-emptively so that once we
later have a use for it we will not need to rebuild the providers to gain
access to it.
The long-term expected use-case for this is to have Terraform Core use
static analysis techniques to trace the path of sensitive data through
interpolations so that intermediate results can be flagged as sensitive
too, but we have a lot more work to do before such a thing would actually
be possible.
As part of moving to the next-generation HCL implementation,
Terraform Core is getting its own representation of configuration schema
that is tailored for configuration-processing use-cases. The capabilities
of this are a subset of the helper/schema model primarily concerned with
the configuration structure and value types, leaving detailed validation
and defaults for helper/schema to still solve.
These new methods allow mechanical creation of a schema in the new Core
schema model from a schema expressed in the helper/schema model. This is
not yet used as of this commit, but will be used later to implement some
new ResourceProvider methods that will allow core to obtain the schema
for provider, resource and data source configuration while remaining
source-compatible with existing provider implementations.
zcldec now has its own function for computing the implied type for a spec,
so we can use that instead of our own logic.
The zcldec logic is more general since its spec model is more general than
our schema model here, but it produces the same results for the subset
of specifications that our DecoderSpec method produces.
This returns a cty.Type that the caller can expect to recieve when
decoding a value using the (not yet implemented) decoder specification
for a given schema.
Internal errors from S3 are usually transient, and can be immediately retried.
Make 2 attempts at retreiving the state object before returning an error.
Terraform has a _lot_ of functions written against HIL's function API, and
we're not ready to rewrite them all yet, so instead we shim the HIL
function API to conform to the HCL2 (really: cty) function API and thus
allow most of our existing functions to work as expected when called from
HCL2-based config files.
Not all of the functions can be fully shimmed in this way due to depending
on HIL implementation details that we can't mimic through the HCL2 API.
We don't attempt to address that yet, and instead just let them fail when
called. We will eventually address this by using first-class HCL2
functions for these few cases, thus avoiding the HIL API altogether where
we need to. (The methodology for that is already illustrated here in the
provision of jsonencode and jsondecode functions that are HCL2-native.)
This early validation uses interpolation of a placeholder value to achieve
some "best effort" validation of the validity of the count attribute.
Since HCL2-specified resources can't be interpolated using the main
interpolator, here we branch and use the HCL2 API to do a
largely-equivalent (though slightly less accurate) check.
In the long run we don't really need this extra check at all, since the
validation walk does a more accurate version of the same thing. However,
we're preserving this for now in the interests of minimizing the amount
of change for the main codepath during our experiment.
Currently the default for tests is to use the old HCL loader, but we need
to be able to test aspects of the new loader as we work through the
experimental phase. This new function testConfigHCL2 is the same as
testConfig except that it forces the use of HCL2 even if the opt-in
comment isn't present, thus allowing us to implement tests that ensure
that the exact same file works in both the old and new cases.
Once the HCL2 loader becomes the default this function will be removed
and callers will start calling into the normal testConfig function.
Use the new HCL2 config loader when the opt-in comment #terraform:hcl2 is
present in a .tf file.
For now this is disabled for "normal" builds and enabled only if
explicitly configured via a linker flag during build. This is because it's
not yet in a good state to be released: the HCL2 loader produces RawConfig
objects that the validator and interpolator can't yet deal with, and so
using HCL2 for anything non-trivial currently causes Terraform to crash
in real use.
This loader uses the HCL2 parser and decoder to process a config file,
and then transforms the result into the same shape as would be produced
by the HCL config loader.
To avoid making changes to the existing config structures (which are
depended on across much of the codebase) we first decode into a set of
HCL2-tailored structs and then process them into the public-facing structs
that a loader is expected to return. This is a compromise to keep the
config package API broadly unchanged for now. Once we're ready to remove
the old HCL loader (which implies that we're ready to support HCL2
natively elsewhere in the codebase) we will be able to simplify this
quite considerably.
Due to some mismatches of abstraction between HCL/HIL and HCL2, some
shimming is required to get the required result.
At this time we're not ready to refactor the various uses of RawConfig
in Terraform core, so we'll smuggle a HCL2 body within a degenerate
RawConfig object that we can then recognize and unpack once this object
is returned to us in an interpolation call.
Martin Atkins