Commit Graph

668 Commits

Author SHA1 Message Date
Martin Atkins b91bd62747 config/configschema: Sensitive flag for attributes
We don't currently have any need for this information, but we're
propagating it out of helper/schema here pre-emptively so that once we
later have a use for it we will not need to rebuild the providers to gain
access to it.

The long-term expected use-case for this is to have Terraform Core use
static analysis techniques to trace the path of sensitive data through
interpolations so that intermediate results can be flagged as sensitive
too, but we have a lot more work to do before such a thing would actually
be possible.
2017-10-04 16:35:11 -07:00
Martin Atkins 0ef985cada config/configschema: Block.ImpliedType delegates to zcldec
zcldec now has its own function for computing the implied type for a spec,
so we can use that instead of our own logic.

The zcldec logic is more general since its spec model is more general than
our schema model here, but it produces the same results for the subset
of specifications that our DecoderSpec method produces.
2017-10-04 16:35:11 -07:00
Martin Atkins 2b622fe31a config/configschema: Block.DecoderSpec
This returns a decoding specification that can be used with the hcldec
package to decode a body into a cty.Value of an object type.
2017-10-04 16:35:11 -07:00
Martin Atkins d0d829848a config/configschema: Block.ImpliedType method
This returns a cty.Type that the caller can expect to recieve when
decoding a value using the (not yet implemented) decoder specification
for a given schema.
2017-10-04 16:35:11 -07:00
Martin Atkins f117906bdb config/configschema: InternalValidate for blocks
This checks that a schema complies with the documented constraints on
which values are valid. It is primarily intended for use in tests.
2017-10-04 16:35:11 -07:00
Martin Atkins d712a04c32 config/configschema: MinItems and MaxItems for nested blocks 2017-10-04 16:35:11 -07:00
Martin Atkins dadec5210b config/configschema: stub out initial API 2017-10-04 16:35:11 -07:00
Martin Atkins b851fa71c9 config: Make HIL-based functions available to HCL2 via a shim
Terraform has a _lot_ of functions written against HIL's function API, and
we're not ready to rewrite them all yet, so instead we shim the HIL
function API to conform to the HCL2 (really: cty) function API and thus
allow most of our existing functions to work as expected when called from
HCL2-based config files.

Not all of the functions can be fully shimmed in this way due to depending
on HIL implementation details that we can't mimic through the HCL2 API.
We don't attempt to address that yet, and instead just let them fail when
called. We will eventually address this by using first-class HCL2
functions for these few cases, thus avoiding the HIL API altogether where
we need to. (The methodology for that is already illustrated here in the
provision of jsonencode and jsondecode functions that are HCL2-native.)
2017-10-03 17:47:01 -07:00
Martin Atkins 34e9de605c config: tests for HIL-to-HCL2 value shimming 2017-10-03 17:47:01 -07:00
Martin Atkins bbf9725134 config: Validate resource "count" for HCL2-specified resources
This early validation uses interpolation of a placeholder value to achieve
some "best effort" validation of the validity of the count attribute.
Since HCL2-specified resources can't be interpolated using the main
interpolator, here we branch and use the HCL2 API to do a
largely-equivalent (though slightly less accurate) check.

In the long run we don't really need this extra check at all, since the
validation walk does a more accurate version of the same thing. However,
we're preserving this for now in the interests of minimizing the amount
of change for the main codepath during our experiment.
2017-10-03 17:47:01 -07:00
Martin Atkins 71e68f06c4 config: allow tests to force using the HCL2 loader
Currently the default for tests is to use the old HCL loader, but we need
to be able to test aspects of the new loader as we work through the
experimental phase. This new function testConfigHCL2 is the same as
testConfig except that it forces the use of HCL2 even if the opt-in
comment isn't present, thus allowing us to implement tests that ensure
that the exact same file works in both the old and new cases.

Once the HCL2 loader becomes the default this function will be removed
and callers will start calling into the normal testConfig function.
2017-10-03 17:47:01 -07:00
Martin Atkins d91327eaa0 config: allow HCL2 experiment opt-in (build-time flag to enable)
Use the new HCL2 config loader when the opt-in comment #terraform:hcl2 is
present in a .tf file.

For now this is disabled for "normal" builds and enabled only if
explicitly configured via a linker flag during build. This is because it's
not yet in a good state to be released: the HCL2 loader produces RawConfig
objects that the validator and interpolator can't yet deal with, and so
using HCL2 for anything non-trivial currently causes Terraform to crash
in real use.
2017-10-03 17:47:01 -07:00
Martin Atkins b0215fcd0f config: HCL2 config loader
This loader uses the HCL2 parser and decoder to process a config file,
and then transforms the result into the same shape as would be produced
by the HCL config loader.

To avoid making changes to the existing config structures (which are
depended on across much of the codebase) we first decode into a set of
HCL2-tailored structs and then process them into the public-facing structs
that a loader is expected to return. This is a compromise to keep the
config package API broadly unchanged for now. Once we're ready to remove
the old HCL loader (which implies that we're ready to support HCL2
natively elsewhere in the codebase) we will be able to simplify this
quite considerably.

Due to some mismatches of abstraction between HCL/HIL and HCL2, some
shimming is required to get the required result.
2017-10-03 17:47:01 -07:00
Martin Atkins edbbe41b44 config: allow a HCL2 body to piggy-back on a RawConfig
At this time we're not ready to refactor the various uses of RawConfig
in Terraform core, so we'll smuggle a HCL2 body within a degenerate
RawConfig object that we can then recognize and unpack once this object
is returned to us in an interpolation call.
2017-10-03 17:47:01 -07:00
James Bardin 08ac4f3a2e Merge pull request #16089 from wgliang/optimization-merge-slice
Optimization merge slice
2017-10-02 16:40:13 -04:00
James Bardin 1fbb2d611a remove extra slash from registry urls
A refactor introduced an extra `/` in the download url, which causes an
extra redirect during discovery.

Improve a registry test to verify that detection doesn't require the
registry after the modules have been fetched.
2017-09-28 14:45:02 -04:00
Ariel Alonso 550ae05819 config: new "transpose" interpolation function
This function takes a map of lists of strings and inverts it so that
the string values become keys and the keys become items within the
corresponding lists.
2017-09-28 10:29:25 -07:00
Martin Atkins a28b5d295e config: improve interpolation function test output
These tests were written before subtest support was available. By running
them as subtests we can get better output in the event of an error, or
in verbose mode.
2017-09-26 14:31:43 -07:00
James Bardin d78b575536 Merge pull request #16160 from hashicorp/jbardin/get-subdir
Handle module source subdirectories in Terraform
2017-09-26 09:23:19 -04:00
James Bardin 2b3e48aa70 fix test comments 2017-09-26 09:12:14 -04:00
Krzysztof Wilczynski 6e7e03f4ea config: new "abs" interpolation function
This new function returns the absolute value for a given number.
2017-09-25 14:19:18 -07:00
James Bardin f6e89bb8c3 record the subdirectory in the FolderStorage
Module detection currently requires calling the registry to determine
the subdirectory. Since we're not directly accessing the subdirectory
through FolderStorage, and now handling it within terraform so modules can
reference sibling paths, we need to call out to the registry every
time we load a configuration to verify the subdirectory for the module,
which is returned during the Detect.

Record the subdirectories for each module in the top-level of the
FolderStorage path for retrieval during Tree.Load. This lets us bypass
Detection altogether, modules can be loaded without redetecting.
2017-09-22 22:03:38 -04:00
James Bardin 1b01f18920 handle go-getter subdirs in Tree.Load
In order to remain backward compatible with some modules, we need to
handle subdirs during Load. This means duplicating part of the go-getter
code path for subDir handling so we can resolve any subDirs and globs
internally, while keeping the entire remote directory structure within
the file storage.
2017-09-22 20:32:44 -04:00
James Bardin 59089ef461 add test loading an archive with `//*` subdir
The registry uses the `//*` subdir format to unpack github archives. Add
a test to cover this pattern.
2017-09-21 15:32:55 -05:00
James Bardin 38569c8508 add tests for get from tar subdir
Test that we can get a subdirectory from a tarball (or any other
"packed" source that we support).

The 'tar-subdir-to-parent' test highlights a regression where the
subdirectory module references a module in its parent directory. This
breaks the intended use ofr the subdirectory and the implementation in
go-getter. We need to fix this in terraform, and possible plan warnings
and deprecations for this type of source.
2017-09-21 13:12:33 -05:00
James Bardin 3d3992454e Add basic ACC test
This just does a lookup in the registry to verify that the download api
works.
2017-09-15 16:16:29 -04:00
James Bardin ba14cf9511 add test fixture for new getter discovery behavior 2017-09-15 16:16:29 -04:00
James Bardin f9fb2b4c9e move local module detection ahead of the registry
The getter.FileDetector was intended to be the final detector, only
converting a path to a file URL and returning a true in all cases. We
want to check for a local module before checking the registry so no
local modules that happen to match a registry module are broken.

Wrap the getter.FileDetector to check the module source's existence
before delegating the search to the registry.
2017-09-15 16:16:28 -04:00
James Bardin d216d19f21 change the FolderStorage key
updating the key will cause the FolderStorage hash to change forcing
modules to be re-fetched. This is required because any configurations
using the subDir notation will have the configuration in the wrong
directory.
2017-09-15 16:16:28 -04:00
James Bardin 6c20141c30 remove subdir handling from Tree.Load
Terraform was redundantly handling `//dir` notation which should be
handled by go-getter. Rather than allowing go-getter to unpack a subdir
as expected, the subdir was stripped off and accessed through the module
configuration.

This scheme will no longer works now that go-getter supports `*`
subdirectories
    (e.g. `//*` would be analogous to `tar --strip-components=1`).

Even though this allows Terraform to use go-getter's native unpacking,
detection is still done separately because Detect requires a `pwd` which
is dependent on the configuration directory and not known to the
global FolderStorage.
2017-09-15 16:16:28 -04:00
James Bardin a83ff57aea Add registry detector
Add a getter.Detector for detecting registry modules and looking up
the download location of the latest version. This is essentially a
temporary API until constraint solving is supported by the registry, as
then we'll have to supply the full set of known contraints to the
registry at once for resolution and we will fetch specific versions of
modules.
2017-09-15 16:16:28 -04:00
Micah Hausler 7fa4b648bb config: multi-line "indent" function
This function prepends a number of spaces on to all but the first line
of a string containing multiple lines of text.
2017-09-14 09:47:48 -07:00
Wang Guoliang 9a97c348f0
optimization:Merge slices 2017-09-13 21:08:41 +08:00
Martin Atkins 0a342e8dc2 config: allow local value interpolations in count
There is some additional, early validation on the "count" meta-argument
that verifies that only suitable variable types are used, and adding local
values to this whitelist was missed in the initial implementation.
2017-09-01 17:54:05 -07:00
Martin Atkins 8cd0ee80e5 config: merge/append for local values
It seems that this somehow got lost in the commit/rebase shuffle and
wasn't caught by the tests that _did_ make it because they were all using
just one file.

As a result of this bug, locals would fail to work correctly in any
configuration with more than one .tf file.

Along with restoring the append/merge behavior, this also reworks some of
the tests to exercise the multi-file case as better insurance against
regressions of this sort in future.

This fixes #15969.
2017-09-01 17:51:13 -07:00
Martin Atkins c12d64f340 Use t.Helper() in our test helpers
Go 1.9 adds this new function which, when called, marks the caller as
being a "helper function". Helper function stack frames are then skipped
when trying to find a line of test code to blame for a test failure, so
that the code in the main test function appears in the test failure output
rather than a line within the helper function itself.

This covers many -- but probaly not all -- of our test helpers across
various packages.
2017-08-28 09:59:30 -07:00
Mitchell Hashimoto adcf41f076
config: parse description field for outputs
We added the description field in 0.9 but we never parsed it because we
didn't have a use for it. As we prepare to use this field, let's start
parsing it out
2017-08-28 09:42:03 -07:00
Sunny 06fd5d7a3e config: urlencode interpolation function
This escapes all characters that might have a special interpretation when embedded into a portion of a URL, including slashes, equals signs and ampersands.
2017-08-22 11:26:09 -07:00
Gorka Lerchundi Osa a303817e03 config: base64gzip interpolation function
Since Terraform's internals are not 8-bit clean (it assumes UTF-8
strings), we can't implement raw gzip directly. We're going to add
support where it makes sense for passing data to attributes as
base64 so that the result of this function can be used.
2017-08-22 10:44:55 -07:00
Martin Atkins d41d58967f config: parsing of local.foo variables for interpolation 2017-08-21 15:15:25 -07:00
Martin Atkins f6797d6cb0 config: parsing of "locals" blocks in configuration 2017-08-21 15:15:25 -07:00
Sebastian Maj 28b5708fef config: jsonencode function to support nested lists and maps 2017-08-18 09:57:57 -07:00
JWal31 117f44bcda config: Add "flatten" interpolation function
This function turns a list of lists or any arbitrary number of nested
lists into a flat list of primitive values.
2017-08-16 13:46:19 -07:00
Radek Simko 67ceb1ab07
config: Make 'id' a reserved field name 2017-08-02 20:02:44 +02:00
Radek Simko 07cbd54fbc Actively disallow reserved field names in schema (#15522) 2017-07-10 21:51:55 -07:00
Robert Liebowitz 006744bfe0 Use all tfvars files in working directory
As a side effect, several commands that previously did not have a failure
state can now fail during meta-parameter processing.
2017-07-05 17:24:17 -07:00
trung 681661a539 #15291: config/interpolate_funcs: Added contains() function to test if a given element is present in the list 2017-06-16 15:05:19 -07:00
Martin Atkins f70318097a config: fix provider version constraint validation
Previously we were using the "semver" library to parse version
constraints, but we switched over to go-version and encapsulated it
inside our own plugin/discovery package to reduce dependency sprawl in
the code.

This particular situation was missed when updating references to the new
path, which meant that our validation code disagreed with the rest of
the code about what is considered a valid version constraint string.
By using the correct function, we ensure that we catch early any invalid
versions.
2017-06-09 14:03:59 -07:00
Martin Atkins a8a64c66c0 config/module: helper to visit all modules in a tree 2017-06-09 14:03:59 -07:00
Martin Atkins 32a5c62639 config: parse provider version constraints into a constraint map 2017-06-09 14:03:59 -07:00