Commit Graph

704 Commits

Author SHA1 Message Date
James Bardin a7df650f01 fix vet error 2017-10-27 09:08:15 -04:00
James Bardin 9bda82ccf1 add example for the reason behind versionedPathKey 2017-10-27 09:08:15 -04:00
James Bardin d6f75cc022 use URLs rather than strings is registry functions
Parse all the registry strings as urls, and compine with path.Join to
for better validation.
2017-10-27 09:08:15 -04:00
James Bardin 4a0de691f5 update test for error condition 2017-10-27 09:08:15 -04:00
James Bardin 6c808347d7 flag off registry ACC test
Was missing the TF_ACC check
2017-10-27 09:08:15 -04:00
James Bardin 1756b3d09f ProviderConfig needs an Inherited flag
If a provider configuration is inherited from another module, any
interpolations in that config won't have variables declared locally. Let
the config only be validated in it's original location.
2017-10-27 09:08:15 -04:00
James Bardin 0afd4a9097 Lookup registry module versions during Tree.Load.
Registry modules can't be handled directly by the getter.Storage
implementation, which doesn't know how to handle versions. First see if
we have a matching module stored that satisfies our constraints. If
not, and we're getting or updating, we can look it up in the registry.

This essentially takes the place of a "registry detector" for go-getter,
but required the intermediate step of resolving the version dependency.

This also starts breaking up the huge Tree.Load method into more
manageable parts. It was sorely needed, as indicated by the difficulty
encountered in this refactor. There's still a lot that can be done to
improve this, but at least there are now a few easier to read methods
when we come back to it.
2017-10-27 09:08:15 -04:00
James Bardin 0d10564a74 remove the registryDetector
The detection of registry modules will have to happen in mutliple
phases. The go-getter interface requires that the detector return the
final URL, while we won't know that until we verify which version we
need. This leaves the regisry sources broken, to be re-integrated in a
following commit.
2017-10-27 09:08:15 -04:00
James Bardin ee36cf28e0 a series of test commits
wire up HTTP so we can test the mock discovery service

test lookupModuleVersions

Add a versions endpoint to the mock registry, and use that to verify the
lookupModuleVersions behavior.

lookupModuleVersions takes a Disco as the argument so a custom Transport
can be injected, and uses that transport for its own client if it set.

test looking up modules with default registry

Add registry.terrform.io to the hostname that the mock registry resolves
to localhost.

ACC test looking up module versions

Lookup a basic module for the available version in the default registry.
2017-10-27 09:08:15 -04:00
James Bardin ae2903b810 missing current module source in versionedPathKey
The reason versionedPathKey didn't solve the previous test case was a
missing source string for the current module.
2017-10-27 09:08:15 -04:00
James Bardin c01dfa7757 failing test for changing intermediate modules
This test highlights how changing an intermediate source path prevents
reloading of submodules. While this is somewhat of an edge case now, it
becomes quite common in the cacse where module versions are updated.
2017-10-27 09:08:15 -04:00
James Bardin a5c86aeff6 Use the new regsrc and response packages
Adds basic detector for registry module source strings. While this isn't
a thorough validation, this will eliminate anything that is definitely
not a registry module, and split out our host and module id strings.

lookupModuleVersions interrogates the registry for the available
versions of a particular module and the tree of dependencies.
2017-10-27 09:08:15 -04:00
James Bardin 0bc279557e add more metadata to the Tree and the storage path
Submodules were located by using their module path as the storage key.
Now that modules may have versions, a submodule needs to know how to
locate the corect source depending on the versions of its ancestors in
the tree.

Add a version field to each Tree, and a pointer back to the parent Tree
to step back through the ancestors. The new versionedPathKey method uses
this information to build a unique key for each module, dependent on the
ancestor versions.

Not only do stored modules need to know their version if it exists, but
any relative source needs to know all the ancestor versions in order to
resolve correctly.
2017-10-27 09:08:15 -04:00
James Bardin 1f44fd8eb2 move getStorage to moduleStorage too
Add a test to ensure we don't store different modules with the same
relative path in the same location.
2017-10-27 09:08:15 -04:00
James Bardin 3d163917c7 move the getter.Storage into our moduleStorge type
Wrap the Storage interfaces together.
2017-10-27 09:08:15 -04:00
James Bardin 7416e7a6e0 update the module manifest implementation
The getter.Storage abstraction is proving entirely inadequate here, but
we can't replace it wholesale at the moment.

The Tree loader needs to know the location of the manifest before it can
start loading any modules. Since the version will have to be part of the
hashed storage key, there is no way to know what version of each module
are stored. The storageDir function will extract the StorageDir field
from the underlying FolderStorage instance for the tree to locate the
manifest.
2017-10-27 09:08:15 -04:00
James Bardin 283d88551a Add a new module storage manifest
To add registry support, a workaround in the local module storage was
added to record the subdirectory containing the module source from
within the archive file. Here we replace that temporary implementation
with the full manifest needed to record the necessary module metadata
for module loading.

In order to support versioned modules, the actual stored version needs
to be recorded. This can't be derived from the configuration, because
the configuration only contains the constraints, and at load time we need
to be able to enumerate the stored modules and all versions in order to
resolve them.

While the local storage key will be derived from the source and version,
that information is lost once it's hashed. While the entire storage
layer could be replaced to encode the needed data in the path itself,
this provides a minimal change to work with the existing storage code.
2017-10-27 09:08:15 -04:00
James Bardin 0962792257 surface registry errors to the user
Now that we can enforce local modules being relative or absolute paths,
we can be assured that any module source matching a registry pattern
must be found in the registry. This allows us to surface more useful
errors to the user, rather than simply stating that a source string
isn't valid.
2017-10-27 09:08:15 -04:00
James Bardin 40c36b48dc missing test-fixture files 2017-10-27 09:08:15 -04:00
James Bardin 8933ef8f70 make //subdir notation work with registry modules
Append the original source subdir to the discovered subdir when locating
the files in the module tree.
2017-10-27 09:08:15 -04:00
James Bardin a359c4ee2f check the registry detector before local files
Breaking change for 0.11.

Local files were checked first to avoid the possibility of breaking a
module with a local source that looked like a registry ID. Now we can
enfore that any source iwth the pattern "namespace/identifier/provider"
must be a registry module.
2017-10-27 09:08:15 -04:00
James Bardin 6088270c66 skip lookup for paths starting with ../
Prevent an extra registry lookup for sources starting with ../
2017-10-27 09:08:15 -04:00
James Bardin b0163e69d1 update comments 2017-10-27 09:08:15 -04:00
James Bardin 4ab70ba09d rename confusing test module name 2017-10-27 09:08:15 -04:00
James Bardin fe0957deb7 hcl2 maps now work 2017-10-27 09:08:15 -04:00
James Bardin 2e505083cd change ProviderConfig.Scope to Path
Though it's intended for "interpolation scope", Path is generally used
for this elsewhere.
2017-10-27 09:08:15 -04:00
James Bardin cb0e37a870 implement provider inheritence during loading
This implements provider inheritance during config loading, rather than
during graph evaluation. At this point it's much simpler to find the
desired configuration, and once all providers are declared, all the
inheritance code in the graph can be removed.

The inheritance is dome by simply copying the RawConfig from the parent
ProviderConfig into the module. Since this happens before any
evaluation, we record the original interpolation scope in the
ProviderConfig so that it can be properly resolved later on.
2017-10-27 09:08:15 -04:00
James Bardin 29e5a355b9 add "version" and "provider" to module config
Add the Version and Providers fields to the module config.

Add ProviderConfig.Scope, which will be used to record the original
path of a ProviderConfig for interpolation.
2017-10-27 09:08:15 -04:00
Radek Simko 7fceccfbf7
helper/schema: Loosen validation for 'id' field 2017-10-26 09:37:38 +01:00
Lyle Franklin 43dcaa1a00 config: Don't panic if config directory contains only overrides 2017-10-23 15:46:15 -07:00
Martin Atkins ccb328cc1f config: source code ranges for InterpolatedVariable
Having a reference to the originating source range will allow us to
generate decent error messages if certain references can't be resolved
at interpolation time.

This is not yet populated or used. It will never be populated nor used by
the current HCL/HIL-based interpolation path, but will be used by the
experimental HCL2-based interpolation path to give it the necessary info
to produce diagnostics.
2017-10-17 07:20:17 -07:00
Martin Atkins 71e989ba3e config/hcl2shim: make some of the HCL2 shim functions public
The value-conversion machinery is also needed in the main "terraform"
package to help us populate our HCL2 evaluation scope, so a subset of the
shim functions move here into a new package where they can be public.

Some of them remain private within the config package since they depend
on some other symbols in the config package, and they are not needed
by outside callers anyway.
2017-10-16 17:54:02 -07:00
Martin Atkins 22fb82963c config: when copying a HCL2 RawConfig, don't corrupt it
Previously we were demoting HCL2 RawConfigs into empty old-school
RawConfigs on copy.
2017-10-16 17:52:23 -07:00
Gauthier Wallet ed9ba576e3 core: New interpolation function "chunklist"
This turns a list into a list of lists with each element (apart from possibly the last) being the given length.
2017-10-10 11:56:13 -07:00
Stefan Staudenmeyer 14792035cf Aliasing the module to omit errors thrown at build time
Signed-off-by: Stefan Staudenmeyer <stefan.staudenmeyer@instana.com>
2017-10-09 16:44:58 +02:00
Martin Atkins fc20f419dd config and command: use errwrap to propagate config load errors
Previously we were using fmt.Sprintf and thus forcing the stringification
of the wrapped error.

Using errwrap allows us to unpack the original error at the top of the
stack, which is useful when the wrapped error is really a hcl.Diagnostics
containing potentially-multiple errors and possibly warnings.
2017-10-06 11:46:07 -07:00
Martin Atkins b91bd62747 config/configschema: Sensitive flag for attributes
We don't currently have any need for this information, but we're
propagating it out of helper/schema here pre-emptively so that once we
later have a use for it we will not need to rebuild the providers to gain
access to it.

The long-term expected use-case for this is to have Terraform Core use
static analysis techniques to trace the path of sensitive data through
interpolations so that intermediate results can be flagged as sensitive
too, but we have a lot more work to do before such a thing would actually
be possible.
2017-10-04 16:35:11 -07:00
Martin Atkins 0ef985cada config/configschema: Block.ImpliedType delegates to zcldec
zcldec now has its own function for computing the implied type for a spec,
so we can use that instead of our own logic.

The zcldec logic is more general since its spec model is more general than
our schema model here, but it produces the same results for the subset
of specifications that our DecoderSpec method produces.
2017-10-04 16:35:11 -07:00
Martin Atkins 2b622fe31a config/configschema: Block.DecoderSpec
This returns a decoding specification that can be used with the hcldec
package to decode a body into a cty.Value of an object type.
2017-10-04 16:35:11 -07:00
Martin Atkins d0d829848a config/configschema: Block.ImpliedType method
This returns a cty.Type that the caller can expect to recieve when
decoding a value using the (not yet implemented) decoder specification
for a given schema.
2017-10-04 16:35:11 -07:00
Martin Atkins f117906bdb config/configschema: InternalValidate for blocks
This checks that a schema complies with the documented constraints on
which values are valid. It is primarily intended for use in tests.
2017-10-04 16:35:11 -07:00
Martin Atkins d712a04c32 config/configschema: MinItems and MaxItems for nested blocks 2017-10-04 16:35:11 -07:00
Martin Atkins dadec5210b config/configschema: stub out initial API 2017-10-04 16:35:11 -07:00
Martin Atkins b851fa71c9 config: Make HIL-based functions available to HCL2 via a shim
Terraform has a _lot_ of functions written against HIL's function API, and
we're not ready to rewrite them all yet, so instead we shim the HIL
function API to conform to the HCL2 (really: cty) function API and thus
allow most of our existing functions to work as expected when called from
HCL2-based config files.

Not all of the functions can be fully shimmed in this way due to depending
on HIL implementation details that we can't mimic through the HCL2 API.
We don't attempt to address that yet, and instead just let them fail when
called. We will eventually address this by using first-class HCL2
functions for these few cases, thus avoiding the HIL API altogether where
we need to. (The methodology for that is already illustrated here in the
provision of jsonencode and jsondecode functions that are HCL2-native.)
2017-10-03 17:47:01 -07:00
Martin Atkins 34e9de605c config: tests for HIL-to-HCL2 value shimming 2017-10-03 17:47:01 -07:00
Martin Atkins bbf9725134 config: Validate resource "count" for HCL2-specified resources
This early validation uses interpolation of a placeholder value to achieve
some "best effort" validation of the validity of the count attribute.
Since HCL2-specified resources can't be interpolated using the main
interpolator, here we branch and use the HCL2 API to do a
largely-equivalent (though slightly less accurate) check.

In the long run we don't really need this extra check at all, since the
validation walk does a more accurate version of the same thing. However,
we're preserving this for now in the interests of minimizing the amount
of change for the main codepath during our experiment.
2017-10-03 17:47:01 -07:00
Martin Atkins 71e68f06c4 config: allow tests to force using the HCL2 loader
Currently the default for tests is to use the old HCL loader, but we need
to be able to test aspects of the new loader as we work through the
experimental phase. This new function testConfigHCL2 is the same as
testConfig except that it forces the use of HCL2 even if the opt-in
comment isn't present, thus allowing us to implement tests that ensure
that the exact same file works in both the old and new cases.

Once the HCL2 loader becomes the default this function will be removed
and callers will start calling into the normal testConfig function.
2017-10-03 17:47:01 -07:00
Martin Atkins d91327eaa0 config: allow HCL2 experiment opt-in (build-time flag to enable)
Use the new HCL2 config loader when the opt-in comment #terraform:hcl2 is
present in a .tf file.

For now this is disabled for "normal" builds and enabled only if
explicitly configured via a linker flag during build. This is because it's
not yet in a good state to be released: the HCL2 loader produces RawConfig
objects that the validator and interpolator can't yet deal with, and so
using HCL2 for anything non-trivial currently causes Terraform to crash
in real use.
2017-10-03 17:47:01 -07:00
Martin Atkins b0215fcd0f config: HCL2 config loader
This loader uses the HCL2 parser and decoder to process a config file,
and then transforms the result into the same shape as would be produced
by the HCL config loader.

To avoid making changes to the existing config structures (which are
depended on across much of the codebase) we first decode into a set of
HCL2-tailored structs and then process them into the public-facing structs
that a loader is expected to return. This is a compromise to keep the
config package API broadly unchanged for now. Once we're ready to remove
the old HCL loader (which implies that we're ready to support HCL2
natively elsewhere in the codebase) we will be able to simplify this
quite considerably.

Due to some mismatches of abstraction between HCL/HIL and HCL2, some
shimming is required to get the required result.
2017-10-03 17:47:01 -07:00
Martin Atkins edbbe41b44 config: allow a HCL2 body to piggy-back on a RawConfig
At this time we're not ready to refactor the various uses of RawConfig
in Terraform core, so we'll smuggle a HCL2 body within a degenerate
RawConfig object that we can then recognize and unpack once this object
is returned to us in an interpolation call.
2017-10-03 17:47:01 -07:00