Commit Graph

130 Commits

Author SHA1 Message Date
James Bardin 9bda82ccf1 add example for the reason behind versionedPathKey 2017-10-27 09:08:15 -04:00
James Bardin d6f75cc022 use URLs rather than strings is registry functions
Parse all the registry strings as urls, and compine with path.Join to
for better validation.
2017-10-27 09:08:15 -04:00
James Bardin 4a0de691f5 update test for error condition 2017-10-27 09:08:15 -04:00
James Bardin 6c808347d7 flag off registry ACC test
Was missing the TF_ACC check
2017-10-27 09:08:15 -04:00
James Bardin 1756b3d09f ProviderConfig needs an Inherited flag
If a provider configuration is inherited from another module, any
interpolations in that config won't have variables declared locally. Let
the config only be validated in it's original location.
2017-10-27 09:08:15 -04:00
James Bardin 0afd4a9097 Lookup registry module versions during Tree.Load.
Registry modules can't be handled directly by the getter.Storage
implementation, which doesn't know how to handle versions. First see if
we have a matching module stored that satisfies our constraints. If
not, and we're getting or updating, we can look it up in the registry.

This essentially takes the place of a "registry detector" for go-getter,
but required the intermediate step of resolving the version dependency.

This also starts breaking up the huge Tree.Load method into more
manageable parts. It was sorely needed, as indicated by the difficulty
encountered in this refactor. There's still a lot that can be done to
improve this, but at least there are now a few easier to read methods
when we come back to it.
2017-10-27 09:08:15 -04:00
James Bardin 0d10564a74 remove the registryDetector
The detection of registry modules will have to happen in mutliple
phases. The go-getter interface requires that the detector return the
final URL, while we won't know that until we verify which version we
need. This leaves the regisry sources broken, to be re-integrated in a
following commit.
2017-10-27 09:08:15 -04:00
James Bardin ee36cf28e0 a series of test commits
wire up HTTP so we can test the mock discovery service

test lookupModuleVersions

Add a versions endpoint to the mock registry, and use that to verify the
lookupModuleVersions behavior.

lookupModuleVersions takes a Disco as the argument so a custom Transport
can be injected, and uses that transport for its own client if it set.

test looking up modules with default registry

Add registry.terrform.io to the hostname that the mock registry resolves
to localhost.

ACC test looking up module versions

Lookup a basic module for the available version in the default registry.
2017-10-27 09:08:15 -04:00
James Bardin ae2903b810 missing current module source in versionedPathKey
The reason versionedPathKey didn't solve the previous test case was a
missing source string for the current module.
2017-10-27 09:08:15 -04:00
James Bardin c01dfa7757 failing test for changing intermediate modules
This test highlights how changing an intermediate source path prevents
reloading of submodules. While this is somewhat of an edge case now, it
becomes quite common in the cacse where module versions are updated.
2017-10-27 09:08:15 -04:00
James Bardin a5c86aeff6 Use the new regsrc and response packages
Adds basic detector for registry module source strings. While this isn't
a thorough validation, this will eliminate anything that is definitely
not a registry module, and split out our host and module id strings.

lookupModuleVersions interrogates the registry for the available
versions of a particular module and the tree of dependencies.
2017-10-27 09:08:15 -04:00
James Bardin 0bc279557e add more metadata to the Tree and the storage path
Submodules were located by using their module path as the storage key.
Now that modules may have versions, a submodule needs to know how to
locate the corect source depending on the versions of its ancestors in
the tree.

Add a version field to each Tree, and a pointer back to the parent Tree
to step back through the ancestors. The new versionedPathKey method uses
this information to build a unique key for each module, dependent on the
ancestor versions.

Not only do stored modules need to know their version if it exists, but
any relative source needs to know all the ancestor versions in order to
resolve correctly.
2017-10-27 09:08:15 -04:00
James Bardin 1f44fd8eb2 move getStorage to moduleStorage too
Add a test to ensure we don't store different modules with the same
relative path in the same location.
2017-10-27 09:08:15 -04:00
James Bardin 3d163917c7 move the getter.Storage into our moduleStorge type
Wrap the Storage interfaces together.
2017-10-27 09:08:15 -04:00
James Bardin 7416e7a6e0 update the module manifest implementation
The getter.Storage abstraction is proving entirely inadequate here, but
we can't replace it wholesale at the moment.

The Tree loader needs to know the location of the manifest before it can
start loading any modules. Since the version will have to be part of the
hashed storage key, there is no way to know what version of each module
are stored. The storageDir function will extract the StorageDir field
from the underlying FolderStorage instance for the tree to locate the
manifest.
2017-10-27 09:08:15 -04:00
James Bardin 283d88551a Add a new module storage manifest
To add registry support, a workaround in the local module storage was
added to record the subdirectory containing the module source from
within the archive file. Here we replace that temporary implementation
with the full manifest needed to record the necessary module metadata
for module loading.

In order to support versioned modules, the actual stored version needs
to be recorded. This can't be derived from the configuration, because
the configuration only contains the constraints, and at load time we need
to be able to enumerate the stored modules and all versions in order to
resolve them.

While the local storage key will be derived from the source and version,
that information is lost once it's hashed. While the entire storage
layer could be replaced to encode the needed data in the path itself,
this provides a minimal change to work with the existing storage code.
2017-10-27 09:08:15 -04:00
James Bardin 0962792257 surface registry errors to the user
Now that we can enforce local modules being relative or absolute paths,
we can be assured that any module source matching a registry pattern
must be found in the registry. This allows us to surface more useful
errors to the user, rather than simply stating that a source string
isn't valid.
2017-10-27 09:08:15 -04:00
James Bardin 40c36b48dc missing test-fixture files 2017-10-27 09:08:15 -04:00
James Bardin 8933ef8f70 make //subdir notation work with registry modules
Append the original source subdir to the discovered subdir when locating
the files in the module tree.
2017-10-27 09:08:15 -04:00
James Bardin a359c4ee2f check the registry detector before local files
Breaking change for 0.11.

Local files were checked first to avoid the possibility of breaking a
module with a local source that looked like a registry ID. Now we can
enfore that any source iwth the pattern "namespace/identifier/provider"
must be a registry module.
2017-10-27 09:08:15 -04:00
James Bardin 6088270c66 skip lookup for paths starting with ../
Prevent an extra registry lookup for sources starting with ../
2017-10-27 09:08:15 -04:00
James Bardin b0163e69d1 update comments 2017-10-27 09:08:15 -04:00
James Bardin 4ab70ba09d rename confusing test module name 2017-10-27 09:08:15 -04:00
James Bardin 2e505083cd change ProviderConfig.Scope to Path
Though it's intended for "interpolation scope", Path is generally used
for this elsewhere.
2017-10-27 09:08:15 -04:00
James Bardin cb0e37a870 implement provider inheritence during loading
This implements provider inheritance during config loading, rather than
during graph evaluation. At this point it's much simpler to find the
desired configuration, and once all providers are declared, all the
inheritance code in the graph can be removed.

The inheritance is dome by simply copying the RawConfig from the parent
ProviderConfig into the module. Since this happens before any
evaluation, we record the original interpolation scope in the
ProviderConfig so that it can be properly resolved later on.
2017-10-27 09:08:15 -04:00
James Bardin 29e5a355b9 add "version" and "provider" to module config
Add the Version and Providers fields to the module config.

Add ProviderConfig.Scope, which will be used to record the original
path of a ProviderConfig for interpolation.
2017-10-27 09:08:15 -04:00
Stefan Staudenmeyer 14792035cf Aliasing the module to omit errors thrown at build time
Signed-off-by: Stefan Staudenmeyer <stefan.staudenmeyer@instana.com>
2017-10-09 16:44:58 +02:00
James Bardin 1fbb2d611a remove extra slash from registry urls
A refactor introduced an extra `/` in the download url, which causes an
extra redirect during discovery.

Improve a registry test to verify that detection doesn't require the
registry after the modules have been fetched.
2017-09-28 14:45:02 -04:00
James Bardin 2b3e48aa70 fix test comments 2017-09-26 09:12:14 -04:00
James Bardin f6e89bb8c3 record the subdirectory in the FolderStorage
Module detection currently requires calling the registry to determine
the subdirectory. Since we're not directly accessing the subdirectory
through FolderStorage, and now handling it within terraform so modules can
reference sibling paths, we need to call out to the registry every
time we load a configuration to verify the subdirectory for the module,
which is returned during the Detect.

Record the subdirectories for each module in the top-level of the
FolderStorage path for retrieval during Tree.Load. This lets us bypass
Detection altogether, modules can be loaded without redetecting.
2017-09-22 22:03:38 -04:00
James Bardin 1b01f18920 handle go-getter subdirs in Tree.Load
In order to remain backward compatible with some modules, we need to
handle subdirs during Load. This means duplicating part of the go-getter
code path for subDir handling so we can resolve any subDirs and globs
internally, while keeping the entire remote directory structure within
the file storage.
2017-09-22 20:32:44 -04:00
James Bardin 59089ef461 add test loading an archive with `//*` subdir
The registry uses the `//*` subdir format to unpack github archives. Add
a test to cover this pattern.
2017-09-21 15:32:55 -05:00
James Bardin 38569c8508 add tests for get from tar subdir
Test that we can get a subdirectory from a tarball (or any other
"packed" source that we support).

The 'tar-subdir-to-parent' test highlights a regression where the
subdirectory module references a module in its parent directory. This
breaks the intended use ofr the subdirectory and the implementation in
go-getter. We need to fix this in terraform, and possible plan warnings
and deprecations for this type of source.
2017-09-21 13:12:33 -05:00
James Bardin 3d3992454e Add basic ACC test
This just does a lookup in the registry to verify that the download api
works.
2017-09-15 16:16:29 -04:00
James Bardin ba14cf9511 add test fixture for new getter discovery behavior 2017-09-15 16:16:29 -04:00
James Bardin f9fb2b4c9e move local module detection ahead of the registry
The getter.FileDetector was intended to be the final detector, only
converting a path to a file URL and returning a true in all cases. We
want to check for a local module before checking the registry so no
local modules that happen to match a registry module are broken.

Wrap the getter.FileDetector to check the module source's existence
before delegating the search to the registry.
2017-09-15 16:16:28 -04:00
James Bardin d216d19f21 change the FolderStorage key
updating the key will cause the FolderStorage hash to change forcing
modules to be re-fetched. This is required because any configurations
using the subDir notation will have the configuration in the wrong
directory.
2017-09-15 16:16:28 -04:00
James Bardin 6c20141c30 remove subdir handling from Tree.Load
Terraform was redundantly handling `//dir` notation which should be
handled by go-getter. Rather than allowing go-getter to unpack a subdir
as expected, the subdir was stripped off and accessed through the module
configuration.

This scheme will no longer works now that go-getter supports `*`
subdirectories
    (e.g. `//*` would be analogous to `tar --strip-components=1`).

Even though this allows Terraform to use go-getter's native unpacking,
detection is still done separately because Detect requires a `pwd` which
is dependent on the configuration directory and not known to the
global FolderStorage.
2017-09-15 16:16:28 -04:00
James Bardin a83ff57aea Add registry detector
Add a getter.Detector for detecting registry modules and looking up
the download location of the latest version. This is essentially a
temporary API until constraint solving is supported by the registry, as
then we'll have to supply the full set of known contraints to the
registry at once for resolution and we will fetch specific versions of
modules.
2017-09-15 16:16:28 -04:00
Martin Atkins a8a64c66c0 config/module: helper to visit all modules in a tree 2017-06-09 14:03:59 -07:00
Mitchell Hashimoto 68ee4e0480
config/module: don't panic when referencing undefined module
Fixes #12788

We would panic when referencing an output from an undefined module. The
panic above this is correct but in this case Load will not catch
interpolated variables that _reference_ an unloaded/undefined module.
Test included.
2017-03-16 20:14:20 -07:00
James Bardin 2a949093ed report all errors from module validation
It can be tedious fixing a new module with many errors when Terraform
only outputs the first random error it encounters.

Accumulate all errors from validation, and format them for the user.
2017-03-02 15:16:05 -05:00
John Murphy 628d46ac53 Fixed broken build for netbsd 2017-02-22 00:30:18 +08:00
Mitchell Hashimoto 7b342100d0
config: add "backend" loading to the Terraform section 2017-01-26 14:33:49 -08:00
Mitchell Hashimoto 095b7e7831
config/module: disallow root modules named "root"
Fixes #11038

This is a **short term fix**.

Terraform core doesn't currently handle root modules named "root" well
because the prefix `[]string{"root"}` has special meaning and Terraform
core [currently] can't disambiguate between the root module and a module
named "root" in the root module.

This PR introduces a short term fix by simply disallowing root modules
named "root". This shouldn't break any BC because since 0.8.0 this
didn't work at all in many broken ways (including crashes).

Longer term, this should be fixed by removing the special prefix at all
and having empty paths be root. I started down this path but the core
changes necessary are far too scary for a patch release. We can aim for
0.9.
2017-01-08 15:39:57 -08:00
Mitchell Hashimoto 0c30caec7d
config: smarter provider alias usage validation
Fixes #4789

This improves the validation that valid provider aliases are used.

Previously, we required that provider aliases be defined in every module
they're used. This isn't correct because the alias may be used in a
parent module and inherited.

This removes that validation and creates the validation that a provider
alias must be defined in the used module or _any parent_. This allows
inheritance to work properly.

We've always had this type of validation for aliases because we believe
its a good UX tradeoff: typo-ing an alias is really painful, so we
require declaration of alias usage. It may add a small burden to
declare, but since relatively few aliases are used, it improves the
scenario where a user fat-fingers an alias name.
2016-12-16 16:47:32 -08:00
Mitchell Hashimoto 55ef966b88
config/module: tree.Child on a nil tree works 2016-10-19 13:38:49 -07:00
Mitchell Hashimoto 098225dc0d
config/module: use the raw source as part of the key
This changes the key for the storage to be the _raw_ source from the
module, not the fully expanded source. Example: it'll be a relative path
instead of an absolute path.

This allows the ".terraform/modules" directory to be portable when
moving to other machines. This was a behavior that existed in <= 0.7.2
and was broken with #8398. This amends that and adds a test to verify.
2016-09-30 10:44:46 -07:00
Mitchell Hashimoto 92336281a5
config/module: use source as part of key 2016-08-22 14:07:28 -07:00
James Nugent 796ce7bdd7 build: Opt-in to building rather than opt-out 2016-07-29 18:40:31 -05:00