Go to file
Chris Marchesi b807505d55 core: New refresh graph building behaviour
Currently, the refresh graph uses the resources from state as a base,
with data sources then layered on. Config is not consulted for resources
and hence new resources that are added with count (or any new resource
from config, for that matter) do not get added to the graph during
refresh.

This is leading to issues with scale in and scale out when the same
value for count is used in both resources, and data sources that may
depend on that resource (and possibly vice versa). While the resources
exist in config and can be used, the fact that ConfigTransformer for
resources is missing means that they don't get added into the graph,
leading to "index out of range" errors and what not.

Further to that, if we add these new resources to the graph for scale
out, considerations need to be taken for scale in as well, which are not
being caught 100% by the current implementation of
NodeRefreshableDataResource. Scale-in resources should be treated as
orphans, which according to the instance-form NodeRefreshableResource
node, should be NodeDestroyableDataResource nodes, but this this logic
is currently not rolled into NodeRefreshableDataResource. This causes
issues on scale-in in the form of race-ish "index out of range" errors
again.

This commit updates the refresh graph so that StateTransformer is no
longer used as the base of the graph. Instead, we add resources from the
state and config in a hybrid fashion:

 * First off, resource nodes are added from config, but only if
   resources currently exist in state.  NodeRefreshableManagedResource
   is a new expandable resource node that will expand count and add
   orphans from state. Any count-expanded node that has config but no
   state is also transformed into a plannable resource, via a new
   ResourceRefreshPlannableTransformer.
 * The NodeRefreshableDataResource node type will now add count orphans
   as NodeDestroyableDataResource nodes. This achieves the same effect
   as if the data sources were added by StateTransformer, but ensures
   there are no races in the dependency chain, with the added benefit of
   directing these nodes straight to the proper
   NodeDestroyableDataResource node.
 * Finally, config orphans (nodes that don't exist in config anymore
   period) are then added, to complete the graph.

This should ensure as much as possible that there is a refresh graph
that best represents both the current state and config with updated
variables and counts.
2017-05-12 15:45:06 -07:00
.github documentation: Add OPC to CONTRIBUTING 2017-05-10 20:25:43 -04:00
backend Merge pull request #13941 from hashicorp/jbardin/sigint-message 2017-04-25 20:16:22 -04:00
builtin provider/vault: Prevent panic when no secret found (#14435) 2017-05-13 01:14:14 +03:00
command http provider and http request data source 2017-05-08 17:37:48 -07:00
communicator Add upload tests 2017-02-10 10:30:05 -05:00
config Add a logarithm function 2017-05-05 17:06:53 +10:00
contrib Shell completions for fish 2017-04-17 15:40:55 +00:00
dag dag: method for filtering a set on arbitrary criteria 2017-05-11 11:57:46 -07:00
digraph Fix TestWriteDot random order error 2014-07-29 10:26:50 -07:00
docs docs: Add Maintainer's etiquette 2017-05-04 15:27:25 +01:00
examples provider/azurerm: Add example of vm from user image (#14231) 2017-05-11 19:40:35 +03:00
flatmap correct spelling mistakes (#13979) 2017-04-27 02:10:04 +12:00
helper provider/google: Log HTTP requests and responses in DEBUG mode (#14281) 2017-05-10 21:16:43 +02:00
plugin plugin: bump protocol version 2017-03-01 14:42:33 -08:00
repl terraform: improve error messages to assist REPL 2016-11-13 23:17:04 -08:00
scripts scripts: update tests for generate plugins to pass new style 2017-01-26 15:10:03 -08:00
state state/remote/swift: Support Openstack request logging (#13583) 2017-04-15 17:11:28 +03:00
terraform core: New refresh graph building behaviour 2017-05-12 15:45:06 -07:00
test-fixtures terraformrc can contain env var references 2017-02-13 17:52:51 -08:00
vendor Update HIL vendor for conditional type checking fix 2017-05-12 15:23:12 -07:00
website Added new evaluation_delay parameter (#14433) 2017-05-12 16:12:21 +03:00
.gitignore gitignore should ignore test files that use .terraform/tfstate 2017-01-26 14:33:49 -08:00
.travis.yml provider/azurerm: Add example of a VNET w/ Two Subnets (#14115) 2017-05-04 10:57:54 +01:00
BUILDING.md Makefile/docs: Lock in 1.6 req, doc vendored deps 2016-02-24 16:13:49 -06:00
CHANGELOG.md Update CHANGELOG.md 2017-05-12 15:26:01 -07:00
LICENSE Adding license 2014-07-28 13:54:06 -04:00
Makefile build: Fix Travis vet issues (#14010) 2017-04-27 13:51:22 +12:00
README.md Fix README logo 2017-04-10 11:54:14 -04:00
Vagrantfile Update Vagrantfile to use go1.8.1 (#14008) 2017-04-27 08:01:59 +12:00
checkpoint.go fixing version numbers RCs should be labeled x.x.x-rcx 2015-02-07 16:56:56 +01:00
commands.go move force-unlock to plumbing 2017-04-01 17:09:20 -04:00
config.go terraformrc can contain env var references 2017-02-13 17:52:51 -08:00
config_test.go terraformrc can contain env var references 2017-02-13 17:52:51 -08:00
config_unix.go core: use !windows instead of a list of unixes 2015-12-30 17:37:24 -05:00
config_windows.go config looks in a plugin directory if it exists 2014-09-27 12:36:13 -07:00
help.go Remind future maintainers to update the docs when changing CLI usage 2016-11-24 09:22:18 -08:00
main.go main: synchronize writes to VT100-faker on Windows 2017-05-04 15:36:51 -07:00
main_test.go support nested subcommands with TF_CLI_ARGS 2017-02-13 15:18:50 -08:00
panic.go panic: Instruct the user to include terraform's version for bug reports. 2015-05-14 18:14:56 -04:00
signal_unix.go Forward SIGTERM and handle that as an interrupt 2016-12-08 12:20:25 -05:00
signal_windows.go Forward SIGTERM and handle that as an interrupt 2016-12-08 12:20:25 -05:00
synchronized_writers.go main: synchronize writes to VT100-faker on Windows 2017-05-04 15:36:51 -07:00
version.go Expose Terraform version internally & externally 2015-06-21 12:24:42 +01:00

README.md

Terraform

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

The key features of Terraform are:

  • Infrastructure as Code: Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.

  • Execution Plans: Terraform has a "planning" step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure.

  • Resource Graph: Terraform builds a graph of all your resources, and parallelizes the creation and modification of any non-dependent resources. Because of this, Terraform builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure.

  • Change Automation: Complex changesets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors.

For more information, see the introduction section of the Terraform website.

Getting Started & Documentation

All documentation is available on the Terraform website.

Developing Terraform

If you wish to work on Terraform itself or any of its built-in providers, you'll first need Go installed on your machine (version 1.8+ is required). Alternatively, you can use the Vagrantfile in the root of this repo to stand up a virtual machine with the appropriate dev tooling already set up for you.

For local dev first make sure Go is properly installed, including setting up a GOPATH. You will also need to add $GOPATH/bin to your $PATH.

Next, using Git, clone this repository into $GOPATH/src/github.com/hashicorp/terraform. All the necessary dependencies are either vendored or automatically installed, so you just need to type make. This will compile the code and then run the tests. If this exits with exit status 0, then everything is working!

$ cd "$GOPATH/src/github.com/hashicorp/terraform"
$ make

To compile a development version of Terraform and the built-in plugins, run make dev. This will build everything using gox and put Terraform binaries in the bin and $GOPATH/bin folders:

$ make dev
...
$ bin/terraform
...

If you're developing a specific package, you can run tests for just that package by specifying the TEST variable. For example below, onlyterraform package tests will be run.

$ make test TEST=./terraform
...

If you're working on a specific provider and only wish to rebuild that provider, you can use the plugin-dev target. For example, to build only the Azure provider:

$ make plugin-dev PLUGIN=provider-azure

If you're working on the core of Terraform, and only wish to rebuild that without rebuilding providers, you can use the core-dev target. It is important to note that some types of changes may require both core and providers to be rebuilt - for example work on the RPC interface. To build just the core of Terraform:

$ make core-dev

Dependencies

Terraform stores its dependencies under vendor/, which Go 1.6+ will automatically recognize and load. We use govendor to manage the vendored dependencies.

If you're developing Terraform, there are a few tasks you might need to perform.

Adding a dependency

If you're adding a dependency, you'll need to vendor it in the same Pull Request as the code that depends on it. You should do this in a separate commit from your code, as makes PR review easier and Git history simpler to read in the future.

To add a dependency:

Assuming your work is on a branch called my-feature-branch, the steps look like this:

  1. Add the new package to your GOPATH:

    go get github.com/hashicorp/my-project
    
  2. Add the new package to your vendor/ directory:

    govendor add github.com/hashicorp/my-project/package
    
  3. Review the changes in git and commit them.

Updating a dependency

To update a dependency:

  1. Fetch the dependency:

    govendor fetch github.com/hashicorp/my-project
    
  2. Review the changes in git and commit them.

Acceptance Tests

Terraform has a comprehensive acceptance test suite covering the built-in providers. Our Contributing Guide includes details about how and when to write and run acceptance tests in order to help contributions get accepted quickly.

Cross Compilation and Building for Distribution

If you wish to cross-compile Terraform for another architecture, you can set the XC_OS and XC_ARCH environment variables to values representing the target operating system and architecture before calling make. The output is placed in the pkg subdirectory tree both expanded in a directory representing the OS/architecture combination and as a ZIP archive.

For example, to compile 64-bit Linux binaries on Mac OS X, you can run:

$ XC_OS=linux XC_ARCH=amd64 make bin
...
$ file pkg/linux_amd64/terraform
terraform: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

XC_OS and XC_ARCH can be space separated lists representing different combinations of operating system and architecture. For example, to compile for both Linux and Mac OS X, targeting both 32- and 64-bit architectures, you can run:

$ XC_OS="linux darwin" XC_ARCH="386 amd64" make bin
...
$ tree ./pkg/ -P "terraform|*.zip"
./pkg/
├── darwin_386
│   └── terraform
├── darwin_386.zip
├── darwin_amd64
│   └── terraform
├── darwin_amd64.zip
├── linux_386
│   └── terraform
├── linux_386.zip
├── linux_amd64
│   └── terraform
└── linux_amd64.zip

4 directories, 8 files

Note: Cross-compilation uses gox, which requires toolchains to be built with versions of Go prior to 1.5. In order to successfully cross-compile with older versions of Go, you will need to run gox -build-toolchain before running the commands detailed above.

Docker

When using docker you don't need to have any of the Go development tools installed and you can clone terraform to any location on disk (doesn't have to be in your $GOPATH). This is useful for users who want to build master or a specific branch for testing without setting up a proper Go environment.

For example, run the following command to build terraform in a linux-based container for macOS.

docker run --rm -v $(pwd):/go/src/github.com/hashicorp/terraform -w /go/src/github.com/hashicorp/terraform -e XC_OS=darwin -e XC_ARCH=amd64 golang:latest bash -c "apt-get update && apt-get install -y zip && make bin"