Catch up the CHANGELOG after the v0.12 development branch merge

Due to the unusual way v0.12 was developed (because of the large amount
of cross-cutting refactoring) we have not built out an incremental
changelog along the way as we normally would.

This is an initial catchup of the changelog for the current tree at the
time of writing, with the intent that we will maintain this in the usual
way moving forward as we head towards the v0.12-alpha1 release.

CHANGELOG.md

@@ -1,13 +1,87 @@
-## 0.12.0 (Unreleased)
+## 0.12.0-alpha1 (Unreleased)
+
+The goal of this release is to give users an early preview of the new language features, and to collect feedback primarily about bugs and usability issues related to the language itself, while the Terraform team addresses the remaining problems. There will be at least one beta and at least one release candidate before final, which should give a more complete impression of how the final v0.12.0 release will behave.
 
 INCOMPATIBILITIES AND NOTES:
 
+The following list contains the most important incompatibillities and notes relative to v0.11.8, but may be incomplete. This alpha release is offered for experimentation purposes only and should not be used to manage real infrastructure. A more complete upgrade guide will be prepared in time for the final v0.12.0 release.
+
+* This release includes a revamped implementation of the configuration language that aims to address a wide array of feedback and known issues with the configuration language handling in prior versions. In order to resolve some ambiguities in the language, the new parser is stricter in some ways about following what was previously just idiomatic usage, and so some unusual constructs will need to be adjusted to be accepted by the new parser.
+
+  The v0.12.0 final release will include a more complete language upgrade guide and a tool that can recognize and automatically upgrade common patterns for the new parser and new idiomatic forms.
+
+* This release introduces new wire protocols for provider and provisioner plugins and a new automatic installation method for provider plugins. At the time of release there are no official plugin releases compatible with these new protocols and so automatic provider installation with `terraform init` is not functional. Instead, the v0.12.0-alpha1 distribution archives contain bundled experimental provider builds for use with the alpha.
+
+* This release introduces new file formats for persisted Terraform state (both on local disk and in remote backends) and for saved plan files. Third-party tools that attempt to parse these files will need to be updated to work with the formats generated by v0.12 releases. Prior to v0.12.0 we will add a new command to obtain a JSON representation of a saved plan intended for outside consumption, but this command is not yet present in v0.12.0-alpha1.
+
 * `terraform validate` now has a smaller scope than before, focusing only on configuration syntax and value type checking. This makes it safe to run e.g. on save in a text editor.
 
+NEW FEATURES:
+
+The overall theme for the v0.12 release is configuration language fixes and improvements. The fixes and improvements are too numerous to list out exhaustively, but the list below covers some highlights:
+
+* **First-class expressions:** Prior to v0.12, expressions could be used only via string interpolation, like `"${var.foo}"`. Expressions are now fully integrated into the language, allowing them to be used directly as argument values, like `ami = var.ami`.
+
+* **`for` expressions:** This new expression construct allows the construction of a list or map by transforming and filtering elements from another list or map. For more information, refer to [the _`for` expressions_ documentation](./website/docs/configuration/expressions.html.md#for-expressions).
+
+* **Dynamic configuration blocks:** For nested configuration blocks accepted as part of a resource configuration, it is now possible to dynamically generate zero or more blocks corresponding to items in a list or map using the special new `dynamic` block construct. This is the official replacement for the common (but buggy) unofficial workaround of treating a block type name as if it were an attribute expecting a list of maps value, which worked occasionally before as a result of some unintended coincidences in the implementation.
+
+* **Generalised "splat" operator:** The `aws_instance.foo.*.id` syntax was previously a special case only for resources with `count` set. It is now an operator within the expression language that can be applied to any list value. There is also an optional new splat variant that allows both index and attribute access operations on each item in the list. For more information, refer to [the _Splat Expressions_ documentation](./website/docs/configuration/expressions.html.md#splat-expressions).
+
+* **Nullable argument values:** It is now possible to use a conditional expression like `var.foo != "" ? var.foo : null` to conditionally leave an argument value unset, whereas before Terraform required the configuration author to provide a specific default value in this case. Assigning `null` to an argument is equivalent to omitting that argument entirely.
+
+* **Rich types in module inputs variables and output values:** Terraform v0.7 added support for returning flat lists and maps of strings, but this is now generalized to allow returning arbitrary nested data structures with mixed types. Module authors can specify a precise expected type for each input variable to allow early validation of caller values.
+
+* **Resource and module object values:** An entire resource or module can now be treated as an object value within expressions, including passing them through input variables and output values to other modules, using an attribute-less reference syntax, like `aws_instance.foo`.
+
+* **Extended template syntax:** The simple interpolation syntax from prior versions is extended to become a simple template language, with support for conditional interpolations and repeated interpolations through iteration. For more information, see [the _String Templates_ documentation](./website/docs/configuration/expressions.html.md#string-templates).
+
+* **`jsondecode` and `csvdecode` interpolation functions:** Due to the richer type system in the new configuration language implementation, we can now offer functions for decoding serialization formats. `jsondecode` is the opposite of `jsonencode`, while `csvdecode` provides a way to load in lists of maps from a compact tabular representation.
+
 IMPROVEMENTS:
 
 * `terraform validate` now accepts an argument `-json` which produces machine-readable output. Please refer to the documentation for this command for details on the format and some caveats that consumers must consider when using this interface. [GH-17539]
 
+* The JSON-based variant of the Terraform language now has a more tightly-specified and reliable mapping to the native syntax variant. In prior versions, certain Terraform configuration features did not function as expected or were not usable via the JSON-based forms. For more information, see [the _JSON Configuration Syntax_ documentation](./website/docs/configuration/syntax-json.html.md).
+
+BUG FIXES:
+
+* The conditional operator `... ? ... : ...` now works with result values of any type and only returns evaluation errors for the chosen result expression, as those familiar with this operator in other languages might expect.
+
+KNOWN ISSUES:
+
+Since v0.12.0-alpha1 is an experimental build, this list is certainly incomplete. Please let us know via GitHub issues if you run into a problem not covered here!
+
+* As noted above, the alpha1 release is bundled with its own special builds of a subset of providers because there are not yet any official upstream releases of providers that are compatible with the new v0.12 provider plugin protocol. Automatic installation of providers with `terraform init` is therefore not functional at the time of release of alpha1.
+
+  Provider developers may wish to try building their plugins against the v0.12-alpha1 tag of Terraform Core to use them with this build. We cannot yet promise that all providers will be buildable in this way and that they will work flawlessly after building. Official releases of all HashiCorp-hosted providers compatible with v0.12 will follow at some point before v0.12.0 final.
+
+* There are several error messages in Terraform Core that claim that a problem is caused by a bug in the provider and ask for an issue to be filed against that provider's repository. For this alpha release, we ask that users disregard this advice and report such problems instead within the Terraform Core repository, since they are more likely to be problems with the new protocol version bridge code that is included in the plugin SDK.
+
+* Some secondary Terraform CLI subcommands are not yet updated for this release and will return errors or produce partial results. Please focus most testing and experimentation with this release on the core workflow commands `terraform init`, `terraform validate`, `terraform plan`,  `terraform apply`, and `terraform destroy`.
+
+* In v0.9.6 we removed the strange requirement to wrap an expression returning a list in another layer of list which had been inherited from earlier versions of Terraform that lacked first-class list support, and deprecated that usage. This was most commonly seen with "splat" expressions, like this:
+
+  ```hcl
+    vpc_security_group_ids = ["${aws_security_group.example.*.id}"]
+  ```
+
+  The backward compatibility shim for this form has been intentionally removed in v0.12 so that the Terraform language can properly support lists of lists, but as of alpha1 the error message produced in this case is an unhelpful prompt that "a list of string is required". Before v0.12.0 final we hope to introduce a better error message for this situation that will suggest removing the redundant list brackets. This older usage will also be fixed automatically by the planned automatic configuration upgrade tool.
+
+* If a `variable` declaration block has a type constraint that is a list, set or map of an object type and the caller uses an incompatible object type in the calling `module` block then the error message is sometimes imprecise and confusing, stating that an object is required when it instead means to say that the object does not have the correct attributes.
+
+* Although Terraform will try to infer a type constraint for a variable that only has `default` set and leaves `type` unset, it currently fails to infer types for some deeply-nested structures. Only flat lists and maps of strings were explicitly supported in prior releases, but some implementation details allowed deeper structures to work as defaults in some cases. If Terraform cannot automatically infer the expected type from the default then it will produce an error; in many cases this can be worked around by explicitly specifying an exact `type` argument using [the new type constraint syntax](https://github.com/hashicorp/terraform/blob/master/website/docs/configuration/variables.html.md#type-constraints).
+
+* In prior versions most of the built-in functions that accepted maps and lists as arguments would require either that the elements all be strings or that the elements all have the same type. Although our new type system allows for more flexibility, not all of the function implementations have been updated so far, and so some functions may still return errors when given collections with inconsistent element types. We intend to update more of these before v0.12.0 final, though some may remain until later releases in the v0.12 line.
+
+* The special `timeouts` block accepted by some resource types to customize how long the provider will wait for operations to complete is not supported in this alpha release.
+
+* The `terraform_remote_state` data source built in to this release has been updated for the new state file format but has not yet been extensively tested. It may not yet return correct results in all cases, and may have other problems such as errors or crashes. The new implementation of this data source will be finalized in a later prerelease version.
+
+* The `template` provider plugin bundled in the release archive is still using the old interpolation language to evaluate template strings in `template_file`, so it does not yet support the new conditional and iteration syntaxes.
+
+* The experimental `remote` backend that was introduced in Terraform v0.11.8 is not included in this release. Since alpha1 is an experimental early release, we do not recommend using it with any "real" infrastructure, and as such we recommend using the default `local` backend only.
+
 ## 0.11.9 (Unreleased)
 
 IMPROVEMENTS: