Commit Graph

23347 Commits

Author SHA1 Message Date
Martin Atkins a4c98d3365 govendor fetch github.com/google.golang.org/grpc/...@v1.14.0 2018-10-16 19:11:09 -07:00
James Bardin 0b76c42ad2 replace provider and provisioner types in tests
The field assignments and methods are incorrect still, but the type
definitions and returns are now updated.
2018-10-16 19:11:09 -07:00
Martin Atkins 549544f201 configschema: Handle nested blocks containing dynamic-typed attributes
We need to make the collection itself be a tuple or object rather than
list or map in this case, since otherwise all of the elements of the
collection are constrained to be of the same type and that isn't the
intent of a provider indicating that it accepts any type.
2018-10-16 19:11:09 -07:00
Martin Atkins 4c78539c2b plans/objchange: AssertObjectSuperset function
This function's goal is to ensure that the "final" plan value produced
by a provider during the apply step is always consistent with the known
parts of the planned value produced during the plan step.

Any error produced here indicates a bug in the provider.
2018-10-16 19:11:09 -07:00
Martin Atkins 0317da9911 plans/objchange: logic for merging prior state with config
This produces a "proposed new state", which already has prior computed
values propagated into it (since that behavior is standard for all
resource types) but could be customized further by the provider to make
the "_planned_ new state".

In the process of implementing this it became clear that our configschema
DecoderSpec behavior is incorrect, since it's producing list values for
NestingList and map values for NestingMap. While that seems like it should
be right, we should actually be using tuple and object types respectively
to allow each block to have a different runtime type in situations where
an attribute is given the type cty.DynamicPseudoType. That's not fixed
here, and so without a further fix list and map blocks will panic here.
The DecoderSpec implementation will be fixed in a subsequent commit.
2018-10-16 19:11:09 -07:00
Martin Atkins dfe0988f10 plans: Record private data bytes as part of resource change
This allows a provider to retain arbitrary extra data in the plan and
make use of it during apply. The contents are not used by Terraform and
never shown to the user.
2018-10-16 19:11:09 -07:00
Martin Atkins fb57801dfe states: object Private data is []byte, not cty.Value
We're going to allow the provider to encode whatever it wants in here, so
a provider can use whatever is most convenient for its implementation
language and to avoid some of the bugs we saw with the prior model where
the forced round-trip through JSON and back into interface{} would cause
some loss of fidelity, leading to bugs.
2018-10-16 19:11:09 -07:00
James Bardin 08a8834882 ResourceProvisioner to provisioners.Interface
Replacing the core types, without touching tests yet.
2018-10-16 19:11:09 -07:00
James Bardin ad7eee7ba4 update Refresh
Use the new ReadResource provider method
2018-10-16 19:11:09 -07:00
James Bardin 16df9c37cf first step in core provider type replacement
Chaange ResourceProvider to providers.Interface starting from the
context, and fix all type errors.

This only replaced some of method calls directly applicable to the
providers themselves. The resource methods will follow.
2018-10-16 19:11:09 -07:00
Martin Atkins c45ebbc493 govendor fetch github.com/golang/protobuf/... 2018-10-16 19:11:09 -07:00
Martin Atkins 5e0b8b9721 govendor fetch google.golang.org/grpc/... 2018-10-16 19:11:09 -07:00
Martin Atkins 249fb9f212 govendor fetch github.com/coreos/etcd/... 2018-10-16 19:11:09 -07:00
Martin Atkins 3ea7e145e1 govendor fetch github.com/hashicorp/hcl2/...
This is a general catchup of some developments in the HCL2 codebase, but
in particular includes:

- Recording expression and evalcontext as part of diagnostics, so that
  variable value information can be included alongside diagnostic
  snippets.

- hcldec supports decoding blocks into tuple and object values as well as
  list and map values, which then allows cty.DynamicPseudoType nested
  attributes to work properly.
2018-10-16 19:11:09 -07:00
Martin Atkins de6054a580 govendor fetch golang.org/x/sys/unix/...
This new version defines some symbols that are required by the grpc
library.
2018-10-16 19:11:09 -07:00
Martin Atkins 594a821ab3 providers: Resolver and Factory types
These new types are adaptations of terraform.ProviderResolver and
terraform.ResourceProviderFactory respectively. In a later commit we will
update users of the old types to use these new types, possibly in a
"gradual repair" sort of fashion by initially using type aliases, though
that won't 100% solve the problem because the types themselves have
changed in this move to return providers.Interface instead of
terraform.ResourceProvider.
2018-10-16 19:11:09 -07:00
James Bardin bddfd0412c add new provider and provisioner mocks
MockProvider and MockProvisioner implement the new plugin interfaces,
and are built following the patterns used by the legacy
MockResourceProvider and MockResourceProvisioner
2018-10-16 19:11:09 -07:00
James Bardin faa46d4727 add Close method to the terraform provider
Close is now part of the plugin interfaces, so needs to be implemented
by the terraform provider.
2018-10-16 19:11:09 -07:00
James Bardin c39e9e107f add Close methods to the plugin interfaces
Close was previously an optional interface, because it's only applicable
on the client side but the types were shared accross all packages. Since
all plugins will now implement Close, it no longer needs to be optional.
2018-10-16 19:11:09 -07:00
James Bardin c07ce1cd4b move conversion functions into separate package
Managing which function need to be shared between the terraform plugin
and the helper plugin without creating cycles was becoming difficult.
Move all functions related to converting between terraform and proto
type into plugin/convert.
2018-10-16 19:11:09 -07:00
James Bardin 88bfbeb9c5 schema version changed type 2018-10-16 19:11:09 -07:00
James Bardin eeb606425d update go-plugin with proto negotiation 2018-10-16 19:11:09 -07:00
Martin Atkins a3403f2766 terraform: Ugly huge change to weave in new State and Plan types
Due to how often the state and plan types are referenced throughout
Terraform, there isn't a great way to switch them out gradually. As a
consequence, this huge commit gets us from the old world to a _compilable_
new world, but still has a large number of known test failures due to
key functionality being stubbed out.

The stubs here are for anything that interacts with providers, since we
now need to do the follow-up work to similarly replace the old
terraform.ResourceProvider interface with its replacement in the new
"providers" package. That work, along with work to fix the remaining
failing tests, will follow in subsequent commits.

The aim here was to replace all references to terraform.State and its
downstream types with states.State, terraform.Plan with plans.Plan,
state.State with statemgr.State, and switch to the new implementations of
the state and plan file formats. However, due to the number of times those
types are used, this also ended up affecting numerous other parts of core
such as terraform.Hook, the backend.Backend interface, and most of the CLI
commands.

Just as with 5861dbf3fc49b19587a31816eb06f511ab861bb4 before, I apologize
in advance to the person who inevitably just found this huge commit while
spelunking through the commit history.
2018-10-16 19:11:09 -07:00
Martin Atkins cf6892275a states: Port stringer implementation from terraform.State
Our existing core tests make extensive use of the string representation
of a state for comparison purposes, because they were written before we
began making use of helper packages like "cmp".

To avoid the need to rewrite all of those tests and potentially break
them, we will instead port that particular rendering as closely as
possible but mark it with a comment sternly warning not to use it for
anything new.

We don't want to use this moving forward for a number of reasons, but
most notably:

 - printing out whole before and after state representations makes it
   hard to find a subtle difference in outcome when a test fails, while
   "cmp" can provide us with a real diff.

 - this string serialization is constrained by the capabilities of
   Terraform prior to our new state models, and so it does not
   comprehensively represent all possibilities in the new world.

 - it will probably behave oddly/poorly when given states containing
   features that arrived after it was written, even though I made a
   best effort here to make it do something reasonable in situations
   I thought about.
2018-10-16 18:58:49 -07:00
Martin Atkins 424afe0ace states: separate types for encoded and decoded state objects
The types here were originally written to allow us to defer decoding of
object values until schemas are available, but it turns out that this was
forcing us to defer decoding longer than necessary and potentially decode
the same value multiple times.

To avoid this, we create pairs of types to represent the encoded and
decoded versions and methods for moving between them. These types are
identical to one another apart from how the dynamic values are
represented.
2018-10-16 18:58:49 -07:00
Martin Atkins b7db32b819 plans: separate types for encoded and decoded changes
The types here were originally written to allow us to defer decoding of
object values until schemas are available, but it turns out that this was
forcing us to defer decoding longer than necessary and potentially decode
the same value multiple times.

To avoid this, we create pairs of types to represent the encoded and
decoded versions and methods for moving between them. These types are
identical to one another apart from how the dynamic values are
represented.
2018-10-16 18:58:49 -07:00
Martin Atkins 3bb731e2d6 statemgr: Helper funcs RefreshAndRead and WriteAndPersist
In practice these pairs of functions are often used together when working
with a "full" statemgr, so these helper wrappers allow us to do that more
conveniently.

This also introduces a new interface statemgr.Storage, which represents
a state manager that has all of the storage capabilities but does not
necessarily support locking. In practice callers will usually just use
statemgr.Full, but these more-specific interfaces allow us to reflect
in APIs which subset of the statemgr functionality each function depends
on.
2018-10-16 18:58:49 -07:00
James Bardin b403023841 serve the new version 5 grpc plugins
Use the new go-plugin version negotiation to server th appropriate
plugin type when the client requests protocol version 5.
2018-10-16 18:58:49 -07:00
James Bardin ff7d51a9b4 update tests to match new test func signature 2018-10-16 18:58:49 -07:00
James Bardin 7188d5cbfe add grpc shims to helper/plugin
Add shim functions to automatically wrap plugins in the new GRPC
implementations.
2018-10-16 18:58:49 -07:00
Kristin Laemmert 5e7956f440 registry/response: rename Collection to ProviderVersionCollection for clarity 2018-10-16 18:58:49 -07:00
James Bardin 44f52e47c5 vendor gomock 2018-10-16 18:58:49 -07:00
James Bardin c9d91dc64e remove old comment 2018-10-16 18:58:49 -07:00
Kristin Laemmert bdaf8290b4 registry/client: added a specific error if the registry client does not
support the requested service.
2018-10-16 18:58:49 -07:00
James Bardin 63dcdbe948 helper/plugin package for grpc servers
The new helper/plugin package contains the grpc servers for handling the
new plugin protocol

The GRPCProviderServer and GRPCProvisionerServer handle the grpc plugin
protocol, and convert the requests to the legacy schema.Provider and
schema.Provisioner methods.
2018-10-16 18:58:49 -07:00
James Bardin 1ec792cabc fix package provisioners package name 2018-10-16 18:58:49 -07:00
James Bardin f9fee83279 minor providers.Interface fixes
Add a missing TerraformVersion field, and fix a TypeName field.
2018-10-16 18:58:49 -07:00
James Bardin c1d4a63fae GRPCProvider and GRPCProvisioner clients
Here we add the GRPCProvisioner and GRPCProvider which implement the
core provisioners.Interface and providers.Interface, and translate
betweeen the core types and the grpc protocol.
2018-10-16 18:58:49 -07:00
James Bardin 81bd3b09d6 generate mocks for some of the grpc interfaces
These will allow easier testing of the grpc endpoints in isolation.
Mocks are generated for ProviderClient, ProvisionerClient,
Provisioner_ProvisionResourceClient, and
Provisioner_ProvisionResourceServer using `go generate`
2018-10-16 18:58:49 -07:00
James Bardin e29053432e grpc proto fixes
Add missing connection field for ProvisionResource.

Add diagnostics to the ProvisionResource stream.
2018-10-16 18:58:49 -07:00
Kristin Laemmert ae015e0f75 plugin/discovery: add function to filter list of versions by those which
support the requested platform.

If the newest version of a provider which matches the version
constraints does not support the requested platform, filter the list of
available versions by platform support and try again.
2018-10-16 18:58:49 -07:00
Kristin Laemmert f83d5866fe plugin/discovery: removing deprecated functions 2018-10-16 18:58:49 -07:00
Kristin Laemmert ce5e66e178 plugin/discovery provider installer: download providers from the registry
Terraform will query the public registry at
https://registry.terraform.io for providers, instead of
https://releases.hashicorp.com.
2018-10-16 18:56:50 -07:00
James Bardin b88410984b legacy provider needs to handle StateUpgraders
In order to not require state migrations to be supported in both
MigrateState and StateUpgraders, the legacy provider codepath needs to
handle the StateUpgraders transparently during Refresh.
2018-10-16 18:56:50 -07:00
James Bardin dcaf5aa262 add some of the shims used for the grpc provider
This adds some of the required shim functions to the schema package.
While this further bloats the already huge package, adding the helpers
here was significantly less disruptive than refactoring types into
separate packages to prevent import cycles.

The majority of tests here are directly adapted from existing schema
tests to provide as many known good values to the shims as possible.
2018-10-16 18:56:50 -07:00
Kristin Laemmert d1f27ce8c2 registry: renaming module-specific registry functions 2018-10-16 18:56:50 -07:00
Kristin Laemmert 082af84131 registry: adding provider functions to registry client 2018-10-16 18:53:51 -07:00
James Bardin 7d24936507 updates to teh StateUpgraders
Fix documentation.
Require StateUpgraders to be add in order, and test in validation.
2018-10-16 18:53:51 -07:00
James Bardin 0c33b26e04 StateUpgrade redux
It turns out that state upgrades need to be handled differently since
providers are going to be backwards compatible. This means that new
state upgrades may still be stored in the flatmap format when used wih
terraform 0.11. Because we can't account for the specific version which
could produce a legacy state, all future state upgrades need to record
the schema types for decoding.

Rather than defining a single Upgrade function for states, we now have a
list of functions, each of which handle upgrading a specific version to
the next. In practice this isn't much different from the way many
resources implement upgrades themselves, with a separate function for
each version dispatched from the MigrateState function. The only added
burden is the recording of the schema type, and we intend to supply
tools and helper function to prevent the need to copy the entire
existing schema in all cases.
2018-10-16 18:53:51 -07:00
James Bardin 9eef5e3f91 implement UpgradeState for schema.Resource
This is the provider-side UpgradeState implementation for a particular
resource. This new function will be called to upgrade a saved state with
an old schema version to the current schema.

UpgradeState also requires a record of the last schema and version that
could have been stored as a flatmapped state. If the stored state is in
the legacy flatmap format, this will allow the provider to properly
decode the flatmapped state into the expected structure for the new json
encoded state. If the stored state's version is below that of the
LegacySchema.Version value, it will first be processed by the legacy
MigrateState function.
2018-10-16 18:53:51 -07:00