terraform/terraform/node_resource_refresh.go

290 lines
9.0 KiB
Go
Raw Normal View History

package terraform
import (
"fmt"
"log"
"github.com/hashicorp/terraform/plans"
"github.com/hashicorp/terraform/providers"
"github.com/hashicorp/terraform/states"
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
"github.com/hashicorp/terraform/addrs"
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
"github.com/hashicorp/terraform/dag"
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
"github.com/hashicorp/terraform/tfdiags"
)
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
// NodeRefreshableManagedResource represents a resource that is expanabled into
// NodeRefreshableManagedResourceInstance. Resource count orphans are also added.
type NodeRefreshableManagedResource struct {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
*NodeAbstractResource
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
}
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
var (
_ GraphNodeSubPath = (*NodeRefreshableManagedResource)(nil)
_ GraphNodeDynamicExpandable = (*NodeRefreshableManagedResource)(nil)
_ GraphNodeReferenceable = (*NodeRefreshableManagedResource)(nil)
_ GraphNodeReferencer = (*NodeRefreshableManagedResource)(nil)
_ GraphNodeResource = (*NodeRefreshableManagedResource)(nil)
_ GraphNodeAttachResourceConfig = (*NodeRefreshableManagedResource)(nil)
)
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
// GraphNodeDynamicExpandable
func (n *NodeRefreshableManagedResource) DynamicExpand(ctx EvalContext) (*Graph, error) {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
var diags tfdiags.Diagnostics
count, countDiags := evaluateResourceCountExpression(n.Config.Count, ctx)
diags = diags.Append(countDiags)
if countDiags.HasErrors() {
return nil, diags.Err()
}
// Next we need to potentially rename an instance address in the state
// if we're transitioning whether "count" is set at all.
fixResourceCountSetTransition(ctx, n.ResourceAddr(), count != -1)
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
// Our graph transformers require access to the full state, so we'll
// temporarily lock it while we work on this.
state := ctx.State().Lock()
defer ctx.State().Unlock()
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
// The concrete resource factory we'll use
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
concreteResource := func(a *NodeAbstractResourceInstance) dag.Vertex {
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
// Add the config and state since we don't do that via transforms
a.Config = n.Config
a.ResolvedProvider = n.ResolvedProvider
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
return &NodeRefreshableManagedResourceInstance{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
NodeAbstractResourceInstance: a,
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
}
}
// Start creating the steps
steps := []GraphTransformer{
// Expand the count.
&ResourceCountTransformer{
Concrete: concreteResource,
Schema: n.Schema,
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
Count: count,
Addr: n.ResourceAddr(),
},
// Add the count orphans to make sure these resources are accounted for
// during a scale in.
&OrphanResourceCountTransformer{
Concrete: concreteResource,
Count: count,
Addr: n.ResourceAddr(),
State: state,
},
// Attach the state
&AttachStateTransformer{State: state},
// Targeting
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
&TargetsTransformer{Targets: n.Targets},
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
// Connect references so ordering is correct
&ReferenceTransformer{},
// Make sure there is a single root
&RootTransformer{},
}
// Build the graph
b := &BasicGraphBuilder{
Steps: steps,
Validate: true,
Name: "NodeRefreshableManagedResource",
}
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
graph, diags := b.Build(ctx.Path())
return graph, diags.ErrWithWarnings()
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
}
// NodeRefreshableManagedResourceInstance represents a resource that is "applyable":
// it is ready to be applied and is represented by a diff.
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
type NodeRefreshableManagedResourceInstance struct {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
*NodeAbstractResourceInstance
}
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
var (
_ GraphNodeSubPath = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeReferenceable = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeReferencer = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeDestroyer = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeResource = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeResourceInstance = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeAttachResourceConfig = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeAttachResourceState = (*NodeRefreshableManagedResourceInstance)(nil)
_ GraphNodeEvalable = (*NodeRefreshableManagedResourceInstance)(nil)
)
// GraphNodeDestroyer
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
func (n *NodeRefreshableManagedResourceInstance) DestroyAddr() *addrs.AbsResourceInstance {
addr := n.ResourceInstanceAddr()
return &addr
}
// GraphNodeEvalable
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
func (n *NodeRefreshableManagedResourceInstance) EvalTree() EvalNode {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
addr := n.ResourceInstanceAddr()
// Eval info is different depending on what kind of resource this is
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
switch addr.Resource.Resource.Mode {
case addrs.ManagedResourceMode:
if n.ResourceState == nil {
log.Printf("[TRACE] NodeRefreshableManagedResourceInstance: %s has no existing state to refresh", addr)
return n.evalTreeManagedResourceNoState()
}
log.Printf("[TRACE] NodeRefreshableManagedResourceInstance: %s will be refreshed", addr)
return n.evalTreeManagedResource()
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
case addrs.DataResourceMode:
// Get the data source node. If we don't have a configuration
// then it is an orphan so we destroy it (remove it from the state).
var dn GraphNodeEvalable
if n.Config != nil {
dn = &NodeRefreshableDataResourceInstance{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
NodeAbstractResourceInstance: n.NodeAbstractResourceInstance,
}
} else {
dn = &NodeDestroyableDataResource{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
NodeAbstractResourceInstance: n.NodeAbstractResourceInstance,
}
}
return dn.EvalTree()
default:
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
panic(fmt.Errorf("unsupported resource mode %s", addr.Resource.Resource.Mode))
}
}
core: New refresh graph building behaviour Currently, the refresh graph uses the resources from state as a base, with data sources then layered on. Config is not consulted for resources and hence new resources that are added with count (or any new resource from config, for that matter) do not get added to the graph during refresh. This is leading to issues with scale in and scale out when the same value for count is used in both resources, and data sources that may depend on that resource (and possibly vice versa). While the resources exist in config and can be used, the fact that ConfigTransformer for resources is missing means that they don't get added into the graph, leading to "index out of range" errors and what not. Further to that, if we add these new resources to the graph for scale out, considerations need to be taken for scale in as well, which are not being caught 100% by the current implementation of NodeRefreshableDataResource. Scale-in resources should be treated as orphans, which according to the instance-form NodeRefreshableResource node, should be NodeDestroyableDataResource nodes, but this this logic is currently not rolled into NodeRefreshableDataResource. This causes issues on scale-in in the form of race-ish "index out of range" errors again. This commit updates the refresh graph so that StateTransformer is no longer used as the base of the graph. Instead, we add resources from the state and config in a hybrid fashion: * First off, resource nodes are added from config, but only if resources currently exist in state. NodeRefreshableManagedResource is a new expandable resource node that will expand count and add orphans from state. Any count-expanded node that has config but no state is also transformed into a plannable resource, via a new ResourceRefreshPlannableTransformer. * The NodeRefreshableDataResource node type will now add count orphans as NodeDestroyableDataResource nodes. This achieves the same effect as if the data sources were added by StateTransformer, but ensures there are no races in the dependency chain, with the added benefit of directing these nodes straight to the proper NodeDestroyableDataResource node. * Finally, config orphans (nodes that don't exist in config anymore period) are then added, to complete the graph. This should ensure as much as possible that there is a refresh graph that best represents both the current state and config with updated variables and counts.
2017-04-30 08:07:01 +02:00
func (n *NodeRefreshableManagedResourceInstance) evalTreeManagedResource() EvalNode {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
addr := n.ResourceInstanceAddr()
// Declare a bunch of variables that are used for state during
// evaluation. Most of this are written to by-address below.
var provider providers.Interface
var providerSchema *ProviderSchema
var state *states.ResourceInstanceObject
// This happened during initial development. All known cases were
// fixed and tested but as a sanity check let's assert here.
if n.ResourceState == nil {
err := fmt.Errorf(
"No resource state attached for addr: %s\n\n"+
"This is a bug. Please report this to Terraform with your configuration\n"+
"and state attached. Please be careful to scrub any sensitive information.",
addr)
return &EvalReturnError{Error: &err}
}
return &EvalSequence{
Nodes: []EvalNode{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
&EvalGetProvider{
Addr: n.ResolvedProvider,
Output: &provider,
Schema: &providerSchema,
},
&EvalReadState{
Addr: addr.Resource,
Provider: &provider,
ProviderSchema: &providerSchema,
Output: &state,
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
},
&EvalRefresh{
Addr: addr.Resource,
ProviderAddr: n.ResolvedProvider,
Provider: &provider,
ProviderSchema: &providerSchema,
State: &state,
Output: &state,
},
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
&EvalWriteState{
Addr: addr.Resource,
ProviderAddr: n.ResolvedProvider,
ProviderSchema: &providerSchema,
State: &state,
},
},
}
}
// evalTreeManagedResourceNoState produces an EvalSequence for refresh resource
// nodes that don't have state attached. An example of where this functionality
// is useful is when a resource that already exists in state is being scaled
// out, ie: has its resource count increased. In this case, the scaled out node
// needs to be available to other nodes (namely data sources) that may depend
// on it for proper interpolation, or confusing "index out of range" errors can
// occur.
//
// The steps in this sequence are very similar to the steps carried out in
// plan, but nothing is done with the diff after it is created - it is dropped,
// and its changes are not counted in the UI.
func (n *NodeRefreshableManagedResourceInstance) evalTreeManagedResourceNoState() EvalNode {
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
addr := n.ResourceInstanceAddr()
// Declare a bunch of variables that are used for state during
// evaluation. Most of this are written to by-address below.
var provider providers.Interface
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
var providerSchema *ProviderSchema
var change *plans.ResourceInstanceChange
var state *states.ResourceInstanceObject
return &EvalSequence{
Nodes: []EvalNode{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
&EvalGetProvider{
Addr: n.ResolvedProvider,
Output: &provider,
Schema: &providerSchema,
},
&EvalReadState{
Addr: addr.Resource,
Provider: &provider,
ProviderSchema: &providerSchema,
Output: &state,
},
&EvalDiff{
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
Addr: addr.Resource,
Config: n.Config,
Provider: &provider,
ProviderAddr: n.ResolvedProvider,
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
ProviderSchema: &providerSchema,
State: &state,
OutputChange: &change,
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
OutputState: &state,
Stub: true,
},
terraform: ugly huge change to weave in new HCL2-oriented types Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.
2018-04-30 19:33:53 +02:00
&EvalWriteState{
Addr: addr.Resource,
ProviderAddr: n.ResolvedProvider,
ProviderSchema: &providerSchema,
State: &state,
},
core: Prune placeholder objects from state after refresh Prior to our refactoring here, we were relying on a lucky coincidence for correct behavior of the plan walk following a refresh in the same run: - The refresh phase created placeholder objects in the state to represent any resource instance pending creation, to allow the interpolator to read attributes from them when evaluating "provider" and "data" blocks. In effect, the refresh walk is creating a partial plan that only covers creation actions, but was immediately discarding the actual diff entries and storing only the planned new state. - It happened that objects pending creation showed up in state with an empty ID value, since that only gets assigned by the provider during apply. - The Refresh function concluded by calling terraform.State.Prune, which deletes from the state any objects that have an empty ID value, which therefore prevented these temporary objects from surviving into the plan phase. After refactoring, we no longer have this special ID field on instance object state, and we instead rely on the Status field for tracking such things. We also no longer have an explicit "prune" step on state, since the state mutation methods themselves keep the structure pruned. To address this, here we introduce a new instance object status "planned", which is equivalent to having an empty ID value in the old world. We also introduce a new method on states.SyncState that deletes from the state any planned objects, which therefore replaces that portion of the old State.prune operation just for this refresh use-case. Finally, we are now expecting the expression evaluator to pull pending objects from the planned changeset rather than from the state directly, and so for correct results these placeholder resource creation changes must also be reported in a throwaway changeset during the refresh walk. The addition of states.ObjectPlanned also permits a previously-missing safety check in the expression evaluator to prevent us from relying on the incomplete value stored in state for a pending object, in the event that some bug prevents the real pending object from being written into the planned changeset.
2018-09-01 01:42:07 +02:00
// We must also save the planned change, so that expressions in
// other nodes, such as provider configurations and data resources,
// can work with the planned new value.
//
// This depends on the fact that Context.Refresh creates a
// temporary new empty changeset for the duration of its graph
// walk, and so this recorded change will be discarded immediately
// after the refresh walk completes.
&EvalWriteDiff{
Addr: addr.Resource,
Change: &change,
ProviderSchema: &providerSchema,
},
},
}
}