backend/remote: Support HCL variable values in local operations
For remote operations, the remote system (Terraform Cloud or Enterprise)
writes the stored variable values into a .tfvars file before running the
remote copy of Terraform CLI.
By contrast, for operations that only run locally (like
"terraform import"), we fetch the stored variable values from the remote
API and add them into the set of available variables directly as part
of creating the local execution context.
Previously in the local-only case we were assuming that all stored
variables are strings, which isn't true: the Terraform Cloud/Enterprise UI
allows users to specify that a particular variable is given as an HCL
expression, in which case the correct behavior is to parse and evaluate
the expression to obtain the final value.
This also addresses a related issue whereby previously we were forcing
all sensitive values to be represented as a special string "<sensitive>".
That leads to type checking errors for any variable specified as having
a type other than string, so instead here we use an unknown value as a
placeholder so that type checking can pass.
Unpopulated sensitive values may cause errors downstream though, so we'll
also produce a warning for each of them to let the user know that those
variables are not available for local-only operations. It's a warning
rather than an error so that operations that don't rely on known values
for those variables can potentially complete successfully.
This can potentially produce errors in situations that would've been
silently ignored before: if a remote variable is marked as being HCL
syntax but is not valid HCL then it will now fail parsing at this early
stage, whereas previously it would've just passed through as a string
and failed only if the operation tried to interpret it as a non-string.
However, in situations like these the remote operations like
"terraform plan" would already have been failing with an equivalent
error message anyway, so it's unlikely that any existing workspace that
is being used for routine operations would have such a broken
configuration.
2019-10-30 01:09:16 +01:00
package remote
import (
2020-04-30 23:56:32 +02:00
"context"
backend/remote: Support HCL variable values in local operations
For remote operations, the remote system (Terraform Cloud or Enterprise)
writes the stored variable values into a .tfvars file before running the
remote copy of Terraform CLI.
By contrast, for operations that only run locally (like
"terraform import"), we fetch the stored variable values from the remote
API and add them into the set of available variables directly as part
of creating the local execution context.
Previously in the local-only case we were assuming that all stored
variables are strings, which isn't true: the Terraform Cloud/Enterprise UI
allows users to specify that a particular variable is given as an HCL
expression, in which case the correct behavior is to parse and evaluate
the expression to obtain the final value.
This also addresses a related issue whereby previously we were forcing
all sensitive values to be represented as a special string "<sensitive>".
That leads to type checking errors for any variable specified as having
a type other than string, so instead here we use an unknown value as a
placeholder so that type checking can pass.
Unpopulated sensitive values may cause errors downstream though, so we'll
also produce a warning for each of them to let the user know that those
variables are not available for local-only operations. It's a warning
rather than an error so that operations that don't rely on known values
for those variables can potentially complete successfully.
This can potentially produce errors in situations that would've been
silently ignored before: if a remote variable is marked as being HCL
syntax but is not valid HCL then it will now fail parsing at this early
stage, whereas previously it would've just passed through as a string
and failed only if the operation tried to interpret it as a non-string.
However, in situations like these the remote operations like
"terraform plan" would already have been failing with an equivalent
error message anyway, so it's unlikely that any existing workspace that
is being used for routine operations would have such a broken
configuration.
2019-10-30 01:09:16 +01:00
"testing"
tfe "github.com/hashicorp/go-tfe"
2019-11-13 17:34:09 +01:00
"github.com/hashicorp/terraform/backend"
2021-02-16 13:19:22 +01:00
"github.com/hashicorp/terraform/command/arguments"
"github.com/hashicorp/terraform/command/clistate"
"github.com/hashicorp/terraform/command/views"
backend/remote: Support HCL variable values in local operations
For remote operations, the remote system (Terraform Cloud or Enterprise)
writes the stored variable values into a .tfvars file before running the
remote copy of Terraform CLI.
By contrast, for operations that only run locally (like
"terraform import"), we fetch the stored variable values from the remote
API and add them into the set of available variables directly as part
of creating the local execution context.
Previously in the local-only case we were assuming that all stored
variables are strings, which isn't true: the Terraform Cloud/Enterprise UI
allows users to specify that a particular variable is given as an HCL
expression, in which case the correct behavior is to parse and evaluate
the expression to obtain the final value.
This also addresses a related issue whereby previously we were forcing
all sensitive values to be represented as a special string "<sensitive>".
That leads to type checking errors for any variable specified as having
a type other than string, so instead here we use an unknown value as a
placeholder so that type checking can pass.
Unpopulated sensitive values may cause errors downstream though, so we'll
also produce a warning for each of them to let the user know that those
variables are not available for local-only operations. It's a warning
rather than an error so that operations that don't rely on known values
for those variables can potentially complete successfully.
This can potentially produce errors in situations that would've been
silently ignored before: if a remote variable is marked as being HCL
syntax but is not valid HCL then it will now fail parsing at this early
stage, whereas previously it would've just passed through as a string
and failed only if the operation tried to interpret it as a non-string.
However, in situations like these the remote operations like
"terraform plan" would already have been failing with an equivalent
error message anyway, so it's unlikely that any existing workspace that
is being used for routine operations would have such a broken
configuration.
2019-10-30 01:09:16 +01:00
"github.com/hashicorp/terraform/configs"
2019-11-13 17:34:09 +01:00
"github.com/hashicorp/terraform/internal/initwd"
2021-02-16 13:19:22 +01:00
"github.com/hashicorp/terraform/internal/terminal"
2020-08-11 17:23:42 +02:00
"github.com/hashicorp/terraform/states/statemgr"
backend/remote: Support HCL variable values in local operations
For remote operations, the remote system (Terraform Cloud or Enterprise)
writes the stored variable values into a .tfvars file before running the
remote copy of Terraform CLI.
By contrast, for operations that only run locally (like
"terraform import"), we fetch the stored variable values from the remote
API and add them into the set of available variables directly as part
of creating the local execution context.
Previously in the local-only case we were assuming that all stored
variables are strings, which isn't true: the Terraform Cloud/Enterprise UI
allows users to specify that a particular variable is given as an HCL
expression, in which case the correct behavior is to parse and evaluate
the expression to obtain the final value.
This also addresses a related issue whereby previously we were forcing
all sensitive values to be represented as a special string "<sensitive>".
That leads to type checking errors for any variable specified as having
a type other than string, so instead here we use an unknown value as a
placeholder so that type checking can pass.
Unpopulated sensitive values may cause errors downstream though, so we'll
also produce a warning for each of them to let the user know that those
variables are not available for local-only operations. It's a warning
rather than an error so that operations that don't rely on known values
for those variables can potentially complete successfully.
This can potentially produce errors in situations that would've been
silently ignored before: if a remote variable is marked as being HCL
syntax but is not valid HCL then it will now fail parsing at this early
stage, whereas previously it would've just passed through as a string
and failed only if the operation tried to interpret it as a non-string.
However, in situations like these the remote operations like
"terraform plan" would already have been failing with an equivalent
error message anyway, so it's unlikely that any existing workspace that
is being used for routine operations would have such a broken
configuration.
2019-10-30 01:09:16 +01:00
"github.com/zclconf/go-cty/cty"
)
func TestRemoteStoredVariableValue ( t * testing . T ) {
tests := map [ string ] struct {
Def * tfe . Variable
Want cty . Value
WantError string
} {
"string literal" : {
& tfe . Variable {
Key : "test" ,
Value : "foo" ,
HCL : false ,
Sensitive : false ,
} ,
cty . StringVal ( "foo" ) ,
` ` ,
} ,
"string HCL" : {
& tfe . Variable {
Key : "test" ,
Value : ` "foo" ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . StringVal ( "foo" ) ,
` ` ,
} ,
"list HCL" : {
& tfe . Variable {
Key : "test" ,
Value : ` [] ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . EmptyTupleVal ,
` ` ,
} ,
"null HCL" : {
& tfe . Variable {
Key : "test" ,
Value : ` null ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . NullVal ( cty . DynamicPseudoType ) ,
` ` ,
} ,
"literal sensitive" : {
& tfe . Variable {
Key : "test" ,
HCL : false ,
Sensitive : true ,
} ,
cty . UnknownVal ( cty . String ) ,
` ` ,
} ,
"HCL sensitive" : {
& tfe . Variable {
Key : "test" ,
HCL : true ,
Sensitive : true ,
} ,
cty . DynamicVal ,
` ` ,
} ,
"HCL computation" : {
// This (stored expressions containing computation) is not a case
// we intentionally supported, but it became possible for remote
// operations in Terraform 0.12 (due to Terraform Cloud/Enterprise
// just writing the HCL verbatim into generated `.tfvars` files).
// We support it here for consistency, and we continue to support
// it in both places for backward-compatibility. In practice,
// there's little reason to do computation in a stored variable
// value because references are not supported.
& tfe . Variable {
Key : "test" ,
Value : ` [for v in ["a"] : v] ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . TupleVal ( [ ] cty . Value { cty . StringVal ( "a" ) } ) ,
` ` ,
} ,
"HCL syntax error" : {
& tfe . Variable {
Key : "test" ,
Value : ` [ ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . DynamicVal ,
` Invalid expression for var.test: The value of variable "test" is marked in the remote workspace as being specified in HCL syntax, but the given value is not valid HCL. Stored variable values must be valid literal expressions and may not contain references to other variables or calls to functions. ` ,
} ,
"HCL with references" : {
& tfe . Variable {
Key : "test" ,
Value : ` foo.bar ` ,
HCL : true ,
Sensitive : false ,
} ,
cty . DynamicVal ,
` Invalid expression for var.test: The value of variable "test" is marked in the remote workspace as being specified in HCL syntax, but the given value is not valid HCL. Stored variable values must be valid literal expressions and may not contain references to other variables or calls to functions. ` ,
} ,
}
for name , test := range tests {
t . Run ( name , func ( t * testing . T ) {
v := & remoteStoredVariableValue {
definition : test . Def ,
}
// This ParseVariableValue implementation ignores the parsing mode,
// so we'll just always parse literal here. (The parsing mode is
// selected by the remote server, not by our local configuration.)
gotIV , diags := v . ParseVariableValue ( configs . VariableParseLiteral )
if test . WantError != "" {
if ! diags . HasErrors ( ) {
t . Fatalf ( "missing expected error\ngot: <no error>\nwant: %s" , test . WantError )
}
errStr := diags . Err ( ) . Error ( )
if errStr != test . WantError {
t . Fatalf ( "wrong error\ngot: %s\nwant: %s" , errStr , test . WantError )
}
} else {
if diags . HasErrors ( ) {
t . Fatalf ( "unexpected error\ngot: %s\nwant: <no error>" , diags . Err ( ) . Error ( ) )
}
got := gotIV . Value
if ! test . Want . RawEquals ( got ) {
t . Errorf ( "wrong result\ngot: %#v\nwant: %#v" , got , test . Want )
}
}
} )
}
}
2019-11-13 17:34:09 +01:00
func TestRemoteContextWithVars ( t * testing . T ) {
catTerraform := tfe . CategoryTerraform
catEnv := tfe . CategoryEnv
tests := map [ string ] struct {
Opts * tfe . VariableCreateOptions
WantError string
} {
"Terraform variable" : {
& tfe . VariableCreateOptions {
Category : & catTerraform ,
} ,
` Value for undeclared variable: A variable named "key" was assigned a value, but the root module does not declare a variable of that name. To use this value, add a "variable" block to the configuration. ` ,
} ,
"environment variable" : {
& tfe . VariableCreateOptions {
Category : & catEnv ,
} ,
` ` ,
} ,
}
for name , test := range tests {
t . Run ( name , func ( t * testing . T ) {
configDir := "./testdata/empty"
b , bCleanup := testBackendDefault ( t )
defer bCleanup ( )
_ , configLoader , configCleanup := initwd . MustLoadConfigForTests ( t , configDir )
defer configCleanup ( )
2020-04-30 23:56:32 +02:00
workspaceID , err := b . getRemoteWorkspaceID ( context . Background ( ) , backend . DefaultStateName )
if err != nil {
t . Fatal ( err )
}
2021-02-16 13:19:22 +01:00
streams , _ := terminal . StreamsForTesting ( t )
view := views . NewStateLocker ( arguments . ViewHuman , views . NewView ( streams ) )
2019-11-13 17:34:09 +01:00
op := & backend . Operation {
ConfigDir : configDir ,
ConfigLoader : configLoader ,
2021-02-16 13:19:22 +01:00
StateLocker : clistate . NewLocker ( 0 , view ) ,
2019-11-13 17:34:09 +01:00
Workspace : backend . DefaultStateName ,
}
v := test . Opts
if v . Key == nil {
key := "key"
v . Key = & key
}
2020-12-01 15:10:01 +01:00
b . client . Variables . Create ( context . TODO ( ) , workspaceID , * v )
2019-11-13 17:34:09 +01:00
_ , _ , diags := b . Context ( op )
if test . WantError != "" {
if ! diags . HasErrors ( ) {
t . Fatalf ( "missing expected error\ngot: <no error>\nwant: %s" , test . WantError )
}
errStr := diags . Err ( ) . Error ( )
if errStr != test . WantError {
t . Fatalf ( "wrong error\ngot: %s\nwant: %s" , errStr , test . WantError )
}
2020-08-11 17:23:42 +02:00
// When Context() returns an error, it should unlock the state,
// so re-locking it is expected to succeed.
stateMgr , _ := b . StateMgr ( backend . DefaultStateName )
if _ , err := stateMgr . Lock ( statemgr . NewLockInfo ( ) ) ; err != nil {
t . Fatalf ( "unexpected error locking state: %s" , err . Error ( ) )
}
2019-11-13 17:34:09 +01:00
} else {
if diags . HasErrors ( ) {
t . Fatalf ( "unexpected error\ngot: %s\nwant: <no error>" , diags . Err ( ) . Error ( ) )
}
2020-08-11 17:23:42 +02:00
// When Context() succeeds, this should fail w/ "workspace already locked"
stateMgr , _ := b . StateMgr ( backend . DefaultStateName )
if _ , err := stateMgr . Lock ( statemgr . NewLockInfo ( ) ) ; err == nil {
t . Fatal ( "unexpected success locking state after Context" )
}
2019-11-13 17:34:09 +01:00
}
} )
}
}