Several changes to lookup to improve how we handle marked values:
- If the entire collection is marked, preserve the marks on any result
(whether successful or fallback)
- If a returned value from the collection is marked, preserve the marks
from only that value, combined with any overall collection marks
- Retain marks on the fallback value when it is returned, combined with
any overall collection marks
- Include marks on the key in the result, as otherwise the result it
ends up selecting could imply what the sensitive value was
- Retain collection marks when returning an unknown value for a not
wholly-known collection
See also https://github.com/zclconf/go-cty/pull/98
Similar to cty's implementation, we only need to preserve marks from the
value itself, not any nested values it may contain. This means that
taking the length of an umarked list with marked elements results in an
unmarked number.
In the Terraform language we typically use lists of zero or one values in
some sense interchangably with single values that might be null, because
various Terraform language constructs are designed to work with
collections rather than with nullable values.
In Terraform v0.12 we made the splat operator [*] have a "special power"
of concisely converting from a possibly-null single value into a
zero-or-one list as a way to make that common operation more concise.
In a sense this "one" function is the opposite operation to that special
power: it goes from a zero-or-one collection (list, set, or tuple) to a
possibly-null single value.
This is a concise alternative to the following clunky conditional
expression, with the additional benefit that the following expression is
also not viable for set values, and it also properly handles the case
where there's unexpectedly more than one value:
length(var.foo) != 0 ? var.foo[0] : null
Instead, we can write:
one(var.foo)
As with the splat operator, this is a tricky tradeoff because it could be
argued that it's not something that'd be immediately intuitive to someone
unfamiliar with Terraform. However, I think that's justified given how
often zero-or-one collections arise in typical Terraform configurations.
Unlike the splat operator, it should at least be easier to search for its
name and find its documentation the first time you see it in a
configuration.
My expectation that this will become a common pattern is also my
justification for giving it a short, concise name. Arguably it could be
better named something like "oneornull", but that's a pretty clunky name
and I'm not convinced it really adds any clarity for someone who isn't
already familiar with it.
Fixes error when calling sum with values not known until apply time.
Also allows sum to cope with numbers too large to represent in float64,
along with correctly handling errors when trying to sum opposing
infinities.
The alltrue/anytrue functions did not correctly handle unknown values.
This commit changes these functions so that the result is unknown if:
- The list argument is unknown
- For alltrue: any elements are unknown
- For anytrue: any elements are unknown and no known elements are true
The last change is a little subtle, so there are test cases to cover it
specifically. Examples:
- anytrue(unknown) -> unknown
- anytrue(false, unknown) -> unknown
- anytrue(false, unknown, true) -> true
Prior to Terraform 0.12 these two functions were the only way to construct
literal lists and maps (respectively) in HIL expressions. Terraform 0.12,
by switching to HCL 2, introduced first-class syntax for constructing
tuple and object values, which can then be converted into list and map
values using the tolist and tomap type conversion functions.
We marked both of these functions as deprecated in the Terraform v0.12
release and have since then mentioned in the docs that they will be
removed in a future Terraform version. The "terraform 0.12upgrade" tool
from Terraform v0.12 also included a rule to automatically rewrite uses
of these functions into equivalent new syntax.
The main motivation for removing these now is just to get this change made
prior to Terraform 1.0. as we'll be doing with various other deprecations.
However, a specific reason for these two functions in particular is that
their existence is what caused us to invent the idea of a "type expression"
as a distinct kind of expression in Terraform v0.12, and so removing them
now would allow potentially unifying type expressions with value
expressions in a future release.
We do not have any current specific plans to make that change, but one
potential motivation for doing so would be to take another attempt at a
generalized "convert" function which takes a type as one of its arguments.
Our previous attempt to implement such a function was foiled by the fact
that Terraform's expression validator doesn't have any way to know to
treat one argument of a particular function as special, and so it was
generating incorrect error messages. We won't necessarily do that, but
having these "list" and "map" functions out of the way leaves the option
open.
This is an analog to the "alltrue" function, using OR as the reduce
operator rather than AND.
This also includes some simplification of the "alltrue" implementation
to implement it similarly as a sort of reduce operation with AND
as the reduce operator, but with the same effective behavior.
Fix for issue #26320 - this allows us to derive known values from
partially known maps where we can, and may prevent unnecessary
destroy/rebuild cycles during apply in some cases.
This commit adds an `alltrue` function to Terraform configuration. A
reason we might want this function is because it will enable more
powerful custom variable validations. For example:
```hcl
variable "amis" {
type = list(object({
id = string
}))
validation {
condition = (alltrue([
for a in var.amis : length(a.id) > 4 && substr(a.id, 0, 4) == "ami-"
]))
error_message = "The ID of at least one AMI was invalid."
}
}
```
The map function assumed that the key arguments were strings, and would
panic if they were not.
After this commit, calling `map(1, 2)` will result in a map `{"1" = 1}`,
and calling `map(null, 1)` will result in a syntax error.
Fixes#23346, fixes#23043
This PR implements 2 changes to the merge function.
- Rather than always defining the merge return type as dynamic, return
a precise type when all argument types match, or all possible object
attributes are known.
- Always return a value containing all keys when the keys are known.
This allows the use of merge output in for_each, even when keys are yet
to be determined.
* lang/funcs: lookup() can work with maps of lists, maps and objects
lookup() can already handle aribtrary objects of (whatever) and should
handle maps of (whatever) similarly.
The function would previously panic when one or more null values were among the arguments.
The new behavior treats nulls as empty strings, therefore, it removes them.
Added higher-level test for matchkeys to exercise mixing
types in searchset. This had to be in the functions tests so the HCL
auto conversion from tuple to list would occur.
`matchkeys` was returning a (false) error if the searchset was a
variable, since then the type of the keylist and searchset parameters
would not match.
This does slightly change the behavior: previously matchkeys would
produce an error if the parameters were not of the same type, for e.g.
if searchset was a list of strings and keylist was a list of integers.
This no longer produces an error.
Previously the type-selection codepath for an input tuple referred
unconditionally to the start and end index values. In the Type callback,
only the types of the arguments are guaranteed to be known, so any access
to the values must be guarded with an .IsKnown check, or else the function
will not short-circuit properly when an unknown value is passed.
Now we will check the start and end indices are in range when we have
enough information to do so, but we'll return an approximate result if
either is unknown.
FlattenFunc can return lists and tuples when individual elements are
unknown. Only return an unknown tuple if the number of elements cannot
be determined because it contains an unknown series.
Make sure flatten can handle non-series elements, which were previously
lost due to passing a slice value as the argument.
When slicing a list containing unknown values, the list itself is known,
and slice should return the proper slice of that list.
Make SliceFunc return the correct type for slices of tuples, and
disallow slicing sets.
* lang/funcs: testing of functions through the lang package API
The function-specific unit tests do not cover the HCL conversion that happens when the functions are called in a terraform configuration. For e.g., HCL converts sets to lists before passing it to the function. This means that we could not test passing a set in the function _unit_ tests.
This adds a higher-level acceptance test, plus a check that every (pure) function has a test.
* website/docs: update function documentation
* funcs/coalesce: return the first non-null, non-empty element from a
sequence.
The go-cty coalesce function, which was originally used here, returns the
first non-null element from a sequence. Terraform 0.11's coalesce,
however, returns the first non-empty string from a list of strings.
This new coalesce function aims to preserve terraform's documented
functionality while adding support for additional argument types. The
tests include those in go-cty and adapted tests from the 0.11 version of
coalesce.
* website/docs: update coalesce function document
In our new world it produces either a set of a tuple type or a list of a
tuple type, depending on the given argument types.
The resulting collection's element tuple type is decided by the element
types of the given collections, allowing type information to propagate
even if unknown values are present.
Now that our language supports tuple/object types in addition to list/map
types, it's convenient for zipmap to be able to produce an object type
given a tuple, since this makes it symmetrical with "keys" and "values"
such the the following identity holds for any map or object value "a"
a == zipmap(keys(a), values(a))
When the values sequence is a tuple, the result has an object type whose
attribute types correspond to the given tuple types.
Since an object type has attribute names as part of its definition, there
is the additional constraint here that the result has an unknown type
(represented by the dynamic pseudo-type) if the given values is a tuple
and the given keys contains any unknown values. This isn't true for values
as a list because we can predict the resulting map element type using
just the list element type.
The "values" function wasn't producing consistently-ordered keys in its
result, leading to crashes. This fixes#19204.
While working on these functions anyway, this also improves slightly their
precision when working with object types, where we can produce a more
complete result for unknown values because the attribute names are part
of the type. We can also produce results for known maps that have unknown
elements; these unknowns will also appear in the values(...) result,
allowing them to propagate through expressions.
Finally, this adds a few more test cases to try different permutations
of empty and unknown values.
Alisdair McDiarmid