Commit Graph

26857 Commits

Author SHA1 Message Date
Martin Atkins ef64df950c getproviders: Prepare for having multiple valid hashes per package
As we continue iterating towards saving valid hashes for a package in a
depsfile lock file after installation and verifying them on future
installation, this prepares getproviders for the possibility of having
multiple valid hashes per package.

This will arise in future commits for two reasons:
- We will need to support both the legacy "zip hash" hashing scheme and
  the new-style content-based hashing scheme because currently the
  registry protocol is only able to produce the legacy scheme, but our
  other installation sources prefer the content-based scheme. Therefore
  packages will typically have a mixture of hashes of both types.
- Installing from an upstream registry will save the hashes for the
  packages across all supported platforms, rather than just the current
  platform, and we'll consider all of those valid for future installation
  if we see both successful matching of the current platform checksum and
  a signature verification for the checksums file as a whole.

This also includes some more preparation for the second case above in that
signatureAuthentication now supports AcceptableHashes and returns all of
the zip-based hashes it can find in the checksums file. This is a bit of
an abstraction leak because previously that authenticator considered its
"document" to just be opaque bytes, but we want to make sure that we can
only end up trusting _all_ of the hashes if we've verified that the
document is signed. Hopefully we'll make this better in a future commit
with some refactoring, but that's deferred for now in order to minimize
disruption to existing codepaths while we work towards a provider locking
MVP.
2020-09-24 14:01:54 -07:00
Martin Atkins 6694cfaa0e getproviders: Add a real type Hash for package hashes
The logic for what constitutes a valid hash and how different hash schemes
are represented was starting to get sprawled over many different files and
packages.

Consistently with other cases where we've used named types to gather the
definition of a particular string into a single place and have the Go
compiler help us use it properly, this introduces both getproviders.Hash
representing a hash value and getproviders.HashScheme representing the
idea of a particular hash scheme.

Most of this changeset is updating existing uses of primitive strings to
uses of getproviders.Hash. The new type definitions are in
internal/getproviders/hash.go.
2020-09-24 14:01:54 -07:00
Martin Atkins 264a3cf031 depsfile: Flatten the "hashes" locks to a single set of strings
Although origin registries return specific [filename, hash] pairs, our
various different installation methods can't produce a structured mapping
from platform to hash without breaking changes.

Therefore, as a compromise, we'll continue to do platform-specific checks
against upstream data in the cases where that's possible (installation
from origin registry or network mirror) but we'll treat the lock file as
just a flat set of equally-valid hashes, at least one of which must match
after we've completed whatever checks we've made against the
upstream-provided checksums/signatures.

This includes only the minimal internal/getproviders updates required to
make this compile. A subsequent commit will update that package to
actually support the idea of verifying against multiple hashes.
2020-09-24 14:01:54 -07:00
Martin Atkins b2c0ccdf96 internal/getproviders: Allow PackageMeta to carry acceptable hashes
The "acceptable hashes" for a package is a set of hashes that the upstream
source considers to be good hashes for checking whether future installs
of the same provider version are considered to match this one.

Because the acceptable hashes are a package authentication concern and
they already need to be known (at least in part) to implement the
authenticators, here we add AcceptableHashes as an optional extra method
that an authenticator can implement.

Because these are hashes chosen by the upstream system, the caller must
make its own determination about their trustworthiness. The result of
authentication is likely to be an input to that, for example by
distrusting hashes produced by an authenticator that succeeds but doesn't
report having validated anything.
2020-09-24 14:01:54 -07:00
Martin Atkins e843097e52 internal/getproviders: Formalize the "ziphash" hashing scheme
This is the pre-existing hashing scheme that was initially built for
releases.hashicorp.com and then later reused for the provider registry
protocol, which takes a SHA256 hash of the official distribution .zip file
and formats it as lowercase hex.

This is a non-ideal hash scheme because it works only for
PackageLocalArchive locations, and so we can't verify package directories
on local disk against such hashes. However, the registry protocol is now
a compatibility constraint and so we're going to need to support this
hashing scheme for the foreseeable future.
2020-09-24 14:01:54 -07:00
Pam Selle 5b549224ae Refactor to call ContainsMarked less and use len() instead 2020-09-24 16:42:03 -04:00
Alisdair McDiarmid 0b632c872e
Update CHANGELOG.md 2020-09-24 15:54:50 -04:00
Alisdair McDiarmid 390a3c1102
Update CHANGELOG.md 2020-09-24 15:53:58 -04:00
Alisdair McDiarmid 60c469b4a5
Merge pull request #26345 from hashicorp/alisdair/taint-should-respect-required-version
command: Taint should respect required_version
2020-09-24 15:52:23 -04:00
Pam Selle 3c9fad0b0e Move plan action check into the sensitivity warning method 2020-09-24 13:49:34 -04:00
Pam Selle 531728f6e9 Sensitive diffs for primitive types
When showing primitive type diffs, hide possibly
sensitive values
2020-09-24 13:27:15 -04:00
Pam Selle 20921dbfb8 Add warning about sensitivity change
This commit adds a warning before displaying
a sensitive diff, and always obfuscates the old value (even
if it was not previously marked as sensitive)
2020-09-24 12:57:40 -04:00
Pam Selle 0a02e7040f
Store sensitive attribute paths in state (#26338)
* Add creation test and simplify in-place test

* Add deletion test

* Start adding marking from state

Start storing paths that should be marked
when pulled out of state. Implements deep
copy for attr paths. This commit also includes some
comment noise from investigations, and fixing the diff test

* Fix apply stripping marks

* Expand diff tests

* Basic apply test

* Update comments on equality checks to clarify current understanding

* Add JSON serialization for sensitive paths

We need to serialize a slice of cty.Path values to be used to re-mark
the sensitive values of a resource instance when loading the state file.
Paths consist of a list of steps, each of which may be either getting an
attribute value by name, or indexing into a collection by string or
number.

To serialize these without building a complex parser for a compact
string form, we render a nested array of small objects, like so:

[
  [
    { type: "get_attr", value: "foo" },
    { type: "index", value: { "type": "number", "value": 2 } }
  ]
]

The above example is equivalent to a path `foo[2]`.

* Format diffs with map types

Comparisons need unmarked values to operate on,
so create unmarked values for those operations. Additionally,
change diff to cover map types

* Remove debugging printing

* Fix bug with marking non-sensitive values

When pulling a sensitive value from state,
we were previously using those marks to remark
the planned new value, but that new value
might *not* be sensitive, so let's not do that

* Fix apply test

Apply was not passing the second state
through to the third pass at apply

* Consistency in checking for length of paths vs inspecting into value

* In apply, don't mark with before paths

* AttrPaths test coverage for DeepCopy

* Revert format changes

Reverts format changes in format/diff for this
branch so those changes can be discussed on a separate PR

* Refactor name of AttrPaths to AttrSensitivePaths

* Rename AttributePaths/attributePaths for naming consistency

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
2020-09-24 12:40:17 -04:00
James Bardin a7c5a72c3d
Merge pull request #26358 from hashicorp/jbardin/no-vendor
Remove vendoring
2020-09-24 10:57:02 -04:00
James Bardin 15dd493139
Merge pull request #26357 from hashicorp/jbardin/go-version
Update go to 1.15
2020-09-24 10:09:45 -04:00
James Bardin 435d8bdbae
Merge pull request #26343 from hashicorp/jbardin/update-cbd-state
Update create_before_destroy in state during refresh
2020-09-24 10:03:18 -04:00
James Bardin a0cee10720 add Addr field for logging 2020-09-24 09:49:22 -04:00
James Bardin eb17d9799b refresh cbd test 2020-09-24 09:43:48 -04:00
James Bardin 27809871ca update create_before_destroy when refreshing
In order to save any changes to lifecycle options, we need to record
those changes during refresh, otherwise they would only be updated when
there is a change in the resource to be applied.
2020-09-24 09:43:45 -04:00
James Bardin 014bd30a67
Merge pull request #26353 from hashicorp/jbardin/refresh-false
Re-implement -refresh=false
2020-09-24 09:40:33 -04:00
James Bardin 37569f5cc3 insert PlanRefresh into the context 2020-09-24 09:34:49 -04:00
James Bardin b16c600edc verify skipRefresh during plan 2020-09-24 09:34:49 -04:00
James Bardin 84f7116ac8 thread skipContext through to the instance node 2020-09-24 09:34:49 -04:00
James Bardin eebb4dfcb2 add SkipRefresh to the terraform context 2020-09-24 09:34:49 -04:00
James Bardin c2566bff7b
Merge pull request #26351 from hashicorp/jbardin/dependencies
we no longer need EvalRefreshDependencies
2020-09-24 09:33:59 -04:00
James Bardin 53a8e321eb remove mod=vendor in statemgr test 2020-09-24 08:44:49 -04:00
James Bardin 2e5366ac69 remove -mod=vendor from command tests 2020-09-23 17:56:19 -04:00
James Bardin 50e547db4d remove vendoring from contribution guide 2020-09-23 17:06:59 -04:00
James Bardin 8bfbafa790 remove vendor from some tooling 2020-09-23 17:06:59 -04:00
James Bardin bf5fc8b913 go1.15 in circleci 2020-09-23 16:54:37 -04:00
James Bardin 35a5b40500 remove vendor folder
Go modules are well understood and supported now, and since our build
pipeline no longer uses the vendored packages, we can remove the extra
overhead of maintaining these files.
2020-09-23 16:44:35 -04:00
James Bardin 6b6499d61c update go.sum for go1.15 2020-09-23 16:43:43 -04:00
James Bardin df1ac404fc fix error string matching in test
This package is no longer used, but we are retaining it for now for its
help in protocol test coverage.
2020-09-23 16:43:40 -04:00
James Bardin 537070ca57 update to go1.15.2 2020-09-23 15:18:22 -04:00
hashicorp-ci 6d86cd46f7 Cleanup after v0.14.0-alpha20200923 release 2020-09-23 18:19:52 +00:00
James Bardin def1f9b084 we no longer need EvalRefreshDependencies
This evaluation was required when refresh ran in a separate walk and
managed resources were only partly handled by configuration. Now that we
have the correct dependency information available when refreshing
configured resources, we can update their state accordingly. Since
orphaned resources are not refreshed, they can retain their stored
dependencies for correct ordering.

This also prevents users from introducing cycles with nodes they can't
"see", since only orphaned nodes will retain their stored dependencies,
and the remaining nodes will be updated according to the configuration.
2020-09-23 14:08:52 -04:00
hashicorp-ci f4a36d8939
Release v0.14.0-alpha20200923 2020-09-23 17:44:31 +00:00
Martin Atkins bd5aeb3d45
Update CHANGELOG.md 2020-09-23 10:22:41 -07:00
Martin Atkins 1510a33d2a
Update CHANGELOG.md 2020-09-23 10:18:57 -07:00
Pam Selle c8169950a8
Merge pull request #26342 from hashicorp/pselle/evaluate_test
tests: Add test for GetInputVariable, with sensitive config
2020-09-23 11:25:47 -04:00
Martin Atkins ddab48a3b4
hashibot: Temporarily disable issue migration
The bot seems to currently be running into some operational problems that are
creating noise for provider development teams by potentially migrating issues
multiple times.

This is just a tactical change to stop the annoying symptoms right now, to
give some time to figure out what's actually going on here.
2020-09-22 14:41:18 -07:00
Alisdair McDiarmid 14a233b019 command: Taint should respect required_version
Despite not requiring the configuration for any other reason, the taint
subcommand should not execute if the required_version constraints cannot
be met. Doing so can result in an undesirable state file upgrade.
2020-09-22 17:33:09 -04:00
Pam Selle a720409ded Add test for GetInputVariable, with sensitive config
This adds a test for GetInputVariable, and includes
a variable with a "sensitive" attribute in configuration,
to test that that value is marked as sensitive
2020-09-22 16:35:40 -04:00
James Bardin ce2e59f835
Merge pull request #26329 from hashicorp/jbardin/remove-refresh
Remove refresh entirely
2020-09-22 15:40:53 -04:00
Pam Selle 9c16724221
Merge pull request #26337 from hashicorp/pselle/docs-vendor-provisioners
docs: Vendor provisioner deprecation version
2020-09-22 14:54:21 -04:00
Alisdair McDiarmid 0a5f2d9047
Merge pull request #26334 from hashicorp/alisdair/deprecate-nested-redundant-interpolation-expressions
configs: Deprecate nested redundant interpolations
2020-09-22 14:11:52 -04:00
Pam Selle 3ba38f9b1f (docs) Vendor provisioner deprecation version 2020-09-22 14:06:39 -04:00
Alisdair McDiarmid 13ed5af5c8 configs: Deprecate nested redundant interpolations
Previous deprecations only included direct assignment of template-only
expressions to arguments. That is, this was not deprecated:

locals {
  foo = ["${var.foo}"]
}

This commit uses hclsyntax.VisitAll to detect and show deprecations for
all template-only expressions, no matter how deep they are in a given
expression.
2020-09-22 13:40:08 -04:00
Alisdair McDiarmid 85ddb386ec
Merge pull request #26318 from hashicorp/alisdair/providers-schema-required-providers
command: Providers schema shows required_providers
2020-09-22 12:42:06 -04:00
James Bardin 87676358e7 update CHANGELOG.md 2020-09-22 10:57:47 -04:00