command/cliconfig: Allow development overrides for providers

For normal provider installation we want to associate each provider with
a selected version number and find a suitable package for that version
that conforms to the official hashes for that release.

Those requirements are very onerous for a provider developer currently
testing a not-yet-released build, though. To allow for that case this new
CLI configuration feature allows overriding specific providers to refer
to give local filesystem directories.

Any provider overridden in this way is not subject to the usual
restrictions about selected versions or checksum conformance, and
activating an override won't cause any changes to the selections recorded
in the lock file because it's intended to be a temporary setting for one
developer only.

This is, in a sense, a spiritual successor of an old capability we had to
override specific plugins in the CLI configuration file. There were
some vestiges of that left in the main package and CLI config package
but nothing has actually been honoring them for several versions now and
so this commit removes them to avoid confusion with the new mechanism.
This commit is contained in:
Martin Atkins 2020-10-14 18:00:23 -07:00
parent 98dc399062
commit 30204ecded
17 changed files with 389 additions and 28 deletions

View File

@ -131,6 +131,11 @@ func (c *ApplyCommand) Run(args []string) int {
return 1
}
// Applying changes with dev overrides in effect could make it impossible
// to switch back to a release version if the schema isn't compatible,
// so we'll warn about it.
diags = diags.Append(c.providerDevOverrideWarnings())
// Before we delegate to the backend, we'll print any warning diagnostics
// we've accumulated here, since the backend will start fresh with its own
// diagnostics.

View File

@ -2,9 +2,12 @@ package cliconfig
import (
"fmt"
"path/filepath"
"github.com/hashicorp/hcl"
hclast "github.com/hashicorp/hcl/hcl/ast"
"github.com/hashicorp/terraform/addrs"
"github.com/hashicorp/terraform/internal/getproviders"
"github.com/hashicorp/terraform/tfdiags"
)
@ -12,6 +15,23 @@ import (
// nested block within the CLI configuration.
type ProviderInstallation struct {
Methods []*ProviderInstallationMethod
// DevOverrides allows overriding the normal selection process for
// a particular subset of providers to force using a particular
// local directory and disregard version numbering altogether.
// This is here to allow provider developers to conveniently test
// local builds of their plugins in a development environment, without
// having to fuss with version constraints, dependency lock files, and
// so forth.
//
// This is _not_ intended for "production" use because it bypasses the
// usual version selection and checksum verification mechanisms for
// the providers in question. To make that intent/effect clearer, some
// Terraform commands emit warnings when overrides are present. Local
// mirror directories are a better way to distribute "released"
// providers, because they are still subject to version constraints and
// checksum verification.
DevOverrides map[addrs.Provider]getproviders.PackageLocalDir
}
// decodeProviderInstallationFromConfig uses the HCL AST API directly to
@ -65,6 +85,7 @@ func decodeProviderInstallationFromConfig(hclFile *hclast.File) ([]*ProviderInst
}
pi := &ProviderInstallation{}
devOverrides := make(map[addrs.Provider]getproviders.PackageLocalDir)
body, ok := block.Val.(*hclast.ObjectType)
if !ok {
@ -188,6 +209,53 @@ func decodeProviderInstallationFromConfig(hclFile *hclast.File) ([]*ProviderInst
location = ProviderInstallationNetworkMirror(bodyContent.URL)
include = bodyContent.Include
exclude = bodyContent.Exclude
case "dev_overrides":
if len(pi.Methods) > 0 {
// We require dev_overrides to appear first if it's present,
// because dev_overrides effectively bypass the normal
// selection process for a particular provider altogether,
// and so they don't participate in the usual
// include/exclude arguments and priority ordering.
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Error,
"Invalid provider_installation method block",
fmt.Sprintf("The dev_overrides block at at %s must appear before all other installation methods, because development overrides always have the highest priority.", methodBlock.Pos()),
))
continue
}
// The content of a dev_overrides block is a mapping from
// provider source addresses to local filesystem paths. To get
// our decoding started, we'll use the normal HCL decoder to
// populate a map of strings and then decode further from
// that.
var rawItems map[string]string
err := hcl.DecodeObject(&rawItems, methodBody)
if err != nil {
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Error,
"Invalid provider_installation method block",
fmt.Sprintf("Invalid %s block at %s: %s.", methodTypeStr, block.Pos(), err),
))
continue
}
for rawAddr, rawPath := range rawItems {
addr, moreDiags := addrs.ParseProviderSourceString(rawAddr)
if moreDiags.HasErrors() {
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Error,
"Invalid provider installation dev overrides",
fmt.Sprintf("The entry %q in %s is not a valid provider source string.", rawAddr, block.Pos()),
))
continue
}
dirPath := filepath.Clean(rawPath)
devOverrides[addr] = getproviders.PackageLocalDir(dirPath)
}
continue // We won't add anything to pi.Methods for this one
default:
diags = diags.Append(tfdiags.Sourceless(
tfdiags.Error,
@ -204,6 +272,10 @@ func decodeProviderInstallationFromConfig(hclFile *hclast.File) ([]*ProviderInst
})
}
if len(devOverrides) > 0 {
pi.DevOverrides = devOverrides
}
ret = append(ret, pi)
}

View File

@ -5,6 +5,8 @@ import (
"testing"
"github.com/google/go-cmp/cmp"
"github.com/hashicorp/terraform/addrs"
"github.com/hashicorp/terraform/internal/getproviders"
)
func TestLoadConfig_providerInstallation(t *testing.T) {
@ -36,6 +38,11 @@ func TestLoadConfig_providerInstallation(t *testing.T) {
Exclude: []string{"example.com/*/*"},
},
},
DevOverrides: map[addrs.Provider]getproviders.PackageLocalDir{
addrs.MustParseProviderSourceString("hashicorp/boop"): getproviders.PackageLocalDir(filepath.FromSlash("/tmp/boop")),
addrs.MustParseProviderSourceString("hashicorp/blorp"): getproviders.PackageLocalDir(filepath.FromSlash("/tmp/blorp")),
},
},
},
}

View File

@ -1,4 +1,8 @@
provider_installation {
dev_overrides {
"hashicorp/boop" = "/tmp/bloop/../boop"
"hashicorp/blorp" = "/tmp/blorp"
}
filesystem_mirror {
path = "/tmp/example1"
include = ["example.com/*/*"]

View File

@ -1,5 +1,9 @@
{
"provider_installation": {
"dev_overrides": {
"hashicorp/boop": "/tmp/bloop/../boop",
"hashicorp/blorp": "/tmp/blorp"
},
"filesystem_mirror": [{
"path": "/tmp/example1",
"include": ["example.com/*/*"]

View File

@ -0,0 +1,76 @@
package e2etest
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strings"
"testing"
"github.com/hashicorp/terraform/e2e"
)
// TestProviderDevOverrides is a test for the special dev_overrides setting
// in the provider_installation section of the CLI configuration file, which
// is our current answer to smoothing provider development by allowing
// developers to opt out of the version number and checksum verification
// we normally do, so they can just overwrite the same local executable
// in-place to iterate faster.
func TestProviderDevOverrides(t *testing.T) {
t.Parallel()
tf := e2e.NewBinary(terraformBin, "testdata/provider-dev-override")
defer tf.Close()
// In order to do a decent end-to-end test for this case we will need a
// real enough provider plugin to try to run and make sure we are able
// to actually run it. For now we'll use the "test" provider for that,
// because it happens to be in this repository and therefore allows
// us to avoid drawing in anything external, but we might revisit this
// strategy in future if other needs cause us to evolve the test
// provider in a way that makes it less suitable for this particular test,
// such as if it stops being buildable into an independent executable.
providerExeDir := filepath.Join(tf.WorkDir(), "pkgdir")
providerExePrefix := filepath.Join(providerExeDir, "terraform-provider-test_")
providerExe := e2e.GoBuild("github.com/hashicorp/terraform/builtin/bins/provider-test", providerExePrefix)
t.Logf("temporary provider executable is %s", providerExe)
err := ioutil.WriteFile(filepath.Join(tf.WorkDir(), "dev.tfrc"), []byte(fmt.Sprintf(`
provider_installation {
dev_overrides {
"example.com/test/test" = %q
}
}
`, providerExeDir)), os.ModePerm)
if err != nil {
t.Fatal(err)
}
tf.AddEnv("TF_CLI_CONFIG_FILE=dev.tfrc")
stdout, stderr, err := tf.Run("providers")
if err != nil {
t.Fatalf("unexpected error: %s\n%s", err, stderr)
}
if got, want := stdout, `provider[example.com/test/test]`; !strings.Contains(got, want) {
t.Errorf("configuration should depend on %s, but doesn't\n%s", want, got)
}
// NOTE: We're intentionally not running "terraform init" here, because
// dev overrides are always ready to use and don't need any special action
// to "install" them. This test is mimicking the a happy path of going
// directly from "go build" to validate/plan/apply without interacting
// with any registries, mirrors, lock files, etc.
stdout, stderr, err = tf.Run("validate")
if err != nil {
t.Fatalf("unexpected error: %s\n%s", err, stderr)
}
if got, want := stdout, `The configuration is valid, but`; !strings.Contains(got, want) {
t.Errorf("stdout doesn't include the success message\nwant: %s\n%s", want, got)
}
if got, want := stdout, `Provider development overrides are in effect`; !strings.Contains(got, want) {
t.Errorf("stdout doesn't include the warning about development overrides\nwant: %s\n%s", want, got)
}
}

View File

@ -0,0 +1 @@
This is where the test will place the temporary build of the test provider.

View File

@ -0,0 +1,14 @@
terraform {
required_providers {
test = {
source = "example.com/test/test"
version = "2.0.0"
}
}
}
provider "test" {
}
data "test_data_source" "test" {
}

View File

@ -730,6 +730,12 @@ func (c *InitCommand) getProviders(config *configs.Config, state *states.State,
},
}
// Dev overrides cause the result of "terraform init" to be irrelevant for
// any overridden providers, so we'll warn about it to avoid later
// confusion when Terraform ends up using a different provider than the
// lock file called for.
diags = diags.Append(c.providerDevOverrideWarnings())
mode := providercache.InstallNewProvidersOnly
if upgrade {
mode = providercache.InstallUpgrades

View File

@ -50,10 +50,9 @@ type Meta struct {
// for some reason.
OriginalWorkingDir string
Color bool // True if output should be colored
GlobalPluginDirs []string // Additional paths to search for plugins
PluginOverrides *PluginOverrides // legacy overrides from .terraformrc file
Ui cli.Ui // Ui for output
Color bool // True if output should be colored
GlobalPluginDirs []string // Additional paths to search for plugins
Ui cli.Ui // Ui for output
// ExtraHooks are extra hooks to add to the context.
ExtraHooks []terraform.Hook
@ -104,7 +103,21 @@ type Meta struct {
// When this channel is closed, the command will be cancelled.
ShutdownCh <-chan struct{}
// UnmanagedProviders are a set of providers that exist as processes predating Terraform, which Terraform should use but not worry about the lifecycle of.
// ProviderDevOverrides are providers where we ignore the lock file, the
// configured version constraints, and the local cache directory and just
// always use exactly the path specified. This is intended to allow
// provider developers to easily test local builds without worrying about
// what version number they might eventually be released as, or what
// checksums they have.
ProviderDevOverrides map[addrs.Provider]getproviders.PackageLocalDir
// UnmanagedProviders are a set of providers that exist as processes
// predating Terraform, which Terraform should use but not worry about the
// lifecycle of.
//
// This is essentially a more extreme version of ProviderDevOverrides where
// Terraform doesn't even worry about how the provider server gets launched,
// just trusting that someone else did it before running Terraform.
UnmanagedProviders map[addrs.Provider]*plugin.ReattachConfig
//----------------------------------------------------------
@ -194,11 +207,6 @@ type Meta struct {
allowMissingConfig bool
}
type PluginOverrides struct {
Providers map[string]string
Provisioners map[string]string
}
type testingOverrides struct {
Providers map[addrs.Provider]providers.Factory
Provisioners map[string]provisioners.Factory

View File

@ -2,9 +2,11 @@ package command
import (
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/go-multierror"
@ -16,6 +18,7 @@ import (
"github.com/hashicorp/terraform/internal/providercache"
tfplugin "github.com/hashicorp/terraform/plugin"
"github.com/hashicorp/terraform/providers"
"github.com/hashicorp/terraform/tfdiags"
)
// The TF_DISABLE_PLUGIN_TLS environment variable is intended only for use by
@ -174,6 +177,35 @@ func (m *Meta) providerInstallSource() getproviders.Source {
return m.ProviderSource
}
// providerDevOverrideWarnings returns a diagnostics that contains at least
// one warning if and only if there is at least one provider development
// override in effect. If not, the result is always empty. The result never
// contains error diagnostics.
//
// Certain commands can use this to include a warning that their results
// may differ from what's expected due to the development overrides. It's
// not necessary to bother the user with this warning on every command, but
// it's helpful to return it on commands that have externally-visible side
// effects and on commands that are used to verify conformance to schemas.
func (m *Meta) providerDevOverrideWarnings() tfdiags.Diagnostics {
if len(m.ProviderDevOverrides) == 0 {
return nil
}
var detailMsg strings.Builder
detailMsg.WriteString("The following provider development overrides are set in the CLI configuration:\n")
for addr, path := range m.ProviderDevOverrides {
detailMsg.WriteString(fmt.Sprintf(" - %s in %s\n", addr.ForDisplay(), path))
}
detailMsg.WriteString("\nThe behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.")
return tfdiags.Diagnostics{
tfdiags.Sourceless(
tfdiags.Warning,
"Provider development overrides are in effect",
detailMsg.String(),
),
}
}
// providerFactories uses the selections made previously by an installer in
// the local cache directory (m.providerLocalCacheDir) to produce a map
// from provider addresses to factory functions to create instances of
@ -209,10 +241,23 @@ func (m *Meta) providerFactories() (map[addrs.Provider]providers.Factory, error)
// and they'll just be ignored if not used.
internalFactories := m.internalProviders()
// The Terraform SDK test harness (and possibly other callers in future)
// We have two different special cases aimed at provider development
// use-cases, which are not for "production" use:
// - The CLI config can specify that a particular provider should always
// use a plugin from a particular local directory, ignoring anything the
// lock file or cache directory might have to say about it. This is useful
// for manual testing of local development builds.
// - The Terraform SDK test harness (and possibly other callers in future)
// can ask that we use its own already-started provider servers, which we
// call "unmanaged" because Terraform isn't responsible for starting
// and stopping them.
// and stopping them. This is intended for automated testing where a
// calling harness is responsible both for starting the provider server
// and orchestrating one or more non-interactive Terraform runs that then
// exercise it.
// Unmanaged providers take precedence over overridden providers because
// overrides are typically a "session-level" setting while unmanaged
// providers are typically scoped to a single unattended command.
devOverrideProviders := m.ProviderDevOverrides
unmanagedProviders := m.UnmanagedProviders
factories := make(map[addrs.Provider]providers.Factory, len(providerLocks)+len(internalFactories)+len(unmanagedProviders))
@ -259,6 +304,11 @@ func (m *Meta) providerFactories() (map[addrs.Provider]providers.Factory, error)
}
factories[provider] = providerFactory(cached)
}
for provider, localDir := range devOverrideProviders {
// It's likely that providers in this map will conflict with providers
// in providerLocks
factories[provider] = devOverrideProviderFactory(provider, localDir)
}
for provider, reattach := range unmanagedProviders {
factories[provider] = unmanagedProviderFactory(provider, reattach)
}
@ -318,6 +368,19 @@ func providerFactory(meta *providercache.CachedProvider) providers.Factory {
}
}
func devOverrideProviderFactory(provider addrs.Provider, localDir getproviders.PackageLocalDir) providers.Factory {
// A dev override is essentially a synthetic cache entry for our purposes
// here, so that's how we'll construct it. The providerFactory function
// doesn't actually care about the version, so we can leave it
// unspecified: overridden providers are not explicitly versioned.
log.Printf("[DEBUG] Provider %s is overridden to load from %s", provider, localDir)
return providerFactory(&providercache.CachedProvider{
Provider: provider,
Version: getproviders.UnspecifiedVersion,
PackageDir: string(localDir),
})
}
// unmanagedProviderFactory produces a provider factory that uses the passed
// reattach information to connect to go-plugin processes that are already
// running, and implements providers.Interface against it.

View File

@ -77,6 +77,12 @@ func (c *ValidateCommand) Run(args []string) int {
validateDiags := c.validate(dir)
diags = diags.Append(validateDiags)
// Validating with dev overrides in effect means that the result might
// not be valid for a stable release, so we'll warn about that in case
// the user is trying to use "terraform validate" as a sort of pre-flight
// check before submitting a change.
diags = diags.Append(c.providerDevOverrideWarnings())
return c.showResults(diags, jsonOutput)
}

View File

@ -30,17 +30,19 @@ var PlumbingCommands map[string]struct{}
// Ui is the cli.Ui used for communicating to the outside world.
var Ui cli.Ui
// PluginOverrides is set from wrappedMain during configuration processing
// and then eventually passed to the "command" package to specify alternative
// plugin locations via the legacy configuration file mechanism.
var PluginOverrides command.PluginOverrides
const (
ErrorPrefix = "e:"
OutputPrefix = "o:"
)
func initCommands(originalWorkingDir string, config *cliconfig.Config, services *disco.Disco, providerSrc getproviders.Source, unmanagedProviders map[addrs.Provider]*plugin.ReattachConfig) {
func initCommands(
originalWorkingDir string,
config *cliconfig.Config,
services *disco.Disco,
providerSrc getproviders.Source,
providerDevOverrides map[addrs.Provider]getproviders.PackageLocalDir,
unmanagedProviders map[addrs.Provider]*plugin.ReattachConfig,
) {
var inAutomation bool
if v := os.Getenv(runningInAutomationEnvName); v != "" {
inAutomation = true
@ -68,11 +70,9 @@ func initCommands(originalWorkingDir string, config *cliconfig.Config, services
Color: true,
GlobalPluginDirs: globalPluginDirs(),
PluginOverrides: &PluginOverrides,
Ui: Ui,
Services: services,
ProviderSource: providerSrc,
BrowserLauncher: webbrowser.NewNativeLauncher(),
RunningInAutomation: inAutomation,
@ -80,8 +80,11 @@ func initCommands(originalWorkingDir string, config *cliconfig.Config, services
PluginCacheDir: config.PluginCacheDir,
OverrideDataDir: dataDir,
ShutdownCh: makeShutdownCh(),
UnmanagedProviders: unmanagedProviders,
ShutdownCh: makeShutdownCh(),
ProviderSource: providerSrc,
ProviderDevOverrides: providerDevOverrides,
UnmanagedProviders: unmanagedProviders,
}
// The command list is included in the terraform -help

View File

@ -254,7 +254,8 @@ func (b *binary) Close() {
}
func GoBuild(pkgPath, tmpPrefix string) string {
tmpFile, err := ioutil.TempFile("", tmpPrefix)
dir, prefix := filepath.Split(tmpPrefix)
tmpFile, err := ioutil.TempFile(dir, prefix)
if err != nil {
panic(err)
}

View File

@ -195,6 +195,7 @@ func wrappedMain() int {
// We continue to run anyway, because most commands don't do provider installation.
}
}
providerDevOverrides := providerDevOverrides(config.ProviderInstallation)
// The user can declare that certain providers are being managed on
// Terraform's behalf using this environment variable. Thsi is used
@ -241,7 +242,7 @@ func wrappedMain() int {
// in case they need to refer back to it for any special reason, though
// they should primarily be working with the override working directory
// that we've now switched to above.
initCommands(originalWd, config, services, providerSrc, unmanagedProviders)
initCommands(originalWd, config, services, providerSrc, providerDevOverrides, unmanagedProviders)
}
// Run checkpoint
@ -300,10 +301,6 @@ func wrappedMain() int {
AutocompleteUninstall: "uninstall-autocomplete",
}
// Pass in the overriding plugin paths from config
PluginOverrides.Providers = config.Providers
PluginOverrides.Provisioners = config.Provisioners
exitCode, err := cliRunner.Run()
if err != nil {
Ui.Error(fmt.Sprintf("Error executing CLI: %s", err.Error()))

View File

@ -225,3 +225,14 @@ func providerSourceForCLIConfigLocation(loc cliconfig.ProviderInstallationLocati
panic(fmt.Sprintf("unexpected provider source location type %T", loc))
}
}
func providerDevOverrides(configs []*cliconfig.ProviderInstallation) map[addrs.Provider]getproviders.PackageLocalDir {
if len(configs) == 0 {
return nil
}
// There should only be zero or one configurations, which is checked by
// the validation logic in the cliconfig package. Therefore we'll just
// ignore any additional configurations in here.
return configs[0].DevOverrides
}

View File

@ -352,6 +352,89 @@ Terraform will never itself delete a plugin from the plugin cache once it has
been placed there. Over time, as plugins are upgraded, the cache directory may
grow to contain several unused versions which you must delete manually.
### Development Overrides for Provider Developers
-> **Note:** Development overrides work only in Terraform v0.14 and later.
Using a `dev_overrides` block in your CLI configuration will cause Terraform
v0.13 to reject the configuration as invalid.
Normally Terraform verifies version selections and checksums for providers
in order to help ensure that all operations are made with the intended version
of a provider, and that authors can gradually upgrade to newer provider versions
in a controlled manner.
These version and checksum rules are inconvenient when developing a provider
though, because we often want to try a test configuration against a development
build of a provider that doesn't even have an associated version number yet,
and doesn't have an official set of checksums listed in a provider registry.
As a convenience for provider development, Terraform supports a special
additional block `dev_overrides` in `provider_installation` blocks. The contents
of this block effectively override all of the other configured installation
methods, so a block of this type must always appear first in the sequence:
```hcl
provider_installation {
# Use /home/developer/tmp/terraform-null as an overridden package directory
# for the hashicorp/null provider. This disables the version and checksum
# verifications for this provider and forces Terraform to look for the
# null provider plugin in the given directory.
dev_overrides {
"hashicorp/null" = "/home/developer/tmp/terraform-null"
}
# For all other providers, install them directly from their origin provider
# registries as normal. If you omit this, Terraform will _only_ use
# the dev_overrides block, and so no other providers will be available.
direct {}
}
```
With development overrides in effect, the `terraform init` command will still
attempt to select a suitable published version of your provider to install and
record in
[the dependency lock file](/docs/configuration/dependency-lock.html)
for future use, but other commands like
`terraform apply` will disregard the lock file's entry for `hashicorp/null` and
will use the given directory instead. Once your new changes are included in a
published release of the provider, you can use `terraform init -upgrade` to
select the new version in the dependency lock file and remove your development
override.
The override path for a particular provider should be a directory similar to
what would be included in a `.zip` file when distributing the provider. At
minimum that includes an executable file named with a prefix like
`terraform-provider-null`, where `null` is the provider type. If your provider
makes use of other files in its distribution package then you can copy those
files into the override directory too.
You may wish to enable a development override only for shell sessions where
you are actively working on provider development. If so, you can write a
local CLI configuration file with content like the above in your development
directory, perhaps called `dev.tfrc` for the sake fo example, and then use the
`TF_CLI_CONFIG_FILE` environment variable to instruct Terraform to use that
localized CLI configuration instead of the default one:
```
export TF_CLI_CONFIG_FILE=/home/developer/tmp/dev.tfrc
```
Development overrides are not intended for general use as a way to have
Terraform look for providers on the local filesystem. If you wish to put
copies of _released_ providers in your local filesystem, see
[Implied Local Mirror Directories](#implied-local-mirror-directories)
or
[Explicit Installation Method Configuration](#explicit-installation-method-configuration)
instead.
This development overrides mechanism is intended as a pragmatic way to enable
smoother provider development. The details of how it behaves, how to
configure it, and how it interacts with the dependency lock file may all evolve
in future Terraform releases, including possible breaking changes. We therefore
recommend using development overrides only temporarily during provider
development work.
## Removed Settings
The following settings are supported in Terraform 0.12 and earlier but are