website: Separate docs for the module registry protocol

We previously had the module registry protocol documented only as an
undefined subset of the full API of the official registry implementation.
However, the vast majority of endpoints documented in the official API
docs are not needed for a headless third-party module registry that only
intends to make modules available to Terraform CLI.

To make this clearer to potential third-party implementors, and also for
consistency with how the provider registry protocol is now documented,
here we create a new page to describe the subset required for all
registries, and then explain in the docs for the offical API that
potential third-party implementors should refer to the new page instead.

The longer page describing the full API of the official implementations
remains for those who wish to write clients for that API, because it is
part of the API surface area for Terraform Cloud and Terraform Enterprise.

I also took this opportunity to address the fact that module addresses
don't really contain "provider names" at all, but rather than the fourth
field in the address is _conventionally_ an official provider name but
can really be any string that serves to differentiate multiple
implementations of the same abstraction. The new docs therefore refer to
this field as "system" rather than "provider".

website/docs/internals/module-registry-protocol.html.md

@@ -0,0 +1,223 @@
+---
+layout: "docs"
+page_title: "Module Registry Protocol"
+sidebar_current: "docs-internals-modules-protocol"
+description: |-
+  The module registry protocol is implemented by a host intending to be the
+  host of one or more Terraform modules, specifying which modules are available
+  and where to find their distribution packages.
+---
+
+# Module Registry Protocol
+
+-> Third-party provider registries are supported only in Terraform CLI 0.11 and later. Prior versions do not support this protocol.
+
+The module registry protocol is what Terraform CLI uses to discover metadata
+about modules available for installation and to locate the distribution
+package for a selected module.
+
+The primary implementation of this protocol is the public
+[Terraform Registry](https://registry.terraform.io/) at `registry.terraform.io`.
+By writing and deploying your own implementation of this protocol, you can
+create a separate registry to distribute your own modules, as an alternative to
+publishing them on the public Terraform Registry.
+
+The public Terraform Registry implements a superset of the API described on
+this page, in order to capture additional information used in the registry UI.
+For information on those extensions, see
+[Terraform Registry HTTP API](/docs/registry/api.html). Third-party registry
+implementations may choose to implement those extensions if desired, but
+Terraform CLI itself does not use them.
+
+## Module Addresses
+
+Each Terraform module has an associated address. A module address has the
+syntax `hostname/namespace/name/system`, where:
+
+* `hostname` is the hostname of the provider registry that serves this module.
+* `namespace` is the name of a namespace, unique on a particular hostname, that
+  can contain one or more modules that are somehow related. On the public
+  Terraform Registry the "namespace" represents the organization that is
+  packaging and distributing the module.
+* `name` is the module name, which generally names the abstraction that the
+  module is intending to create.
+* `system` is the name of a system that the module is primarily written to
+  target. For multi-cloud abstractions, there can be multiple modules with
+  addresses that differ only in "system" to reflect system-specific
+  implementations of the abstraction, like
+  `registry.terraform.io/hashicorp/consul/aws` vs.
+  `registry.terraform.io/hashicorp/consul/azurerm`. The system name commonly
+  matches the type portion of the address of an official provider, but that
+  is not required.
+
+The `hostname/` portion of a provider address (including its slash delimiter)
+is optional, and if omitted defaults to `registry.terraform.io/`.
+
+For example:
+
+* `hashicorp/consul/aws` is a shorthand for
+  `registry.terraform.io/hashicorp/consul/aws`, which is a module on the
+  public registry for deploying Consul clusters in Amazon Web Services.
+* `example.com/awesomecorp/consul/happycloud` is a hypothetical module published
+  on a third-party registry.
+
+If you intend only to share a module you've developed for use by all
+Terraform users, please consider publishing it into the public
+[Terraform Registry](https://registry.terraform.io/), which will make your
+module discoverable. You only need to implement this module registry
+protocol if you wish to publish modules whose addresses include a different
+hostname that is under your control.
+
+## Module Versions
+
+Each distinct module address has associated with it a set of versions, each
+of which has an associated version number. Terraform assumes version numbers
+follow the [Semantic Versioning 2.0](https://semver.org/) conventions, with
+the user-facing behavior of the module serving as the "public API".
+
+Each `module` block may select a distinct version of a module, even if multiple
+blocks have the same source address.
+
+## Service Discovery
+
+The providers protocol begins with Terraform CLI using
+[./remote-service-discovery.html](Terraform's remote service discovery protocol),
+with the hostname in the module address acting as the "User-facing Hostname".
+
+The service identifier for the module registry protocol is `modules.v1`.
+Its associated string value is the base URL for the relative URLs defined in
+the sections that follow.
+
+For example, the service discovery document for a host that _only_ implements
+the module registry protocol might contain the following:
+
+```json
+{
+  "modules.v1": "/terraform/modules/v1/"
+}
+```
+
+If the given URL is a relative URL then Terraform will interpret it as relative
+to the discovery document itself. The specific module registry protocol
+endpoints are defined as URLs relative to the given base URL, and so the
+specified base URL should generally end with a slash to ensure that those
+relative paths will be resolved as expected.
+
+The following sections describe the various operations that a module
+registry must implement to be compatible with Terraform CLI's module
+installer. The indicated URLs are all relative to the URL resulting from
+service discovery, as described above. We use the current URLs on
+Terraform Registry as working examples, assuming that the caller already
+performed service discovery on `registry.terraform.io` to learn the base URL.
+
+The URLs are shown with the convention that a path portion with a colon `:`
+prefix is a placeholder for a dynamically-selected value, while all other
+path portions are literal. For example, in `:namespace/:type/versions`,
+the first two path portions are placeholders while the third is literally
+the string "versions".
+
+## List Available Versions for a Specific Module
+
+This is the primary endpoint for resolving module sources, returning the
+available versions for a given fully-qualified module.
+
+| Method | Path                                  | Produces                   |
+| ------ | ------------------------------------- | -------------------------- |
+| `GET`  | `:namespace/:name/:provider/versions` | `application/json`         |
+
+### Parameters
+
+- `namespace` `(string: <required>)` - The user or organization the module is
+  owned by. This is required and is specified as part of the URL path.
+
+- `name` `(string: <required>)` - The name of the module.
+  This is required and is specified as part of the URL path.
+
+- `system` `(string: <required>)` - The name of the target system.
+  This is required and is specified as part of the URL path.
+
+### Sample Request
+
+```text
+$ curl 'https://registry.terraform.io/v1/modules/hashicorp/consul/aws/versions'
+```
+
+### Sample Response
+
+The `modules` array in the response always includes the requested module as the
+first element.
+
+Other elements of this list are not currently used. Third-party implementations
+should always use a single-element list for forward compatiblity with possible
+future extensions to the protocol.
+
+Each returned module has an array of available versions, which Terraform
+matches against any version constraints given in configuration.
+
+```json
+{
+   "modules": [
+      {
+         "versions": [
+            {"version": "1.0.0"},
+            {"version": "1.1.0"},
+            {"version": "2.0.0"}
+         ]
+      }
+   ]
+}
+```
+
+Return `404 Not Found` to indicate that no module is available with the
+requested namespace, name, and provider
+
+## Download Source Code for a Specific Module Version
+
+This endpoint downloads the specified version of a module for a single provider.
+
+| Method | Path                                                   | Produces                   |
+| ------ | ------------------------------------------------------ | -------------------------- |
+| `GET`  | `:namespace/:name/:provider/:system/:version/download` | `application/json`         |
+
+### Parameters
+
+- `namespace` `(string: <required>)` - The user the module is owned by.
+  This is required and is specified as part of the URL path.
+
+- `name` `(string: <required>)` - The name of the module.
+  This is required and is specified as part of the URL path.
+
+- `provider` `(string: <required>)` - The name of the provider.
+  This is required and is specified as part of the URL path.
+
+- `system` `(string: <required>)` - The name of the target system.
+  This is required and is specified as part of the URL path.
+
+- `version` `(string: <required>)` - The version of the module.
+  This is required and is specified as part of the URL path.
+
+### Sample Request
+
+```text
+$ curl -i 'https://registry.terraform.io/v1/modules/hashicorp/consul/aws/0.0.1/download'
+```
+
+### Sample Response
+
+```text
+HTTP/1.1 204 No Content
+Content-Length: 0
+X-Terraform-Get: https://api.github.com/repos/hashicorp/terraform-aws-consul/tarball/v0.0.1//*?archive=tar.gz
+```
+
+A successful response has no body, and includes the location from which the
+module version's source can be downloaded in the `X-Terraform-Get` header.
+The value of this header accepts the same values as the `source` argument
+in a `module` block in Terraform configuration, as described in
+[Module Sources](https://www.terraform.io/docs/modules/sources.html),
+except that it may not recursively refer to another module registry address.
+
+The value of `X-Terraform-Get` may instead be a relative URL, indicated by
+beginning with `/`, `./` or `../`, in which case it is resolved relative to
+the full URL of the download endpoint to produce
+[an HTTP URL module source](/docs/modules/sources.html#http-urls).

website/docs/internals/remote-service-discovery.html.md

@@ -86,7 +86,7 @@ version 1 of the module registry protocol:
 At present, the following service identifiers are in use:
 
 * `login.v1`: [login protocol version 1](/docs/commands/login.html#protocol-v1)
-* `modules.v1`: [module registry API version 1](/docs/registry/api.html)
+* `modules.v1`: [module registry API version 1](module-registry-protocol.html)
 * `providers.v1`: [provider registry API version 1](provider-registry-protocol.html)
 
 ## Authentication

website/docs/registry/api.html.md

@@ -9,21 +9,33 @@ description: |-
 # HTTP API
 
 When downloading modules from registry sources such as the public
-[Terraform Registry](https://registry.terraform.io), Terraform expects
+[Terraform Registry](https://registry.terraform.io/), Terraform CLI expects
-the given hostname to support the following module registry protocol.
+the given hostname to support
+[the module registry protocol](/docs/internals/module-registry-protocol.html),
+which is the minimal API required for Terraform CLI to successfully retrieve
+a module.
 
-A registry module source is of the form `hostname/namespace/name/provider`,
+The public Terraform Registry and the private registry included in Terraform
-where the initial hostname portion is implied to be `registry.terraform.io/`
+Cloud and Terraform Enterprise implement a superset of that minimal module
-if not specified. The public Terraform Registry is therefore the default
+registry API to support additional use-cases such as searching for modules
-module source.
+across the whole registry, retrieving documentation and schemas for modules,
+and so on.
 
-[Terraform Registry](https://registry.terraform.io) implements a superset
+This page describes the extended API implemented by the official module
-of this API to allow for importing new modules, etc, but any endpoints not
+registry implementations, and is aimed at those intending to build clients
-documented on this page are subject to change over time.
+to work with registry data. Third-party implementations of the registry
+protocol are not required to implement these extensions. If you intend to
+implement your own module registry, please refer to
+[the module registry protocol](/docs/internals/module-registry-protocol.html)
+instead.
+
+Terraform Registry also has some additional internal API endpoints used to
+support its UI. Any endpoints or properties not documented on this page are
+subject to change over time.
 
 ## Service Discovery
 
-The hostname portion of a module source string is first passed to
+The hostname portion of a module source address is first passed to
 [the service discovery protocol](/docs/internals/remote-service-discovery.html)
 to determine if the given host has a module registry and, if so, the base
 URL for its module registry endpoints.
@@ -36,6 +48,10 @@ For example, if discovery produces the URL `https://modules.example.com/v1/`
 then this API would use full endpoint URLs like
 `https://modules.example.com/v1/{namespace}/{name}/{provider}/versions`.
 
+A module source address with no hostname is a shorthand for an address
+on `registry.terraform.io`. You can perform service discovery on that hostname
+to find the public Terraform Registry's module API endpoints.
+
 ## Base URL
 
 The example request URLs shown in this document are for the public [Terraform