8209b40526
The original contents of `vendor` were inadvertently captured with an older version of `godep`. Here, we recapture dependencies by running the following: ``` godep restore -v cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath' | xargs godep update -v ``` The newer godep makes the following changes as it captures dependencies: * Skips test files * Copies `LICENSE` / `PATENTS` files There is also an additional diff in `golang.org/x/sys/unix` that looks very similar to the diff between `master..c65f27f` in that repo, so I'm guessing that dependency was accidentally captured from master instead of the commit saved to `Godeps.json`. All in all, these changes should all be "more correct" and result in smaller diffs for any future updates made to dependencies. |
||
---|---|---|
.. | ||
helper/url | ||
test-fixtures | ||
.travis.yml | ||
LICENSE | ||
README.md | ||
client.go | ||
copy_dir.go | ||
decompress.go | ||
decompress_bzip2.go | ||
decompress_gzip.go | ||
decompress_tbz2.go | ||
decompress_testing.go | ||
decompress_tgz.go | ||
decompress_zip.go | ||
detect.go | ||
detect_bitbucket.go | ||
detect_file.go | ||
detect_github.go | ||
detect_s3.go | ||
folder_storage.go | ||
get.go | ||
get_file.go | ||
get_git.go | ||
get_hg.go | ||
get_http.go | ||
get_mock.go | ||
get_s3.go | ||
source.go | ||
storage.go |
README.md
go-getter
go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.
The power of this library is being flexible in being able to download from a number of different sources (file paths, Git, HTTP, Mercurial, etc.) using a single string as input. This removes the burden of knowing how to download from a variety of sources from the implementer.
The concept of a detector automatically turns invalid URLs into proper URLs. For example: "github.com/hashicorp/go-getter" would turn into a Git URL. Or "./foo" would turn into a file URL. These are extensible.
This library is used by Terraform for downloading modules, Otto for dependencies and Appfile imports, and Nomad for downloading binaries.
Installation and Usage
Package documentation can be found on GoDoc.
Installation can be done with a normal go get
:
$ go get github.com/hashicorp/go-getter
URL Format
go-getter uses a single string URL as input to downlaod from a variety of protocols. go-getter has various "tricks" with this URL to do certain things. This section documents the URL format.
Supported Protocols and Detectors
Protocols are used to download files/directories using a specific mechanism. Example protocols are Git and HTTP.
Detectors are used to transform a valid or invalid URL into another URL if it matches a certain pattern. Example: "github.com/user/repo" is automatically transformed into a fully valid Git URL. This allows go-getter to be very user friendly.
go-getter out of the box supports the following protocols. Additional protocols
can be augmented at runtime by implementing the Getter
interface.
- Local files
- Git
- Mercurial
- HTTP
- Amazon S3
In addition to the above protocols, go-getter has what are called "detectors." These take a URL and attempt to automatically choose the best protocol for it, which might involve even changing the protocol. The following detection is built-in by default:
- File paths such as "./foo" are automatically changed to absolute file URLs.
- GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically changed to Git protocol over HTTP.
- BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically changed to a Git or mercurial protocol using the BitBucket API.
Forced Protocol
In some cases, the protocol to use is ambiguous depending on the source URL. For example, "http://github.com/mitchellh/vagrant.git" could reference an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this URL.
Forced protocol can be done by prefixing the URL with the protocol followed
by double colons. For example: git::http://github.com/mitchellh/vagrant.git
would download the given HTTP URL using the Git protocol.
Forced protocols will also override any detectors.
In the absense of a forced protocol, detectors may be run on the URL, transforming the protocol anyways. The above example would've used the Git protocol either way since the Git detector would've detected it was a GitHub URL.
Checksumming
For file downloads of any protocol, go-getter can automatically verify a checksum for you. Note that checksumming only works for downloading files, not directories, but checksumming will work for any protocol.
To checksum a file, append a checksum
query parameter to the URL.
The paramter value should be in the format of type:value
, where
type is "md5", "sha1", "sha256", or "sha512". The "value" should be
the actual checksum value. go-getter will parse out this query parameter
automatically and use it to verify the checksum. An example URL
is shown below:
./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21
The checksum query parameter is never sent to the backend protocol implementation. It is used at a higher level by go-getter itself.
Unarchiving
go-getter will automatically unarchive files into a file or directory based on the extension of the file being requested (over any protocol). This works for both file and directory downloads.
go-getter looks for an archive
query parameter to specify the format of
the archive. If this isn't specified, go-getter will use the extension of
the path to see if it appears archived. Unarchiving can be explicitly
disabled by setting the archive
query parameter to false
.
The following archive formats are supported:
tar.gz
andtgz
tar.bz2
andtbz2
zip
gz
bz2
For example, an example URL is shown below:
./foo.zip
This will automatically be inferred to be a ZIP file and will be extracted. You can also be explicit about the archive type:
./some/other/path?archive=zip
And finally, you can disable archiving completely:
./some/path?archive=false
You can combine unarchiving with the other features of go-getter such
as checksumming. The special archive
query parameter will be removed
from the URL before going to the final protocol downloader.