2017-06-26 23:46:34 +02:00
|
|
|
// Copyright 2016-2017, Pulumi Corporation. All rights reserved.
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
|
|
|
|
package resource
|
|
|
|
|
2017-04-17 22:34:19 +02:00
|
|
|
import (
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
"archive/tar"
|
|
|
|
"archive/zip"
|
2017-04-17 22:34:19 +02:00
|
|
|
"bytes"
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
"compress/gzip"
|
2017-10-22 22:39:21 +02:00
|
|
|
"crypto/sha256"
|
|
|
|
"encoding/hex"
|
2017-04-17 22:34:19 +02:00
|
|
|
"io"
|
|
|
|
"io/ioutil"
|
|
|
|
"net/http"
|
|
|
|
"net/url"
|
|
|
|
"os"
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
"path/filepath"
|
2017-10-22 22:39:21 +02:00
|
|
|
"reflect"
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
"sort"
|
|
|
|
"strings"
|
2017-04-17 22:34:19 +02:00
|
|
|
|
2017-04-19 23:46:50 +02:00
|
|
|
"github.com/pkg/errors"
|
|
|
|
|
2017-09-22 04:18:21 +02:00
|
|
|
"github.com/pulumi/pulumi/pkg/util/contract"
|
2017-10-22 22:39:21 +02:00
|
|
|
"github.com/pulumi/pulumi/pkg/workspace"
|
2017-04-17 22:34:19 +02:00
|
|
|
)
|
|
|
|
|
|
|
|
// Asset is a serialized asset reference. It is a union: thus, only one of its fields will be non-nil. Several helper
|
|
|
|
// routines exist as members in order to easily interact with the assets referenced by an instance of this type.
|
2017-10-20 22:21:04 +02:00
|
|
|
//nolint: lll
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
type Asset struct {
|
2017-10-20 22:21:04 +02:00
|
|
|
Sig string `json:"4dabf18193072939515e22adb298388d" yaml:"4dabf18193072939515e22adb298388d"` // the unique asset signature (see properties.go).
|
2017-10-23 03:30:42 +02:00
|
|
|
Hash string `json:"hash,omitempty" yaml:"hash,omitempty"` // the SHA256 hash of the asset contents.
|
2017-10-20 22:21:04 +02:00
|
|
|
Text string `json:"text,omitempty" yaml:"text,omitempty"` // a textual asset.
|
|
|
|
Path string `json:"path,omitempty" yaml:"path,omitempty"` // a file on the current filesystem.
|
|
|
|
URI string `json:"uri,omitempty" yaml:"uri,omitempty"` // a URI (file://, http://, https://, or custom).
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
}
|
|
|
|
|
2017-07-14 21:28:43 +02:00
|
|
|
const (
|
2017-07-17 19:38:57 +02:00
|
|
|
AssetSig = "c44067f5952c0a294b673a41bacd8c17" // a randomly assigned type hash for assets.
|
2017-10-22 22:39:21 +02:00
|
|
|
AssetHashProperty = "hash" // the dynamic property for an asset's hash.
|
2017-07-17 19:38:57 +02:00
|
|
|
AssetTextProperty = "text" // the dynamic property for an asset's text.
|
|
|
|
AssetPathProperty = "path" // the dynamic property for an asset's path.
|
|
|
|
AssetURIProperty = "uri" // the dynamic property for an asset's URI.
|
2017-07-14 21:28:43 +02:00
|
|
|
)
|
|
|
|
|
2017-10-23 03:30:42 +02:00
|
|
|
// NewTextAsset produces a new asset and its corresponding SHA256 hash from the given text.
|
2017-10-22 22:39:21 +02:00
|
|
|
func NewTextAsset(text string) (*Asset, error) {
|
|
|
|
a := &Asset{Sig: AssetSig, Text: text}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
|
|
|
|
2017-10-23 03:30:42 +02:00
|
|
|
// NewPathAsset produces a new asset and its corresponding SHA256 hash from the given filesystem path.
|
2017-10-22 22:39:21 +02:00
|
|
|
func NewPathAsset(path string) (*Asset, error) {
|
|
|
|
a := &Asset{Sig: AssetSig, Path: path}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
2017-05-29 22:08:30 +02:00
|
|
|
|
2017-10-23 03:30:42 +02:00
|
|
|
// NewURIAsset produces a new asset and its corresponding SHA256 hash from the given network URI.
|
2017-10-22 22:39:21 +02:00
|
|
|
func NewURIAsset(uri string) (*Asset, error) {
|
|
|
|
a := &Asset{Sig: AssetSig, URI: uri}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) IsText() bool { return a.Text != "" }
|
|
|
|
func (a *Asset) IsPath() bool { return a.Path != "" }
|
|
|
|
func (a *Asset) IsURI() bool { return a.URI != "" }
|
|
|
|
|
|
|
|
func (a *Asset) GetText() (string, bool) {
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
if a.IsText() {
|
2017-07-17 19:38:57 +02:00
|
|
|
return a.Text, true
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
}
|
|
|
|
return "", false
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) GetPath() (string, bool) {
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
if a.IsPath() {
|
2017-07-17 19:38:57 +02:00
|
|
|
return a.Path, true
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
}
|
|
|
|
return "", false
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) GetURI() (string, bool) {
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
if a.IsURI() {
|
2017-07-17 19:38:57 +02:00
|
|
|
return a.URI, true
|
Introduce assets
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
2017-04-17 22:00:26 +02:00
|
|
|
}
|
|
|
|
return "", false
|
|
|
|
}
|
2017-04-17 22:34:19 +02:00
|
|
|
|
2017-04-18 20:02:04 +02:00
|
|
|
// GetURIURL returns the underlying URI as a parsed URL, provided it is one. If there was an error parsing the URI, it
|
|
|
|
// will be returned as a non-nil error object.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) GetURIURL() (*url.URL, bool, error) {
|
2017-04-18 20:02:04 +02:00
|
|
|
if uri, isuri := a.GetURI(); isuri {
|
|
|
|
url, err := url.Parse(uri)
|
|
|
|
if err != nil {
|
|
|
|
return nil, true, err
|
|
|
|
}
|
|
|
|
return url, true, nil
|
|
|
|
}
|
|
|
|
return nil, false, nil
|
|
|
|
}
|
|
|
|
|
2017-07-17 21:11:15 +02:00
|
|
|
// Equals returns true if a is value-equal to other.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) Equals(other *Asset) bool {
|
|
|
|
if a == nil {
|
|
|
|
return other == nil
|
|
|
|
} else if other == nil {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
return a.Hash == other.Hash && a.Text == other.Text && a.Path == other.Path && a.URI == other.URI
|
2017-07-17 21:11:15 +02:00
|
|
|
}
|
|
|
|
|
2017-07-17 19:38:57 +02:00
|
|
|
// Serialize returns a weakly typed map that contains the right signature for serialization purposes.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) Serialize() map[string]interface{} {
|
|
|
|
result := map[string]interface{}{
|
|
|
|
string(SigKey): AssetSig,
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
if a.Hash != "" {
|
|
|
|
result[AssetHashProperty] = a.Hash
|
|
|
|
}
|
|
|
|
if a.Text != "" {
|
|
|
|
result[AssetTextProperty] = a.Text
|
|
|
|
}
|
|
|
|
if a.Path != "" {
|
|
|
|
result[AssetPathProperty] = a.Path
|
|
|
|
}
|
|
|
|
if a.URI != "" {
|
|
|
|
result[AssetURIProperty] = a.URI
|
|
|
|
}
|
|
|
|
return result
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// DeserializeAsset checks to see if the map contains an asset, using its signature, and if so deserializes it.
|
2017-10-23 00:54:44 +02:00
|
|
|
func DeserializeAsset(obj map[string]interface{}) (*Asset, bool, error) {
|
2017-10-22 22:39:21 +02:00
|
|
|
// If not an asset, return false immediately.
|
2017-07-17 19:38:57 +02:00
|
|
|
if obj[string(SigKey)] != AssetSig {
|
2017-10-22 22:39:21 +02:00
|
|
|
return &Asset{}, false, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// Else, deserialize the possible fields.
|
|
|
|
var hash string
|
|
|
|
if v, has := obj[AssetHashProperty]; has {
|
|
|
|
hash = v.(string)
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
var text string
|
|
|
|
if v, has := obj[AssetTextProperty]; has {
|
|
|
|
text = v.(string)
|
|
|
|
}
|
|
|
|
var path string
|
|
|
|
if v, has := obj[AssetPathProperty]; has {
|
|
|
|
path = v.(string)
|
|
|
|
}
|
|
|
|
var uri string
|
|
|
|
if v, has := obj[AssetURIProperty]; has {
|
|
|
|
uri = v.(string)
|
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
if text == "" && path == "" && uri == "" {
|
|
|
|
return &Asset{}, false, errors.New("asset is missing one of text, path, or URI")
|
|
|
|
}
|
|
|
|
|
|
|
|
return &Asset{Hash: hash, Text: text, Path: path, URI: uri}, true, nil
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// Read reads an asset's contents into memory.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) Read() (*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if a.IsText() {
|
|
|
|
return a.readText()
|
|
|
|
} else if a.IsPath() {
|
|
|
|
return a.readPath()
|
|
|
|
} else if a.IsURI() {
|
|
|
|
return a.readURI()
|
|
|
|
}
|
|
|
|
contract.Failf("Invalid asset; one of Text, Path, or URI must be non-nil")
|
|
|
|
return nil, nil
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) readText() (*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
text, istext := a.GetText()
|
|
|
|
contract.Assertf(istext, "Expected a text-based asset")
|
|
|
|
return NewByteBlob([]byte(text)), nil
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) readPath() (*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
path, ispath := a.GetPath()
|
|
|
|
contract.Assertf(ispath, "Expected a path-based asset")
|
2017-10-22 22:39:21 +02:00
|
|
|
|
|
|
|
// Do a quick check to make sure it's a file, so we can fail gracefully if someone passes a directory.
|
|
|
|
info, err := os.Stat(path)
|
|
|
|
if err != nil {
|
|
|
|
return nil, errors.Wrapf(err, "failed to open asset file '%v'", path)
|
|
|
|
} else if info.IsDir() {
|
|
|
|
return nil, errors.Errorf("asset path '%v' is a directory; try using an archive", path)
|
|
|
|
}
|
|
|
|
|
2017-08-29 22:33:02 +02:00
|
|
|
byts, err := ioutil.ReadFile(path)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if err != nil {
|
2017-10-22 22:39:21 +02:00
|
|
|
return nil, errors.Wrapf(err, "failed to read asset file '%v'", path)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
2017-08-29 22:33:02 +02:00
|
|
|
return NewByteBlob(byts), nil
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) readURI() (*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
url, isurl, err := a.GetURIURL()
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
contract.Assertf(isurl, "Expected a URI-based asset")
|
|
|
|
switch s := url.Scheme; s {
|
|
|
|
case "http", "https":
|
2017-05-14 02:04:35 +02:00
|
|
|
resp, err := http.Get(url.String())
|
|
|
|
if err != nil {
|
2017-04-17 22:34:19 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
return NewReadCloserBlob(resp.Body)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
case "file":
|
|
|
|
contract.Assert(url.User == nil)
|
|
|
|
contract.Assert(url.RawQuery == "")
|
|
|
|
contract.Assert(url.Fragment == "")
|
2017-10-22 22:39:21 +02:00
|
|
|
if url.Host != "" && url.Host != "localhost" {
|
|
|
|
return nil, errors.Errorf("file:// host '%v' not supported (only localhost)", url.Host)
|
|
|
|
}
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
f, err := os.Open(url.Path)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
2017-04-17 22:34:19 +02:00
|
|
|
}
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return NewFileBlob(f)
|
|
|
|
default:
|
|
|
|
return nil, errors.Errorf("Unrecognized or unsupported URI scheme: %v", s)
|
2017-04-17 22:34:19 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-23 03:30:42 +02:00
|
|
|
// EnsureHash computes the SHA256 hash of the asset's contents and stores it on the object.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Asset) EnsureHash() error {
|
|
|
|
if a.Hash == "" {
|
|
|
|
blob, err := a.Read()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
defer contract.IgnoreClose(blob)
|
|
|
|
|
|
|
|
hash := sha256.New()
|
|
|
|
_, err = io.Copy(hash, blob)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
a.Hash = hex.EncodeToString(hash.Sum(nil))
|
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// SeekableReadCloser combines Read, Close, and Seek functionality into one interface.
|
|
|
|
type SeekableReadCloser interface {
|
2017-04-17 22:34:19 +02:00
|
|
|
io.Seeker
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
io.ReadCloser
|
2017-04-17 22:34:19 +02:00
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// Blob is a blob that implements ReadCloser, Seek, and offers Len functionality.
|
|
|
|
type Blob struct {
|
|
|
|
rd SeekableReadCloser // an underlying reader.
|
|
|
|
sz int64 // the size of the blob.
|
|
|
|
}
|
|
|
|
|
2017-05-22 06:45:28 +02:00
|
|
|
func (blob *Blob) Close() error { return blob.rd.Close() }
|
2017-05-04 18:53:52 +02:00
|
|
|
func (blob *Blob) Read(p []byte) (int, error) { return blob.rd.Read(p) }
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
func (blob *Blob) Reader() SeekableReadCloser { return blob.rd }
|
2017-05-04 18:53:52 +02:00
|
|
|
func (blob *Blob) Seek(offset int64, whence int) (int64, error) { return blob.rd.Seek(offset, whence) }
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
func (blob *Blob) Size() int64 { return blob.sz }
|
|
|
|
|
|
|
|
// NewByteBlob creates a new byte blob.
|
|
|
|
func NewByteBlob(data []byte) *Blob {
|
|
|
|
return &Blob{
|
|
|
|
rd: newBytesReader(data),
|
|
|
|
sz: int64(len(data)),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// NewFileBlob creates a new asset blob whose size is known thanks to stat.
|
|
|
|
func NewFileBlob(f *os.File) (*Blob, error) {
|
2017-05-14 02:04:35 +02:00
|
|
|
stat, err := f.Stat()
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
return &Blob{
|
|
|
|
rd: f,
|
|
|
|
sz: stat.Size(),
|
|
|
|
}, nil
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// NewReadCloserBlob turn any old ReadCloser into an Blob, usually by making a copy.
|
|
|
|
func NewReadCloserBlob(r io.ReadCloser) (*Blob, error) {
|
|
|
|
if f, isf := r.(*os.File); isf {
|
|
|
|
// If it's a file, we can "fast path" the asset creation without making a copy.
|
|
|
|
return NewFileBlob(f)
|
|
|
|
}
|
|
|
|
// Otherwise, read it all in, and create a blob out of that.
|
Tidy up more lint
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 21:09:46 +02:00
|
|
|
defer contract.IgnoreClose(r)
|
2017-05-14 02:04:35 +02:00
|
|
|
data, err := ioutil.ReadAll(r)
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
return NewByteBlob(data), nil
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// bytesReader turns a *bytes.Reader into a SeekableReadCloser by adding an empty Close method.
|
2017-04-17 22:34:19 +02:00
|
|
|
type bytesReader struct {
|
|
|
|
*bytes.Reader
|
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
func newBytesReader(b []byte) SeekableReadCloser {
|
2017-04-17 22:34:19 +02:00
|
|
|
return bytesReader{
|
|
|
|
Reader: bytes.NewReader(b),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func (b bytesReader) Close() error {
|
|
|
|
return nil // intentionally blank
|
|
|
|
}
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
// Archive is a serialized archive reference. It is a union: thus, only one of its fields will be non-nil. Several
|
|
|
|
// helper routines exist as members in order to easily interact with archives of different kinds.
|
2017-10-20 22:21:04 +02:00
|
|
|
//nolint: lll
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
type Archive struct {
|
2017-10-22 22:39:21 +02:00
|
|
|
Sig string `json:"4dabf18193072939515e22adb298388d" yaml:"4dabf18193072939515e22adb298388d"` // the unique asset signature (see properties.go).
|
2017-10-23 03:30:42 +02:00
|
|
|
Hash string `json:"hash,omitempty" yaml:"hash,omitempty"` // the SHA256 hash of the archive contents.
|
2017-10-22 22:39:21 +02:00
|
|
|
Assets map[string]interface{} `json:"assets,omitempty" yaml:"assets,omitempty"` // a collection of other assets/archives.
|
|
|
|
Path string `json:"path,omitempty" yaml:"path,omitempty"` // a file on the current filesystem.
|
|
|
|
URI string `json:"uri,omitempty" yaml:"uri,omitempty"` // a remote URI (file://, http://, https://, etc).
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
2017-07-14 21:28:43 +02:00
|
|
|
const (
|
2017-07-17 19:38:57 +02:00
|
|
|
ArchiveSig = "0def7320c3a5731c473e5ecbe6d01bc7" // a randomly assigned archive type signature.
|
2017-10-22 22:39:21 +02:00
|
|
|
ArchiveHashProperty = "hash" // the dynamic property for an archive's hash.
|
2017-07-17 19:38:57 +02:00
|
|
|
ArchiveAssetsProperty = "assets" // the dynamic property for an archive's assets.
|
|
|
|
ArchivePathProperty = "path" // the dynamic property for an archive's path.
|
|
|
|
ArchiveURIProperty = "uri" // the dynamic property for an archive's URI.
|
2017-07-14 21:28:43 +02:00
|
|
|
)
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func NewAssetArchive(assets map[string]interface{}) (*Archive, error) {
|
|
|
|
// Ensure all elements are either assets or archives.
|
|
|
|
for _, asset := range assets {
|
|
|
|
switch t := asset.(type) {
|
|
|
|
case *Asset, *Archive:
|
|
|
|
// ok
|
|
|
|
default:
|
|
|
|
return &Archive{}, errors.Errorf("type %v is not a valid archive element", t)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
a := &Archive{Sig: ArchiveSig, Assets: assets}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewPathArchive(path string) (*Archive, error) {
|
|
|
|
a := &Archive{Sig: ArchiveSig, Path: path}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewURIArchive(uri string) (*Archive, error) {
|
|
|
|
a := &Archive{Sig: ArchiveSig, URI: uri}
|
|
|
|
err := a.EnsureHash()
|
|
|
|
return a, err
|
|
|
|
}
|
2017-07-14 21:28:43 +02:00
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) IsAssets() bool { return a.Assets != nil }
|
|
|
|
func (a *Archive) IsPath() bool { return a.Path != "" }
|
|
|
|
func (a *Archive) IsURI() bool { return a.URI != "" }
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) GetAssets() (map[string]interface{}, bool) {
|
2017-07-17 19:38:57 +02:00
|
|
|
if a.IsAssets() {
|
|
|
|
return a.Assets, true
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
return nil, false
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) GetPath() (string, bool) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if a.IsPath() {
|
2017-07-17 19:38:57 +02:00
|
|
|
return a.Path, true
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
return "", false
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) GetURI() (string, bool) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if a.IsURI() {
|
2017-07-17 19:38:57 +02:00
|
|
|
return a.URI, true
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
return "", false
|
|
|
|
}
|
|
|
|
|
|
|
|
// GetURIURL returns the underlying URI as a parsed URL, provided it is one. If there was an error parsing the URI, it
|
|
|
|
// will be returned as a non-nil error object.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) GetURIURL() (*url.URL, bool, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if uri, isuri := a.GetURI(); isuri {
|
|
|
|
url, err := url.Parse(uri)
|
|
|
|
if err != nil {
|
|
|
|
return nil, true, err
|
|
|
|
}
|
|
|
|
return url, true, nil
|
|
|
|
}
|
|
|
|
return nil, false, nil
|
|
|
|
}
|
|
|
|
|
2017-07-17 21:11:15 +02:00
|
|
|
// Equals returns true if a is value-equal to other.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) Equals(other *Archive) bool {
|
|
|
|
if a == nil {
|
|
|
|
return other == nil
|
|
|
|
} else if other == nil {
|
|
|
|
return false
|
|
|
|
}
|
2017-07-17 21:11:15 +02:00
|
|
|
if a.Assets != nil {
|
|
|
|
if other.Assets == nil {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
if len(a.Assets) != len(other.Assets) {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
for key, value := range a.Assets {
|
2017-10-22 22:39:21 +02:00
|
|
|
otherv := other.Assets[key]
|
|
|
|
switch valuet := value.(type) {
|
|
|
|
case *Asset:
|
|
|
|
if othera, isAsset := otherv.(*Asset); isAsset {
|
|
|
|
if !valuet.Equals(othera) {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
case *Archive:
|
|
|
|
if othera, isArchive := otherv.(*Archive); isArchive {
|
|
|
|
if !valuet.Equals(othera) {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
default:
|
2017-07-17 21:11:15 +02:00
|
|
|
return false
|
|
|
|
}
|
|
|
|
}
|
|
|
|
} else if other.Assets != nil {
|
|
|
|
return false
|
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
return a.Hash == other.Hash && a.Path == other.Path && a.URI == other.URI
|
2017-07-17 21:11:15 +02:00
|
|
|
}
|
|
|
|
|
2017-07-17 19:38:57 +02:00
|
|
|
// Serialize returns a weakly typed map that contains the right signature for serialization purposes.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) Serialize() map[string]interface{} {
|
|
|
|
result := map[string]interface{}{
|
|
|
|
string(SigKey): ArchiveSig,
|
|
|
|
}
|
|
|
|
if a.Hash != "" {
|
|
|
|
result[ArchiveHashProperty] = a.Hash
|
|
|
|
}
|
2017-07-17 19:38:57 +02:00
|
|
|
if a.Assets != nil {
|
2017-10-22 22:39:21 +02:00
|
|
|
assets := make(map[string]interface{})
|
2017-07-17 19:38:57 +02:00
|
|
|
for k, v := range a.Assets {
|
2017-10-22 22:39:21 +02:00
|
|
|
switch t := v.(type) {
|
|
|
|
case *Asset:
|
|
|
|
assets[k] = t.Serialize()
|
|
|
|
case *Archive:
|
|
|
|
assets[k] = t.Serialize()
|
|
|
|
default:
|
|
|
|
contract.Failf("Unrecognized asset map type %v", reflect.TypeOf(t))
|
|
|
|
}
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
result[ArchiveAssetsProperty] = assets
|
|
|
|
}
|
|
|
|
if a.Path != "" {
|
|
|
|
result[ArchivePathProperty] = a.Path
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
if a.URI != "" {
|
|
|
|
result[ArchiveURIProperty] = a.URI
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
return result
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// DeserializeArchive checks to see if the map contains an archive, using its signature, and if so deserializes it.
|
2017-10-23 00:54:44 +02:00
|
|
|
func DeserializeArchive(obj map[string]interface{}) (*Archive, bool, error) {
|
2017-10-22 22:39:21 +02:00
|
|
|
// If not an archive, return false immediately.
|
2017-07-17 19:38:57 +02:00
|
|
|
if obj[string(SigKey)] != ArchiveSig {
|
2017-10-22 22:39:21 +02:00
|
|
|
return &Archive{}, false, nil
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
|
|
|
|
var hash string
|
|
|
|
if v, has := obj[ArchiveHashProperty]; has {
|
|
|
|
hash = v.(string)
|
|
|
|
}
|
|
|
|
|
|
|
|
var assets map[string]interface{}
|
2017-07-17 19:38:57 +02:00
|
|
|
if v, has := obj[ArchiveAssetsProperty]; has {
|
2017-10-22 22:39:21 +02:00
|
|
|
assets = make(map[string]interface{})
|
|
|
|
if v != nil {
|
|
|
|
for k, elem := range v.(map[string]interface{}) {
|
|
|
|
switch t := elem.(type) {
|
|
|
|
case *Asset:
|
|
|
|
assets[k] = t
|
|
|
|
case *Archive:
|
|
|
|
assets[k] = t
|
|
|
|
case map[string]interface{}:
|
2017-10-23 00:54:44 +02:00
|
|
|
a, isa, err := DeserializeAsset(t)
|
2017-10-22 22:39:21 +02:00
|
|
|
if err != nil {
|
|
|
|
return &Archive{}, false, err
|
|
|
|
} else if isa {
|
|
|
|
assets[k] = a
|
|
|
|
} else {
|
2017-10-23 00:54:44 +02:00
|
|
|
arch, isarch, err := DeserializeArchive(t)
|
2017-10-22 22:39:21 +02:00
|
|
|
if err != nil {
|
|
|
|
return &Archive{}, false, err
|
|
|
|
} else if !isarch {
|
|
|
|
return &Archive{}, false, errors.Errorf("archive member '%v' is not an asset or archive", k)
|
|
|
|
}
|
|
|
|
assets[k] = arch
|
|
|
|
}
|
|
|
|
default:
|
|
|
|
return &Archive{}, false, nil
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
var path string
|
|
|
|
if v, has := obj[ArchivePathProperty]; has {
|
|
|
|
path = v.(string)
|
|
|
|
}
|
|
|
|
var uri string
|
|
|
|
if v, has := obj[ArchiveURIProperty]; has {
|
|
|
|
uri = v.(string)
|
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
if assets == nil && path == "" && uri == "" {
|
|
|
|
return &Archive{}, false, errors.New("archive is missing one of assets, path, or URI")
|
|
|
|
}
|
|
|
|
|
|
|
|
return &Archive{Hash: hash, Assets: assets, Path: path, URI: uri}, true, nil
|
2017-07-17 19:38:57 +02:00
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// Read returns a map of asset name to its associated reader object (which can be used to perform reads/IO).
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) Read() (map[string]*Blob, error) {
|
2017-07-17 19:38:57 +02:00
|
|
|
if a.IsAssets() {
|
|
|
|
return a.readAssets()
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
} else if a.IsPath() {
|
|
|
|
return a.readPath()
|
|
|
|
} else if a.IsURI() {
|
|
|
|
return a.readURI()
|
|
|
|
}
|
|
|
|
contract.Failf("Invalid archive; one of Assets, Path, or URI must be non-nil")
|
|
|
|
return nil, nil
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) readAssets() (map[string]*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// To read a map-based archive, just produce a map from each asset to its associated reader.
|
2017-07-17 19:38:57 +02:00
|
|
|
m, isassets := a.GetAssets()
|
|
|
|
contract.Assertf(isassets, "Expected an asset map-based archive")
|
2017-10-22 22:39:21 +02:00
|
|
|
|
2017-05-23 07:57:55 +02:00
|
|
|
result := map[string]*Blob{}
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
for name, asset := range m {
|
2017-10-22 22:39:21 +02:00
|
|
|
switch t := asset.(type) {
|
|
|
|
case *Asset:
|
|
|
|
// An asset can be added directly to the result.
|
|
|
|
blob, err := t.Read()
|
|
|
|
if err != nil {
|
|
|
|
return nil, errors.Wrapf(err, "failed to expand archive asset '%v'", name)
|
|
|
|
}
|
|
|
|
result[name] = blob
|
|
|
|
case *Archive:
|
|
|
|
// An archive must be recursively walked in order to turn it into a flat result map.
|
|
|
|
subs, err := t.Read()
|
|
|
|
if err != nil {
|
|
|
|
return nil, errors.Wrapf(err, "failed to expand sub-archive '%v'", name)
|
|
|
|
}
|
|
|
|
for sub, blob := range subs {
|
|
|
|
result[filepath.Join(name, sub)] = blob
|
2017-06-12 19:15:20 +02:00
|
|
|
}
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return result, nil
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) readPath() (map[string]*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// To read a path-based archive, read that file and use its extension to ascertain what format to use.
|
|
|
|
path, ispath := a.GetPath()
|
|
|
|
contract.Assertf(ispath, "Expected a path-based asset")
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
format := detectArchiveFormat(path)
|
|
|
|
|
|
|
|
if format == NotArchive {
|
|
|
|
// If not an archive, it could be a directory; if so, simply expand it out uncompressed as an archive.
|
|
|
|
info, err := os.Stat(path)
|
|
|
|
if err != nil {
|
|
|
|
return nil, errors.Wrapf(err, "couldn't read archive path '%v'", path)
|
|
|
|
} else if info.IsDir() {
|
|
|
|
return nil, errors.Wrapf(err, "'%v' is neither a recognized archive type nor a directory", path)
|
|
|
|
}
|
|
|
|
results := make(map[string]*Blob)
|
|
|
|
if walkerr := filepath.Walk(path, func(filePath string, f os.FileInfo, fileerr error) error {
|
|
|
|
// If there was an error, exit.
|
|
|
|
if fileerr != nil {
|
|
|
|
return fileerr
|
|
|
|
}
|
|
|
|
// If this was a directory or a symlink, skip it.
|
|
|
|
if f.IsDir() || f.Mode()&os.ModeSymlink != 0 {
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
// Finally, if this was a .pulumi directory, we will skip this by default.
|
|
|
|
// TODO[pulumi/pulumi#122]: when we support .pulumiignore, this will be customizable.
|
|
|
|
if !f.IsDir() && f.Name() == workspace.Dir {
|
|
|
|
return filepath.SkipDir
|
|
|
|
}
|
|
|
|
|
|
|
|
// Otherwise, add this asset to the list of blobs, and keep going.
|
|
|
|
file := &Asset{Path: filePath}
|
|
|
|
blob, err := file.Read()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
results[filePath] = blob
|
|
|
|
return nil
|
|
|
|
}); walkerr != nil {
|
|
|
|
return nil, walkerr
|
|
|
|
}
|
|
|
|
return results, nil
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
// Otherwise, it's an archive file, and we will go ahead and open it up and read it.
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
file, err := os.Open(path)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return readArchive(file, format)
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) readURI() (map[string]*Blob, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// To read a URI-based archive, fetch the contents remotely and use the extension to pick the format to use.
|
|
|
|
url, isurl, err := a.GetURIURL()
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
contract.Assertf(isurl, "Expected a URI-based asset")
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
format := detectArchiveFormat(url.Path)
|
|
|
|
if format == NotArchive {
|
2017-06-06 03:11:51 +02:00
|
|
|
// IDEA: support (a) hints and (b) custom providers that default to certain formats.
|
2017-10-22 22:39:21 +02:00
|
|
|
return nil, errors.Errorf("file at URL '%v' is not a recognized archive format", url)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
ar, err := a.openURLStream(url)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return readArchive(ar, format)
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) openURLStream(url *url.URL) (io.ReadCloser, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
switch s := url.Scheme; s {
|
|
|
|
case "http", "https":
|
2017-05-14 02:04:35 +02:00
|
|
|
resp, err := http.Get(url.String())
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
return resp.Body, nil
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
case "file":
|
|
|
|
contract.Assert(url.Host == "")
|
|
|
|
contract.Assert(url.User == nil)
|
|
|
|
contract.Assert(url.RawQuery == "")
|
|
|
|
contract.Assert(url.Fragment == "")
|
|
|
|
return os.Open(url.Path)
|
|
|
|
default:
|
|
|
|
return nil, errors.Errorf("Unrecognized or unsupported URI scheme: %v", s)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-05-01 18:38:23 +02:00
|
|
|
// Bytes fetches the archive contents as a byte slices. This is almost certainly the least efficient way to deal with
|
|
|
|
// the underlying streaming capabilities offered by assets and archives, but can be used in a pinch to interact with
|
|
|
|
// APIs that demand []bytes.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) Bytes(format ArchiveFormat) ([]byte, error) {
|
2017-05-01 18:38:23 +02:00
|
|
|
var data bytes.Buffer
|
2017-06-12 19:15:20 +02:00
|
|
|
if err := a.Archive(format, &data); err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
return data.Bytes(), nil
|
2017-05-01 18:38:23 +02:00
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// Archive produces a single archive stream in the desired format. It prefers to return the archive with as little
|
|
|
|
// copying as is feasible, however if the desired format is different from the source, it will need to translate.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) Archive(format ArchiveFormat, w io.Writer) error {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// If the source format is the same, just return that.
|
|
|
|
if sf, ss, err := a.ReadSourceArchive(); sf != NotArchive && sf == format {
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
_, err := io.Copy(w, ss)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
switch format {
|
|
|
|
case TarArchive:
|
|
|
|
return a.archiveTar(w)
|
|
|
|
case TarGZIPArchive:
|
|
|
|
return a.archiveTarGZIP(w)
|
|
|
|
case ZIPArchive:
|
|
|
|
return a.archiveZIP(w)
|
|
|
|
default:
|
|
|
|
contract.Failf("Illegal archive type: %v", format)
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) archiveTar(w io.Writer) error {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// Read the archive.
|
|
|
|
arch, err := a.Read()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
2017-08-29 03:52:51 +02:00
|
|
|
defer (func() {
|
|
|
|
// Ensure we close all files before exiting this function, no matter the outcome.
|
|
|
|
for _, blob := range arch {
|
|
|
|
contract.IgnoreClose(blob)
|
|
|
|
}
|
|
|
|
})()
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
// Sort the file names so we emit in a deterministic order.
|
|
|
|
var files []string
|
|
|
|
for file := range arch {
|
|
|
|
files = append(files, file)
|
|
|
|
}
|
|
|
|
sort.Strings(files)
|
|
|
|
|
|
|
|
// Now actually emit the contents, file by file.
|
|
|
|
tw := tar.NewWriter(w)
|
|
|
|
for _, file := range files {
|
|
|
|
data := arch[file]
|
|
|
|
sz := data.Size()
|
|
|
|
if err := tw.WriteHeader(&tar.Header{
|
|
|
|
Name: file,
|
|
|
|
Mode: 0600,
|
|
|
|
Size: sz,
|
|
|
|
}); err != nil {
|
|
|
|
return err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
n, err := io.Copy(tw, data)
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return err
|
|
|
|
}
|
Tidy up more lint
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 21:09:46 +02:00
|
|
|
contract.Assert(n == sz)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
return tw.Close()
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) archiveTarGZIP(w io.Writer) error {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
z := gzip.NewWriter(w)
|
|
|
|
return a.archiveTar(z)
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) archiveZIP(w io.Writer) error {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// Read the archive.
|
|
|
|
arch, err := a.Read()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
2017-08-29 03:52:51 +02:00
|
|
|
defer (func() {
|
|
|
|
// Ensure we close all files before exiting this function, no matter the outcome.
|
|
|
|
for _, blob := range arch {
|
|
|
|
contract.IgnoreClose(blob)
|
|
|
|
}
|
|
|
|
})()
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
// Sort the file names so we emit in a deterministic order.
|
|
|
|
var files []string
|
|
|
|
for file := range arch {
|
|
|
|
files = append(files, file)
|
|
|
|
}
|
|
|
|
sort.Strings(files)
|
|
|
|
|
|
|
|
// Now actually emit the contents, file by file.
|
|
|
|
zw := zip.NewWriter(w)
|
|
|
|
for _, file := range files {
|
|
|
|
fw, err := zw.Create(file)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
if _, err = io.Copy(fw, arch[file]); err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return zw.Close()
|
|
|
|
}
|
|
|
|
|
2017-10-22 22:39:21 +02:00
|
|
|
// ReadSourceArchive returns a stream to the underlying archive, if there is one.
|
|
|
|
func (a *Archive) ReadSourceArchive() (ArchiveFormat, io.ReadCloser, error) {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
if path, ispath := a.GetPath(); ispath {
|
2017-10-22 22:39:21 +02:00
|
|
|
if format := detectArchiveFormat(path); format != NotArchive {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
f, err := os.Open(path)
|
|
|
|
return format, f, err
|
|
|
|
}
|
2017-06-14 01:47:55 +02:00
|
|
|
} else if url, isurl, urlerr := a.GetURIURL(); urlerr == nil && isurl {
|
2017-10-22 22:39:21 +02:00
|
|
|
if format := detectArchiveFormat(url.Path); format != NotArchive {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
s, err := a.openURLStream(url)
|
|
|
|
return format, s, err
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return NotArchive, nil, nil
|
|
|
|
}
|
|
|
|
|
2017-10-23 03:30:42 +02:00
|
|
|
// EnsureHash computes the SHA256 hash of the archive's contents and stores it on the object.
|
2017-10-22 22:39:21 +02:00
|
|
|
func (a *Archive) EnsureHash() error {
|
|
|
|
if a.Hash == "" {
|
|
|
|
hash := sha256.New()
|
|
|
|
|
|
|
|
// Attempt to compute the hash in the most efficient way. First try to open the archive directly and copy it
|
|
|
|
// to the hash. This avoids traversing any of the contents and just treats it as a byte stream.
|
|
|
|
f, r, err := a.ReadSourceArchive()
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
if f != NotArchive && r != nil {
|
|
|
|
defer contract.IgnoreClose(r)
|
|
|
|
_, err = io.Copy(hash, r)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
// Otherwise, it's not an archive; we'll need to transform it into one. Pick tar since it avoids
|
|
|
|
// any superfluous compression which doesn't actually help us in this situation.
|
|
|
|
err := a.Archive(TarArchive, hash)
|
|
|
|
if err != nil {
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Finally, encode the resulting hash as a string and we're done.
|
|
|
|
a.Hash = hex.EncodeToString(hash.Sum(nil))
|
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
// ArchiveFormat indicates what archive and/or compression format an archive uses.
|
|
|
|
type ArchiveFormat int
|
|
|
|
|
|
|
|
const (
|
|
|
|
NotArchive = iota // not an archive.
|
|
|
|
TarArchive // a POSIX tar archive.
|
|
|
|
TarGZIPArchive // a POSIX tar archive that has been subsequently compressed using GZip.
|
|
|
|
ZIPArchive // a multi-file ZIP archive.
|
|
|
|
)
|
|
|
|
|
|
|
|
// ArchiveExts maps from a file extension and its associated archive and/or compression format.
|
|
|
|
var ArchiveExts = map[string]ArchiveFormat{
|
|
|
|
".tar": TarArchive,
|
|
|
|
".tgz": TarGZIPArchive,
|
|
|
|
".tar.gz": TarGZIPArchive,
|
|
|
|
".zip": ZIPArchive,
|
|
|
|
}
|
|
|
|
|
|
|
|
// detectArchiveFormat takes a path and infers its archive format based on the file extension.
|
2017-10-22 22:39:21 +02:00
|
|
|
func detectArchiveFormat(path string) ArchiveFormat {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
ext := filepath.Ext(path)
|
|
|
|
if moreext := filepath.Ext(strings.TrimRight(path, ext)); moreext != "" {
|
|
|
|
ext = moreext + ext // this ensures we detect ".tar.gz" correctly.
|
|
|
|
}
|
|
|
|
format, has := ArchiveExts[ext]
|
|
|
|
if !has {
|
2017-10-22 22:39:21 +02:00
|
|
|
return NotArchive
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
2017-10-22 22:39:21 +02:00
|
|
|
return format
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// readArchive takes a stream to an existing archive and returns a map of names to readers for the inner assets.
|
|
|
|
// The routine returns an error if something goes wrong and, no matter what, closes the stream before returning.
|
|
|
|
func readArchive(ar io.ReadCloser, format ArchiveFormat) (map[string]*Blob, error) {
|
Tidy up more lint
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 21:09:46 +02:00
|
|
|
defer contract.IgnoreClose(ar) // consume the input stream
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
switch format {
|
|
|
|
case TarArchive:
|
|
|
|
return readTarArchive(ar)
|
|
|
|
case TarGZIPArchive:
|
|
|
|
return readTarGZIPArchive(ar)
|
|
|
|
case ZIPArchive:
|
|
|
|
// Unfortunately, the ZIP archive reader requires ReaderAt functionality. If it's a file, we can recovera this
|
|
|
|
// with a simple stat. Otherwise, we will need to go ahead and make a copy in memory.
|
|
|
|
var ra io.ReaderAt
|
|
|
|
var sz int64
|
|
|
|
if f, isf := ar.(*os.File); isf {
|
2017-05-14 02:04:35 +02:00
|
|
|
stat, err := f.Stat()
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
ra = f
|
|
|
|
sz = stat.Size()
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
} else if data, err := ioutil.ReadAll(ar); err != nil {
|
|
|
|
return nil, err
|
|
|
|
} else {
|
|
|
|
ra = bytes.NewReader(data)
|
|
|
|
sz = int64(len(data))
|
|
|
|
}
|
|
|
|
return readZIPArchive(ra, sz)
|
|
|
|
default:
|
|
|
|
contract.Failf("Illegal archive type: %v", format)
|
|
|
|
return nil, nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func readTarArchive(ar io.ReadCloser) (map[string]*Blob, error) {
|
Tidy up more lint
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 21:09:46 +02:00
|
|
|
defer contract.IgnoreClose(ar) // consume the input stream
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
// Create a tar reader and walk through each file, adding each one to the map.
|
|
|
|
assets := make(map[string]*Blob)
|
|
|
|
tr := tar.NewReader(ar)
|
|
|
|
for {
|
|
|
|
file, err := tr.Next()
|
|
|
|
if err == io.EOF {
|
|
|
|
break
|
|
|
|
} else if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
switch file.Typeflag {
|
|
|
|
case tar.TypeDir:
|
|
|
|
continue // skip directories
|
|
|
|
case tar.TypeReg:
|
|
|
|
data := make([]byte, file.Size)
|
2017-05-14 02:04:35 +02:00
|
|
|
n, err := tr.Read(data)
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
contract.Assert(int64(n) == file.Size)
|
|
|
|
assets[file.Name] = NewByteBlob(data)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
default:
|
|
|
|
contract.Failf("Unrecognized tar header typeflag: %v", file.Typeflag)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return assets, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
func readTarGZIPArchive(ar io.ReadCloser) (map[string]*Blob, error) {
|
Tidy up more lint
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 21:09:46 +02:00
|
|
|
defer contract.IgnoreClose(ar) // consume the input stream
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
|
|
|
|
// First decompress the GZIP stream.
|
|
|
|
gz, err := gzip.NewReader(ar)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
// Now read the tarfile.
|
|
|
|
return readTarArchive(gz)
|
|
|
|
}
|
|
|
|
|
|
|
|
func readZIPArchive(ar io.ReaderAt, size int64) (map[string]*Blob, error) {
|
|
|
|
// Create a ZIP reader and iterate over the files inside of it, adding each one.
|
|
|
|
assets := make(map[string]*Blob)
|
|
|
|
z, err := zip.NewReader(ar, size)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
for _, file := range z.File {
|
|
|
|
body, err := file.Open()
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
size := file.UncompressedSize64
|
|
|
|
data := make([]byte, size)
|
2017-05-14 02:04:35 +02:00
|
|
|
n, err := body.Read(data)
|
|
|
|
if err != nil {
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
return nil, err
|
|
|
|
}
|
2017-05-14 02:04:35 +02:00
|
|
|
contract.Assert(uint64(n) == size)
|
|
|
|
assets[file.Name] = NewByteBlob(data)
|
Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 21:37:24 +02:00
|
|
|
}
|
|
|
|
return assets, nil
|
|
|
|
}
|