maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3170 commits 18 branches 277 tags 146 MiB
Commit graph

65 commits

Author SHA1 Message Date
Harshavardhana f248089523 api: Implement bucket notification. (#2271) 
* Implement basic S3 notifications through queues

Supports multiple queues and three basic queue types:

1. NilQueue -- messages don't get sent anywhere
2. LogQueue -- messages get logged
3. AmqpQueue -- messages are sent to an AMQP queue

* api: Implement bucket notification.

Supports two different queue types

- AMQP
- ElasticSearch.

* Add support for redis
 2016-07-23 22:51:12 -07:00
Harshavardhana 86d31e99d5 api: use checkAuth now at PutBucket, DeleteBucket handlers. (#2225) 
Additionally add a unit test for isReqAuthenticated function.
 2016-07-18 23:56:27 -07:00
Harshavardhana 1f706e067d api: xmlDecoder should honor contentLength. (#2226) 
This is needed so that we avoid reading large amounts
of data from compromised clients.
 2016-07-18 21:20:17 -07:00
Harshavardhana aeac902747 API: ListBuckets doesn't have a body, we should never read the body. (#2218) 
ListBuckets was incorrectly reading the body of the request, fix it.
 2016-07-17 13:23:15 -07:00
Harshavardhana aaf7803831 api: Requests should be differentiated if possible based on http router. (#2219) 
In current master ListObjectsV2 was merged into ListObjectsHandler
which also implements V1 API as well.

Move the detection of ListObject types to its rightful place
in http router.
 2016-07-17 12:32:05 -07:00
Krishna Srinivas 614c770b5d List Objects version 2. (#1815) 
object: List Objects v2 support
 2016-05-31 22:10:55 -07:00
Harshavardhana 9fdb69563d handler: CopyObject should save metadata. (#1698) 
- Content-Type
- Content-Encoding
- ETag

Fixes #1682
 2016-05-19 17:10:08 -07:00
Harshavardhana 9472299308 logging: Log only for unhandled errors, remove all the debug logging. (#1652) 
This patch brings in the removal of debug logging altogether, instead
we bring in the functionality of being able to trace the errors properly
pointing back to the origination of the problem.

To enable tracing you need to enable "MINIO_TRACE" set to "1" or "true"
environment variable which would print back traces whenever there is an
error which is unhandled or at the handler layer.

By default this tracing is turned off and only user level logging is
provided.
 2016-05-16 14:31:28 -07:00
Krishna Srinivas c06b9abc15 bucket-handlers: do not unescape marker as gorilla layer would have already done it. (#1495) (#1510) 2016-05-06 16:04:46 -07:00
Harshavardhana 17868ccd7f handlers: overhaul entire writErrorResponse, simplify. (#1472) 2016-05-05 20:24:29 -07:00
Harshavardhana ad40036cba posix: filepath shouldn't be used anymore use path.Join (#1486) 2016-05-05 01:39:26 -07:00
karthic rao 82113b747c Resource matching fix to overcome issues with regular expression based match (#1476) 2016-05-04 16:56:57 -07:00
Rajiv Makhijani 321aefa026 Add Response for PostPolicyBucketHandler (#1477) (#1483) 2016-05-04 15:24:10 -07:00
Yurii bba5468368 minio: Replace 'bucket already exists' error by 'bucket already owned by you'. (#1465) 
S3 API returns BucketAlreadyExists error when some another user has such bucket.
If user that creates the bucket already has it, s3 returns BucketAlreadyOwnedByYou.
As minio has only one user, it should behave accordingly.
Otherwise it causes failures in the applications that ignore creation of already existing bucket in the account, but fail when bucket name is used by someone else.
 2016-05-03 03:19:04 -07:00
Harshavardhana 4e34e03dd4 xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
Harshavardhana f3784d1087 xl: Handle read quorum for StatVol, ListVols 2016-04-25 12:47:31 -07:00
karthic rao 560c3bd153 Adding return statement after error response in the lastest commit to verify location constraint (#1348) 2016-04-21 20:08:08 -07:00
karthic rao cb1116725b api: verify Location constraint for make bucket. (#1342) 2016-04-20 17:35:38 -07:00
Harshavardhana e0f8fed011 object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
Harshavardhana be002ac01e fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
Harshavardhana 30b0b4deba storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
Krishna Srinivas 3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
karthic rao 188bb92d8a bucket-policy parset tests, and bug fixes (#1317) 2016-04-15 18:23:19 -07:00
Harshavardhana b182e94acc signature: Handle presigned payload if set. 
Validate payload with incoming content.



Fixes #1288
 2016-04-07 03:04:18 -07:00
Donald Guy e8cd1aad8d accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes 
* accessPolicy: copy object should require PutObject

* accessPolicy: cite mpu perms doc only for relevant operations

* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
 2016-04-06 18:31:40 -07:00
Donald Guy 8b4a5f07b4 accessPolicy: allow anonymous HEAD for Getable objects 
* accessPolicy: allow anonymous HEAD for Getable objects

* accessPolicy: allow anonymous HEAD of Listable Buckets
 2016-04-06 16:40:54 -07:00
Harshavardhana 3fcc60de91 Move the files and rename some functions. 
- Rename dir.go as 'fs-multipart-dir.go'
- Move the push/pop to fs-multipart.go and rename them as save/lookup.
- Rename objectInfo instances in fs-multipart as multipartObjInfo.
 2016-04-05 12:26:19 -07:00
Harshavardhana c69fdf0cf2 listObjects: Cleanup and naming conventions. 
- Marker should be escaped outside in handlers.

- Delimiter should be handled outside in handlers.

- Add missing comments and change the function names.

- Handle case of 'maxKeys' when its set to '0', its a valid

  case and should be treated as such.
 2016-04-04 19:55:07 -07:00
Harshavardhana 0479d4976b objectAPI: Fix object API interface, remove unnecessary structs. 
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
 2016-04-03 15:25:01 -07:00
Harshavardhana a6a4e7e297 signature: No need to validate region for getBucketLocation and listBuckets. 
This type of check is added for making sure that we can support
custom regions.

ListBuckets and GetBucketLocation are always "us-east-1" rest
should look for the configured region.

Fixes #1278
 2016-04-02 18:42:32 -07:00
Harshavardhana efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface. 
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
 2016-04-01 15:58:39 -07:00
Harshavardhana 02ad48466d error: Signature errors should be returned with APIErrorCode. 
The reasoning is that we can reply back with wide range of
S3 error responses, which would provide more richer context
to S3 client.

Fixes #1267
 2016-03-31 23:28:40 -07:00
Harshavardhana 9dca46e156 signature: Use a layered approach for signature verification. 
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.

In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
 2016-03-26 15:21:05 -07:00
Harshavardhana aaf97ea02c config/main: Re-write config files - add to new config v3 
- New config format.

```
{
	"version": "3",
	"address": ":9000",
    "backend": {
          "type": "fs",
          "disk": "/path"
    },
	"credential": {
		"accessKey": "WLGDGYAQYIGI833EV05A",
		"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
	},
	"region": "us-east-1",
	"logger": {
		"file": {
			"enable": false,
			"fileName": "",
			"level": "error"
		},
		"syslog": {
			"enable": false,
			"address": "",
			"level": "debug"
		},
		"console": {
			"enable": true,
			"level": "fatal"
		}
	}
}
```

New command lines in lieu of supporting XL.

Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~

Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~

For 'fs' backend it starts the server.
~~~
$ minio server
~~~

For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~

Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
 2016-03-23 19:16:09 -07:00
Harshavardhana 76bda0d8f1 routers: Fix order of PostPolicyHandlers and headers. 2016-03-22 17:54:44 -07:00
Harshavardhana 88714e7c8e bucketpolicy: Improve bucket policy validation, avoid nested rules. 
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.

This patch implements stricter checks.

Example policy to be generally avoided.
```
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"s3:GetObject",
				"s3:DeleteObject"
			],
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"*"
				]
			},
			"Resource": [
				"arn:aws:s3:::jarjarbing/*"
			]
		},
		{
			"Action": [
				"s3:GetObject",
				"s3:DeleteObject"
			],
			"Effect": "Deny",
			"Principal": {
				"AWS": [
					"*"
				]
			},
			"Resource": [
				"arn:aws:s3:::jarjarbing/restic/key/*"
			]
		}
	]
}
```
 2016-03-15 17:50:23 -07:00
Harshavardhana 52751d81cb cleanup: Rename ObjectMetadata as ObjectInfo. 
Fixes #1215
 2016-03-11 16:58:08 -08:00
Harshavardhana fdf3d64793 error: Add proper prefixes for s3Error codes. 
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
 2016-03-10 18:38:46 -08:00
Harshavardhana d5057b3c51 accessPolicy: Implement Put, Get, Delete access policy. 
This patch implements Get,Put,Delete bucket policies

Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html

Currently supports following actions.

   "*":                             true,
   "s3:*":                          true,
   "s3:GetObject":                  true,
   "s3:ListBucket":                 true,
   "s3:PutObject":                  true,
   "s3:CreateBucket":               true,
   "s3:GetBucketLocation":          true,
   "s3:DeleteBucket":               true,
   "s3:DeleteObject":               true,
   "s3:AbortMultipartUpload":       true,
   "s3:ListBucketMultipartUploads": true,
   "s3:ListMultipartUploadParts":   true,

following conditions for "StringEquals" and "StringNotEquals"

   "s3:prefix", "s3:max-keys"
 2016-03-08 17:44:50 -08:00
Harshavardhana aed62788d9 api: Implement multiple objects Delete api - fixes #956 
This API takes input XML input in following form.

```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
         <Key>Key</Key>
    </Object>
    <Object>
         <Key>Key</Key>
    </Object>
    ...
</Delete>
```

and responds the list of successful deletes, list of errors
for all the deleted objects.

```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>sample1.txt</Key>
  </Deleted>
  <Error>
    <Key>sample2.txt</Key>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
  </Error>
</DeleteResult>
```
 2016-03-06 18:31:50 -08:00
Brendan Ashworth adf74ffdb0 api: DRY code and add new test 
This commit makes code cleaner and reduces the repetitions in the code
base. Specifically, it reduces the clutter in setObjectHeaders. It also
merges encodeSuccessResponse and encodeErrorResponse together because
they served no purpose differently. Finally, it adds a simple test for
generateRequestID.
 2016-03-06 13:26:27 -08:00
Harshavardhana f111997184 multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible. 
This increases the performance phenominally.
 2016-03-02 14:20:49 -08:00
Harshavardhana 17d145df3a signature: Fix signature handling of parallel requests. 
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
 2016-03-02 11:49:50 -08:00
Harshavardhana 653ceee9ee signV4: Move pkg/signature to pkg/s3/signature4 
Cleanup and move this to relevant path.
 2016-02-22 22:47:09 -08:00
Harshavardhana 91a092792a presigned: Fix a bug in presigned request verification. 
Additionally add Docker proxy configuration.
 2016-02-18 02:23:12 -08:00
Harshavardhana dd9aaa855c web/rpc: Merge ports with API server. 
Fixes #1081 and #1130
 2016-02-17 20:28:15 -08:00
Harshavardhana 5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana 62f6ffb6db xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Harshavardhana a4c005ce30 multipart: Code cleanup 
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
 2016-02-05 14:42:09 -08:00
Bala.FA d79fcb1800 fix: handle Transfer-Encoding for make bucket 
In case of make bucket, there is a chance of Transfer-Encoding is sent
where Content-Length is missing.  This patch fixes the problem by
checking whether Transfer-Encoding: chunked is set along with
Content-Length.
 2016-02-05 19:01:39 +05:30