Commit graph

69 commits

Author SHA1 Message Date
Pontus Leitzler 9631d65552 Fix goroutine test fatalf (#6682)
Use t.Error/t.ErrorF instead if t.Fatal/t.Fatalf

Add returns to achieve same behaviour as earlier
2018-10-23 09:44:20 -07:00
Praveen raj Mani ce9d36d954 Add object compression support (#6292)
Add support for streaming (golang/LZ77/snappy) compression.
2018-09-28 09:06:17 +05:30
Aditya Manthramurthy 36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
poornas 5c0b98abf0 Add ObjectOptions to ObjectLayer calls (#6382) 2018-09-10 09:42:43 -07:00
kannappanr 76ddf4d32f Log x-amz-request-id as log and XML error response (#6173)
Currently, requestid field in logEntry is not populated, as the
requestid field gets set at the very end.
It is now set before regular handler functions. This is also
useful in setting it as part of the XML error response.

Travis build for ppc64le has been quite inconsistent and stays queued
for most of the time. Removing this build as part of Travis.yml for
the time being.
2018-07-20 18:46:32 -07:00
Bala FA 0d52126023 Enhance policy handling to support SSE and WORM (#5790)
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests

This patch brings support to bucket policy to have more control not
limiting to anonymous.  Bucket owner controls to allow/deny any rest
API.

For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
2018-04-24 15:53:30 -07:00
kannappanr f8a3fd0c2a
Create logger package and rename errorIf to LogIf (#5678)
Removing message from error logging
Replace errors.Trace with LogIf
2018-04-05 15:04:40 -07:00
Krishna Srinivas 9ede179a21 Use context.Background() instead of nil
Rename Context[Get|Set] -> [Get|Set]Context
2018-03-15 16:28:25 -07:00
Krishna Srinivas e452377b24 Add context to the object-interface methods.
Make necessary changes to xl fs azure sia
2018-03-15 16:28:25 -07:00
Bala FA 0e4431725c make notification as separate package (#5294)
* Remove old notification files

* Add net package

* Add event package

* Modify minio to take new notification system
2018-03-15 13:03:41 -07:00
Harshavardhana e4f6877c8b Handle incoming proxy requests ip, scheme (#5591)
This PR implements functions to get the right ip, scheme
from the incoming proxied requests.
2018-03-02 15:23:04 -08:00
poornas 4f73fd9487 Unify gateway and object layer. (#5487)
* Unify gateway and object layer. Bring bucket policies into
object layer.
2018-02-09 15:19:30 -08:00
poornas 0bb6247056 Move nslocking from s3 layer to object layer (#5382)
Fixes #5350
2018-01-13 10:04:52 +05:30
Nitish Tiwari 545a9e4a82 Fix storage class related issues (#5322)
- Add storage class metadata validation for request header
- Change storage class header values to be consistent with AWS S3
- Refactor internal method to take only the reqd argument
2017-12-27 10:06:16 +05:30
Harshavardhana a4d6195244 Add public data-types for easier external loading (#5170)
This change brings public data-types such that
we can ask projects to implement gateway projects
externally than maintaining in our repo.

All publicly exported structs are maintained in object-api-datatypes.go

completePart --> CompletePart
uploadMetadata --> MultipartInfo

All other exported errors are at object-api-errors.go
2017-11-14 13:55:10 +05:30
Bala FA 32c6b62932 move credentials as separate package (#5115) 2017-10-31 11:54:32 -07:00
Harshavardhana 1d8a8c63db Simplify data verification with HashReader. (#5071)
Verify() was being called by caller after the data
has been successfully read after io.EOF. This disconnection
opens a race under concurrent access to such an object.
Verification is not necessary outside of Read() call,
we can simply just do checksum verification right inside
Read() call at io.EOF.

This approach simplifies the usage.
2017-10-22 11:00:34 +05:30
Harshavardhana c3ff402fcb Fix signature v2 and presigned query unescaping. (#4936)
Simplifies the testing code by using s3signer
package from minio-go library.

Fixes #4927
2017-09-24 14:20:12 -07:00
Andreas Auernhammer 79ba4d3f33 refactor ObjectLayer PutObject and PutObjectPart (#4925)
This change refactor the ObjectLayer PutObject and PutObjectPart
functions. Instead of passing an io.Reader and a size to PUT operations
ObejectLayer expects an HashReader.
A HashReader verifies the MD5 sum (and SHA256 sum if required) of the object.
This change updates all all PutObject(Part) calls and removes unnecessary code
in all ObjectLayer implementations.

Fixes #4923
2017-09-19 12:40:27 -07:00
Aaron Walker 5db533c024 bucket-policy: Add IPAddress/NotIPAddress conditions support (#4736) 2017-08-05 01:00:05 -07:00
Krishnan Parthasarathi 75c43bfb6c ListMultipartUploads, ListObjectParts return empty response (#4694)
Also, periodically removes incomplete multipart uploads older than 2 weeks.
2017-08-04 10:45:57 -07:00
Harshavardhana a86c2e2ce1 xl/fs: Return InvalidPart{} error for part ETag mismatch. (#4541)
Fixes #4539
2017-06-17 11:20:39 -07:00
Harshavardhana 99ddd35343 docs: use IEC format such as iB everywhere. (#4247) 2017-05-05 08:28:08 -07:00
Harshavardhana 48aa2ac392 server: Validate path for bad components in a handler. (#4170) 2017-04-24 18:13:46 -07:00
Anis Elleuch e4bd882f11 handlers: Ignore malformatted datetime type header (#4097)
Ignore headers, such as If-Modified-Since, If-Unmodified-Since, etc.. when they
are received with a format other than HTTP date.
2017-04-12 12:34:57 -07:00
Anis Elleuch e2aba9196f obj-handlers: Rewrite src & dst path cmp in Copy() (#3998)
CopyObjectHandler() was incorrectly performing comparison
between destination and source object paths, which sometimes
leads to a lock race. This PR simplifies comparaison and add
one test case.
2017-03-29 09:21:38 -07:00
Harshavardhana b62cd8ed84 sign/streaming: Content-Encoding is not set in newer aws-java-sdks (#3986)
We can't use Content-Encoding to verify if `aws-chunked` is set
or not. Just use 'streaming' signature header instead.

While this is considered mandatory, on the contrary aws-sdk-java
doesn't set this value

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

```
Set the value to aws-chunked.
```

We will relax it and behave appropriately. Also this PR supports
saving custom encoding after trimming off the `aws-chunked`
parameter.

Fixes #3983
2017-03-27 17:02:04 -07:00
Harshavardhana 6509589adb Use canonicalETag helper wherever needed. (#3910) 2017-03-15 20:48:49 -07:00
Harshavardhana 05e53f1b34 api: CopyObjectPart was copying wrong offsets due to shadowing. (#3838)
startOffset was re-assigned to '0' so it would end up
copying wrong content ignoring the requested startOffset.

This also fixes the corruption issue we observed while
using docker registry.

Fixes https://github.com/docker/distribution/issues/2205

Also fixes #3842 - incorrect routing.
2017-03-03 16:32:04 -08:00
Harshavardhana bc52d911ef api: Increase the maximum object size limit from 5GiB to 16GiB. (#3834)
The globalMaxObjectSize limit is instilled in S3 spec perhaps
due to certain limitations on S3 infrastructure. For minio we
don't have such limitations and we can stream a larger file
instead.

So we are going to bump this limit to 16GiB.

Fixes #3825
2017-03-03 10:14:17 -08:00
Harshavardhana 6b68c0170f For streaming signature do not save content-encoding in PutObject() (#3776)
Content-Encoding is set to "aws-chunked" which is an S3 specific
API value which is no meaning for an object. This is how S3
behaves as well for a streaming signature uploaded object.
2017-02-20 12:07:03 -08:00
Anis Elleuch 70e70446bb signv4: Read always returns EOF when stream ends (#3692)
When EOF is reached, further calls of Read() doesn't return io.EOF
but continue to work as it expects to have more data, this PR fixes
the behavior
2017-02-06 14:19:27 -08:00
Harshavardhana 77a192a7b5 Implement CopyObjectPart API (#3663)
This API is implemented to allow copying data from an
existing source object to an ongoing multipart operation

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html

Fixes #3662
2017-01-31 09:38:34 -08:00
Harshavardhana 69559aa101 objAPI: Implement CopyObject API. (#3487)
This is written so that to simplify our handler code
and provide a way to only update metadata instead of
the data when source and destination in CopyObject
request are same.

Fixes #3316
2016-12-26 16:29:26 -08:00
Bala FA e8ce3b64ed Generate and use access/secret keys properly (#3498) 2016-12-26 10:21:23 -08:00
Bala FA 825000bc34 Use humanize constants for KiB, MiB and GiB units. (#3322) 2016-11-22 18:18:22 -08:00
Bala FA bef0a50bc1 Cleanup and fixes (#3273)
* newRequestID() (previously generateUploadID()) returns string than byte array.
* Remove unclear comments and added appropriate comments.
* SHA-256, MD5 Hash functions return Hex/Base64 encoded string than byte array.
* Remove duplicate MD5 hasher functions.
* Rename listObjectsValidateArgs() into validateListObjectsArgs()
* Remove repeated auth check code in all bucket request handlers.
* Remove abbreviated names in bucket-metadata
* Avoid nested if in bucketPolicyMatchStatement()
* Use ioutil.ReadFile() instead of os.Open() and ioutil.ReadAll()
* Set crossDomainXML as constant.
2016-11-21 13:51:05 -08:00
Harshavardhana a8ab02a73a v4/presign: Fix presign requests when there are more signed headers. (#3222)
This fix removes a wrong logic which fails for requests which
have more signed headers in a presign request.

Fixes #3217
2016-11-10 21:57:15 -08:00
Harshavardhana 51d1e6f75b Fix missing CompleteMultipartUpload Etag. (#3227)
Fixes #3224
2016-11-10 07:41:02 -08:00
Bala FA 9c2cfb5cb6 tests: Add missing unit test cases for AbortMultipartUploadHandler(). (#3200)
Fixes #3070
2016-11-08 16:25:00 -08:00
Bala FA 91a0ade908 tests: add unit test for HeadObjectHandler (#3197)
Fixes #3068
2016-11-07 16:02:27 -08:00
Karthic Rao 286a8924fd Add leak detection to object-handler tests. (#3195) 2016-11-06 21:53:50 -08:00
Karthic Rao 3e8cb8c937 build: ineffassign fixes. (#3134) 2016-10-30 23:32:46 -07:00
Karthic Rao 63f1b4fdf4 mispell fixes. (#3100) 2016-10-26 08:46:14 -07:00
Krishnan Parthasarathi 31f2db6880 Remove leftover debug statement from PutObject StreamingSignature unit-test (#3099) 2016-10-26 03:17:47 -07:00
Krishnan Parthasarathi 12cd2da265 Add PutObjectHandler unit tests covering failure cases (#3096) 2016-10-26 02:06:22 -07:00
Bala FA 36639b65a9 rename completeMultipartMD5() into getCompleteMultipartMD5(). (#3051) 2016-10-24 13:56:13 -07:00
Karthic Rao 070d3610ff tests: V2 Signature tests for object-handlers. (#2931) 2016-10-14 20:52:46 -07:00
Krishnan Parthasarathi a5921b5743 Use same timestamp for all chunks in chunked signature (#2908) 2016-10-11 23:46:51 -07:00
Karthic Rao ff91ecb177 tests: Adding unknown signature type test for API handlers. (#2905) 2016-10-11 20:38:10 -07:00