Commit graph

5222 commits

Author SHA1 Message Date
Aditya Manthramurthy 16a100b597 Fix out-of-bound array access crash in select processing (#6594)
Fix test case.
2018-10-09 09:45:56 -07:00
Anis Elleuch cbc5d78a09 Handle read/quorum errors when initializing all subsystems (#6585)
- Only require len(disks)/2 to initialize the cluster
- Fix checking of read/write quorm in subsystems init
- Add retry mechanism in policy and notification to avoid aborting in case of read/write quorums errors
2018-10-08 15:47:13 -07:00
Minio Trusted d8a2975a68 Update yaml files to latest version RELEASE.2018-10-06T00-15-16Z 2018-10-06 00:19:47 +00:00
Anis Elleuch 66d911653f xl: Fix typo in PutObjectPart when part size is 10Mb (#6574)
PutObjectPart forgot to allocate buffer memory when the size
of the uploaded part is exactly equal to blockSizeV1 = 10 Mb.
2018-10-05 17:09:50 -07:00
Anis Elleuch ca6f795504 xl: check for correct err variable when prepareFile fails in PutObjectPart (#6573)
in xl.PutObjectPart call, prepareFile detected an error when the storage
is exhausted but we were returning the wrong error.

With this commit, users can see the correct error message when their disks
become full.
2018-10-05 17:09:20 -07:00
Eco 2af0f11731 Update readme.md (#6568) 2018-10-05 16:25:22 -07:00
Harshavardhana c3408f4f04 Send correct bucket notifications from web handlers (#6572)
Upload, Download, DownloadZip were incomplete
2018-10-05 11:20:00 -07:00
Minio Trusted b92c324254 Update yaml files to latest version RELEASE.2018-10-05T01-03-03Z 2018-10-05 01:08:39 +00:00
Krishna Srinivas 81bee93b8d Move remote disk StorageAPI abstraction from RPC to REST (#6464) 2018-10-04 17:44:06 -07:00
Ashish Kumar Sinha 670f9788e3 Count(*) to give integer value (#6564)
The Max, Min functions were giving float value even when they were integers.  
Resolved max and Min to return integers in that scenario.

Fixes #6472
2018-10-04 17:33:53 -07:00
Anis Elleuch f187a16962 xl: Require full write quorum in rename() as general rule (#6535)
Simplify the logic of using rename() in xl. Currently, renaming
doesn't require the source object/dir to be existent in at least
read quorum disks, since there is no apparent reason for it
to be as a general rule, this commit will just simplify the
logic to avoid possible inconsistency in the backend in the future.
2018-10-04 17:22:49 -07:00
Anis Elleuch e031f2b614 xl: Fix typo in PutObject when uploading a 10Mb file (#6567)
PutObject forgot to allocate memory when the size of the uploaded
file is exactly equal to blockSizeV1 = 10 Mb.
2018-10-04 15:37:15 -07:00
Harshavardhana c2b7b82ef4 Verify bitrot in ReadFile correctly when verifier is set (#6563)
This is a major regression introduced in this commit

ce02ab613d is the first bad commit
commit ce02ab613d
Author: Krishna Srinivas <634494+krishnasrinivas@users.noreply.github.com>
Date:   Mon Aug 6 15:14:08 2018 -0700

    Simplify erasure code by separating bitrot from erasure code (#5959)

:040000 040000 794f58d82a d2201ebfc8 M	cmd

This effects all distributed server deployments since this commit

All the following releases are affected

- RELEASE.2018-09-25T21-34-43Z
- RELEASE.2018-09-12T18-49-56Z
- RELEASE.2018-09-11T01-39-21Z
- RELEASE.2018-09-01T00-38-25Z
- RELEASE.2018-08-25T01-56-38Z
- RELEASE.2018-08-21T00-37-20Z
- RELEASE.2018-08-18T03-49-57Z

Thanks to Anis for reproducing the issue
2018-10-04 10:16:38 -07:00
Harshavardhana 02ad9d6072 Avoid unsolicited response over idle http client (#6562)
Without this PR minio server is writing an erroneous
response to clients on an idle connections which ends
up printing following message

```
Unsolicited response received on idle HTTP channel
```

This PR would avoid sending responses on idle connections
.i.e routine network errors.
2018-10-04 10:13:12 -07:00
Anis Elleuch ea9408ccbb worm: when enabled, avoid renaming the existing object in tmp directory (#6560)
In XL PutObject & CompleteMultipartUpload, the existing object is renamed
to the temporary directory before checking if worm is enabled or not.

Most of the times, this doesn't cause an issue unless two uploads to the
same location occurs at the same time. Since there is no locking in object
handlers, both uploads will reach XL layer. The second client acquiring
write lock in put object or complete upload in XL will rename the object
to the temporary directory before doing the check and returning the error (wrong!).

This commit fixes then the behavior: no rename to temporary directory if
worm is enabled.
2018-10-03 20:39:24 -07:00
Pontus Leitzler 307765591d Use GetObjectInfo instead of GetObjectNInfo before cache decision (#6553) 2018-10-03 11:02:32 -07:00
Harshavardhana 5d859b2178
gateway/azure: allow putObject to support block based upload (#6552)
Current implementation simply uses all the memory locally
and crashes when a large upload is initiated using Minio
browser UI.

This PR uploads stream in blocks and finally commits the blocks
allowing for low memory footprint while uploading large objects
through Minio browser UI.

This PR also adds ETag compatibility for single PUT operations.

Fixes #6542
Fixes #6550
2018-10-02 23:08:16 -07:00
Harshavardhana 223967fd32 Return always a default heal item upon unexpected error (#6556)
Never return an empty result item even upon error,
choose all the default values and based on the errors
make sure to send right result reply.
2018-10-02 17:13:51 -07:00
Pontus Leitzler 274b35154c Fix go1.11 vet shadow errors (#6555) 2018-10-02 15:21:34 -07:00
Harshavardhana c05ced08bb
Handle Range requests on empty objects (#6557)
Return a proper error on empty objects, S3 returns
416 Invalid Range on objects which are empty.

We should return the same.
2018-10-02 12:48:51 -07:00
Praveen raj Mani c7722fbb1b Simplify pkg mimedb (#6549)
Content-Type resolution can now use a function `TypeByExtension(extension)` 
to resolve to the respective content-type.
2018-10-02 11:48:17 +05:30
Harshavardhana f163bed40d
Add Vault support for custom CAs directory (#6527) 2018-10-01 13:49:10 -07:00
Harshavardhana b4772849f9 Heal recursively all entries in config/ prefix (#6545)
This to ensure that we heal all entries in config/
prefix, we will have IAM and STS related files which
are being introduced in #6168 PR

This is a change to ensure that we heal all of them
properly, not just `config.json`
2018-10-01 22:24:26 +05:30
Harshavardhana 839a758a36
Fix a crash due to race between Abort/CompleteMultipart (#6544)
Fixes #6429
2018-10-01 09:50:09 -07:00
Harshavardhana aebfceeafb Heal backend configuration file (#6532)
Fixes #6461
2018-09-29 13:47:01 +05:30
Anis Elleuch 83d7ec09c1 Disable restarting server after setting a new config (#6521)
Also disable listening to service restart event in tests since
we don't do this anymore.
2018-09-28 12:10:51 -07:00
Harshavardhana 8c29f69b00 Fix racy error communication inside go-routine (#6539)
Use CloseWithError to communicate errors in pipe,
this PR also fixes potential shadowing of error
2018-09-28 13:14:59 +05:30
Praveen raj Mani ce9d36d954 Add object compression support (#6292)
Add support for streaming (golang/LZ77/snappy) compression.
2018-09-28 09:06:17 +05:30
Anis Elleuch 5c765bc63e Fix one possible data race in admin tests (#6537)
go test shows the following warning:
```
WARNING: DATA RACE
Write at 0x000002909e18 by goroutine 276:
  github.com/minio/minio/cmd.testAdminCmdRunnerSignalService()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-rpc_test.go:44 +0x94

 Previous read at 0x000002909e18 by goroutine 194:
  github.com/minio/minio/cmd.testServiceSignalReceiver()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-handlers_test.go:467 +0x70
```

The reason for this data race is that some admin tests are not waiting for go routines
that they created to be properly exited, which triggers the race detector.
2018-09-27 17:16:30 -07:00
Anis Elleuch 6c7c6bec91 profiler: Download API returns error when all nodes fail (#6525)
When download profiling data API fails to gather profiling data
from all nodes for any reason (including profiler not enabled),
return 400 http code with the appropriate json message.
2018-09-27 10:34:37 -07:00
poornas ed703c065d Add ObjectOptions to GetObjectNInfo (#6533) 2018-09-27 15:36:45 +05:30
Aditya Manthramurthy 387584356f Remove unused range parsing code and update tests (#6530) 2018-09-27 15:24:07 +05:30
Harshavardhana 1111419d4a Add debugging for mutex, tracing (#6522) 2018-09-27 09:32:05 +05:30
Anis Elleuch 20378821cf madmin: close http response when returning an error (#6526)
httpRespToErrorResponse() usually reads the http response when
the http error code is not expected to parse the json error
response in the http body, however it was never properly closing
the connection. This PR fixes the behavior.
2018-09-26 11:03:35 -07:00
Pontus Leitzler ce1bfa6de8 Removed unused vendored dependencies (#6520) 2018-09-25 14:51:51 -07:00
Minio Trusted 6c26227081 Update yaml files to latest version RELEASE.2018-09-25T21-34-43Z 2018-09-25 21:39:03 +00:00
Anis Elleuch aa4e2b1542 Use GetObjectNInfo in CopyObject and CopyObjectPart (#6489) 2018-09-25 12:39:46 -07:00
Harshavardhana 1e5ac39ff3 Return HTTP response error for malformed requests (#6517) 2018-09-25 11:07:26 -07:00
Harshavardhana ec2295c3dc Quickly build dev docker images using 'make docker' (#6505)
This PR simplifies the process of developer build of local
docker containers using `make docker`.

You need to provide a TAG i.e

```
TAG=y4m4/minio:exp make docker
```
2018-09-25 10:33:25 -07:00
Andreas Auernhammer 8cf7b88cc5 add functions to remove confidential information (#6516)
This commit adds two functions for removing
confidential information - like SSE-C keys -
from HTTP headers / object metadata.

This creates a central point grouping all
headers/entries which must be filtered / removed.

See also https://github.com/minio/minio/pull/6489#discussion_r219797993
of #6489
2018-09-24 21:02:51 +05:30
Harshavardhana 48bfebe442 HEAD on an object should mimic GET without body (#6508)
Add "Range" header support etc.
2018-09-23 22:54:10 +05:30
Pontus Leitzler df60b3c733 Remove unnecessary contexts passed as data to FatalIf. No need to log an empty context. (#6487) 2018-09-21 16:04:11 -07:00
Aditya Manthramurthy 584cb61bb8 Switch back to GetObjectInfo for HEAD requests (#6513) 2018-09-21 13:48:58 -07:00
kannappanr c1798cc89a
Update GCS storage library to support GetObject API on gzipped objects (#6506)
Fixes #6280
2018-09-21 11:43:10 -07:00
Aditya Manthramurthy 3c8fabd116 Fix cleanup of pipe in GetObjectNInfo handlers (#6509) 2018-09-21 11:42:06 -07:00
Aditya Manthramurthy 36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
Harshavardhana 7d0645fb3a Deprecate domain, browser as config entries (#6498) 2018-09-20 14:56:32 -07:00
Kanagaraj M 7c339e248a fix success alert not shown after object is uploaded (#6500) 2018-09-20 22:54:09 +05:30
Harshavardhana b62ed5dc90 select API CSV may not be specified (#6493)
This should be present until we support JSON
2018-09-20 15:04:26 +05:30
Harshavardhana f0641a0406 Avoid changing creds in streaming signature (#6495)
This PR fixes a potential issue where credentials can be
changed in middle of different chunks during data transfer
which can lead to unexpected bugs.
2018-09-19 16:52:05 -07:00