Commit graph

3878 commits

Author SHA1 Message Date
Bala FA 825000bc34 Use humanize constants for KiB, MiB and GiB units. (#3322) 2016-11-22 18:18:22 -08:00
Bala FA c1ebcbcda2 Remove usused code. (#3321) 2016-11-22 17:44:18 -08:00
Anis Elleuch 41a3a9e402 server: forbid zero port in address flag since it confuses clients and (#3318) 2016-11-22 17:01:15 -08:00
Bala FA 1d4ac4b084 Rename getUUID() into mustGetUUID() (#3320)
In case of UUID generation failure mustGetUUID() will panic than
infinitely trying in for loop.
2016-11-22 16:52:37 -08:00
Bala FA 71b357e4f2 Remove uploadIDChange structure. (#3309)
addUploadID() and removeUploadID() are wrappers to updateUploadJSON()
which is called with respective arguments.
2016-11-22 15:29:39 -08:00
Anis Elleuch 339c9019b9 Protect multipart directory from removing when it is empty (#3315) 2016-11-22 13:15:06 -08:00
Harshavardhana dd93f808c8 web: Add more data for jsonrpc responses. (#3296)
This change adds more richer error response
for JSON-RPC by interpreting object layer
errors to corresponding meaningful errors
for the web browser.

```go
&json2.Error{
   Message: "Bucket Name Invalid, Only lowercase letters, full stops, and numbers are allowed.",
}
```

Additionally this patch also allows PresignedGetObject()
to take expiry parameter to have variable expiry.
2016-11-22 11:12:38 -08:00
Anis Elleuch 4098025c11 Remove XL multipart tmp files when the latter is canceled (#3214)
XL multipart fails to remove tmp files when an error occurs during upload, this case covers the scenario where an upload is canceled manually by the client in the middle of job.
2016-11-21 16:34:57 -08:00
Bala FA bef0a50bc1 Cleanup and fixes (#3273)
* newRequestID() (previously generateUploadID()) returns string than byte array.
* Remove unclear comments and added appropriate comments.
* SHA-256, MD5 Hash functions return Hex/Base64 encoded string than byte array.
* Remove duplicate MD5 hasher functions.
* Rename listObjectsValidateArgs() into validateListObjectsArgs()
* Remove repeated auth check code in all bucket request handlers.
* Remove abbreviated names in bucket-metadata
* Avoid nested if in bucketPolicyMatchStatement()
* Use ioutil.ReadFile() instead of os.Open() and ioutil.ReadAll()
* Set crossDomainXML as constant.
2016-11-21 13:51:05 -08:00
Anis Elleuch 71ada9d6f8 commitXLMetadata() expects src and dst buckets in its arguments (#3307) 2016-11-21 12:45:02 -08:00
tibbes 33c022fcec Fix checkdeps.sh on Mac (#3306)
Update the check_minimum_version function to use numeric comparison (not
string comparison) on components of version numbers. Fixes the following
output:

```
$ make
Checking deps:
ERROR
OSX version '10.11.6' not supported.
Minimum supported version: 10.8
make: *** [checks] Error 1
```
2016-11-21 12:25:46 -08:00
Karthic Rao 273228fafa Vendor update for dsync, fixing major go routine leak issue. (#3308) 2016-11-21 10:47:48 -08:00
Harshavardhana aa98702908 api: Handle content-length-range policy properly. (#3297)
content-length-range policy in postPolicy API was
not working properly handle it. The reflection
strategy used has changed in recent version of Go.
Any free form interface{} of any integer is treated
as `float64` this caused a bug where content-length-range
parsing failed to provide any value.

Fixes #3295
2016-11-21 04:15:26 -08:00
Harshavardhana 5197649081 utils: reduceErrs returns and validates quorum errors. (#3300)
This is needed as explained by @krisis

Lets say we have following errors.

```
[]error{nil, errFileNotFound, errDiskAccessDenied, errDiskAccesDenied}
```

Since the last two errors are filtered, the maximum is nil,
depending on map order.

Let's say we get nil from reduceErr. Clearly at this point
we don't have quorum nodes agreeing about the data and since
GetObject only requires N/2 (Read quorum) and isDiskQuorum
would have returned true. This is problematic and can lead to
undersiable consequences.

Fixes #3298
2016-11-21 01:47:26 -08:00
Harshavardhana 066f64d34a bootup: MetaVolume init should use isErrIngored helper. (#3303) 2016-11-21 01:46:55 -08:00
Krishna Srinivas afa4c7c3ef fs/multipart: Append multipart parts in a proper Go routine in background. (#3282) 2016-11-20 23:42:53 -08:00
Krishna Srinivas 38537c7df2 Print line numbers to give more info on the failed tests in ExecObjectLayerAPIAnonTest() (#3302) 2016-11-20 23:41:39 -08:00
Krishnan Parthasarathi eed9ab0464 XL: pickValidXLMeta should return error instead of panic'ing (#3277) 2016-11-20 20:56:44 -08:00
Harshavardhana 0b9f0d14a1 auth/rpc: Take remote disk offline after maximum allowed attempts. (#3288)
Disks when are offline for a long period of time, we should
ignore the disk after trying Login upto 5 times.

This is to reduce the network chattiness, this also reduces
the overall time spent on `net.Dial`.

Fixes #3286
2016-11-20 16:57:12 -08:00
Anis Elleuch ffbee70e04 Avoid removing 'tmp' directory inside '.minio.sys' (#3294) 2016-11-20 14:25:43 -08:00
Harshavardhana 2c3a2241e7 update: Change update notifier for new style banner. (#3289)
For binary releases and operating systems it would be

All operating systems.
```
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Minio is 25 days 12 hours 30 minutes old                           ┃
┃ Update: https://dl.minio.io/server/minio/release/linux-amd64/minio ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```

On docker.
```
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Minio is 25 days 12 hours 32 minutes old ┃
┃ Update: docker pull minio/minio          ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
2016-11-19 23:20:13 -08:00
Harshavardhana 1c47365445 xl/bootup: Upon bootup handle errors loading bucket and event configs. (#3287)
In a situation when we have lots of buckets the bootup time
might have slowed down a bit but during this situation the
servers quickly going up and down would be an in-transit state.

Certain calls which do not use quorum like `readXLMetaStat`
might return an error saying `errDiskNotFound` this is returned
in place of expected `errFileNotFound` which leads to an issue
where server doesn't start.

To avoid this situation we need to ignore them as safe values
to be ignored, for the most part these are network related errors.

Fixes #3275
2016-11-19 17:37:57 -08:00
Krishnan Parthasarathi 81eb7c0301 Have separate directory names for docker-compose and Dockerfile (#3292) 2016-11-19 10:42:22 -08:00
Anis Elleuch 4a926d19a7 Add GnuTLS config documentation for Windows (#3285) 2016-11-18 12:03:23 -08:00
Bala FA 05dc52a206 fix: use constants for access/secret key min/max length (#3271) 2016-11-16 17:33:55 -08:00
Harshavardhana c91d3791f9 heal: Add healing support for bucket, bucket metadata files. (#3252)
This patch implements healing in general but it is only used
as part of quickHeal().

Fixes #3237
2016-11-16 16:42:23 -08:00
Bala FA df8153859c docs: Add comments of using canonical import paths. (#3269) 2016-11-16 16:23:32 -08:00
Aditya Manthramurthy 2f43709f85 Prevent gorilla mux from normalizing path (Fixes #3256) (#3268)
Also fix test to not use a bucket name with a leading slash - this
causes the bucket name to become empty and go to an unintended API
call (listbuckets).
2016-11-16 16:23:22 -08:00
Bala FA 61d67a061c tests: add unit test for DeleteMultipleObjectsHandler. (#3267)
Fixes #3058
2016-11-16 09:46:09 -08:00
Harshavardhana 1b85302161 Fix spelling and golint errors. (#3266)
Fixes #3263
2016-11-15 18:14:23 -08:00
Anis Elleuch 6512d9978e Add support of user self signed certificates
Additionally add documentation about how to configure TLS with Minio
2016-11-15 16:15:23 -08:00
Aditya Manthramurthy e216201901 Remove control command from minio binary (Fixes #3264) (#3265) 2016-11-15 13:39:02 -08:00
Krishnan Parthasarathi 7abcededf2 Add tests for storage rpc handlers (#3262) 2016-11-15 12:12:06 -08:00
Harshavardhana 398421b9f5 xl/bootup: Server bootup shouldn't return for missing buckets. (#3255)
Ref #3196
2016-11-14 15:45:00 -08:00
Anis Elleuch b8f0d9352f signature-v2: encode path and query strings when calculating signature (#3253) 2016-11-14 10:23:21 -08:00
Harshavardhana f234c35020 lock: slice length of lock clients should be precisely urls. (#3254)
This patch fixes a possible bug, reproduced rarely only seen
once.

```
panic: runtime error: index out of range

goroutine 136 [running]:
panic(0xac1a40, 0xc4200120b0)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1(0xc4203d2240, 0x4, 0xc420474080, 0x4, 0x4, 0xc4202abb60, 0x0, 0xa86d01, 0xefcfc0, 0xc420417a80)
    /go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:170 +0x69b
created by github.com/minio/minio/vendor/github.com/minio/dsync.lock
    /go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0xf4
```
2016-11-14 10:18:56 -08:00
koolhead17 3bcf7b7593 Docs: Modified Docker instruction adding distributed image tag. (#3250) 2016-11-13 13:40:56 -08:00
Anis Elleuch 0c042a622a Return objects content types in Web List Objects handler (#3249) 2016-11-13 12:26:40 -08:00
Harshavardhana c57a358c9d Validate date header only for Signed{,V2} and StreamingSigned. (#3248)
Fixes #2941
2016-11-13 12:08:24 -08:00
Anis Elleuch 380d6c6435 Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information (#3247) 2016-11-13 11:48:02 -08:00
Harshavardhana 716316f711 Reduce number of envs and options from command line. (#3230)
Ref #3229

After review with @abperiasamy we decided to remove all the unnecessary options

- MINIO_BROWSER (Implemented as a security feature but now deemed obsolete
  since even if blocking access to MINIO_BROWSER, s3 API port is open)
- MINIO_CACHE_EXPIRY (Defaults to 72h)
- MINIO_MAXCONN (No one used this option and we don't test this)
- MINIO_ENABLE_FSMETA (Enable FSMETA all the time)

Remove --ignore-disks option - this option was implemented when XL layer
 would initialize the backend disks and heal them automatically to disallow
 XL accidentally using the root partition itself this option was introduced.

This behavior has been changed XL no longer automatically initializes
`format.json`  a HEAL is controlled activity, so ignore-disks is not
useful anymore. This change also addresses the problems of our documentation
going forward and keeps things simple. This patch brings in reduction of
options and defaulting them to a valid known inputs.  This patch also
serves as a guideline of limiting many ways to do the same thing.
2016-11-11 16:40:55 -08:00
Anis Elleuch 98e79b4b50 Use 307 StatusTemporaryRedirect to redirect clients from http to https with forcing them to keep HTTP Verb (#3246) 2016-11-11 15:04:51 -08:00
Anis Elleuch e2216a0936 exponentialBackoffWait returns zero after some retries, limit attempt number inside (#3245) 2016-11-11 09:49:30 -08:00
Harshavardhana e8d9d710d0 rpc: Protect racy access of internal auth states. (#3238)
Fixes #3232
2016-11-11 00:14:32 -08:00
Harshavardhana a8ab02a73a v4/presign: Fix presign requests when there are more signed headers. (#3222)
This fix removes a wrong logic which fails for requests which
have more signed headers in a presign request.

Fixes #3217
2016-11-10 21:57:15 -08:00
Aditya Manthramurthy cf022de4d5 Add tests for s3PeerAPIHandlers (Fixes #3067) (#3242) 2016-11-10 16:43:04 -08:00
Bala FA 3995e21c5b fix: Ignore object not found error in RemoveObject() in web-handler. (#3228)
Fixes #3181
2016-11-10 15:02:03 -08:00
Harshavardhana 2f7fb78692 rpc: Our rpcClient should make an attempt to reconnect. (#3221)
rpcClient should attempt a reconnect if the call fails
with 'rpc.ErrShutdown' this is needed since at times when
the servers are taken down and brought back up.

The hijacked connection from net.Dial is usually closed.

So upon first attempt rpcClient might falsely indicate that
disk to be down, to avoid this state make another dial attempt
to really fail.

Fixes #3206
Fixes #3205
2016-11-10 07:44:41 -08:00
Bala FA cf2fb30ac7 event: Add event notification for object deletion in web browser. (#3226)
Fixes #3201
2016-11-10 07:42:55 -08:00
Harshavardhana 51d1e6f75b Fix missing CompleteMultipartUpload Etag. (#3227)
Fixes #3224
2016-11-10 07:41:02 -08:00