Commit graph

5082 commits

Author SHA1 Message Date
kannappanr
4f52d22c36
Fix make test failure (#6185)
Fix shadowing errors.
2018-07-24 14:17:58 -07:00
Kaan Kabalak
cbe8df198e Replace pointer with default cursor on filename hover (#6193) 2018-07-24 13:34:20 +05:30
Harshavardhana
157ed65c35 Fix healthcheck handler to check errors in local disks only (#6184)
Healthcheck handler in current implementation was
performing ListBuckets() to check for liveness of Minio
service. ListBuckets() implementation on the other hand
doesn't do quorum based listing and if one of the disks
returned error, an I/O error it would be lead to kubernetes
taking the minio pod down prematurely even if the disk
is not local to that minio server.

The reason is ListBuckets() call cannot be trusted to
provide us the valid information that we need, Minio is a
clustered application which is designed to handle disk
failures. Error on one of the disks doesn't mean the pod
should become fully non-operational.

This PR attempts to fix this by only checking for alive
disks which are local to each setup and also by simply
performing a Stat() operation, if the Stat() returned
error on all disks local to a particular server then
we can let kubernetes safely take it down, until then
we should be operational.
2018-07-23 12:21:25 -07:00
Minio Trusted
869018ad14 Update yaml files to latest version RELEASE.2018-07-23T18-34-49Z 2018-07-23 18:39:08 +00:00
Harshavardhana
5acc2a6db1 S3 gateway signature probe use a unique bucket (#6190)
This fixes an issue because someone is
using `probe-bucket-sign` bucket name in
region 'eu-central-1'
2018-07-23 10:16:56 -07:00
kannappanr
2cd14f567c Do not set Key and BucketName in ErrorResponse, if empty (#6174) 2018-07-23 14:09:09 +05:30
Harshavardhana
f1be356cc6 Do not use parallel deletes to avoid random I/O (#6178)
The current code for deleting 1000 objects simultaneously
causes significant random I/O, which on slower drives
leads to servers disconnecting in a distributed setup.

Simplify this by serially deleting and reducing the
chattiness of this operation.
2018-07-20 21:21:01 -07:00
kannappanr
76ddf4d32f Log x-amz-request-id as log and XML error response (#6173)
Currently, requestid field in logEntry is not populated, as the
requestid field gets set at the very end.
It is now set before regular handler functions. This is also
useful in setting it as part of the XML error response.

Travis build for ppc64le has been quite inconsistent and stays queued
for most of the time. Removing this build as part of Travis.yml for
the time being.
2018-07-20 18:46:32 -07:00
kannappanr
7b91bd71fe Remove ppc64le support from .travis.yml (#6180)
Travis CI for ppc64le stays in queued state.
Removing the support for now.
2018-07-20 18:39:55 -07:00
Harshavardhana
36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
kannappanr
963a70053b
Do not trace HTTP body of PostPolicyBuckethandler (#6177) 2018-07-20 07:37:46 -07:00
Anis Elleuch
9c5e971a58 Add new console/http loggers (#6066)
- Add console target logging, enabled by default.
- Add http target logging, which supports an endpoint
  with basic authentication (username/password are passed
  in the endpoint url itself)
- HTTP target logging is asynchronous and some logs can be
  dropped if channel buffer (10000) is full
2018-07-19 15:55:06 -07:00
Anis Elleuch
b1c9eb0e01 Disable splitting lines in pretty error messages (#6171)
In a small window, UI error tries to split lines for an eye candy
error message. However, since we show some docs.minio.io links in some
error messages, these links are actually broken and not easily selected
in a X terminal. This PR changes the behavior and won't split lines
anymore.
2018-07-19 15:49:02 -07:00
Nitish Tiwari
b8f4f26cf6 Add S3 gateway documentation (#6165)
Fixes #4830
2018-07-19 11:54:38 -07:00
kannappanr
43cc0096fa
Add support for deployment ID (#6144)
deployment ID helps in identifying a minio deployment in the case of remote
logging targets.
2018-07-18 20:17:35 -07:00
Anis Elleuch
e8a008f5b5 Better validation of all config file fields (#6090)
Add Validate() to serverConfig to call it at server
startup and in Admin SetConfig handler to minimize
errors scenario after server restart.
2018-07-18 11:22:29 -07:00
Andreas Auernhammer
758a80e39b crypto: add basic functionality for parsing SSE-C headers (#6148)
This commit adds basic support for SSE-C / SSE-C copy.
This includes functions for determining whether SSE-C
is requested by the S3 client and functions for parsing
such HTTP headers.

All S3 SSE-C parsing errors are exported such that callers
can pattern-match to forward the correct error to S3
clients.

Further the SSE-C related internal metadata entry-keys
are added by this commit.
2018-07-18 10:49:26 -07:00
wd256
3ec4738955 gcs: Use Pager to iterate results in ListObjectsV1/V2 (#6162)
Fixes #6052
2018-07-18 21:49:16 +05:30
Andreas Auernhammer
6c93c60424 crypto: add a basic KMS implementation (#6161)
This commit adds a basic KMS implementation for an
operator-specified SSE-S3 master key. The master key
is wrapped as KMS such that using SSE-S3 with master key
and SSE-S3 with KMS can use the same code.

Bindings for a remote / true KMS (like hashicorp vault)
will be added later on.
2018-07-17 22:40:34 -07:00
Krishna Srinivas
0c9f4c9092 formatMetaV1 should be "inherited" by disk format structs (#6134) 2018-07-16 20:26:42 -07:00
Andreas Auernhammer
289d6ce1d7 crypto: update SSE-S3 and SSE-C key derivation (#6152)
This commit updates the key derivation to reflect the
latest change of crypto/doc.go. This includes handling
the insecure legacy KDF.

Since #6064 is fixed, the 3. test case for object key
generation is enabled again.
2018-07-16 07:49:50 -07:00
Rob Girard
2a12e694f3 Changed command line examples (#6149)
Order for server:disk originally provided wouldn't stripe 
wide and may lead to availability issues.

Also added Short-form examples using {1...n} and a 
warning about {1..2} vs {1...3}
2018-07-14 20:48:38 +05:30
Harshavardhana
db26d3c9e2 Fix handling files at leaf attempting disk.ListDir() (#6155)
Return an ignorable error upon readDir() failure on
a file.
2018-07-14 12:11:48 +05:30
Stefan Husch
914c76a801 Implement lock.Open() to fix #5642 (#6150) 2018-07-13 18:37:02 -07:00
Krishnan Parthasarathi
a1ef90be52 gcs: Limit number of objects listed to max-keys (#6133) 2018-07-13 10:27:26 -07:00
Minio Trusted
7c4a41b933 Update yaml files to latest version RELEASE.2018-07-13T00-09-07Z 2018-07-13 00:43:51 +00:00
Nitish Tiwari
2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Andreas Auernhammer
adf7340394 fix size computation for en/decrypted objects (#6147)
This PR fixes the size calculation for encrypted multipart
objects.
2018-07-12 11:23:32 -07:00
Harshavardhana
b11a8eb3f4
Support multiple values for x-amz-meta header (#6145)
Fixes #5595
2018-07-12 09:40:14 -07:00
Andreas Auernhammer
15771ebe8d Fix decrypted object size and key derivation in CopyObjectPart (#6141)
This commit fixes the size calculation for multipart
objects. The decrypted size of an encrypted multipart
object is the sum of the decrypted part sizes.

Also fixes the key derivation in CopyObjectPart.
Instead of using the same object-encryption-key for each
part now an unique per-part key is derived.

Updates #6139
2018-07-12 21:59:56 +05:30
Praveen raj Mani
44865596db SignatureV4 validation with Metadata in the presignedUrl (#5894)
The `X-Amz-Meta-`/`X-Minio-Meta-` will now be recognized in query string also.

Fixes #5857 #5950
2018-07-10 20:27:10 -07:00
Nitish Tiwari
c9bc7e47b9
Update distributed docs (#6123)
We need to clarify that distributed Minio doesn't strictly need a
fresh drive, instead it just needs a fresh directory on the drive.
2018-07-10 07:32:24 +05:30
Anis Elleuch
be1700f595 Avoid startup abort when a notify target is down (#6126)
Minio server was preventing itself to start when any notification
target is down and not running. The PR changes the behavior by
avoiding startup abort in that case, so the user will still
be able to access Minio server using mc admin commands after
a restart or set config commands.
2018-07-10 07:20:31 +05:30
Minio Trusted
42c5b64e4e Update yaml files to latest version RELEASE.2018-07-10T01-42-11Z 2018-07-10 01:46:03 +00:00
Krishna Srinivas
40ed0d1f5d Support 1GB disk size (#6137)
Pivotal CF by default has 1GB disk option which causes minio to not start
2018-07-09 18:23:49 -07:00
Andreas Auernhammer
b181a693fb fix object rebinding SSE-C security guarantee violation (#6121)
This commit fixes a weakness of the key-encryption-key
derivation for SSE-C encrypted objects. Before this
change the key-encryption-key was not bound to / didn't
depend on the object path. This allows an attacker to
repalce objects - encrypted with the same
client-key - with each other.

This change fixes this issue by updating the
key-encryption-key derivation to include:
 - the domain (in this case SSE-C)
 - a canonical object path representation
 - the encryption & key derivation algorithm

Changing the object path now causes the KDF to derive a
different key-encryption-key such that the object-key
unsealing fails.
Including the domain (SSE-C) and encryption & key
derivation algorithm is not directly neccessary for this
fix. However, both will be included for the SSE-S3 KDF.
So they are included here to avoid updating the KDF
again when we add SSE-S3.

The leagcy KDF 'DARE-SHA256' is only used for existing
objects and never for new objects / key rotation.
2018-07-09 17:18:28 -07:00
Anis Elleuch
4ddc222f46 fix: Propagate bucket policy update in a distributed setup (#6135)
Commit 0d52126023 caused a regression in setting
a new bucket policy in a distributed setup. The reason is that gob is not able
to encode fields declared as interfaces unless we provide GobEncode() and GobDecode()
This PR adds them by using json marshaller and unmarshaller that are already
implemented for Functions interface.
2018-07-09 02:18:48 -07:00
Mike Scarlett
c310cbbe89 Update comments regarding GCS component count (#6131) 2018-07-06 17:07:11 -07:00
Harshavardhana
0ef0d7e685 pkg/certs: On windows watch for directory changes to load certs (#6128)
This PR fixes an issue when configuring Minio TLS on windows
2018-07-05 16:33:37 -07:00
wksw
c62813c887 completed sample config and fix wrong json format (#6096)
missing worm configuration and wrong json format
2018-07-05 13:40:07 +05:30
poornas
1da362538b cache: allow ellipse style entries for MINIO_CACHE_DRIVES (#6088)
Fixes #5863
2018-07-03 16:54:10 -07:00
Krishna Srinivas
e40a5e05e1 Do notification in background to not block S3 client REST calls (#6005) 2018-07-03 11:09:36 -07:00
ebozduman
b0b0fb4c8d Validate Minio config.json file on the client side (#6067) 2018-07-03 11:07:46 -07:00
kannappanr
726e75611e
Do not log BucketNotFound errors on minio console (#6114) 2018-07-03 11:04:55 -07:00
Rafael Peria de Sene
317e648c0d Add support for ppc64le (#6116)
Signed-off-by: Rafael Peria de Sene <rpsene@br.ibm.com>
2018-07-03 09:42:19 -07:00
Harshavardhana
80b3e9cb03 use appropriate HTTP status for storage is full (#6117) 2018-07-03 09:40:14 -07:00
Harshavardhana
6c85706c24 Use GetSourceIP for source ip as request params (#6109)
Fixes #6108
2018-07-02 14:40:18 -07:00
Harshavardhana
d7ced9a8b5 Upgrade gjson to bring in new performance improvements (#6106) 2018-07-02 13:31:11 -07:00
Praveen raj Mani
360f3f9335 Checking the existence of the bucket in DeleteObjectHandler (#6085)
Fixes #6077
2018-06-30 22:35:43 -07:00
wd256
25f9b0bc3b Handle ListObjectsV2 start-after parameter in ObjectLayer (#6078) 2018-07-01 09:52:45 +05:30