Commit graph

6375 commits

Author SHA1 Message Date
Krishnan Parthasarathi 026265f8f7
Add support for bucket encryption feature (#8890)
- pkg/bucket/encryption provides support for handling bucket 
  encryption configuration
- changes under cmd/ provide support for AES256 algorithm only

Co-Authored-By: Poorna  <poornas@users.noreply.github.com>
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 15:12:34 +05:30
Andreas Auernhammer f91c072f61
re-write the KMS get started guide (#8936)
This commit updates the KMS getting started guide
and replaces the legacy MinIO<-->Vault setup with a
MinIO<-->KES<-->Vault setup.

Therefore, add some architecture ASCII diagrams and
provide a step-by-step guide to setup Vault, KES and
MinIO such that MinIO can encrypt objects with KES +
Vault.

The legacy Vault guide has been moved to `./vault-legacy.md`.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 12:38:47 +05:30
Anis Elleuch 52bdbcd046
Add new admin API to return Accounting Usage (#8689) 2020-02-04 18:20:39 -08:00
poornas 301c50b721
Add canned diagnostics policy for admin users (#8937) 2020-02-04 17:58:38 -08:00
Harshavardhana e9c111c8d0
Avoid unnecessary statPart() calls in PutObjectPart (#8905)
Assume `xl.json` as the source of truth for all operations.
2020-02-04 10:04:37 +05:30
poornas 278a165674
Allow caching based on a configurable number of hits. (#8891)
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-04 09:10:01 +05:30
ebozduman a05af9bb98
Fixes several typos and clarifies some explanations in MySQL doc. (#8908) 2020-02-03 18:55:05 -08:00
Anis Elleuch e934c3e2a2
usage: Fix buckets count calculation when no object is present (#8929)
XL crawling wrongly returns a zero buckets count when
there are no objects uploaded in the server yet. The reason is 
data of the crawler of posix returns invalid result when all 
disks has zero objects.

A simple fix is to always pick the crawling result of the first 
disk but choose over the result of the disk which has the most 
objects in it.
2020-02-04 06:57:47 +05:30
Harshavardhana 2d295a31de
Avoid select inside a recursive function to avoid CPU spikes (#8923)
Additionally also allow configurable go-routines
2020-02-03 16:45:59 -08:00
Harshavardhana 9bbf5cb74f
fix: Avoid re-reading bucket names from etcd (#8924)
This helps improve performance when there are
1000+ bucket entries on etcd, improves the
startup time significantly.
2020-02-03 13:54:20 +05:30
Harshavardhana 680e493065
fix a crash in base64 buffer pool (#8925)
looks like 1024 buffer size is not enough in
all situations, use 8192 instead which
can satisfy all the rare situations that
may arise in base64 decoding.
2020-02-03 08:42:32 +05:30
poornas 1ea2449269
NAS gateway: fix notification initialization (#8920)
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-02 15:22:07 +05:30
Harshavardhana 7ce63b3078
fix: multi-delete API write quorum failures (#8926)
multi-delete API failed with write quorum errors
under following situations

- list of files requested for delete doesn't exist
  anymore can lead to quorum errors and failure
- due to usage of query param for paths, for really
  long paths MinIO server rejects these requests as
  malformed as unexpected.

This was reproduced with warp
2020-02-01 18:11:29 -08:00
Anis Elleuch 7432b5c9b2
Use user CAs in checkEndpoint() call (#8911)
The server info handler makes a http connection to other
nodes to check if they are up but does not load the custom
CAs in ~/.minio/certs/CAs.

This commit fix it.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-02 07:15:29 +05:30
Harshavardhana d76160c245
Initialize only one retry timer for all sub-systems (#8913)
Also make sure that we create buckets on all zones
successfully, do not run quick heal buckets if not
running with expansion.
2020-02-02 06:37:43 +05:30
poornas 5d838edcef
Fix panic in ServerInfoHandler when (#8915)
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-01 17:50:04 +05:30
poornas c9116e6bd7
trace - log request body (#8917) 2020-02-01 02:39:49 -08:00
Harshavardhana d7dc9aaf52
fix: remove response header timeout (#8919)
Adding respone header timeout seems to have
premature timeout like consequences which
leads to potential disconnections.
2020-02-01 08:31:55 +05:30
Harshavardhana bfe8a9bccc
jwt: Simplify JWT parsing (#8802)
JWT parsing is simplified by using a custom claim
data structure such as MapClaims{}, also writes
a custom Unmarshaller for faster unmarshalling.

- Avoid as much reflections as possible
- Provide the right types for functions as much
  as possible
- Avoid strings.Join, strings.Split to reduce
  allocations, rely on indexes directly.
2020-01-31 08:29:22 +05:30
Klaus Post 9990464cd5
Fix recursive deep scan of buckets (#8900) 2020-01-30 17:20:07 +05:30
poornas 881e983ed9
Fix Retention, ObjectLock, LegalHold struct namespaces correctly. (#8909)
Reverts #8903 to allow structs to be unmarshalled 
even if the namespace is missing.
2020-01-30 09:58:05 +05:30
Nesty92 f6a7d4d29b
Add CreateBucketError, ListMultipartUploads functional test (#8886) 2020-01-29 07:10:03 -08:00
Harshavardhana f98616dce7
heal: Optimize heal listing by avoiding batches (#8901)
Also limit the heal per object if there is incoming
requests by suspending heal for longer periods of time.
2020-01-29 12:05:44 +05:30
Ashish Kumar Sinha 5bd0e95eef
Set default namespace for necessary structs (#8903) 2020-01-29 10:19:38 +05:30
Li Yi 0414f01b6e doc: match code on sysctl -w vm.swappiness=1 (#8894) 2020-01-27 16:19:15 -08:00
Harshavardhana 0cbebf0f57 Rename pkg/{tagging,lifecycle} to pkg/bucket sub-directory (#8892)
Rename to allow for more such features to come in a more
proper hierarchical manner.
2020-01-27 14:12:34 -08:00
Harshavardhana 4cb6ebcfa2 test: print more relevant info in healing failure (#8895) 2020-01-27 14:56:36 +05:30
poornas 2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Minio Trusted aae337f5b5 Update yaml files to latest version RELEASE.2020-01-25T02-50-51Z 2020-01-25 02:57:23 +00:00
poornas a78e5d4763 Add missing error check in cache GetObjectNInfo (#8889) 2020-01-24 15:49:16 -08:00
Harshavardhana cf37c7997e Heal bucket only on missing drives in quorum (#8883)
MakeVol shouldn't be called in heal bucket
when bucket doesn't really exist in quorum.
2020-01-24 15:38:07 -08:00
Harshavardhana 1ffbb5c24c fix racy tests when editing xl.getDisks (#8879) 2020-01-23 11:50:09 -08:00
Harshavardhana b9c48e0ab0 fix return appropriate error for MakeBucket in federation (#8878) 2020-01-22 08:25:28 -08:00
Harshavardhana fe5d599802 fix: STS creds without "aud" should be honored with STS checks (#8868)
Fixes #8865
2020-01-22 15:09:46 +05:30
Aditya Manthramurthy 55063906b5 Fix group add/remove membership bug (#8877) 2020-01-21 19:00:41 -08:00
Klaus Post c7178d2066 Profiling: Add base, fix memory profiling (#8850)
For 'snapshot' type profiles, record a 'before' profile that can be used 
as `go tool pprof -base=before ...` to compare before and after.

"Before" profiles are included in the zipped package.

[`runtime.MemProfileRate`](https://golang.org/pkg/runtime/#pkg-variables) 
should not be updated while the application is running, so we set it at startup.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-21 15:49:25 -08:00
Harshavardhana f14f60a487 fix: Avoid double usage calculation on every restart (#8856)
On every restart of the server, usage was being
calculated which is not useful instead wait for
sufficient time to start the crawling routine.

This PR also avoids lots of double allocations
through strings, optimizes usage of string builders
and also avoids crawling through symbolic links.

Fixes #8844
2020-01-21 14:07:49 -08:00
Harshavardhana e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Anis Elleuch 3011692d93 doc: Add missing ``` in compression documentation (#8872) 2020-01-21 15:50:54 +05:30
Harshavardhana 86252ec7e1
fix: document _ENABLE for all notification targets (#8864)
Fixes #8863
2020-01-20 16:48:19 -08:00
Harshavardhana ef1aa870c5 cleanup unneeded files, update credits (#8858)
additionally add code of conduct
2020-01-20 10:38:58 -08:00
bijialin c260182412 zh_CN docs: add Expanding existing distributed setup (#8862) 2020-01-20 08:51:09 -08:00
Nitish Tiwari 61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
Forest Lovewood dd93eee1e3 Implement bucket caching for b2 gateway (#8820)
fixes #8739 #6806
2020-01-20 22:13:38 +05:30
Harshavardhana 7a400542ae Remove usage of GORPOXY let 'go' decide the defaults (#8855) 2020-01-19 09:17:42 -08:00
Harshavardhana 88286cf8d0 fix: support pre-sign signature for STS tokens (#8826)
Fixes #8391
2020-01-18 17:04:50 -08:00
Klaus Post 8cb6184f1d Fix erasure block allocation (#8851)
Small blocks are undersized when file size isn't divisible by the 
shard could leading to allocation in *reedsolomon.Split()*
2020-01-18 14:21:58 -08:00
Zhongpeng Lin b849fd7a75 remove gorilla/rpc replacement (#8852) 2020-01-18 14:21:13 -08:00
Harshavardhana 09ee145e9c gw/hdfs: indicate hdfs gateway is production ready (#8848) 2020-01-18 07:25:03 -08:00
Harshavardhana 04dfcbfdd7
update minio-go to latest master (#8849) 2020-01-18 04:05:17 -08:00