Commit graph

1421 commits

Author SHA1 Message Date
Kody A Kantor 06e30b5aa1
Skip building directio on platforms that don't support Direct IO (#9059) 2020-03-12 18:57:41 -07:00
Harshavardhana 603cf2a8bb
fix: broken gzip handling with Select API (#9128)
This PR fixes a regression introduced in a1c7c9ea73
2020-03-12 15:34:11 -07:00
Anis Elleuch fdf65aa9b9
heal: Add info about the next background healing round (#9122)
- avoid setting last heal activity when starting self-healing

This can be confusing to users thinking that the self healing
cycle was already performed.

- add info about the next background healing round
2020-03-11 23:00:31 -07:00
Aditya Manthramurthy cec8cdb35e
S3Select: Handle array selection in from clause (#9076) 2020-03-10 22:34:58 -07:00
kannappanr 33edb072a3
Add TopLocksAdminAction to diagsnostics canned policy (#9104) 2020-03-08 18:32:39 -07:00
ebozduman a1c7c9ea73
Matches s3 invalid compression format error for 'mc sql' (#9067) 2020-03-05 19:34:04 -08:00
kannappanr 2f2c7d91a8
Add new extended list of JWT keys from OpenID group (#9087)
https://www.iana.org/assignments/jwt/jwt.xhtml#claims
2020-03-05 05:05:36 -08:00
Klaus Post f1b2462193
Add goroutine profiles (#9078)
Allow downloading goroutine dump to help detect leaks
or overuse of goroutines.

Extensions are now type dependent.

Change `profiling` -> `profile` prefix, since that is what they are 
not the abstract concept.
2020-03-04 06:58:12 -08:00
Hamid 5b8975bf4b
Simplify redis access event format to faciliate parsing (#9046) 2020-02-26 09:23:32 -08:00
Kody A Kantor deb3911f5e
cpu package fails to build on illumos (#9036) 2020-02-25 10:58:18 -08:00
Harshavardhana 23a8411732
Add a generic Walk()'er to list a bucket, optinally prefix (#9026)
This generic Walk() is used by likes of Lifecyle, or
KMS to rotate keys or any other functionality which
relies on this functionality.
2020-02-25 21:22:28 +05:30
Harshavardhana ab7d3cd508
fix: Speed up multi-object delete by taking bulk locks (#8974)
Change distributed locking to allow taking bulk locks
across objects, reduces usually 1000 calls to 1.

Also allows for situations where multiple clients sends
delete requests to objects with following names

```
{1,2,3,4,5}
```

```
{5,4,3,2,1}
```

will block and ensure that we do not fail the request
on each other.
2020-02-21 11:29:57 +05:30
poornas 02a59a04d1
Fix error messages returned by (Put)GetObjectLegalHold (#9013)
fiixing some minor discrepancies between aws s3 responses
vs minio server
2020-02-19 08:15:48 +05:30
Harshavardhana 16a6e68d7b
fix: indicate PutBucketEncryption as a valid policy action (#9009) 2020-02-18 10:32:53 -08:00
Praveen raj Mani 1b427ddb69
Support for Kafka version in the config (#9001)
Add a field for the Kafka version in the config. The user can explicitly 
set the version of the Kafka cluster.

Fixes #8768
2020-02-17 07:56:33 +05:30
Krishnan Parthasarathi 9f298d2311
Omit empty algorithm tags in bucket encryption XML (#8987)
- Bucket encryption config returned by MinIO would always have the xml namespace
set
- Make unit tests in pkg/bucket/encryption more robust
2020-02-13 22:12:42 -08:00
Klaus Post e4020fb41f
SIMDJSON S3 select input (#8401) 2020-02-13 14:03:52 -08:00
Nitish Tiwari 64ec17b463
Fix Legal Hold request parsing (#8981)
AWS S3 doesn't enforce the URL in XMLNS, accordingly, removing the
URL in XMLNS for ObjectLegalHold.

This was found while testing https://github.com/minio/minio-go/pull/1226
2020-02-13 01:18:29 -08:00
Nitish Tiwari 7e819d00ea
Fix Error Code for ObjectTagging Parsing (#8971)
Also add Mint tests
2020-02-11 17:42:28 -08:00
Nitish Tiwari 33767266e7
Fix Tagging XML Unmarshalling (#8977)
AWS S3 doesn't strictly enforce the presence of URL in tagging XML.
This PR updates MinIO to behave similarly.

Fixes #8976
2020-02-11 14:35:33 -08:00
Anis Elleuch de924605a1
Import CSV parser library (#8927)
The CSV library code is imported from Go 1.13.6
2020-02-07 16:25:36 +05:30
Nitish Tiwari e5951e30d0
Add support for Object Tagging in LifeCycle configuration (#8880)
Fixes #8870

Co-Authored-By: Krishnan Parthasarathi <krisis@users.noreply.github.com>
2020-02-06 13:20:10 +05:30
Andreas Auernhammer 086fbb745e
fix and improve KMS server info (#8944)
This commit fixes typos in the displayed server info
w.r.t. the KMS and removes the update status.

For more information about why the update status
is removed see: PR #8943
2020-02-06 06:18:34 +05:30
Andreas Auernhammer 4f37c8ccf2
refine the KMS admin API (#8943)
This commit removes the `Update` functionality
from the admin API. While this is technically
a breaking change I think this will not cause
any harm because:
 - The KMS admin API is not complete, yet.
   At the moment only the status can be fetched.
 - The `mc` integration hasn't been merged yet.
   So no `mc` client could have used this API
   in the past.

The `Update`/`Rewrap` status is not useful anymore.
It provided a way to migrate from one master key version
to another. However, KES does not support the concept of
key versions. Instead, key migration should be implemented
as migration from one master key to another.

Basically, the `Update` functionality has been implemented just
for Vault.
2020-02-05 22:47:35 +05:30
Krishnan Parthasarathi 026265f8f7
Add support for bucket encryption feature (#8890)
- pkg/bucket/encryption provides support for handling bucket 
  encryption configuration
- changes under cmd/ provide support for AES256 algorithm only

Co-Authored-By: Poorna  <poornas@users.noreply.github.com>
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-05 15:12:34 +05:30
Anis Elleuch 52bdbcd046
Add new admin API to return Accounting Usage (#8689) 2020-02-04 18:20:39 -08:00
poornas 301c50b721
Add canned diagnostics policy for admin users (#8937) 2020-02-04 17:58:38 -08:00
Anis Elleuch 7432b5c9b2
Use user CAs in checkEndpoint() call (#8911)
The server info handler makes a http connection to other
nodes to check if they are up but does not load the custom
CAs in ~/.minio/certs/CAs.

This commit fix it.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-02 07:15:29 +05:30
poornas 881e983ed9
Fix Retention, ObjectLock, LegalHold struct namespaces correctly. (#8909)
Reverts #8903 to allow structs to be unmarshalled 
even if the namespace is missing.
2020-01-30 09:58:05 +05:30
Ashish Kumar Sinha 5bd0e95eef
Set default namespace for necessary structs (#8903) 2020-01-29 10:19:38 +05:30
Harshavardhana 0cbebf0f57 Rename pkg/{tagging,lifecycle} to pkg/bucket sub-directory (#8892)
Rename to allow for more such features to come in a more
proper hierarchical manner.
2020-01-27 14:12:34 -08:00
poornas 2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Klaus Post c7178d2066 Profiling: Add base, fix memory profiling (#8850)
For 'snapshot' type profiles, record a 'before' profile that can be used 
as `go tool pprof -base=before ...` to compare before and after.

"Before" profiles are included in the zipped package.

[`runtime.MemProfileRate`](https://golang.org/pkg/runtime/#pkg-variables) 
should not be updated while the application is running, so we set it at startup.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-21 15:49:25 -08:00
Harshavardhana e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Nitish Tiwari 61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
poornas 60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Harshavardhana 5aa5dcdc6d
lock: improve locker initialization at init (#8776)
Use reference format to initialize lockers
during startup, also handle `nil` for NetLocker
in dsync and remove *errorLocker* implementation

Add further tuning parameters such as

 - DialTimeout is now 15 seconds from 30 seconds
 - KeepAliveTimeout is not 20 seconds, 5 seconds
   more than default 15 seconds
 - ResponseHeaderTimeout to 10 seconds
 - ExpectContinueTimeout is reduced to 3 seconds
 - DualStack is enabled by default remove setting
   it to `true`
 - Reduce IdleConnTimeout to 30 seconds from
   1 minute to avoid idleConn build up

Fixes #8773
2020-01-10 02:35:06 -08:00
Harshavardhana 0a70bc24ac
Disallow only policy statements which are exactly same (#8785) 2020-01-09 19:29:57 -08:00
Praveen raj Mani 4cd1bbb50a This PR fixes two things (#8772)
- Stop spawning store replay routines when testing the notification targets
- Properly honor the target.Close() to clean the resources used

Fixes #8707

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-09 19:45:44 +05:30
Harshavardhana c2cde6beb5
policy: Allow duplicate statements with different effects (#8775)
This allows "Allow" and "Deny" conflicting statements,
where we evaluate to implicit "Deny".
2020-01-08 23:00:54 -08:00
Harshavardhana abc1c1070a Add custom policy claim name (#8764)
In certain organizations policy claim names
can be not just 'policy' but also things like
'roles', the value of this field might also
be *string* or *[]string* support this as well

In this PR we are still not supporting multiple
policies per STS account which will require a
more comprehensive change.
2020-01-08 17:21:58 -08:00
Harshavardhana 60813bef29
Allow proper setCount SLAs across zones (#8752)
Fixes scenario where zones are appropriately
handled, along with supporting overriding set
count. The new fix also ensures that we handle
the various setup types properly.

Update documentation to properly indicate the
behavior.

Fixes #8750

Co-authored-by: Nitish Tiwari <nitish@minio.io>
2020-01-07 09:13:44 -08:00
Harshavardhana 933c60bc3a Add crypto context errors (#8740)
Currently when connections to vault fail, client
perpetually retries this leads to assumptions that
the server has issues and masks the problem.

Re-purpose *crypto.Error* type to send appropriate
errors back to the client.
2020-01-06 16:15:22 -08:00
Harshavardhana 6695fd6a61
Add more context aware error for policy parsing errors (#8726)
In existing functionality we simply return a generic
error such as "MalformedPolicy" which indicates just
a generic string "invalid resource" which is not very
meaningful when there might be multiple types of errors
during policy parsing. This PR ensures that we send
these errors back to client to indicate the actual
error, brings in two concrete types such as

 - iampolicy.Error
 - policy.Error

Refer #8202
2020-01-03 11:28:52 -08:00
Harshavardhana 0b7bd024fb Fix dependencies graph for minio source compilation (#8717)
We had messy cyclical dependency problem with `mc`
due to dependencies in pkg/console, moved the pkg/console
to minio for more control and also to avoid any further
cyclical dependencies of `mc` clobbering up the
dependencies on server.

Fixes #8659
2019-12-31 09:36:13 +05:30
Haines Chan b2044dd22f Fix typo in comments (#8704) 2019-12-27 15:51:32 +05:30
kannappanr cd59a945d8
Fix: Policy action name for Get/Put BucketLifeCycle (#8698)
S3:GetBucketLifeCycle should be changed to S3:GetLifeCycleConfiguration
S3:PutBucketLifeCycle should be changed to S3:PutLifeCycleConfiguration
2019-12-25 22:04:46 -08:00
Bruce Wang c476b27a65 Comment typo "index max" to "index map" (#8700) 2019-12-24 21:57:43 -08:00
Harshavardhana 586614c73f fix: temp credentials shouldn't allow policy/group changes (#8675)
This PR fixes the issue where we might allow policy changes
for temporary credentials out of band, this situation allows
privilege escalation for those temporary credentials. We
should disallow any external actions on temporary creds
as a practice and we should clearly differentiate which
are static and which are temporary credentials.

Refer #8667
2019-12-19 14:21:21 -08:00
Harshavardhana 471a3a650a
fix: Don't allow to set unconfigured notification ARNs (#8643)
Fixes #8642
2019-12-13 12:36:45 -08:00