maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
6251 commits 18 branches 277 tags 146 MiB
Commit graph

6251 commits

This branch
This branch
All branches
Author SHA1 Message Date
poornas be0c8b1ec0 Add support for missing Cache-Control directives (#8619) 
no-cache, only-if-cached and no-store directives are
being enforced in this PR.
 2019-12-07 07:49:36 +05:30
Harshavardhana 476111968a Update help messages with new wording (#8616) 
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
 2019-12-06 13:53:51 -08:00
Klaus Post f1e2e1cc9e S3 Select: Mismatched types don't match (#8608) 
When comparing for equality, if types cannot be matched, they don't match.
 2019-12-06 07:24:41 -08:00
Harshavardhana 97deba2a7c GetKVS should add new keys automatically, preserve order (#8612) 2019-12-06 16:13:10 +05:30
Nitish Tiwari 3df7285c3c Add Support for Cache and S3 related metrics in Prometheus endpoint (#8591) 
This PR adds support below metrics

- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3

Fixes #8549
 2019-12-05 23:16:06 -08:00
Aleksandr Petruhin d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana d8e3de0cae Ensure comment is always a valid key (#8604) 
Also fix LDAP leaky connection
 2019-12-05 18:17:42 +05:30
Harshavardhana c9940d8c3f Final changes to config sub-system (#8600) 
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
 2019-12-04 15:32:37 -08:00
Harshavardhana 794eb54da8 Export command prints turned-off sub-sys as comments (#8594) 
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
 2019-12-03 10:50:20 -08:00
Harshavardhana 2ab8d5e47f Enable build verification with race (#8583) 2019-12-02 15:54:26 -08:00
Clemens Wolff 947bc8c7d3 Update Azure Gateway to azure-storage-blob SDK (#8537) 
The azure-sdk-for-go/storage package has been in maintenance-
only mode since February 2018 (see [1]) and will be deprecated in the future.
 2019-12-02 09:32:19 -08:00
Harshavardhana 5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588) 
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
 2019-12-02 09:28:01 -08:00
Harshavardhana 0bfd20a8e3
Add client_id support for OpenID (#8579) 
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
 2019-11-29 21:37:42 -08:00
Klaus Post db3dbcce3a Print goroutines when shutdown hangs (#8574) 2019-11-29 19:40:08 +05:30
Harshavardhana b21835f195 Honor DurationSeconds properly for WebIdentity (#8581) 
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
 2019-11-29 18:57:54 +05:30
Klaus Post c7844fb1fb posix: cache disk ID for a short while (#8564) 
`*posix.getDiskID()` takes up to 30% of all CPU due to the `os.Stat` call on `GET` calls.

Before:
```
Operation: GET - Concurrency: 12
* Average: 1333.97 MB/s, 1365.99 obj/s, 1365.98 ops ended/s (4m59.975s)
* First Byte: Average: 7.801487ms, Median: 7.9974ms, Best: 1.9822ms, Worst: 110.0021ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1453.50 MB/s, 1488.38 obj/s, 1492.00 ops ended/s (1s)
* 50% Median: 1360.47 MB/s, 1393.12 obj/s, 1393.00 ops ended/s (1s)
* Slowest: 978.68 MB/s, 1002.17 obj/s, 1004.00 ops ended/s (1s)
```

After:
```
Operation: GET - Concurrency: 12
* Average: 1706.07 MB/s, 1747.02 obj/s, 1747.01 ops ended/s (4m59.985s)
* First Byte: Average: 5.797886ms, Median: 5.9959ms, Best: 996.3µs, Worst: 84.0007ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1830.03 MB/s, 1873.96 obj/s, 1872.00 ops ended/s (1s)
* 50% Median: 1735.04 MB/s, 1776.68 obj/s, 1776.00 ops ended/s (1s)
* Slowest: 994.94 MB/s, 1018.82 obj/s, 1018.00 ops ended/s (1s)
```

TLDR; `os.Stat` is not free.
 2019-11-29 02:57:14 -08:00
Harshavardhana 2ff8132e2d Fix the regression introduced in #8580 2019-11-27 16:13:07 -08:00
Harshavardhana 30e80d0a86
Add ReadFrom,WriteTo helpers for server config (#8580) 2019-11-27 09:36:08 -08:00
Klaus Post 87443af49e Cache Windows mount point info (#8572) 
It seems like every upload creates a new StorageInfo object.

This makes it very sensitive to slow code paths.

A particular slow function is `IsLikelyMountPoint` on Windows since `windows.GetVolumePathName` is slow.

We cache the result of this operation.

Before:

```
Operation: PUT - Concurrency: 12
* Average: 70.46 MB/s, 70.46 obj/s, 70.46 ops ended/s (59.58s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 77.67 MB/s, 77.67 obj/s, 79.00 ops ended/s (1s)
* 50% Median: 70.84 MB/s, 70.84 obj/s, 70.00 ops ended/s (1s)
* Slowest: 61.51 MB/s, 61.51 obj/s, 61.00 ops ended/s (1s)
```

After:
```
Operation: PUT - Concurrency: 12
* Average: 166.80 MB/s, 166.80 obj/s, 166.74 ops ended/s (59.8s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 183.83 MB/s, 183.83 obj/s, 186.00 ops ended/s (1s)
* 50% Median: 167.47 MB/s, 167.47 obj/s, 166.00 ops ended/s (1s)
* Slowest: 139.30 MB/s, 139.30 obj/s, 141.00 ops ended/s (1s)
```
 2019-11-27 20:10:13 +05:30
Harshavardhana 5d65428b29
Handle localhost distributed setups properly (#8577) 
Fixes an issue reported by @klauspost and @vadmeste

This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
 2019-11-26 11:42:10 -08:00
Harshavardhana 78eb3b78bb
Repurpose Get/SetConfig as import/export support (#8578) 2019-11-26 10:08:25 -08:00
Harshavardhana 720442b1a2
Add lock expiry handler to expire state locks (#8562) 2019-11-25 16:39:43 -08:00
Harshavardhana e542084c37
Add etcd path prefix for all IAM assets (#8569) 
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.

Fixes #8567
 2019-11-25 16:33:34 -08:00
poornas f931fc7bfb Fix retention enforcement in Compliance mode (#8556) 
In compliance mode, the retention date can be extended with 
governance bypass permissions
 2019-11-25 10:58:39 -08:00
Harshavardhana 0a56e33ce1 Preserve client sent config appropriately (#8566) 2019-11-22 13:46:05 -08:00
Harshavardhana c3771df641
Add bootstrap REST handler for verifying server config (#8550) 2019-11-22 12:45:13 -08:00
Klaus Post 890b493a2e Use random file name for write check (#8563) 
Since there may be multiple writes going on concurrently
Use a random file name for the write check to avoid collisions.
 2019-11-22 09:50:17 -08:00
Bruce Wang fb708b6b64 minimum Go version 1.13 (#8561) 2019-11-22 12:02:34 +05:30
Harshavardhana f96e902f63 Do not rely on quorum for StorageInfo() (#8557) 
StorageInfo() call is supposed to give each
server/disk information independently, rely
on this appropriately so that `mc admin info server`
gets correct information all the time.
 2019-11-21 22:08:41 -08:00
Sergey Morgunov 06bd1e582a Log in with OIDC not work with MINIO_DOMAIN (#8558) (#8559) 2019-11-21 17:45:15 -08:00
Harshavardhana fb43d64dc3
Fix healing on multiple zones (#8555) 
It is expected in zone healing underlying
callers should return appropriate errors
 2019-11-21 13:18:32 -08:00
Harshavardhana fd0fa4e5c5 Add NTP retention time (#8548) 2019-11-21 18:22:35 +05:30
Harshavardhana 4e9de58675 Avoid pointer based copy, instead use Clone() (#8547) 
This PR adds functional test to test expanded
cluster syntax.
 2019-11-21 17:54:51 +05:30
Harshavardhana 9565641b9b
Enhance ListObjectsV2 API to return UserDefined metadata (#8539) 2019-11-21 01:54:49 -08:00
poornas 4da68cfcfc Handle indexes correctly in DeleteMultipleObjectsHandler (#8544) 
Regression from #8509 which changes objectsToDelete entry
from a list to map. This will cause index out of range
panic if object is not selected for delete.
 2019-11-20 17:51:10 -08:00
poornas a9b87c0a16 Fix typo in event name (#8545) 2019-11-20 16:35:01 -08:00
Harshavardhana 5ac4b517c9
Order all keys in config (#8541) 
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
 2019-11-20 15:10:24 -08:00
poornas ca96560d56 Add object retention at the per object (#8528) 
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
 2019-11-20 13:18:09 -08:00
Nitish Tiwari cc1a84b62e Fix heal result item output to properly count drives and sets (#8543) 2019-11-20 10:10:26 -08:00
Harshavardhana 8392d2f510 Preserve same deploymentID on all zones (#8542) 2019-11-20 15:39:30 +05:30
Harshavardhana 347b29d059 Implement bucket expansion (#8509) 2019-11-19 17:42:27 -08:00
Harshavardhana 3a34d98db8
Initialize local nsLocker for gateway instances (#8540) 2019-11-19 16:45:35 -08:00
Harshavardhana 7cdb67680e
Add help with order of keys (#8535) 2019-11-19 13:48:13 -08:00
poornas 929951fd49 Add support for multiple admins (#8487) 
Also define IAM policies for administering
MinIO server
 2019-11-19 02:03:18 -08:00
Harshavardhana 13a3d17321
Do not add comments after migration (#8530) 
Also filter out empty comments from being
printed.
 2019-11-16 14:57:36 -08:00
Harshavardhana a8e156d6a5
Fix cache locking to use local namespace locking (#8529) 2019-11-16 13:44:28 -08:00
svistoi c9be601988 NATS TLS specify CA and client TLS authentication (#8389) 
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
 2019-11-15 09:13:23 -08:00
poornas 13e2b97ad9 Fix regression in caching on single PUT (#8526) 
Regression caused by #8120
 2019-11-15 15:46:27 +05:30
Ville Skyttä 95e5d7a9c3 Improve access and secret key validation error, sync with implementation (#8516) 2019-11-14 14:47:35 -08:00
Harshavardhana 32c200fe12 Fix console logger crash in gateway mode (#8525) 
This PR also fixes config migration only
for credentials and region which are valid
and set.

Also fix implicit `state="on"` behavior
 2019-11-14 14:19:57 -08:00