maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
6217 commits 18 branches 277 tags 146 MiB
Commit graph

2187 commits

Author SHA1 Message Date
Harshavardhana c364f0af6c Start using custom HTTP transport for webhook endpoints (#8630) 
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
 2019-12-12 06:53:50 -08:00
Anis Elleuch 555969ee42 Add data usage collect with its new admin API (#8553) 
Admin data usage info API returns the following

(Only FS & XL, for now)

- Number of buckets
- Number of objects
- The total size of objects
- Objects histogram
- Bucket sizes
 2019-12-12 06:02:37 -08:00
Ashish Kumar Sinha e2c5d29017 Bucket,Object count & Usage removed if set to default (#8638) 2019-12-11 21:56:47 -08:00
Harshavardhana fa00a84709
Avoid crashes on peers if IAMSys is not initialized (#8636) 2019-12-11 20:46:57 -08:00
kannappanr d266b3a066
Admin Info: Modify Uptime to return seconds (#8635) 2019-12-11 17:56:02 -08:00
Ashish Kumar Sinha 24fb1bf258 New Admin Info (#8497) 2019-12-11 14:27:03 -08:00
Harshavardhana 8b803491af
fix: CacheOpts parsing tests (#8632) 2019-12-11 13:26:18 -08:00
Harshavardhana 10b2f15f6f Add randomize sleep times for lock checkers (#8628) 2019-12-11 10:57:05 -08:00
Harshavardhana 3e9ab5f4a9
Fix k8s replica set deployment (#8629) 
In replica sets, hosts resolve to localhost
IP automatically until the deployment fully
comes up. To avoid this issue we need to
wait for such resolution.
 2019-12-10 20:28:22 -08:00
Krishna Srinivas 3b67f629a4 Retry peer notification of events (#8621) 2019-12-09 05:29:37 -08:00
poornas 3c30e4503d Cache only the range requested for range GETs (#8599) 2019-12-08 13:58:04 -08:00
poornas 8390bc26db Fix cache hit metrics. (#8617) 2019-12-07 23:14:33 +05:30
Nitish Tiwari 24ad59316d
Use atomic.Uint64 for gateway metrics count instead of mutex (#8615) 2019-12-07 11:21:52 +05:30
poornas be0c8b1ec0 Add support for missing Cache-Control directives (#8619) 
no-cache, only-if-cached and no-store directives are
being enforced in this PR.
 2019-12-07 07:49:36 +05:30
Harshavardhana 476111968a Update help messages with new wording (#8616) 
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
 2019-12-06 13:53:51 -08:00
Harshavardhana 97deba2a7c GetKVS should add new keys automatically, preserve order (#8612) 2019-12-06 16:13:10 +05:30
Nitish Tiwari 3df7285c3c Add Support for Cache and S3 related metrics in Prometheus endpoint (#8591) 
This PR adds support below metrics

- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3

Fixes #8549
 2019-12-05 23:16:06 -08:00
Aleksandr Petruhin d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana d8e3de0cae Ensure comment is always a valid key (#8604) 
Also fix LDAP leaky connection
 2019-12-05 18:17:42 +05:30
Harshavardhana c9940d8c3f Final changes to config sub-system (#8600) 
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
 2019-12-04 15:32:37 -08:00
Harshavardhana 794eb54da8 Export command prints turned-off sub-sys as comments (#8594) 
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
 2019-12-03 10:50:20 -08:00
Harshavardhana 2ab8d5e47f Enable build verification with race (#8583) 2019-12-02 15:54:26 -08:00
Clemens Wolff 947bc8c7d3 Update Azure Gateway to azure-storage-blob SDK (#8537) 
The azure-sdk-for-go/storage package has been in maintenance-
only mode since February 2018 (see [1]) and will be deprecated in the future.
 2019-12-02 09:32:19 -08:00
Harshavardhana 5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588) 
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
 2019-12-02 09:28:01 -08:00
Harshavardhana 0bfd20a8e3
Add client_id support for OpenID (#8579) 
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
 2019-11-29 21:37:42 -08:00
Klaus Post db3dbcce3a Print goroutines when shutdown hangs (#8574) 2019-11-29 19:40:08 +05:30
Harshavardhana b21835f195 Honor DurationSeconds properly for WebIdentity (#8581) 
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
 2019-11-29 18:57:54 +05:30
Klaus Post c7844fb1fb posix: cache disk ID for a short while (#8564) 
`*posix.getDiskID()` takes up to 30% of all CPU due to the `os.Stat` call on `GET` calls.

Before:
```
Operation: GET - Concurrency: 12
* Average: 1333.97 MB/s, 1365.99 obj/s, 1365.98 ops ended/s (4m59.975s)
* First Byte: Average: 7.801487ms, Median: 7.9974ms, Best: 1.9822ms, Worst: 110.0021ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1453.50 MB/s, 1488.38 obj/s, 1492.00 ops ended/s (1s)
* 50% Median: 1360.47 MB/s, 1393.12 obj/s, 1393.00 ops ended/s (1s)
* Slowest: 978.68 MB/s, 1002.17 obj/s, 1004.00 ops ended/s (1s)
```

After:
```
Operation: GET - Concurrency: 12
* Average: 1706.07 MB/s, 1747.02 obj/s, 1747.01 ops ended/s (4m59.985s)
* First Byte: Average: 5.797886ms, Median: 5.9959ms, Best: 996.3µs, Worst: 84.0007ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1830.03 MB/s, 1873.96 obj/s, 1872.00 ops ended/s (1s)
* 50% Median: 1735.04 MB/s, 1776.68 obj/s, 1776.00 ops ended/s (1s)
* Slowest: 994.94 MB/s, 1018.82 obj/s, 1018.00 ops ended/s (1s)
```

TLDR; `os.Stat` is not free.
 2019-11-29 02:57:14 -08:00
Harshavardhana 2ff8132e2d Fix the regression introduced in #8580 2019-11-27 16:13:07 -08:00
Harshavardhana 30e80d0a86
Add ReadFrom,WriteTo helpers for server config (#8580) 2019-11-27 09:36:08 -08:00
Harshavardhana 5d65428b29
Handle localhost distributed setups properly (#8577) 
Fixes an issue reported by @klauspost and @vadmeste

This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
 2019-11-26 11:42:10 -08:00
Harshavardhana 78eb3b78bb
Repurpose Get/SetConfig as import/export support (#8578) 2019-11-26 10:08:25 -08:00
Harshavardhana 720442b1a2
Add lock expiry handler to expire state locks (#8562) 2019-11-25 16:39:43 -08:00
Harshavardhana e542084c37
Add etcd path prefix for all IAM assets (#8569) 
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.

Fixes #8567
 2019-11-25 16:33:34 -08:00
poornas f931fc7bfb Fix retention enforcement in Compliance mode (#8556) 
In compliance mode, the retention date can be extended with 
governance bypass permissions
 2019-11-25 10:58:39 -08:00
Harshavardhana 0a56e33ce1 Preserve client sent config appropriately (#8566) 2019-11-22 13:46:05 -08:00
Harshavardhana c3771df641
Add bootstrap REST handler for verifying server config (#8550) 2019-11-22 12:45:13 -08:00
Klaus Post 890b493a2e Use random file name for write check (#8563) 
Since there may be multiple writes going on concurrently
Use a random file name for the write check to avoid collisions.
 2019-11-22 09:50:17 -08:00
Harshavardhana f96e902f63 Do not rely on quorum for StorageInfo() (#8557) 
StorageInfo() call is supposed to give each
server/disk information independently, rely
on this appropriately so that `mc admin info server`
gets correct information all the time.
 2019-11-21 22:08:41 -08:00
Sergey Morgunov 06bd1e582a Log in with OIDC not work with MINIO_DOMAIN (#8558) (#8559) 2019-11-21 17:45:15 -08:00
Harshavardhana fb43d64dc3
Fix healing on multiple zones (#8555) 
It is expected in zone healing underlying
callers should return appropriate errors
 2019-11-21 13:18:32 -08:00
Harshavardhana fd0fa4e5c5 Add NTP retention time (#8548) 2019-11-21 18:22:35 +05:30
Harshavardhana 4e9de58675 Avoid pointer based copy, instead use Clone() (#8547) 
This PR adds functional test to test expanded
cluster syntax.
 2019-11-21 17:54:51 +05:30
Harshavardhana 9565641b9b
Enhance ListObjectsV2 API to return UserDefined metadata (#8539) 2019-11-21 01:54:49 -08:00
poornas 4da68cfcfc Handle indexes correctly in DeleteMultipleObjectsHandler (#8544) 
Regression from #8509 which changes objectsToDelete entry
from a list to map. This will cause index out of range
panic if object is not selected for delete.
 2019-11-20 17:51:10 -08:00
Harshavardhana 5ac4b517c9
Order all keys in config (#8541) 
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
 2019-11-20 15:10:24 -08:00
poornas ca96560d56 Add object retention at the per object (#8528) 
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
 2019-11-20 13:18:09 -08:00
Nitish Tiwari cc1a84b62e Fix heal result item output to properly count drives and sets (#8543) 2019-11-20 10:10:26 -08:00
Harshavardhana 8392d2f510 Preserve same deploymentID on all zones (#8542) 2019-11-20 15:39:30 +05:30
Harshavardhana 347b29d059 Implement bucket expansion (#8509) 2019-11-19 17:42:27 -08:00