Commit graph

7407 commits

Author SHA1 Message Date
Harshavardhana fb78283c0a add GOMAXPROCS back 2021-02-08 22:29:10 -08:00
Harshavardhana f07c9c58e7 fix: handle setIndexes properly 2021-02-08 22:25:06 -08:00
Harshavardhana bc89e47066 remove GOMAXPROCS requirement 2021-02-08 21:54:00 -08:00
Harshavardhana 0615d85384 heal sets with optional prefix input 2021-02-05 11:15:42 -08:00
Harshavardhana 42157eb218 listing also match sets index for proper quorum 2021-02-01 22:48:08 -08:00
Harshavardhana fa1cd6dcce heal multiple buckets in parallel 2021-02-01 22:45:34 -08:00
Harshavardhana 745a4b31ba add support for concurrent heals 2021-01-29 21:59:49 -08:00
Harshavardhana 5151c429e4 fix: add api level throttler for LIST calls 2021-01-28 22:59:15 -08:00
Klaus Post dc1a46e5d2 crawler: Stream bucket usage cache data (#11068)
Stream bucket caches to storage and through RPC calls.
2021-01-25 21:27:28 -08:00
Harshavardhana 8724d49116 implement Heal sets API to heal erasure sets independently 2021-01-24 19:05:56 -08:00
Anis Elleuch 006c69f716 make: Add hotfix target to generate hotfix binaries (#11053)
hotfix target will fetch the release tag prior to the latest commit and create a binary
with the same release tag plus '.hotfix' suffix

e.g.   RELEASE.2020-12-03T05-49-24Z.hotfix
2021-01-22 11:01:46 -08:00
Harshavardhana 28974fb5da fix: release locks if the client timedout (#11030)
situations where client indeed timedout there was
a potential to falsely think that lock is still
active.
2020-12-05 00:01:56 -08:00
Harshavardhana 123cfa7573 re-route requests if IAM is not initialized (#10850) 2020-11-08 18:35:33 -08:00
Klaus Post 2439d4fb3c Don't retain context in locker (#10515)
Use the context for internal timeouts, but disconnect it from outgoing
calls so we always receive the results and cancel it remotely.
2020-11-04 10:08:58 -08:00
Harshavardhana 6bd9057bb1 initialize IAM after etcd has initialized 2020-11-03 08:49:27 -08:00
Harshavardhana 2d878b7081 allow requests to be proxied when server is booting up (#10790)
when server is booting up there is a possibility
that users might see '503' because object layer
when not initialized, then the request is proxied
to neighboring peers first one which is online.
2020-10-31 19:38:23 -07:00
Harshavardhana 0570c21671 fix: replaced drive properly by healing the entire drive
Bonus fixes, we do not need reload format anymore
as the replaced drive is healed locally we only need
to ensure that drive heal reloads the drive properly.

We preserve the UUID of the original order, this means
that the replacement in `format.json` doesn't mean that
the drive needs to be reloaded into memory anymore.

fixes #10791
2020-10-31 00:30:14 -07:00
Klaus Post 2c0a81bc91 Optimize decryptObjectInfo (#10726)
`decryptObjectInfo` is a significant bottleneck when listing objects.

Reduce the allocations for a significant speedup.

https://github.com/minio/sio/pull/40

```
λ benchcmp before.txt after.txt
benchmark                          old ns/op     new ns/op     delta
Benchmark_decryptObjectInfo-32     24260928      808656        -96.67%

benchmark                          old MB/s     new MB/s     speedup
Benchmark_decryptObjectInfo-32     0.04         1.24         31.00x

benchmark                          old allocs     new allocs     delta
Benchmark_decryptObjectInfo-32     75112          48996          -34.77%

benchmark                          old bytes     new bytes     delta
Benchmark_decryptObjectInfo-32     287694772     4228076       -98.53%
```
2020-10-31 00:19:53 -07:00
Klaus Post b0698b4b98 rest client: Expect context timeouts for locks (#10782)
Add option for rest clients to not mark a remote offline for context timeouts.

This can be used if context timeouts are expected on the call.
2020-10-29 10:15:35 -07:00
Harshavardhana 7ec6214e6e fix: A possible crash when fi.Erasure.Distribution is empty (#10779) 2020-10-28 21:00:36 -07:00
Krishna Srinivas f53c5a020e
fix: heal object shards with ec.index and ec.distribution mismatches (#10773)
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-10-28 00:10:20 -07:00
Harshavardhana 5b30bbda92
fix: add more protection distribution to match EcIndex (#10772)
allows for more stricter validation in picking up the right
set of disks for reconstruction.
2020-10-28 00:09:15 -07:00
Shireesh Anjal 858e2a43df
Remove logging info from OBDInfoHandler (#10727)
A lot of logging data is counterproductive. A better implementation with
precise useful log data can be introduced later.
2020-10-27 17:41:48 -07:00
Kaloyan Raev df9894e275
avoid caching http ranges in background goroutine (#10724) 2020-10-26 23:04:48 -07:00
Minio Trusted ca77ee1c0e Update yaml files to latest version RELEASE.2020-10-27T04-03-55Z 2020-10-27 04:20:10 +00:00
Krishna Srinivas 592f2f23a3
fix: heal rejects objects with disk re-ordering issue (#10766) 2020-10-26 18:48:47 -07:00
Krishna Srinivas c49a80db41
fix: use meta.Erasure.Index for GetObject() to reconstruct object (#10764) 2020-10-26 16:19:42 -07:00
Poorna Krishnamoorthy 46275c6547
cache: rename function declarations (#10763) 2020-10-26 15:41:24 -07:00
Poorna Krishnamoorthy 0994ed9783
cache: fix call in GetObjectNInfo (#10762)
Fixes: #10751
2020-10-26 12:30:40 -07:00
Anis Elleuch eb95353cb1
fix: Get/HeadObject return 404 on non quorum objects (#10753) 2020-10-26 10:30:46 -07:00
Harshavardhana 029758cb20
fix: retain the previous UUID for newly replaced drives (#10759)
only newly replaced drives get the new `format.json`,
this avoids disks reloading their in-memory reference
format, ensures that drives are online without
reloading the in-memory reference format.

keeping reference format in-tact means UUIDs
never change once they are formatted.
2020-10-26 10:29:29 -07:00
Harshavardhana 649035677f fix: arm64 Dockerfile with missing \ for multiple ENVs
fixes #10758
2020-10-25 13:52:55 -07:00
Harshavardhana 646d6917ed
turn-off checking for updates completely if MINIO_UPDATE=off (#10752) 2020-10-24 22:39:44 -07:00
Harshavardhana d9db7f3308
expire lockers if lockers are offline (#10749)
lockers currently might leave stale lockers,
in unknown ways waiting for downed lockers.

locker check interval is high enough to safely
cleanup stale locks.
2020-10-24 13:23:16 -07:00
Harshavardhana 6a8c62f9fd
make sure to preserve UUID from reference format (#10748)
reference format should be source of truth
for inconsistent drives which reconnect,
add them back to their original position

remove automatic fix for existing offline
disk uuids
2020-10-24 13:23:08 -07:00
Harshavardhana 4442382c16 update STS examples to use latest v7 APIs 2020-10-24 11:16:59 -07:00
Anis Elleuch 00124c56d9
erasure: Commit data before xl.meta in RenameData() (#10734)
This will reduce the chance to have updated xl.meta without data.
2020-10-23 21:54:58 -07:00
Anis Elleuch 2c32c2149e
tests: Avoid running TestNSRace in short test mode (#10735) 2020-10-23 21:23:12 -07:00
Harshavardhana 734f258878
fix: slow down auto healing more aggressively (#10730)
Bonus fixes

- logging improvements to ensure that we don't use
  `go logger.LogIf` to avoid runtime.Caller missing
  the function name. log where necessary.
- remove unused code at erasure sets
2020-10-22 13:36:24 -07:00
Anis Elleuch 0e0c53bba4
tests: Lower expectation in addr selection in rand cache dialer (#10739)
Test TestDialContextWithDNSCacheRand was failing sometimes because it depends
on a random selection of addresses when testing random DNS resolution from cache.

Lower addr selection exception to 10%
2020-10-22 09:35:32 -07:00
Poorna Krishnamoorthy 5cc23ae052
validate if iam store is initialized (#10719)
Fixes panic - regression from d6d770c1b1
2020-10-20 21:28:24 -07:00
Anis Elleuch 6fd088f448
docs: Update ILM doc with versioning features (#10714) 2020-10-20 09:23:27 -07:00
Harshavardhana d6d770c1b1 initialize object layer right after config has loaded 2020-10-19 22:04:59 -07:00
Harshavardhana b07df5cae1
initialize IAM as soon as object layer is initialized (#10700)
Allow requests to come in for users as soon as object
layer and config are initialized, this allows users
to be authenticated sooner and would succeed automatically
on servers which are yet to fully initialize.
2020-10-19 09:54:40 -07:00
Harshavardhana c107728676
fix: s3 gateway DNS cache initialization (#10706)
fixes #10705
2020-10-19 01:34:23 -07:00
Minio Trusted ba5215561f Update yaml files to latest version RELEASE.2020-10-18T21-54-12Z 2020-10-18 22:15:30 +00:00
Márk Sági-Kazár 4eb45c9a0f
fix: dex getting started guide URL (#10701) 2020-10-17 16:47:28 -07:00
Harshavardhana 187129a907 fix comment in bucket bandwidth package 2020-10-17 00:38:54 -07:00
Anis Elleuch 284a2b9021
ilm: Send delete marker creation event when appropriate (#10696)
Before this commit, the crawler ILM will always send object delete event
notification though this is wrong.
2020-10-16 21:22:12 -07:00
Ritesh H Shukla 0b53e30ecb
Clean up monitor on delete bucket (#10698) 2020-10-16 17:59:31 -07:00