Commit graph

6358 commits

Author SHA1 Message Date
Harshavardhana d7dc9aaf52
fix: remove response header timeout (#8919)
Adding respone header timeout seems to have
premature timeout like consequences which
leads to potential disconnections.
2020-02-01 08:31:55 +05:30
Harshavardhana bfe8a9bccc
jwt: Simplify JWT parsing (#8802)
JWT parsing is simplified by using a custom claim
data structure such as MapClaims{}, also writes
a custom Unmarshaller for faster unmarshalling.

- Avoid as much reflections as possible
- Provide the right types for functions as much
  as possible
- Avoid strings.Join, strings.Split to reduce
  allocations, rely on indexes directly.
2020-01-31 08:29:22 +05:30
Klaus Post 9990464cd5
Fix recursive deep scan of buckets (#8900) 2020-01-30 17:20:07 +05:30
poornas 881e983ed9
Fix Retention, ObjectLock, LegalHold struct namespaces correctly. (#8909)
Reverts #8903 to allow structs to be unmarshalled 
even if the namespace is missing.
2020-01-30 09:58:05 +05:30
Nesty92 f6a7d4d29b
Add CreateBucketError, ListMultipartUploads functional test (#8886) 2020-01-29 07:10:03 -08:00
Harshavardhana f98616dce7
heal: Optimize heal listing by avoiding batches (#8901)
Also limit the heal per object if there is incoming
requests by suspending heal for longer periods of time.
2020-01-29 12:05:44 +05:30
Ashish Kumar Sinha 5bd0e95eef
Set default namespace for necessary structs (#8903) 2020-01-29 10:19:38 +05:30
Li Yi 0414f01b6e doc: match code on sysctl -w vm.swappiness=1 (#8894) 2020-01-27 16:19:15 -08:00
Harshavardhana 0cbebf0f57 Rename pkg/{tagging,lifecycle} to pkg/bucket sub-directory (#8892)
Rename to allow for more such features to come in a more
proper hierarchical manner.
2020-01-27 14:12:34 -08:00
Harshavardhana 4cb6ebcfa2 test: print more relevant info in healing failure (#8895) 2020-01-27 14:56:36 +05:30
poornas 2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Minio Trusted aae337f5b5 Update yaml files to latest version RELEASE.2020-01-25T02-50-51Z 2020-01-25 02:57:23 +00:00
poornas a78e5d4763 Add missing error check in cache GetObjectNInfo (#8889) 2020-01-24 15:49:16 -08:00
Harshavardhana cf37c7997e Heal bucket only on missing drives in quorum (#8883)
MakeVol shouldn't be called in heal bucket
when bucket doesn't really exist in quorum.
2020-01-24 15:38:07 -08:00
Harshavardhana 1ffbb5c24c fix racy tests when editing xl.getDisks (#8879) 2020-01-23 11:50:09 -08:00
Harshavardhana b9c48e0ab0 fix return appropriate error for MakeBucket in federation (#8878) 2020-01-22 08:25:28 -08:00
Harshavardhana fe5d599802 fix: STS creds without "aud" should be honored with STS checks (#8868)
Fixes #8865
2020-01-22 15:09:46 +05:30
Aditya Manthramurthy 55063906b5 Fix group add/remove membership bug (#8877) 2020-01-21 19:00:41 -08:00
Klaus Post c7178d2066 Profiling: Add base, fix memory profiling (#8850)
For 'snapshot' type profiles, record a 'before' profile that can be used 
as `go tool pprof -base=before ...` to compare before and after.

"Before" profiles are included in the zipped package.

[`runtime.MemProfileRate`](https://golang.org/pkg/runtime/#pkg-variables) 
should not be updated while the application is running, so we set it at startup.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-21 15:49:25 -08:00
Harshavardhana f14f60a487 fix: Avoid double usage calculation on every restart (#8856)
On every restart of the server, usage was being
calculated which is not useful instead wait for
sufficient time to start the crawling routine.

This PR also avoids lots of double allocations
through strings, optimizes usage of string builders
and also avoids crawling through symbolic links.

Fixes #8844
2020-01-21 14:07:49 -08:00
Harshavardhana e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Anis Elleuch 3011692d93 doc: Add missing ``` in compression documentation (#8872) 2020-01-21 15:50:54 +05:30
Harshavardhana 86252ec7e1
fix: document _ENABLE for all notification targets (#8864)
Fixes #8863
2020-01-20 16:48:19 -08:00
Harshavardhana ef1aa870c5 cleanup unneeded files, update credits (#8858)
additionally add code of conduct
2020-01-20 10:38:58 -08:00
bijialin c260182412 zh_CN docs: add Expanding existing distributed setup (#8862) 2020-01-20 08:51:09 -08:00
Nitish Tiwari 61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
Forest Lovewood dd93eee1e3 Implement bucket caching for b2 gateway (#8820)
fixes #8739 #6806
2020-01-20 22:13:38 +05:30
Harshavardhana 7a400542ae Remove usage of GORPOXY let 'go' decide the defaults (#8855) 2020-01-19 09:17:42 -08:00
Harshavardhana 88286cf8d0 fix: support pre-sign signature for STS tokens (#8826)
Fixes #8391
2020-01-18 17:04:50 -08:00
Klaus Post 8cb6184f1d Fix erasure block allocation (#8851)
Small blocks are undersized when file size isn't divisible by the 
shard could leading to allocation in *reedsolomon.Split()*
2020-01-18 14:21:58 -08:00
Zhongpeng Lin b849fd7a75 remove gorilla/rpc replacement (#8852) 2020-01-18 14:21:13 -08:00
Harshavardhana 09ee145e9c gw/hdfs: indicate hdfs gateway is production ready (#8848) 2020-01-18 07:25:03 -08:00
Harshavardhana 04dfcbfdd7
update minio-go to latest master (#8849) 2020-01-18 04:05:17 -08:00
Harshavardhana 23e46f9dba
log formatting only the first time (#8846) 2020-01-17 15:39:07 -08:00
Harshavardhana fc5213258e
posix: Do not take disk offline on I/O errors (#8836)
Choosing maxAllowedIOError is arbitrary and
prone to errors, when drives might be perfectly
capable of taking I/O with only few locations
return I/O error. This is a hindrance of sort
where backend filesystems like ZFS can automatically
fix and handle these scenarios.

The added problem with current approach that we
take the drive offline, making it virtually impossible
to bring it online without restart the server which
is not desirable on a busy cluster. Remove this state
such that let the backend return error appropriately
to caller and let the caller decide what to do with
the error.
2020-01-17 13:34:43 -08:00
kannappanr 005ebbb9b2 Typo: Add missing '<' character (#8845)
Fixes #8833
2020-01-17 13:31:46 -08:00
Anis Elleuch 017067e11f data-usage: Avoid crawling duplicated call (#8843)
This fix will also picks 3 and not 4 disks from a single erasure set.
2020-01-17 09:59:37 -08:00
Harshavardhana 2bb69033e5 http: fail appropriately and return standard Go error (#8837)
return http.ErrServerClosed with proper body when
server is shutting down, allowing more context instead
of just returning '503' which doesn't mean the same
thing.
2020-01-17 05:48:39 -08:00
Harshavardhana fca4ee84c9
gw/hdfs: listing should list directories properly (#8827)
Fixes #8822
2020-01-16 17:11:25 -08:00
poornas 60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Minio Trusted ba758361b3 Update yaml files to latest version RELEASE.2020-01-16T22-40-29Z 2020-01-16 22:47:41 +00:00
Harshavardhana c6b218e5df
fix: readiness should return 200 OK with first zone online (#8834) 2020-01-16 13:49:25 -08:00
Anis Elleuch c18fbdb29a posix: Remove a non needed nil check in DiskInfo() (#8830)
posix.DiskInfo() returns errFaultyDisk when posix is nil,
but there is no way that this would happen any time, therefore
removing un-needed code.
2020-01-16 11:27:50 -08:00
Harshavardhana b1ad99edbf
fix: avoid crash copy map before reading (#8825)
code of this form is always racy, when the
map itself is being written to as well

```
func (r Map) retMap() map[string]string {
     .. lock ..
     return r.internalMap
}

func (r Map) addMap(k, v string) {
     .. lock ..
     r.internalMap[k] = v
}
```

Anyone reading from `retMap()` is not protected
because of locking and we need to make sure
to avoid code in this manner. Always safe to
copy the map and return.
2020-01-16 01:35:30 -08:00
Minio Trusted 080e0c2323 Update yaml files to latest version RELEASE.2020-01-16T03-05-44Z 2020-01-16 03:11:40 +00:00
Anis Elleuch 935546d5ca xl: Implement MRF healing (#8470) 2020-01-15 18:30:32 -08:00
Harshavardhana 64fde1ab95
xl/zones: return errNoHealRequired when no heal is required (#8821)
Zone abstraction of object layer was returning `nil`
incorrectly under situations where disk healing is
not required. Returning `nil` is considered as healing
successful, which leads to unexpected ReloadFormat()
peer notification calls during startup.

This PR fixes this behavior properly for zones.
2020-01-15 17:19:13 -08:00
Anis Elleuch 169e8742fc mint: Use https for access to maven reposity (#8824)
Maven repository requires HTTPS now. This lead to issues
building mint image in aws-sdk-java & minio-java.

The PR fixes the issue and also bump aws sdk version in
aws-sdk-java to the latest.
2020-01-15 15:48:49 -08:00
Anis Elleuch 069876e262 xl: All nodes create meta volumes in its local disks (#8786)
Meta volumes directories, tmp/, background-ops/, etc..
undr .minio.sys are created when disks are formatted
but also when the cluster is started.

However using MakeVolBulk() is not appropriate in the
case of a user migrating from a version which does not
have .minio.sys/background-ops/. The reason is that
MakeVolBulk() exits early when an error is occured:
errVolumeExists in this case, which is expected since
some directories such as tmp/ already exist.

This commit will avoid use MakeVolBulk and use MakeVol
instead.

Also the PR will make each node creates meta volumes
in its local disks and stop relying on the first disk
since the first node could be offline.
2020-01-15 12:36:52 -08:00
Harshavardhana 442e1698cb
heal: Avoid spinning up object healing during startup (#8819)
auto-heal disks, metadata and buckets in background but
not objects, let the auto heal kick in for objects after
the cluster has been up for a while.
2020-01-15 01:08:39 -08:00