Commit graph

8858 commits

Author SHA1 Message Date
Harshavardhana f4b55ea7a7 update console to v0.12.2 2021-11-08 19:21:45 -08:00
Aditya Manthramurthy 8dfd1f03e9
fix: IAM initialization crash with etcd store (#13612) 2021-11-08 12:55:27 -08:00
Harshavardhana acf26c5ab7 re-arrange metacache struct to be optimal (#13609) 2021-11-08 10:26:08 -08:00
Klaus Post d9800c8135
fix: make sure to log panic in handlers (#13611) 2021-11-08 09:28:13 -08:00
Harshavardhana 02bef7560f add missing Copyright header 2021-11-08 09:13:15 -08:00
Daniel A. Ochoa 07dd0692b6
Fix hdfs gateway concurrent map writes (#13596)
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-11-08 09:07:58 -08:00
Klaus Post 4f3317effe
Close stream on panic (#13605)
Always close streamHTTPResponse on panic on main thread to avoid 
write/flush after response handler has returned.
2021-11-08 08:41:27 -08:00
Klaus Post 9afdbe3648
fix: RLock UID memory leak (#13607)
UID were misnamed in RLock, leading to memory buildup.

Regression in #13430
2021-11-08 07:35:50 -08:00
Aditya Manthramurthy fe0df01448
fix: locking in some situations for IAM store (#13595)
- Fix a bug where read locks were taken instead of write locks in some situations
- Remove an unnecessary lock when updating based on notifications.
2021-11-07 17:42:32 -08:00
Harshavardhana 12e6907512
apply spelling checks for US locale (#13599) 2021-11-07 01:22:59 -08:00
Harshavardhana 5aef492b4c update disk-caching design guide 2021-11-07 01:21:34 -08:00
Harshavardhana 5d7ed8ff7d update S3 gateway limitation docs 2021-11-06 23:24:48 -07:00
Harshavardhana b1754fc5ff update go.mod and CREDITS 2021-11-06 11:39:17 -07:00
Harshavardhana 19bbf3e142 update CREDITS 2021-11-05 13:53:21 -07:00
jiangfucheng e1755275a0
resume heal from previous object instead of bucket after server restart (#13581) 2021-11-05 13:10:41 -07:00
Harshavardhana 520037e721
move to jwt-go v4 with correct releases (#13586) 2021-11-05 12:20:08 -07:00
Minio Trusted cbb0828ab8 Update yaml files to latest version RELEASE.2021-11-05T09-16-26Z 2021-11-05 10:03:56 +00:00
Andreas Auernhammer 8774d10bdf
sts: always verify the key usage of client certificates (#13583)
This commit makes the MinIO server behavior more consistent
w.r.t. key usage verification.

When MinIO verifies the client certificates it also checks
that the client certificate is valid of client authentication
(or any (i.e. wildcard) usage).

However, the MinIO server used to not verify the client key usage
when client certificate verification was disabled.
Now, the MinIO server verifies the client key usage even when
client certificate verification has been disabled. This makes
the MinIO behavior more consistent from a client's perspective.

Now, a client certificate has to be valid for client authentication
in all cases.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-11-05 02:16:26 -07:00
Harshavardhana df9f479d58 update console to v0.12.1 2021-11-04 16:43:59 -07:00
Harshavardhana 8bb52c9c2a
fix: ignore disks that are available but not writable (#13585)
This is to allow replacing drives while some drives
while available are not writable.
2021-11-04 16:42:49 -07:00
Aditya Manthramurthy 947c423824
fix: user DN filtering that causes some unnecessary logs (#13584)
Additionally, remove the unnecessary `isUsingLookupBind` field in the LDAP struct
2021-11-04 13:11:20 -07:00
Harshavardhana c3d24fb26d
use single encoder for sending speedtest results (#13579)
Bonus: if runs have PUT higher then capture it anyways
to display an unexpected result, which provides a way
to understand what might be slowing things down on the
system.

For example on a Data24 WDC setup it is clearly visible
there is a bug in the hardware.

```
./mc admin speedtest wdc/
⠧ Running speedtest (With 64 MiB object size, 32 concurrency) PUT: 31 GiB/s GET: 24 GiB/s
⠹ Running speedtest (With 64 MiB object size, 48 concurrency) PUT: 38 GiB/s GET: 24 GiB/s

MinIO 2021-11-04T06:08:33Z, 6 servers, 48 drives
PUT: 38 GiB/s, 605 objs/s
GET: 24 GiB/s, 383 objs/s
```

Reads are almost 14GiB/sec slower than Writes which
is practically not possible.
2021-11-04 12:11:52 -07:00
Pavel M 112f9ae087
claim exp should be integer (#13582)
claim exp can be 

- float64
- json.Number

As per OIDC spec https://openid.net/specs/openid-connect-core-1_0.html#IDToken

Avoid using strings since the upstream library only supports these two types now.
2021-11-04 12:03:43 -07:00
Aditya Manthramurthy 01b9ff54d9
Add LDAP STS tests and workflow for CI (#13576)
Runs LDAP tests with openldap container on GH Actions
2021-11-04 08:16:30 -07:00
Aditya Manthramurthy 64a1904136
Remove unused GlobalServiceDoneCh (#13578) 2021-11-04 08:15:10 -07:00
Aditya Manthramurthy bce6864785
Add tests to verify default server policies (#13575)
Check that they are present and that they can be modified by user
2021-11-03 19:49:05 -07:00
Aditya Manthramurthy ecd54b4cba
Move all IAM storage functionality into iam store type (#13567)
This reverts commit 091a7ae359.

- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they
  are not associated with any active credentials.

Also adds fix for accidental canned policy removal that was present in the
reverted version of the change.
2021-11-03 19:47:49 -07:00
Harshavardhana ca2b288a4b update console to v0.12.0 2021-11-03 16:23:45 -07:00
Harshavardhana 1016fbb8f9
feat: detect starting from windows explorer (#13570)
Windows users often click on the binary without
knowing MinIO is a command-line tool and should be
run from a terminal. Throw a message to guide them
on what to do.

Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-11-03 14:22:13 -07:00
Harshavardhana be3f81c7ec
remove unused activeIOCount in single drive mode (#13574) 2021-11-03 12:29:45 -07:00
Minio Trusted 9f3c151c3c Update yaml files to latest version RELEASE.2021-11-03T03-36-36Z 2021-11-03 06:48:34 +00:00
Harshavardhana 9735f3d8f2 fix missing go.sum changes 2021-11-02 20:36:36 -07:00
Harshavardhana 34680c5ccf
fix: SQL select to honor limits properly for array queries (#13568)
added tests to cover the scenarios as well.
2021-11-02 19:14:46 -07:00
Krishna Srinivas 58934e5881
Support live updates for clients during speedtest (#13566) 2021-11-02 15:27:03 -07:00
Harshavardhana ad3f98b8e7 add util-linux RPM for setpriv command 2021-11-02 14:25:01 -07:00
Poorna K 7c33a33ef3
cache: fix commit value lookup in config (#13551) 2021-11-02 14:20:52 -07:00
Poorna K 3dfcca68e6
fix race in TestComputeActions test (#13564) 2021-11-02 14:20:15 -07:00
Harshavardhana 73b74c94a1
remove unnecessary RPMs to reduce security reports (#13565) 2021-11-02 14:15:46 -07:00
Harshavardhana 18338d60d5 treat all 2xx, 3xx as good status-codes
fixes #13560
2021-11-02 14:12:43 -07:00
Harshavardhana e106070640 update docs to mention the expected behavior for requests_max
fixes #13561
2021-11-02 14:10:21 -07:00
Harshavardhana 091a7ae359 Revert "Move all IAM storage functionality into iam store type (#13541)"
This reverts commit caadcc3ed8.
2021-11-02 13:51:42 -07:00
Krishna Srinivas 70160aeab3
Remove IOPS autotuning and simplify autotune code (#13554) 2021-11-02 13:03:00 -07:00
jandres - moscardo 1aa08f594d
Update README.md prometheus (#13514)
Modify the doc to warn users about Prometheus sending `domain:port`
2021-11-02 12:27:30 -07:00
Harshavardhana 14d8a931fe
re-use io.Copy buffers with 32k pools (#13553)
Borrowed idea from Go's usage of this
optimization for ReadFrom() on client
side, we should re-use the 32k buffers
io.Copy() allocates for generic copy
from a reader to writer.

the performance increase for reads for
really tiny objects is at this range
after this change.

> * Fastest: +7.89% (+1.3 MiB/s) throughput, +7.89% (+1308.1) obj/s
2021-11-02 08:11:50 -07:00
Harshavardhana 30ba85bc67
no need to write storageClass globally (#13555)
fixes #13548
2021-11-02 08:11:20 -07:00
Aditya Manthramurthy caadcc3ed8
Move all IAM storage functionality into iam store type (#13541)
- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they 
  are not associated with any active credentials.
2021-11-01 21:58:07 -07:00
Poorna K 26f55472c6
fix: clean up dangling buckets during bucket delete (#13523) 2021-11-01 21:52:45 -07:00
Aditya Manthramurthy 79a58e275c
fix: race in delete user functionality (#13547)
- The race happens with a goroutine that refreshes IAM cache data from storage.
- It could lead to deleted users re-appearing as valid live credentials.
- This change also causes CI to run tests without a race flag (in addition to
running it with).
2021-11-01 15:03:07 -07:00
Aditya Manthramurthy 900e584514
CI: Cancel in-progress jobs when a PR is updated (#13552)
- This should lead to faster results as jobs will be queued for shorter periods
when PRs are updated.

- Current behavior is that previously running CI jobs for an updated PR run to
completion needlessly, and cause new CI jobs to be queued.

Ref: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#concurrency
2021-11-01 13:42:48 -07:00
Harshavardhana bb639d9f29
remove double reads delete versions (#13544)
deleting collection of versions belonging
to same object, we can avoid re-reading
the xl.meta from the disk instead purge
all the requested versions in-memory,

the tradeoff is to allocate a map to de-dup
the versions, allow disks to be read only
once per object.

additionally reduce the data transfer between
nodes by shortening msgp data values.
2021-11-01 10:50:07 -07:00