Commit graph

8542 commits

Author SHA1 Message Date
Matt Sarrel 109c8acf4f
fixed typo in metrics README.md (#12874) 2021-08-04 12:48:57 -07:00
Klaus Post cc60d66909
Fix incremental usage accounting (#12871)
Remote caches were not returned correctly, so they would not get updated on save.

Furthermore make some tweaks for more reliable updates.

Invalidate bloom filter to ensure rescan.
2021-08-04 09:14:14 -07:00
Krishnan Parthasarathi 4197870287
Storage-class headers should depend only on ObjectInfo state (#12864) 2021-08-03 17:35:52 -07:00
Harshavardhana 035882d292
fix: remove parentIsObject() check (#12851)
we will allow situations such as

```
a/b/1.txt
a/b
```

and

```
a/b
a/b/1.txt
```

we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.

rest of the situations where the prefixes get
created across sets are supported as is.
2021-08-03 13:26:57 -07:00
Harshavardhana 9371852c7d
fix: getAPIEndpoints() should return public_url (#12852)
fixes #12850
2021-08-02 21:50:20 -07:00
Harshavardhana ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Harshavardhana 510c67de4a update console to v0.8.2 2021-08-02 13:07:49 -07:00
Dmitry Eliseev 27dbe30ecb
Allow to specify secrets separately (#12813) 2021-07-31 21:44:51 -07:00
Krishnan Parthasarathi 0a62ae4e61
Revert ignoring inlined objects for transition (#12843) 2021-07-30 16:45:17 -07:00
Harshavardhana a51799d9f0
feat: Add support for audit notifications for transition (#12842)
This PR adds audit notifications for transitioning objects,
similar to audit logging for expiration and replication
traffic.
2021-07-30 12:45:25 -07:00
Harshavardhana e7baf2d7d2 stop all console logging 2021-07-29 23:06:05 -07:00
Shireesh Anjal 56097dfca8
Include system errors in health report (#12776)
Some examples of system errors:

- audit is enabled
- updatedb is installed
2021-07-29 23:05:34 -07:00
Harshavardhana 7281e86d9e update docker-compose to RELEASE.2021-07-30T00-02-00Z 2021-07-29 18:07:06 -07:00
Harshavardhana d004828b20 fix: speedtest Endpoint formatting 2021-07-29 17:51:30 -07:00
Harshavardhana 4567160e4d update madmin-go to v1.0.20 2021-07-29 17:02:00 -07:00
Harshavardhana bc6a2b579b update to console v0.8.1 2021-07-29 16:14:10 -07:00
Harshavardhana 3c34e18a4e
allow multipart uploads for single part multipart (#12821)
its possible that some multipart uploads would have
uploaded only single parts so relying on `len(o.Parts)`
alone is not sufficient, we need to look for ETag
pattern to be absolutely sure.
2021-07-28 22:11:55 -07:00
Poorna Krishnamoorthy b6cd54779c
Increase context timeout for bandwidth throttled reader (#12820)
increase default timeout up to one hour for toy setups.

fixes #12812
2021-07-28 15:20:01 -07:00
Harshavardhana bfbdb8f0a8
fix: incorrect O_DIRECT behavior for reads (#12811)
O_DIRECT behavior was broken and it was still
caching all the reads, this change properly fixes
this behavior.
2021-07-28 11:20:16 -07:00
Harshavardhana 397637a042 remove x-frame-options HTTP header
fixes #12817
2021-07-28 08:57:22 -07:00
Anis Elleuch 20cacc3b04
fs: Update object modtime when metadata is updated (#12818)
Follow S3 behavior of updating object modtime when object metadata is
updated.
2021-07-28 07:36:38 -07:00
Harshavardhana a78bc7bfdb rename all access token to id_tokens 2021-07-27 22:38:12 -07:00
Harshavardhana 3735450e7e
fix: allow audience claim to be an array (#12810)
Some incorrect setups might have multiple audiences
where they are trying to use a single authentication
endpoint for multiple services.

Nevertheless OpenID spec allows it to make it
even more confusin for no good reason.

> It MUST contain the OAuth 2.0 client_id of the
> Relying Party as an audience value. It MAY also
> contain identifiers for other audiences. In the
> general case, the aud value is an array of case
> sensitive strings. In the common special case
> when there is one audience, the aud value MAY
> be a single case sensitive string.

fixes #12809
2021-07-27 18:37:51 -07:00
Krishna Srinivas aa0c28809b
Server side speedtest implementation (#12750) 2021-07-27 12:55:56 -07:00
Minio Trusted 471b4fd0c9 Update yaml files to latest version RELEASE.2021-07-27T02-40-15Z 2021-07-27 05:02:00 +00:00
Harshavardhana ddcd419b4c
always validate JWT token audience (#12797)
audience for the JWT token should match
the configured client_id, this allows
rejecting valid JWTs not meant for MinIO.
2021-07-26 19:40:15 -07:00
Harshavardhana a9d9b520ec
remove short circuited healing optimization (#12796)
this healing optimization caused multiple
regressions in healing

- delete-markers incorrectly missing
  heal and returning incorrect healing
  results to client.

- missing individual 'parts' such
  as for restored object or simply
  for all objects just missing few parts.

This optimization is not necessary, we
should proceed to verify all cases possible
not just when metadata is inconsistent.
2021-07-26 16:51:09 -07:00
Harshavardhana 0c666379fe
fix: avoid removing healed parts on dstDataPath (#12795)
destination path and old path will be similar
when healing occurs, this can lead to healed
parts being again purged leading to always an
inconsistent state on an object which might
further cause reduction in quorum eventually.
2021-07-26 15:15:34 -07:00
Harshavardhana a3f7d575e0
improve delete-marker healing (#12794)
delete-markers missing on drives were
not healed due to few things

disksWithAllParts() does not know-how
to deal with delete markers, add support
for that.

fixes #12787
2021-07-26 11:48:09 -07:00
Harshavardhana f175ff8f66
add healing fixes for delete-marker (#12788)
- delete-markers are incorrectly reported
  as corrupt with wrong data sent to client
  'mc admin heal -r' on objects with delete
  marker will report as 'grey' incorrectly.

- do not heal delete-markers during HeadObject()
  this can lead to inconsistent order of heals
  on the object, although this is not an issue
  in terms of order of versions it is rather
  simpler to keep the same order on all drives.

- defaultHealResult() should handle 'err == nil'
  case such that valid cases should be handled
  as 'drive' status OK.
2021-07-26 08:01:41 -07:00
Anis Elleuch 39874b77ed
mrf: Avoid rare data race and more simplification (#12791)
This change avoids a rare data race and simplify the function that
returns MRF last activity information.
2021-07-26 08:00:59 -07:00
Aditya Manthramurthy 7106e9394c
[LDAP] Document new LDAP synchronization behavior (#12786) 2021-07-25 10:45:53 -07:00
Harshavardhana e124d88788
optimize listing operation concurrency (#12728)
- remove use of getOnlineDisks() instead rely on fallbackDisks()
  when disk return errors like diskNotFound, unformattedDisk
  use other fallback disks to list from, instead of paying the
  price for checking getOnlineDisks()

- optimize getDiskID() further to avoid large write locks when
  looking formatLastCheck time window

This new change allows for a more relaxed fallback for listing
allowing for more tolerance and also eventually gain more
consistency in results even if using '3' disks by default.
2021-07-24 22:03:38 -07:00
Aditya Manthramurthy de00b641da
[LDAP] Support syncing user-group memberships with LDAP service (#12785)
When configured in Lookup Bind mode, the server now periodically queries the
LDAP IDP service to find changes to a user's group memberships, and saves this
info to update the access policies for all temporary and service account
credentials belonging to LDAP users.
2021-07-24 11:57:36 -07:00
Harshavardhana e936871b83 update to console v0.8.0 2021-07-23 19:53:56 -07:00
Harshavardhana 922f5da8dc update nats.io to fix critical bug 2021-07-23 18:33:31 -07:00
Harshavardhana 294bafd58b remove unused WebTrace and minio/rpc dependency 2021-07-23 15:50:21 -07:00
Anis Elleuch 23ef25b57a
profiling: Return goroutines with sleep duration (#12775)
Add a new goroutine file which has another printing format. We need it
to see how much time each goroutine was blocked. Easier to detect stops.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-07-23 13:16:53 -07:00
Aditya Manthramurthy 9a31030e74
DEPRECATION Warning: Avoid configuring default LDAP STS Expiry (#12781)
- Show notice when `MINIO_IDENTITY_LDAP_STS_EXPIRY` or the 
  corresponding to the configuration option is used at server startup.
- Once support is removed, the default will be fixed at 1 hour.
- Users may specify expiry directly in the STS API.
- Update docs and help message
- Adds example in ldap.go to configure expiry in STS API.
2021-07-22 16:43:57 -07:00
AlexHuang2021 df2871de53
fix: return error when listing fails to retry healing (#12765) 2021-07-22 12:14:44 -07:00
Aditya Manthramurthy a3079a7de2
fix: Add support for DurationSeconds in LDAP STS API (#12778) 2021-07-22 12:13:21 -07:00
Aditya Manthramurthy 17a37241f0
fix: regression in AssumeRole session policy handling (fixes #12756) (#12773) 2021-07-22 08:42:07 -07:00
Minio Trusted 026a005b64 Update yaml files to latest version RELEASE.2021-07-22T05-23-32Z 2021-07-22 06:12:29 +00:00
Harshavardhana b865b9f43d update console to fix assets 2021-07-21 22:23:32 -07:00
Krishnan Parthasarathi 209e6d00c6
Use ObjectInfo.ToLifecycleOpts instead of literal values (#12772)
Promote getLifecycleTransitionTier to a method on lifecycle.Lifecycle.
2021-07-21 19:12:44 -07:00
Minio Trusted dbd7f74bb9 Update yaml files to latest version RELEASE.2021-07-21T22-15-23Z 2021-07-21 23:46:28 +00:00
Harshavardhana 4a2cb23690 update console to master 2021-07-21 15:15:23 -07:00
Harshavardhana 320e1533c4
use expected MinIO URLs for console (#12770)
when TLS is configured using IPs directly
might interfere and not work properly when
the server is configured with TLS certs but
the certs only have domain certs.

Also additionally allow users to specify
a public accessible URL for console to talk
to MinIO i.e `MINIO_SERVER_URL` this would
allow them to use an external ingress domain
to talk to MinIO. This internally fixes few
problems such as presigned URL generation on
the console UI etc.

This needs to be done additionally for any
MinIO deployments that might have a much more
stricter requirement when running in standalone
mode such as FS or standalone erasure code.
2021-07-21 14:51:16 -07:00
Harshavardhana 7f45e80755 update console to master with more UI fixes 2021-07-21 11:19:36 -07:00
Aditya Manthramurthy 0db1c94e7d
[IDP:LDAP] Cleanup creds for removed LDAP user entries (#12759) 2021-07-20 23:33:12 -07:00