Commit graph

8531 commits

Author SHA1 Message Date
johnnyaug 17e0de1e87
fix: typo in CONTRIBUTING.md (#12907) 2021-08-17 09:20:10 -07:00
Krishnan Parthasarathi b7e3651d3c
Set free-version id in case of version/version-suspended buckets (#12982)
This free-version id may be used to track tiered object contents of the
object (version) being deleted.
2021-08-17 08:59:48 -07:00
Harshavardhana ef4d023c85
fix: various performance improvements to tiering (#12965)
- deletes should always Sweep() for tiering at the
  end and does not need an extra getObjectInfo() call
- puts, copy and multipart writes should conditionally
  do getObjectInfo() when tiering targets are configured
- introduce 'TransitionedObject' struct for ease of usage
  and understanding.
- multiple-pools optimization deletes don't need to hold
  read locks verifying objects across namespace and pools.
2021-08-17 07:50:00 -07:00
Harshavardhana 654a6e9871
always set the filter to skip navigating baseDir (#12984)
baseDir is empty if the top level prefix does not
end with `/` this causes large recursive listings
without any filtering, to fix this filtering make
sure to set the filter prefix appropriately.

also do not navigate folders at top level that do
not match the filter prefix, entries don't need
to match prefix since they are never prefixed
with the prefix anyways.
2021-08-17 07:43:24 -07:00
Aditya Manthramurthy 2ca5ee026d
Remove default value of STSExpiry for LDAP (#12985)
This ensures that the deprecation warning is shown when the setting is actually
used in a configuration - instead of showing up whenever LDAP is enabled.
2021-08-17 02:25:05 -07:00
Aditya Manthramurthy 9b7d593e28
Add helper script to call assume role (#12978) 2021-08-17 01:46:59 -07:00
Klaus Post ad928f0078
Return list request when canceled (#12977)
* Return list request when canceled
* Cancel list if abandoned
2021-08-16 11:59:16 -07:00
Klaus Post 92bb2928e4
Compress better on amd64 (#12974)
Since S2 has amd64 assembly, it now operates at a reasonable 
speed to use by default.

Here are some examples of stream compression speed, 16 cores:
```
nyc-taxi-data-10M.csv	s2	1	3325605752	-> 1095998837	312ms	10139.07MB/s		67.04% reduction
nyc-taxi-data-10M.csv	s2	2	3325605752	-> 917905514	428ms	7393.74MB/s		72.40%

github-june-2days-2019.json	s2	1	6273951764	-> 1043196283	391ms	15301.99 MB/s		83.37%
github-june-2days-2019.json	s2	2	6273951764	-> 955924506	519ms	11510.81MB/s		84.76%

github-ranks-backup.bin	s2	1	1862623243	-> 623911363	146ms	12133MB/s		66.50%
github-ranks-backup.bin	s2	2	1862623243	-> 563752759	230ms	7705.26MB/s		69.73%
```

We keep non-assembly platforms on the faster, but less efficient mode.
2021-08-16 11:55:07 -07:00
Anis Elleuch 47dfc1b1b0
ldap: Reevalute filter when searching for non eligible users (#12953)
The previous code removes SVC/STS accounts for ldap users that do not
exist anymore in LDAP server. This commit will actually re-evaluate
filter as well if it is changed and remove all local SVC/STS accounts
beloning to the ldap user if the latter is not eligible for the
search filter anymore.

For example: the filter selects enabled users among other criteras in
the LDAP database, if one ldap user changes his status to disabled
later, then associated SVC/STS accounts will be removed because that user
does not meet the filter search anymore.
2021-08-13 11:40:04 -07:00
Klaus Post 7d8413a589
Reuse more metadata buffers (#12955)
Reuse metadata buffers when no longer referenced.

Takes care of most of the happy paths.
2021-08-13 11:39:27 -07:00
Klaus Post 24722ddd02
Remove inline data hack (#12946)
move the code down to the storage layer,
this logic decouples the inline data from the 
size parameter making it flexible and future
proof.
2021-08-13 08:25:54 -07:00
Klaus Post f31a00de01
fix: http stats race in traffic metering (#12956)
Traffic metering was not protected against concurrent updates.

```
WARNING: DATA RACE
Read at 0x00c02b0dace8 by goroutine 235:
  github.com/minio/minio/cmd.setHTTPStatsHandler.func1()
      d:/minio/minio/cmd/generic-handlers.go:360 +0x27d
  net/http.HandlerFunc.ServeHTTP()
...

Previous write at 0x00c02b0dace8 by goroutine 994:
  github.com/minio/minio/internal/http/stats.(*IncomingTrafficMeter).Read()
      d:/minio/minio/internal/http/stats/http-traffic-recorder.go:34 +0xd2

```
2021-08-13 07:30:03 -07:00
Shireesh Anjal d44e4399e6
Add admin api to return sys services info (#12939)
The intention is to provide status of any sys services that can
potentially impact the performance of minio.

At present, it will return information about the `selinux` service
(not-installed/disabled/permissive/enforcing)

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
2021-08-12 18:58:40 -07:00
Harshavardhana f9ae71fd17
fix: deleteMultiObjects performance regression (#12951)
fixes performance regression found in deleteObjects(),
putObject(), copyObject and completeMultipart calls.
2021-08-12 18:57:37 -07:00
Harshavardhana ce28e904c9 pass the current credentials for claims 2021-08-12 18:24:04 -07:00
Harshavardhana 77b8885a24 update console to latest release v0.9.0 2021-08-12 18:08:30 -07:00
Harshavardhana 8f2a3efa85
disallow sub-credentials based on root credentials to gain priviledges (#12947)
This happens because of a change added where any sub-credential
with parentUser == rootCredential i.e (MINIO_ROOT_USER) will
always be an owner, you cannot generate credentials with lower
session policy to restrict their access.

This doesn't affect user service accounts created with regular
users, LDAP or OpenID
2021-08-12 18:07:08 -07:00
Klaus Post 89febdb3d6
Reuse small buffers (#12948)
When reading metadata allow reuse of buffers 
in certain cases. Take the low-hanging fruit.

Reduce GC overhead when listing.
2021-08-12 14:27:22 -07:00
Klaus Post 3eac02f676
Use metadata reader in ReadVersion (#12942)
Use `readMetadata` when reading version 
information without data requested. 

Reduces IO on inlined data.

Bonus: Inline compressed data as well when 
compression is enabled.
2021-08-12 10:05:24 -07:00
Ricardo Katz a526ad2e80
Add headers into AMQP notifications (#12911)
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
2021-08-11 22:24:19 -07:00
Krishnan Parthasarathi 65b6f4aa31
Add dynamic reconfiguration of number of transition workers (#12926) 2021-08-11 22:23:56 -07:00
Harshavardhana 9e88941515
fix: skip disks that are offline when healing the drives (#12931) 2021-08-11 12:57:18 -07:00
Harshavardhana 3becee9e5d
use sync map instead of local DNS cache (#12925)
also enable PreferGo only resolver for FIPS builds
2021-08-10 21:20:09 -07:00
Harshavardhana 40a2fa8e81
fix: add more optimizations to putMetacacheObject() (#12916)
- avoid extra lookup for 'xl.meta' since we are
  definitely sure that it doesn't exist.

- use this in newMultipartUpload() as well

- also additionally do not write with O_DSYNC
  to avoid loading the drives, instead create
  'xl.meta' for listing operations without
  O_DSYNC since these are ephemeral objects.

- do the same with newMultipartUpload() since
  it gets synced when the PutObjectPart() is
  attempted, we do not need to tax newMultipartUpload()
  instead.
2021-08-10 11:12:22 -07:00
Harshavardhana 39f81d2c5b update max_delay to max_sleep under healing docs 2021-08-10 08:52:42 -07:00
Aditya Manthramurthy 59bb54ed6a
Use common function for authenticating admin requests (#12915) 2021-08-09 18:14:38 -07:00
Klaus Post 9ab5e0312d
Simplify gzhttp wrapper (#12912)
The wrapper now accepts interfaces so we don't need a wrapper for that any more.
2021-08-09 12:45:59 -07:00
Harshavardhana 54ab3a1d5b
implement putMetacacheObject() optimizing List operations (#12903)
removes unexpected features from regular putObject() such as

- increasing parity when disks are down, avoids
  a lot of DiskInfo() calls.

- triggering MRF for metacache objects
  if disks are offline

- avoiding renames from temporary location
  to actual namespace, not needed since
  metacache files are unique.
2021-08-09 06:58:54 -07:00
Klaus Post 92c94011f1
Skip downed interfaces on Windows (#12910)
Disregard interfaces that are down when selecting bind addresses

Windows often has a number of disabled NICs used for VPN and other services.

This often causes minio to select an address for contacting the console that is on a disabled (virtual) NIC.

This checks if the interface is up before adding it to the pool on Windows.
2021-08-09 06:57:54 -07:00
Anis Elleuch 35cbe43b6d
Start gateway when KMS is enabled and encryption is unsupported (#12808)
Before, the gateway will complain that it found KMS configured in the
environment but the gateway mode does not support encryption. This
commit will allow starting of the gateway but ensure that S3 operations
with encryption headers will fail when the gateway doesn't support
encryption. That way, the user can use etcd + KMS and have IAM data
encrypted in the etcd store.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-08 12:51:48 -07:00
Harshavardhana a2cd3c9a1d
use ParseForm() to allow query param lookups once (#12900)
```
cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkURLQueryForm
BenchmarkURLQueryForm-4         247099363                4.809 ns/op           0 B/op          0 allocs/op
BenchmarkURLQuery
BenchmarkURLQuery-4              2517624               462.1 ns/op           432 B/op          4 allocs/op
PASS
ok      github.com/minio/minio/cmd      3.848s
```
2021-08-07 22:43:01 -07:00
Minio Trusted 7b0b0f9101 Update yaml files to latest version RELEASE.2021-08-05T22-01-19Z 2021-08-06 02:19:42 +00:00
Harshavardhana 6c0757eea6
fix: simplify APIEndpoints() usage (#12893)
improvements include

- skip IPv6 correctly
- do not set default value for
  MINIO_SERVER_URL, let it be
  configured if not use local IPs

Bonus:

- In healing return error from listPathRaw()
- update console to v0.8.3
2021-08-05 15:01:19 -07:00
Anis Elleuch 37bef900fd
ilm: More compliance to spec related to expired delete markers (#12887)
<Days> tag can remove expired delete markers according to AWS S3 spec.

<NoncurrentVersionExpiration> cannot remove expired delete markers in
anyway.
2021-08-05 11:21:21 -07:00
Harshavardhana 3863a96bdf remove unexpected spaces and newlines 2021-08-04 17:21:32 -07:00
Ravind Kumar c89135fd13
Update License section for documentation (#12878) 2021-08-04 17:07:38 -07:00
Harshavardhana b10f823907
allow diskFillFraction to be 99% (#12879)
larger 4-8TiB sized disks would return
error prematurely even with sufficient
amount of disk space left, increase
diskFillFraction to 1%
2021-08-04 15:23:39 -07:00
Harshavardhana 4d8f81a992
remove replicate set fallback behavior (#12865)
replica set deployment style is an incorrect
style of distributed MinIO deployment, since
statefulsets have been around for many releases
in k8s now.
2021-08-04 15:23:18 -07:00
Harshavardhana c13cbc64d1 fix multiple typos in documentation 2021-08-04 14:15:45 -07:00
Matt Sarrel 109c8acf4f
fixed typo in metrics README.md (#12874) 2021-08-04 12:48:57 -07:00
Klaus Post cc60d66909
Fix incremental usage accounting (#12871)
Remote caches were not returned correctly, so they would not get updated on save.

Furthermore make some tweaks for more reliable updates.

Invalidate bloom filter to ensure rescan.
2021-08-04 09:14:14 -07:00
Krishnan Parthasarathi 4197870287
Storage-class headers should depend only on ObjectInfo state (#12864) 2021-08-03 17:35:52 -07:00
Harshavardhana 035882d292
fix: remove parentIsObject() check (#12851)
we will allow situations such as

```
a/b/1.txt
a/b
```

and

```
a/b
a/b/1.txt
```

we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.

rest of the situations where the prefixes get
created across sets are supported as is.
2021-08-03 13:26:57 -07:00
Harshavardhana 9371852c7d
fix: getAPIEndpoints() should return public_url (#12852)
fixes #12850
2021-08-02 21:50:20 -07:00
Harshavardhana ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Harshavardhana 510c67de4a update console to v0.8.2 2021-08-02 13:07:49 -07:00
Dmitry Eliseev 27dbe30ecb
Allow to specify secrets separately (#12813) 2021-07-31 21:44:51 -07:00
Krishnan Parthasarathi 0a62ae4e61
Revert ignoring inlined objects for transition (#12843) 2021-07-30 16:45:17 -07:00
Harshavardhana a51799d9f0
feat: Add support for audit notifications for transition (#12842)
This PR adds audit notifications for transitioning objects,
similar to audit logging for expiration and replication
traffic.
2021-07-30 12:45:25 -07:00
Harshavardhana e7baf2d7d2 stop all console logging 2021-07-29 23:06:05 -07:00