maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
7535 commits 18 branches 277 tags 146 MiB
Commit graph

160 commits

Author SHA1 Message Date
Andreas Auernhammer 8771e83545 Update klauspost/reedsolomon dependency to 1.9.3 (#8328) 
This commit updates the reedsolomon dependency
since it contains an fix for an unexpected property
of the `Split` function.

See: klauspost/reedsolomon#109
 2019-10-01 02:31:00 +05:30
Klaus Post ff726969aa Switch to Snappy -> S2 compression (#8189) 2019-09-25 23:08:24 -07:00
Harshavardhana be70ef59e7
Update minio-go to latest release (#8297) 
- Change LDAP XML names to follow server changes
- Fix listObjects hang when listing special characters
- set encoding type to url in listobjects API
 2019-09-24 01:01:54 -07:00
Klaus Post ddea0bdf11 Concurrent CSV parsing and reduce S3 select allocations (#8200) 
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12         	    2842	    407533 ns/op	  397860 B/op	     957 allocs/op
BenchmarkReaderReplace-12       	    2718	    429914 ns/op	  397844 B/op	     957 allocs/op
BenchmarkReaderReplaceTwo-12    	    2718	    435556 ns/op	  397855 B/op	     957 allocs/op
BenchmarkAggregateCount_100K-12    	     171	   6798974 ns/op	16667102 B/op	  308077 allocs/op
BenchmarkAggregateCount_1M-12    	      19	  65657411 ns/op	168057743 B/op	 3146610 allocs/op
BenchmarkSelectAll_10M-12    	       1	20882119900 ns/op	2758799896 B/op	41978762 allocs/op

CSV parsing, AFTER:
BenchmarkReaderBasic-12         	    3721	    312549 ns/op	  101920 B/op	     338 allocs/op
BenchmarkReaderReplace-12       	    3776	    318810 ns/op	  101993 B/op	     340 allocs/op
BenchmarkReaderReplaceTwo-12    	    3610	    330967 ns/op	  102012 B/op	     341 allocs/op
BenchmarkAggregateCount_100K-12    	     295	   4149588 ns/op	 3553623 B/op	  103261 allocs/op
BenchmarkAggregateCount_1M-12    	      30	  37746503 ns/op	33827931 B/op	 1049435 allocs/op
BenchmarkSelectAll_10M-12    	       1	17608495800 ns/op	1416504040 B/op	21007082 allocs/op

~ benchcmp old.txt new.txt
benchmark                           old ns/op       new ns/op       delta
BenchmarkReaderBasic-12             407533          312549          -23.31%
BenchmarkReaderReplace-12           429914          318810          -25.84%
BenchmarkReaderReplaceTwo-12        435556          330967          -24.01%
BenchmarkAggregateCount_100K-12     6798974         4149588         -38.97%
BenchmarkAggregateCount_1M-12       65657411        37746503        -42.51%
BenchmarkSelectAll_10M-12           20882119900     17608495800     -15.68%

benchmark                           old allocs     new allocs     delta
BenchmarkReaderBasic-12             957            338            -64.68%
BenchmarkReaderReplace-12           957            340            -64.47%
BenchmarkReaderReplaceTwo-12        957            341            -64.37%
BenchmarkAggregateCount_100K-12     308077         103261         -66.48%
BenchmarkAggregateCount_1M-12       3146610        1049435        -66.65%
BenchmarkSelectAll_10M-12           41978762       21007082       -49.96%

benchmark                           old bytes      new bytes      delta
BenchmarkReaderBasic-12             397860         101920         -74.38%
BenchmarkReaderReplace-12           397844         101993         -74.36%
BenchmarkReaderReplaceTwo-12        397855         102012         -74.36%
BenchmarkAggregateCount_100K-12     16667102       3553623        -78.68%
BenchmarkAggregateCount_1M-12       168057743      33827931       -79.87%
BenchmarkSelectAll_10M-12           2758799896     1416504040     -48.66%
```

```
BenchmarkReaderHuge/97K-12         	    2200	    540840 ns/op	 184.32 MB/s	 1604450 B/op	     687 allocs/op
BenchmarkReaderHuge/194K-12        	    1522	    752257 ns/op	 265.04 MB/s	 2143135 B/op	    1335 allocs/op
BenchmarkReaderHuge/389K-12        	    1190	    947858 ns/op	 420.69 MB/s	 3221831 B/op	    2630 allocs/op
BenchmarkReaderHuge/778K-12        	     806	   1472486 ns/op	 541.61 MB/s	 5201856 B/op	    5187 allocs/op
BenchmarkReaderHuge/1557K-12       	     426	   2575269 ns/op	 619.36 MB/s	 9101330 B/op	   10233 allocs/op
BenchmarkReaderHuge/3115K-12       	     286	   4034656 ns/op	 790.66 MB/s	12397968 B/op	   16099 allocs/op
BenchmarkReaderHuge/6230K-12       	     172	   6830563 ns/op	 934.05 MB/s	16008416 B/op	   26844 allocs/op
BenchmarkReaderHuge/12461K-12      	     100	  11409467 ns/op	1118.39 MB/s	22655163 B/op	   48107 allocs/op
BenchmarkReaderHuge/24922K-12      	      66	  19780395 ns/op	1290.19 MB/s	35158559 B/op	   90216 allocs/op
BenchmarkReaderHuge/49844K-12      	      34	  37282559 ns/op	1369.03 MB/s	60528624 B/op	  174497 allocs/op
```
 2019-09-13 14:18:35 -07:00
Krishnan Parthasarathi 6ba323b009 Add ability to test drive speeds on a MinIO setup (#7664) 
- Extends existing Admin API to measure disk performance
 2019-09-13 03:22:30 +05:30
Harshavardhana 5512baab21 Fix go mod tidy errors and fix platform independent downloads (#8222) 
Refer https://github.com/gin-gonic/gin/issues/1673
 2019-09-12 04:57:59 +05:30
Minio Trusted 3d65dc8d94 Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z 2019-09-11 19:59:05 +00:00
Aditya Manthramurthy a0456ce940 LDAP STS API (#8091) 
Add LDAP based users-groups system

This change adds support to integrate an LDAP server for user
authentication. This works via a custom STS API for LDAP. Each user
accessing the MinIO who can be authenticated via LDAP receives
temporary credentials to access the MinIO server.

LDAP is enabled only over TLS.

User groups are also supported via LDAP. The administrator may
configure an LDAP search query to find the group attribute of a user -
this may correspond to any attribute in the LDAP tree (that the user
has access to view). One or more groups may be returned by such a
query.

A group is mapped to an IAM policy in the usual way, and the server
enforces a policy corresponding to all the groups and the user's own
mapped policy.

When LDAP is configured, the internal MinIO users system is disabled.
 2019-09-10 04:42:29 +05:30
Harshavardhana 94e5cb7576
Migrate to go1.13 to avail all new features (#8203) 
Read more https://blog.golang.org/go1.13
 2019-09-08 16:44:15 -07:00
Klaus Post 33aec08e8c Clean up go. mod (#8197) 
Golang proposes to clean up these and is fairly insisting about it ;)
 2019-09-06 14:57:55 -07:00
Minio Trusted 5bde31d021 Update yaml files to latest version RELEASE.2019-09-05T23-24-38Z 2019-09-05 23:30:40 +00:00
Harshavardhana b52a3e523c Avoid using fastjson parser pool, move back to jsoniter (#8190) 
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.

The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
 2019-09-06 04:21:27 +05:30
Minio Trusted daa57e8e5a Update yaml files to latest version RELEASE.2019-08-29T00-25-01Z 2019-08-29 00:30:50 +00:00
Minio Trusted de736f2864 Update yaml files to latest version RELEASE.2019-08-21T19-40-07Z 2019-08-21 19:46:01 +00:00
Harshavardhana 9ca7470ccc
Avoid using jsoniter, move to fastjson (#8063) 
This is to avoid using unsafe.Pointer type
code dependency for MinIO, this causes
crashes on ARM64 platforms

Refer #8005 collection of runtime crashes due
to unsafe.Pointer usage incorrectly. We have
seen issues like this before when using
jsoniter library in the past.

This PR hopes to fix this using fastjson
 2019-08-19 08:35:52 -10:00
Minio Trusted 26640033b0 Update yaml files to latest version RELEASE.2019-08-14T20-37-41Z 2019-08-14 20:42:16 +00:00
Minio Trusted 55b385beee Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z 2019-08-07 02:05:41 +00:00
kannappanr 930943f058
Fix IAM users migration regression in etcd (#8029) 
PR #8008 did not migrate user data stored in etcd.
This PR fixes that.
 2019-08-06 17:06:31 -07:00
Minio Trusted dfa8835720 Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z 2019-08-01 22:27:41 +00:00
Minio Trusted c301f5882d Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z 2019-07-31 19:06:20 +00:00
Harshavardhana 6f2b4675fa
Add krb5 support for HDFS gateway (#7933) 2019-07-24 18:05:48 -07:00
Minio Trusted 5b71c21330 Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z 2019-07-24 02:09:19 +00:00
Krishnan Parthasarathi 559a59220e Add initial support for bucket lifecycle (#7563) 
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)

N B the code for actual expiration of objects will be included in a
subsequent PR.
 2019-07-19 21:20:33 +01:00
Minio Trusted b058e32348 Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z 2019-07-17 22:59:33 +00:00
Harshavardhana 16a45e5aff
Fix dynamic help vars for sub-commands (#7925) 
The fix in #7646 introduced a regression which
was left unnoticed, the fix didn't work for
sub-commands unfortunately. This fixes it
by moving v1.21.0 version of the minio/cli
package.

Fixes #7924
 2019-07-12 23:32:27 -07:00
Praveen raj Mani bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Minio Trusted 2d96745156 Update yaml files to latest version RELEASE.2019-07-10T00-34-56Z 2019-07-10 00:39:45 +00:00
mzukowski-reef 9d49688c87 Switch to kurin/blazer from minio/blazer fork for b2 gateway (#7879) 2019-07-09 08:14:02 -07:00
Minio Trusted 22bc15d89b Update yaml files to latest version RELEASE.2019-07-05T21-20-21Z 2019-07-05 21:24:43 +00:00
Krishna Srinivas a2e904b966 Support any string as delimiter for listing (#7882) 2019-07-05 14:06:12 -07:00
Krishna Srinivas 338e9a9be9 Put object client disconnect (#7824) 
Fail putObject  and postpolicy in case client prematurely disconnects
Use request's context to cancel lock requests on client disconnects
 2019-06-28 22:09:17 -07:00
Minio Trusted 5db60a6c59 Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z 2019-06-27 21:18:52 +00:00
Minio Trusted e43d3a075c Update yaml files to latest version RELEASE.2019-06-19T18-24-42Z 2019-06-19 18:31:37 +00:00
Minio Trusted 59f7266081 Update yaml files to latest version RELEASE.2019-06-15T23-07-18Z 2019-06-15 23:17:28 +00:00
Harshavardhana 99bf4d0c42 [sec] Match ${aws:username} exactly instead of prefix match (#7791) 
This PR fixes a security issue where an IAM user based
on his policy is granted more privileges than restricted
by the users IAM policy.

This is due to an issue of prefix based Matcher() function
which was incorrectly matching prefix based on resource
prefixes instead of exact match.
 2019-06-15 11:27:17 -07:00
Minio Trusted da2887f914 Update yaml files to latest version RELEASE.2019-06-13T01-41-13Z 2019-06-13 01:46:00 +00:00
Harshavardhana c22439c82e Update minio-go v6.0.29 (#7778) 
Bring improved retry logic
 2019-06-12 18:09:21 -07:00
Harshavardhana b4ab778cb2 Fix user IAM policy regression, reload policy appropriately (#7770) 
Introduce in commit 7e4c9a9e1e

Fixes #7769
 2019-06-12 14:49:45 -07:00
Minio Trusted d3a2efbf91 Update yaml files to latest version RELEASE.2019-06-11T00-44-33Z 2019-06-11 00:50:31 +00:00
Harshavardhana cb1566c6e6 S3 Gateway: Handle restricted access credentials (#7757) 2019-06-07 15:49:13 -07:00
Minio Trusted df19546cc8 Update yaml files to latest version RELEASE.2019-06-04T01-15-58Z 2019-06-04 01:22:32 +00:00
Harshavardhana ff7799452b Fix build failure due to nats project migration 2019-06-03 17:49:49 -07:00
Minio Trusted 48ffb7731a Update yaml files to latest version RELEASE.2019-06-01T03-46-14Z 2019-06-01 03:52:47 +00:00
Harshavardhana 2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Minio Trusted 7a02faab72 Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z 2019-05-23 00:35:13 +00:00
Harshavardhana 39b3e4f9b3 Avoid using io.ReadFull() for WriteAll and CreateFile (#7676) 
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.

Also adds readahead for disk reads, which also increases
performance for reads by 3x.
 2019-05-22 13:47:15 -07:00
Andrei Mikhalenia 59e847aebe Signature v4: Allow signed headers from GET parameters 2019-05-21 21:00:02 -07:00
Harshavardhana c5f26d5cdd Fix hdfsReader fd leak upon GetObject() (#7596) 
Also migrate to minio/hdfs/v3@v3.0.0
 2019-05-01 14:43:21 -07:00
Krishna Srinivas a3ec71bc28 Use O_DIRECT while writing to disk (#7479) 
- Use O_DIRECT while writing to disk
- Remove MINIO_DRIVE_SYNC option
 2019-04-23 21:25:06 -07:00
Minio Trusted ab711fe1a2 Update yaml files to latest version RELEASE.2019-04-23T23-50-36Z 2019-04-23 23:55:31 +00:00
Harshavardhana f767a2538a
Optimize listing with leaf check offloaded to posix (#7541) 
Other listing optimizations include

- remove double sorting while filtering object entries
- improve error message when upload-id is not in quorum
- use jsoniter for full unmarshal json, instead of gjson
- remove unused code
 2019-04-23 14:54:28 -07:00
Minio Trusted d42496cc74 Update yaml files to latest version RELEASE.2019-04-18T21-44-59Z 2019-04-18 21:52:22 +00:00
Minio Trusted 7686340621 Update yaml files to latest version RELEASE.2019-04-18T01-15-57Z 2019-04-18 01:22:00 +00:00
Harshavardhana 620e462413 Implement S3-HDFS gateway (#7440) 
- [x] Support bucket and regular object operations
- [x] Supports Select API on HDFS
- [x] Implement multipart API support
- [x] Completion of ListObjects support
 2019-04-17 09:52:08 -07:00
Krishnan Parthasarathi 35ef5eb236 Don't exit background append if backend specific files show up (#7519) 2019-04-12 15:51:32 -07:00
Minio Trusted ea052a2a00 Update yaml files to latest version RELEASE.2019-04-09T01-22-30Z 2019-04-09 01:30:27 +00:00
Minio Trusted 082bd5f31f Update yaml files to latest version RELEASE.2019-04-04T18-31-46Z 2019-04-04 18:39:37 +00:00
Harshavardhana 313a3a286a Migrate to go1.12 to simplify our cmd/http package (#7302) 
Simplify the cmd/http package overall by removing
custom plain text v/s tls connection detection, by
migrating to go1.12 and choose minimum version
to be go1.12

Also remove all the vendored deps, since they
are not useful anymore.
 2019-04-02 18:28:39 -07:00
Minio Trusted 8cf707779c Update yaml files to latest version RELEASE.2019-03-27T22-35-21Z 2019-03-27 22:41:57 +00:00
Sidhartha Mani 6bc0de2a75 add go modules file and start running go 1.11 style builds (#7354) 2019-03-19 13:50:58 -07:00