maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
6032 commits 18 branches 277 tags 146 MiB
Commit graph

6032 commits

This branch
This branch
All branches
Author SHA1 Message Date
poornas 2e02e1889b Cleanup ResponseWriter function for audit and trace (#8283) 2019-09-24 02:04:28 +05:30
Julio Pintos 3b05e175d7 fix link to let's encrypt doc (#8294) 2019-09-23 23:53:11 +05:30
Harshavardhana b512241300
Add metrics healthcheck test with JWT (#8287) 2019-09-22 23:30:38 -07:00
ebozduman dbf7b1e573 starts-with policy condition support issue (#7937) 2019-09-22 14:20:49 -07:00
Harshavardhana 26985ac632 Fix all failing tests with -race 2019-09-22 11:01:46 -07:00
Praveen raj Mani ad75683bde Authorize prometheus endpoint with bearer token (#7640) 2019-09-22 20:27:12 +05:30
poornas 4925bc3e80 log server startup messages to admin console api (#8264) 2019-09-22 13:54:32 +05:30
Andreas Auernhammer ffded5a930 make the crypto error type a native go type (#8267) 
This commit makes the `crypto.Error` type a native go (string)
type. That allows us to define error values as constants instead
of variables.

For reference see:
 - https://twitter.com/_aead_/status/1118170258215514115?s=20
 - https://dave.cheney.net/2016/04/07/constant-errors
 2019-09-22 01:12:51 -07:00
Krishnan Parthasarathi 1127293863 Add a basic .dockerignore file to reduce docker context in make docker (#8282) 
- Add useful default tag to dev docker image build using `make docker`
 2019-09-21 04:49:56 +05:30
Andreas Auernhammer 2b51fe9f26 make SSE request header check comprehensive (#8276) 
This commit refactors the SSE header check
by moving it into the `crypto` package, adds
a unit test for it and makes the check comprehensive.
 2019-09-21 03:26:12 +05:30
Harshavardhana 4780fa5a58 Remove setting net.Conn Deadlines as its not needed anymore (#8269) 
This commit fixes a bug introduced in af6c6a2b35.

Setting deadlines in Go results in arbitrary hangs as reported here
https://github.com/golang/go/issues/34385

Fixes https://github.com/minio/minio/issues/7852
 2019-09-20 23:37:24 +05:30
Klaus Post 520552ffa9 S3 select: flush when reaching limit (#8279) 
Add missing flush when reaching select limit.
 2019-09-20 11:00:17 -07:00
Andreas Auernhammer b823d6d7bd remove the unused code for decrypting io.Writer (#8277) 
This commit removes unused code for decrypting
`io.Writer` since the actual implementation only
decrypts `io.Reader`
 2019-09-20 14:51:07 +05:30
Andreas Auernhammer a9d724120f remove TLS 1.3 opt-in code (#8275) 
This commit removes the TLS 1.3 opt-in code.
Since TLS 1.3 is opt-out for >= Go 1.13 this
code is not needed anymore.
 2019-09-20 01:51:44 +05:30
Praveen raj Mani dc0dce9beb Check for connection errors during event transit in elasticsearch and webhook (#8273) 2019-09-19 08:23:43 -07:00
Yao Zongyou 0e1408844b fix get container id bug (#8196) 2019-09-19 10:19:51 +05:30
Andreas Auernhammer e34369c860 prepare SSE-S3 metadata parsing for K/V data key store (#8259) 
This commit allows the MinIO server to parse the metadata if:
 - either the `X-Minio-Internal-Server-Side-Encryption-S3-Key-Id`
   and the `X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key`
   entries are present.
 - or *both* headers are not present.

This is in service to support a K/V data key store.
 2019-09-19 04:08:09 +05:30
Minio Trusted 8dc897b5f5 Update yaml files to latest version RELEASE.2019-09-18T21-55-05Z 2019-09-18 22:02:41 +00:00
Praveen raj Mani 456ce4cc92 Add rootCAs support to Kafka & MQTT (#8236) 
Fixes #8211
 2019-09-18 23:43:04 +05:30
Aditya Manthramurthy 696f4ceee2 Add notes on configuring LDAP STS with Microsoft Active Directory (#8260) 2019-09-17 17:21:37 -07:00
Klaus Post dac1cf5a9a S3 Select: Parsing tweaks (#8261) 
* Don't output empty lines.
* Trim whitespace from byte to int/float/bool conversions.
 2019-09-17 17:21:23 -07:00
Harshavardhana cb01516a26 In HDFS gateway fix non-empty folder behavior (#8254) 
To be compatible with our FS and Erasure coded
mode deployments, make sure that we do not send
200 OK for folders which have files inside.

Fixes #8143
 2019-09-18 01:59:59 +05:30
Aditya Manthramurthy dfa1b417a8 Update comments in ldap example client code (#8257) 2019-09-18 01:59:31 +05:30
Krishnan Parthasarathi 31bee6b6ed Remove size query parameter from PerfInfo handler (#8258) 2019-09-18 01:59:12 +05:30
poornas 04b92124c5 fs/xl: Log warning if cache config specified (#8251) 
in non-gateway mode.
 2019-09-16 19:55:52 -07:00
Harshavardhana 5392eee250 Avoid recursion and use a simple loop to merge entries (#8239) 
This avoids stack overflows when there are
lot of entries to be skipped, this PR also
optimizes the code to reuse the buffers.
 2019-09-17 06:08:37 +05:30
Vivian Kong fa32c71a56 Build staticcheck as pre-built binaries are not available for s390x (#8220) 2019-09-17 05:59:15 +05:30
Klaus Post c9b8bd8de2 S3 Select: optimize output (#8238) 
Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.

In best case this is more than an order of magnitude speedup:

```
BenchmarkSelectAll_1M-12    	       1	1841049400 ns/op	274299728 B/op	 4198522 allocs/op
BenchmarkSelectAll_1M-12    	      14	  84833400 ns/op	169228346 B/op	 3146541 allocs/op
```
 2019-09-17 05:56:27 +05:30
Klaus Post 017456df63 Wait clearing the close channel (#8250) 
Close channel should not be nilled before goroutines have exited.

Fixes potential hang on closing.
 2019-09-16 16:18:01 -07:00
Harshavardhana 14b137aa66 posix/readDir should populate name for DT_UKNOWN (#8240) 
In commit a8296445ad we changed the code to handle
some corner cases on ARM and other platforms, this
PR just avoids the return for unknown filetypes
prematurely and let the name be populated appropriately.

This fixes bug for older XFS implementations such as
in Ubuntu 14.04
 2019-09-17 03:04:01 +05:30
Andreas Auernhammer 3064da7b08 return error during part listing when no quorum (#8241) 
This commit fixes a subtle bug that (probably)
caused an issue affecting encrypted multipart objects.

When a cluster has no quorum this bug causes `ListObjectParts`
to return nil as error instead of a quorum error.

Thanks to @harshavardhana for detecting this.
 2019-09-17 02:57:34 +05:30
poornas 76df027264 Allow caching only in gateway mode. (#8232) 
This PR changes cache on PUT behavior to background fill the cache
after PutObject completes. This will avoid concurrency issues as in #8219.

Added cleanup of partially filled cache to prevent cache corruption
- Fixes #8208
 2019-09-17 02:54:04 +05:30
k-tipp 208efb843b Update Kafka notification example doc (#8212) 2019-09-16 19:24:55 +05:30
Harshavardhana 9ac12cf898
Remove unusued Set/GetConfigKeys API (#8235) 2019-09-13 16:34:34 -07:00
Klaus Post ddea0bdf11 Concurrent CSV parsing and reduce S3 select allocations (#8200) 
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12         	    2842	    407533 ns/op	  397860 B/op	     957 allocs/op
BenchmarkReaderReplace-12       	    2718	    429914 ns/op	  397844 B/op	     957 allocs/op
BenchmarkReaderReplaceTwo-12    	    2718	    435556 ns/op	  397855 B/op	     957 allocs/op
BenchmarkAggregateCount_100K-12    	     171	   6798974 ns/op	16667102 B/op	  308077 allocs/op
BenchmarkAggregateCount_1M-12    	      19	  65657411 ns/op	168057743 B/op	 3146610 allocs/op
BenchmarkSelectAll_10M-12    	       1	20882119900 ns/op	2758799896 B/op	41978762 allocs/op

CSV parsing, AFTER:
BenchmarkReaderBasic-12         	    3721	    312549 ns/op	  101920 B/op	     338 allocs/op
BenchmarkReaderReplace-12       	    3776	    318810 ns/op	  101993 B/op	     340 allocs/op
BenchmarkReaderReplaceTwo-12    	    3610	    330967 ns/op	  102012 B/op	     341 allocs/op
BenchmarkAggregateCount_100K-12    	     295	   4149588 ns/op	 3553623 B/op	  103261 allocs/op
BenchmarkAggregateCount_1M-12    	      30	  37746503 ns/op	33827931 B/op	 1049435 allocs/op
BenchmarkSelectAll_10M-12    	       1	17608495800 ns/op	1416504040 B/op	21007082 allocs/op

~ benchcmp old.txt new.txt
benchmark                           old ns/op       new ns/op       delta
BenchmarkReaderBasic-12             407533          312549          -23.31%
BenchmarkReaderReplace-12           429914          318810          -25.84%
BenchmarkReaderReplaceTwo-12        435556          330967          -24.01%
BenchmarkAggregateCount_100K-12     6798974         4149588         -38.97%
BenchmarkAggregateCount_1M-12       65657411        37746503        -42.51%
BenchmarkSelectAll_10M-12           20882119900     17608495800     -15.68%

benchmark                           old allocs     new allocs     delta
BenchmarkReaderBasic-12             957            338            -64.68%
BenchmarkReaderReplace-12           957            340            -64.47%
BenchmarkReaderReplaceTwo-12        957            341            -64.37%
BenchmarkAggregateCount_100K-12     308077         103261         -66.48%
BenchmarkAggregateCount_1M-12       3146610        1049435        -66.65%
BenchmarkSelectAll_10M-12           41978762       21007082       -49.96%

benchmark                           old bytes      new bytes      delta
BenchmarkReaderBasic-12             397860         101920         -74.38%
BenchmarkReaderReplace-12           397844         101993         -74.36%
BenchmarkReaderReplaceTwo-12        397855         102012         -74.36%
BenchmarkAggregateCount_100K-12     16667102       3553623        -78.68%
BenchmarkAggregateCount_1M-12       168057743      33827931       -79.87%
BenchmarkSelectAll_10M-12           2758799896     1416504040     -48.66%
```

```
BenchmarkReaderHuge/97K-12         	    2200	    540840 ns/op	 184.32 MB/s	 1604450 B/op	     687 allocs/op
BenchmarkReaderHuge/194K-12        	    1522	    752257 ns/op	 265.04 MB/s	 2143135 B/op	    1335 allocs/op
BenchmarkReaderHuge/389K-12        	    1190	    947858 ns/op	 420.69 MB/s	 3221831 B/op	    2630 allocs/op
BenchmarkReaderHuge/778K-12        	     806	   1472486 ns/op	 541.61 MB/s	 5201856 B/op	    5187 allocs/op
BenchmarkReaderHuge/1557K-12       	     426	   2575269 ns/op	 619.36 MB/s	 9101330 B/op	   10233 allocs/op
BenchmarkReaderHuge/3115K-12       	     286	   4034656 ns/op	 790.66 MB/s	12397968 B/op	   16099 allocs/op
BenchmarkReaderHuge/6230K-12       	     172	   6830563 ns/op	 934.05 MB/s	16008416 B/op	   26844 allocs/op
BenchmarkReaderHuge/12461K-12      	     100	  11409467 ns/op	1118.39 MB/s	22655163 B/op	   48107 allocs/op
BenchmarkReaderHuge/24922K-12      	      66	  19780395 ns/op	1290.19 MB/s	35158559 B/op	   90216 allocs/op
BenchmarkReaderHuge/49844K-12      	      34	  37282559 ns/op	1369.03 MB/s	60528624 B/op	  174497 allocs/op
```
 2019-09-13 14:18:35 -07:00
Harshavardhana e7f491a14b Use optimized sha256-simd whenever possible (#8227) 
Avoid using `crypto/sha256` and use always
`github.com/minio/sha256-simd`
 2019-09-14 00:39:39 +05:30
Praveen raj Mani 8700945cdf Handle connection failures on webhook/url pings (#8204) 
Properly handle connection failures while replaying events

Fixes #8194
 2019-09-12 16:44:51 -07:00
Harshavardhana ff6aabd9c0 Honor standard HTTP headers for sourceIP (#8233) 
Behind load balancers we should be tracing sourceIP
preserved by load balancers.
 2019-09-13 03:59:59 +05:30
Krishnan Parthasarathi 6ba323b009 Add ability to test drive speeds on a MinIO setup (#7664) 
- Extends existing Admin API to measure disk performance
 2019-09-13 03:22:30 +05:30
Anis Elleuch e7b3f39064 xl: Fix verifying non streaming highway algo with a dist setup (#8230) 
VerifyFile in the distributed setup does not work with
the non streaming highway hash. The reason is that the
internode mux router did not expect `storageRESTBitrotHash`
parameter.
 2019-09-12 13:08:02 -07:00
Harshavardhana 9fa727d154 Provide a friendlier error when an update fails (#8228) 
Add upgrading documentation as well
 2019-09-13 01:33:42 +05:30
Harshavardhana 73e4e99942 Hosts should be skipped, when calculating local info (#8191) 
endpoint.IsLocal will not have .Host entries so
using them to skip double entries will never work.

change the code such that we look for endpoint.Host
outside of endpoint.IsLocal logic to skip double
hosts appropriately.

Move these functions to their appropriate file.
 2019-09-12 23:36:12 +05:30
Alex Pardoe a87fc7d09b Use the B2 'list' endpoint to determine file ID (#8169) 
- More effective deletion and checking for existence.
- Rever Dockerfile.
- Add a 'GOPROXY' to the Dockerfile to workaround Apache issues.
 2019-09-12 22:48:47 +05:30
Harshavardhana 475df52a19 Fix etcd watch regression in IAM subsystem (#8224) 
Fixes #8223
 2019-09-12 07:24:25 +05:30
Harshavardhana 5512baab21 Fix go mod tidy errors and fix platform independent downloads (#8222) 
Refer https://github.com/gin-gonic/gin/issues/1673
 2019-09-12 04:57:59 +05:30
Harshavardhana 77963078a2 Remove maintainers.md use only contributing.md (#8215) 2019-09-12 04:57:11 +05:30
Anis Elleuch 3f258062d8 bitrot: Verify file size inside storage interface (#7932) 2019-09-12 02:19:53 +05:30
Minio Trusted 3d65dc8d94 Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z 2019-09-11 19:59:05 +00:00
Harshavardhana 53e4887e02 Simplify and cleanup metadata r/w functions (#8146) 2019-09-11 22:52:12 +05:30
Harshavardhana a7be313230 Start using new errors package (#8207) 2019-09-11 22:51:43 +05:30