maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3086 commits 18 branches 277 tags 146 MiB
Commit graph

3086 commits

This branch
This branch
All branches
Author SHA1 Message Date
Harshavardhana
734e779b19 XL/erasureCreate: Create a limit reader if size is specified. (#2059) 
This is needed so that we only write data which was requested
for, using a limit reader avoids spurious reads on the incoming
client data. Additionally using limit reader provides server
safety from rogue clients sending copious amounts of data (for
example a denial of service attack).

This patch also caters for size == -1 when content encoding from
a client is set as chunked, we happily read till io.EOF
 2016-07-01 14:33:28 -07:00
Krishna Srinivas
3f2b4d9dc2 Show "https" in the "minio server export/" output if certificates are available. (#2065) 
fixes #2036
 2016-07-01 14:32:53 -07:00
Bala FA
cd1c2db864 posix-utils: fix path handling in windows. (#2053) 2016-07-01 11:44:07 -07:00
karthic rao
48aa5f2199 api/bucket-policy: Add unit tests for more coverage, fixes couple of bugs. (#2055) 
Changes to ResourceMatch logic.
Test for action match function.
 2016-06-30 23:49:59 -07:00
Krishnan Parthasarathi
bcb822c390 Send XML header before the first of whitespace chars (#2046) 
* Sent XML header before the first of whitespace chars

XML parsing fails in aws cli due to unexpected whitespace character. To
fix this, we send the xml header before we send the first whitespace
character, if any.

* Fix race between sendWhiteSpaceChars and completeMultiUploadpart
 2016-06-30 18:48:50 -07:00
Krishnan Parthasarathi
285a94d2c0 Ignored errors in cleanup in commitXLMetadata (#2044) 
Deletion of tmp files where xl metadata was saved before the commit
operation doesn't change the error returned to the caller. So, it is to
be ignored.
 2016-06-30 16:37:51 -07:00
Krishnan Parthasarathi
64899e5197 xl: Used unique tmp file to update xl.json in putObjectPart (#2043) 
An in-place update to xl.json amidst concurrent PutObjectPart operations
lead to racy updates to xl.json making it un-parseable. To avoid this,
we create a unique tmp file where updates to xl.json are staged before
renaming it to the final location.
 2016-06-30 16:28:01 -07:00
Danilo Pereira
812554087f getCertsPath should use getConfigPath instead of defaulting to users homedir. (#2039) 
Fixes #2028
 2016-06-30 15:49:18 -07:00
Bala FA
57bc08cc7e posix: remove disk free space check for read-only and delete methods. (#2033) 2016-06-29 11:25:35 -07:00
Harshavardhana
0e3907072c XL/fs: Initialize export paths supplied on command line (#2020) 
Fixes #2013
 2016-06-29 03:13:44 -07:00
karthic rao
8e8f6f90a4 adding detailed comments to server_test (#2032) 2016-06-29 02:30:36 -07:00
Bala FA
4c1a11aae6 XL: allow meta bucket name appended with tmp meta prefix. (#2007) 2016-06-29 02:28:46 -07:00
Harshavardhana
ae936a0147 XL: Relax write quorum further to N/2 + 1. (#2018) 
This changes behavior in some parts of the code
as well address it.

Fixes #2016
 2016-06-29 02:10:40 -07:00
Harshavardhana
d484157d67 XL/bitrot: Migrate to new blake2b-simd SIMD optimized implementation. (#2031) 
Thanks for Frank Wessels <fwessels@xs4all.nl> for all the heavy lifting work.

Comparative benchmarks are as below.
```
benchmark                old ns/op     new ns/op     delta
BenchmarkHash64-4        742           411           -44.61%
BenchmarkHash128-4       681           346           -49.19%
BenchmarkWrite1K-4       4239          1497          -64.69%
BenchmarkWrite8K-4       33633         11514         -65.77%
BenchmarkWrite32K-4      134091        45947         -65.73%
BenchmarkWrite128K-4     537976        183643        -65.86%

benchmark                old MB/s     new MB/s     speedup
BenchmarkHash64-4        86.18        155.51       1.80x
BenchmarkHash128-4       187.96       369.10       1.96x
BenchmarkWrite1K-4       241.55       683.87       2.83x
BenchmarkWrite8K-4       3897.06      11383.41     2.92x
BenchmarkWrite32K-4      977.48       2852.63      2.92x
BenchmarkWrite128K-4     243.64       713.73       2.93x
```

Fixes #2030
 2016-06-29 02:06:35 -07:00
Harshavardhana
796fe165c7 server: Minor command line doc change for XL. (#2022) 2016-06-29 02:05:57 -07:00
Harshavardhana
3ac39ff107 XL: Change minimum disks supported to 6 now. (#2023) 
This change co-incides with another patch set which
reduces the writeQuorum requirement. With the
write quorum change it is now possible to support
6 disk configuration.
 2016-06-29 02:05:29 -07:00
Krishnan Parthasarathi
b6b9e88e47 Added unit-tests for treeWalkPool (#1969) 
* Added unit-tests for treeWalkPool
* Added unit tests for tree-walk-fs
* Added period at the end of all comments.
* FS/XL: Unified tree walk tests for both backends
* Added disk failure related tests for treewalk

Replaced removeRandomDisks with removeDiskN. There is no need to
randomize disks that fail while the distribution of chunks in XL during
erasure coding data is random.
 2016-06-28 22:32:00 -07:00
karthic rao
59366d8f4c Benchmarks for ObjectLayer.PutObject() (#2029) 2016-06-28 22:12:36 -07:00
Harshavardhana
748dc80047 API: add writePartTooSmallErrorResponse to extend standard error responses. (#2005) 
This function is added to extend the standard error responses.
Which is needed in some cases for example CompleteMultipartUpload
should respond with ErrPartTooSmall error when parts uploaded are
lesser than 5MB (i.e minimum allowed size per part).

Fixes #1536
 2016-06-28 14:51:49 -07:00
karthic rao
6dcfa7b046 Fix for tests leaving out temp directories (#2025) 2016-06-28 04:21:52 -07:00
Krishnan Parthasarathi
a854e8cc5c api: Sent ErrPreconditionFailed on If-Match failure (#2009) 
* api: Sent ErrPreconditionFailed on If-Match failure

ref:
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

* tests: Added functional tests for GetObject w/ If-Match headers set

* tests: Used verifyError to simplify errorCode and description matching on error
 2016-06-28 01:18:18 -07:00
karthic rao
76f6533f8d Adding detailed comments for server_xl_test. (#2011) 2016-06-27 23:54:56 -07:00
Harshavardhana
4db2b03312 XL: Rename objectN to part.N (#2019) 
Fixes #2015
 2016-06-27 21:42:33 -07:00
Krishna Srinivas
5291db60c6 XL/erasure: refactor erasureReadFile. Move parallelRead into a separate function. (#2008) 2016-06-27 13:24:55 -07:00
Harshavardhana
2e1f66c37d XL: Handle quorum situations properly for write operations. (#1986) 
Adds two test cases one for

 - PutObject when write quorum is not available.
 - PutObjectPart when write quorum is not available.

Fixes #1951
 2016-06-27 10:01:09 -07:00
Bala FA
c88720ea2c XL/listObjects: Ignore entry if getObjectInfo() returns errFileNotFound (#2004) 
Fixes #1956
 2016-06-26 22:10:22 -07:00
karthic rao
ce7d5eddbc Misspell warnings fix (#2001) 2016-06-26 22:05:48 -07:00
Harshavardhana
0d3a9c9438 XL: Add tests for checkSufficientDisks, storageInfo. (#1988) 2016-06-26 19:48:02 -07:00
Harshavardhana
293ba00249 posix: Re-do tests for readDir(). (#1996) 2016-06-26 19:31:53 -07:00
Krishnan Parthasarathi
d0be09fdd3 object: checkETag compares quoted ETags properly (#1997) 
Previously, checkETag didn't handle ETags with leading and trailing
double quotes. e.g "abcdef1234" == "\"abcdef1234\"" would return false.
Now, checkETag function canonicalizes the ETags passed as arguments by
removing one leading/trailing double quote.
 2016-06-26 18:10:08 -07:00
Remco Verhoef
1e52759c3c fix typo (#1987) 2016-06-26 11:27:04 -07:00
Harshavardhana
9add048f3c erasure: Add erasure encode/decode unit tests. (#1911) 
Fixes #1819
 2016-06-26 03:32:49 -07:00
Harshavardhana
936a916e78 server: Add connection rate limiter for server. (#1977) 2016-06-26 03:18:07 -07:00
Harshavardhana
57146fbbb8 main: minio --help should print regardless of root. (#1985) 
Remove root check entirely.

Fixes #1964
 2016-06-26 03:03:52 -07:00
karthic rao
3d02f7471e Benchmarks for various object sizes for FS/XL GetObject (#1984) 2016-06-25 20:22:04 -07:00
karthic rao
b2d9a46cbb Cleaning up/refactoring tests to be more extensible (#1970) 2016-06-25 19:07:44 -07:00
Harshavardhana
42286cba70 XL: Implement new ReadAll API for files which are read in single call. (#1974) 
Add a unit test as well.
 2016-06-25 14:51:06 -07:00
karthic rao
ed2fdd90b0 fs: Fix GetObject failure to read large blocks. (#1982) 
Add relevant test cases as well for verifying this
part of the codebase.

Fixes #1979
 2016-06-25 03:03:27 -07:00
Krishna Srinivas
cb1200a66d XL/erasure-read: read disks in parallel. (#1975) 
On read failure, fallback to reads from other
disks also happen in parallel.
 2016-06-24 18:02:10 -07:00
Krishnan Parthasarathi
a3a310cde8 Moved tree-walk-fs to use tree-walk-pool (#1978) 2016-06-24 16:41:57 -07:00
Bala FA
f625392211 tests: add unit test for posix.readDir() (#1967) 
Fixes #1820
 2016-06-24 14:32:08 -07:00
Harshavardhana
e8990e42c2 XL: Make allocations simpler avoid redundant allocs. (#1961) 
- Reduce 10MiB buffers for loopy calls to use 128KiB.
- start using 128KiB buffer where needed.
 2016-06-24 02:06:23 -07:00
Harshavardhana
ff9fc22c72 posix: Mkdir() and OpenFile() should honor umask. (#1972) 
Adds two unit tests for validation as well.

Fixes #1965
 2016-06-23 20:19:27 -07:00
Harshavardhana
41c089a7e0 XL: Add mis-aligned GetObject() test. (#1960) 2016-06-22 21:42:24 -07:00
Harshavardhana
5725f3c809 Merge pull request #1958 from krisis/unittest/unc-path 
Added unit tests for UNC path handling in windows
 2016-06-22 17:19:40 -07:00
Harshavardhana
f4830162a4 XL: Format heal should re-allocate new UUIDs not reuse. (#1953) 
This patch also supports writing to a temporary file and renaming
rather than appending to an existing file. This helps in avoiding
inconsistent files.
 2016-06-22 17:18:31 -07:00
Harshavardhana
e10934a88e bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957) 
Fixes #1952
 2016-06-22 17:13:26 -07:00
Krishnan Parthasarathi
a07751f61b Added tests to validate disk name length near MAX_PATH 2016-06-22 15:30:41 -07:00
Krishnan Parthasarathi
0766e903e3 Added unit tests for UNC path handling in windows 2016-06-22 15:30:41 -07:00
Harshavardhana
75dddfb2ae Merge pull request #1959 from krishnasrinivas/parallel-reads 
Parallel reads in erasure-read
 2016-06-22 15:05:35 -07:00