maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3102 commits 18 branches 277 tags 146 MiB
Commit graph

50 commits

Author SHA1 Message Date
Anis Elleuch 0fddf3fe17 Avoid creating tmp directories under .minio/tmp/ to facilitate cleaning (#2187) 2016-07-12 09:38:45 -07:00
Krishnan Parthasarathi 1c82b81408 Rename parts/objects only on onlineDisks (#2185) 2016-07-11 22:53:54 -07:00
Krishnan Parthasarathi bef72f26db xl: Make locking more granular for PutObjectPart requests (#2168) 2016-07-11 17:24:49 -07:00
Harshavardhana c0c8a8430e XL/PutObject: Add single putObject and multipart caching. (#2115) 
- Additionally adds test cases as well for object cache.
- Adds auto-expiry with expiration and cleanup time interval.

Fixes #2080
Fixes #2091
 2016-07-08 20:34:27 -07:00
Harshavardhana 6266328a85 XL/metadata: use new hashOrder algorithm for newXLMeta. (#2147) 2016-07-08 15:39:21 -07:00
Harshavardhana ca1b1921c4 XL: Implement ignore errors. (#2136) 
Each metadata ops have a list of errors which can be
ignored, this is essentially needed when

  - disks are not found
  - disks are found but cannot be accessed (permission denied)
  - disks are there but fresh disks were added

This is needed since we don't have healing code in place where
it would have healed the fresh disks added.

Fixes #2072
 2016-07-07 22:10:27 -07:00
Krishna Srinivas f55093cdd6 multipart: During multipart list the listing go-routine should be saved to the List-pool. (#2130) 2016-07-07 09:06:35 -07:00
Harshavardhana 8a028a9efb handler/PUT: Handle signature verification through a custom reader. (#2066) 
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.

Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().

Thanks to Krishna for his inputs on this.

Fixes #2058
Fixes #2054
Fixes #2087
 2016-07-05 01:04:50 -07:00
Krishna Srinivas 7a8b8cd0a1 tree-walk: unify FS and XL tree-walk with functional approach. (#2027) 2016-07-04 01:49:27 -07:00
Harshavardhana 734e779b19 XL/erasureCreate: Create a limit reader if size is specified. (#2059) 
This is needed so that we only write data which was requested
for, using a limit reader avoids spurious reads on the incoming
client data. Additionally using limit reader provides server
safety from rogue clients sending copious amounts of data (for
example a denial of service attack).

This patch also caters for size == -1 when content encoding from
a client is set as chunked, we happily read till io.EOF
 2016-07-01 14:33:28 -07:00
Krishnan Parthasarathi 64899e5197 xl: Used unique tmp file to update xl.json in putObjectPart (#2043) 
An in-place update to xl.json amidst concurrent PutObjectPart operations
lead to racy updates to xl.json making it un-parseable. To avoid this,
we create a unique tmp file where updates to xl.json are staged before
renaming it to the final location.
 2016-06-30 16:28:01 -07:00
Harshavardhana ae936a0147 XL: Relax write quorum further to N/2 + 1. (#2018) 
This changes behavior in some parts of the code
as well address it.

Fixes #2016
 2016-06-29 02:10:40 -07:00
Harshavardhana 748dc80047 API: add writePartTooSmallErrorResponse to extend standard error responses. (#2005) 
This function is added to extend the standard error responses.
Which is needed in some cases for example CompleteMultipartUpload
should respond with ErrPartTooSmall error when parts uploaded are
lesser than 5MB (i.e minimum allowed size per part).

Fixes #1536
 2016-06-28 14:51:49 -07:00
Harshavardhana 4db2b03312 XL: Rename objectN to part.N (#2019) 
Fixes #2015
 2016-06-27 21:42:33 -07:00
Harshavardhana 2e1f66c37d XL: Handle quorum situations properly for write operations. (#1986) 
Adds two test cases one for

 - PutObject when write quorum is not available.
 - PutObjectPart when write quorum is not available.

Fixes #1951
 2016-06-27 10:01:09 -07:00
Krishnan Parthasarathi 393c504de0 Renaming a part from tmp namespace needs to be handled different from… (#1944) 
* Renaming a part from tmp namespace needs to be handled different from renaming of an object

* Made argument passing in xl.rename and xl.undoRename explicit
 2016-06-20 19:11:55 -07:00
Aakash Muttineni 4ee2136b28 Unit tests for PUT object when object already exists (#1904) 
* fs/xl tests for multiple put object requests
* xl fix for put object on directory
* Unit tests fix windows test issue.
 2016-06-20 06:18:47 -07:00
Harshavardhana e1aad066c6 XL: CompleteMultipart should ignore last part is 0bytes. (#1931) 
Fixes #1917
 2016-06-19 14:51:20 -07:00
Harshavardhana 8c0942bf0d XL: Remove usage of reduceErr and make it isQuorum verification. (#1909) 
Fixes #1908
 2016-06-18 00:27:51 +05:30
Krishna Srinivas de1c7d33eb XL: appendFile should return error if quorum is not met. (#1898) 
Fixes #1890
 2016-06-15 00:24:49 +05:30
karthic rao 276282957e Test for Complete Multipart Upload. (#1888) 2016-06-10 18:43:16 +05:30
Bala FA 61598ed02f posix: return errFaultyDisk on I/O errors. (#1885) 
When I/O error is occured more than allowed limit, posix returns
errFaultyDisk.

Fixes #1884
 2016-06-08 22:02:10 -07:00
Harshavardhana 51f3d4e0ca XL/multipart: statPart should ignore errDiskNotFound. (#1862) 
startPart should also take uploadId and partName as arguments.
 2016-06-07 18:15:04 -07:00
Harshavardhana 1cf1532ca3 XL: Implement ListObjects channel and pool management. 2016-06-03 12:17:54 -07:00
Harshavardhana 82fd907933 XL/PutObject: Handle all pending cases of DiskNotFound. 2016-06-03 11:40:44 -07:00
Krishna Srinivas b00ac40c35 XL/PutObject: Calculate size if not provided by the client and update xl.json with the correct size. (#1844) 2016-06-02 17:09:47 -07:00
Harshavardhana fb95c1fad3 XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
Krishna Srinivas aa1d769b1e FS/Multipart: remove uploads.json on complete-multipart if no more uploadIDs are present for the object. (#1843) 
Fixes #1835
 2016-06-02 15:54:00 -07:00
Harshavardhana ae311aa53b XL: Cleanup, comments and all the updated functions. (#1830) 2016-06-01 16:43:31 -07:00
Harshavardhana c493ab5d0d XL: Bring in sha512 checksum support. (#1797) 2016-05-31 20:23:31 -07:00
Harshavardhana 445dc22118 XL: Cleanup and add more comments. (#1807) 2016-05-30 16:51:59 -07:00
Krishna Srinivas b466f27705 Nslock fixes (#1803) 
* XL/Multipart: Support parallel upload of parts by doing NS locking appropriately.

* XL/Multipart: hold lock on the multipart upload while aborting.
 2016-05-30 11:26:10 -07:00
Harshavardhana a4a0ea605b XL: Fix GetObject erasure decode issues. (#1793) 2016-05-29 15:38:14 -07:00
Harshavardhana 5e8de786b3 XL: Truly use unique id's in temp directory. (#1790) 
This also helps in avoiding cleaning up directories after.

Additionally this patch also fixes the problem of Range offsets.
 2016-05-29 00:42:09 -07:00
Harshavardhana feb337098d XL: bring in new storage API. (#1780) 
Fixes #1771
 2016-05-28 16:12:51 -07:00
Krishnan Parthasarathi c87f259820 Remove parts that are missing in CompleteMultipartUpload (#1786) 
* Remove parts that are missing in CompleteMultipartUpload

* Moved isUploadIDExists under proper namespace locks

* Moved code that deletes part files to a function
 2016-05-28 15:15:53 -07:00
Krishna Srinivas 41a5b3908b XL/ListParts: take the size from xl.json instead of backend file size as it will be different. (#1781) 
Fixes #1779
 2016-05-28 15:15:53 -07:00
Krishna Srinivas 3fb0b5e455 XL/Multipart: check existance upload uploadID after lock. (#1778) 
Fixes #1767
 2016-05-28 15:15:53 -07:00
Krishnan Parthasarathi 302ec27fa2 Fixed race during parallel PutObjectPart requests (#1775) 
The race is between two parallel PutObjectPart requests updating partsInfo in xl.json.
Previously, it was being updated under a RLock().
 2016-05-28 15:15:53 -07:00
Harshavardhana d65101a8c8 XL: Implement strided erasure distribution. (#1772) 
Strided erasure distribution uses a new randomized
block distribution for each Put operation. This
information is captured inside `xl.json` for subsequent
Get operations.
 2016-05-28 15:15:53 -07:00
Krishna Srinivas 6dc8323684 FS/ListMultipart: Fix FS list-multipart to work for unit test cases. 2016-05-28 15:15:53 -07:00
Harshavardhana b2293c2bf4 XL: Rename, cleanup and add more comments. (#1769) 
- xl-v1-bucket.go - removes a whole bunch of code.
- {xl-v1,fs-v1}-metadata.go - add a lot of comments and rename functions
   appropriately.
 2016-05-28 15:15:53 -07:00
Harshavardhana 553fdb9211 XL: Bring in support for object versions written during writeQuorum. (#1762) 
Erasure is initialized as needed depending on the quorum and onlineDisks.
This way we can manage the quorum at the object layer.
 2016-05-28 15:15:53 -07:00
Harshavardhana cae4782973 XL: explicit deleteObject is not needed after rename failure. (#1760) 
Reason is renameObject() does deleteObject() upon writeQuorum
failure if not keeps the successfully renamed parts if we have
reached readQuorum.
 2016-05-28 15:15:53 -07:00
Harshavardhana a4771265cf XL: Abortmultipart should update uploads.json properly. (#1757) 2016-05-28 15:15:53 -07:00
Harshavardhana a9e778f460 XL/fs: initObjectLayer should cleanup tmpMetaPrefix in parallel. (#1752) 
Fixes #1747
 2016-05-28 15:15:53 -07:00
Harshavardhana ee6645f421 XL: Add additional PartNumber variable as part of xl.json (#1750) 
This is needed for verification of incoming parts and to
support variadic part uploads. Which should be sorted
properly.

Fixes #1740
 2016-05-28 15:15:53 -07:00
Krishna Srinivas 6d84e84b3c XL/mutltipart: fix partnumber to partname association. (#1739) 
Fixes #1738
 2016-05-28 15:15:53 -07:00
Harshavardhana a00a5c6e7e XL: Multipart update uploads.json properly. (#1741) 2016-05-28 15:15:53 -07:00
Harshavardhana 293d246f95 XL/FS: Rewrite in new format. 2016-05-28 15:15:53 -07:00