Commit graph

40 commits

Author SHA1 Message Date
Krishna Srinivas 303f216150 tests: xl-v1-metadata.go, xl-v1-multipart-common.go - remove unused methods, add enhance tests to improve code coverage. (#2260) 2016-07-21 15:00:11 -07:00
Harshavardhana a0635dcdd9 XL: Do not rely on getLoadBalancedQuorumDisks for NS consistency. (#2243)
The reason is any function relying on `getLoadBalancedQuorumDisks`
cannot possibly have an idempotent behavior.

The problem comes from given a set of N disks returning just a
shuffled N/2 disks.  In case of a scenario where we have N/2
number of failed disks, the returned value of `getLoadBalancedQuorumDisks`
is not equal to the same failed disks so essentially calls using such
disks might succeed or fail randomly at different intervals in time.

This proposal change is we move to `getLoadBalancedDisks()`
and use the shuffled N disks as a whole. Since most of the time we might
hit a good disk since we are not reducing our solution space. This
also provides consistent behavior for all the functions which rely
on shuffled disks.

Fixes #2242
2016-07-21 00:27:08 -07:00
Harshavardhana cef26fd6ea XL: Refactor usage of reduceErrs and consistent behavior. (#2240)
This refactor is also needed in lieu of our quorum
requirement change for the newly understood logic behind
klauspost/reedsolom implementation.
2016-07-19 19:24:32 -07:00
Krishna Srinivas b090c7112e Refactor of xl.PutObjectPart and erasureCreateFile. (#2193)
* XL: Refactor of xl.PutObjectPart and erasureCreateFile.

* GetCheckSum and AddCheckSum methods for xlMetaV1

* Simple unit test case for erasureCreateFile()
2016-07-14 14:59:01 -07:00
Harshavardhana cdf1373f8e XL: Ignore and continue for cases when bucket does not exist. (#2205)
Fixes #2201
Fixes #2204
2016-07-13 13:44:33 -07:00
Harshavardhana dc3bafb194 XL: isQuorum rename as isDiskQuorum, word it properly. (#2196) 2016-07-13 00:29:48 -07:00
Harshavardhana 623e0f9243 XL: listOnlineDisks should use modTime instead of version. (#2166)
This change is needed to make reading from objects future proof
in-terms of handling online disks. Our current counter is not
based on affirmative knowledge and relies on arithmetic sequence
which can lead to bugs.

Using modTime simplifies the understanding of `xl.json` and future
tooling / debugging of the format.
2016-07-12 15:20:31 -07:00
Krishnan Parthasarathi bef72f26db xl: Make locking more granular for PutObjectPart requests (#2168) 2016-07-11 17:24:49 -07:00
Harshavardhana de468f92ec posix: ReadAll should handle the case when parent is not a dir. (#2163)
It can happen so that a read request can come for a file which
already has a parent i.e a file.

This fix handles this scenario - fixes #2047
2016-07-11 00:15:37 -07:00
Krishna Srinivas aa7079fc7b XL/GetObject: If quorum not available during GetObject appropriate error should be returned. (#2135) 2016-07-10 17:12:22 -07:00
Harshavardhana 6266328a85 XL/metadata: use new hashOrder algorithm for newXLMeta. (#2147) 2016-07-08 15:39:21 -07:00
frankw 63b3f1dcfd Use new algorithm to get fixed random order of disks (#2147) 2016-07-08 15:38:47 -07:00
Harshavardhana ca1b1921c4 XL: Implement ignore errors. (#2136)
Each metadata ops have a list of errors which can be
ignored, this is essentially needed when

  - disks are not found
  - disks are found but cannot be accessed (permission denied)
  - disks are there but fresh disks were added

This is needed since we don't have healing code in place where
it would have healed the fresh disks added.

Fixes #2072
2016-07-07 22:10:27 -07:00
Harshavardhana ae936a0147 XL: Relax write quorum further to N/2 + 1. (#2018)
This changes behavior in some parts of the code
as well address it.

Fixes #2016
2016-06-29 02:10:40 -07:00
Harshavardhana e8990e42c2 XL: Make allocations simpler avoid redundant allocs. (#1961)
- Reduce 10MiB buffers for loopy calls to use 128KiB.
- start using 128KiB buffer where needed.
2016-06-24 02:06:23 -07:00
Harshavardhana 50d25ca94a XL: Change AppendFile() to return only error (#1932)
AppendFile ensures that it appends the entire buffer. Returns
an error otherwise, this patch removes the necessity for the
caller to look for 'n' return on short writes.

Ref #1893
2016-06-19 15:31:13 -07:00
Harshavardhana e1aad066c6 XL: CompleteMultipart should ignore last part is 0bytes. (#1931)
Fixes #1917
2016-06-19 14:51:20 -07:00
Harshavardhana 8c0942bf0d XL: Remove usage of reduceErr and make it isQuorum verification. (#1909)
Fixes #1908
2016-06-18 00:27:51 +05:30
Bala FA 61598ed02f posix: return errFaultyDisk on I/O errors. (#1885)
When I/O error is occured more than allowed limit, posix returns
errFaultyDisk.

Fixes #1884
2016-06-08 22:02:10 -07:00
Harshavardhana 82fd907933 XL/PutObject: Handle all pending cases of DiskNotFound. 2016-06-03 11:40:44 -07:00
Harshavardhana fb95c1fad3 XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
Harshavardhana ae311aa53b XL: Cleanup, comments and all the updated functions. (#1830) 2016-06-01 16:43:31 -07:00
Harshavardhana c493ab5d0d XL: Bring in sha512 checksum support. (#1797) 2016-05-31 20:23:31 -07:00
Harshavardhana 445dc22118 XL: Cleanup and add more comments. (#1807) 2016-05-30 16:51:59 -07:00
Harshavardhana a4a0ea605b XL: Fix GetObject erasure decode issues. (#1793) 2016-05-29 15:38:14 -07:00
Harshavardhana 5e8de786b3 XL: Truly use unique id's in temp directory. (#1790)
This also helps in avoiding cleaning up directories after.

Additionally this patch also fixes the problem of Range offsets.
2016-05-29 00:42:09 -07:00
Harshavardhana feb337098d XL: bring in new storage API. (#1780)
Fixes #1771
2016-05-28 16:12:51 -07:00
Harshavardhana d65101a8c8 XL: Implement strided erasure distribution. (#1772)
Strided erasure distribution uses a new randomized
block distribution for each Put operation. This
information is captured inside `xl.json` for subsequent
Get operations.
2016-05-28 15:15:53 -07:00
Harshavardhana b2293c2bf4 XL: Rename, cleanup and add more comments. (#1769)
- xl-v1-bucket.go - removes a whole bunch of code.
- {xl-v1,fs-v1}-metadata.go - add a lot of comments and rename functions
   appropriately.
2016-05-28 15:15:53 -07:00
Harshavardhana 553fdb9211 XL: Bring in support for object versions written during writeQuorum. (#1762)
Erasure is initialized as needed depending on the quorum and onlineDisks.
This way we can manage the quorum at the object layer.
2016-05-28 15:15:53 -07:00
Harshavardhana a9e778f460 XL/fs: initObjectLayer should cleanup tmpMetaPrefix in parallel. (#1752)
Fixes #1747
2016-05-28 15:15:53 -07:00
Harshavardhana ee6645f421 XL: Add additional PartNumber variable as part of xl.json (#1750)
This is needed for verification of incoming parts and to
support variadic part uploads. Which should be sorted
properly.

Fixes #1740
2016-05-28 15:15:53 -07:00
Harshavardhana 1e393c6c5b XL: Add new metadata for checksum. (#1743) 2016-05-28 15:15:53 -07:00
Harshavardhana a00a5c6e7e XL: Multipart update uploads.json properly. (#1741) 2016-05-28 15:15:53 -07:00
Harshavardhana 293d246f95 XL/FS: Rewrite in new format. 2016-05-28 15:15:53 -07:00
Harshavardhana 4e34e03dd4 xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
Harshavardhana a1a667ae5d xl: Change fileMetadata to xlMetadata. (#1404)
Finalized backend format

```
{
    "version": "1.0.0",
    "stat": {
        "size": 24256,
        "modTime": "2016-04-28T00:11:37.843Z"
    },
    "erasure": {
        "data": 5,
        "parity": 5,
        "blockSize": 4194304
    ],
    "minio": {
        "release": "RELEASE.2016-04-28T00-09-47Z"
    }
}
```
2016-04-28 19:27:02 -07:00
Krishna Srinivas 8c85815106 xl: refactor functions to xl-v1-common.go xl-v1-utils.go. (#1357) 2016-04-25 12:47:31 -07:00
Krishna Srinivas becc814531 Xl layer selfheal quorum2
* xl/selfheal: selfheal based on read quorum on GET

* xl: getReadableDisks() also returns whether self-heal is needed so that this info can be used by ReadFile/SelfHeal/StatFile.

* xl: trigger selfheal from StatFile.
2016-04-25 12:47:31 -07:00
Harshavardhana 9bd9441107 xl: Simplify reading metadata and add a new fileMetadata type. (#1346) 2016-04-25 12:47:31 -07:00