Commit graph

962 commits

Author SHA1 Message Date
Krishnan Parthasarathi c8f57133a4 Implement list, clear locks REST API w/ pkg/madmin support (#3491)
* Filter lock info based on bucket, prefix and time since lock was held
* Implement list and clear locks REST API
* madmin: Add list and clear locks API
* locks: Clear locks matching bucket, prefix, relTime.
* Gather lock information across nodes for both list and clear locks admin REST API.
* docs: Add lock API to management APIs
2017-01-03 23:39:22 -08:00
Harshavardhana f57f773189 admin: Add missing madmin examples and API docs. (#3483) 2016-12-20 18:49:48 -08:00
Harshavardhana 1d134c1a94 admin: Fix wrong vendorized location for s3signer and s3utils. 2016-12-20 15:18:37 -08:00
Anis Elleuch 329a910b86 Admin Lib: Implement Service API (#3426)
Three APIs were added to control a minio server

* NewAdminClient()
* ServiceStop()
* ServiceRestart()
* ServiceStatus()
2016-12-20 14:45:17 -08:00
Harshavardhana b363709c11 caching: Optimize memory allocations. (#3405)
This change brings in changes at multiple places

 - Reuse buffers at almost all locations ranging
   from rpc, fs, xl, checksum etc.
 - Change caching behavior to disable itself
   under low memory conditions i.e < 8GB of RAM.
 - Only objects cached are of size 1/10th the size
   of the cache for example if 4GB is the cache size
   the maximum object size which will be cached
   is going to be 400MB. This change is an
   optimization to cache more objects rather
   than few larger objects.
 - If object cache is enabled default GC
   percent has been reduced to 20% in lieu
   with newly found behavior of GC. If the cache
   utilization reaches 75% of the maximum value
   GC percent is reduced to 10% to make GC
   more aggressive.
 - Do not use *bytes.Buffer* due to its growth
   requirements. For every allocation *bytes.Buffer*
   allocates an additional buffer for its internal
   purposes. This is undesirable for us, so
   implemented a new cappedWriter which is capped to a
   desired size, beyond this all writes rejected.

Possible fix for #3403.
2016-12-08 20:35:07 -08:00
Anis Elleuch 0cef971832 Fix max cache size calculation when system RAM is inferior to the default cache size (#3410) 2016-12-06 16:09:26 -08:00
Harshavardhana 1b85302161 Fix spelling and golint errors. (#3266)
Fixes #3263
2016-11-15 18:14:23 -08:00
Harshavardhana 2f373684f5 Fix the server startup messages and help text. (#3211) 2016-11-09 23:37:12 -08:00
Harshavardhana f3c6c55719 posix: Fix windows performance issues. (#3132)
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.

This verification is only needed during startup and we
let things fail at a later stage on windows.
2016-10-31 09:34:44 -07:00
Anis Elleuch 9417614a8e Recalculate free minimum disk space (#2788)
* Fix calculating free space disk by using blocks available for unprivileged user

* Use fixed minimal free disk space instead of percentage
2016-09-27 12:46:38 -07:00
Krishnan Parthasarathi 669783f875 Purge stale object cache entry (#2770) 2016-09-23 19:55:28 -07:00
Harshavardhana 7270ca4157 pkg/wildcard: Simplify the wildcard logic further. (#2555) 2016-08-27 00:27:17 -07:00
Harshavardhana ec4260d260 api: BucketNotification should disallow duplicate notification. (#2539)
Added checks to look for duplicated notification configs.

Fixes #2472
2016-08-23 18:42:30 -07:00
Harshavardhana 4dec50ba51 build: Add platform specific fixes. 2016-08-16 14:40:41 -07:00
Harshavardhana 76d56c6ff2 typo: Fix typos across the codebase. (#2442) 2016-08-15 02:44:48 -07:00
Harshavardhana 7e46055a15 api/handlers: Implement streaming signature v4 support. (#2370)
* api/handlers: Implement streaming signature v4 support.

Fixes #2326

* tests: Add tests for quick/safe
2016-08-08 20:56:29 -07:00
Harshavardhana b23605a2b5 pkg/objcache: Add more tests. (#2371) 2016-08-06 10:22:14 -07:00
Harshavardhana 064c51162d api: Add new ListenBucketNotificationHandler. (#2336)
This API is precursor before implementing `minio lambda` and `mc` continous replication.

This new api is an extention to BucketNofication APIs.

// Request
```
GET /bucket?notificationARN=arn:minio:lambda:us-east-1:10:minio HTTP/1.1
...
...
```

// Response
```

{"Records": ...}
...
...
...
{"Records": ...}
```
2016-08-04 22:01:58 -07:00
Harshavardhana de5d5ff241 pkg/crypto: Deprecate cgo sha256 version. (#2354) 2016-08-04 03:19:36 -07:00
karthic rao 2e0742e309 bucket policy: Support for '?' wildcard. (#2353)
- Support for '?' wildcard for resource matching.

- Wildcard package is added with Match functions.

- Wildcard.Match supports '*' and wild.MatchExtended supports both '*'
  and '?' wildcards in the pattern string.

- Tests for the same for the wide range of cases.
2016-08-04 00:41:32 -07:00
Krishna Srinivas 69fd196471 Object-cache: enforce cache size to be less than RAM. (#2338) 2016-08-02 10:04:35 -07:00
Krishna Srinivas 043ddbd834 optimize memory allocation during erasure-read by using temporary buffer pool. (#2259)
* XL/erasure-read: optimize memory allocation during erasure-read by using temporary buffer pool.

With the change the buffer needed during GetObject by erasureReadFile is allocated only once.
2016-07-25 14:17:01 -07:00
Harshavardhana c0c8a8430e XL/PutObject: Add single putObject and multipart caching. (#2115)
- Additionally adds test cases as well for object cache.
- Adds auto-expiry with expiration and cleanup time interval.

Fixes #2080
Fixes #2091
2016-07-08 20:34:27 -07:00
Krishna Srinivas 01cbacd803 object-cache: use golang bytes.Buffer and bytes.NewReader instead of custom implementation. (#2108) 2016-07-06 01:29:49 -07:00
Harshavardhana 8a028a9efb handler/PUT: Handle signature verification through a custom reader. (#2066)
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.

Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().

Thanks to Krishna for his inputs on this.

Fixes #2058
Fixes #2054
Fixes #2087
2016-07-05 01:04:50 -07:00
Bala FA 1ad5fb8f76 posix: checkDiskFree() also checks free inodes. (#2086)
Previously checkDiskFree() checks for free available space.  This
patch enables checkDiskFree() also checks for free inodes in linux and
free clusters in windows.

Fixes #2075
2016-07-03 22:34:45 -07:00
Harshavardhana d2557bb538 XL: GetObject caching implemented for XL. (#2017)
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.

Currently expiration is not implemented.
2016-07-03 17:15:38 -07:00
Harshavardhana e10934a88e bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957)
Fixes #1952
2016-06-22 17:13:26 -07:00
Harshavardhana 4ab57f7d60 server: terminal width should fallback to 80x25. (#1895)
Some environments might disable access to `/dev/tty`, fall
back to '80' in such scenarios.

Move to 'cheggaaa/pb' package for better cross platform
support on fetching terminal width.

Fixes #1891
2016-06-12 19:35:28 -07:00
Harshavardhana 51f3d4e0ca XL/multipart: statPart should ignore errDiskNotFound. (#1862)
startPart should also take uploadId and partName as arguments.
2016-06-07 18:15:04 -07:00
Bala FA 51bb613fdf pkg/safe: remove temporary file on failure (#1774) 2016-05-28 15:15:53 -07:00
Harshavardhana 64b0976e1b Remove probe and tasker. (#1733)
Fixes #1717
2016-05-24 18:43:33 -07:00
Harshavardhana 49141eb3e4 http: Remove minhttp package and use standard Golang http. (#1587)
The functionality provided by minhttp will be implemented
cleanly through our own APIs. Since we are not going to
send SIGUSR2 and manage configuration in that manner, it
doesn't make sense to use minhttp.

Fixes #1586
2016-05-10 18:03:00 -07:00
Harshavardhana d1fa1d9352 Remove binary files from previous commit. 2016-05-10 15:49:17 -07:00
karthic rao 26e2c4bf4d Replacing fastsha256 with crypto/sha256 package from golang standard package (#1584) 2016-05-10 14:20:11 -07:00
Harshavardhana 5133ea50bd xl/fs: Make i/o operations atomic. (#1496) 2016-05-05 20:28:22 -07:00
Harshavardhana 7ae40eb1bb minhttp: Remove probe usage, move to golang error. (#1459)
Fixes #1454
2016-05-03 01:07:34 -07:00
Harshavardhana 4e34e03dd4 xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
Harshavardhana a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
Harshavardhana ff4e04d942 atomic/fs: use safe package for atomic writes, even in multipart. 2016-04-06 16:05:30 -07:00
Harshavardhana 379e0abf03 cleanup: Remove old donut/xl code and erasure implementation.
This is a change to bring in 'klauspost/reedsolomon' library
in #1270 patch.
2016-04-02 17:30:35 -07:00
Harshavardhana 6037fe66e9 minio: Simplify for gosimple tool complaints. 2016-04-02 17:28:54 -07:00
Harshavardhana efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface.
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00
Harshavardhana 1502e2f29f Merge pull request #1265 from vadmeste/add_fbsd_support
Add simple FreeBSD support, make the minio project compilable
2016-03-26 22:28:07 -07:00
Harshavardhana 9dca46e156 signature: Use a layered approach for signature verification.
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.

In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
2016-03-26 15:21:05 -07:00
Anis Elleuch 663f24064b Add simple FreeBSD support, make the minio project compilable 2016-03-26 22:39:34 +01:00
Harshavardhana aaf97ea02c config/main: Re-write config files - add to new config v3
- New config format.

```
{
	"version": "3",
	"address": ":9000",
    "backend": {
          "type": "fs",
          "disk": "/path"
    },
	"credential": {
		"accessKey": "WLGDGYAQYIGI833EV05A",
		"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
	},
	"region": "us-east-1",
	"logger": {
		"file": {
			"enable": false,
			"fileName": "",
			"level": "error"
		},
		"syslog": {
			"enable": false,
			"address": "",
			"level": "debug"
		},
		"console": {
			"enable": true,
			"level": "fatal"
		}
	}
}
```

New command lines in lieu of supporting XL.

Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~

Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~

For 'fs' backend it starts the server.
~~~
$ minio server
~~~

For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~

Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
2016-03-23 19:16:09 -07:00
Karthic Rao c8570edaab Issue of 'mc ls' when prefix is a directory fixed, tests added for GetObjectInfo 2016-03-23 19:42:04 +05:30
Karthic Rao 7be79b507b Changing to empty response when prefixDir doesn't exist 2016-03-23 04:46:10 +05:30
Karthic Rao ff41c050d5 Formatting issues fixed. 2016-03-22 15:55:29 +05:30