Commit graph

85 commits

Author SHA1 Message Date
Frank ea406754a6 New dsync and added ForceUnlock to lock rpc server (#2956)
* Update dsync and added ForceUnlock function
* Added test cases for ForceUnlock
2016-10-17 01:53:29 -07:00
Harshavardhana a681af6953 Update minio browser with new changes. (#2940)
- Bucket policy set/unset support.
- Shareable URL support.
- Delete object support.
2016-10-15 08:51:53 -07:00
Harshavardhana 2f520ed92f Remove errors package, add comments and simplify. (#2925) 2016-10-14 12:31:00 -07:00
Mateusz Gajewski c03ce0f74a Display SSL expiry warnings (#2925) 2016-10-14 12:30:36 -07:00
Anis Elleuch df59967f59 Avoid checking date header of web requests by properly applying generic handlers (#2914) 2016-10-12 12:58:36 -07:00
Frank e53a9f6cab Update vendorized version of dsync that relaxes read quorum to N/2 (#2874) 2016-10-07 08:02:59 -07:00
Harshavardhana ed676667d0 vendor: update reedsolomon package with new changes. (#2870)
- Cached inverse matrices for better reconstruct performance.
- New error reconstruction required is returned, helpful in
  initiating healing.
2016-10-06 21:57:42 -07:00
Aditya Manthramurthy 315e66858c Add PostgreSQL notifier (#2739) (#2824)
* The user is required to specify a table name and database connection
  information in the configuration file.

* INSERTs and DELETEs are done via prepared statements for speed.

* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
  the use of UPSERT.

* Creates the table if it does not exist with the given table name using
  a query like:

    CREATE TABLE myminio (
        key varchar PRIMARY KEY,
        value JSONB
    );

* Vendors some required libraries.
2016-10-03 17:29:55 -07:00
Anis Elleuch 1e6afac3bd Add NATS notifier (#2795) 2016-09-29 23:42:10 -07:00
Bala FA aa579bbc20 web: add method to get all policies for given bucket name. (#2756)
Refer #1858
2016-09-22 23:06:45 -07:00
Frank df2ef64d20 Upgrade to new dsync version incl. stale lock detection (#2708) 2016-09-16 00:30:55 -07:00
Karthic Rao 8bd78fbdfb performance: gjson parsing for readXLMeta, listParts, getObjectInfo. (#2631)
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
  for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
2016-09-13 21:18:30 -07:00
Harshavardhana f655592ff5 vendorize new dsync with new changes of logging. 2016-09-13 21:18:30 -07:00
Harshavardhana 9605fde04d controller/auth: Implement JWT based authorization for controller. (#2544)
Fixes #2474
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi bda6bcd5be Layered rpc-client implementation (#2512) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi c33d1b8ee6 Vendorize minio/dsync for server-side read lock (#2484)
- Prevention of stale lock accumulation.
- Removal of dead code.
2016-09-13 21:18:30 -07:00
Harshavardhana 83074ed57e Add missing rpc-client.go - missing in previous commit. 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi 64a7e6992e Vendorize rpc reconnect changes from minio/dsync (#2405) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi b0f3f94163 unify single-node and distributed namespace locking (#2401) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi b7c169d71d object layer initialization using distributed locking (#2397)
* vendorized latest minio/dsync

* wip - object layer initialization using distributed locking
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi e55926e8cf distribute: Make server work with multiple remote disks
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.

Also adds lock RPC service vendorized minio/dsync
2016-09-13 21:18:30 -07:00
Bala FA 7431acb2c4 web: add handlers for set/get bucket policy. (#2486)
Refer #1858
2016-08-30 10:04:50 -07:00
Harshavardhana 975eb31973 api/bucketPolicy: Use minio-go/pkg/set and fix bucket policy regression. (#2506)
Current master has a regression 'mc policy <policy-type> alias/bucket/prefix'
does not work anymore, due to the way new minio-go changes do json marshalling.
This led to a regression on server side when a ``prefix`` is provided
policy is rejected as malformed from th server which is not the case with
AWS S3.

This patch uses the new ``minio-go/pkg/set`` package to address the
unmarshalling problems.

Fixes #2503
2016-08-20 03:16:38 -07:00
Harshavardhana 0a3d43273f vendor: sha256 32bit updated. (#2459) 2016-08-16 16:19:29 -07:00
Frank f239fcac67 Switched to faster minio/sha256-simd implementation (#2320) 2016-07-28 14:44:37 -07:00
Harshavardhana f248089523 api: Implement bucket notification. (#2271)
* Implement basic S3 notifications through queues

Supports multiple queues and three basic queue types:

1. NilQueue -- messages don't get sent anywhere
2. LogQueue -- messages get logged
3. AmqpQueue -- messages are sent to an AMQP queue

* api: Implement bucket notification.

Supports two different queue types

- AMQP
- ElasticSearch.

* Add support for redis
2016-07-23 22:51:12 -07:00
Harshavardhana 35d438e0ff vendorize: update all vendorized packages. (#2206)
Bring in new changes from upstream for all the packages.

Important ones include
   - gorilla/mux
   - logrus
   - jwt
2016-07-14 14:59:20 -07:00
Harshavardhana 169c72cdab vendor: Bring new updates from blake2b-simd repo. (#2094)
This vendorization is needed to bring in new improvements
and support for AVX2 and SSE.

Fixes #2081
2016-07-06 18:24:31 -07:00
Harshavardhana 48ac34919f browser: Add new release for ui-assets.go (#2070)
update `ui-assets.go` using `x-amz-date` for JSON rpc.
2016-07-02 10:54:17 -07:00
Harshavardhana d484157d67 XL/bitrot: Migrate to new blake2b-simd SIMD optimized implementation. (#2031)
Thanks for Frank Wessels <fwessels@xs4all.nl> for all the heavy lifting work.

Comparative benchmarks are as below.
```
benchmark                old ns/op     new ns/op     delta
BenchmarkHash64-4        742           411           -44.61%
BenchmarkHash128-4       681           346           -49.19%
BenchmarkWrite1K-4       4239          1497          -64.69%
BenchmarkWrite8K-4       33633         11514         -65.77%
BenchmarkWrite32K-4      134091        45947         -65.73%
BenchmarkWrite128K-4     537976        183643        -65.86%

benchmark                old MB/s     new MB/s     speedup
BenchmarkHash64-4        86.18        155.51       1.80x
BenchmarkHash128-4       187.96       369.10       1.96x
BenchmarkWrite1K-4       241.55       683.87       2.83x
BenchmarkWrite8K-4       3897.06      11383.41     2.92x
BenchmarkWrite32K-4      977.48       2852.63      2.92x
BenchmarkWrite128K-4     243.64       713.73       2.93x
```

Fixes #2030
2016-06-29 02:06:35 -07:00
Harshavardhana e10934a88e bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957)
Fixes #1952
2016-06-22 17:13:26 -07:00
Harshavardhana 7f38f46e20 vendor: update klauspost/reedsomon package with upstream changes. (#1912) 2016-06-17 15:16:26 +05:30
Harshavardhana 4ab57f7d60 server: terminal width should fallback to 80x25. (#1895)
Some environments might disable access to `/dev/tty`, fall
back to '80' in such scenarios.

Move to 'cheggaaa/pb' package for better cross platform
support on fetching terminal width.

Fixes #1891
2016-06-12 19:35:28 -07:00
Harshavardhana 2e4ab71303 Web: Update with ui changes. (#1808) 2016-05-31 02:01:02 -07:00
karthic rao 2da34e4668 vendor changes to pkg/profile (#1671) 2016-05-18 09:22:06 -07:00
Harshavardhana 8828fd1e5c vendor: Remove unused packages. 2016-05-15 00:04:56 -07:00
Harshavardhana b66c3bf35e server: Enable server profiling as needed. (#1565) 2016-05-09 16:18:56 -07:00
Harshavardhana ba7a55c321 xl: ReedSolomon code fix small file erasure bug. (#1431)
For files less than 'dataBlocks', erasure encoding would fail
with short data due to a bug in the implementation itself.

Relax the error return, even a single byte can be properly
erasure coded without issues.

Fixes #1413
2016-05-01 15:30:40 -07:00
Harshavardhana a9935f886c vendor: update reedsolomon package with new perm improvements. (#1422) 2016-04-29 14:59:03 -07:00
Harshavardhana a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
Harshavardhana 444d1f8a65 miniobrowser: Vendorize to new changes in miniobrowser. 2016-04-21 20:35:48 -07:00
Harshavardhana 8603185f2f browser: Add new ui-assets.go 2016-04-08 01:47:30 -07:00
Harshavardhana fbd02d530d web: Change /rpc to /webrpc 2016-04-08 01:46:46 -07:00
Bala.FA 083e4e9479 backend/fs: Refactor multipart upload
This patch modifies multipart upload related functions as below

* New multipart upload call creates file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.uploadid

* Put object part call creates file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.PART_NUMBER.MD5SUM_STRING

* Abort multipart call removes all files matching
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.*

* Complete multipart call does
  1. creates a staging file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete.TEMP_NAME
  then renames to
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
  2. rename staging file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
  to EXPORT_DIR/BUCKET/PATH/TO/OBJECT
2016-04-05 22:22:29 +05:30
Harshavardhana ba3a5805c1 vendorize: Add updated ui-assets.go. 2016-04-02 17:27:36 -07:00
Krishna Srinivas 331890c4c8 UI-handler: remove minio-go dependancy. 2016-04-01 13:56:32 +05:30
Harshavardhana aaf97ea02c config/main: Re-write config files - add to new config v3
- New config format.

```
{
	"version": "3",
	"address": ":9000",
    "backend": {
          "type": "fs",
          "disk": "/path"
    },
	"credential": {
		"accessKey": "WLGDGYAQYIGI833EV05A",
		"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
	},
	"region": "us-east-1",
	"logger": {
		"file": {
			"enable": false,
			"fileName": "",
			"level": "error"
		},
		"syslog": {
			"enable": false,
			"address": "",
			"level": "debug"
		},
		"console": {
			"enable": true,
			"level": "fatal"
		}
	}
}
```

New command lines in lieu of supporting XL.

Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~

Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~

For 'fs' backend it starts the server.
~~~
$ minio server
~~~

For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~

Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
2016-03-23 19:16:09 -07:00
Harshavardhana 5606232567 vendor: Update ui-assets with new changes and release. 2016-03-10 17:36:32 -08:00
Harshavardhana e781959d5b vendor: Add minio-go vendor updates. 2016-03-10 14:33:15 -08:00
Harshavardhana 164dfe2ec9 console: Fix console color printing on windows.
Print colored accessKeyID and secretAccessKey are unreadable on windows
on command prompts and powershell.

Use the console package from minio client.
2016-03-04 10:07:19 -08:00