Krishnan Parthasarathi
c8dfc4cda4
auth-rpc: Reset token on disconnect ( #2542 )
2016-09-13 21:18:30 -07:00
Bala FA
7922a54c9a
rpc-client: remove unwanted nil check of rpcClient. ( #2538 )
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
bda6bcd5be
Layered rpc-client implementation ( #2512 )
2016-09-13 21:18:30 -07:00
Harshavardhana
7e3e24b394
rpc: client login should ignore server versions.
2016-09-13 21:18:30 -07:00
Harshavardhana
bb0466f4ce
control: Fix controller CLI handling with distributed server object layer.
...
Object layer initialization is done lazily fix it.
2016-09-13 21:18:30 -07:00
Harshavardhana
8797952409
server: Add server command line for running in distributed mode
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
a4691611a7
Move initGracefulShutdown before objectLayer initialization ( #2468 )
2016-09-13 21:18:30 -07:00
awwalker
7c7eb1475d
splitNetPath: Add support for windows paths including volumeNames e.g ip:C:\network\path
2016-09-13 21:18:30 -07:00
Harshavardhana
0bce3d6d63
Fix web handler initialize with distributed lazy init.
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
804d91ef61
storage/rpc-client: Reconnect on network disconnect ( #2436 )
2016-09-13 21:18:30 -07:00
Harshavardhana
43098df9d8
rpc: Re-factor ReadFile behavior client <--> server.
...
Current code did not marshal/unmarshal buffers properly from
the server, The reason being buffers have to be allocated and
sent properly back to client to be consumable.
2016-09-13 21:18:30 -07:00
Harshavardhana
6908a0dcd4
Extract rpc server wrapped errors and translate to storage error.
2016-09-13 21:18:30 -07:00
Harshavardhana
cae5761f16
rpc/client: Add missing rpcTokens for each rpc calls.
2016-09-13 21:18:30 -07:00
Harshavardhana
61af764f8a
Add rpc layer authentication.
2016-09-13 21:18:30 -07:00
Harshavardhana
b4172ad3c8
Rename rpc-{client,server} storage-rpc-{client,server}
2016-09-13 21:18:30 -07:00
Harshavardhana
4917038f55
Move the ObjectAPI() resource to be beginning of each handlers.
...
We should return back proper errors so that the clients can
retry until server has been initialized.
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
b0f3f94163
unify single-node and distributed namespace locking ( #2401 )
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
b7c169d71d
object layer initialization using distributed locking ( #2397 )
...
* vendorized latest minio/dsync
* wip - object layer initialization using distributed locking
2016-09-13 21:18:30 -07:00
Frank
3939c75345
Added distributed RWMutex ( #2369 )
2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
e55926e8cf
distribute: Make server work with multiple remote disks
...
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.
Also adds lock RPC service vendorized minio/dsync
2016-09-13 21:18:30 -07:00
Anis Elleuch
f82f535509
Fix possible race in shutdown callbacks process and simplify shuttting down profiler ( #2684 )
2016-09-13 18:43:45 -07:00
Anis Elleuch
51e337228e
Avoid hiding disk errors in some cases in FS Shutdown ( #2668 )
2016-09-13 11:01:10 -07:00
Kevin Qiu
241c56e6d7
Use Set instead of Add in the event that the request already contains the content-length ( #2683 )
2016-09-13 10:59:40 -07:00
Aditya Manthramurthy
895471afa1
Fixes #2678 ( #2679 )
...
Refactor `ratelimit.acquire()` to properly enforce the *globalMaxConn*
limit.
2016-09-12 15:53:54 -07:00
Harshavardhana
85e2d886bc
bucketPolicy: Trim incoming paths properly for policy verification.
2016-09-11 10:36:54 -07:00
Anis Elleuch
8acf4d112a
Return Owner information in ListObjectV2 only when fetch-owner is specified ( #2654 )
2016-09-10 10:44:38 -07:00
Anis Elleuch
e6abfb3b67
More tests for format-config for a better coverage ( #2655 )
2016-09-10 09:54:22 -07:00
Harshavardhana
040af08473
server: Startup message now prints configured ARNs. ( #2653 )
...
Fixes #2628
2016-09-10 02:23:28 -07:00
Harshavardhana
e3de237eb8
Move config to v7 to fix previous v6 bug on null Notification entries. ( #2650 )
...
Thanks to @koolhead17 for reporting this.
2016-09-10 00:51:25 -07:00
Anis Elleuch
11785529fc
Add Request Not Ready error when presigned date request is not valid ( #2646 )
2016-09-10 00:38:07 -07:00
Anis Elleuch
32201a18ab
Forbid requests generated at least 15 min in the past or in the future ( #2648 )
2016-09-09 15:33:35 -07:00
Anis Elleuch
421cccb1d7
Add naughty disk to simulate disk errors ( #2645 )
2016-09-09 12:53:09 -07:00
Krishna Srinivas
400e9309f1
Signature-V4: Include content-length for signature calculation. ( #2643 )
...
This is to be compatible with clients which includes content-length for signature calculation
(and hence deviating from AWS Signature-v4 spec)
2016-09-09 09:38:07 -07:00
Krishna Srinivas
a4afb312d4
S3-layer: Multiple-delete should return success for non-existent objects. ( #2633 )
...
Fixes #2630
2016-09-07 11:49:12 -07:00
Harshavardhana
afe874f15a
api: CopyObject pipe should be closed pro-actively upon error. ( #2625 )
...
Fixes a deadlock reproduced while running s3verify during
RemoveObject(). Previously held lock by GetObject() inside
the go-routine was never relenquished.
2016-09-06 02:23:32 -07:00
Harshavardhana
b291dbe9c5
s3: Fix compliance bugs reported by s3verify. ( #2624 )
...
- Fixes couple of error strings reported are mismatching.
- Fixes a error HTTP status which was wrong fixed.
- Remove usage of an deprecated PostResponse, au contraire
to their documentation there is no response body in
PostPolicy.
2016-09-06 02:18:02 -07:00
Harshavardhana
bc8f34bfe7
server/mux: Remove unused waitgroup from listenerMux. ( #2614 )
...
Change struct names to be more meaningful.
2016-09-02 19:07:42 -07:00
Harshavardhana
2dc7ecc59b
notifications: Fix bucket notifications for DeleteMultipleObjects. ( #2609 )
...
Now reports bucket notifications for DeleteMultipleObjects API.
Also deletes multiple objects in parallel.
2016-09-02 01:59:08 -07:00
Harshavardhana
7398d737b5
profiler: Fix it properly and generate/save profiles even failure situations. ( #2607 )
...
Fixes #2594
2016-09-01 20:13:11 -07:00
Harshavardhana
511a8cbe04
profiler: Fix how we generate profiles. ( #2596 )
...
Fixes #2594
2016-09-01 03:50:31 -07:00
Anis Elleuch
e79d2381fc
Fix rare 'go test -race' failure in ListenServe{Plain,TLS} ( #2588 )
2016-08-31 16:07:44 -07:00
Krishna Srinivas
a3b4199e9b
FS/Multipart: Allow for parallel uploads of parts with same part number. ( #2587 )
...
CBL client does not close connection when the backup process is stopped, this causes
read() on the stream on the server side to block and hence the lock held on the part
is not released. When the backup process is restarted, we again try to lock on the
part and this will block. Using a unique tmp name and not locking it fixes the problem.
2016-08-31 16:01:32 -07:00
Krishna Srinivas
3aa0574c6b
FS/multipart: Append the parts to a file as and when the parts arrive. ( #2513 )
2016-08-31 13:42:57 -07:00
Harshavardhana
5716f1c199
Fix incorrect error strings. ( #2580 )
2016-08-30 13:47:09 -07:00
Harshavardhana
2020afd200
server: http.Server do not add deadlines causes issues. ( #2581 )
...
Adding deadlines is a no go since Golang doesn't back off
the timers if there is an active i/o in progress.
It is meant to be for applications to handle this themselves
and manually progress the deadlines.
Fixes #2561
2016-08-30 12:52:19 -07:00
Bala FA
7431acb2c4
web: add handlers for set/get bucket policy. ( #2486 )
...
Refer #1858
2016-08-30 10:04:50 -07:00
Harshavardhana
2f9975c76c
notifications: GetObjectInfo() later if bucket notifications are set. ( #2575 )
2016-08-30 03:42:37 -07:00
Harshavardhana
7270ca4157
pkg/wildcard: Simplify the wildcard logic further. ( #2555 )
2016-08-27 00:27:17 -07:00
Brendan Ashworth
b67c8970c9
bucket-handlers: fix missing return statement ( #2553 )
...
If the location was invalid, it would write an error response but then
continue to attempt to make the bucket. Whether or not it would succeed,
it would attempt to call response.WriteHeaders twice in a row, which
would cause a message to be logged to the server console (bad).
Here is the relevant Go code:
c80e0d374b/src/net/http/server.go (L878-L881)
2016-08-25 20:00:47 -07:00
Mohit Agarwal
418921de89
minor cleanup
...
- Reused contains() from utils.go at a couple of places
- Cleanup in return statements and boolean checks
2016-08-24 22:54:34 -07:00
Harshavardhana
ec4260d260
api: BucketNotification should disallow duplicate notification. ( #2539 )
...
Added checks to look for duplicated notification configs.
Fixes #2472
2016-08-23 18:42:30 -07:00
Harshavardhana
c39d3db7a0
server/tls: allocate tls.Config{} properly. ( #2537 )
...
There is a golang bug which crashes the server, allocate tls.Config properly to avoid
this problem.
```
2016/08/22 20:21:03 http: panic serving 127.0.0.1:40744: runtime error: invalid memory address or nil pointer dereference
goroutine 38 [running]:
net/http.(*conn).serve.func1(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1389 +0xc1
panic(0xbefa80, 0xc820010140)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/runtime/panic.go:443 +0x4e9
crypto/tls.(*Conn).serverHandshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/handshake_server.go:43 +0x4d6
crypto/tls.(*Conn).Handshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/conn.go:1035 +0x169
net/http.(*conn).serve(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1405 +0x382
created by net/http.(*Server).Serve
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:2137 +0x44e
```
Fixes #2536
2016-08-23 00:20:15 -07:00
Harshavardhana
2da0cfc904
signature: Use the client sent region if region is properly validated. ( #2534 )
...
A properly validated region string from the client should be once
validated properly.
Fixes #2521
2016-08-22 19:43:36 -07:00
Harshavardhana
9dd0e3dc44
config: Initialize notify for fresh configs. ( #2531 )
2016-08-22 16:04:04 -07:00
Krishna Srinivas
45c928e2f5
boot: checkPortAvailability() should fail only for EADDRINUSE error and ignore other errors. ( #2527 )
...
fixes #2510
2016-08-22 10:20:01 -07:00
Anis Elleuch
07506358ff
Refactor Heal RPC and add Shutdown RPC ( #2488 )
2016-08-21 12:06:53 -07:00
Harshavardhana
975eb31973
api/bucketPolicy: Use minio-go/pkg/set and fix bucket policy regression. ( #2506 )
...
Current master has a regression 'mc policy <policy-type> alias/bucket/prefix'
does not work anymore, due to the way new minio-go changes do json marshalling.
This led to a regression on server side when a ``prefix`` is provided
policy is rejected as malformed from th server which is not the case with
AWS S3.
This patch uses the new ``minio-go/pkg/set`` package to address the
unmarshalling problems.
Fixes #2503
2016-08-20 03:16:38 -07:00
Jesse Lucas
f2fd8b0265
Adding test coverage for server-mux.go by creating TestListenAndServeTLS(). ( #2493 )
...
Editing config.go to add lock on customConfigPath to avoid race condition
when setGlobalConfigPath() and getConfigPath() are called concurrently.
2016-08-19 01:29:50 -07:00
Jesse Lucas
a8052889fe
server-mux_test.go updating test to fix race condition with TestListenAndServe ( #2485 )
2016-08-18 19:45:03 -07:00
Harshavardhana
bccf549463
server: Move all the top level files into cmd folder. ( #2490 )
...
This change brings a change which was done for the 'mc'
package to allow for clean repo and have a cleaner
github drop in experience.
2016-08-18 16:23:42 -07:00
Harshavardhana
aabfd541e1
Merge cmd/donut into minio cmd, deprecate controller RPC request
2015-07-31 12:57:15 -07:00
Harshavardhana
f15375426a
minor: Add commands into donut template
2015-07-28 19:40:02 -07:00
Harshavardhana
0eefbdef0c
use new app.ExtraInfo inside minio and donut commands properly
2015-07-24 23:55:18 -07:00
Harshavardhana
63c9cf0c4b
Move from Minimalist Object Storage to Minio Cloud Storage
2015-07-24 17:51:40 -07:00
Harshavardhana
4498662c16
Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut
2015-07-15 11:55:57 -07:00
Harshavardhana
1e80925ca7
Remove dependency on minio/cli for make.go fixes #738
2015-07-13 11:39:28 -07:00
Harshavardhana
f360ee0ab5
Add mkdonut examples
2015-07-12 21:37:57 -07:00
Harshavardhana
55e4d0c6a5
mkdonut now creates a donut processing cli args
2015-07-12 21:21:31 -07:00
Harshavardhana
535bcc3eac
Add mkdonut command
2015-07-12 19:16:36 -07:00
Harshavardhana
432275e966
Full restructure in accordance with
...
- pkg/{subsystem}/{package} style
- modify Makefile to reflect the new style,
consolidate various entries
- add a dummy ``main.go`` at top level
2015-01-14 11:29:04 -08:00
Harshavardhana
462808b87a
Disable sha1,sha256,sha512 avx,avx2,sse3 crypto implementations.
...
Re-implement them later, once stable
2015-01-06 17:02:25 -08:00
Frederick F. Kautz IV
992dea1bf3
Adding index command
2015-01-06 22:26:09 +13:00
Harshavardhana
fff6b589cf
Rename split-file --> split
2015-01-03 18:03:21 -08:00
Frederick F. Kautz IV
f5010e1ee2
Adding initial split-file command
2015-01-04 14:47:37 +13:00
Harshavardhana
0212079cd5
Remove template files, make them const strings instead
2015-01-01 17:31:51 -08:00
Harshavardhana
0e4a26e3b4
Add Diskattrmap, Scsiattrmap for probed scsi devices
...
Additional changes
- Use ``iota`` for constants
- Remove unncessary C header files
- ``new-cmd`` now depends on codegangsta cli
2014-12-30 02:37:47 -08:00
Harshavardhana
6b36b5c551
A full restructure
2014-12-29 21:22:26 -08:00
Harshavardhana
13650e088c
Implement md5c function, slower than Golang's implementation
...
keeping it in repo to make further improvements and also rename
minio-hash ---> crypto
2014-12-21 03:50:33 -08:00
Frederick F. Kautz IV
2278df9910
Adding minio-hash with streaming crypto hashes
2014-12-21 13:04:41 +13:00
Harshavardhana
d4c05f11c7
Update documentation
2014-12-20 09:09:35 -08:00
Harshavardhana
b0e986c82c
Rename appname to commandname for consistency
2014-12-19 00:16:41 -08:00
Harshavardhana
5268a62166
Add minio-cli stub generator
2014-12-19 00:00:14 -08:00
Harshavardhana
65e2031cfc
Implement add, commit - instead of put, encoded
2014-12-15 21:55:54 -08:00
Harshavardhana
98119707fb
Move wrappers at pkgs/minio back to cmd/minio - as their usage is limited
2014-12-15 20:09:50 -08:00
Harshavardhana
3ec5c2fb8f
Implement commands
...
Commands implemented
$ minio put <objectname> <file>
$ minio encode <objectname>
$ minio get <objectname>
$ minio list
TODO :
$ minio init
$ minio verify
...
...
2014-12-15 14:59:15 -08:00
Frederick F. Kautz IV
5efc0d54f8
storage.ObjectStorage List() is now List(objectPath string)
2014-12-15 19:30:08 +13:00
Frederick F. Kautz IV
05baa6d9a2
Removing fsstorage
2014-12-15 19:15:22 +13:00
Harshavardhana
fc50291cac
fsStorage should use appendStorage with offsets
...
- also takes in 'blockSize' cli option.
- robustness fixes
2014-12-12 17:18:16 -08:00
Harshavardhana
0a0e1111cd
Add list() object support for erasure and append storage drivers
...
- Reply back objects with their protectionlevel and md5sum
- // TODO hash value
- Calculate md5sum after "storeBlocks()", to make sure data is committed
2014-12-12 02:52:37 -08:00
Harshavardhana
f71b323e32
Provide new cli options for 'put' command
...
- --protection-level
- --block-size
2014-12-11 22:56:12 -08:00
Harshavardhana
0bd2b8516e
Build fix, fixing without a PR
2014-12-11 20:44:06 -08:00
Harshavardhana
dddc555491
Use objectName with basename for the filepath
2014-12-11 15:35:25 -08:00
Frederick F. Kautz IV
568ca4e1bc
erasure encoded demo
2014-12-11 00:32:32 -08:00
Frederick F. Kautz IV
a1040ae657
erasure-demo put now takes 1 or 2 arguments
2014-12-10 20:59:49 -08:00
Frederick F. Kautz IV
19da9760b3
ObjectStorage refactor to take io.Reader instead of []byte
2014-12-10 20:57:59 -08:00
Frederick F. Kautz IV
cae9b288b0
Adding List(path) to object storage definition and accompanying definition to fs
2014-12-10 19:15:03 -08:00
Harshavardhana
9dd3dd8b53
Merge pull request #100 from harshavardhana/pr_out_add_crc32c_checksum_support_for_erasure_chunks
...
Add crc32c checksum support for erasure chunks
2014-12-10 11:53:23 -08:00
Harshavardhana
01c0e45b5c
Add crc32c checksum support for erasure chunks
...
Now upon conditions
- if checksum '.cksum' file is missing
- or corrupted chunk (mismatching chunk)
- or missing chunk entirely
Decoding works consistently
2014-12-10 00:58:15 -08:00
Harshavardhana
f6a5b61b00
Make get/put more robust
...
- init folder if not present already in 'get()' call
- put.go was referencing wrong args - fix it
- if no OBJECTNAME is specified during 'get' - grab a
list of files under "~/.minio"
- Provide new API GetList() for ObjectStorage struct
2014-12-09 03:32:31 -08:00
Frederick F. Kautz IV
d1ebf935da
Adding extremely simple fs based get and put in erasure-demo
2014-12-08 20:20:25 -08:00