Harshavardhana
636788997b
Minor optimization avoiding one unnecessary copy
2015-06-11 14:23:05 -07:00
Harshavardhana
6d44274db6
Make sure to reply back ETag with quotes for s3 compliance
2015-06-09 01:13:02 -07:00
Harshavardhana
a947767339
Remove unnecessary tlsHelpers, SSL works with domain certificate
2015-06-08 20:54:41 -07:00
Harshavardhana
c6ecd94420
Enable TLS and use it
2015-06-08 20:10:59 -07:00
Harshavardhana
64b014369c
Return back entity too large for createObject requests bigger than cache size
2015-06-08 17:10:07 -07:00
Harshavardhana
ab1976bc14
Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server
2015-06-08 15:15:37 -07:00
Harshavardhana
2976f109df
More multipart fixes, filter out parts and multipart metadata from listObjects
2015-06-08 14:08:36 -07:00
Harshavardhana
e08d59be00
Make sure to delete uploadid's from active session file properly
2015-06-08 12:25:51 -07:00
Harshavardhana
0d3710cb9b
Store in json file instead of gob for fs metadata
2015-06-08 12:08:01 -07:00
Harshavardhana
8efc842b59
Add delete handlers and reply back as 'NotImplemented' instead of 404
2015-06-08 11:06:06 -07:00
Harshavardhana
8a320c4447
Enhance logging handler to print in humanize form for ContentLength
2015-06-08 02:14:09 -07:00
Harshavardhana
cefd740002
Put back the older conn_limit
2015-06-06 18:11:38 -07:00
Harshavardhana
371651bde6
Add rate limiter instead of connection limit for now
2015-06-06 18:04:55 -07:00
Harshavardhana
21916dc0a1
Fix a deadlock in trove when evicting objects when we reach the maxSize used
2015-06-06 16:02:52 -07:00
Harshavardhana
8b43655d9f
Make connection limit configurable
2015-06-06 14:31:50 -07:00
Harshavardhana
a344f5b34d
Misc changes before capturing proper video
...
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
2015-06-03 18:27:15 -07:00
Harshavardhana
097f70a3c5
Enable filesystem for API tests, this patch also implements ListMultipartUploads()
2015-06-01 20:57:13 -07:00
Harshavardhana
bfb9209689
Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service
2015-06-01 16:43:43 -07:00
Harshavardhana
d86ba5f336
Implement multipart support
...
- implemented NewMultipartUpload()
- implemented CreateObjectPart()
- implemented ListObjectParts()
- implemented AbortMultipartUpload()
- implemented CompleteMultipartUpload()
2015-06-01 16:25:11 -07:00
Harshavardhana
c07d48e7e8
ErrorResponse from minio server needed proper xml translations
2015-06-01 12:19:27 -07:00
Harshavardhana
a91e519692
Make caching a package trove, and use it inside memory driver
2015-05-31 18:41:56 -07:00
Anis Elleuch
7e5a26f42c
Protect the exported method Delete()
2015-05-31 20:29:42 +01:00
Harshavardhana
82a0eac659
Add filesystem factory functions and add related cli options
2015-05-30 03:23:06 -07:00
Harshavardhana
0cc63706bb
Resurrect back fileystem driver
2015-05-30 03:12:01 -07:00
Anis Elleuch
d360d0fec3
Forbid the upload of files bigger than the memory backend capacity
2015-05-28 22:07:02 +01:00
Harshavardhana
554a244b47
Add donut OSX support
2015-05-26 02:14:34 -07:00
Harshavardhana
e46660a966
Use http.TimeFormat to return, S3 expects us to return in GMT not UTC
2015-05-25 20:17:04 -07:00
Harshavardhana
19250296c6
Handle authorization header better
2015-05-24 20:59:53 -07:00
Harshavardhana
9666f2e5bf
Relax minium object size limit, one can really upload a 0byte file
...
For example
~~~
$ mc ls https://s3.amazonaws.com/ferenginar/test.go
[2015-05-23 12:02:06 PDT] 0B test.go
$ curl -i -X HEAD https://s3.amazonaws.com/ferenginar/test.go
HTTP/1.1 200 OK
x-amz-id-2: ZMWLriPH+uQJ8IsaMHmuNOU/FXvdSq+s6O7ugUI9hZ695XJTNAZ1utKxh03w5Jcf
x-amz-request-id: F5A11F533B74DD8F
Date: Sat, 23 May 2015 19:19:29 GMT
Last-Modified: Sat, 23 May 2015 19:02:06 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3
~~~
2015-05-23 12:19:16 -07:00
Harshavardhana
e11f9110b6
add missing validate access keys after being extracted from AuthHeader
2015-05-21 19:23:47 -07:00
Harshavardhana
ad60b97d52
Fix some donut TODO's regarding bucket created date
2015-05-21 11:13:49 -07:00
Harshavardhana
cd817d34df
Finish last bits of api testsuite on server
2015-05-21 02:57:03 -07:00
Harshavardhana
f00bfd38da
Bump gocyclo back to 16
2015-05-19 18:32:50 -07:00
Harshavardhana
3d61ba5930
Fix duplicate map allocation
2015-05-18 13:14:01 -07:00
Harshavardhana
e4c0d574dc
Update totalnumber of multiparts inside CreateObjectPart(), also introduce support for UploadIDMarker and Prefix for listMultipartUploads
2015-05-18 12:24:36 -07:00
Harshavardhana
7ce47130fd
HEAD request should have Content-Length for only successful response, there is no response body for errors, just header is sufficient - fixes #603
...
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13
""" in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. """
2015-05-15 21:55:59 -07:00
Harshavardhana
317637fcb8
Remove redundant WriteHeader calls
2015-05-15 20:58:28 -07:00
Harshavardhana
5c5761ae16
Remove unnecessary GetObjectMetadata prefix argument
2015-05-15 19:25:17 -07:00
Harshavardhana
6275238a47
For multiparts use its own cache avoids deadlocks during object expiration
...
Using same object cache as regular objects for multiparts causes deadlocks during object expiration routine.
As a side affect this also fixes the bug of listing multiparts during regular 'listing' of objects.
2015-05-14 20:26:59 -07:00
Harshavardhana
c41ebf7535
Fixing a race in NewMultipartUpload() session for simultaneous requests, observed at play.minio.io
2015-05-14 19:11:19 -07:00
Harshavardhana
8fb427a892
Add more tests for multipart now in driver api_testsuite
2015-05-14 17:28:39 -07:00
Harshavardhana
dba7c94571
Add tests for abortMultipartUpload, listobjectparts, listmultipartuploads
2015-05-14 17:07:30 -07:00
Harshavardhana
1bd94ec6ab
An attempt to implement ListMultipartUploads()
2015-05-14 16:22:47 -07:00
Harshavardhana
82b043005a
Avoid one last memory copy
2015-05-14 11:26:55 -07:00
Harshavardhana
5498c90788
Add a new validateContentTypeHandler{}, verify Accept header earlier
2015-05-13 12:22:15 -07:00
Harshavardhana
de56909d55
More scripts and code from Minio-io to minio
2015-05-11 16:43:38 -07:00
Harshavardhana
8d06504068
Change all minio-io path to minio
2015-05-11 16:23:10 -07:00
Harshavardhana
e39de4f603
Fix typo introduced in previous commit
2015-05-11 14:24:31 -07:00
Harshavardhana
f45635524f
Remove SignatureV2 support, bring in SignatureV4 header only validation for now
2015-05-11 13:38:40 -07:00
Harshavardhana
b48b7f51ef
Use convenience functions for url.Values over looping through keys, fix a bug in listObjectParts()
2015-05-10 11:32:41 -07:00
Harshavardhana
14b5828505
Handle partNumberMarker with listObjectParts now and other fixes
2015-05-09 19:39:00 -07:00
Frederick F. Kautz IV
cf559fa48e
Limiting upload id size
2015-05-09 18:28:50 -07:00
Frederick F. Kautz IV
61a9ea0f4b
Adding support for memory driver test
2015-05-09 18:15:27 -07:00
Frederick F. Kautz IV
058604aa21
Adding tests and fixes for multipart uploads uncovered from tests
2015-05-09 17:42:14 -07:00
Harshavardhana
8860aa0c8c
Implement AbortMultipart
2015-05-09 16:09:50 -07:00
Harshavardhana
18c7f620cc
Add listparts support
2015-05-09 13:55:55 -07:00
Harshavardhana
04e388d620
Handle two different styles of time format, s3cmd now compatible
2015-05-08 16:01:59 -07:00
Harshavardhana
01b474fca3
Move expiration code into its own and add ticker select case
2015-05-08 12:37:49 -07:00
Harshavardhana
23a545673d
Verify incoming md5sum for invidual parts in complete multipart request and some simplification of the code
...
Add two new functions for session cleanup
2015-05-08 03:59:05 -07:00
Harshavardhana
616241a2c1
Re-enable bucket deletion, this time with uploadID sitting there
2015-05-08 02:28:38 -07:00
Harshavardhana
2ea10c798b
Implement proper errors for Multipart
2015-05-08 02:02:51 -07:00
Harshavardhana
82c3656f79
Reply back CompleteMultipartUploadResult properly with final ETag computed
...
- Now s3 libraries and also objectstorage-go work properly
2015-05-07 23:01:20 -07:00
Frederick F. Kautz IV
ae5c99d7bb
Router should work with ?uploads unescaped now
2015-05-07 21:37:34 -07:00
Frederick F. Kautz IV
9dc7b82b39
Merge pull request #576 from fkautz/pr_out_adding_multipart_support
2015-05-07 20:00:20 -07:00
Frederick F. Kautz IV
f050d5e974
Adding multipart support
2015-05-07 19:55:30 -07:00
Frederick F. Kautz IV
ed71c28c37
Fixing panic in intelligent, attempted to delete nonexistent key
2015-05-07 19:12:49 -07:00
Frederick F. Kautz IV
d0c4334834
Adding feature flags
2015-05-07 13:56:54 -07:00
Harshavardhana
1df28f3060
Get rid of sub-domain code and some other formatting changes
2015-05-07 00:21:40 -07:00
Harshavardhana
6d3d92694f
Add Marker support for donut, also add a testing code
2015-05-05 22:18:24 -07:00
Harshavardhana
75028c2ad1
Add lexicographic Marker/NextMarker support for recursive listing of objects.
...
Also update times when an object is accessed logic
2015-05-05 18:05:34 -07:00
Frederick F. Kautz IV
24af2222c9
Adding xmlns
2015-05-05 12:54:34 -07:00
Harshavardhana
193a6606db
Fix a potential race between ExpireObjects() and Set() over items map
2015-05-04 22:36:47 -07:00
Harshavardhana
ed1259d6f0
Deprecate LRU use just map[string]interface{} and call it intelligent 'cache'
2015-05-04 22:01:56 -07:00
Harshavardhana
63edb1e9a0
Nullify list memory to nil as we remove the element
2015-05-04 17:22:08 -07:00
Harshavardhana
d63064b8af
Optimize memory usage in GetPartialObject() for memory driver
2015-05-04 15:26:56 -07:00
Harshavardhana
c8f31d97a8
Modify LRU further to add object expiration
2015-05-04 04:32:44 -07:00
Harshavardhana
670f997b07
Bring blockingWriter from client and use it here
2015-05-04 03:55:43 -07:00
Harshavardhana
d0df548eb5
Use new LRU inside memory driver
2015-05-04 03:55:43 -07:00
Harshavardhana
f7caef2d26
Change CreateObject() to take size argument from content-length
2015-05-04 03:55:43 -07:00
Harshavardhana
c342ce1588
New LRU based on GroupCache which keeps track of memory rather than entries
2015-05-04 03:55:19 -07:00
Frederick F. Kautz IV
3cf045fd1b
Deleting key from lastAccessedObjects on eviction
2015-05-01 22:01:07 -07:00
Harshavardhana
9ec8925f1c
Rename IsValidObject() to IsValidObjectName()
2015-05-01 20:31:26 -07:00
Frederick F. Kautz IV
33ecba8e49
Reimagining memory settings cli input
2015-05-01 18:12:58 -07:00
Frederick F. Kautz IV
25c268ac88
Attempt to fix memory lock on Get preventing write
2015-05-01 17:45:40 -07:00
Frederick F. Kautz IV
85f1b46492
Removing log message from test
2015-05-01 17:32:18 -07:00
Frederick F. Kautz IV
7cf86e6114
Fix docker support
2015-05-01 15:23:39 -07:00
Harshavardhana
2c7b30e5e1
Even mux vars don't help, handle it by looking at req.URL.Path
2015-05-01 12:41:17 -07:00
Frederick F. Kautz IV
b86b26e7e5
Exposing expiration for memory driver
2015-04-30 21:20:54 -07:00
Harshavardhana
c01fb92eca
Merge pull request #557 from harshavardhana/pr_out_make_builddate_universal
2015-04-30 21:19:04 -07:00
Harshavardhana
bbc32d6ac0
Disable domain based routing for now
2015-04-30 21:15:53 -07:00
Harshavardhana
f3078d047c
Get bucketName from mux.Vars instead of heuristic code, resolves subdomain issues
2015-04-30 21:11:32 -07:00
Frederick F. Kautz IV
e911cc332e
Merge pull request #554 from fkautz/pr_out_disable_eviction_on_0_expiration_time
2015-04-30 19:40:46 -07:00
Harshavardhana
8e3d48bf35
Use UTC() everywhere
...
ref - https://github.com/golang/go/issues/1988 , no standard function to set timeZone
2015-04-30 19:38:32 -07:00
Frederick F. Kautz IV
94e24f1a05
Disable eviction on 0 expiration time
2015-04-30 19:34:35 -07:00
Harshavardhana
72395798c8
Merge pull request #552 from harshavardhana/pr_out_minio_config_directory_should_be_0700_not_os_modedir
2015-04-30 17:40:15 -07:00
Harshavardhana
2de81c8a3c
Minio config directory should be 0700 not os.ModeDir
2015-04-30 15:42:49 -07:00
Frederick F. Kautz IV
8e2f7f9493
Wrapping functions that require gc so that references are released before running
2015-04-30 12:38:12 -07:00
Harshavardhana
d815e6adfd
Changes to CreateObject() now returns back md5 along with any error
...
- This change is necessary to avoid the racy calls to GetObjectMetadata()
- This change is also necessary since one has to reply back md5sum with
PUT object response header
2015-04-30 03:38:14 -07:00
Harshavardhana
fd2203b1b7
Some more improvements to connection limit
2015-04-30 01:23:44 -07:00
Harshavardhana
f8a16dd22b
Memory now evicts bucket if no more objects in memory struct
...
- To avoid race in expiration while accessing memory driver structs with
in two competing write locks. Use lru Len() instead to know exact length
to schedule for eviction.
- squash both bucket and object structs, instead use a separate map to keep
mutable info of lastAccessTime which can be independently used inside the
expiration routine.
2015-04-30 00:31:52 -07:00
Frederick F. Kautz IV
ed0a9a9e06
Removing entity too large errors from memory driver
2015-04-30 00:09:54 -07:00
Frederick F. Kautz IV
aff2b3b06a
Adding debug statements for testing
2015-04-29 21:13:32 -07:00
Frederick F. Kautz IV
82a16b8848
Make minio work with curl and browsers again
2015-04-29 20:17:26 -07:00
Frederick F. Kautz IV
3a1386165f
Fixing race condition
2015-04-29 18:54:59 -07:00
Harshavardhana
03bca9a001
Fix out of memory issue
2015-04-29 18:33:19 -07:00
Harshavardhana
25d4c0f6fa
A small logical change in messaging and logging
2015-04-29 17:30:54 -07:00
Frederick F. Kautz IV
f596c744ed
Adding error message on connection limit met
2015-04-29 17:21:23 -07:00
Harshavardhana
a89c4a57d3
Return resources as its available, not a new value
2015-04-29 16:47:14 -07:00
Harshavardhana
251c99176a
Merge pull request #537 from harshavardhana/pr_out_add_proper_content_length_for_error_and_success_responses
2015-04-29 15:32:12 -07:00
Frederick F. Kautz IV
0da04c6f17
Expirations are now based on last accessed instead of time created
2015-04-29 15:28:45 -07:00
Harshavardhana
92e4301414
Add proper content-length for error and success responses
...
- All compliance issues with S3 API for Put,Get,List (Bucket,Object) respectively
- Encodes and returns back proper HTTP headers
2015-04-29 15:28:28 -07:00
Frederick F. Kautz IV
5fc5f79ab7
Adding lru
2015-04-29 15:08:26 -07:00
Frederick F. Kautz IV
8fcb697844
Simplifying memory sleep logic
2015-04-29 13:53:09 -07:00
Frederick F. Kautz IV
5cc9418ca7
Adding expiration
2015-04-29 13:02:21 -07:00
Frederick F. Kautz IV
a608524ebf
api logging is now in its own package
2015-04-29 12:24:53 -07:00
Harshavardhana
d1d5f7a78d
Collate success response into writeSuccessResponse(), add docs
2015-04-29 10:52:04 -07:00
Harshavardhana
b18bb230da
maxObjectSize and minObjectSize limitation added at putObjectHandler()
...
Put() replies back with
- EntityTooLarge with > 5GB in single PUT operation
- EntityTooSmall with < 1B in single PUT operation
- IncompleteBody with ho Content-Length found in HTTP request header
2015-04-29 02:19:53 -07:00
Frederick F. Kautz IV
172cd18455
Adding initial logging framework to api_router
2015-04-28 20:37:08 -07:00
Frederick F. Kautz IV
88cca647ea
Adding json logging to stderr
2015-04-28 18:11:53 -07:00
Frederick F. Kautz IV
de091068e8
Removing quotas, will handle with iptables
2015-04-28 12:14:34 -07:00
Frederick F. Kautz IV
f2257a643c
Critical sections are now protected by lock
2015-04-27 19:59:58 -07:00
Harshavardhana
605edbc307
Add some logging with debug
2015-04-27 19:26:24 -07:00
Harshavardhana
7bdcf4bbf3
Merge pull request #515 from fkautz/pr_out_fix_multiple_response_writeheader_calls
...
Fix multiple response.WriteHeader calls
2015-04-27 19:21:47 -07:00
Frederick F. Kautz IV
db5870be28
Fix multiple response.WriteHeader calls
2015-04-27 18:30:21 -07:00
Frederick F. Kautz IV
fc2537eef9
Merge pull request #513 from fkautz/pr_out_adding_software_fallback
2015-04-27 14:41:41 -07:00
Frederick F. Kautz IV
289c859675
Adding software fallback
2015-04-27 13:55:40 -07:00
Harshavardhana
ade803a923
Verify if request date is 5minutes late, reject such a request as
...
it could be a possible replay attack.
This commit also fixes #505 , by returning MethodNotAllowed instead of NotImplemented
2015-04-27 03:55:10 -07:00
Harshavardhana
9232ce3b4e
Now client requests for ACL changes are honored through PutBucketACL API
2015-04-27 03:04:29 -07:00
Harshavardhana
c99d96dbc2
Now donut supports bucket ACL's, bringing in this support for dl.minio.io
2015-04-27 02:09:32 -07:00
Harshavardhana
8beca83090
Limit memory mode at 100 buckets and return error appropriately
...
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101 ]. Reason: [You have attempted to create more buckets than allowed.].
~~~
Response from server
2015-04-26 22:15:39 -07:00
Harshavardhana
2d96d5ad57
Enhance error responses for request limit and bring some code from api errors
...
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101 ]. Reason: [Reduce your request rate.].
~~~
Client sees proper errors now.
2015-04-26 22:10:59 -07:00
Frederick F. Kautz IV
8a57006b36
Merge pull request #504 from fkautz/pr_out_making_lock_much_less_aggressive_fixes_multiple_uploads_blocking
2015-04-26 20:11:50 -07:00
Frederick F. Kautz IV
a974e4d7c7
Making lock much less aggressive, fixes multiple uploads blocking
2015-04-26 20:08:37 -07:00
Frederick F. Kautz IV
258bf55e31
Bandwidth quota now supports 100-Continue
2015-04-26 20:07:05 -07:00
Frederick F. Kautz IV
9431a54a27
finish me
2015-04-26 19:25:48 -07:00
Frederick F. Kautz IV
202fc26719
Revert "Exporting WriteErrorResponse"
...
This reverts commit e05d46a0b5
.
2015-04-26 17:02:49 -07:00
Frederick F. Kautz IV
e05d46a0b5
Exporting WriteErrorResponse
2015-04-26 16:28:54 -07:00
Frederick F. Kautz IV
64f7d04cd1
Adding connection limits
2015-04-26 16:14:51 -07:00
Frederick F. Kautz IV
7867ee89fa
Bandwidth quotas now work on data returned from server
2015-04-26 15:49:55 -07:00
Frederick F. Kautz IV
e2403cbc56
Quotas are more accurate, occur on Read()
2015-04-26 15:07:57 -07:00
Frederick F. Kautz IV
bbb493080f
Switching back to BigEndian
2015-04-25 18:16:21 -07:00
Frederick F. Kautz IV
3a48f9fe4d
Adding quotas based upon type
2015-04-25 18:07:02 -07:00
Harshavardhana
b75f46aaa0
Use standard functions to achieve byteorder converstion
2015-04-25 13:18:15 -07:00
Frederick F. Kautz IV
e2475925ea
quota handlers now log against ip properly against a duration
2015-04-24 20:49:33 -07:00
Harshavardhana
b3a9d8b058
Merge pull request #483 from harshavardhana/pr_out_memory_now_handles_submitting_large_files_fixes_482
2015-04-24 15:25:38 -07:00
Harshavardhana
c5d8ca245d
Memory now handles submitting large files - fixes #482
2015-04-24 15:22:22 -07:00
Frederick F. Kautz IV
fda50d52c8
Add initial framework for quota
2015-04-24 00:16:29 -07:00
Frederick F. Kautz IV
5a607a6762
Moving factories and start minio method to server package
2015-04-23 16:13:50 -07:00
Frederick F. Kautz IV
5c515d3138
Simplifying server config handling
2015-04-23 12:44:22 -07:00
Harshavardhana
5ae8d34f76
Donut mode will now take multiple paths as argument, updated docs as well
2015-04-23 11:36:14 -07:00