maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1839 commits 18 branches 277 tags 146 MiB
Commit graph

720 commits

Author SHA1 Message Date
Harshavardhana
bbb89b5776 Add multi-thread protection and also allow atomic file creates, rename upon Close() 2015-07-06 18:54:32 -07:00
Harshavardhana
c2c7bdf0cd Cleanup nimble http 2015-07-06 18:51:20 -07:00
Harshavardhana
b029d0a5f0 Avoid config reload all the time, reload is manually triggerred from outside 2015-07-06 17:26:35 -07:00
Harshavardhana
8b94c53345 Fix issues with multipart upload 2015-07-06 16:22:27 -07:00
Harshavardhana
474954022e Add modified grace library from facebookgo, rename it as nimble 2015-07-06 15:40:12 -07:00
Harshavardhana
1d64e4b6c1 Add Donut rpc service for sending changes to configuration files 2015-07-06 11:10:06 -07:00
Harshavardhana
10b082144e Add updateConfig code to load config changes if possible for every function 2015-07-05 22:46:42 -07:00
Harshavardhana
ba0a5ed416 Add sighup, sigusr2 into trapping code, to trap signals for reloading configuration. 
Need to still figure out a way of graceful restarts - gave facebookgo/httpdown a shot,
but it is not suitable.
 2015-07-05 21:40:53 -07:00
Harshavardhana
4a27ab0e58 Fix another deadlock inside CreateObjectPart() code, premature return without Unlocking() 
Also this patch changes the cache key element to be interface{} type not string.
 2015-07-05 20:26:32 -07:00
Harshavardhana
75788c7a1d Fix go installation check on amazon instance 2015-07-05 18:12:58 -07:00
Harshavardhana
7f0c14f2b7 Add basic controller code, initiating json rpc connection getting list of disks and memstats for now. 2015-07-05 17:17:41 -07:00
Harshavardhana
a3ccb9d405 Rename stuttered service names and make them appropriate 2015-07-05 10:19:23 -07:00
Harshavardhana
e66a84242a Add disk detection for Linux, add new RPC service GetDiskInfoService(), remove dummy HelloService() 2015-07-05 02:08:33 -07:00
Harshavardhana
bab4a47525 Move to container/list datastructure from map[string][]byte 2015-07-04 17:08:23 -07:00
Harshavardhana
aa67a19e99 Implement new CPU detection using cpuid, cpuidex plan9 instructions from klauspost/cpuid project, remove C code 2015-07-04 14:28:16 -07:00
Harshavardhana
0a827305ad Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache 
We should use it internally everywhere.
 2015-07-03 21:09:57 -07:00
Harshavardhana
14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana
30fc14e703 Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana
8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana
7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana
12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel 
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
 2015-07-02 21:04:04 -07:00
Harshavardhana
5cfb05465e Add cache, donut tests separately - fix behavior differences 
Remove priority queue, implement it using a simpler channels
 2015-07-02 21:04:04 -07:00
Harshavardhana
ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana
bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana
0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana
38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient 
Remove it and use map instead
 2015-07-02 21:04:04 -07:00
Harshavardhana
84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana
14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana
eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana
701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana
4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana
335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana
dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana
72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana
c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV
cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana
8f61d6b6be Move memory code out, add it as layer on top of existing donut code 
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
 2015-06-30 10:09:12 -07:00
Harshavardhana
ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana
12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana
2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana
f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana
3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana
d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana
10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana
6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana
d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana
42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana
91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana
ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions 
There are still some more out there
 2015-06-27 20:43:25 -07:00
Harshavardhana
05f8654e3d Add Append() method to trove cache for appending data to an existing key 
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
 2015-06-27 20:25:24 -07:00
Harshavardhana
367772b988 Make sure to populate on disk data into memory upon first API requests 2015-06-27 18:25:21 -07:00
Harshavardhana
45a7eab804 An attempt to bring in memory layer into donut driver 2015-06-27 17:23:34 -07:00
Harshavardhana
f3c25bcfc4 Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects() 2015-06-27 13:12:44 -07:00
Harshavardhana
ae66ae42c4 Rename functions for their purpose 2015-06-27 12:39:11 -07:00
Harshavardhana
9a4680475f Object metadata was wrongly misconstrued to be mutable, handle it 2015-06-26 23:22:53 -07:00
Harshavardhana
3aa6d90c5e Avoid racy maps, read from disk on success return quickly. Many more optimizations 2015-06-26 19:49:37 -07:00
Harshavardhana
68974918ac Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions 2015-06-26 16:23:12 -07:00
Harshavardhana
767d3743ee Add new metadata definitions and use them 2015-06-26 13:34:09 -07:00
Harshavardhana
9958e34772 Import quick key value store from Minio Client for persistent state files, primarily for donut 2015-06-25 19:57:31 -07:00
Harshavardhana
fb9adb5524 Donut cleanup another set 
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
 2015-06-25 18:54:34 -07:00
Harshavardhana
eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana
45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana
82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana
03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana
57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana
a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00
Harshavardhana
841ff01412 Move disk into its own package, remove bloated code 2015-06-24 20:13:47 -07:00
Harshavardhana
1682c748ac Remove unnecessary interfaces from donut, cleanup 2015-06-24 19:43:38 -07:00
Harshavardhana
3498872467 Add sha256 and sha512 windows compatibility layer 2015-06-24 14:39:41 -07:00
Harshavardhana
77d35b87d4 Fix a bug on windows regarding blockSSE3 calculation 2015-06-24 14:24:33 -07:00
Harshavardhana
f1410731db Add windows code for sha1 and crc32c 2015-06-24 14:16:32 -07:00
Harshavardhana
ba2d3dea74 Fix wrong TMPFS listing in donut filesystem map 2015-06-23 23:16:06 -07:00
Harshavardhana
e818bc7187 Trim iodine path properly, so that now errors have github.com/minio/minio prefixed 2015-06-23 13:36:54 -07:00
Harshavardhana
2fd52ca284 Add missing strongly typed errors for Donut 2015-06-23 11:54:44 -07:00
Harshavardhana
7a060110ff Fix BUILDDEPS paths for golang installation 2015-06-22 12:07:23 -07:00
Harshavardhana
3bf64f5669 Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659 2015-06-20 11:03:17 -07:00
Harshavardhana
641f07cecf Use filepath everywhere instead of path.{} functions for portability - fixes #656 2015-06-18 16:02:45 -07:00
Harshavardhana
573a6134b2 Remove redundant !ok for map 2015-06-17 22:36:46 -07:00
Harshavardhana
e9a3fd677a Hold lock on GetGlobalStateKey() - fixes #652 2015-06-17 20:37:49 -07:00
Harshavardhana
b793f53d48 Minior change filter() to filterObjects() 2015-06-17 20:35:44 -07:00
Harshavardhana
d4e71beb54 Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects 2015-06-13 11:28:04 -07:00
Harshavardhana
636788997b Minor optimization avoiding one unnecessary copy 2015-06-11 14:23:05 -07:00
Harshavardhana
6d44274db6 Make sure to reply back ETag with quotes for s3 compliance 2015-06-09 01:13:02 -07:00
Harshavardhana
a947767339 Remove unnecessary tlsHelpers, SSL works with domain certificate 2015-06-08 20:54:41 -07:00
Harshavardhana
c6ecd94420 Enable TLS and use it 2015-06-08 20:10:59 -07:00
Harshavardhana
64b014369c Return back entity too large for createObject requests bigger than cache size 2015-06-08 17:10:07 -07:00
Harshavardhana
ab1976bc14 Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server 2015-06-08 15:15:37 -07:00
Harshavardhana
2976f109df More multipart fixes, filter out parts and multipart metadata from listObjects 2015-06-08 14:08:36 -07:00
Harshavardhana
e08d59be00 Make sure to delete uploadid's from active session file properly 2015-06-08 12:25:51 -07:00
Harshavardhana
0d3710cb9b Store in json file instead of gob for fs metadata 2015-06-08 12:08:01 -07:00
Harshavardhana
8efc842b59 Add delete handlers and reply back as 'NotImplemented' instead of 404 2015-06-08 11:06:06 -07:00
Harshavardhana
8a320c4447 Enhance logging handler to print in humanize form for ContentLength 2015-06-08 02:14:09 -07:00
Harshavardhana
cefd740002 Put back the older conn_limit 2015-06-06 18:11:38 -07:00
Harshavardhana
371651bde6 Add rate limiter instead of connection limit for now 2015-06-06 18:04:55 -07:00
Harshavardhana
21916dc0a1 Fix a deadlock in trove when evicting objects when we reach the maxSize used 2015-06-06 16:02:52 -07:00
Harshavardhana
8b43655d9f Make connection limit configurable 2015-06-06 14:31:50 -07:00
Harshavardhana
a344f5b34d Misc changes before capturing proper video 
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
 2015-06-03 18:27:15 -07:00
Harshavardhana
097f70a3c5 Enable filesystem for API tests, this patch also implements ListMultipartUploads() 2015-06-01 20:57:13 -07:00
Harshavardhana
bfb9209689 Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service 2015-06-01 16:43:43 -07:00
Harshavardhana
d86ba5f336 Implement multipart support 
- implemented NewMultipartUpload()
 - implemented CreateObjectPart()
 - implemented ListObjectParts()
 - implemented AbortMultipartUpload()
 - implemented CompleteMultipartUpload()
 2015-06-01 16:25:11 -07:00