maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1708 commits 18 branches 277 tags 146 MiB
Commit graph

654 commits

Author SHA1 Message Date
Harshavardhana 14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana 30fc14e703 Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana 8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana 7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana 12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel 
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
 2015-07-02 21:04:04 -07:00
Harshavardhana 5cfb05465e Add cache, donut tests separately - fix behavior differences 
Remove priority queue, implement it using a simpler channels
 2015-07-02 21:04:04 -07:00
Harshavardhana ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana 0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana 38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient 
Remove it and use map instead
 2015-07-02 21:04:04 -07:00
Harshavardhana 84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana 14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana 701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana 4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana 335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana 72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana 8f61d6b6be Move memory code out, add it as layer on top of existing donut code 
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
 2015-06-30 10:09:12 -07:00
Harshavardhana ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana 12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana 2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana 3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana 10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana 6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana 42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana 91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions 
There are still some more out there
 2015-06-27 20:43:25 -07:00
Harshavardhana 05f8654e3d Add Append() method to trove cache for appending data to an existing key 
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
 2015-06-27 20:25:24 -07:00
Harshavardhana 367772b988 Make sure to populate on disk data into memory upon first API requests 2015-06-27 18:25:21 -07:00
Harshavardhana 45a7eab804 An attempt to bring in memory layer into donut driver 2015-06-27 17:23:34 -07:00
Harshavardhana f3c25bcfc4 Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects() 2015-06-27 13:12:44 -07:00
Harshavardhana ae66ae42c4 Rename functions for their purpose 2015-06-27 12:39:11 -07:00
Harshavardhana 9a4680475f Object metadata was wrongly misconstrued to be mutable, handle it 2015-06-26 23:22:53 -07:00
Harshavardhana 3aa6d90c5e Avoid racy maps, read from disk on success return quickly. Many more optimizations 2015-06-26 19:49:37 -07:00
Harshavardhana 68974918ac Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions 2015-06-26 16:23:12 -07:00
Harshavardhana 767d3743ee Add new metadata definitions and use them 2015-06-26 13:34:09 -07:00
Harshavardhana 9958e34772 Import quick key value store from Minio Client for persistent state files, primarily for donut 2015-06-25 19:57:31 -07:00
Harshavardhana fb9adb5524 Donut cleanup another set 
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
 2015-06-25 18:54:34 -07:00
Harshavardhana eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana 45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana 82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana 03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana 57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00