Harshavardhana
bce93c1b3a
Integrate cache with donut, add tests
2015-07-02 21:04:04 -07:00
Harshavardhana
0533abf6a8
Make priority queue lambda function return error over a channel
2015-07-02 21:04:04 -07:00
Harshavardhana
38a6ce36e5
Remove slow AppendUniq code, rolling through over a slice is in-efficient
...
Remove it and use map instead
2015-07-02 21:04:04 -07:00
Harshavardhana
14ec42d646
Add initial implementation of priority queue, uses container/heap
2015-07-02 21:04:04 -07:00
Harshavardhana
335c7827eb
More donut, cache, api cleanup
2015-07-02 21:04:04 -07:00
Harshavardhana
dc0df3dc0e
Breakaway from driver model, move cache into donut
2015-07-02 21:04:03 -07:00
Harshavardhana
8f61d6b6be
Move memory code out, add it as layer on top of existing donut code
...
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().
drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.
API <----> cache <----> donut <----> donut(format)
2015-06-30 10:09:12 -07:00
Harshavardhana
ab6e16bb41
Add dummy driver for community to submit new drivers
2015-06-29 16:43:50 -07:00
Harshavardhana
12de98fb62
Rename memory driver as cache
2015-06-29 16:43:50 -07:00
Harshavardhana
2571342451
Filesystem goes the high road *again*
2015-06-29 16:43:42 -07:00
Harshavardhana
f74d6138da
IsValidBucket() is sufficient we don't need to verify for "."
2015-06-29 15:15:54 -07:00
Harshavardhana
3109909355
Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject()
2015-06-29 12:28:50 -07:00
Harshavardhana
d07d0c670a
Return back proper errors in writeObjectData(), rename few functions
2015-06-29 11:46:35 -07:00
Harshavardhana
10c807f233
Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver
2015-06-29 11:15:46 -07:00
Harshavardhana
6921328b93
Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects()
2015-06-29 11:14:58 -07:00
Harshavardhana
42c0287943
Use errorChannels only for services not for drivers, reduce them to use simple functions
2015-06-28 23:59:47 -07:00
Harshavardhana
91e5f648cb
Move to Set() not Append() due to large memory reference copy
2015-06-28 10:13:12 -07:00
Harshavardhana
ac4d8fe478
Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions
...
There are still some more out there
2015-06-27 20:43:25 -07:00
Harshavardhana
05f8654e3d
Add Append() method to trove cache for appending data to an existing key
...
This largely avoids a large buffer copy which would accumulate inside proxyReader{}
This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
2015-06-27 20:25:24 -07:00
Harshavardhana
367772b988
Make sure to populate on disk data into memory upon first API requests
2015-06-27 18:25:21 -07:00
Harshavardhana
45a7eab804
An attempt to bring in memory layer into donut driver
2015-06-27 17:23:34 -07:00
Harshavardhana
f3c25bcfc4
Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects()
2015-06-27 13:12:44 -07:00
Harshavardhana
ae66ae42c4
Rename functions for their purpose
2015-06-27 12:39:11 -07:00
Harshavardhana
9a4680475f
Object metadata was wrongly misconstrued to be mutable, handle it
2015-06-26 23:22:53 -07:00
Harshavardhana
3aa6d90c5e
Avoid racy maps, read from disk on success return quickly. Many more optimizations
2015-06-26 19:49:37 -07:00
Harshavardhana
68974918ac
Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions
2015-06-26 16:23:12 -07:00
Harshavardhana
767d3743ee
Add new metadata definitions and use them
2015-06-26 13:34:09 -07:00
Harshavardhana
fb9adb5524
Donut cleanup another set
...
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
2015-06-25 18:54:34 -07:00
Harshavardhana
eec66f195a
Take all the ListObjects into bucket handlers
...
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
2015-06-25 18:04:29 -07:00
Harshavardhana
45e9d25931
Remove more bloated code - simplify
2015-06-25 13:02:08 -07:00
Harshavardhana
82dcbf262d
Add simple locking for donut API for now - fixes #671
2015-06-25 11:29:11 -07:00
Harshavardhana
03b4d3b308
Go vet fixes for donut
2015-06-24 21:07:03 -07:00
Harshavardhana
57a2b53178
Removing further bloated code simplifying
2015-06-24 21:03:39 -07:00
Harshavardhana
a2c205ff2e
Use external package disk for donut.
2015-06-24 21:03:02 -07:00
Harshavardhana
841ff01412
Move disk into its own package, remove bloated code
2015-06-24 20:13:47 -07:00
Harshavardhana
1682c748ac
Remove unnecessary interfaces from donut, cleanup
2015-06-24 19:43:38 -07:00
Harshavardhana
ba2d3dea74
Fix wrong TMPFS listing in donut filesystem map
2015-06-23 23:16:06 -07:00
Harshavardhana
2fd52ca284
Add missing strongly typed errors for Donut
2015-06-23 11:54:44 -07:00
Harshavardhana
3bf64f5669
Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659
2015-06-20 11:03:17 -07:00
Harshavardhana
641f07cecf
Use filepath everywhere instead of path.{} functions for portability - fixes #656
2015-06-18 16:02:45 -07:00
Harshavardhana
b793f53d48
Minior change filter() to filterObjects()
2015-06-17 20:35:44 -07:00
Harshavardhana
d4e71beb54
Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects
2015-06-13 11:28:04 -07:00
Harshavardhana
636788997b
Minor optimization avoiding one unnecessary copy
2015-06-11 14:23:05 -07:00
Harshavardhana
64b014369c
Return back entity too large for createObject requests bigger than cache size
2015-06-08 17:10:07 -07:00
Harshavardhana
ab1976bc14
Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server
2015-06-08 15:15:37 -07:00
Harshavardhana
2976f109df
More multipart fixes, filter out parts and multipart metadata from listObjects
2015-06-08 14:08:36 -07:00
Harshavardhana
e08d59be00
Make sure to delete uploadid's from active session file properly
2015-06-08 12:25:51 -07:00
Harshavardhana
0d3710cb9b
Store in json file instead of gob for fs metadata
2015-06-08 12:08:01 -07:00
Harshavardhana
21916dc0a1
Fix a deadlock in trove when evicting objects when we reach the maxSize used
2015-06-06 16:02:52 -07:00
Harshavardhana
097f70a3c5
Enable filesystem for API tests, this patch also implements ListMultipartUploads()
2015-06-01 20:57:13 -07:00
