maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1014 commits 18 branches 277 tags 146 MiB
Commit graph

343 commits

Author SHA1 Message Date
Frederick F. Kautz IV 679c6ace3a Fragment reader implemented 2015-03-07 18:04:20 -08:00
Harshavardhana df69038b04 Merge pull request #267 from harshavardhana/pr_out_add_md5_support_for_fsstorage_also_re_purpose_getobjectmetadata_for_listobjects 2015-03-07 17:16:00 -08:00
Harshavardhana 4751ed0e30 Add MD5 support for fsStorage, also re-purpose GetObjectMetadata for ListObjects 2015-03-07 17:15:00 -08:00
Frederick F. Kautz IV 1ec0cd6683 Merge pull request #266 from fkautz/pr_out_adding_read_header_for_erasure 2015-03-07 15:54:17 -08:00
Frederick F. Kautz IV ca1a4b616c Adding read header for erasure 2015-03-07 15:53:41 -08:00
Frederick F. Kautz IV 00381c887a Adding erasure package to wrap versioned erasure. Ensures we always write latest version. 2015-03-07 13:04:02 -08:00
Harshavardhana bf5a314144 clang lacks proper gas support, implement stubs for sha256,sha512 for darwin. 
REF: http://llvm.org/bugs/show_bug.cgi?id=18918
 2015-03-07 02:55:48 -08:00
Harshavardhana 1637138f86 If incoming request IP bucketfromHostname needs to be empty string 2015-03-07 02:53:29 -08:00
Harshavardhana a18bfa2fda Rename test-data to testdata, go will ignore it automatically 2015-03-07 01:32:35 -08:00
Harshavardhana d992bccd9a Move pkg/storage/erasure to pkg/encoding/erasure - and other cleanups 2015-03-07 00:16:48 -08:00
Frederick F. Kautz IV 856e0100c0 Erasure layer now writes using new technique 2015-03-06 16:37:44 -08:00
Harshavardhana 856b7680df Now able to use crc32c and sha1 on Darwin, with OSX specific code 2015-03-06 02:38:29 -08:00
Harshavardhana e5af8a3f5d Further fixes - 
- All test files have been renamed to their respective <package>_test name,
    this is done in accordance with
      - https://github.com/golang/go/wiki/CodeReviewComments#import-dot
        imports are largely used in testing, but to avoid namespace collision
        and circular dependencies

  - Never use _* in package names other than "_test" change fragment_v1 to expose
    fragment just like 'gopkg.in/check.v1'
 2015-03-06 02:04:21 -08:00
Harshavardhana c00d1461b9 Golint cleanup from top level 2015-03-05 23:37:37 -08:00
Harshavardhana 540723d7ae Golint cleanup pkg/api 2015-03-05 21:09:19 -08:00
Harshavardhana 76e601b26b Golint cleanup pkg/storage 2015-03-05 21:09:19 -08:00
Harshavardhana 256faddab5 Golint cleanup utils/crypto/sha*,cpu,md5 2015-03-05 21:09:19 -08:00
Harshavardhana 66e31445ff Golint cleanup pkg/utils/crypto/keys 2015-03-05 21:09:19 -08:00
Harshavardhana b33e2d2f9b Golint cleanup pkg/utils/config 2015-03-05 21:09:19 -08:00
Harshavardhana 861f13d5bd Golint cleanup pkg/utils/cpu 2015-03-05 21:09:19 -08:00
Harshavardhana c36450a83a Golint cleanup pkg/utils/policy 2015-03-05 21:09:19 -08:00
Harshavardhana 3e321b6631 Golint cleanup pkg/utils/split 2015-03-05 21:09:19 -08:00
Frederick F. Kautz IV 1186c47603 Fragment now implements new version 2015-03-05 20:56:02 -08:00
Frederick F. Kautz IV 665af44c70 Renaming donut packages to match layout 2015-03-05 19:44:01 -08:00
Frederick F. Kautz IV 6ec7894312 Removing innerxml 2015-03-04 14:24:56 -08:00
Frederick F. Kautz IV 6f7840677a Fixing xml tags 2015-03-04 11:02:37 -08:00
Harshavardhana f2e238199b Handle individual objects too 2015-03-04 01:54:35 -08:00
Harshavardhana e400e85639 Add proper path prefix to be delimited 2015-03-04 00:54:39 -08:00
Harshavardhana df135d7307 Merge pull request #249 from harshavardhana/pr_out_pathprefix_is_filtered_out_properly_and_delimited_as_well 2015-03-03 23:08:30 -08:00
Harshavardhana cc46475c86 Pathprefix is filtered out properly and delimited as well 2015-03-03 19:45:54 -08:00
Frederick F. Kautz IV 34c1e58332 Renaming Package to goPackage to fix golint error 2015-03-03 12:56:19 -08:00
Harshavardhana 0ebaac3594 Janitor duty v2 2015-03-03 02:43:24 -08:00
Harshavardhana 3a3c8645fc Update comments across the codebase 2015-03-03 02:39:38 -08:00
Harshavardhana 137584d658 Add comments 2015-03-03 01:25:45 -08:00
Harshavardhana c3ad0906e0 Add deadcode code which recursivley goes into all directories and verifies dangling variables. 2015-03-02 18:44:20 -08:00
Frederick F. Kautz IV 32f8c17174 Renaming AppendUstr to AppendUniqStr 2015-03-01 17:06:00 -08:00
Anand Babu (AB) Periasamy 81ede6c3fe Merge pull request #241 from abperiasamy/janitor 
On janitor duty.
 2015-03-01 16:43:04 -08:00
Frederick F. Kautz IV de6131cc7c Renaming AppendUInt to AppendUniqInt 2015-03-01 16:39:45 -08:00
Anand Babu (AB) Periasamy d9c9634138 removed custom Assert 2015-03-01 15:28:42 -08:00
Frederick F. Kautz IV 6f300c9ef0 Adding test at byte level 2015-03-01 15:13:09 -08:00
Anand Babu (AB) Periasamy 177121868b removed HomeDir 2015-03-01 15:13:08 -08:00
Frederick F. Kautz IV 0b08312ddd Specifying more exact types for data 2015-03-01 14:43:11 -08:00
Harshavardhana 9f43659135 Add tests, add NewHeader() for populating Header and ValidateHeader() 2015-03-01 13:19:06 -08:00
Anand Babu (AB) Periasamy 6515678e87 Merge pull request #236 from fkautz/pr_out_adding_data_writer 
Adding data writer
 2015-03-01 11:36:35 -08:00
Frederick F. Kautz IV 7414dc9b75 Adding data writer 2015-03-01 11:32:30 -08:00
Harshavardhana eb9a4a83ac Move AppendUstr(), AppendUint() into helpers 2015-03-01 11:19:03 -08:00
Frederick F. Kautz IV be12a6b3d9 Changing K and M in data.go to match new datatype in erasure package 2015-03-01 01:28:33 -08:00
Harshavardhana 33601ff567 Merge pull request #233 from harshavardhana/pr_out_make_k_m_to_be_uint8_and_technique_becomes_its_own_type 2015-03-01 01:26:42 -08:00
Harshavardhana b77c62e25d Make K,M to be uint8 and Technique becomes its own type 2015-03-01 01:25:28 -08:00
Frederick F. Kautz IV bb0441917c Making changes to data based on discussion 2015-03-01 01:16:22 -08:00
Harshavardhana 2f04a1299e Avoid allocating 1gig memory, avoid such mistakes 2015-03-01 00:43:12 -08:00
Frederick F. Kautz IV 2916b1278e Adding data.go, does nothing for now. 2015-03-01 00:27:52 -08:00
Frederick F. Kautz IV b4ef522a9b Donut frame is now its own package 2015-03-01 00:02:14 -08:00
Frederick F. Kautz IV 6c2428833a Renaming Write to WriteFrame 2015-02-28 23:52:56 -08:00
Harshavardhana 6b397844b9 Add donut benchmark for 64MB, 128MB, 256MB, 512MB 
Current average values

~~~
> minio/pkg/storage/donut/v1 *> go test -bench .
OK: 2 passed
PASS
BenchmarkDonut64M              5         266249634 ns/op         252.05 MB/s
BenchmarkDonut128M             3         436507394 ns/op         307.48 MB/s
BenchmarkDonut256M             2         836116359 ns/op         321.05 MB/s
BenchmarkDonut512M             1        1676738951 ns/op         320.19 MB/s
ok      github.com/minio-io/minio/pkg/storage/donut/v1  8.430s
> minio/pkg/storage/donut/v1 *>
~~~
 2015-02-28 17:05:46 -08:00
Harshavardhana 65f22083ca donut_gen works with new Donut Frame 2015-02-28 16:30:18 -08:00
Frederick F. Kautz IV 4d98c355c9 Merge pull request #224 from abperiasamy/units 2015-02-28 16:12:10 -08:00
Frederick F. Kautz IV ddc7cf835e Donut now tests amount written should match expected amount to write 2015-02-28 16:09:52 -08:00
Frederick F. Kautz IV d3d1205eb1 Updating comment describing data type 2015-02-28 15:59:52 -08:00
Frederick F. Kautz IV 58a04ee831 New donut frame implemented 2015-02-28 15:54:13 -08:00
Anand Babu (AB) Periasamy 317a1141c0 removed unitconv 2015-02-28 15:53:44 -08:00
Frederick F. Kautz IV 90cd8b7b47 Adding streaming sha512 2015-02-28 15:51:33 -08:00
Harshavardhana 56bd413a16 Merge pull request #219 from harshavardhana/pr_out_add_1mb_block_benchmarks 2015-02-28 15:25:24 -08:00
Harshavardhana 7feca3f0d1 Add 1MB block benchmarks 2015-02-28 15:00:24 -08:00
Harshavardhana 0c2d58bc6d Implement proper delimiter and prefix handling 
With this change Minio server now responds with, delimited
'object names' in conjunction with prefix filtering

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>sample.html</Key>
    <LastModified>2011-02-26T01:56:20.000Z</LastModified>
    <ETag>example-bucket#sample.html</ETag>
    <Size>142863</Size>
    <Owner>
      <ID>minio</ID>
      <DisplayName>minio</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <CommonPrefixes>
    <Prefix>photos/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix>photos/2006/</Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>

  <CommonPrefixes>
    <Prefix>photos/2006/feb/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>photos/2006/jan/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~
 2015-02-28 14:47:19 -08:00
Harshavardhana d5e5a3b3e6 Use validString 2015-02-27 19:49:18 -08:00
Harshavardhana 6ebb48b4ea Add object name validation 2015-02-27 19:42:04 -08:00
Harshavardhana 9acb1f728c Sort list of objects 2015-02-27 17:52:55 -08:00
Harshavardhana 53669a0854 Implement delimiter, path prefix 2015-02-27 16:23:05 -08:00
Harshavardhana 3f8b3462e6 Remove donut_gen binary and update crc32c function 2015-02-27 14:41:53 -08:00
Harshavardhana 9aea7fd97b Merge pull request #213 from harshavardhana/pr_out_crc32c_re_implementation_with_convenience_functions 2015-02-27 14:35:11 -08:00
Harshavardhana 8ce9b84b69 Crc32c re-implementation with Convenience functions 
- Sum32([]byte) --> uint32
   - Sum(io.Reader) --> uint32, error
 2015-02-27 13:39:17 -08:00
Frederick F. Kautz IV 2e37791681 Merge pull request #212 from fkautz/pr_out_adding_comparison_for_footer_s_inim 2015-02-27 12:40:10 -08:00
Frederick F. Kautz IV 1f7a3c13a5 Adding comparison for footer's INIM 2015-02-27 12:39:52 -08:00
Frederick F. Kautz IV 2ca62f13a5 Moving donut_gen to its own dir and renaming it as donut_gen_v1 2015-02-27 12:39:14 -08:00
Frederick F. Kautz IV be768d01a2 Seek to beginning for tempbuffer 2015-02-27 12:18:27 -08:00
Frederick F. Kautz IV 5c52da6ebe Adding crc32c to write 2015-02-26 23:04:07 -08:00
Harshavardhana a9aaff0ee0 Remove HTTP server timeouts write and read. 
Implement it later with a proper timer thread. Large file
transfers from 'mc' fail with i/o timeout without this change.
 2015-02-24 13:55:24 -08:00
Frederick F. Kautz IV ef442289da Making donut use io.ReadWriteSeek instead of io.Writer 2015-02-23 20:27:25 -08:00
Harshavardhana 5e1e5ad786 More updates on documentation 2015-02-23 17:44:55 -08:00
Harshavardhana b7868d23ac Update minioapi documentation 2015-02-23 16:46:48 -08:00
Harshavardhana f4b5519451 Update erasure documentation 2015-02-23 13:39:16 -08:00
Anand Babu (AB) Periasamy de41e465c8 move httpserver under server 2015-02-23 11:03:01 -08:00
Harshavardhana 51e80eaa6d Add domain and subdomain support for MinioAPI 
This change brings in domain and subdomain support

   - ./minio --domain "yourminiodomain.com"

This change brings in a much needed feature by keeping
bucketnames as part of your 'DNS' name.

All your existing applications can be migrated off from s3 to
Minio without little to no modifications.

NOTE: Setting up DNS for your `buckets` is out of scope of this feature
 2015-02-23 02:25:01 -08:00
Harshavardhana a92c6cb2b2 Purge tiedot wrapper 2015-02-22 20:02:09 -08:00
Harshavardhana 49a714d1a4 Add license header for Minio modifications, improvements for sha1,sha256,sha512 implementations 
Also bring in SSE3 optimized public domain implementation of SHA1 from Intel ``FIPS PUB 180-1``
 2015-02-21 15:50:47 -08:00
Harshavardhana 12cff1be58 Implement sha1 as intel optimized set 2015-02-21 11:25:09 -08:00
Anis Elleuch 1d4d7ff274 Add missing asm instruction to correct sha256 calculation in avx mode 2015-02-21 19:34:36 +01:00
Anis Elleuch e89271b8f5 Use Intel optimized code (ssse3, avx, avx2) to calculate sha256 2015-02-21 13:11:15 +01:00
Harshavardhana 3151f9b6c3 Add proper entries for GobHeader, Data copy. 
This commit also adds 'donut_gen.go' for generating sample donut file
 2015-02-21 01:44:27 -08:00
Frederick F. Kautz IV 8129d8fd0b Writing data out with error handling 2015-02-20 23:24:01 -08:00
Frederick F. Kautz IV 98ddbd840e Moving donut code to v1 in donut package 2015-02-20 23:07:43 -08:00
Frederick F. Kautz IV 9833647fb7 Fixing magic number for INIM 2015-02-20 22:42:58 -08:00
Anand Babu (AB) Periasamy ec36a598c9 Magic as numbers are faster to encode and compare 2015-02-20 18:20:21 -08:00
Harshavardhana 741e14f749 Remove go cover and remove unncessary Makefiles 2015-02-20 17:44:26 -08:00
Harshavardhana 19a4998fcb SHA512 Implemention with Intel assembly code 2015-02-20 16:32:19 -08:00
Harshavardhana c4f0aad9b5 Add donut tests, just for verification 2015-02-18 19:46:14 -08:00
Frederick F. Kautz IV f7f34d08e8 Donut structure refined 2015-02-18 17:18:33 -08:00
Frederick F. Kautz IV ab80c0ce12 Merge pull request #180 from abperiasamy/donut 2015-02-18 15:57:16 -08:00
Anand Babu (AB) Periasamy 9e304b53de donut spec v1 prototype 2015-02-18 15:52:12 -08:00
Harshavardhana fab954f13f Run govet and fix 2015-02-18 15:33:55 -08:00
Harshavardhana f7a06a5d1e Some more cleanup 2015-02-18 12:15:33 -08:00
Frederick F. Kautz IV 05ac09bf69 Adding header version 2015-02-18 10:21:02 -08:00
Harshavardhana a3e5121f7b Make channels more aware of errors and shutdown on error - some cleanup 2015-02-18 00:24:19 -08:00
Harshavardhana cf6d03b907 Expose policy code, for api router usage 2015-02-17 19:06:01 -08:00
Harshavardhana 5235badec9 Merge pull request #169 from harshavardhana/pr_out_add_policy_validation_code_more_rigorous 2015-02-17 18:10:00 -08:00
Frederick F. Kautz IV fe9a5c14a0 Adding scaffolding for writing donut 2015-02-17 17:47:10 -08:00
Harshavardhana 980d229272 Add policy validation code more rigorous 2015-02-17 13:23:16 -08:00
Harshavardhana f04beab7c2 Skip policy files 2015-02-16 18:37:09 -08:00
Harshavardhana eeae64935e Implement bucket policy handler and with galore of cleanup 2015-02-16 18:20:40 -08:00
Harshavardhana ba3e2d41f7 Add license and fix development scripts 2015-02-15 01:59:48 -08:00
Harshavardhana 966cdd4f05 Reply in xml from validate and ignore handlers 2015-02-11 22:00:45 -08:00
Harshavardhana ac4f07906c Implement S3 Style ErrorCodes and Response 
This patchset also brings in lot of cleanup in terms of minioapi codebase
 2015-02-11 21:27:28 -08:00
Harshavardhana 680848bdcb Add license header 2015-02-08 03:00:32 -08:00
Harshavardhana 8087ca2450 Verify both 'x-amz-date' and 'date' header before discarding request 2015-02-08 02:37:19 -08:00
Harshavardhana 92f83982db Changes to tests pending from #142 2015-02-06 20:47:21 -08:00
Harshavardhana ecb70a4c19 Avoid trailing '/' for buckets and also re-directs - in accordance with aws clients 2015-02-06 20:42:09 -08:00
Harshavardhana 81fc11ee5d Implement authorization support 2015-02-06 02:08:52 -08:00
Harshavardhana 0e8b16a55d Ignore EOF in ReadConfig() 2015-02-05 16:10:49 -08:00
Harshavardhana 72d0999dcb Reply back AccessKey, Secretkey through json 2015-02-05 15:59:44 -08:00
Harshavardhana 95f214c99b Append to Users list 2015-02-05 15:40:51 -08:00
Harshavardhana b3b73ff4cb Merge pull request #136 from harshavardhana/pr_out_store_credentials_through_webui 2015-02-05 14:20:06 -08:00
Harshavardhana 4c90017b02 Store credentials through webui 2015-02-05 14:19:48 -08:00
Frederick F. Kautz IV 620435cea6 Content type is now passed from minioapi to storage 2015-02-05 14:05:43 -08:00
Frederick F. Kautz IV 288bc3fcc7 Adding content type to backend storage drivers 2015-02-04 17:32:40 -08:00
Frederick F. Kautz IV 816310825e Removing XMLName from json request 2015-02-04 14:21:47 -08:00
Frederick F. Kautz IV 7152712f34 Accessing dir as object should return ObjectNotFound 2015-02-03 16:45:00 -08:00
Harshavardhana 582ce3df36 Make sure config testing is done in tempdir 2015-02-03 00:25:33 -08:00
Harshavardhana b1db70c807 Add json config reader/writer 2015-02-03 00:12:46 -08:00
Harshavardhana aff4c5826b Use helper HomeDir() 2015-02-02 21:44:17 -08:00
Harshavardhana dc47ca1431 Fix build issue 2015-02-01 17:21:17 -08:00
Frederick F. Kautz IV a8eef13835 Minioapi TestPutObject properly creates bucket at start of test 2015-02-01 14:46:45 -08:00
Frederick F. Kautz IV a092b44cf1 Adding test to ensure list objects returns an error for non existant buckets to match previous bug fix and fix for inmemory 2015-02-01 10:48:31 -08:00
Harshavardhana 5d197dd063 Reply back errors properly 2015-02-01 03:10:28 -08:00
Harshavardhana b635501fb7 Add StrictSlash for path re-direction 2015-02-01 02:49:09 -08:00
Harshavardhana e1b3836c60 Service a new WebUI server at port 8081. 
It is provided to avoid the router clutter on the Object API
end. This is just a first cut sample we are going to build on
top of this the new management console UI.

General interest comes from subsequently writing a small UI to
generate `AccessKey` and `SecretKey` to build authorization
layer.
 2015-02-01 02:44:53 -08:00
Frederick F. Kautz IV 54026dbc78 Setting order when retrieving buckets 2015-02-01 00:16:25 -08:00
Harshavardhana 26aac9a439 Add bucket resources to ignore handler 2015-01-31 22:45:52 -08:00
Frederick F. Kautz IV 94e124470c Converting unimplemented methods list into an http handler 2015-01-31 19:50:32 -08:00
Harshavardhana 46751f0b28 Add database query function, and some re-factoring 2015-01-31 15:51:18 -08:00
Frederick F. Kautz IV 0cd54031a7 Adding list api test 2015-01-30 17:36:53 -08:00
Harshavardhana 8a4128d304 Force clients to have certs 2015-01-30 16:51:33 -08:00
Frederick F. Kautz IV 84c5df8441 Using generic make test ./... and make build ./..., disabled signers code for now to fix build. 2015-01-30 10:58:47 -08:00
Frederick F. Kautz IV 7ca87e06aa Adding list bucket tests and fixing inmemory behavior. 2015-01-29 11:23:14 -08:00
Harshavardhana 82baccb8bf Prefix if empty skip it before IsValidBucket 2015-01-29 01:07:37 -08:00
Frederick F. Kautz IV c6d537e66e Reducing code duplication relating to content type 2015-01-28 20:47:23 -08:00
Frederick F. Kautz IV 96a5fc0ee7 Updating godep 2015-01-28 20:26:52 -08:00
Harshavardhana 57d764778d Merge pull request #88 from harshavardhana/pr_out_add_x509_tests_and_should_be_part_of_the_build 2015-01-28 17:13:10 -08:00
Harshavardhana ac99968796 Add x509 tests and should be part of the build 2015-01-28 17:12:59 -08:00
Frederick F. Kautz IV b170f3f5fc Reordering minioapi functions 2015-01-28 16:51:44 -08:00