Commit graph

60 commits

Author SHA1 Message Date
Harshavardhana
556a51120c Deprecate ListLocks and ClearLocks (#6233)
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.

Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
2018-08-02 23:09:42 +05:30
Harshavardhana
36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
ebozduman
b0b0fb4c8d Validate Minio config.json file on the client side (#6067) 2018-07-03 11:07:46 -07:00
Harshavardhana
e5e522fc61
docs: fix all Chinese doc links for the new docs site (#6097)
Additionally fix typos, default to US locale words
2018-06-28 16:02:02 -07:00
Harshavardhana
d9d13c898c docs: Fix admin heal API description (#6073) 2018-06-26 10:53:14 -07:00
Praveen raj Mani
c0cfe21c00 Ignore region in the case of admin API (#5919)
Admin API is not an S3 API and hence it is not required
to honor server region while validating admin API calls.

Fixes #2411
2018-06-07 10:37:31 -07:00
Andreas Auernhammer
9fb94e6aa8 ensure authenticated request bodies for Admin-API (#5984)
This commit adds a check to the server's admin-API such that it only
accepts Admin-API requests with authenticated bodies. Further this
commit updates the `madmin` package to always add the
`X-Amz-Content-Sha256` header.

This change improves the Admin-API security since the server does not
accept unauthenticated request bodies anymore.

After this commit `mc` must be updated to the new `madmin` api because
requests over TLS connections will fail.
2018-05-30 14:49:03 -07:00
Harshavardhana
000e360196 Deprecate showing drive capacity and total free (#5976)
This addresses a situation that we shouldn't be
displaying Total/Free anymore, instead we should simply
show the total usage.
2018-05-23 17:30:25 -07:00
Harshavardhana
e6ec645035 Implement support for calculating disk usage per tenant (#5969)
Fixes #5961
2018-05-23 15:41:29 +05:30
Acid Chicken (硫酸鶏)
483fe4bed5 Fix typo (#5960) 2018-05-22 08:09:30 +05:30
Harshavardhana
e09d97abaf Fix docs in admin API (#5559) 2018-02-21 12:00:46 -08:00
Harshavardhana
fb96779a8a Add large bucket support for erasure coded backend (#5160)
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.

This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.

Some design details and restrictions:

- Objects are distributed using consistent ordering
  to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
  properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
  requirement, you can start with multiple
  such sets statically.
- Static sets set of disks and cannot be
  changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
  changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
  slower since List happens on all servers,
  and is merged at this sets layer.

Fixes #5465
Fixes #5464
Fixes #5461
Fixes #5460
Fixes #5459
Fixes #5458
Fixes #5460
Fixes #5488
Fixes #5489
Fixes #5497
Fixes #5496
2018-02-15 17:45:57 -08:00
Aditya Manthramurthy
5cdcc73bd5 Admin API auth and heal related fixes (#5445)
- Fetch region for auth from global state
- Fix SHA256 handling for empty body in heal API
2018-01-25 19:24:00 +05:30
Aditya Manthramurthy
a003de72bf Update madmin doc (fixes #5432) (#5433) 2018-01-22 16:10:43 -08:00
Aditya Manthramurthy
a337ea4d11 Move admin APIs to new path and add redesigned heal APIs (#5351)
- Changes related to moving admin APIs
   - admin APIs now have an endpoint under /minio/admin
   - admin APIs are now versioned - a new API to server the version is
     added at "GET /minio/admin/version" and all API operations have the
     path prefix /minio/admin/v1/<operation>
   - new service stop API added
   - credentials change API is moved to /minio/admin/v1/config/credential
   - credentials change API and configuration get/set API now require TLS
     so that credentials are protected
   - all API requests now receive JSON
   - heal APIs are disabled as they will be changed substantially

- Heal API changes
   Heal API is now provided at a single endpoint with the ability for a
   client to start a heal sequence on all the data in the server, a
   single bucket, or under a prefix within a bucket.

   When a heal sequence is started, the server returns a unique token
   that needs to be used for subsequent 'status' requests to fetch heal
   results.

   On each status request from the client, the server returns heal result
   records that it has accumulated since the previous status request. The
   server accumulates upto 1000 records and pauses healing further
   objects until the client requests for status. If the client does not
   request any further records for a long time, the server aborts the
   heal sequence automatically.

   A heal result record is returned for each entity healed on the server,
   such as system metadata, object metadata, buckets and objects, and has
   information about the before and after states on each disk.

   A client may request to force restart a heal sequence - this causes
   the running heal sequence to be aborted at the next safe spot and
   starts a new heal sequence.
2018-01-22 14:54:55 -08:00
Andreas Auernhammer
d0a43af616 replace all "crypto/sha256" with "github.com/minio/sha256-simd" (#5391)
This change replaces all imports of "crypto/sha256" with
"github.com/minio/sha256-simd". The sha256-simd package
is faster on ARM64 (NEON instructions) and can take advantage
of AVX-512 in certain scenarios.

Fixes #5374
2018-01-17 10:54:31 -08:00
Aditya Manthramurthy
aa7e5c71e9 Remove upload healing related dead code (#5404) 2018-01-15 18:20:39 -08:00
Nitish Tiwari
42633748db
Update madmin package to return storage class parity (#5387)
After the addition of Storage Class support, readQuorum
and writeQuorum are decided on a per object basis, instead
of deployment wide static quorums.

This PR updates madmin api to remove readQuorum/writeQuorum
and add Standard storage class and reduced redundancy storage
class parity as return values. Since these parity values are
used to decide the quorum for each object.

Fixes #5378
2018-01-12 07:52:52 +05:30
Aditya Manthramurthy
cd22feecf8 Remove healing of incomplete multipart uploads (#5390)
Since the server performs automatic clean-up of multipart uploads that
have not been resumed for more than a couple of weeks, it was decided
to remove functionality to heal multipart uploads.
2018-01-11 15:07:43 -08:00
Matthieu Paret
374feda237 add HTTPStats to madmin (#5299) 2017-12-22 17:47:30 -08:00
Nitish Tiwari
34a1b58a75
Remove redirectHeaders method (#5120)
As of go 1.8, headers are copied on redirect, so we no longer need
to do this manually.

See https://github.com/golang/go/issues/4800 and
https://go-review.googlesource.com/c/go/+/28930 for more context on go
behaviour.

Fixes #5042
2017-11-01 12:43:13 +05:30
Bala FA
88938340b3 remove all dead codes (#5019)
Fixes #5012
2017-10-05 12:25:45 -07:00
Harshavardhana
f5b4b0765a Update minio-go dependency (#4551)
This updates dependency for

 - AWS S3 backend.
 - pkg/madmin

```
- Relax isValidBucketName to allow reading existing buckets. (#708) (3 minutes ago) <Harshavardhana>
- For GCS the size limit of S3 is not useful. (#711) (3 days ago) <Harshavardhana>
- s3utils: Support AWS S3 US GovCloud endpoint. (#701) (3 days ago) <Harshavardhana>
- api: Always strip 80/443 port from host (#709) (3 days ago) <Anis Elleuch>
- Redact signature strings properly. (#706) (4 days ago) <Harshavardhana>
- api: Single putObject can use temporary file always. (#703) (6 days ago) <Harshavardhana>
- Spelling fix (#704) (7 days ago) <Jacob Taylor>
- api/encrypt: Get() on encrypted object should be a reader. (#699) (2 weeks ago) <Harshavardhana>
- get: Fix reading an object if its size is unknown (#694) (3 weeks ago) <Anis Elleuch>
- fixes #696 by updating the examples for put-encrypted-object and get-encrypted-object (#697) (3 weeks ago) <Tejay Cardon>
- fix InvalidAccessKeyId error according to amazon documentation (#692) (4 weeks ago) <samkevich>
- Add AWS S3 SSE-C example. (#689) (4 weeks ago) <Harshavardhana>
- According to RFC7232 Etag should be in quotes for If-Match. (#688) (5 weeks ago) <Harshavardhana>
- api: getReaderSize() should honor seeked file descriptors. (#681) (5 weeks ago) <Harshavardhana>
- tests: Use bytes.Repeat() when generating big data (#683) (5 weeks ago) <Anis Elleuch>
- api: Failed call retry with region only when http.StatusBadRequest. (#678) (5 weeks ago) <Harshavardhana>
- api: Add NewWithCredentials() (#646) (5 weeks ago) <Harshavardhana>
```
2017-06-19 16:02:35 -07:00
Anis Elleuch
465274cd21 server-info: Change Error type to string (#4346)
Golang std error type doesn't marshal/unmarshal with json. So errors
are not actually being sent when a client calls ServerInfo() API.
2017-05-15 07:28:47 -07:00
Anis Elleuch
83abad0b37 admin: ServerInfo() returns info for each node (#4150)
ServerInfo() will gather information from all nodes before returning
it back to the client.
2017-04-21 07:15:53 -07:00
Krishnan Parthasarathi
3032f0f505 Remove duration field from lock instrumentation (#4111)
Duration for which a lock was held can be computed from the `Since`
field of `OpsLockState`. It is the difference between current time and
time at which the namespace lock was held. This change avoids
superfluous instrumentation.
2017-04-15 11:40:01 -07:00
Krishnan Parthasarathi
ca64b86112 Return possible states a heal operation (#4045) 2017-04-14 10:28:35 -07:00
Krishnan Parthasarathi
96c46c15e7 madmin: Rename HealObjectResult to HealResult (#4035)
madmin.HealObjectResult is used in HealObject and HealUpload. It only
makes sense to rename it to HealResult.
2017-04-03 08:25:32 -07:00
Krishnan Parthasarathi
2bd694dbc8 Add disksUnavailable healStatus const (#3990)
`disksUnavailable` healStatus constant indicates that a given object
needs healing but one or more of disks requiring heal are offline. This
can be used by admin heal API consumers to distinguish between a
successful heal and a no-op since the outdated disks were offline.
2017-03-31 17:55:15 -07:00
Krishnan Parthasarathi
c27ece409b heal: Check if all parts are available and valid (#3967)
In the algorithm to check if an object requires healing, in addition to
checking if all disks have xl.json present we should check if all parts
of the object are present and have valid blake2b checksums.

Also fixed a minor compilation error in heal-objects-list.go.
2017-03-24 08:40:44 -07:00
Krishnan Parthasarathi
181e002c56 pkg/madmin: Set UploadID in ListUploadsHeal (#3945)
Without this fix, `mc admin heal -I` wouldn't be able to heal ongoing
uploads. `mc` depends on `ListUploadsHeal` API to identify ongoing
uploads to heal given a bucket and an object.
2017-03-21 10:32:02 -07:00
Anis Elleuch
3a6111eff5 admin: Export HealStatus data type (#3930)
`healStatus` can be returned to the API caller. This commit will help
developers to declare a variable with HealStatus type.
2017-03-18 11:27:27 -07:00
Krishnan Parthasarathi
c192e5c9b2 Implement heal-upload admin API (#3914)
This API is meant for administrative tools like mc-admin to heal an
ongoing multipart upload on a Minio server.  N B This set of admin
APIs apply only for Minio servers.

`github.com/minio/minio/pkg/madmin` provides a go SDK for this (and
other admin) operations.  Specifically,

  func HealUpload(bucket, object, uploadID string, dryRun bool) error

Sample admin API request:
POST
/?heal&bucket=mybucket&object=myobject&upload-id=myuploadID&dry-run
- Header(s): ["x-minio-operation"] = "upload"

Notes:
- bucket, object and upload-id are mandatory query parameters
- if dry-run is set, API returns success if all parameters passed are
  valid.
2017-03-17 09:25:49 -07:00
Krishnan Parthasarathi
051f9bb5c6 Implement list uploads heal admin API (#3885) 2017-03-16 00:15:06 -07:00
Anis Elleuch
d602495600 madmin: Do not require SSL to set credentials (#3879)
We need to relax this requirement and let the client decides
if it can allow to set credentials API over plain connection.
2017-03-09 14:08:33 -08:00
Harshavardhana
85cbd875fc cleanup: All conditionals simplified under pkg. (#3875)
Address all the changes reported/recommended by
`gosimple` tool.
2017-03-09 10:13:30 -08:00
Anis Elleuch
cddc684559 admin: Set Config returns errSet and errMsg (#3822)
There is no way to see if a node encountered an error
when trying to set a new config set, this commit adds
a bool errSet field.
2017-03-03 02:53:48 -08:00
Krishnan Parthasarathi
c9619673fb Implement SetConfig admin API handler. (#3792) 2017-02-27 11:40:27 -08:00
Anis Elleuch
461b2bbd37 admin: Move SetCredentials from Service to Generic (#3805)
Setting credentials doesn't belong to service management API
anymore.
2017-02-25 11:06:08 -08:00
Krishnan Parthasarathi
2745bf2f1f Implement ServerConfig admin REST API (#3741)
Returns a valid config.json of the setup. In case of distributed
setup, it checks if quorum or more number of nodes have the same
config.json.
2017-02-20 12:58:50 -08:00
Anis Elleuch
7f86a21317 admin: Add ServerInfo API() (#3743) 2017-02-15 10:45:45 -08:00
Harshavardhana
9df01035da Remove XL references in public docs to Erasure. (#3725)
Ref #3722
2017-02-09 23:26:44 -08:00
Krishnan Parthasarathi
ce9aa2f2b2 Add uptime to ServiceStatus (#3690) 2017-02-08 00:13:02 -08:00
Anis Elleuch
f612a7dd85 madmin: Fix a typo in Locks duration query name (#3673) 2017-02-01 11:46:49 -08:00
Krishnan Parthasarathi
0472e5c1e1 Change query param name to duration in list/clear locks API (#3664)
Following is a sample list lock API request schematic,

  /?lock&bucket=mybucket&prefix=myprefix&duration=holdDuration
  x-minio-operation: list

The response would contain the list of locks held on mybucket matching
myprefix for a duration longer than holdDuration.
2017-02-01 11:17:30 -08:00
Anis Elleuch
c753b5dfeb madmin: All APIs return ErrorResponse error type (#3632)
Avoid returning errors.New("Got HTTP Status") in all APIs
2017-01-25 16:01:29 -08:00
Anis Elleuch
76b8abcd8e madmin: Fix args order in listObjectsHeal() (#3629)
The order of marker and delimiter and in listObjectsHeal() internal function
are switched. That will give wrong result in case of a non recursive objects
heal list.
2017-01-25 09:58:49 -08:00
Krishnan Parthasarathi
0e693e0284 Add dry-run query param for HealFormat API (#3618) 2017-01-24 08:11:05 -08:00
Anis Elleuch
d1d89116f1 admin: Add version to service Status API response (#3605)
Add server's version field to service status API:

"version":{
	"version":"DEVELOPMENT.GOGET",
	"commitID":"DEVELOPMENT.GOGET"
}
2017-01-23 08:56:06 -08:00
Krishnan Parthasarathi
586058f079 Implement mgmt REST APIs to heal storage format. (#3604)
* Implement heal format REST API handler
* Implement admin peer rpc handler to re-initialize storage
* Implement HealFormat API in pkg/madmin
* Update pkg/madmin API.md to incl. HealFormat
* Added unit tests for ReInitDisks rpc handler and HealFormatHandler
2017-01-23 00:32:55 -08:00