Commit graph

1290 commits

Author SHA1 Message Date
Harshavardhana
a7be313230 Start using new errors package (#8207) 2019-09-11 22:51:43 +05:30
Yao Zongyou
18fedc67d5 friendly prompt for s3select MalformedXML error (#8171)
partly fix #7911
2019-09-09 21:33:27 -07:00
Harshavardhana
b52a3e523c Avoid using fastjson parser pool, move back to jsoniter (#8190)
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.

The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
2019-09-06 04:21:27 +05:30
Praveen raj Mani
428836d4e1 Ignore "connection reset" errors while initializing the targets with queue store enabled (#8185)
Fixes #8178
2019-09-05 13:55:48 -07:00
Andreas Auernhammer
810a44e951 KMS Admin-API: add route and handler for KMS key info (#7955)
This commit adds an admin API route and handler for
requesting status information about a KMS key.

Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
2019-09-05 01:49:44 +05:30
poornas
8a71b0ec5a Add admin API to send console log messages (#7784)
Utilized by mc admin console command.
2019-09-03 23:40:48 +05:30
Yao Zongyou
ec9bfd3aef speed up the performance of s3select on csv (#7945) 2019-08-31 00:07:40 -07:00
Bala FA
fa3546bb03 Add NetPerfInfo() API in madmin (#8112) 2019-08-31 08:27:53 +05:30
Harshavardhana
83d4c5763c
Decouple ServiceUpdate to ServerUpdate to be more native (#8138)
The change now is to ensure that we take custom URL as
well for updating the deployment, this is required for
hotfix deliveries for certain deployments - other than
the community release.

This commit changes the previous work d65a2c6725
with newer set of requirements.

Also deprecates PeerUptime()
2019-08-28 15:04:43 -07:00
Ashish Kumar Sinha
c6f86d35d3 Rectify Examples and README.MD (#8141) 2019-08-28 10:56:02 -07:00
Harshavardhana
d65a2c6725
Implement cluster-wide in-place updates (#8070)
This PR is a breaking change and also deprecates
`minio update` command, from this release onwards
all users are advised to just use `mc admin update`
2019-08-27 11:37:47 -07:00
Krishna Srinivas
c38ada1a26 write() to disk in 4MB blocks for better performance (#7888) 2019-08-23 15:36:46 -07:00
poornas
7bf1caa0fe Fix broken link to go install docs (#8090) 2019-08-15 16:00:50 -07:00
Aditya Manthramurthy
bf9b619d86 Set the policy mapping for a user or group (#8036)
Add API to set policy mapping for a user or group

Contains a breaking Admin APIs change.

- Also enforce all applicable policies
- Removes the previous /set-user-policy API

 Bump up peerRESTVersion

Add get user info API to show groups of a user
2019-08-13 13:41:06 -07:00
maihde
0ed6daab59 fix: #8051 so that stale DNS entries are cleaned-up (#8053) 2019-08-13 08:49:26 -07:00
Praveen raj Mani
e96f19c867 Use default ports for the schemes when dialing (#7960)
Fixes #7946
2019-08-13 16:43:11 +05:30
Harshavardhana
aaef18b1a3 Fail for disks which overflow upon usage calculation (#8056)
Fixes #8035
2019-08-13 02:58:43 -07:00
Anis Elleuch
1ce8d2c476 Add bucket lifecycle expiry feature (#7834) 2019-08-09 10:02:41 -07:00
Harshavardhana
d759a7ce99
Fix time formatting of Expiration field in STS (#8043)
Without explicit conversion to UTC() from Unix
time the zone information is lost, this leads
to XML marshallers marshaling the time into
a wrong format.

This PR fixes the compatibility issue with AWS STS
API by keeping Expiration format close to ISO8601
or RFC3339

Fixes #8041
2019-08-08 15:44:57 -07:00
Aditya Manthramurthy
414a7eca83 Add IAM groups support (#7981)
This change adds admin APIs and IAM subsystem APIs to:

- add or remove members to a group (group addition and deletion is
  implicit on add and remove)

- enable/disable a group

- list and fetch group info
2019-08-02 14:25:00 -07:00
maihde
5cd9f10a02 Support Federation on a single machine (#8009)
When checking if federation is necessary, the code compares
the SRV record stored in etcd against the list of endpoints
that the MinIO server is exposing.  If there is an intersection
in this list the request is forwarded.

The SRV record includes both the host and the port, but the
intersection check previously only looked at the IP address.  This
would prevent federation from working in situations where the endpoint
IP is the same for multiple MinIO servers.  Some examples of where this
can occur are:
 - running mulitiple copies of MinIO on the same host
 - using multiple MinIO servers behind a NAT with port-forwarding
2019-08-02 12:40:51 -07:00
maihde
d966d29fed fix: add integer code for Windows Subsystem for Linux filesystem (#8010) 2019-08-01 06:00:57 -07:00
Anis Elleuch
cbd02c58be federation: Avoid printing context canceled error (#7997)
Golang proactively prints this error
        `http: proxy error: context canceled`

when a request arrived to the current deployment and
redirected to another deployment in a federated setup.

Since this error can confuse users, this commit will
just hide it.
2019-07-31 11:08:10 -07:00
Praveen raj Mani
63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Harshavardhana
8d47ef503c Fix crash observed in OPA initialization (#7990)
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.

This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
2019-07-29 15:58:25 -07:00
Harshavardhana
94c88890b8 Add additional logging for OPA connections (#7982) 2019-07-28 08:33:25 +05:30
Praveen raj Mani
efb8b00db0 Preserve tailing backslash in URL paths (#7678)
Fixes #7649
2019-07-25 20:55:09 -07:00
Praveen raj Mani
b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Praveen raj Mani
55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Praveen raj Mani
c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani
2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Christian Muehlhaeuser
38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Harshavardhana
8b5e6e338c Fix: Only add SRV records that match the bucket name exactly (#7957)
Problem: MinIO incorrectly appends DNS SRV records of buckets that have a prefix match with a given bucket. E.g bucket1 would incorrectly get bucket's DNS records too.
Solution: This fix ensures that we only add SRV records that match the key exactly
2019-07-20 11:29:05 +01:00
poornas
0373a1699b Add error filter to admin trace API (#7923)
This allows MinIO to have the ability to send back only error trace
2019-07-20 01:38:26 +01:00
Krishnan Parthasarathi
559a59220e Add initial support for bucket lifecycle (#7563)
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)

N B the code for actual expiration of objects will be included in a
subsequent PR.
2019-07-19 21:20:33 +01:00
poornas
041a812ba0 trace api: add call stats to trace (#7915)
Stats such as call latency, bytes received and sent have been added
2019-07-18 23:29:17 +01:00
Kanagaraj M
12353caf35 Fix: Support Unicode delimiters in s3 select (#7931) 2019-07-17 19:10:17 +01:00
Praveen raj Mani
bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Praveen raj Mani
bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Yao Zongyou
c4f480a839 fix csv read bug (#7885) 2019-07-05 12:08:56 -07:00
Yao Zongyou
60831e3299 aggregation functions' argument may already has been cast to numeric (#7876) 2019-07-05 10:38:38 -07:00
Yao Zongyou
037319066f fix unicode support related bugs in s3select (#7877) 2019-07-05 09:43:10 -07:00
Praveen raj Mani
bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
Ryan Tam
bd56f80250 Fix ignored alias for aggregate result in S3 Select (#7849)
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.

Also improve test for S3 select

On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).

Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.

[1] - https://github.com/golang/go/wiki/TableDrivenTests
2019-07-03 16:34:54 -07:00
iliul
d3f9f8be88 golint: fix redundant code logic (#7842)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-06-27 15:18:33 +05:30
Krishna Srinivas
183ec094c4 Simplify HTTP trace related code (#7833) 2019-06-26 22:41:12 -07:00
Anis Elleuch
48f2c98052 admin: Add Background heal status info API (#7774)
This API returns the information related to the self healing routine.

For the moment, it returns:
- The total number of objects that are scanned
- The last time when an item was scanned
2019-06-25 16:42:24 -07:00
Yao Zongyou
941fed8e4a s3Select: call Close on error to release the read lock (#7830) 2019-06-25 13:30:48 -07:00
Yao Zongyou
55092bede1 add timestamp compare support (#7832) 2019-06-25 11:05:37 -07:00
Yao Zongyou
90a3b830f4 fix typo and the string representation of the time.Time value (#7831) 2019-06-25 09:54:14 -07:00