Commit graph

1269 commits

Author SHA1 Message Date
maihde
d966d29fed fix: add integer code for Windows Subsystem for Linux filesystem (#8010) 2019-08-01 06:00:57 -07:00
Anis Elleuch
cbd02c58be federation: Avoid printing context canceled error (#7997)
Golang proactively prints this error
        `http: proxy error: context canceled`

when a request arrived to the current deployment and
redirected to another deployment in a federated setup.

Since this error can confuse users, this commit will
just hide it.
2019-07-31 11:08:10 -07:00
Praveen raj Mani
63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Harshavardhana
8d47ef503c Fix crash observed in OPA initialization (#7990)
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.

This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
2019-07-29 15:58:25 -07:00
Harshavardhana
94c88890b8 Add additional logging for OPA connections (#7982) 2019-07-28 08:33:25 +05:30
Praveen raj Mani
efb8b00db0 Preserve tailing backslash in URL paths (#7678)
Fixes #7649
2019-07-25 20:55:09 -07:00
Praveen raj Mani
b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Praveen raj Mani
55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Praveen raj Mani
c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani
2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Christian Muehlhaeuser
38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Harshavardhana
8b5e6e338c Fix: Only add SRV records that match the bucket name exactly (#7957)
Problem: MinIO incorrectly appends DNS SRV records of buckets that have a prefix match with a given bucket. E.g bucket1 would incorrectly get bucket's DNS records too.
Solution: This fix ensures that we only add SRV records that match the key exactly
2019-07-20 11:29:05 +01:00
poornas
0373a1699b Add error filter to admin trace API (#7923)
This allows MinIO to have the ability to send back only error trace
2019-07-20 01:38:26 +01:00
Krishnan Parthasarathi
559a59220e Add initial support for bucket lifecycle (#7563)
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)

N B the code for actual expiration of objects will be included in a
subsequent PR.
2019-07-19 21:20:33 +01:00
poornas
041a812ba0 trace api: add call stats to trace (#7915)
Stats such as call latency, bytes received and sent have been added
2019-07-18 23:29:17 +01:00
Kanagaraj M
12353caf35 Fix: Support Unicode delimiters in s3 select (#7931) 2019-07-17 19:10:17 +01:00
Praveen raj Mani
bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Praveen raj Mani
bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Yao Zongyou
c4f480a839 fix csv read bug (#7885) 2019-07-05 12:08:56 -07:00
Yao Zongyou
60831e3299 aggregation functions' argument may already has been cast to numeric (#7876) 2019-07-05 10:38:38 -07:00
Yao Zongyou
037319066f fix unicode support related bugs in s3select (#7877) 2019-07-05 09:43:10 -07:00
Praveen raj Mani
bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
Ryan Tam
bd56f80250 Fix ignored alias for aggregate result in S3 Select (#7849)
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.

Also improve test for S3 select

On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).

Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.

[1] - https://github.com/golang/go/wiki/TableDrivenTests
2019-07-03 16:34:54 -07:00
iliul
d3f9f8be88 golint: fix redundant code logic (#7842)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-06-27 15:18:33 +05:30
Krishna Srinivas
183ec094c4 Simplify HTTP trace related code (#7833) 2019-06-26 22:41:12 -07:00
Anis Elleuch
48f2c98052 admin: Add Background heal status info API (#7774)
This API returns the information related to the self healing routine.

For the moment, it returns:
- The total number of objects that are scanned
- The last time when an item was scanned
2019-06-25 16:42:24 -07:00
Yao Zongyou
941fed8e4a s3Select: call Close on error to release the read lock (#7830) 2019-06-25 13:30:48 -07:00
Yao Zongyou
55092bede1 add timestamp compare support (#7832) 2019-06-25 11:05:37 -07:00
Yao Zongyou
90a3b830f4 fix typo and the string representation of the time.Time value (#7831) 2019-06-25 09:54:14 -07:00
Yao Zongyou
23b9df0694 Fix s3select TRIM function's nil pointer dereference bug (#7817) 2019-06-24 16:59:33 -07:00
Ashish Kumar Sinha
3022f60561 Stop duplicate entry in Notification.xml (#7690)
fixes #7671
2019-06-21 13:28:02 +05:30
Harshavardhana
1af6e8cb72
Add support for session policies in STS APIs (#7747)
This PR adds support for adding session policies
for further restrictions on STS credentials, useful
in situations when applications want to generate
creds for multiple interested parties with different
set of policy restrictions.

This session policy is not mandatory, but optional.

Fixes #7732
2019-06-20 15:28:33 -07:00
poornas
299ef9b188 Trace: Replace function name with API prefix (#7794)
This change is required for `Admin Trace`
2019-06-18 13:55:13 -07:00
Harshavardhana
99bf4d0c42 [sec] Match ${aws:username} exactly instead of prefix match (#7791)
This PR fixes a security issue where an IAM user based
on his policy is granted more privileges than restricted
by the users IAM policy.

This is due to an issue of prefix based Matcher() function
which was incorrectly matching prefix based on resource
prefixes instead of exact match.
2019-06-15 11:27:17 -07:00
Harshavardhana
002a205c9c Fix OPA result response handling (#7763)
Also update the document with updated rego policy
and updated OPA agent REST API.

This PR is to fix a regression caused by PR #7637
2019-06-10 17:06:32 -07:00
poornas
97090aa16c Add admin API to send trace notifications to registered (#7128)
Remove current functionality to log trace to file
using MINIO_HTTP_TRACE env, and replace it with
mc admin trace command on mc client.
2019-06-08 15:54:41 -07:00
Joe Stevens
a19cf063b5 Fixes for multiplatform dev and testing from forks (#7734)
Add support for correct dependency URLs on all platforms

only build mountinfo.go on linux

make testfile path relative to support fork work
2019-06-04 00:59:40 -07:00
Harshavardhana
ff7799452b Fix build failure due to nats project migration 2019-06-03 17:49:49 -07:00
Harshavardhana
d16a409943
Ignore srvRecords from domain level entries (#7742)
Fixes #7741
2019-06-03 16:16:05 -07:00
Harshavardhana
2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani
763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Harshavardhana
7906f464fd
Skip non-bucket dns entry (#7699)
Fixes #7697
2019-05-28 19:13:49 -07:00
Harshavardhana
39b3e4f9b3 Avoid using io.ReadFull() for WriteAll and CreateFile (#7676)
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.

Also adds readahead for disk reads, which also increases
performance for reads by 3x.
2019-05-22 13:47:15 -07:00
Praveen raj Mani
c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Krishna Srinivas
502456db91 Accept simpler response from OPA (#7637) 2019-05-12 18:40:49 +05:30
Harshavardhana
ac3b59645e
Add API retry functionality in mc admin (#7602) 2019-05-09 17:41:54 -07:00
Harshavardhana
72929ec05b
Turn off md5sum optionally if content-md5 is not set (#7609)
This PR also brings --compat option to run MinIO in strict
S3 compatibility mode, MinIO by default will now try to run
high performance mode.
2019-05-08 18:35:40 -07:00
Praveen raj Mani
998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
Harshavardhana
ae002aa724 Deprecate updating admin credentials using API calls (#7570)
Root credentials are not allowed to change in all of our
distributed setup deployments, this PR simply removes
that behavior.
2019-04-24 12:54:44 -07:00
Krishna Srinivas
a3ec71bc28 Use O_DIRECT while writing to disk (#7479)
- Use O_DIRECT while writing to disk
- Remove MINIO_DRIVE_SYNC option
2019-04-23 21:25:06 -07:00