Commit graph

2056 commits

Author SHA1 Message Date
Harshavardhana 4155f4e49b
trace: Print either Transfer-Encoding or Content-Length (#8314)
If Transfer-Encoding is set client would have
never set Content-Length as its considered
malformed HTTP request
2019-09-27 10:19:27 -07:00
Bala FA 2a2ff96ee1 change ReadPerf into ReadThroughput in NetPerfInfo. (#8316)
Previously `ReadPerf` was in time.Duration is changed to `ReadThroughput` in uint64.
2019-09-27 00:01:18 +05:30
Harshavardhana fd53057654 Add InfoCannedPolicy API to fetch only necessary policy (#8307)
This PR adds
- InfoCannedPolicy() API for efficiency in fetching policies
- Send group memberships for LDAPUser if available
2019-09-26 23:53:13 +05:30
Klaus Post ff726969aa Switch to Snappy -> S2 compression (#8189) 2019-09-25 23:08:24 -07:00
Harshavardhana c8fbc94329
Fix writing 'format.json' and make it atomic (#8296)
- Choose a unique uuid such that under situations of duplicate
  mounts we do not append to an existing json entry.
- Avoid AppendFile instead use WriteAll() to write the entire
  byte array atomically.
2019-09-24 18:47:26 -07:00
Anis Elleuch a790877c01 s3: Encode continuation & next continuation tokens when asked (#8292)
When url encoding is passed in v2 listing handler, continuationToken
and nextContinuationToken needs to be encoded. The reason is that
both represents an object name/prefix in Minio server and it could
contain a character unsupported by XML specification.
2019-09-24 05:30:53 +05:30
Harshavardhana 77dc2031a2 Fix LDAP responseXML to be named appropriately (#8285)
This PR additionally also adds support for missing

- Session policy support for AD/LDAP
- Add API request/response parameters detail
- Update example to take ldap username,
  password input from the command line
- Fixes session policy handling for
  ClientGrants and WebIdentity
2019-09-24 03:51:16 +05:30
Harshavardhana 975134e42b
Add checks in DiskInfo() to protect against changing mounts (#8286) 2019-09-23 15:16:55 -07:00
Andreas Auernhammer cb7d23cb17 remove SSE-S3 key rotation in CopyObject (#8278)
This commit removes the SSE-S3 key rotation functionality
from CopyObject since there will be a dedicated Admin-API
for this purpose.

Also update the security documentation to link to mc and
the admin documentation.
2019-09-24 02:05:04 +05:30
poornas 2e02e1889b Cleanup ResponseWriter function for audit and trace (#8283) 2019-09-24 02:04:28 +05:30
ebozduman dbf7b1e573 starts-with policy condition support issue (#7937) 2019-09-22 14:20:49 -07:00
Harshavardhana 26985ac632 Fix all failing tests with -race 2019-09-22 11:01:46 -07:00
Praveen raj Mani ad75683bde Authorize prometheus endpoint with bearer token (#7640) 2019-09-22 20:27:12 +05:30
poornas 4925bc3e80 log server startup messages to admin console api (#8264) 2019-09-22 13:54:32 +05:30
Andreas Auernhammer ffded5a930 make the crypto error type a native go type (#8267)
This commit makes the `crypto.Error` type a native go (string)
type. That allows us to define error values as constants instead
of variables.

For reference see:
 - https://twitter.com/_aead_/status/1118170258215514115?s=20
 - https://dave.cheney.net/2016/04/07/constant-errors
2019-09-22 01:12:51 -07:00
Andreas Auernhammer 2b51fe9f26 make SSE request header check comprehensive (#8276)
This commit refactors the SSE header check
by moving it into the `crypto` package, adds
a unit test for it and makes the check comprehensive.
2019-09-21 03:26:12 +05:30
Harshavardhana 4780fa5a58 Remove setting net.Conn Deadlines as its not needed anymore (#8269)
This commit fixes a bug introduced in af6c6a2b35.

Setting deadlines in Go results in arbitrary hangs as reported here
https://github.com/golang/go/issues/34385

Fixes https://github.com/minio/minio/issues/7852
2019-09-20 23:37:24 +05:30
Andreas Auernhammer b823d6d7bd remove the unused code for decrypting io.Writer (#8277)
This commit removes unused code for decrypting
`io.Writer` since the actual implementation only
decrypts `io.Reader`
2019-09-20 14:51:07 +05:30
Andreas Auernhammer a9d724120f remove TLS 1.3 opt-in code (#8275)
This commit removes the TLS 1.3 opt-in code.
Since TLS 1.3 is opt-out for >= Go 1.13 this
code is not needed anymore.
2019-09-20 01:51:44 +05:30
Andreas Auernhammer e34369c860 prepare SSE-S3 metadata parsing for K/V data key store (#8259)
This commit allows the MinIO server to parse the metadata if:
 - either the `X-Minio-Internal-Server-Side-Encryption-S3-Key-Id`
   and the `X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key`
   entries are present.
 - or *both* headers are not present.

This is in service to support a K/V data key store.
2019-09-19 04:08:09 +05:30
Praveen raj Mani 456ce4cc92 Add rootCAs support to Kafka & MQTT (#8236)
Fixes #8211
2019-09-18 23:43:04 +05:30
Harshavardhana cb01516a26 In HDFS gateway fix non-empty folder behavior (#8254)
To be compatible with our FS and Erasure coded
mode deployments, make sure that we do not send
200 OK for folders which have files inside.

Fixes #8143
2019-09-18 01:59:59 +05:30
Krishnan Parthasarathi 31bee6b6ed Remove size query parameter from PerfInfo handler (#8258) 2019-09-18 01:59:12 +05:30
poornas 04b92124c5 fs/xl: Log warning if cache config specified (#8251)
in non-gateway mode.
2019-09-16 19:55:52 -07:00
Harshavardhana 5392eee250 Avoid recursion and use a simple loop to merge entries (#8239)
This avoids stack overflows when there are
lot of entries to be skipped, this PR also
optimizes the code to reuse the buffers.
2019-09-17 06:08:37 +05:30
Harshavardhana 14b137aa66 posix/readDir should populate name for DT_UKNOWN (#8240)
In commit a8296445ad we changed the code to handle
some corner cases on ARM and other platforms, this
PR just avoids the return for unknown filetypes
prematurely and let the name be populated appropriately.

This fixes bug for older XFS implementations such as
in Ubuntu 14.04
2019-09-17 03:04:01 +05:30
Andreas Auernhammer 3064da7b08 return error during part listing when no quorum (#8241)
This commit fixes a subtle bug that (probably)
caused an issue affecting encrypted multipart objects.

When a cluster has no quorum this bug causes `ListObjectParts`
to return nil as error instead of a quorum error.

Thanks to @harshavardhana for detecting this.
2019-09-17 02:57:34 +05:30
poornas 76df027264 Allow caching only in gateway mode. (#8232)
This PR changes cache on PUT behavior to background fill the cache
after PutObject completes. This will avoid concurrency issues as in #8219.

Added cleanup of partially filled cache to prevent cache corruption
- Fixes #8208
2019-09-17 02:54:04 +05:30
Harshavardhana 9ac12cf898
Remove unusued Set/GetConfigKeys API (#8235) 2019-09-13 16:34:34 -07:00
Harshavardhana e7f491a14b Use optimized sha256-simd whenever possible (#8227)
Avoid using `crypto/sha256` and use always
`github.com/minio/sha256-simd`
2019-09-14 00:39:39 +05:30
Praveen raj Mani 8700945cdf Handle connection failures on webhook/url pings (#8204)
Properly handle connection failures while replaying events

Fixes #8194
2019-09-12 16:44:51 -07:00
Harshavardhana ff6aabd9c0 Honor standard HTTP headers for sourceIP (#8233)
Behind load balancers we should be tracing sourceIP
preserved by load balancers.
2019-09-13 03:59:59 +05:30
Krishnan Parthasarathi 6ba323b009 Add ability to test drive speeds on a MinIO setup (#7664)
- Extends existing Admin API to measure disk performance
2019-09-13 03:22:30 +05:30
Anis Elleuch e7b3f39064 xl: Fix verifying non streaming highway algo with a dist setup (#8230)
VerifyFile in the distributed setup does not work with
the non streaming highway hash. The reason is that the
internode mux router did not expect `storageRESTBitrotHash`
parameter.
2019-09-12 13:08:02 -07:00
Harshavardhana 9fa727d154 Provide a friendlier error when an update fails (#8228)
Add upgrading documentation as well
2019-09-13 01:33:42 +05:30
Harshavardhana 73e4e99942 Hosts should be skipped, when calculating local info (#8191)
endpoint.IsLocal will not have .Host entries so
using them to skip double entries will never work.

change the code such that we look for endpoint.Host
outside of endpoint.IsLocal logic to skip double
hosts appropriately.

Move these functions to their appropriate file.
2019-09-12 23:36:12 +05:30
Alex Pardoe a87fc7d09b Use the B2 'list' endpoint to determine file ID (#8169)
- More effective deletion and checking for existence.
- Rever Dockerfile.
- Add a 'GOPROXY' to the Dockerfile to workaround Apache issues.
2019-09-12 22:48:47 +05:30
Harshavardhana 475df52a19 Fix etcd watch regression in IAM subsystem (#8224)
Fixes #8223
2019-09-12 07:24:25 +05:30
Anis Elleuch 3f258062d8 bitrot: Verify file size inside storage interface (#7932) 2019-09-12 02:19:53 +05:30
Harshavardhana 53e4887e02 Simplify and cleanup metadata r/w functions (#8146) 2019-09-11 22:52:12 +05:30
Harshavardhana a7be313230 Start using new errors package (#8207) 2019-09-11 22:51:43 +05:30
Harshavardhana e12f52e2c6 Enhancements to daily-sweeper routine to reduce CPU load (#8209)
- ListObjectsHeal should list only objects
  which need healing, not the entire namespace.
- DeleteObjects() to be used to delete 1000s of
  objects in bulk instead of serially.
2019-09-11 00:38:44 +05:30
Aditya Manthramurthy a0456ce940 LDAP STS API (#8091)
Add LDAP based users-groups system

This change adds support to integrate an LDAP server for user
authentication. This works via a custom STS API for LDAP. Each user
accessing the MinIO who can be authenticated via LDAP receives
temporary credentials to access the MinIO server.

LDAP is enabled only over TLS.

User groups are also supported via LDAP. The administrator may
configure an LDAP search query to find the group attribute of a user -
this may correspond to any attribute in the LDAP tree (that the user
has access to view). One or more groups may be returned by such a
query.

A group is mapped to an IAM policy in the usual way, and the server
enforces a policy corresponding to all the groups and the user's own
mapped policy.

When LDAP is configured, the internal MinIO users system is disabled.
2019-09-10 04:42:29 +05:30
Harshavardhana b52a3e523c Avoid using fastjson parser pool, move back to jsoniter (#8190)
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.

The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
2019-09-06 04:21:27 +05:30
poornas 259a5d825b cache - fix corruption when client prematurely terminates request (#8155) 2019-09-05 23:33:32 +05:30
poornas 29f64355ce Allow caching on single PutObject (#8100) 2019-09-05 19:50:16 +05:30
Nitish Tiwari 496fba3e9a
Return 200 OK for liveness checks while distributed cluster starts (#8176)
With this PR, liveness check responds with 200 OK with "server-not-
initialized" header while objectLayer gets initialized. The header
is removed as objectLayer is initialized. This is to allow
MinIO distributed cluster to get started when running on an
orchestration platforms like Docker Swarm.

This PR also updates sample Swarm yaml files to use correct values
for healthcheck fields.

Fixes #8140
2019-09-05 14:50:56 +05:30
Andreas Auernhammer 810a44e951 KMS Admin-API: add route and handler for KMS key info (#7955)
This commit adds an admin API route and handler for
requesting status information about a KMS key.

Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
2019-09-05 01:49:44 +05:30
Praveen raj Mani 341d61e3d8 Fix for web-uploads in federated mode (#8175)
Fixes #8173
2019-09-04 23:14:02 +05:30
poornas 8a71b0ec5a Add admin API to send console log messages (#7784)
Utilized by mc admin console command.
2019-09-03 23:40:48 +05:30