Commit graph

1148 commits

Author SHA1 Message Date
Aarushi Arya 89b14639a9 avoid using URL encoding to generate keys (#6731) 2018-10-31 15:07:20 -07:00
Harshavardhana 3f744c0361 Fix mimedb update files (#6744) 2018-10-31 14:15:27 -07:00
Harshavardhana f162d7bd97 Performance improvements by re-using record buffer (#6622)
Avoid unnecessary pointer reference allocations
when not needed, for example

- *SelectFuncs{}
- *Row{}
2018-10-31 08:48:01 +05:30
Harshavardhana bab4c90c45 Fix broken links in docs (#6700) 2018-10-25 11:39:31 +05:30
Harshavardhana fde8c38638 Add default canned policies (#6690) 2018-10-24 17:14:27 -07:00
Anis Elleuch e29009d347 Register postgre driver in pkg/event/target (#6689)
Commit 5c13765168 removed postgre registration triggerd
by the automatic gofmt command but it was the only where pg is registered. This commit
fixes behavior and adds unit tests to check whether postgre & sql are registered or not.
2018-10-23 11:44:46 -07:00
Pontus Leitzler 9631d65552 Fix goroutine test fatalf (#6682)
Use t.Error/t.ErrorF instead if t.Fatal/t.Fatalf

Add returns to achieve same behaviour as earlier
2018-10-23 09:44:20 -07:00
Harshavardhana b99aaab42e Sid value can be any unicode character support it (#6676)
Fixes #6476
2018-10-23 16:11:06 +05:30
Ashish Kumar Sinha c0b4bf0a3e SQL select query for CSV/JSON (#6648)
select * , select column names have been implemented for CSV.
select * is implemented for JSON.
2018-10-22 12:12:22 -07:00
Praveen raj Mani cef044178c Treat columns with spaces inbetween [s3Select] (#6597)
replace the double/single quotes with backticks for the xwb1989/sqlparser
to recognise such queries.

Fixes #6589
2018-10-17 11:01:26 -07:00
Harshavardhana b251454dd6 Fix toggling users status (#6640) 2018-10-16 14:55:23 -07:00
Harshavardhana 1e7e5e297c
Add canned policy support (#6637)
This PR adds an additional API where we can create
a new set of canned policies which can be used with one
or many users.
2018-10-16 12:48:19 -07:00
Harshavardhana 3ef3fefd54 Add ListUsers API to list all configured users in IAM (#6619) 2018-10-13 12:48:43 +05:30
Aditya Manthramurthy e3eec89d24 Optimize string processing in select (#6593)
Reduce allocations during string concatenation and simplify some
processing code.
2018-10-09 14:02:19 -07:00
Harshavardhana 54ae364def Introduce STS client grants API and OPA policy integration (#6168)
This PR introduces two new features

- AWS STS compatible STS API named AssumeRoleWithClientGrants

```
POST /?Action=AssumeRoleWithClientGrants&Token=<jwt>
```

This API endpoint returns temporary access credentials, access
tokens signature types supported by this API

  - RSA keys
  - ECDSA keys

Fetches the required public key from the JWKS endpoints, provides
them as rsa or ecdsa public keys.

- External policy engine support, in this case OPA policy engine

- Credentials are stored on disks
2018-10-09 14:00:01 -07:00
Aditya Manthramurthy 16a100b597 Fix out-of-bound array access crash in select processing (#6594)
Fix test case.
2018-10-09 09:45:56 -07:00
Ashish Kumar Sinha 670f9788e3 Count(*) to give integer value (#6564)
The Max, Min functions were giving float value even when they were integers.  
Resolved max and Min to return integers in that scenario.

Fixes #6472
2018-10-04 17:33:53 -07:00
Praveen raj Mani c7722fbb1b Simplify pkg mimedb (#6549)
Content-Type resolution can now use a function `TypeByExtension(extension)` 
to resolve to the respective content-type.
2018-10-02 11:48:17 +05:30
Anis Elleuch 83d7ec09c1 Disable restarting server after setting a new config (#6521)
Also disable listening to service restart event in tests since
we don't do this anymore.
2018-09-28 12:10:51 -07:00
Praveen raj Mani ce9d36d954 Add object compression support (#6292)
Add support for streaming (golang/LZ77/snappy) compression.
2018-09-28 09:06:17 +05:30
Harshavardhana 1111419d4a Add debugging for mutex, tracing (#6522) 2018-09-27 09:32:05 +05:30
Anis Elleuch 20378821cf madmin: close http response when returning an error (#6526)
httpRespToErrorResponse() usually reads the http response when
the http error code is not expected to parse the json error
response in the http body, however it was never properly closing
the connection. This PR fixes the behavior.
2018-09-26 11:03:35 -07:00
Aditya Manthramurthy 36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
Anis Elleuch 9531cddb06 Add Profiler Admin API (#6463)
Two handlers are added to admin API to enable profiling and disable
profiling of a server in a standalone mode, or all nodes in the
distributed mode.

/minio/admin/profiling/start/{cpu,block,mem}:
  - Start profiling and return starting JSON results, e.g. one
    node is offline.

/minio/admin/profiling/download:
  - Stop the on-going profiling task
  - Stream a zip file which contains all profiling files that can
    be later inspected by go tool pprof
2018-09-18 16:46:35 -07:00
Harshavardhana a0683d3c1f Send progress only when requested by client in SelectObject (#6467) 2018-09-17 11:52:46 +05:30
Harshavardhana 0c2b708484 for O_RDONLY mode hold shared locks on windows (#6454)
Fixes #6401
2018-09-12 09:29:41 -07:00
Praveen raj Mani 30d4a2cf53 s3select should honour custom record delimiter (#6419)
Allow custom delimiters like `\r\n`, `a`, `\r` etc in input csv and 
replace with `\n`.

Fixes #6403
2018-09-10 21:50:28 +05:30
Annanay Agarwal 7cb87f863e Kafka (sarama) authentication with user/pass (#6291) 2018-09-07 00:01:58 -07:00
Anis Elleuch 5c13765168 postgresql: Disable validation of connectionString field (#6397)
A bug concerning the validation of connectionString is found,
however there is no solution to fix it for now, postgresql API
doesn't help to do that hence disabling validation of that field.
2018-09-06 20:34:52 +05:30
Anis Elleuch 3099af70a3 Add admin get/set config keys API (#6113)
This PR adds two new admin APIs in Minio server and madmin package:
- GetConfigKeys(keys []string) ([]byte, error)
- SetConfigKeys(params map[string]string) (err error)

A key is a path in Minio configuration file, (e.g. notify.webhook.1)

The user will always send a string value when setting it in the config file,
the API will know how to convert the value to the appropriate type. The user
is also able to set a raw json.

Before setting a new config, Minio will validate all fields and try to connect
to notification targets if available.
2018-09-06 20:33:18 +05:30
Harshavardhana fd1b8491db
Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
Harshavardhana 4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
Aditya Manthramurthy e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
Raphael Randschau 8601f29d95 select: fix int overflow of math.MaxInt64 on ARM (#6317) 2018-08-22 16:16:04 +05:30
Harshavardhana 9f14433cbd Ensure that setConfig uses latest functionality (#6302) 2018-08-17 18:51:34 -07:00
Harshavardhana 5a4a57700b Add select docs and fix return values for Select API (#6300) 2018-08-17 17:11:39 -07:00
Harshavardhana 1103ad2d08 Watch for symlinked certs in container envs (#6282)
Fixes #6278
2018-08-16 18:37:21 -07:00
Arjun Mishra 7c14cdb60e S3 Select API Support for CSV (#6127)
Add support for trivial where clause cases
2018-08-15 03:30:19 -07:00
Harshavardhana 0e02328c98 Migrate config.json from config-dir to backend (#6195)
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes #6182
2018-08-15 10:11:47 +05:30
Anis Elleuch 6df20734f9 Avoid logging the body of the http 206 response (#6258)
When an S3 client issues a GET request with range specified, Minio
server returns some partial data with 206 http code. The latter
is sent in MINIO_HTTP_TRACE output which is incorrect. This PR
fixes the issue.
2018-08-08 12:34:42 -07:00
Harshavardhana a091b1a3ee Fix admin API doc formatting (#6235) 2018-08-02 14:21:38 -07:00
Harshavardhana 556a51120c Deprecate ListLocks and ClearLocks (#6233)
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.

Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
2018-08-02 23:09:42 +05:30
Harshavardhana 36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
Anis Elleuch e8a008f5b5 Better validation of all config file fields (#6090)
Add Validate() to serverConfig to call it at server
startup and in Admin SetConfig handler to minimize
errors scenario after server restart.
2018-07-18 11:22:29 -07:00
Stefan Husch 914c76a801 Implement lock.Open() to fix #5642 (#6150) 2018-07-13 18:37:02 -07:00
Nitish Tiwari 2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Anis Elleuch 4ddc222f46 fix: Propagate bucket policy update in a distributed setup (#6135)
Commit 0d52126023 caused a regression in setting
a new bucket policy in a distributed setup. The reason is that gob is not able
to encode fields declared as interfaces unless we provide GobEncode() and GobDecode()
This PR adds them by using json marshaller and unmarshaller that are already
implemented for Functions interface.
2018-07-09 02:18:48 -07:00
Harshavardhana 0ef0d7e685 pkg/certs: On windows watch for directory changes to load certs (#6128)
This PR fixes an issue when configuring Minio TLS on windows
2018-07-05 16:33:37 -07:00
ebozduman b0b0fb4c8d Validate Minio config.json file on the client side (#6067) 2018-07-03 11:07:46 -07:00
Harshavardhana e5e522fc61
docs: fix all Chinese doc links for the new docs site (#6097)
Additionally fix typos, default to US locale words
2018-06-28 16:02:02 -07:00