Commit graph

116 commits

Author SHA1 Message Date
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588)
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
2019-12-02 09:28:01 -08:00
Harshavardhana
4e9de58675 Avoid pointer based copy, instead use Clone() (#8547)
This PR adds functional test to test expanded
cluster syntax.
2019-11-21 17:54:51 +05:30
poornas
a9b87c0a16 Fix typo in event name (#8545) 2019-11-20 16:35:01 -08:00
poornas
ca96560d56 Add object retention at the per object (#8528)
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
2019-11-20 13:18:09 -08:00
svistoi
c9be601988 NATS TLS specify CA and client TLS authentication (#8389)
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
2019-11-15 09:13:23 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515)
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
2019-11-13 17:38:05 -08:00
Harshavardhana
822eb5ddc7 Bring in safe mode support (#8478)
This PR refactors object layer handling such
that upon failure in sub-system initialization
server reaches a stage of safe-mode operation
wherein only certain API operations are enabled
and available.

This allows for fixing many scenarios such as

 - incorrect configuration in vault, etcd,
   notification targets
 - missing files, incomplete config migrations
   unable to read encrypted content etc
 - any other issues related to notification,
   policies, lifecycle etc
2019-11-09 09:27:23 -08:00
Harshavardhana
9e7a3e6adc Extend further validation of config values (#8469)
- This PR allows config KVS to be validated properly
  without being affected by ENV overrides, rejects
  invalid values during set operation

- Expands unit tests and refactors the error handling
  for notification targets, returns error instead of
  ignoring targets for invalid KVS

- Does all the prep-work for implementing safe-mode
  style operation for MinIO server, introduces a new
  global variable to toggle safe mode based operations
  NOTE: this PR itself doesn't provide safe mode operations
2019-10-30 23:39:09 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Harshavardhana
c33bae057f Fix build issues with event target logger 2019-10-11 19:07:09 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381)
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Praveen raj Mani
dc0dce9beb Check for connection errors during event transit in elasticsearch and webhook (#8273) 2019-09-19 08:23:43 -07:00
Praveen raj Mani
456ce4cc92 Add rootCAs support to Kafka & MQTT (#8236)
Fixes #8211
2019-09-18 23:43:04 +05:30
Praveen raj Mani
8700945cdf Handle connection failures on webhook/url pings (#8204)
Properly handle connection failures while replaying events

Fixes #8194
2019-09-12 16:44:51 -07:00
Praveen raj Mani
428836d4e1 Ignore "connection reset" errors while initializing the targets with queue store enabled (#8185)
Fixes #8178
2019-09-05 13:55:48 -07:00
Praveen raj Mani
e96f19c867 Use default ports for the schemes when dialing (#7960)
Fixes #7946
2019-08-13 16:43:11 +05:30
Praveen raj Mani
63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Praveen raj Mani
b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Praveen raj Mani
55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Praveen raj Mani
c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani
2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Christian Muehlhaeuser
38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Praveen raj Mani
bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Praveen raj Mani
bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Praveen raj Mani
bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
iliul
d3f9f8be88 golint: fix redundant code logic (#7842)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-06-27 15:18:33 +05:30
Ashish Kumar Sinha
3022f60561 Stop duplicate entry in Notification.xml (#7690)
fixes #7671
2019-06-21 13:28:02 +05:30
Harshavardhana
ff7799452b Fix build failure due to nats project migration 2019-06-03 17:49:49 -07:00
Harshavardhana
2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani
763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Praveen raj Mani
c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Praveen raj Mani
998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
Praveen raj Mani
47ca411163 Enhance the event store interface to support channeling (#7343)
- Avoids code duplication across the other targets. By having a
  centralized function call.

- Reduce the room for race.
2019-04-10 18:16:01 +05:30
kannappanr
5ecac91a55
Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
poornas
dd1d69ab5c fix mysql config for native password authentication (#7436)
fixes #7430
2019-03-28 10:45:22 -07:00
Praveen raj Mani
78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
Harshavardhana
df35d7db9d Introduce staticcheck for stricter builds (#7035) 2019-02-13 18:29:36 +05:30
Nitish Tiwari
13c3b8afe2
Update Redis client used for bucket notifications (#7213)
Current Redis client used https://github.com/garyburd/redigo/
has been archived and further development is done at
https://github.com/gomodule/redigo

This commit updates the client library accordingly.

Fixes #6392
2019-02-11 19:06:27 +05:30
Praveen raj Mani
fad59da29d clientID removed in the MQTT config (#7157)
More than one client can't use the same clientID for MQTT connection. 
This causes problem in distributed deployments where config is shared 
across nodes, as each Minio instance tries to connect to MQTT using the
same clientID.

This commit removes the clientID field in config, and allows
MQTT client to create random clientID for each node.
2019-01-29 15:00:15 +05:30
Harshavardhana
e8791ae274 Remove Minio server arch, version from Server: header (#7074) 2019-01-15 13:16:11 +05:30
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Praveen raj Mani
e7af31c2ff Removed clientID from NATS-Streaming Config (#6391)
clientID must be a unique `UUID` for each connections. Now, the
server generates it, rather considering the config.

Removing it as it is non-beneficial right now.

Fixes #6364
2018-11-30 10:46:17 +05:30
Harshavardhana
eddf468aef Lock the targetList properly in go-routines (#6838)
Fixes #6483
2018-11-21 21:25:54 +05:30
Ashish Kumar Sinha
b0d04b9a81 Retry Connection for RabbitMQ (#6837)
Add retries to connect to RabbitMQ 5 times 
with 2s interval

Fixes #6807
2018-11-21 08:37:29 +05:30
Pontus Leitzler
f9779b24ad Enable default vet flags (#6810)
Enable default vet flags except experimental
2018-11-14 10:23:44 -08:00
Harshavardhana
b24b320807 Set notification namespace for NotificationConfiguration (#6789) 2018-11-09 10:40:03 -08:00
Harshavardhana
d58fc68137 Fix shadowing issue in elasticsearch target (#6774) 2018-11-07 12:09:03 -08:00
Matthias Schneider
71c66464c1 feature: added nsq as broker for events (#6740) 2018-11-07 10:23:13 -08:00
Ashish Kumar Sinha
572719872d Event Notification for ElasticSearch (#6764)
Using access format for Event Notification for Elastic Search
2018-11-06 11:38:54 -08:00
Anis Elleuch
e29009d347 Register postgre driver in pkg/event/target (#6689)
Commit 5c13765168 removed postgre registration triggerd
by the automatic gofmt command but it was the only where pg is registered. This commit
fixes behavior and adds unit tests to check whether postgre & sql are registered or not.
2018-10-23 11:44:46 -07:00
Annanay Agarwal
7cb87f863e Kafka (sarama) authentication with user/pass (#6291) 2018-09-07 00:01:58 -07:00
Anis Elleuch
5c13765168 postgresql: Disable validation of connectionString field (#6397)
A bug concerning the validation of connectionString is found,
however there is no solution to fix it for now, postgresql API
doesn't help to do that hence disabling validation of that field.
2018-09-06 20:34:52 +05:30
Harshavardhana
fd1b8491db
Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
Anis Elleuch
e8a008f5b5 Better validation of all config file fields (#6090)
Add Validate() to serverConfig to call it at server
startup and in Admin SetConfig handler to minimize
errors scenario after server restart.
2018-07-18 11:22:29 -07:00
Harshavardhana
4886bfbc72 fix: Avoid more crashes due to concurrent map usage (#5912)
This PR fixes another situation where a crash occurs
thanks to @krishnasrinivas for reproducing this

Fixes #5897
2018-05-09 15:11:51 -07:00
Anis Elleuch
9439dfef64 Use defer style to stop tickers to avoid current/possible misuse (#5883)
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
2018-05-04 10:43:20 -07:00
ebozduman
f16bfda2f2 Remove panic() and handle it appropriately (#5807)
This is an effort to remove panic from the source. 
Add a new call called CriticialIf, that calls LogIf and exits. 
Replace panics with one of CriticalIf, FatalIf and a return of error.
2018-04-19 17:24:43 -07:00
Bala FA
650c6ee8fb event/target/webhook: treat all 2xx as success. (#5792)
Fixes #5769
2018-04-10 17:45:54 -07:00
Takeshi Watanabe
2182c1a4f7 Use paho configuration API instead of setting it directly with struct initializer. (#5707) 2018-03-26 16:45:21 -07:00
Harshavardhana
c726145baf Fix regression in removing notification (#5673)
fixes a regression introduced in 0e4431725c
when removing a previously applied notification configuration.

event.ParseConfig() was stricter in terms of handling notification
configuration, we need to allow when notification configuration is
sent empty, this is the way to remove notification configuration.
2018-03-20 12:02:56 -07:00
Bala FA
0e4431725c make notification as separate package (#5294)
* Remove old notification files

* Add net package

* Add event package

* Modify minio to take new notification system
2018-03-15 13:03:41 -07:00