maxmustermann/minio
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Support IPv6 hosts for postgres connections (#7168)

Browse source 
Fixes #7145
This commit is contained in:
Praveen raj Mani 2019-05-03 17:31:33 +05:30 committed by Nitish Tiwari
parent 4b858b562a
commit 998f01fadc
5 changed files with 235 additions and 127 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
dockerscripts/healthcheck Executable file
View file

Binary file not shown.

278
docs/bucket/notifications/README.md
View file

@ -2,29 +2,30 @@
Events occurring on objects in a bucket can be monitored using bucket event notifications. Event types supported by MinIO server are
| Supported Event Types | | |
|:---------------------------|--------------------------------------------|-------------------------|
| `s3:ObjectCreated:Put` | `s3:ObjectCreated:CompleteMultipartUpload` | `s3:ObjectAccessed:Head`|
| `s3:ObjectCreated:Post` | `s3:ObjectRemoved:Delete` |
| `s3:ObjectCreated:Copy` | `s3:ObjectAccessed:Get` |
| Supported Event Types | | |
| :---------------------- | ------------------------------------------ | ------------------------ |
| `s3:ObjectCreated:Put` | `s3:ObjectCreated:CompleteMultipartUpload` | `s3:ObjectAccessed:Head` |
| `s3:ObjectCreated:Post` | `s3:ObjectRemoved:Delete` |
| `s3:ObjectCreated:Copy` | `s3:ObjectAccessed:Get` |
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://docs.min.io/docs/minio-client-complete-guide#events). MinIO SDK's [`BucketNotification` APIs](https://docs.min.io/docs/golang-client-api-reference#SetBucketNotification) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Bucket events can be published to the following targets:
| Supported Notification Targets | | |
|:---------------------------|--------------------------------------------|-------------------------|
| [`AMQP`](#AMQP) | [`Redis`](#Redis) | [`MySQL`](#MySQL) |
| [`MQTT`](#MQTT) | [`NATS`](#NATS) | [`Apache Kafka`](#apache-kafka) |
| [`Elasticsearch`](#Elasticsearch) | [`PostgreSQL`](#PostgreSQL) | [`Webhooks`](#webhooks) |
| [`NSQ`](#NSQ) | | |
| Supported Notification Targets | | |
| :-------------------------------- | --------------------------- | ------------------------------- |
| [`AMQP`](#AMQP) | [`Redis`](#Redis) | [`MySQL`](#MySQL) |
| [`MQTT`](#MQTT) | [`NATS`](#NATS) | [`Apache Kafka`](#apache-kafka) |
| [`Elasticsearch`](#Elasticsearch) | [`PostgreSQL`](#PostgreSQL) | [`Webhooks`](#webhooks) |
| [`NSQ`](#NSQ) | | |
## Prerequisites
* Install and configure MinIO Server from [here](https://docs.min.io/docs/minio-quickstart-guide).
* Install and configure MinIO Client from [here](https://docs.min.io/docs/minio-client-quickstart-guide).
- Install and configure MinIO Server from [here](https://docs.min.io/docs/minio-quickstart-guide).
- Install and configure MinIO Client from [here](https://docs.min.io/docs/minio-client-quickstart-guide).
<a name="AMQP"></a>
## Publish MinIO events via AMQP
Install RabbitMQ from [here](https://www.rabbitmq.com/).
@ -33,20 +34,20 @@ Install RabbitMQ from [here](https://www.rabbitmq.com/).
The MinIO server configuration file is stored on the backend in json format. The AMQP configuration is located in the `amqp` key under the `notify` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `url` | _string_ | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `exchange` | _string_ | Name of the exchange. |
| `routingKey` | _string_ | Routing key for publishing. |
| `exchangeType` | _string_ | Kind of exchange. |
| `deliveryMode` | _uint8_ | Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `mandatory` | _bool_ | Publishing related bool. |
| `immediate` | _bool_ | Publishing related bool. |
| `durable` | _bool_ | Exchange declaration related bool. |
| `internal` | _bool_ | Exchange declaration related bool. |
| `noWait` | _bool_ | Exchange declaration related bool. |
| `autoDeleted` | _bool_ | Exchange declaration related bool. |
| Parameter | Type | Description |
| :------------- | :------- | :------------------------------------------------------------------------------ |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `url` | _string_ | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `exchange` | _string_ | Name of the exchange. |
| `routingKey` | _string_ | Routing key for publishing. |
| `exchangeType` | _string_ | Kind of exchange. |
| `deliveryMode` | _uint8_ | Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `mandatory` | _bool_ | Publishing related bool. |
| `immediate` | _bool_ | Publishing related bool. |
| `durable` | _bool_ | Exchange declaration related bool. |
| `internal` | _bool_ | Exchange declaration related bool. |
| `noWait` | _bool_ | Exchange declaration related bool. |
| `autoDeleted` | _bool_ | Exchange declaration related bool. |
An example configuration for RabbitMQ is shown below:
@ -68,22 +69,26 @@ An example configuration for RabbitMQ is shown below:
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the AMQP configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
After updating the AMQP configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange.
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:amqp``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:amqp`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
@ -94,7 +99,7 @@ arn:minio:sqs::1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.j
### Step 3: Test on RabbitMQ
The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this.
The python program below waits on the queue exchange `bucketevents` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this.
```py
#!/usr/bin/env python
@ -131,7 +136,7 @@ Execute this example python program to watch for RabbitMQ events on the console.
python rabbit.py
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
@ -145,6 +150,7 @@ python rabbit.py
```
<a name="MQTT"></a>
## Publish MinIO events MQTT
Install an MQTT Broker from [here](https://mosquitto.org/).
@ -153,17 +159,16 @@ Install an MQTT Broker from [here](https://mosquitto.org/).
The MinIO server configuration file is stored on the backend in json format. The MQTT configuration is located in the `mqtt` key under the `notify` top-level key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `broker` | _string_ | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `topic` | _string_ | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `qos` | _int_ | Set the Quality of Service Level |
| `username` | _string_ | Username to connect to the MQTT server (if required) |
| `password` | _string_ | Password to connect to the MQTT server (if required) |
| `queueDir` | _string_ | Persistent store for events when MQTT broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
| Parameter | Type | Description |
| :----------- | :------- | :------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `broker` | _string_ | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `topic` | _string_ | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `qos` | _int_ | Set the Quality of Service Level |
| `username` | _string_ | Username to connect to the MQTT server (if required) |
| `password` | _string_ | Password to connect to the MQTT server (if required) |
| `queueDir` | _string_ | Persistent store for events when MQTT broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
An example configuration for MQTT is shown below:
@ -181,25 +186,28 @@ An example configuration for MQTT is shown below:
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the MQTT configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using ``tcp://``, ``tls://``, or ``ws://`` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:mqtt``.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:mqtt`.
```
mc mb myminio/images
@ -210,7 +218,7 @@ arn:minio:sqs::1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.j
### Step 3: Test on MQTT
The python program below waits on mqtt topic ``/minio`` and prints event notifications on the console. We use [paho-mqtt](https://pypi.python.org/pypi/paho-mqtt/) library to do this.
The python program below waits on mqtt topic `/minio` and prints event notifications on the console. We use [paho-mqtt](https://pypi.python.org/pypi/paho-mqtt/) library to do this.
```py
#!/usr/bin/env python3
@ -243,7 +251,7 @@ Execute this example python program to watch for MQTT events on the console.
python mqtt.py
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
@ -257,6 +265,7 @@ python mqtt.py
```
<a name="Elasticsearch"></a>
## Publish MinIO events via Elasticsearch
Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) server.
@ -277,12 +286,12 @@ MinIO requires a 5.x series version of Elasticsearch. This is the latest major r
The MinIO server configuration file is stored on the backend in json format. The Elasticsearch configuration is located in the `elasticsearch` key under the `notify` top-level key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `url` | _string_ | (Required) The Elasticsearch server's address, with optional authentication info. For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`. |
| `index` | _string_ | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
| Parameter | Type | Description |
| :-------- | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `url` | _string_ | (Required) The Elasticsearch server's address, with optional authentication info. For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`. |
| `index` | _string_ | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
An example of Elasticsearch configuration is as follows:
@ -300,10 +309,13 @@ An example of Elasticsearch configuration is as follows:
If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the Elasticsearch configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
After updating the Elasticsearch configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
@ -327,13 +339,13 @@ arn:minio:sqs::1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suf
### Step 4: Test on Elasticsearch
Upload a JPEG image into ``images`` bucket.
Upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
```
Use curl to view contents of ``minio_events`` index.
Use curl to view contents of `minio_events` index.
```
$ curl "http://localhost:9200/minio_events/_search?pretty=true"
@ -408,6 +420,7 @@ This output shows that a document has been created for the event in Elasticsearc
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
<a name="Redis"></a>
## Publish MinIO events via Redis
Install [Redis](http://redis.io/download) server. For illustrative purposes, we have set the database password as "yoursecret".
@ -424,13 +437,13 @@ The steps below show how to use this notification target in `namespace` and `acc
The MinIO server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `address` | _string_ | (Required) The Redis server's address. For example: `localhost:6379`. |
| `password` | _string_ | (Optional) The Redis server's password. |
| `key` | _string_ | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format.|
| Parameter | Type | Description |
| :--------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `address` | _string_ | (Required) The Redis server's address. For example: `localhost:6379`. |
| `password` | _string_ | (Optional) The Redis server's password. |
| `key` | _string_ | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format. |
An example of Redis configuration is as follows:
@ -445,11 +458,15 @@ An example of Redis configuration is as follows:
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the Redis configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
After updating the Redis configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
@ -481,7 +498,7 @@ redis-cli -a yoursecret
OK
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
@ -501,13 +518,14 @@ Here we see that MinIO performed `HSET` on `minio_events` key.
In case, `access` format was used, then `minio_events` would be a list, and the MinIO server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
<a name="NATS"></a>
## Publish MinIO events via NATS
Install NATS from [here](http://nats.io/).
### Step 1: Add NATS endpoint to MinIO
The NATS configuration block in ``config.json`` is as follows:
The NATS configuration block in `config.json` is as follows:
```
"nats": {
@ -530,14 +548,19 @@ The NATS configuration block in ``config.json`` is as follows:
}
},
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the NATS configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example.
After updating the NATS configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `Message/event persistence`, `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
```
@ -561,11 +584,12 @@ MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/st
}
},
```
Read more about sections `clusterID`, `clientID` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/go-nats-streaming#publisher-rate-limiting).
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:nats``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nats`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
@ -614,13 +638,13 @@ go run nats.go
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
```
The example ``nats.go`` program prints event notification to console.
The example `nats.go` program prints event notification to console.
```
go run nats.go
@ -664,19 +688,21 @@ go run nats.go
2017/07/07 11:47:40 Connected
2017/07/07 11:47:40 Subscribing to subject 'bucketevents'
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
```
The example ``nats.go`` program prints event notification to console.
The example `nats.go` program prints event notification to console.
```
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"MinIO (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
```
<a name="PostgreSQL"></a>
## Publish MinIO events via PostgreSQL
Install [PostgreSQL](https://www.postgresql.org/) database server. For illustrative purposes, we have set the "postgres" user password as `password` and created a database called `minio_events` to store the events.
@ -697,17 +723,17 @@ MinIO requires PostgreSQL version 9.5 or above. MinIO uses the [`INSERT ON CONFL
The MinIO server configuration file is stored on the backend in json format. The PostgreSQL configuration is located in the `postgresql` key under the `notify` top-level key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| Parameter | Type | Description |
| :----------------- | :------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `connectionString` | _string_ | (Optional) [Connection string parameters](https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters) for the PostgreSQL server. Can be used to set `sslmode` for example. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `host` | _string_ | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`|
| `port` | _string_ | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `user` | _string_ | (Optional) Database user name. Defaults to user running the server process. |
| `password` | _string_ | (Optional) Database password. |
| `database` | _string_ | (Optional) Database name. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | _string_ | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`. IPv6 host should be enclosed with `[` and `]` |
| `port` | _string_ | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `user` | _string_ | (Optional) Database user name. Defaults to user running the server process. |
| `password` | _string_ | (Optional) Database password. |
| `database` | _string_ | (Optional) Database name. |
An example of PostgreSQL configuration is as follows:
@ -729,15 +755,18 @@ An example of PostgreSQL configuration is as follows:
Note that for illustration here, we have disabled SSL. In the interest of security, for production this is not recommended.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the Postgres configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
After updating the Postgres configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
### Step 3: Enable bucket notification using MinIO client
@ -760,7 +789,7 @@ arn:minio:sqs::1:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix
### Step 4: Test on PostgreSQL
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
@ -779,6 +808,7 @@ key | value
```
<a name="MySQL"></a>
## Publish MinIO events via MySQL
Install MySQL from [here](https://dev.mysql.com/downloads/mysql/). For illustrative purposes, we have set the root password as `password` and created a database called `miniodb` to store the events.
@ -799,17 +829,17 @@ MinIO requires MySQL version 5.7.8 or above. MinIO uses the [JSON](https://dev.m
The MinIO server configuration file is stored on the backend in json format. The MySQL configuration is located in the `mysql` key under the `notify` top-level key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| Parameter | Type | Description |
| :---------- | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `dsnString` | _string_ | (Optional) [Data-Source-Name connection string](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server. If not specified, the connection information specified by the `host`, `port`, `user`, `password` and `database` parameters are used. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `host` | _string_ | Host name of the MySQL server (used only if `dsnString` is empty). |
| `port` | _string_ | Port on which to connect to the MySQL server (used only if `dsnString` is empty). |
| `user` | _string_ | Database user-name (used only if `dsnString` is empty). |
| `password` | _string_ | Database password (used only if `dsnString` is empty). |
| `database` | _string_ | Database name (used only if `dsnString` is empty). |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | _string_ | Host name of the MySQL server (used only if `dsnString` is empty). |
| `port` | _string_ | Port on which to connect to the MySQL server (used only if `dsnString` is empty). |
| `user` | _string_ | Database user-name (used only if `dsnString` is empty). |
| `password` | _string_ | Database password (used only if `dsnString` is empty). |
| `database` | _string_ | Database name (used only if `dsnString` is empty). |
An example of MySQL configuration is as follows:
@ -827,16 +857,20 @@ An example of MySQL configuration is as follows:
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the MySQL configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
After updating the MySQL configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MySQL instance and an object of per-server configuration parameters.
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MySQL instance and an object of per-server configuration parameters.
### Step 3: Enable bucket notification using MinIO client
@ -858,7 +892,7 @@ arn:minio:sqs::1:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix
### Step 4: Test on MySQL
Open another terminal and upload a JPEG image into ``images`` bucket:
Open another terminal and upload a JPEG image into `images` bucket:
```
mc cp myphoto.jpg myminio/images
@ -879,6 +913,7 @@ mysql> select * from minio_images;
```
<a name="apache-kafka"></a>
## Publish MinIO events via Kafka
Install Apache Kafka from [here](http://kafka.apache.org/).
@ -889,7 +924,7 @@ MinIO requires Kafka version 0.10 or 0.9. Internally MinIO uses the [Shopify/sar
### Step 2: Add Kafka endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the kafka configuration block in ``config.json`` as follows:
The MinIO server configuration file is stored on the backend in json format. Update the kafka configuration block in `config.json` as follows:
```
"kafka": {
@ -900,17 +935,22 @@ The MinIO server configuration file is stored on the backend in json format. Upd
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the Kafka configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.``bucketevents`` is the topic used by kafka in this example.
After updating the Kafka configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
### Step 3: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:kafka``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:kafka`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
@ -927,13 +967,13 @@ We used [kafkacat](https://github.com/edenhill/kafkacat) to print all notificati
kafkacat -C -b localhost:9092 -t bucketevents
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp myphoto.jpg myminio/images
```
``kafkacat`` prints the event notification to the console.
`kafkacat` prints the event notification to the console.
```
kafkacat -b localhost:9092 -t bucketevents
@ -941,13 +981,14 @@ kafkacat -b localhost:9092 -t bucketevents
```
<a name="webhooks"></a>
## Publish MinIO events via Webhooks
[Webhooks](https://en.wikipedia.org/wiki/Webhook) are a way to receive information when it happens, rather than continually polling for that data.
### Step 1: Add Webhook endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the Webhook configuration block in ``config.json`` as follows
The MinIO server configuration file is stored on the backend in json format. Update the Webhook configuration block in `config.json` as follows
```
"webhook": {
@ -956,17 +997,22 @@ The MinIO server configuration file is stored on the backend in json format. Upd
"endpoint": "http://localhost:3000/"
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the webhook configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Here the endpoint is the server listening for webhook notifications. Save the file and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:webhook``. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:webhook`. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
mc mb myminio/images
@ -995,18 +1041,19 @@ git clone https://github.com/minio/thumbnailer/
npm install
```
Then open the Thumbnailer config file at ``config/webhook.json`` and add the configuration for your MinIO server and then start Thumbnailer by
Then open the Thumbnailer config file at `config/webhook.json` and add the configuration for your MinIO server and then start Thumbnailer by
```
NODE_ENV=webhook node thumbnail-webhook.js
```
Thumbnailer starts running at ``http://localhost:3000/``. Next, configure the MinIO server to send notifications to this URL (as mentioned in step 1) and use ``mc`` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to MinIO server by
Thumbnailer starts running at `http://localhost:3000/`. Next, configure the MinIO server to send notifications to this URL (as mentioned in step 1) and use `mc` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to MinIO server by
```
mc cp ~/images.jpg myminio/images
.../images.jpg: 8.31 KB / 8.31 KB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 59.42 KB/s 0s
```
Wait a few moments, then check the bucket’s contents with mc ls — you will see a thumbnail appear.
```
@ -1015,6 +1062,7 @@ mc ls myminio/images-thumbnail
```
<a name="NSQ"></a>
## Publish MinIO events to NSQ
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
@ -1043,21 +1091,24 @@ An example configuration for NSQ is shown below:
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the NSQ configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:nsq``.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nsq`.
```
mc mb myminio/images
@ -1074,7 +1125,7 @@ The simplest test is to download `nsq_tail` from [nsq github](https://github.com
./nsq_tail -nsqd-tcp-address 127.0.0.1:4150 -topic minio
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
mc cp gopher.jpg myminio/images
@ -1086,5 +1137,4 @@ You should receive the following event notification via NSQ once the upload comp
{"EventName":"s3:ObjectCreated:Put","Key":"images/gopher.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2018-10-31T09:31:11Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"requestParameters":{"sourceIPAddress":"10.1.1.1"},"responseElements":{"x-amz-request-id":"1562A792DAA53426","x-minio-origin-endpoint":"http://10.0.3.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"arn":"arn:aws:s3:::images"},"object":{"key":"gopher.jpg","size":162023,"eTag":"5337769ffa594e742408ad3f30713cd7","contentType":"image/jpeg","userMetadata":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"1562A792DAA53426"}},"source":{"host":"","port":"","userAgent":"MinIO (linux; amd64) minio-go/v6.0.8 mc/DEVELOPMENT.GOGET"}}]}
```
*NOTE* If you are running [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration.
_NOTE_ If you are running [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide), modify `~/.minio/config.json` on all the nodes with your bucket event notification backend configuration.

18
pkg/event/target/postgresql.go
View file

@ -80,15 +80,15 @@ const (
// PostgreSQLArgs - PostgreSQL target arguments.
type PostgreSQLArgs struct {
Enable bool `json:"enable"`
Format string `json:"format"`
ConnectionString string `json:"connectionString"`
Table string `json:"table"`
Host xnet.URL `json:"host"` // default: localhost
Port string `json:"port"` // default: 5432
User string `json:"user"` // default: user running minio
Password string `json:"password"` // default: no password
Database string `json:"database"` // default: same as user
Enable bool `json:"enable"`
Format string `json:"format"`
ConnectionString string `json:"connectionString"`
Table string `json:"table"`
Host xnet.Host `json:"host"` // default: localhost
Port string `json:"port"` // default: 5432
User string `json:"user"` // default: user running minio
Password string `json:"password"` // default: no password
Database string `json:"database"` // default: same as user
}
// Validate PostgreSQLArgs fields

31
pkg/net/host.go
View file

@ -118,7 +118,6 @@ func ParseHost(s string) (*Host, error) {
if !strings.Contains(err.Error(), "missing port in address") {
return nil, err
}
host = s
portStr = ""
} else {
@ -129,11 +128,23 @@ func ParseHost(s string) (*Host, error) {
isPortSet = true
}
if i := strings.Index(host, "%"); i > -1 {
host = host[:i]
if host != "" {
host, err = trimIPv6(host)
if err != nil {
return nil, err
}
}
if !isValidHost(host) {
// IPv6 requires a link-local address on every network interface.
// `%interface` should be preserved.
trimmedHost := host
if i := strings.LastIndex(trimmedHost, "%"); i > -1 {
// `%interface` can be skipped for validity check though.
trimmedHost = trimmedHost[:i]
}
if !isValidHost(trimmedHost) {
return nil, errors.New("invalid hostname")
}
@ -143,3 +154,15 @@ func ParseHost(s string) (*Host, error) {
IsPortSet: isPortSet,
}, nil
}
// IPv6 can be embedded with square brackets.
func trimIPv6(host string) (string, error) {
// `missing ']' in host` error is already handled in `SplitHostPort`
if host[len(host)-1] == ']' {
if host[0] != '[' {
return "", errors.New("missing '[' in host")
}
return host[1:][:len(host)-2], nil
}
return host, nil
}

35
pkg/net/host_test.go
View file

@ -138,6 +138,13 @@ func TestHostUnmarshalJSON(t *testing.T) {
{[]byte(`"play-minio-io"`), &Host{"play-minio-io", 0, false}, false},
{[]byte(`"play--min.io"`), &Host{"play--min.io", 0, false}, false},
{[]byte(`":9000"`), nil, true},
{[]byte(`"[fe80::8097:76eb:b397:e067%wlp2s0]"`), &Host{"fe80::8097:76eb:b397:e067%wlp2s0", 0, false}, false},
{[]byte(`"[fe80::8097:76eb:b397:e067]:9000"`), &Host{"fe80::8097:76eb:b397:e067", 9000, true}, false},
{[]byte(`"fe80::8097:76eb:b397:e067%wlp2s0"`), nil, true},
{[]byte(`"fe80::8097:76eb:b397:e067%wlp2s0]"`), nil, true},
{[]byte(`"[fe80::8097:76eb:b397:e067%wlp2s0"`), nil, true},
{[]byte(`"[[fe80::8097:76eb:b397:e067%wlp2s0]]"`), nil, true},
{[]byte(`"[[fe80::8097:76eb:b397:e067%wlp2s0"`), nil, true},
{[]byte(`"play:"`), nil, true},
{[]byte(`"play::"`), nil, true},
{[]byte(`"play:90000"`), nil, true},
@ -207,3 +214,31 @@ func TestParseHost(t *testing.T) {
}
}
}
func TestTrimIPv6(t *testing.T) {
testCases := []struct {
IP string
expectedIP string
expectErr bool
}{
{"[fe80::8097:76eb:b397:e067%wlp2s0]", "fe80::8097:76eb:b397:e067%wlp2s0", false},
{"fe80::8097:76eb:b397:e067%wlp2s0]", "fe80::8097:76eb:b397:e067%wlp2s0", true},
{"[fe80::8097:76eb:b397:e067%wlp2s0]]", "fe80::8097:76eb:b397:e067%wlp2s0]", false},
{"[[fe80::8097:76eb:b397:e067%wlp2s0]]", "[fe80::8097:76eb:b397:e067%wlp2s0]", false},
}
for i, testCase := range testCases {
ip, err := trimIPv6(testCase.IP)
expectErr := (err != nil)
if expectErr != testCase.expectErr {
t.Fatalf("test %v: error: expected: %v, got: %v", i+1, testCase.expectErr, expectErr)
}
if !testCase.expectErr {
if ip != testCase.expectedIP {
t.Fatalf("test %v: IP: expected: %#v, got: %#v", i+1, testCase.expectedIP, ip)
}
}
}
}