Commit graph

51 commits

Author SHA1 Message Date
Harshavardhana 1027afa853
Indicate backend encrypted only if encryption is requested (#8508) 2019-11-11 18:42:10 -08:00
Harshavardhana aa04f97f95 Config migration should handle plain-text (#8506)
This PR fixes issues found in config migration

 - StorageClass migration error when rrs is empty
 - Plain-text migration of older config
 - Do not run in safe mode with incorrect credentials
 - Update logger_http documentation for _STATE env

Refer more reported issues at #8434
2019-11-11 12:01:21 -08:00
Harshavardhana ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
poornas d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Harshavardhana bd10640846 Fix logger and audit http endpoint config lookup 2019-10-11 18:33:25 -07:00
Harshavardhana 6a4ef2e48e Initialize configs correctly, move notification config (#8367)
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Harshavardhana 290ad0996f Move etcd, logger, crypto into their own packages (#8366)
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
Harshavardhana 589e32a4ed Refactor config and split them in packages (#8351)
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
2019-10-04 23:05:33 +05:30
poornas 2e02e1889b Cleanup ResponseWriter function for audit and trace (#8283) 2019-09-24 02:04:28 +05:30
Krishnan Parthasarathi bbb56739bd Add User-Agent header with MinIO release details in http logs (#7843)
This would allow http log target server to distinguish between log
messages across different versions of MinIO deployments.
2019-08-14 11:43:43 -07:00
Harshavardhana 8ce424bacd Enhance audit logging to capture responseTimes (#8067)
Audit logging requires to have

- timeToFirstByte
- timeToResponse

timing information
2019-08-12 20:32:34 -07:00
Krishnan Parthasarathi fbfc9a61ec Add node address information to logs (#7941) 2019-07-18 09:58:37 -07:00
Harshavardhana c43f745449
Ensure that we use constants everywhere (#7845)
This allows for canonicalization of the strings
throughout our code and provides a common space
for all these constants to reside.

This list is rather non-exhaustive but captures
all the headers used in AWS S3 API operations
2019-07-02 22:34:32 -07:00
kannappanr 70b350c383
Remove DeploymentID from response headers (#7815)
Response headers need not contain deployment ID.
2019-07-01 12:22:01 -07:00
Krishnan Parthasarathi 74efbb4153 Add deploymentID to web handler logs (#7712) 2019-06-03 15:40:04 -07:00
Harshavardhana 2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
kannappanr a436f2baa5 Change order of trace source in error log (#7599)
Change the order of trace source that gets
printed on the console.
2019-04-29 14:56:30 -07:00
kannappanr 5ecac91a55
Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
kannappanr 7154b8a568
Error log: Correct error type in anonymous mode (#7414)
Currently message is set to error type value.
Message field is not used in error logs. it is used only in the case of info logs.

This PR sets error message field to store error type correctly.
2019-03-25 13:40:08 -07:00
Harshavardhana df35d7db9d Introduce staticcheck for stricter builds (#7035) 2019-02-13 18:29:36 +05:30
Harshavardhana 817269475f Make sure to drain body upon an error (#7197)
Also cleanup redundant code and use it at a common place
2019-02-06 12:07:03 -08:00
kannappanr 8ee8ad777c
logger: do not interpret encoded url as format string (#7110)
Error logger currently interprets encoded url in the error as a format string.
2019-01-24 00:30:00 -08:00
Anis Elleuch 99b843a64e Add anonymous flag to prevent logging sensitive information (#6899) 2018-12-18 16:08:11 -08:00
Harshavardhana 9e3fce441e Audit log claims from token (#6847) 2018-11-22 09:33:24 +05:30
Harshavardhana bfb505aa8e Refactor logging in more Go idiomatic style (#6816)
This refactor brings a change which allows
targets to be added in a cleaner way and also
audit is now moved out.

This PR also simplifies logger dependency for auditing
2018-11-19 14:47:03 -08:00
Harshavardhana bef0318c36 Support audit logs with additional fields (#6738)
This PR adds support

- Request query params
- Request headers
- Response headers

AuditLogEntry is exported and versioned as well
starting with this PR.
2018-11-02 18:40:08 -07:00
kannappanr 6869f6d9dd
Remove unwanted logs (#6708) 2018-10-26 14:41:25 -07:00
Harshavardhana b0c9ae7490 Add audit logging for S3 and Web handlers (#6571)
This PR brings an additional logger implementation
called AuditLog which logs to http targets

The intention is to use AuditLog to log all incoming
requests, this is used as a mechanism by external log
collection entities for processing Minio requests.
2018-10-12 12:25:59 -07:00
Harshavardhana a63bc9254d Add 'disk' tag to log output to enhance 'disk not found' errors (#6460) 2018-09-13 21:42:50 -07:00
Anis Elleuch 3099af70a3 Add admin get/set config keys API (#6113)
This PR adds two new admin APIs in Minio server and madmin package:
- GetConfigKeys(keys []string) ([]byte, error)
- SetConfigKeys(params map[string]string) (err error)

A key is a path in Minio configuration file, (e.g. notify.webhook.1)

The user will always send a string value when setting it in the config file,
the API will know how to convert the value to the appropriate type. The user
is also able to set a raw json.

Before setting a new config, Minio will validate all fields and try to connect
to notification targets if available.
2018-09-06 20:33:18 +05:30
Harshavardhana fd1b8491db
Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
kannappanr 0286e61aee Log disk not found error just once (#6059)
Modified the LogIf function to log only if the error passed
is not on the ignored errors list.

Currently, only disk not found error is added to the list.
Added a new function in logger package called LogAlwaysIf, 
which will print on any error.

Fixes #5997
2018-08-14 13:58:48 -07:00
Oleg Kovalov 37de2dbd3b simplifying if-else chains to switches (#6208) 2018-08-06 10:26:40 -07:00
Harshavardhana a82500f162 Support dumb terminals by turning off color (#6246)
ANSI colors do not work on dumb terminals, in situations
when minio is running as a service under systemd.

This PR ensures we turn off color in those situations.
2018-08-06 18:16:49 +05:30
Harshavardhana 36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
Anis Elleuch 9c5e971a58 Add new console/http loggers (#6066)
- Add console target logging, enabled by default.
- Add http target logging, which supports an endpoint
  with basic authentication (username/password are passed
  in the endpoint url itself)
- HTTP target logging is asynchronous and some logs can be
  dropped if channel buffer (10000) is full
2018-07-19 15:55:06 -07:00
Anis Elleuch b1c9eb0e01 Disable splitting lines in pretty error messages (#6171)
In a small window, UI error tries to split lines for an eye candy
error message. However, since we show some docs.minio.io links in some
error messages, these links are actually broken and not easily selected
in a X terminal. This PR changes the behavior and won't split lines
anymore.
2018-07-19 15:49:02 -07:00
kannappanr 43cc0096fa
Add support for deployment ID (#6144)
deployment ID helps in identifying a minio deployment in the case of remote
logging targets.
2018-07-18 20:17:35 -07:00
Andreas Auernhammer cd152f404a replace os.Exit with panic for logger.CriticalIf (#6065)
This commit prevents complete server failures caused by
`logger.CriticalIf` calls. Instead of calling `os.Exit(1)`
the function now executes a panic with a special value
indicating that a critical error happend. At the top HTTP
handler layer panics are recovered and if its a critical
error the client gets an InternalServerError status code.

Further this allows unit tests to cover critical-error code
paths.
2018-06-25 13:51:49 -07:00
kannappanr 577d10674d
Cleanup stack trace in error logs (#6045)
Add compile time GOROOT path to the list of prefix
of file paths to be removed.

Add webhandler function names to the slice that
stores function names to terminate logging.
2018-06-18 12:04:46 -07:00
Krishna Srinivas cc8178cdc4 Log errors only once for event notification errors (#5905) 2018-05-09 15:59:45 -07:00
Anis Elleuch 32700fca52 Enhance fatal errors printing of common issues seen by users (#5878) 2018-05-08 19:04:36 -07:00
Krishna Srinivas 11b8e292a7 Disable logging during unit tests (#5861) 2018-04-27 15:03:19 -07:00
ebozduman f16bfda2f2 Remove panic() and handle it appropriately (#5807)
This is an effort to remove panic from the source. 
Add a new call called CriticialIf, that calls LogIf and exits. 
Replace panics with one of CriticalIf, FatalIf and a return of error.
2018-04-19 17:24:43 -07:00
Harshavardhana c332b4fb5e fix shadowing issue in logger (#5830) 2018-04-16 13:36:45 -07:00
Harshavardhana adf9a9d300 Remove all unused variables and functions (#5823) 2018-04-15 19:26:04 +05:30
kannappanr 57a3d9c16c
Modify fatalIf, startup and update message logging code (#5780)
Use a common logging framework to log fatalIf, startup, Info and Update
messages.
2018-04-10 09:37:14 -07:00
kannappanr f8a3fd0c2a
Create logger package and rename errorIf to LogIf (#5678)
Removing message from error logging
Replace errors.Trace with LogIf
2018-04-05 15:04:40 -07:00
Krishna Srinivas 9ede179a21 Use context.Background() instead of nil
Rename Context[Get|Set] -> [Get|Set]Context
2018-03-15 16:28:25 -07:00
Ersan 33fe42df8a Adds context support for nas, oss and s3 2018-03-15 16:28:25 -07:00