* Initial version of CCR monitoring UI
* Adding missing files
* Use icons
* Use new column header text
* Update tests
* Basic of shard detail page
* Do these in parallel
* Disable time picker on ccr page
* Remove summary for now
* Remove unnecessary code here
* Fix a few things on the shard page
* Only send down what we need
* update snapshot
* Handle no ccr_stats documents
* Ensure we fetch the latest
* Updates
* Format the time
* Add api integration tests
* Adding pagination and sorting
* Updated query logic
* Change this back
* Add specific information about the follower and leader lag ops
* Update tests
* UI updates
* Address PR issues
* Fix tests
* Update shapshots
* Add timestamp
* Update tests
* Add a few snapshot tests
* Use timezone formatter
* Fix tests
* Fix aligment of shard table
* PR feedback
* Update snapshots
* Update snapshot
With Metricbeat shipping Elasticsearch monitoring data (instead of internal collection by Elasticsearch), there are some subtle changes to the format of monitoring docs that are indexed into `.monitoring-es-6-*`. One such change is that metricbeat won't index fields with `null` values; instead it simply doesn't index such fields at all.
As a result, in the context of Elasticsearch monitoring docs, when it comes to docs with `type` = `shards` representing unassigned shards, the `shard.node` field was being indexed as `null` by internal Elasticearch collection, whereas the field was absent when the doc was indexed by Metricbeat.
Since both cases represent the same case — the shard being unassigned — this PR relaxes the check in the UI code to look for either case.
### Sample `shards` document indexed by internal ES collection
```js
{
"_index":".monitoring-es-6-2018.10.02",
"_type":"doc",
"_id":"WUf_htOeSXOJQmiesyF5Bw:_na:metricbeat-7.0.0-alpha1-2018.10.01:0:r",
"_source":{
"cluster_uuid":"zXO1GjA6SJGsrPnCPkOoyA",
"timestamp":"2018-10-02T03:54:43.364Z",
"interval_ms":10000,
"type":"shards",
"source_node":null,
"state_uuid":"WUf_htOeSXOJQmiesyF5Bw",
"shard":{
"state":"UNASSIGNED",
"primary":false,
"node":null,
"relocating_node":null,
"shard":0,
"index":"metricbeat-7.0.0-alpha1-2018.10.01"
}
}
}
```
### Sample `shards` document indexed by Metricbeat collection
```js
{
"_index":".monitoring-es-6-mb-2018.10.02",
"_type":"doc",
"_id":"FhDRTPjkQJqsgawYbxjQzw:_na:metricbeat-7.0.0-alpha1-2018.10.01:0:r",
"_source":{
"@timestamp":"2018-10-02T04:00:03.361Z",
"interval_ms":10000,
"shard":{
"state":"UNASSIGNED",
"primary":false,
"index":"metricbeat-7.0.0-alpha1-2018.10.01",
"shard":0
},
"state_uuid":"FhDRTPjkQJqsgawYbxjQzw",
"beat":{
"hostname":"Shaunaks-MBP-2",
"version":"7.0.0-alpha1",
"name":"Shaunaks-MBP-2"
},
"timestamp":"2018-10-02T04:00:03.375Z",
"type":"shards",
"metricset":{
"name":"shard",
"module":"elasticsearch",
"host":"localhost:9200",
"rtt":14254,
"namespace":"elasticsearch.shard"
},
"host":{
"name":"Shaunaks-MBP-2"
},
"cluster_uuid":"zXO1GjA6SJGsrPnCPkOoyA"
}
}
```
* Add basic support for new K7 navigation
* Make visibility and app title work
* Allow nav controls on right side of navbar
* Use render callback w/ el
* Add support for multiple sides
* Remove fake spaces nav control
* Breadcrumb support
* Hide breadcrumbs in plugins when k7design is enabled:
* Fix units
* Rename k7 -> header
* Add tests
* Fix tests
* Fix loading indicator
* PR comments
* Move ts-ignore
* Use canvasApp icon type
### Review notes
This is generally ready for review. We are awaiting https://github.com/elastic/elasticsearch/issues/32777 to improve handling when users do not have any access to Kibana, but this should not hold up the overall review for this PR.
This PR is massive, there's no denying that. Here's what to focus on:
1) `x-pack/plugins/spaces`: This is, well, the Spaces plugin. Everything in here is brand new. The server code is arguably more important, but feel free to review whatever you see fit.
2) `x-pack/plugins/security`: There are large and significant changes here to allow Spaces to be securable. To save a bit of time, you are free to ignore changes in `x-pack/plugins/security/public`: These are the UI changes for the role management screen, which were previously reviewed by both us and the design team.
3) `x-pack/test/saved_object_api_integration` and `x-pack/test/spaces_api_integration`: These are the API test suites which verify functionality for:
a) Both security and spaces enabled
b) Only security enabled
c) Only spaces enabled
What to ignore:
1) As mentioned above, you are free to ignore changes in `x-pack/plugins/security/public`
2) Changes to `kibana/src/server/*`: These changes are part of a [different PR that we're targeting against master](https://github.com/elastic/kibana/pull/23378) for easier review.
## Saved Objects Client Extensions
A bulk of the changes to the saved objects service are in the namespaces PR, but we have a couple of important changes included here.
### Priority Queue for wrappers
We have implemented a priority queue which allows plugins to specify the order in which their SOC wrapper should be applied: `kibana/src/server/saved_objects/service/lib/priority_collection.ts`. We are leveraging this to ensure that both the security SOC wrapper and the spaces SOC wrapper are applied in the correct order (more details below).
### Spaces SOC Wrapper
This wrapper is very simple, and it is only responsible for two things:
1) Prevent users from interacting with any `space` objects (use the Spaces client instead, described below)
2) Provide a `namespace` to the underlying Saved Objects Client, and ensure that no other wrappers/callers have provided a namespace. In order to accomplish this, the Spaces wrapper uses the priority queue to ensure that it is the last wrapper invoked before calling the underlying client.
### Security SOC Wrapper
This wrapper is responsible for performing authorization checks. It uses the priority queue to ensure that it is the first wrapper invoked. To say another way, if the authorization checks fail, then no other wrappers will be called, and the base client will not be called either. This wrapper authorizes users in one of two ways: RBAC or Legacy. More details on this are below.
### Examples:
`GET /s/marketing/api/saved_objects/index-pattern/foo`
**When both Security and Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object at this space.
3) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
4) The underlying client/repository are invoked to retrieve the object from ES.
**When only Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
3) The underlying client/repository are invoked to retrieve the object from ES.
**When only Security is enabled:**
(assume `/s/marketing` is no longer part of the request)
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object globally.
3) The underlying client/repository are invoked to retrieve the object from ES.
## Authorization
Authorization changes for this project are centered around Saved Objects, and builds on the work introduced in RBAC Phase 1.
### Saved objects client
#### Security without spaces
When security is enabled, but spaces is disabled, then the authorization model behaves the same way as before: If the user is taking advantage of Kibana Privileges, then we check their privileges "globally" before proceeding. A "global" privilege check specifies `resources: ['*']` when calling the [ES _has_privileges api.](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-has-privileges.html). Legacy users (non-rbac) will continue to use the underlying index privileges for authorization.
#### Security with spaces
When both plugins are enabled, then the authorization model becomes more fine-tuned. Rather than checking privileges globally, the privileges are checked against a specific resource that matches the user's active space. In order to accomplish this, the Security plugin needs to know if Spaces is enabled, and if so, it needs to ask Spaces for the user's active space. The subsequent call to the `ES _has_privileges api` would use `resources: ['space:marketing']` to verify that the user is authorized at the `marketing` space. Legacy users (non-rbac) will continue to use the underlying index privileges for authorization. **NOTE** The legacy behavior implies that those users will have access to all spaces. The read/write restrictions are still enforced, but there is no way to restrict access to a specific space for legacy auth users.
#### Spaces without security
No authorization performed. Everyone can access everything.
### Spaces client
Spaces, when enabled, prevents saved objects of type `space` from being CRUD'd via the Saved Objects Client. Instead, the only "approved" way to work with these objects is through the new Spaces client (`kibana/x-pack/plugins/spaces/lib/spaces_client.ts`).
When security is enabled, the Spaces client performs its own set of authorization checks before allowing the request to proceed. The Spaces client knows which authorization checks need to happen for a particular request, but it doesn't know _how_ to check privileges. To accomplish this, the spaces client will delegate the check security's authorization service.
#### FAQ: Why oh why can't you used the Saved Objects Client instead!?
That's a great question! We did this primarily to simplify the authorization model (at least for our initial release). Accessing regular saved objects follows a predictible authorization pattern (described above). Spaces themselves inform the authorization model, and this interplay would have greatly increased the complexity. We are brainstorming ideas to obselete the Spaces client in favor of using the Saved Objects Client everywhere, but that's certainly out of scope for this release.
## Test Coverage
### Saved Objects API
A bulk of the changes to enable spaces are centered around saved objects, so we have spent a majority of our time automating tests against the saved objects api.
**`x-pack/test/saved_object_api_integration/`** contains the test suites for the saved objects api. There is a `common/suites` subfolder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
3) Security only: `./security_only`
Each of these test configurations will start up ES/Kibana with the appropriate license and plugin set. Each set runs through the entire test suite described in `common/suites`. Each test with in each suite is run multiple times with different inputs, to test the various permutations of authentication, authorization type (legacy vs RBAC), space-level privileges, and the user's active space.
### Spaces API
Spaces provides an experimental public API.
**`x-pack/test/spaces_api_integration`** contains the test suites for the Spaces API. Similar to the Saved Objects API tests described above, there is a `common/suites` folder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
### Role Management UI
We did not provide any new functional UI tests for role management, but the existing suite was updated to accomidate the screen rewrite.
We do have a decent suite of jest unit tests for the various components that make up the new role management screen. They're nested within `kibana/x-pack/plugins/security/public/views/management/edit_role`
### Spaces Management UI
We did not provide any new functional UI tests for spaces management, but the components that make up the screens are well-tested, and can be found within `kibana/x-pack/plugins/spaces/public/views/management/edit_space`
### Spaces Functional UI Tests
There are a couple of UI tests that verify _basic_ functionality. They assert that a user can login, select a space, and then choose a different space once inside: `kibana/x-pack/test/functional/apps/spaces`
## Reference
Notable child PRs are listed below for easier digesting. Note that some of these PRs are built on other PRs, so the deltas in the links below may be outdated. Cross reference with this PR when in doubt.
### UI
- Reactify Role Management Screen: https://github.com/elastic/kibana/pull/19035
- Space Aware Privileges UI: https://github.com/elastic/kibana/pull/21049
- Space Selector (in Kibana Nav): https://github.com/elastic/kibana/pull/19497
- Recently viewed Widget: https://github.com/elastic/kibana/pull/22492
- Support Space rename/delete: https://github.com/elastic/kibana/pull/22586
### Saved Objects Client
- ~~Space Aware Saved Objects: https://github.com/elastic/kibana/pull/18862~~
- ~~Add Space ID to document id: https://github.com/elastic/kibana/pull/21372~~
- Saved object namespaces (supercedes #18862 and #21372): https://github.com/elastic/kibana/pull/22357
- Securing saved objects: https://github.com/elastic/kibana/pull/21995
- Dedicated Spaces client (w/ security): https://github.com/elastic/kibana/pull/21995
### Other
- Public Spaces API (experimental): https://github.com/elastic/kibana/pull/22501
- Telemetry: https://github.com/elastic/kibana/pull/20581
- Reporting: https://github.com/elastic/kibana/pull/21457
- Spencer's original Spaces work: https://github.com/elastic/kibana/pull/18664
- Expose `spaceId` to "Add Data" tutorials: https://github.com/elastic/kibana/pull/22760Closes#18948
"Release Note: Create spaces within Kibana to organize dashboards, visualizations, and other saved objects. Secure access to each space when X-Pack Security is enabled"
* Merge in boilerplate branch
* Manually copy over the specific metrics and UIs
* Add api integration tests
* Fix tests
* Remove unused metrics
* Update snapshot
* Fix tests
* Remove types agg
* Use ApmClusterMetric
* provide description for apm-server monitoring metrics (#23331)
* Vis LESS to SASS (cont.) (#23199)
* Tweak migrations integraiton tests to have a stable sort (#23265)
* Fix: plugin api route with security enabled (#23334)
Closes https://github.com/elastic/kibana/issues/23266
This is more of a quick fix than the final solution. The issue was that Canvas tries to check the plugins API without checking to see if the user it logged in. As a result, instead of the plugins response, it gets the HTML from the login page and that causes an error to be thrown when attempting to parse the results.
For now, this PR just disables the auth requirement on the Canvas plugin API endpoint.
* [migrations/tests] sort results before assertion (#23347)
There have been several failures in this test, seemingly caused by a lack of sorting in the results. It makes sense that since both migrations are run simultaneously that sometimes one would succeed and sometimes another would, so I've just sorted the results before checking.
cc: @chrisdavies
* [ML] Moves custom URL editor Add button and form to top of flyout (#23326)
* [ML] Moves custom URL editor Add button and form to top of flyout
* [ML] Edits to custom URL editor class name
* Graph LESS to SASS (#23348)
* Developer documentation for integrating with the telemetry service (#23295)
* Developer documentation for integrating with the telemetry service
* open with a bang
* more faqs
* thing about tracking ui interactions
* talk to the plat team
* create and register
* Fix a bug where ES sends a string and migrations expect a boolean (#23313)
* chore: use cheerio in i18n.html.getDirectiveMessages (#23342)
this was only using jsdom to parse html, but cheerio allows parsing html without requiring a dom. cheerio was also already in the dependency list.
* [core/utils] add shareWeakReplay() operator (#23333)
* Chore: fix canvas test runner (#23336)
Blocked by https://github.com/elastic/kibana/pull/23342
This fixes the local test runner in Canvas. It should not affect anything else, including the CI test runner.
- Bumps JSDOM to ^12.0.0
- I matched Kibana's version on migration, but nothing else in X-Pack uses JSDOM, so we can use the newer version (which has a very different API)
- I had to match it because of a script that enforces version matching, but #23342 removed jsdom from Kibana, so we no longer have a version to match
- Restores the local `.babelrc` file
- I thought it was only used for building plugins; I was wrong 😢
* Convert Discover open top nav to EUI flyout (#22971)
* move find logic to SavedObjectFinder component since savedObjectClient is no longer coupled to angular
* implement flyout open saved searches
* remove old open stuff
* add jest test for OpenSearchPanel and simplify panel title
* fix functional tests
* fix _lab_mode functional test
* Migrate save top nav in Discover and Visualize to EUI (#23190)
* extract reusable save component from DashboardSaveModal
* update discover search to use SavedObjectSaveModal
* create generic show_save_model that works for both discover and dashboard
* fix last bits of discover save
* remove old save functionallity
* migrate visualize save to EUI
* fix functional tests
* disable save button if title is empty
* mark title input as invalid when title is not provided
* fix funtional tests
* Moves styleSheetPath to uiExports (#23007)
This was previously defined in uiExports.app, which limited plugins which are not an app of providing a stylesheet. This allows any plugin to define a stylesheet which will be available on page load.
* Timelion less to sass (#23339)
* Consistent casing
* Fix snapshot
* Update tests
This was previously defined in uiExports.app, which limited plugins which are not an app of providing a stylesheet. This allows any plugin to define a stylesheet which will be available on page load.
This PR fixes the logic for logging the deprecation warning introduced with https://github.com/elastic/kibana/pull/22765. Previously Kibana would log the warning if the new `xpack.monitoring.cluster_alerts.email_notifications.email_address` setting was not defined, regardless of `xpack:defaultAdminEmail`'s setting.
Now, we will only log the deprecation warning if all of the following are true:
1) `xpack.monitoring.cluster_alerts.email_notifications.email_address` is not set.
2) `xpack:defaultAdminEmail` is set. (**<-- this is the new part**)
3) We haven't already logged the deprecation warning
* Move timezone settings into autoload file
* Remove applying setting from timelion
* Remove manual set from ML
* Remove manual set from monitoring
* Remove now obsolete code from embedding test plugin
* If the settings collector returns nothing, ensure the settings api still returns a null value for default_admin_email
* Update test
* Feedback from PR
* Retrofit the Bulk Uploader types combiner [ch2198]
fix usage collector, add comments to formatForBulk
remove unnecessary customizations
* override default format for bulk upload for usage type collectors
* rename to ignoreForInternalUploader
* collectors -> collectorSet
* use constant for kibana_stats type
* example of data formatting for bulk in function comment
* Add screen reader only text for monitoring charts
* Add aria label for completed recoveries
* Ensure we have aria label coverage for monitoring chart tooltips
* Ensure table rows are tabbable and include the number of results at the top
* Use EuiScreenReaderOnly
* Updated copy
* Use EUI
* Remove kui usage
* Use an id instead of aria-label
* Show results in the table footer
* Revert "Show results in the table footer"
This reverts commit d622eb7eb4.
* Show total row count within the monitoring table
* PR feedback
* PR feedback
* Ensure all charts show an interval in the tooltip
* Fix padding issue with the cluster status
* Convert info button and the rest of the page to EUI
* Update other overview pages
* Use Page directive
* Give the tooltip some more space
* Add page body to a couple places
* PR feedback
* Convert beats overview page to use EUI components
* More UI components for beats overview
* Set background color to white
* Update snapshots
* Add PageBody wrapper
* Update snapshots
* Use panels
* PR feedback
* Update snapshots
* set pause to 'true' when refresh interval is zero
* keep original assertion
* fix expect message in functional tests now that pause is set to true
* fix executorProvider mocha test
* Ensure we pass down the expiry_date for the license
* Update tests
* Update fixtues
* Update fixture
* Revert "Update fixture"
This reverts commit 0a1e7a5b29.
* Revert "Update fixtues"
This reverts commit b0c0e70200.
* Revert "Update tests"
This reverts commit 6a4421bb4e.
* Revert "Ensure we pass down the expiry_date for the license"
This reverts commit 2562395a78.
* Use expiry_date_in_millis value instead of expiry_date
This is a breaking change that removes the
`xpack.monitoring.node_resolver` setting. This setting was deprecated in
5.6, and in 6.1 the setting was limited to `uuid` explicitly. Beginning in
7.0, after this is merged, the setting will no longer exist.
* Replace notify.warning with toastNotifications in region map, vega, index_pattern, redirect_when_missing, graph, monitoring, and ML
* Link to index patterns from Graph toast.
* Delete RouteBasedNotifier.
* Remove courierNotifier and SearchTimeout and ShardFailure errors.
* Remove warning and custom notifier types.
* [Stats API] Set API field names per spec
* fix jest tests
* fix api integration test
* trash the original metrics collector
- constantly accumulating stats over time does not align with the existing behavior, which is to reset the stats to 0 whenever they are pulled
* move some logic out of the collector types combiner into inline
- change the signature of sourceKibana
* Make a new stats collector for the API
- to not clear the data when pulling via the api
- fetching is a read-only thing
* isolate data transforms for api data and upload data
* no static methods
* remove external in bytes
* remove the _stats prefix for kibana and reporting
* update jest test snapshot
* fix collector_types_combiner test
* fix usage api
* add test suite todo comment
* reduce some loc change
* roll back mysterious change
* reduce some more loc change
* comment correction
* reduce more loc change
* whitespace
* comment question
* fix cluster_uuid
* fix stats integration test
* fix bulk uploader test, combineTypes is no longer external
* very important comments about the current nature of stats represented and long-term goals
* add stats api tests with/without authentication
* fix more fields to match data model
* fix more tests
* fix jest test
* remove TODO
* remove sockets
* use snake_case for api field names
* restore accidental removal + copy/paste error
* sourceKibana -> getKibanaInfoForStats
* skip usage test on legacy endpoint
* fix api tests
* more comment
* stop putting a field in that used to be omitted
* fix the internal type to ID the usage data for bulk uploader
* correct the kibana usage type value, which is shown as-is in the API
* more fixes for the constants identifying collector types + test against duplicates
* add a comment on a hack, and a whitespace fix
* bump eui
* Fixed breaking `EuiPage` changes
Mainly adding `EuiPageBody`’s where there were none
* bump to 3.0, remove duplicate declaration of EuiFlyoutBody, update jest snapshots
* bump eui
* bump to 3.0, remove duplicate declaration of EuiFlyoutBody, update jest snapshots
* Update jest snapshots in xpack
* [Monitoring/Telemetry] Move Usage service from Monitoring to Kibana core
* fix tests
* fix reporting integration
* roll back more diffs
* roll logger into bulk uploader to remove file duplication
* fix xpack usage api
* subclass constructor is not needed
* collectorSet has factory methods for collector object creation
* fix reporting usage jest test
* Rename config view to PipelineViewer.
* Decouple CollapsibleStatement from if/else using props.children.
* Add tests for PipelineViewer component.
* Test Metric component.
* Test CollapsibleStatement component.
* Test PluginStatement component.
* Test Queue component.
* Test StatementListHeading component.
* Test StatementSection component. Move StatementList component to dedicated file.
* Test StatementList component.
* Test Statement component.
* Run prettier on edited files.
* Rename config view to PipelineViewer.
* Decouple CollapsibleStatement from if/else using props.children.
* Modify function to take params instead of single object.
* use uiSettingsClient instead of angular config service
* replace timefilter service with singleton
* remove all other uses of timefilter service, get timefilter singleton working
* remove Private from brush event
* globalState
* fix timefilter jest test
* clean up rebase artifacts
* lint errors
* another lint error
* fix broken functional test _shared_links
* fix broken mocha test doc.js
* fix service load order breaking status page
* remove Provider from change_time_filter
* another round of mocha test updates
* convert create_brush_handler test to jest
* fix kbnGlobalTimepicker toogle mocha test
* remove _root_search_source - removed in another PR and somehow I added it back during rebase
* better comments around Object.assign and proper js-docs format
* use so listerners get cleaned up when scope is destroyed
* listen to correct timefilter update event
* update APM to use timefilter singleton
* update angular scope when timefilter changes
* fix APM jest test
* add listenAndDigestAsync to listen.js
* fix apm breadcrumbs jest test
* move diffTime and diffInterval into timefilter
* pass mode to updateFilter to resolve functional test failure
* call registerTimefilterWithGlobalState from APM application
* remove rootScope from kbn_global_timepicker
* use to subscribe to timefilter in cases where results need to happen in digest async
* spalger review changes
* ensure monitoring timefilter callbacks are executed inside evalAsync
* remove unneeded evalAsync calls around setTime
