With Metricbeat shipping Elasticsearch monitoring data (instead of internal collection by Elasticsearch), there are some subtle changes to the format of monitoring docs that are indexed into `.monitoring-es-6-*`. One such change is that metricbeat won't index fields with `null` values; instead it simply doesn't index such fields at all.
As a result, in the context of Elasticsearch monitoring docs, when it comes to docs with `type` = `shards` representing unassigned shards, the `shard.node` field was being indexed as `null` by internal Elasticearch collection, whereas the field was absent when the doc was indexed by Metricbeat.
Since both cases represent the same case — the shard being unassigned — this PR relaxes the check in the UI code to look for either case.
### Sample `shards` document indexed by internal ES collection
```js
{
"_index":".monitoring-es-6-2018.10.02",
"_type":"doc",
"_id":"WUf_htOeSXOJQmiesyF5Bw:_na:metricbeat-7.0.0-alpha1-2018.10.01:0:r",
"_source":{
"cluster_uuid":"zXO1GjA6SJGsrPnCPkOoyA",
"timestamp":"2018-10-02T03:54:43.364Z",
"interval_ms":10000,
"type":"shards",
"source_node":null,
"state_uuid":"WUf_htOeSXOJQmiesyF5Bw",
"shard":{
"state":"UNASSIGNED",
"primary":false,
"node":null,
"relocating_node":null,
"shard":0,
"index":"metricbeat-7.0.0-alpha1-2018.10.01"
}
}
}
```
### Sample `shards` document indexed by Metricbeat collection
```js
{
"_index":".monitoring-es-6-mb-2018.10.02",
"_type":"doc",
"_id":"FhDRTPjkQJqsgawYbxjQzw:_na:metricbeat-7.0.0-alpha1-2018.10.01:0:r",
"_source":{
"@timestamp":"2018-10-02T04:00:03.361Z",
"interval_ms":10000,
"shard":{
"state":"UNASSIGNED",
"primary":false,
"index":"metricbeat-7.0.0-alpha1-2018.10.01",
"shard":0
},
"state_uuid":"FhDRTPjkQJqsgawYbxjQzw",
"beat":{
"hostname":"Shaunaks-MBP-2",
"version":"7.0.0-alpha1",
"name":"Shaunaks-MBP-2"
},
"timestamp":"2018-10-02T04:00:03.375Z",
"type":"shards",
"metricset":{
"name":"shard",
"module":"elasticsearch",
"host":"localhost:9200",
"rtt":14254,
"namespace":"elasticsearch.shard"
},
"host":{
"name":"Shaunaks-MBP-2"
},
"cluster_uuid":"zXO1GjA6SJGsrPnCPkOoyA"
}
}
```
* Removed server functions from webpack bundle. Copies server files from canvas_plugins_src to canvas_plugins
* Moved server functions
* Installed CopyWebpackPlugin to copy server functions from canvas_plugin_src to canvas_plugin
* Added canvas_plugin_src to cluster manager ignore list
* Revert plugins task changes
* ignores __tests__ folder
* Added task to delete canvas_plugin before build
* Fixed bug in canvas:plugins:build-prod
* Updated yarn lock
* Limiting the number of spaces
* Adding docs
* Adding forgotten fixture
* Fixing tslint error
* Adjusting docs
* Changing test descriptions from Boom.badRequest to bad request
* Updating error snapshots
* Added check for mounted workpad loader before setState calls
* Added check for mounted page manager before setState calls
* Added check for mounted arg form before setState calls
* Resets onmousemove and onmouseup handlers when workpad page unmounts
This PR updates the role management screen so that changes to space privileges are correctly tracked when adding/updating/deleting both new and existing privilege associations.
We were not tracking state correctly when both existing and in-progress privileges existed on screen.
Closes#23541
* Add basic support for new K7 navigation
* Make visibility and app title work
* Allow nav controls on right side of navbar
* Use render callback w/ el
* Add support for multiple sides
* Remove fake spaces nav control
* Breadcrumb support
* Hide breadcrumbs in plugins when k7design is enabled:
* Fix units
* Rename k7 -> header
* Add tests
* Fix tests
* Fix loading indicator
* PR comments
* Move ts-ignore
* Use canvasApp icon type
Kibana now keeps a constant connection between the browser and the
server from Canvas's websocket. When there's a reverse proxy between the
server and the browser, the fallback is XHR polling. This open polling
connection was keeping the network alive all the time, never idle, which
resulted in the Chromium browser driver kept waiting. Eventually, the
Report job would fail with a timeout error.
* Add space id to all filebeat and metricbeat tutorials
* Do not show if default or does not exist. Also, move to a helper method as the logic is fairly complex now.
* Add comment
* Provide a boolean indicating if the current space is the default one on the context object
* Remove debug
* PR feedback
* Fix prettier issue
### Review notes
This is generally ready for review. We are awaiting https://github.com/elastic/elasticsearch/issues/32777 to improve handling when users do not have any access to Kibana, but this should not hold up the overall review for this PR.
This PR is massive, there's no denying that. Here's what to focus on:
1) `x-pack/plugins/spaces`: This is, well, the Spaces plugin. Everything in here is brand new. The server code is arguably more important, but feel free to review whatever you see fit.
2) `x-pack/plugins/security`: There are large and significant changes here to allow Spaces to be securable. To save a bit of time, you are free to ignore changes in `x-pack/plugins/security/public`: These are the UI changes for the role management screen, which were previously reviewed by both us and the design team.
3) `x-pack/test/saved_object_api_integration` and `x-pack/test/spaces_api_integration`: These are the API test suites which verify functionality for:
a) Both security and spaces enabled
b) Only security enabled
c) Only spaces enabled
What to ignore:
1) As mentioned above, you are free to ignore changes in `x-pack/plugins/security/public`
2) Changes to `kibana/src/server/*`: These changes are part of a [different PR that we're targeting against master](https://github.com/elastic/kibana/pull/23378) for easier review.
## Saved Objects Client Extensions
A bulk of the changes to the saved objects service are in the namespaces PR, but we have a couple of important changes included here.
### Priority Queue for wrappers
We have implemented a priority queue which allows plugins to specify the order in which their SOC wrapper should be applied: `kibana/src/server/saved_objects/service/lib/priority_collection.ts`. We are leveraging this to ensure that both the security SOC wrapper and the spaces SOC wrapper are applied in the correct order (more details below).
### Spaces SOC Wrapper
This wrapper is very simple, and it is only responsible for two things:
1) Prevent users from interacting with any `space` objects (use the Spaces client instead, described below)
2) Provide a `namespace` to the underlying Saved Objects Client, and ensure that no other wrappers/callers have provided a namespace. In order to accomplish this, the Spaces wrapper uses the priority queue to ensure that it is the last wrapper invoked before calling the underlying client.
### Security SOC Wrapper
This wrapper is responsible for performing authorization checks. It uses the priority queue to ensure that it is the first wrapper invoked. To say another way, if the authorization checks fail, then no other wrappers will be called, and the base client will not be called either. This wrapper authorizes users in one of two ways: RBAC or Legacy. More details on this are below.
### Examples:
`GET /s/marketing/api/saved_objects/index-pattern/foo`
**When both Security and Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object at this space.
3) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
4) The underlying client/repository are invoked to retrieve the object from ES.
**When only Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
3) The underlying client/repository are invoked to retrieve the object from ES.
**When only Security is enabled:**
(assume `/s/marketing` is no longer part of the request)
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object globally.
3) The underlying client/repository are invoked to retrieve the object from ES.
## Authorization
Authorization changes for this project are centered around Saved Objects, and builds on the work introduced in RBAC Phase 1.
### Saved objects client
#### Security without spaces
When security is enabled, but spaces is disabled, then the authorization model behaves the same way as before: If the user is taking advantage of Kibana Privileges, then we check their privileges "globally" before proceeding. A "global" privilege check specifies `resources: ['*']` when calling the [ES _has_privileges api.](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-has-privileges.html). Legacy users (non-rbac) will continue to use the underlying index privileges for authorization.
#### Security with spaces
When both plugins are enabled, then the authorization model becomes more fine-tuned. Rather than checking privileges globally, the privileges are checked against a specific resource that matches the user's active space. In order to accomplish this, the Security plugin needs to know if Spaces is enabled, and if so, it needs to ask Spaces for the user's active space. The subsequent call to the `ES _has_privileges api` would use `resources: ['space:marketing']` to verify that the user is authorized at the `marketing` space. Legacy users (non-rbac) will continue to use the underlying index privileges for authorization. **NOTE** The legacy behavior implies that those users will have access to all spaces. The read/write restrictions are still enforced, but there is no way to restrict access to a specific space for legacy auth users.
#### Spaces without security
No authorization performed. Everyone can access everything.
### Spaces client
Spaces, when enabled, prevents saved objects of type `space` from being CRUD'd via the Saved Objects Client. Instead, the only "approved" way to work with these objects is through the new Spaces client (`kibana/x-pack/plugins/spaces/lib/spaces_client.ts`).
When security is enabled, the Spaces client performs its own set of authorization checks before allowing the request to proceed. The Spaces client knows which authorization checks need to happen for a particular request, but it doesn't know _how_ to check privileges. To accomplish this, the spaces client will delegate the check security's authorization service.
#### FAQ: Why oh why can't you used the Saved Objects Client instead!?
That's a great question! We did this primarily to simplify the authorization model (at least for our initial release). Accessing regular saved objects follows a predictible authorization pattern (described above). Spaces themselves inform the authorization model, and this interplay would have greatly increased the complexity. We are brainstorming ideas to obselete the Spaces client in favor of using the Saved Objects Client everywhere, but that's certainly out of scope for this release.
## Test Coverage
### Saved Objects API
A bulk of the changes to enable spaces are centered around saved objects, so we have spent a majority of our time automating tests against the saved objects api.
**`x-pack/test/saved_object_api_integration/`** contains the test suites for the saved objects api. There is a `common/suites` subfolder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
3) Security only: `./security_only`
Each of these test configurations will start up ES/Kibana with the appropriate license and plugin set. Each set runs through the entire test suite described in `common/suites`. Each test with in each suite is run multiple times with different inputs, to test the various permutations of authentication, authorization type (legacy vs RBAC), space-level privileges, and the user's active space.
### Spaces API
Spaces provides an experimental public API.
**`x-pack/test/spaces_api_integration`** contains the test suites for the Spaces API. Similar to the Saved Objects API tests described above, there is a `common/suites` folder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
### Role Management UI
We did not provide any new functional UI tests for role management, but the existing suite was updated to accomidate the screen rewrite.
We do have a decent suite of jest unit tests for the various components that make up the new role management screen. They're nested within `kibana/x-pack/plugins/security/public/views/management/edit_role`
### Spaces Management UI
We did not provide any new functional UI tests for spaces management, but the components that make up the screens are well-tested, and can be found within `kibana/x-pack/plugins/spaces/public/views/management/edit_space`
### Spaces Functional UI Tests
There are a couple of UI tests that verify _basic_ functionality. They assert that a user can login, select a space, and then choose a different space once inside: `kibana/x-pack/test/functional/apps/spaces`
## Reference
Notable child PRs are listed below for easier digesting. Note that some of these PRs are built on other PRs, so the deltas in the links below may be outdated. Cross reference with this PR when in doubt.
### UI
- Reactify Role Management Screen: https://github.com/elastic/kibana/pull/19035
- Space Aware Privileges UI: https://github.com/elastic/kibana/pull/21049
- Space Selector (in Kibana Nav): https://github.com/elastic/kibana/pull/19497
- Recently viewed Widget: https://github.com/elastic/kibana/pull/22492
- Support Space rename/delete: https://github.com/elastic/kibana/pull/22586
### Saved Objects Client
- ~~Space Aware Saved Objects: https://github.com/elastic/kibana/pull/18862~~
- ~~Add Space ID to document id: https://github.com/elastic/kibana/pull/21372~~
- Saved object namespaces (supercedes #18862 and #21372): https://github.com/elastic/kibana/pull/22357
- Securing saved objects: https://github.com/elastic/kibana/pull/21995
- Dedicated Spaces client (w/ security): https://github.com/elastic/kibana/pull/21995
### Other
- Public Spaces API (experimental): https://github.com/elastic/kibana/pull/22501
- Telemetry: https://github.com/elastic/kibana/pull/20581
- Reporting: https://github.com/elastic/kibana/pull/21457
- Spencer's original Spaces work: https://github.com/elastic/kibana/pull/18664
- Expose `spaceId` to "Add Data" tutorials: https://github.com/elastic/kibana/pull/22760Closes#18948
"Release Note: Create spaces within Kibana to organize dashboards, visualizations, and other saved objects. Secure access to each space when X-Pack Security is enabled"
* Adds progress function and elements
Added progress elements
Added progress view
Added unit tests for progress common function
Fixed prop type in toggle arg
Renamed vert -> vertical and horiz -> horizontal
Adjusted progress element dimensions
Removed check for null context in progress function
Refactored progress shapes
Added unicorn shape
Adds labelPosition arg
Added tests for labelPosition
* Added percentage column to demodata
* Updated elements to use percent_uptime in demodata
* Updated demodata percent values
* Refactored progress to use SVGs instead of shape defs
* Added barWeight arg to progress function
* Removed labelPosition arg. Set static label position for each progress shape
* Added label to unicorn shape
* Fixed element images
* removed passing of ID in to creation of layout and added a new test without a layout param
* renamed test constant so it reflects what we are testing and added comment
* Recommended Changes
* removed */ from end of line
* Begin replacing pipeline editor KUI elements.
* WIP build out EUI rendering of Create Pipeline view.
* Add settings components.
* Add close functionality.
* Add save functionality.
* Add temporary dependency hack for testing purposes until EUI XY Chart replaces jquery-flot.
* Add delete pipeline button/capability.
* Add delete modal.
* Remove TODO comment.
* Added toasts.
* Switch to global toast system.
* Add toast for inactive license and readonly state.
* Remove pipeline edit template.
* Add notify on PUT and DELETE errors.
* Add null check for username prop of securityService return value.
* Add disable save button if invalid ID.
* Remove pipeline id field when editing existing pipeline.
* Remove obsolete code.
* Move PipelineEditor component to dedicated file.
* Add EUI table to pipeline list view.
* Add search to pipelines table.
* Add create/delete pipelines buttons.
* Add pagination stubs. Complete after EUI bug resolved.
* Added unselectable for non-centrally-managed pipelines.
* Add clone button to pipelines list.
* Add min page height. Fix bug with edit pipeline link.
* Remove obsolete pipeline list code.
* Remove obsolete tooltip, edit, list code.
* Disable create pipeline if id is empty.
* Move PipelineList component to dedicated file.
* Add empty state to pipeline list. Add selection messages.
* Update loading message.
* Move methods to more logical positions in component.
* Add info alerts to pipeline list.
* Remove obsolete angular template.
* Remove obsolete imports from pipeline list directive.
* Define UpgradeFailure component.
* Move UpgradeFailure subcomponents to dedicated files. Write tests.
* Move PipelineEditor subcomponents to dedicated files.
* Write tests for pipeline editor subcomponents.
* Move bare strings into constants.
* Move PipelineEditor constant values into constants file.
* Break subcomponents of InfoAlerts component into dedicated files.
* Remove obsolete constants.
* WIP - write tests for PipelineList, break table into separate component, add error empty prompt message.
* Move ConfirmDeleteModal component to dedicated file and test.
* Add TODO comment.
* Add test tags to react components.
* Add 'data-test-subj' prop to fields for func tests. Minor layout update. Run prettier on some files.
* Add data-test-subj prop to button. Disable two tests until pagination is re-added.
* Re-enabled pagination for pipeline list.
* Remove wallaby hack.
* Update pagination options, remove obsolete code.
* Fix bug introduced in refactor to display delete button at appropriate time.
* Handle max_bytes setting correctly. Add theme/mode to code editor.
* Update snapshot for new pagination.
* Remove angular template for UpgradeFailure view.
* Move bare text from Modal functional component to constant file. Update test + snapshot.
* Ran prettier on all changed documents.
* Remove obsolete TODO comments.
* Re-enable disabled functional test. Clean up TODO code.
* Fix unresolved promise in functional tests.
* Pipeline delete button hidden unless enabled, move to left.
* Make filter title more readable.
* Apply width to clone column on Pipeline List.
* Modify pipeline edit view to use 's'-size icons.
* Change pipeline editor delete button to empty button.
* Move pipeline edit actions to bottom left of form.
* Add propTypes for PipelineEditor.
* Update test snapshots.
* Update pipeline list delete button func test.
* Add pipeline edit test. Add heading to pipeline edit page.
* Move constant files to modules where they are consumed.
* Move UPGRADE_FAILURE constants into module that consumes them.
* Remove redundant tests and remove text constant imports from tests.
* Give initial values to id and description text fields to make them controlled components.
* Clean up pipeline ID form regex validation and add tests.
* EUIify report management page
* wire ReportListing component together
* fetch jobs and display content in EuiPage
* display jobs in table
* add title and remove page size dropdown
* format date and display date in status column
* add poller
* add download button
* report error button
* remove old reporting table
* fix page styling
* create type for job
* remove job queue service
* remove angular-paging dependency from x-pack
* make download lib, update job notification hack to use jobQueueClient
* fix some more typescript stuff
* remove last angular service
* make report object type subdued color and small text
* update import in canvas
* stricter typing
* fix stuff lost in branch merge
* add return types to JobQueueClient
* wrap javascript code in {} in JSX
Improves the display of the Explorer Chart labels to fix the following issues:
- Long chart labels could be cut off, so it's not possible to tell what entity fields a chart is referring to. A workaround is to hover the info icon tooltip but that's really slow and cumbersome if you have to do it for every chart.
- The list of entity fields and its values is an unformatted text blob which makes it hard to read and tell which values refer to which field.
Changes:
- If any of the chart labels is longer than 60 chars, the entity fields will wrap to a new line (for all charts to a achieve a consistent look).
- Entity fields use EuiBadge and some custom formatting to make it easier to see field/value pairs.
- If the detector description is too long, it still uses ellipsis for text-overflow:
- If the entity badges are too long, they will be just cut off to the right. There's no simple CSS fix for that, we cannot use ellipsis and we don't want to wrap those badges again because then multiple charts could have different heights. I experimented with gradients but that turned out to be somewhat unreliable. I still consider this a good enough improvement compare to the previous version and would like to leave a tweak for that to a follow up PR.
- If there are mixed detectors with and without entity fields and the existing one wrap, multiple charts are aligned considered the height of the entity fields on display:
- Additionally, this changes the link to the single series viewer from custom code using a Font Awesome icon to use EuiButtonEmpty with the same EUI based icon and a tooltip.
Fixes two issues in IE11 for Anomaly Explorer:
- The format of the string returned from element.attr('transform') is different in IE11 so the regex based on it would fail. This fixes the issue and adds tests for the different formats. The code was also changed to gracefully return NaN in case the regex wouldn't return results, the previous version triggered a JS error.
- The migration of the swimlanes to React caused the cell selection to malfunction in IE11. This fixes it by updating the dragSelect library to use the new method setSelectables. The previous method we used (addSelectables) didn't play well with how React rerenders the swimlanes. Note this lib update using the new method will require to run yarn kbn bootstrap.
* Feat: ad-hoc grouping
* Feat: deleting ad-hoc group constituents
* Chore: deleted the former removeElement action
* Feat: make group snap to guides
* Feat: make group snap to guides 2
* Feat: make group snap to guides 3
* Use an instance of SavedObjectsSerializer for migrations and the repository
* Fixing spelling of serialization
* Making the serializer conditionally include and prepend id with ns
* Adding repository tests for the namespaces
* Implementing find
* Modifying the SOCs to pass the options with the namespace
* Centralizing omitting the namespace when using serializer.rawToSavedObject
* Passing the schema through to the SavedObjectRepositoryProvider
* Changing the schema to work with undefined ui exports schemas
* Adding schema tests
* Making the complimentary serialization test use the namespace
* Fixing uiExports
* Fixing some tests
* Fixing included fields for the find
* Fixing include field tests, they're checking length also...
* Updating Repository test after adding namespace to always included
fields
* Renaming UIExportsSavedObjectTypeSchema to SavedObjectsSchemaDefinition
* Completing rename... forgot to save usages
* Fixing issue with the serialization.isRawSavedObject and the trailing :
* Merge in boilerplate branch
* Manually copy over the specific metrics and UIs
* Add api integration tests
* Fix tests
* Remove unused metrics
* Update snapshot
* Fix tests
* Remove types agg
* Use ApmClusterMetric
* provide description for apm-server monitoring metrics (#23331)
* Vis LESS to SASS (cont.) (#23199)
* Tweak migrations integraiton tests to have a stable sort (#23265)
* Fix: plugin api route with security enabled (#23334)
Closes https://github.com/elastic/kibana/issues/23266
This is more of a quick fix than the final solution. The issue was that Canvas tries to check the plugins API without checking to see if the user it logged in. As a result, instead of the plugins response, it gets the HTML from the login page and that causes an error to be thrown when attempting to parse the results.
For now, this PR just disables the auth requirement on the Canvas plugin API endpoint.
* [migrations/tests] sort results before assertion (#23347)
There have been several failures in this test, seemingly caused by a lack of sorting in the results. It makes sense that since both migrations are run simultaneously that sometimes one would succeed and sometimes another would, so I've just sorted the results before checking.
![image](https://user-images.githubusercontent.com/1329312/45791153-44e9cc80-bc3d-11e8-88c4-760d4c7b35bd.png)
cc: @chrisdavies
* [ML] Moves custom URL editor Add button and form to top of flyout (#23326)
* [ML] Moves custom URL editor Add button and form to top of flyout
* [ML] Edits to custom URL editor class name
* Graph LESS to SASS (#23348)
* Developer documentation for integrating with the telemetry service (#23295)
* Developer documentation for integrating with the telemetry service
* open with a bang
* more faqs
* thing about tracking ui interactions
* talk to the plat team
* create and register
* Fix a bug where ES sends a string and migrations expect a boolean (#23313)
* chore: use cheerio in i18n.html.getDirectiveMessages (#23342)
this was only using jsdom to parse html, but cheerio allows parsing html without requiring a dom. cheerio was also already in the dependency list.
* [core/utils] add shareWeakReplay() operator (#23333)
* Chore: fix canvas test runner (#23336)
Blocked by https://github.com/elastic/kibana/pull/23342
This fixes the local test runner in Canvas. It should not affect anything else, including the CI test runner.
- Bumps JSDOM to ^12.0.0
- I matched Kibana's version on migration, but nothing else in X-Pack uses JSDOM, so we can use the newer version (which has a very different API)
- I had to match it because of a script that enforces version matching, but #23342 removed jsdom from Kibana, so we no longer have a version to match
- Restores the local `.babelrc` file
- I thought it was only used for building plugins; I was wrong 😢
* Convert Discover open top nav to EUI flyout (#22971)
* move find logic to SavedObjectFinder component since savedObjectClient is no longer coupled to angular
* implement flyout open saved searches
* remove old open stuff
* add jest test for OpenSearchPanel and simplify panel title
* fix functional tests
* fix _lab_mode functional test
* Migrate save top nav in Discover and Visualize to EUI (#23190)
* extract reusable save component from DashboardSaveModal
* update discover search to use SavedObjectSaveModal
* create generic show_save_model that works for both discover and dashboard
* fix last bits of discover save
* remove old save functionallity
* migrate visualize save to EUI
* fix functional tests
* disable save button if title is empty
* mark title input as invalid when title is not provided
* fix funtional tests
* Moves styleSheetPath to uiExports (#23007)
This was previously defined in uiExports.app, which limited plugins which are not an app of providing a stylesheet. This allows any plugin to define a stylesheet which will be available on page load.
* Timelion less to sass (#23339)
* Consistent casing
* Fix snapshot
* Update tests