* Add KQL functionality in the find function of the saved objects
wip
rename variable from KQL to filter, fix unit test + add new ones
miss security pluggins
review I
fix api changes
refactor after reviewing with Rudolf
fix type
review III
review IV
for security put back allowed logic back to return empty results
remove StaticIndexPattern
review V
fix core_api_changes
fix type
* validate filter to match requirement type.attributes.key or type.savedObjectKey
* Fix types
* fix a bug + add more api integration test
* fix types in test until we create package @kbn/types
* fix type issue
* fix api integration test
* export nodeTypes from packages @kbn/es-query instead of the function buildNodeKuery
* throw 400- bad request when validation error in find
* fix type issue
* accept api change
* renove _ to represent private
* fix unit test + add doc
* add comment to explain why we removed the private
* Re-split ciGroups after pipeline rollout
Revert "Revert "Revert "Revert "Revert "[ci] compress jobs for CI stab… (#45454)"
This reverts commit 9a109f2170.
Revert "set IS_PIPELINE_JOB in intake jobs (#45850)"
This reverts commit b1a01effa8.
* Split one of the slow test suites up to try to make overall CI faster
* Disable visualRegression groups, they are being handled in other work
* Revert "Split one of the slow test suites up to try to make overall CI faster"
This reverts commit 1213239545.
* Move some different xpack ciGroup8 suites around
* Pipeline
* WIP some work for parallelization with ciGroups
* Fix xpack kibana install dir, and add some debugging
* Attempt to quick fix a few tests
* Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)""
This reverts commit 078ac2897f.
* Recombine test groups, and try runbld again
* Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline
* Fix typo in shell script
* Remove some debug code
* Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT
* Fix test that uses hard-coded es transport port and add it back in to parallel groups
* Disable checks reporter again for now
* Set env var for TEST_ES_TRANSPORT_PORT in pipeline
* Update Jenkinsfile for shorter testrunner labels
* Fix another hard-coded transport port
* Fix a new test with hard-coded URLs
* Jenkinsfile cleanup and fix one of the groups
* Fix double slash
* Testing vault credentials on jenkins server
* Add a non-existent credential
* Revert "Add a non-existent credential"
This reverts commit 0dc234c465a5483b1a994cb510a182fef766e9cc.
* Try github-checks-reporter again
* github-checks-reporter should only run for elastic/kibana, forks won't work
* Clean up some debug code
* Changing names around to try to make BlueOcean UI a little better
* Add more stages
* Make some changes to stage structure to mirror a nested example from CloudBees
* Handle TODOs, and some cleanup in Jenkinsfile
* Pass GIT_BRANCH when started without GHPRB, fix branch check
* Fix mailer problem and add code that ensures all tests are in cigroups back in
* Test adding worker/job name to junit report paths
* Remove some duplication from ci_setup scripts
* Fix unit test that uses junit path
* Don't reinstall node every time setup_env is run
* Fix yarn install logic
* Fix another unit test that uses junit output dir
* Download latest ES snapshot after kibana builds
* Make sure junit reports are always processed
* Add two failing tests for testing purposes
* Add support to Jenkinsfile for kibana build e-mails
* Remove some debug code for email sending
* Change JOB env handling in junit paths and move it to a sub-directory
* Revert "Add two failing tests for testing purposes"
This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c.
* Fix junit report path in test
* Don't send kibana emails on build abort
* Address PR feedback, formatting and use built-in url formatting library
* Fix path formatting for functional test
* Add email sending back in to Jenkinsfile
* Fix another unit test with path problem
* make space a hidden saved object type
* bulk_create api tests
* bulk_get tests
* create functional tests
* delete space tests
* export space tests
* find space tests
* get space tests
* import space tests
* resolve_import_errirs space tests
* update space tests
* standardize test names where appropriate
* remove unused import
* Switching tests from using the space type directly to a "hidden t… (#21)
* add space saved object api tests
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
* split large x-pack ciGroup5 into 3 groups
* split large x-pack ciGroup3 into 2 groups
* Add groups to the jenkins xpack ci group file
* Re-level some more work
* move es_search_source test to new describe block
* move es_search_source to first in ciGroup7
* Modify the relationships API and UI
* Remove type validation on export
* Update relationship test snapshots
* Change relationships table titles
* Change relationships UI to share one table
* Add server side logic to inject meta data into saved objects from plugins
* Manually enable each type of saved object to support
* Use injected vars to determine what types are import / exportable
* Fix some broken tests
* Remove unused translations
* Fix relationships mocha tests
* Remove tests that ensured types are restricted, functionality removed
* Move kfetch logic into separate file
* Add inAppUrl to missing types
* Add tooltip to management table titles that aren't links
* Make relationships screen support filtering by type
* Fix failing tests
* Add refresh support for inAppUrls
* Add error notifications when export API call fails
* Add relationship direction
* Fix broken tests
* Remove graph workspace from import / export
* Use parent / child terminology for relationships
* Use direct relationship terminology
* Flip view / edit logic in saved object management app
* Make config saved object redirect to advanced settings
* Fix broken tests
* Remove unused translations
* Code cleanup
* Add tests
* Add fallback overwrite confirmation object title
* Enforce supported types on import, export and resolve import errors
* Fix broken tests
* Fix broken tests pt2
* Fix broken tests pt3
* Test cleanup
* Use server.decorate to access savedobjectschemas
* Fix some broken tests
* Fix broken tests, add new title to relationships screen
* Fix some broken tests
* Handle dynamic versions
* Fix inAppUrl structure in tests
* Re-use generic canGoInApp
* Fix broken tests
* Apply maps PR feedback
* Apply PR feedback pt1
* Apply PR feedback pt2
* Add savedObjectsManagement to uiExports
* Fix broken tests
* Fix encodeURIComponent implementation
* Merge 403 and unsupported type errors into single error
* Apply suggestion
* Remove import / exportable by default, opt-in instead
* Fix type config to show up properly in the table
* Change config type title and fix tests
* Remove isImportableAndExportable where set to false (new default)
* Remove comments referencing to authorization
* Add unit tests for spaces
* Add unit tests for security plugin
* Change can* signature to be the same as their equivalent function, apply PR feedback
* Cleanup git diff
* Revert "Change can* signature to be the same as their equivalent function, apply PR feedback"
This reverts commit b657ac8fc1.
* Revert "Add unit tests for security plugin"
This reverts commit 6287a8cecf.
* Revert "Add unit tests for spaces"
This reverts commit 2674a9d78f.
* Revert "Remove comments referencing to authorization"
This reverts commit 9618c2cc3a.
* Revert "Merge 403 and unsupported type errors into single error"
This reverts commit 99aea10c0f.
* Add CUSTOM_ELEMENT_TYPE for import / export
* Fix broken tests
* Fix broken tests pt2
* Prevent crashing app when inAppUrl is undefined
enable security on file dataviz and import (ML plugin)
update unit tests
add api test coverage for security in basic
move audit logging to standard+ license level
* Initial work
* Add overwrite and skip support
* Cleanup and add tests
* Move code into separate files
* Remove reduce
* New API parameters
* Add support to replace references
* Add better error handling
* Add spaces tests
* Fix return type in collectSavedObjects
* Apply PR feedback
* Update jest tests due to jest version upgrade
* Add docs
* WIP
* Split import routes pt1
* Add tests
* Fix broken tests
* Update docs and fix broken test
* Add successCount to _import endpoint
* Make skip by default in resolution API
* Update tests for removal of skips
* Add back support for skips
* Add success count
* Add back resolve import conflicts x-pack tests
* Remove writev from filter stream
* Delete _mock_server.d.ts file
* Rename lib/import_saved_objects to lib/import
* Filter records at stream level for conflict resolution
* Update docs
* Add tests to validate documentation
* Return 200 instead of other code for errors, include errors array
* Change [] to {}
* Apply PR feedback
* Fix import object limit to not return 500
* Change some wording in the docs
* Fix status code
* Apply PR feedback pt2
* Lower maxImportPayloadBytes to 10MB
* Add unknown type tests for import
* Add unknown type tests for resolve_import_conflicts
* Fix tslint issues
* Initial work for new server side export API
* Revert UI changes, API only in this PR
* Remove whitespace at top of export.asciidoc
* Add tests around limitations
* Add comment
* Convert some files to typescript
* Move Boom.boomify to where the errors are created
* Use Boom.badRequest for now
* Fix lint issue
* Move files
* Update tests
* Add functional test
* Export all documents by default
* Update test assertions
* Use ~10000 saved objects in export api integration test
* Convert route to typescript, add content-type response header
* Move some tests to api_integration
* Use new sort and rename functions/variables
* Move tests to API integration
* Cleanup and finalize api integration tests
* Make type or objects required but not both in the same call
* Add spaces / security tests
* Add noTypeOrObjects to security / spaces tests
* Use json-stable-stringify and add tests for export ordering
* Address self feedback, add without kibana index test
* Only allow export API to export index-pattern, dashboard, visualization and search type objects
* Make import export size configurable and fix broken tests
* Fix broken tests
* Move test config to mock server
* Add more typescript types instead of using any
* Convert request from GET to POST
* Fix saved objects mixin test
* Update src/legacy/server/saved_objects/lib/export.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Apply PR feedback
* Fix lint error
* Update test snapshots due to jest upgrade
* Add error handling for bulkGet
* Split export API into two endpoints
* Update src/legacy/server/saved_objects/routes/export_by_type.test.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update docs/api/saved-objects/export_by_type.asciidoc
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update docs/api/saved-objects/export_by_type.asciidoc
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update src/legacy/server/saved_objects/routes/export_objects.test.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Apply PR feedback
* MockServer -> createMockServer
* Revert back to single API
* Re-apply PR feedback
* Saved Objects routes and service should be able to hide objects.
* Remove context providers as a feature.
* Respository should be creatable to include hidden types.
* Fixes failing unit tests.
* Fixes issues with filter method.
* Adds check to get method for allowed types.
* Adds tests for get,delete,bulkGet,find
* Remove need for schema in saved objects api.
* Remove more traces of schema validation added to rest api.
* Remove inclusion of hidden types in route specific client.
* Removes getAvailableTypes as it is no longer used.
* Fixing up initialization of components.
* Moves default setting for includeHiddenTypes
* Allows for single value or array in assertAllowedTypes.
* Adds type assertion to bulkCreate, and incrementCounter with tests.
* Conversion to boolean should be more explicit.
* Repository should restrict types allowed to be manipulated.
* Saved objects should use the right root type.
Privileges should have unit test.
* All saved objects APIs should validate types.
* No need for test to be typescript if under test not ts.
* Handle extraTypes being undefined.
* Routes should verify that they do not allow invalid types.
* Bulk create should be tested.
* Saved objects mixin does not need extra blank lines.
* Saved objects integration tests should test unknown types.
* Integration tests should test for bad request with unknown type.
* Adds missing privileges to global all.
* Tests should use valid types.
Tests should have accurate expectations.
* Fix bulkCreate to assert allowed types.
* Fix unknown search field tests.
* Adjust expectations for unknown type in saved object api.
* Saved object integration should return proper responses.
* Fix expect to use a separated matcher.
* Should expect forbidden responses for unauthorized users.
* Should expect 400 when trying to use unknown types.
* Removes unwanted .only call.
* Adjust repository to throw error unless it has allowed types.
* Unknown types should return 403s and empty results where applicable
* Removes type validation from saved object API.
* Captures and returns appropriate exceptions for type assertion.
* Properly filtter c'tor params to repository.
* Checking allowed type should be bool check function.
* Cleanup test situation descriptions.
* Updating snapshot file for jest tests.
* Changes expected results for find from saved object service.
* Expect an empty response when attempting to access an unsupported type.
* Adds test coverage for new error methods added.
Adds create test to repository.
* Adds bulkGet, bulkCreate unsupported type errors.
* First step in refactoring saved object service.
Adding missing test coverage of saved object service creation.
* Move extra saved objects test to legacy folder.
* Adds references filtering by allowed type.
* Adds more coverage for mixin repository creation.
* Removes unnecessary decorate on server object for unused method.
* Revert reworking how kibana migrator uses mappings.
* Revert "Adds references filtering by allowed type."
This reverts commit 92b07d4b92.
* Adds check for unexpected callCluster type.
* Should cover as many parts of the mixin as possible.
* More expectations to tests.
* Keeps ordering of created items but does not pass unsupported types.
* Fix a failing before hook test.
* Should not use escaped single quotes inside template literals.
Co-Authored-By: njd5475 <njd5475@gmail.com>
* Changed how check is done in repository.
* Remove unused mappings file.
* Cleans up a couple of nits.
* Adds test for overwrite option being passed if it is in the url.
* Missed semicolon.
* Remove mode.initialize and change useRbacForRequest to useRbac
* Updating saved object api tests
* Fixing spaces api integration tests
* Removing unused "expect legacy forbidden" declarations and imports
* Updating docs
* Update docs/migration/migrate_7_0.asciidoc
Co-Authored-By: kobelb <brandon.kobel@gmail.com>
* Update docs/migration/migrate_7_0.asciidoc
Co-Authored-By: kobelb <brandon.kobel@gmail.com>
* Updating comment that mentions the scenario when we aren't using RBAC
* Adding back the authorization section of the config
When a config setting is marked as unused using the deprecations, it's
still required to show up in the config declarations so an error isn't
thrown on startup.
* Adding note about watcher jobs
* Update docs/migration/migrate_7_0.asciidoc
Co-Authored-By: kobelb <brandon.kobel@gmail.com>
* Added ES Archiver Tests for uncommon processes
* Re-initialized host api integration tests with new data
* Fixed ci build to work with api integration tests again
* Fixed ci build to run tslint, eslint, type_check again
* Fixed import with ui/theme to work with the dll optimizer
* Minor word changes
* https://github.com/elastic/ingest-dev/issues/193
* https://github.com/elastic/ingest-dev/issues/185
* [ci][ftr][kbn/test] split up CI jobs
* [ci] run --assert-none-excluded in ci groups, before builds
* [ftr] improve error message when excluded tests found
* [ci] document other places the ciGroups live
* Making it easier and more terse to specify the user for a test
* Using ES list privileges API to determine the authorization mode
This let's us correct use RBAC authorization for the proper users when
security is enabled, and spaces is disabled to detect whether they have
privileges of any kind and if so use RBAC.
* Fixing authorization service test
* Fixing tests referencing wrong expects
* Putting create test back
* Update x-pack/plugins/security/server/lib/authorization/mode.js
* Update x-pack/plugins/security/server/lib/authorization/mode.js
### Review notes
This is generally ready for review. We are awaiting https://github.com/elastic/elasticsearch/issues/32777 to improve handling when users do not have any access to Kibana, but this should not hold up the overall review for this PR.
This PR is massive, there's no denying that. Here's what to focus on:
1) `x-pack/plugins/spaces`: This is, well, the Spaces plugin. Everything in here is brand new. The server code is arguably more important, but feel free to review whatever you see fit.
2) `x-pack/plugins/security`: There are large and significant changes here to allow Spaces to be securable. To save a bit of time, you are free to ignore changes in `x-pack/plugins/security/public`: These are the UI changes for the role management screen, which were previously reviewed by both us and the design team.
3) `x-pack/test/saved_object_api_integration` and `x-pack/test/spaces_api_integration`: These are the API test suites which verify functionality for:
a) Both security and spaces enabled
b) Only security enabled
c) Only spaces enabled
What to ignore:
1) As mentioned above, you are free to ignore changes in `x-pack/plugins/security/public`
2) Changes to `kibana/src/server/*`: These changes are part of a [different PR that we're targeting against master](https://github.com/elastic/kibana/pull/23378) for easier review.
## Saved Objects Client Extensions
A bulk of the changes to the saved objects service are in the namespaces PR, but we have a couple of important changes included here.
### Priority Queue for wrappers
We have implemented a priority queue which allows plugins to specify the order in which their SOC wrapper should be applied: `kibana/src/server/saved_objects/service/lib/priority_collection.ts`. We are leveraging this to ensure that both the security SOC wrapper and the spaces SOC wrapper are applied in the correct order (more details below).
### Spaces SOC Wrapper
This wrapper is very simple, and it is only responsible for two things:
1) Prevent users from interacting with any `space` objects (use the Spaces client instead, described below)
2) Provide a `namespace` to the underlying Saved Objects Client, and ensure that no other wrappers/callers have provided a namespace. In order to accomplish this, the Spaces wrapper uses the priority queue to ensure that it is the last wrapper invoked before calling the underlying client.
### Security SOC Wrapper
This wrapper is responsible for performing authorization checks. It uses the priority queue to ensure that it is the first wrapper invoked. To say another way, if the authorization checks fail, then no other wrappers will be called, and the base client will not be called either. This wrapper authorizes users in one of two ways: RBAC or Legacy. More details on this are below.
### Examples:
`GET /s/marketing/api/saved_objects/index-pattern/foo`
**When both Security and Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object at this space.
3) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
4) The underlying client/repository are invoked to retrieve the object from ES.
**When only Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
3) The underlying client/repository are invoked to retrieve the object from ES.
**When only Security is enabled:**
(assume `/s/marketing` is no longer part of the request)
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object globally.
3) The underlying client/repository are invoked to retrieve the object from ES.
## Authorization
Authorization changes for this project are centered around Saved Objects, and builds on the work introduced in RBAC Phase 1.
### Saved objects client
#### Security without spaces
When security is enabled, but spaces is disabled, then the authorization model behaves the same way as before: If the user is taking advantage of Kibana Privileges, then we check their privileges "globally" before proceeding. A "global" privilege check specifies `resources: ['*']` when calling the [ES _has_privileges api.](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-has-privileges.html). Legacy users (non-rbac) will continue to use the underlying index privileges for authorization.
#### Security with spaces
When both plugins are enabled, then the authorization model becomes more fine-tuned. Rather than checking privileges globally, the privileges are checked against a specific resource that matches the user's active space. In order to accomplish this, the Security plugin needs to know if Spaces is enabled, and if so, it needs to ask Spaces for the user's active space. The subsequent call to the `ES _has_privileges api` would use `resources: ['space:marketing']` to verify that the user is authorized at the `marketing` space. Legacy users (non-rbac) will continue to use the underlying index privileges for authorization. **NOTE** The legacy behavior implies that those users will have access to all spaces. The read/write restrictions are still enforced, but there is no way to restrict access to a specific space for legacy auth users.
#### Spaces without security
No authorization performed. Everyone can access everything.
### Spaces client
Spaces, when enabled, prevents saved objects of type `space` from being CRUD'd via the Saved Objects Client. Instead, the only "approved" way to work with these objects is through the new Spaces client (`kibana/x-pack/plugins/spaces/lib/spaces_client.ts`).
When security is enabled, the Spaces client performs its own set of authorization checks before allowing the request to proceed. The Spaces client knows which authorization checks need to happen for a particular request, but it doesn't know _how_ to check privileges. To accomplish this, the spaces client will delegate the check security's authorization service.
#### FAQ: Why oh why can't you used the Saved Objects Client instead!?
That's a great question! We did this primarily to simplify the authorization model (at least for our initial release). Accessing regular saved objects follows a predictible authorization pattern (described above). Spaces themselves inform the authorization model, and this interplay would have greatly increased the complexity. We are brainstorming ideas to obselete the Spaces client in favor of using the Saved Objects Client everywhere, but that's certainly out of scope for this release.
## Test Coverage
### Saved Objects API
A bulk of the changes to enable spaces are centered around saved objects, so we have spent a majority of our time automating tests against the saved objects api.
**`x-pack/test/saved_object_api_integration/`** contains the test suites for the saved objects api. There is a `common/suites` subfolder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
3) Security only: `./security_only`
Each of these test configurations will start up ES/Kibana with the appropriate license and plugin set. Each set runs through the entire test suite described in `common/suites`. Each test with in each suite is run multiple times with different inputs, to test the various permutations of authentication, authorization type (legacy vs RBAC), space-level privileges, and the user's active space.
### Spaces API
Spaces provides an experimental public API.
**`x-pack/test/spaces_api_integration`** contains the test suites for the Spaces API. Similar to the Saved Objects API tests described above, there is a `common/suites` folder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
### Role Management UI
We did not provide any new functional UI tests for role management, but the existing suite was updated to accomidate the screen rewrite.
We do have a decent suite of jest unit tests for the various components that make up the new role management screen. They're nested within `kibana/x-pack/plugins/security/public/views/management/edit_role`
### Spaces Management UI
We did not provide any new functional UI tests for spaces management, but the components that make up the screens are well-tested, and can be found within `kibana/x-pack/plugins/spaces/public/views/management/edit_space`
### Spaces Functional UI Tests
There are a couple of UI tests that verify _basic_ functionality. They assert that a user can login, select a space, and then choose a different space once inside: `kibana/x-pack/test/functional/apps/spaces`
## Reference
Notable child PRs are listed below for easier digesting. Note that some of these PRs are built on other PRs, so the deltas in the links below may be outdated. Cross reference with this PR when in doubt.
### UI
- Reactify Role Management Screen: https://github.com/elastic/kibana/pull/19035
- Space Aware Privileges UI: https://github.com/elastic/kibana/pull/21049
- Space Selector (in Kibana Nav): https://github.com/elastic/kibana/pull/19497
- Recently viewed Widget: https://github.com/elastic/kibana/pull/22492
- Support Space rename/delete: https://github.com/elastic/kibana/pull/22586
### Saved Objects Client
- ~~Space Aware Saved Objects: https://github.com/elastic/kibana/pull/18862~~
- ~~Add Space ID to document id: https://github.com/elastic/kibana/pull/21372~~
- Saved object namespaces (supercedes #18862 and #21372): https://github.com/elastic/kibana/pull/22357
- Securing saved objects: https://github.com/elastic/kibana/pull/21995
- Dedicated Spaces client (w/ security): https://github.com/elastic/kibana/pull/21995
### Other
- Public Spaces API (experimental): https://github.com/elastic/kibana/pull/22501
- Telemetry: https://github.com/elastic/kibana/pull/20581
- Reporting: https://github.com/elastic/kibana/pull/21457
- Spencer's original Spaces work: https://github.com/elastic/kibana/pull/18664
- Expose `spaceId` to "Add Data" tutorials: https://github.com/elastic/kibana/pull/22760Closes#18948
"Release Note: Create spaces within Kibana to organize dashboards, visualizations, and other saved objects. Secure access to each space when X-Pack Security is enabled"
