* Revert "[FTR] Use importExport for saved_object/basic archive (#100244) (#102016)"
This reverts commit 9851b7bcfa.
* Revert "[7.x] [FTR][CI] Use default distribution for all tests (#94968) (#101118)"
This reverts commit 73225daa87.
* [FTR][CI] Use default distribution for all tests (#94968)
* [FTR] Use importExport for saved_object/basic archive (#100244)
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
In 7.x, when saved objects point to an index and not an alias it will
assume it's pre-6.8 and perform a legacy migration. This causes issues
with the removal of oss builds. This update prevents that from happening
by using an alias.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>
* [SOM] display invalid references in the relationship flyout (#88814)
* return invalid relations and display them in SOM
* add FTR test
* fix mappings for 7.x
* SavedObjects tagging MVP (#79096)
* create xpack plugin skeleton, start to implement management section
* add tag creation modal
* first implementation of the tags table
* use InMemoryTable
* add edit modal and delete action
* update plugin list
* add tag list, fix types
* add capabilities check on client-side
* add tag combo box component
* add missing i18n keys
* fix privilege FTR tests
* add base structure for FTR tests
* fix feature ftr test
* use string literals for i18n
* create savedObjectsTaggingOss plugin, move API types to oss plugin, start to wire to SO management page.
* update plugin list
* fix types
* allow to use `_find` with multiple references
* add FTR test for _find API on references fields
* add _find integration tests
* update generated doc
* start to implement tag filtering on SO management section
* update generated docs
* wire tagging API to dashboard listing page
* fix i18n namespace
* fix type & tests
* update dashboard listing snapshots
* adapt FTR listingTable service to search for parsable queries
* wite tagging API to visualize listing
* update tagging plugin limits
* add server-side and client-side validation for tag create/edit
* rename title field to name
* fix types
* fix types bis
* add removeReferencesTo API to SOR/SOC
* update generated doc
* add server-side unit test for `savedObjectsTagging` plugin
* move tagging API types to its own file
* add savedObjectsTaggingOss mock
* add tags_cache tests
* add tests for client-side tag client
* extract uiApi to distinct files
* various API improvements
* add more tests
* add link between tag and so management sections + add connection counts
* add base functional test suite for tagging
* add more FTR tests
* improve feature control func test
* update codeowners
* update generated doc
* fix access to proxy modal
* adapt SO save modal to allow to add tag field
* add SO decorator registry and tag implementation
* add unit tests for SO tag decorator
* add functional tests for visualize integration
* add tag SO read permission for vis/dash feature
* add RBAC api integ tests
* add API integration tests
* add test for getTagConnectionsUrl
* add SOM test suite
* add dashboard integration suite
* remove test line
* add missing unit tests
* improve API types doc
* fix create modal save button label
* remove console.log
* improve doc
* self review
* add refresh interval for tag cache
* improve page object doc
* minor cleanup
* address review comments
* small layout fixes
* add initial focus
* use lazy accessor for tag request handler context
* adapt SOM export and export route to handle references
* remove icon from feature config due to master changes
* fix SO table tests
* update generated docs
* sort tags by name in filter dropdown and listing component
* wire SO tagging to dashboard save modal
* fix types
* - add 'create tag' action in tag selector
- add notifications on update/create/delete from management
- delete modal wording
* add description max length validation
* remove real-time validation
* fix i18n bundle id
* update expected size of savedObjectsTagging plugin
* use own useIfMounted
* update limit again, contract components cannot be lazy loaded atm.
* math is hard
* remove single usage of lodash for bundle size
* add async imports for create/edit modal
* add FTR test for 'create tag' action from tag selector
* allow 'create new' option to prepopulate name field
* extract savedObjectToTag
* add advancedSettings read user for security api_integ suite
* add audit login for security client wrapper
* use import type when possible
* wire SO tagging to lens visualization
* fix lens jest test
* Fix `create tag` option being selected when closing the selector dropdown
* add sorting to tag column from getTableColumnDef
* address some of restrry comments
* rename tag selector's setSelected option to onTagsSelected
* fix audit logging even type for saved_object_remove_references
* update plugin size limit to current size
* adapt maxlength validation wording
* remove selection column until we have batch action menu
* remove connections link when user lack read privilege to savedObjectManagement
* forbid registering multiple SO decorators with the same priority
* add so decorator test
* extract getTagFindReferences and create API mock
* update audit-logging ascidoc
* doc nit
* throw conflict error if update returns any failure
* use refresh=true as default
* wording nits
* export: rename `references` to `hasReference`
* update generated doc
* set description max length to 100
* do not initialize tag cache on anonymous pages
* split fetchObjectsToExport into two distinct functions
* change tag client `delete` call order
* tsdoc nits
* more nits
* add README for oss plugin
* add oss plugin start tests
* SavedObject.find: rename `references` to `hasReference`
* change section description label
* remove url prefix constants
* last nits and comments
* update generated doc
# Conflicts:
# .github/CODEOWNERS
# packages/kbn-optimizer/limits.yml
# x-pack/scripts/functional_tests.js
* fix FTR mapping files for 7.x
* unexpose SavedObjectsManagement from legacy server
* migrate saved object management routes to new plugin
* fix endpoint methods
* adapt code due to rebase
* extract types
* improve findAll params
* adapt existing api integration tests and migrate to TS
* update generated doc
* add API integration tests for /scroll/count
* add unit tests for plugin and routes
* add injectMetaAttributes tests
* extract relation type
* add find_relationships tests
* add find_all tests
* do not complete migrator$ to avoid unhandled promise rejection
* fix data for search endpoint integration tests
* remove falsy comment
* rename plugin folder to match plugin id
* address review comments
* update CODEOWNERS
This PR adds a new syntax to KQL for querying nested fields.
Nested fields can be queried in two different ways:
Parts of the query may only match a single nested doc (bool inside nested). This is what most people want when querying on a nested field.
Parts of the query may match different nested docs (nested inside bool). This is how a regular object field works but nested fields can be queried in the same way. Although generally less useful, there are occasions where one might want to query a nested field in this way.
The new KQL syntax supports both.
* Initial work
* Add overwrite and skip support
* Cleanup and add tests
* Move code into separate files
* Remove reduce
* New API parameters
* Add support to replace references
* Add better error handling
* Add spaces tests
* Fix return type in collectSavedObjects
* Apply PR feedback
* Update jest tests due to jest version upgrade
* Add docs
* WIP
* Split import routes pt1
* Add tests
* Fix broken tests
* Update docs and fix broken test
* Add successCount to _import endpoint
* Make skip by default in resolution API
* Update tests for removal of skips
* Add back support for skips
* Add success count
* Add back resolve import conflicts x-pack tests
* Remove writev from filter stream
* Delete _mock_server.d.ts file
* Rename lib/import_saved_objects to lib/import
* Filter records at stream level for conflict resolution
* Update docs
* Add tests to validate documentation
* Return 200 instead of other code for errors, include errors array
* Change [] to {}
* Apply PR feedback
* Fix import object limit to not return 500
* Change some wording in the docs
* Fix status code
* Apply PR feedback pt2
* Lower maxImportPayloadBytes to 10MB
* Add unknown type tests for import
* Add unknown type tests for resolve_import_conflicts
* Fix tslint issues
* Initial work for new server side export API
* Revert UI changes, API only in this PR
* Remove whitespace at top of export.asciidoc
* Add tests around limitations
* Add comment
* Convert some files to typescript
* Move Boom.boomify to where the errors are created
* Use Boom.badRequest for now
* Fix lint issue
* Move files
* Update tests
* Add functional test
* Export all documents by default
* Update test assertions
* Use ~10000 saved objects in export api integration test
* Convert route to typescript, add content-type response header
* Move some tests to api_integration
* Use new sort and rename functions/variables
* Move tests to API integration
* Cleanup and finalize api integration tests
* Make type or objects required but not both in the same call
* Add spaces / security tests
* Add noTypeOrObjects to security / spaces tests
* Use json-stable-stringify and add tests for export ordering
* Address self feedback, add without kibana index test
* Only allow export API to export index-pattern, dashboard, visualization and search type objects
* Make import export size configurable and fix broken tests
* Fix broken tests
* Move test config to mock server
* Add more typescript types instead of using any
* Convert request from GET to POST
* Fix saved objects mixin test
* Update src/legacy/server/saved_objects/lib/export.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Apply PR feedback
* Fix lint error
* Update test snapshots due to jest upgrade
* Add error handling for bulkGet
* Split export API into two endpoints
* Update src/legacy/server/saved_objects/routes/export_by_type.test.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update docs/api/saved-objects/export_by_type.asciidoc
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update docs/api/saved-objects/export_by_type.asciidoc
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Update src/legacy/server/saved_objects/routes/export_objects.test.ts
Co-Authored-By: mikecote <mikecote@users.noreply.github.com>
* Apply PR feedback
* MockServer -> createMockServer
* Revert back to single API
* Re-apply PR feedback
### Review notes
This is generally ready for review. We are awaiting https://github.com/elastic/elasticsearch/issues/32777 to improve handling when users do not have any access to Kibana, but this should not hold up the overall review for this PR.
This PR is massive, there's no denying that. Here's what to focus on:
1) `x-pack/plugins/spaces`: This is, well, the Spaces plugin. Everything in here is brand new. The server code is arguably more important, but feel free to review whatever you see fit.
2) `x-pack/plugins/security`: There are large and significant changes here to allow Spaces to be securable. To save a bit of time, you are free to ignore changes in `x-pack/plugins/security/public`: These are the UI changes for the role management screen, which were previously reviewed by both us and the design team.
3) `x-pack/test/saved_object_api_integration` and `x-pack/test/spaces_api_integration`: These are the API test suites which verify functionality for:
a) Both security and spaces enabled
b) Only security enabled
c) Only spaces enabled
What to ignore:
1) As mentioned above, you are free to ignore changes in `x-pack/plugins/security/public`
2) Changes to `kibana/src/server/*`: These changes are part of a [different PR that we're targeting against master](https://github.com/elastic/kibana/pull/23378) for easier review.
## Saved Objects Client Extensions
A bulk of the changes to the saved objects service are in the namespaces PR, but we have a couple of important changes included here.
### Priority Queue for wrappers
We have implemented a priority queue which allows plugins to specify the order in which their SOC wrapper should be applied: `kibana/src/server/saved_objects/service/lib/priority_collection.ts`. We are leveraging this to ensure that both the security SOC wrapper and the spaces SOC wrapper are applied in the correct order (more details below).
### Spaces SOC Wrapper
This wrapper is very simple, and it is only responsible for two things:
1) Prevent users from interacting with any `space` objects (use the Spaces client instead, described below)
2) Provide a `namespace` to the underlying Saved Objects Client, and ensure that no other wrappers/callers have provided a namespace. In order to accomplish this, the Spaces wrapper uses the priority queue to ensure that it is the last wrapper invoked before calling the underlying client.
### Security SOC Wrapper
This wrapper is responsible for performing authorization checks. It uses the priority queue to ensure that it is the first wrapper invoked. To say another way, if the authorization checks fail, then no other wrappers will be called, and the base client will not be called either. This wrapper authorizes users in one of two ways: RBAC or Legacy. More details on this are below.
### Examples:
`GET /s/marketing/api/saved_objects/index-pattern/foo`
**When both Security and Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object at this space.
3) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
4) The underlying client/repository are invoked to retrieve the object from ES.
**When only Spaces are enabled:**
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Spaces wrapper is invoked.
a) Spaces applies a `namespace` to be used by the underlying client
3) The underlying client/repository are invoked to retrieve the object from ES.
**When only Security is enabled:**
(assume `/s/marketing` is no longer part of the request)
1) Saved objects API retrieves an instance of the SOC via `savedObjects.getScopedClient()`, and invokes its `get` function
2) The Security wrapper is invoked.
a) Authorization checks are performed to ensure user can access this particular saved object globally.
3) The underlying client/repository are invoked to retrieve the object from ES.
## Authorization
Authorization changes for this project are centered around Saved Objects, and builds on the work introduced in RBAC Phase 1.
### Saved objects client
#### Security without spaces
When security is enabled, but spaces is disabled, then the authorization model behaves the same way as before: If the user is taking advantage of Kibana Privileges, then we check their privileges "globally" before proceeding. A "global" privilege check specifies `resources: ['*']` when calling the [ES _has_privileges api.](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-has-privileges.html). Legacy users (non-rbac) will continue to use the underlying index privileges for authorization.
#### Security with spaces
When both plugins are enabled, then the authorization model becomes more fine-tuned. Rather than checking privileges globally, the privileges are checked against a specific resource that matches the user's active space. In order to accomplish this, the Security plugin needs to know if Spaces is enabled, and if so, it needs to ask Spaces for the user's active space. The subsequent call to the `ES _has_privileges api` would use `resources: ['space:marketing']` to verify that the user is authorized at the `marketing` space. Legacy users (non-rbac) will continue to use the underlying index privileges for authorization. **NOTE** The legacy behavior implies that those users will have access to all spaces. The read/write restrictions are still enforced, but there is no way to restrict access to a specific space for legacy auth users.
#### Spaces without security
No authorization performed. Everyone can access everything.
### Spaces client
Spaces, when enabled, prevents saved objects of type `space` from being CRUD'd via the Saved Objects Client. Instead, the only "approved" way to work with these objects is through the new Spaces client (`kibana/x-pack/plugins/spaces/lib/spaces_client.ts`).
When security is enabled, the Spaces client performs its own set of authorization checks before allowing the request to proceed. The Spaces client knows which authorization checks need to happen for a particular request, but it doesn't know _how_ to check privileges. To accomplish this, the spaces client will delegate the check security's authorization service.
#### FAQ: Why oh why can't you used the Saved Objects Client instead!?
That's a great question! We did this primarily to simplify the authorization model (at least for our initial release). Accessing regular saved objects follows a predictible authorization pattern (described above). Spaces themselves inform the authorization model, and this interplay would have greatly increased the complexity. We are brainstorming ideas to obselete the Spaces client in favor of using the Saved Objects Client everywhere, but that's certainly out of scope for this release.
## Test Coverage
### Saved Objects API
A bulk of the changes to enable spaces are centered around saved objects, so we have spent a majority of our time automating tests against the saved objects api.
**`x-pack/test/saved_object_api_integration/`** contains the test suites for the saved objects api. There is a `common/suites` subfolder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
3) Security only: `./security_only`
Each of these test configurations will start up ES/Kibana with the appropriate license and plugin set. Each set runs through the entire test suite described in `common/suites`. Each test with in each suite is run multiple times with different inputs, to test the various permutations of authentication, authorization type (legacy vs RBAC), space-level privileges, and the user's active space.
### Spaces API
Spaces provides an experimental public API.
**`x-pack/test/spaces_api_integration`** contains the test suites for the Spaces API. Similar to the Saved Objects API tests described above, there is a `common/suites` folder which contains a bulk of the test logic. The suites defined here are used in the following test configurations:
1) Spaces only: `./spaces_only`
2) Security and spaces: `./security_and_spaces`
### Role Management UI
We did not provide any new functional UI tests for role management, but the existing suite was updated to accomidate the screen rewrite.
We do have a decent suite of jest unit tests for the various components that make up the new role management screen. They're nested within `kibana/x-pack/plugins/security/public/views/management/edit_role`
### Spaces Management UI
We did not provide any new functional UI tests for spaces management, but the components that make up the screens are well-tested, and can be found within `kibana/x-pack/plugins/spaces/public/views/management/edit_space`
### Spaces Functional UI Tests
There are a couple of UI tests that verify _basic_ functionality. They assert that a user can login, select a space, and then choose a different space once inside: `kibana/x-pack/test/functional/apps/spaces`
## Reference
Notable child PRs are listed below for easier digesting. Note that some of these PRs are built on other PRs, so the deltas in the links below may be outdated. Cross reference with this PR when in doubt.
### UI
- Reactify Role Management Screen: https://github.com/elastic/kibana/pull/19035
- Space Aware Privileges UI: https://github.com/elastic/kibana/pull/21049
- Space Selector (in Kibana Nav): https://github.com/elastic/kibana/pull/19497
- Recently viewed Widget: https://github.com/elastic/kibana/pull/22492
- Support Space rename/delete: https://github.com/elastic/kibana/pull/22586
### Saved Objects Client
- ~~Space Aware Saved Objects: https://github.com/elastic/kibana/pull/18862~~
- ~~Add Space ID to document id: https://github.com/elastic/kibana/pull/21372~~
- Saved object namespaces (supercedes #18862 and #21372): https://github.com/elastic/kibana/pull/22357
- Securing saved objects: https://github.com/elastic/kibana/pull/21995
- Dedicated Spaces client (w/ security): https://github.com/elastic/kibana/pull/21995
### Other
- Public Spaces API (experimental): https://github.com/elastic/kibana/pull/22501
- Telemetry: https://github.com/elastic/kibana/pull/20581
- Reporting: https://github.com/elastic/kibana/pull/21457
- Spencer's original Spaces work: https://github.com/elastic/kibana/pull/18664
- Expose `spaceId` to "Add Data" tutorials: https://github.com/elastic/kibana/pull/22760Closes#18948
"Release Note: Create spaces within Kibana to organize dashboards, visualizations, and other saved objects. Secure access to each space when X-Pack Security is enabled"
* Not working proto code
* More proto code
* Work in progress
* Just go back to non interactive searching, much easier
* This should be on the server
* Revert "[@kbn/ui-framework] move ui-framework to a package (#17085)"
This reverts commit ef3339bd7a.
* Revert "Revert "[@kbn/ui-framework] move ui-framework to a package (#17085)""
This reverts commit ce9ce14e1060c426090b55a5367de3ff4329e681.
* Use BasicTable properly
* Table improvements
* Small tweaks to the table
* Improvements
* Flyout mostly working
* Remove in memory table
* Getting close
* Tweaks
* Revamping server code, still need to support editing
* Progress
* Fix export
* Updates and passing functional tests
* Better links in relationships flyout
* Add skip import option
* Fixes around importing and removing unnecessary code
* Remove tags for now
* Tests for lib/
* Some fixes
* Ensure we clear index pattern cache
* Parity with master
* Revert any changes in package.json
* Reset any changes in this file
* Move the new argumen to the end to prevent test failures
* Fix functional tests
* Add relationship tests
* Fix tests
* API integration tests for relationships
* Ensure we're properly waiting for things to happen
* Fix test issue
* Wait for the table to finish loading instead of the whole page
* Tests for objects_table
* Componentry tests
* Ensure this is grabbing the right field
* Update snapshot
* Fixes with importing index patterns
* PR feedback
* PR feedback
* PR feedback
* Update snapshot
* PR feedback
* Update snapshot
* Respect the savedObjects:perPage config
* Updates from PR feedback
* More updates from PR feedback
* Make this more efficient
* Add debugging for functional test failures
* Wait longer
* Wrap each button accessor with a retry.try
* Try wrapping this in a retry.try
* Debug
* Lets make sure it is visible
* Maybe the short timeout is affecting this - use the default timeout which should be higher and allow more time for the animation to finish
* Rewrite this per suggestions from stacey
Integration tests should go into the corresponding directory under
`test` and should use the appropriate testing framework that is designed
for long-running tests like those. Tests under __tests__ directories
should be fast-running unit tests only.
* [savedObjects/delete+bulk_get] add failing tests
* [savedObjects/delete+bulk_get] improve 404 handling
* [savedObjects/client] fix mocha tests
* [savedObjects/tests] remove extra test wrapper
* [apiIntegration/kbnServer] basically disable es healthcheck
* [savedObjects/create] add integration test
* [savedObjects/find] add failing integration tests
* [savedObjects/find] fix failing test
* [savedObjects/client] explain reason for generic 404s
* [savedObjects/get] add integration tests
* [savedObjects/find] test request with unkown type
* [savedObjects/find] add some more weird param tests
* [savedObjects/find] test that weird params pass when no index
* [savedObjects/update] use generic 404
* fix typos
* [savedObjects/update] add integration tests
* remove debugging uncomment
* [savedObjects/tests] move backup kibana index delete out of tests
* [savedObjects/tests/esArchives] remove logstash data
* [savedObjects] update test
* [uiSettings] remove detailed previously leaked from API
* [functional/dashboard] wrap check that is only failing on Jenkins
* [savedObjects/error] replace decorateNotFound with createGenericNotFound
* fix typo
* [savedObjectsClient/errors] fix decorateEsError() test
* [savedObjectsClient] fix typos
* [savedObjects/tests/functional] delete document that would normally exist
* [savedObjectsClient/tests] use sinon assertions
* [savedObjects/apiTests] create without index responds with 503 after #14202
* Re-enable filter editor suggestions
* Use search instead of include
* Escape query
* Show spinner
* Use include rather than search
* Add additional regex and explanation for parameters
* Add suggestions API test
* Make sure test actually runs
* Use send instead of query
* Fix suggestions API test
* Ensure that conflict fields can be searchable and/or aggregatable in the UI
* Use `some` instead of `reduce`
* Revert UI changes
* Attempt to convert multiple ES types to kibana types, and if they all resolve to the same kibana type, there is no conflict
* Add comma back
* Cleaner code
* Add tests
* Update failing test to handle searchable and aggregatable properly
* Add functional test to ensure similar ES types are properly merged
* Update tests
* Revert shard size
* [indexPatterns] support cross cluster patterns
* [vis] remove unused `hasTimeField` param
* [indexPatterns/create] fix method name in view
* [indexPatterns/create] disallow expanding with ccs
* [indexPatterns/create] field fetching is cheaper, react faster
* [indexPatterns/resolveTimePattern/tests] increase readability
* [tests/apiIntegration/indexPatterns] test conflict field output
* [indexPatterns/fieldCaps/readFieldCapsResponse] add unit tests
* [test/apiIntegration] ensure random word will not be valid
* [indexPatterns/ui/client] remove unused import
* remove use of auto-release-sinon
* [indexPatterns/create] don't allow expand when cross cluster
* [indexPatternsApiClient/stub] use angular promises
* [indexPatterns/create] add tests for base create ui behaviors
