## [SIEM] Remove placeholder from pinned event tooltips
Similar to signals, pinned timeline events should be copied from source indexes, which are subject to ILM, to separate (space-aware) indexes (with different ILM), such that pinned events can be viewed in a timeline after the events have aged out of the original indexes.
The backend APIs and UI patterns in development now for signals can likely be reused to implement the above, but until then, the placeholder tooltip text for unpinned / pinned events, which mentions persistence, should be removed from the SIEM beta.
- [x] Changed the _unpinned_ event tooltip text from (sic) `This is event is NOT persisted with the timeline` to `Unpinned event`
- [x] Changed the pinned event tooltip text from `This event is persisted with the timeline` to `Pinned event`
https://github.com/elastic/siem-team/issues/482
Part of #44151,
Continuation of #51835,
Just moves existing state related url utils to kibana_utils plugin
Also fixes small regression introduced in #51835,
When sharing hashed url directly it should show error toast instead of full page fatal error
## Summary
Changes the signals output index to be based on the user's space
* Adds the ability to create a space based index through `POST /api/detection_engine/index`
* Adds the existence API for the index through `HEAD /api/detection_engine/index`
* Adds an index check during the creation of a rule, `POST api/detection_engine/rules` that will return a status of 400 with an error message if the index does not exist
* Adds a new optional key in kibana.dev.yml of `xpack.siem.signalsIndex` for developers working together who need to segregate signals indexes.
* Splits apart the ECS mappings and the signal mappings into separate files for easier maintenance.
* Deprecates the defaultSignalsIndex (will remove it once the UI is updated)
* Updates the README.md to remove the SIGNALS_INDEX environment variable
* Updates the existing unit tests
* Adds more unit tests unit tests
For people writing the UI:
---
How do I check for the existence of a signals index?
See [scripts/signal_index_exists.sh](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/signal_index_exists.sh)
```sh
HEAD /api/detection_engine/index
```
How do I create a new signals index if my user has correct privileges?
See [scripts/post_signal_index.sh](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/post_signal_index.sh)
```sh
POST /api/detection_engine/index
```
How do I delete _everything_ of all signal indexes, policies, and templates for a particular space?
See [scripts/delete_signal_index.sh](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/delete_signal_index.sh)
```sh
DELETE /api/detection_engine/index
```
FAQ for people testing
---
What is the name of the index, policy, etc... per space?
If you're using the default space the index, policies, etc... will be:
```sh
.siem-signals-default
```
If you're using a custom space such as `test-space` they will be:
```sh
.siem-signals-test-space
```
If you set your `xpack.siem.signalsIndex` in your `kibana.dev.yml` to something such as:
```yml
xpack.siem.signalsIndex: .siem-signals-frank-hassanabad
```
And use the default space it will be:
```sh
.siem-signals-frank-hassanabad-default
```
And for a custom space such as `test-space` they will be:
```sh
.siem-signals-frank-hassanabad-test-space
```
What is the policy that is being set?
See: [signals_policy.json](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_policy.json)
```json
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "10gb",
"max_age": "7d"
}
}
}
}
}
}
```
What is the boot strap index that is being set look like?
See: [create_bootstrap_index.ts](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/index/create_bootstrap_index.ts)
You should see this when running:
```sh
./get_signal_index.sh | less
```
```json
".siem-signals-default-000001": {
"aliases": {
".siem-signals-default": {
"is_write_index": true
}
},
```
What is the template that is being set look like?
See: [get_signals_template.ts](28937ebe00/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/get_signals_template.ts)
You should see this at the bottom when running:
```sh
./get_signal_index.sh
```
```json
"settings": {
"index": {
"lifecycle": {
"name": ".siem-signals-default",
"rollover_alias": ".siem-signals-default"
},
"number_of_shards": "1",
"provided_name": ".siem-signals-default-000001",
"creation_date": "1575502837772",
"number_of_replicas": "1",
"uuid": "GB0h3AYRQD6AWl8OfNonJA",
"version": {
"created": "8000099"
}
}
}
```
For more in-depth of testing of spaces using dev tools of Kibana
---
Different testing scenarios involving having spaces set in the URL, vs not having spaces set. Also different testing scenarios involving having a developer based `xpack.siem.signalsIndex` being set vs not having one set and gettin the default of `.siem-signals`
With a default space and kibana.dev.yml setting of:
* xpack.siem.signalsIndex: .siem-signals-frank-hassanabad
You can use dev tools to check the results after doing a `./post_signal_index.sh`
``` sh
GET /_template/.siem-signals-frank-hassanabad-default
GET /.siem-signals-frank-hassanabad-default-000001
GET /_ilm/policy/.siem-signals-frank-hassanabad-default
GET /_alias/.siem-signals-frank-hassanabad-default
```
With a default space and no `kibana.dev.yml` setting, you can use dev tools to check the results after doing a `./post_signal_index.sh`
```sh
GET /.siem-signals-default
GET /_template/.siem-signals-default
GET /.siem-signals-default-000001
GET /_ilm/policy/.siem-signals-default
GET /_alias/.siem-signals-default
```
Setting a space through:
```sh
export SPACE_URL=/s/test-space
```
With a default space and `kibana.dev.yml` setting using a user name such as mine:
* xpack.siem.signalsIndex: .siem-signals-frank-hassanabad
You can use dev tools to check the results after doing a `./post_signal_index.sh`
```
GET /.siem-signals-frank-hassanabad-test-space
GET /_template/.siem-signals-frank-hassanabad-test-space
GET /.siem-signals-frank-hassanabad-test-space-000001
GET /_ilm/policy/.siem-signals-frank-hassanabad-test-space
GET /_alias/.siem-signals-frank-hassanabad-test-space
```
With a default space and no `kibana.dev.yml` setting, you can use dev tools to check the results after doing a `./post_signal_index.sh`
```
GET /.siem-signals-test-space
GET /_template/.siem-signals-test-space
GET /.siem-signals-default-test-space-000001
GET /_ilm/policy/.siem-signals-test-space
GET /_alias/.siem-signals-test-space
```
### Checklist
Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.
~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~
~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~
~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~
### For maintainers
~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
- [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
* Move a REST endpoint and the GQL endpoint to NP routing.
* Delete obsolete REST endpoint.
* Update remaining REST routes to work with NP router.
* Remove obsolete code, update some unit tests.
* Simplify route creation.
* Remove tests of API decommissioned API endpoint.
* Rename domain check.
* Make return shape of index pattern endpoint correspond to required NP resp body.
* Move validate to appropriate level of route definition object for monitor details endpoint.
* Update snapshot count route.
* Fix broken lint rule.
* Remove usages of Boom.
* Fix license router creation.
This instructs mapbox-gl to load its workers using a static script. It removes the corresponding child-src CSP-rule from the policy. It retains the worker-src rule as it is required by the ace-editor in Dev-tools.
* [ci/reportFailures] --dry-run is overloaded, split it up
* force some failures to verify the fix
* Revert "force some failures to verify the fix"
This reverts commit cf2a58e139.
* update readme to mention new flags
* remove unnecessary commas
* add job description input to creation form
* wip add description to expanded row
* add description to analytics list table
* update jest test
* description input to 2 rows and update types
* update type
* [xpack/ftr/common] follow conventions, expose a config file
In order to make xpack/test/common ftr config easier to work with, this updates it to follow the conventions established by most other configs of exposing all services from the services module as an object that can be easily merges into local services modules.
* common config file and FtrProviderConfig are unused and unnecessary
* turns out FtrProviderContext was used...
* [failed_tests_cli] update reports with links to github issues
* reorder test report hooks so that published Junit includes modified reports
* force failures and enable dry-run mode for debugging
* auto-switch to --dry-run when running in non-tracked branches/prs
* add --skip-junit-update flag to skip mutating the reports
* remove comma after URL to support auto-linking in Jenkins
* Revert "force failures and enable dry-run mode for debugging"
This reverts commit ac0c287a3f.
* fix method call
* extend TestResult to include relevence flag rather than wrapping
* fix createFailureIssue() tests
* make report messages more consistent, append when not dry-run
* rename module
* update snapshots to not contain valid xml
* don't send authorization header if no token defined
* merge with master modified fixtures
* [NP] Expose global config to the plugins
* globalConfig in Plugin context: expose read-only methods only
* SharedGlobalConfig rework + Moving pkg, fromRoot & path utils from legacy to NP
* Updated API docs
* Fix test references to the moved utils
* Replace zip with combineLatest
* Change tests to describe/it + remove "(deprecated)" from the test description
* Moving path files to a folder + exposing the config path in the contract
* deepFreeze the globalConfig in the pluginContext
* Fix types in tests with new path.config
* Move fromRoot and package_json utils to core/server/utils
* Rename globalConfig to legacy.globalConfig$
* path.config renamed to path.configDir (not renaming path.data because it might be a breaking change)
* Change configDir in mocker as well
* Fix test after config renamed to configDir
* Fix API docs conflicts
* Rename the path properties when exposing them
* path.configDir removed from the path config-schema
* Remove path.configDir. It is already in env.configs
* Add Migration documentation and examples
* Fix 'kibana/server' imports in the MIGRATION docs
resolves https://github.com/elastic/kibana/issues/50721
note this branch was previously merged into master and then reverted: https://github.com/elastic/kibana/pull/51489 (prior PR made shape changes this one didn't take into account)
Uses the same whitelist config value / utilities that the webhook
action already uses. Was already mentioned in the README doc
that email uses this whitelist config value :-)
Required a change to the functional tests to use a host already
whitelisted in config, made for the the webhook action tests.
Also realized some jest tests on email were bogus, so fixed those
(was passing `user` in config, which is invalid, and masking the
actual thing being tested).
* Port graphql query for snapshot to hapijs
* fix type
* Remove graphql from node details.
* Use boom for error handling.
* Remove unused imports
* Remove dead code. Merge types
* Fix tests
* Remove unused import
* Rename types
* Go all in on io-ts. Cleanup types
* Fix typecheck
* Move plugin init calls to new plugin file, update refs and bind to legacy var. Assume ref for license init
* Bind server calls to __LEGACY var and pass to plugin setup
* Expose map config externally
* Pass in NP features and licensing plugins to replace some xpack_main functionality
* Pass through legacy server config for routes dependency
* Expose map config for legacy plugins
* Remove unused legacy refs
* Update license check to conform with latest NP license practices
* Clean up
* Revert to old license check. Handle NP license checking in separate PR
* Add back the license check file
* add links to plugin manifest format
* remove obsolete comment
* address comments. add registry folder to file structure tree
* revert back not confirmed conventions
resolves https://github.com/elastic/kibana/issues/50721
Uses the same whitelist config value / utilities that the webhook
action already uses. Was already mentioned in the README doc
that email uses this whitelist config value :-)
Required a change to the functional tests to use a host already
whitelisted in config, made for the the webhook action tests.
Also realized some jest tests on email were bogus, so fixed those
(was passing `user` in config, which is invalid, and masking the
actual thing being tested).
