* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
## Summary
This adds utilities and two strategies for merging using the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) and the `_source` document during signal generation. This gives us the ability to support `constant_keyword`, field alias value support, some runtime fields support, and `copy_to` support. Previously we did not copy any of these values and only generated signals based on the `_source` record values. This changes the behavior to allow us to copy some of the mentioned values above.
The folder of `source_fields_merging` contains a `strategy` folder and a `utils` folder which contains both the strategies and the utilities for this implementation. The two strategies are `merge_all_fields_with_source` and `merge_missing_fields_with_source`. The defaulted choice for this PR is we use `merge_missing_fields_with_source` and not the `merge_all_fields_with_source`. The reasoning is that this is much lower risk and lower behavior changes to the signals detection engine.
The main driving force behind this PR is that ECS has introduced `constant_keyword` and that field has the possibility of only showing up in the fields section of a document and not `_source` when index authors do not push the `constant_keyword` into the `_source` section. The secondary driving forces behind this behavioral change is that some users have been expecting their runtime fields, `copy_to` fields, and field alias values of their indexes to be copied into the signals index.
Both strategies of `merge_missing_fields_with_source` and `merge_all_fields_with_source` are considered Best Effort meaning that both strategies will not always merge as expected when they encounter ambiguous use cases as outlined in the `README.md` text at the top of `source_fields_merging` in detail.
The default used strategy of `merge_missing_fields_with_source` which has the simplest behavior will work in most common use cases. This is simply if the `_source` document is missing a value that is present in the `fields`, and the `fields` value is a primitive concrete value such as a `string` or `number` or `boolean` and the `_source` document does not contain an existing object or ambiguous array, then the value will be merged into `_source` and a new reference is returned. If you call the strategy twice it should be idempotent meaning that the second call will detect a value is now present in `_source` and not re-merge a second time.
* 301 unit tests were added
* Extensive README.md docs are added
* e2e tests are updated to test scenarios and ambiguity and conflicts from previously to support this effort.
* Other e2e tests were updated
* One bug with EQL and fields was found with a workaround implemented. See https://github.com/elastic/elasticsearch/issues/74582
* SearchTypes adjusted to use recursive TypeScript types
* Changed deprecated for `@deprecated` in a few spots
* Removed some `ts-expect-error` in favor of `??` in a few areas
* Added a new handling of epoch strings and tests to `detection_engine/signals/utils.ts` since fields returns `epoch_millis` as a string instead of as a number.
* Uses lodash safer set to reduce changes of prototype pollution
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Prototype pollution | Low | High | Used lodash safer set |
| Users which have existing rules that work, upgrade and now we do not generate signals due to bad merging of fields and _source | Mid | High | We use the safer strategy method, `merge_missing_fields_with_source `, that is lighter weight to start with. We might add a follow up PR which enables a key in Kibana to turn off merging of fields with source. We added extensive unit tests and e2e tests. However, unexpected unknowns and behaviors from runtime fields and fields API such as geo-points looking like nested fields or `epoch_milliseconds` being a string value or runtime fields allowing invalid values were uncovered and tests and utilities around that have been added which makes this PR risky |
| Found a bug with using fields and EQL which caused EQL rules to not run. | Low | High | Implemented workaround for tests to pass and created an Elastic ticket and communicated the bug to EQL developers. |
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
* cancel the previous session
* split to 3 tasks
* fixes
* cancellation
* updated tests
* split out and improve jest tests
* cleanup previous session properly
* don't fail delete and cancel if item was already cleaned up
* test
* test
* ignore resource_not_found_exception when deleting an already cleared \ expired async search
* jest
* update jest
* api int
* fix jest
* testssss
* Code review @dosant
* types
* remove any
* Fix merge
* type
* test
* jest
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Liza Katz <lizka.k@gmail.com>
Co-authored-by: Liza K <liza.katz@elastic.co>
* Automatically install and update the security_detection_engine package
* Remove security_detection_engine from required Fleet packages
* Update fleet package-registry image
* Add sha256: to the distribution package
* Use distribution from https://beats-ci.elastic.co/job/Ingest-manager/job/release-distribution/152
* Change fleet required packag
* Fix bad merge
* Update rules to 0.13.1 package
* Fix NOTICE.txt
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
* [Lens] Escape field names in formula
* Fix handling of partially typed fields with invalid chars
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Wylie Conlon <william.conlon@elastic.co>
* Reporting: Task Manager integration for 7.x
* update test assertions, use more explicit types
* update comment
* Update x-pack/plugins/reporting/server/lib/store/store.ts
Co-authored-by: Vadim Dalecky <streamich@gmail.com>
* fix field mapping
* Update x-pack/plugins/reporting/server/lib/store/store.ts
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
* Report also implements ReportDocumentHead
* the actual ID of the task is prefixed with `task:`
* remove pointless update to the report instance after failing
* comment clarification
* sync with 8.0
Co-authored-by: Vadim Dalecky <streamich@gmail.com>
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
resolves#98634
This adds a new object property to the event log kibana object named
task, with two properties to track the time the task was scheduled to
run, and the delay between when it was supposed to run and when it
actually started. This task property is only added to the appropriate
events.
task: schema.maybe(
schema.object({
scheduled: ecsDate(),
schedule_delay: ecsNumber(),
})
),
Note that these changes were previously merged to master in https://github.com/elastic/kibana/pull/102252 which had to be reverted - this PR contains the same commits, plus some additional ones to resolve the tests that were broken during the bad merge.
* wip
* First pass at standalone and embedded redux stores and usage
* wip
* First pass at standalone and embedded redux stores and usage
* wip
* clean up
* wip
* refact(NA): remove extra pkg_npm target and add specific target folders on @kbn/i18n
* cleanup
* - fixes type errors in tests
* WIP remove use_manage_timeline
* wip add query + selector
* finishing integrating timeline manage context from redux
* integrating t-grid in security solution
* fix RowRender type
* WIP begin to move components from package to plugin
* integration of t-grid inside of security solution
* wip to make redux work
* little trick to make it render
* - fixes a few type errors
* better integration betwen tgrid and security solutions
* bringing back tsconfig on timeline
* wip integration t-grid in observability
* fix types
* fix type in security solutions
* add type to import + trie dto get the bundle size as small as possible
* fix type in integration test
* fix type in integration test
* - fix tests
* clean up to use technical fields
* - fixes unit tests
* - mocks the `useDateFormat` function of the `useKibana` service to fix unit tests
* fix t-grid settings vs create timeline + fix inspect button
* fix last suites test
* Update unit tests, snapshots and lint
* Fix bad merge
* fix plugin export
* Fix some failing tests
* fix unit tets in timelines plugins
* fix latest test
* fix i18n
* free obs from t-grid
* Fix timeline functional plugin types
* fix store provider
* Update failing defaultHeader test
* Fix i18n usage in security solution
* Fix remaining i18n errors in timelines plugin
* Dedupe common shared types
* move drag and drop utils in package to avoid duplication
* More shared type cleanup
* add feature flag
* review I
* fix merge with master
* fix i18n translation
* More type deduping
* Use @kbn/common-utils, fix remaining types
* fix types
* fix tests
* missing type
* fix cypress tests
Co-authored-by: Kevin Qualters <kevin.qualters@elastic.co>
Co-authored-by: Tiago Costa <tiagoffcc@hotmail.com>
Co-authored-by: Andrew Goldstein <andrew.goldstein@elastic.co>
Co-authored-by: Kevin Qualters <kevin.qualters@elastic.co>
Co-authored-by: Tiago Costa <tiagoffcc@hotmail.com>
Co-authored-by: Andrew Goldstein <andrew.goldstein@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Closes#90454
Closes https://github.com/elastic/kibana/issues/72959
* Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings`
* When any package is installed, add a component template named `${templateName}@custom`
* Any of above templates also include a `_meta` property with `{ package: { name: packageName } }`
* On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object
* On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object
<details><summary>Kibana logs showing component templates added for package</summary>
```
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom]
│ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom]
```
</details>
<details><summary>screenshot - component templates are editable in the Stack Management UI</summary>
<img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png">
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: John Schulz <john.schulz@elastic.co>
* Allow restored session to run missing searches and show a warning
* tests and docs
* improve warning
* tests for new functionality
NoSearchIdInSessionError type
* managmeent tests
* Update texts
* fix search service pus
* link to docs
* imports
* format import
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Liza Katz <lizka.k@gmail.com>
* refactor cert alerts from batched to individual
* remove old translations
* create new certificate alert rule type and transition old cert rule type to legacy
* update translations
* maintain legacy tls rule UI to support legacy rule editing
* update translations
* update TLS alert content, rule type id, and alert instance id schema
* remove extraneous logic and format date content
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Dominique Clarke <doclarke71@gmail.com>
* Adding feature flag for auth
* Hiding SOs and adding consumer field
* First pass at adding security changes
* Consumer as the app's plugin ID
* Create addConsumerToSO migration helper
* Fix mapping's SO consumer
* Add test for CasesActions
* Declare hidden types on SO client
* Restructure integration tests
* Init spaces_only integration tests
* Implementing the cases security string
* Adding security plugin tests for cases
* Rough concept for authorization class
* Adding comments
* Fix merge
* Get requiredPrivileges for classes
* Check privillages
* Ensure that all classes are available
* Success if hasAllRequested is true
* Failure if hasAllRequested is false
* Adding schema updates for feature plugin
* Seperate basic from trial
* Enable SIR on integration tests
* Starting the plumbing for authorization in plugin
* Unit tests working
* Move find route logic to case client
* Create integration test helper functions
* Adding auth to create call
* Create getClassFilter helper
* Add class attribute to find request
* Create getFindAuthorizationFilter
* Ensure savedObject is authorized in find method
* Include fields for authorization
* Combine authorization filter with cases & subcases filter
* Fix isAuthorized flag
* Fix merge issue
* Create/delete spaces & users before and after tests
* Add more user and roles
* [Cases] Convert filters from strings to KueryNode (#95288)
* [Cases] RBAC: Rename class to scope (#95535)
* [Cases][RBAC] Rename scope to owner (#96035)
* [Cases] RBAC: Create & Find integration tests (#95511)
* [Cases] Cases client enchantment (#95923)
* [Cases] Authorization and Client Audit Logger (#95477)
* Starting audit logger
* Finishing auth audit logger
* Fixing tests and types
* Adding audit event creator
* Renaming class to scope
* Adding audit logger messages to create and find
* Adding comments and fixing import issue
* Fixing type errors
* Fixing tests and adding username to message
* Addressing PR feedback
* Removing unneccessary log and generating id
* Fixing module issue and remove expect.anything
* [Cases] Migrate sub cases routes to a client (#96461)
* Adding sub cases client
* Move sub case routes to case client
* Throw when attempting to access the sub cases client
* Fixing throw and removing user ans soclients
* [Cases] RBAC: Migrate routes' unit tests to integration tests (#96374)
Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
* [Cases] Move remaining HTTP functionality to client (#96507)
* Moving deletes and find for attachments
* Moving rest of comment apis
* Migrating configuration routes to client
* Finished moving routes, starting utils refactor
* Refactoring utilites and fixing integration tests
* Addressing PR feedback
* Fixing mocks and types
* Fixing integration tests
* Renaming status_stats
* Fixing test type errors
* Adding plugins to kibana.json
* Adding cases to required plugin
* [Cases] Refactoring authorization (#97483)
* Refactoring authorization
* Wrapping auth calls in helper for try catch
* Reverting name change
* Hardcoding the saved object types
* Switching ensure to owner array
* [Cases] Add authorization to configuration & cases routes (#97228)
* [Cases] Attachments RBAC (#97756)
* Starting rbac for comments
* Adding authorization to rest of comment apis
* Starting the comment rbac tests
* Fixing some of the rbac tests
* Adding some integration tests
* Starting patch tests
* Working tests for comments
* Working tests
* Fixing some tests
* Fixing type issues from pulling in master
* Fixing connector tests that only work in trial license
* Attempting to fix cypress
* Mock return of array for configure
* Fixing cypress test
* Cleaning up
* Addressing PR comments
* Reducing operations
* [Cases] Add RBAC to remaining Cases APIs (#98762)
* Starting rbac for comments
* Adding authorization to rest of comment apis
* Starting the comment rbac tests
* Fixing some of the rbac tests
* Adding some integration tests
* Starting patch tests
* Working tests for comments
* Working tests
* Fixing some tests
* Fixing type issues from pulling in master
* Fixing connector tests that only work in trial license
* Attempting to fix cypress
* Mock return of array for configure
* Fixing cypress test
* Cleaning up
* Working case update tests
* Addressing PR comments
* Reducing operations
* Working rbac push case tests
* Starting stats apis
* Working status tests
* User action tests and fixing migration errors
* Fixing type errors
* including error in message
* Addressing pr feedback
* Fixing some type errors
* [Cases] Add space only tests (#99409)
* Starting spaces tests
* Finishing space only tests
* Refactoring createCaseWithConnector
* Fixing spelling
* Addressing PR feedback and creating alert tests
* Fixing mocks
* [Cases] Add security only tests (#99679)
* Starting spaces tests
* Finishing space only tests
* Refactoring createCaseWithConnector
* Fixing spelling
* Addressing PR feedback and creating alert tests
* Fixing mocks
* Starting security only tests
* Adding remainder security only tests
* Using helper objects
* Fixing type error for null space
* Renaming utility variables
* Refactoring users and roles for security only tests
* Adding sub feature
* [Cases] Cleaning up the services and TODOs (#99723)
* Cleaning up the service intialization
* Fixing type errors
* Adding comments for the api
* Working test for cases client
* Fix type error
* Adding generated docs
* Adding more docs and cleaning up types
* Cleaning up readme
* More clean up and links
* Changing some file names
* Renaming docs
* Integration tests for cases privs and fixes (#100038)
* [Cases] RBAC on UI (#99478)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fixing case ids by alert id route call
* [Cases] Fixing UI feature permissions and adding UI tests (#100074)
* Integration tests for cases privs and fixes
* Fixing ui cases permissions and adding tests
* Adding test for collection failure and fixing jest
* Renaming variables
* Fixing type error
* Adding some comments
* Validate cases features
* Fix new schema
* Adding owner param for the status stats
* Fix get case status tests
* Adjusting permissions text and fixing status
* Address PR feedback
* Adding top level feature back
* Fixing feature privileges
* Renaming
* Removing uneeded else
* Fixing tests and adding cases merge tests
* [Cases][Security Solution] Basic license security solution API tests (#100925)
* Cleaning up the fixture plugins
* Adding basic feature test
* renaming to unsecuredSavedObjectsClient (#101215)
* [Cases] RBAC Refactoring audit logging (#100952)
* Refactoring audit logging
* Adding unit tests for authorization classes
* Addressing feedback and adding util tests
* return undefined on empty array
* fixing eslint
* conditional rendering the recently created cases
* Remove unnecessary Array.from
* Cleaning up overview page for permissions
* Fixing log message for attachments
* hiding add to cases button
* Disable the Cases app from the global nav
* Hide the add to cases button from detections
* Fixing merge
* Making progress on removing icons
* Hding edit icons on detail view
* Trying to get connector error msg tests working
* Removing test
* Disable error callouts
* Fixing spacing and removing cases tab one no read
* Adding read only badge
* Cleaning up and adding badge
* Wrapping in use effect
* Default toasting permissions errors
* Removing actions icon on comments
* Addressing feedback
* Fixing type
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP add filtering capabilities to XY legend
* Fix filter by legend on xy axis charts
* Filter pie and xy axis by legend
* create a shared component
* Add functional test
* Add functional test for pie
* Make the buttons keyboard accessible
* Fix functional test
* move function to retry
* Give another try
* Enable the rest od the tests
* Address PR comments
* Address PR comments
* Apply PR comments, fix popover label for alreadyformatted layers
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
