Commit graph

45309 commits

Author SHA1 Message Date
Matthias Wilhelm 45ec64f99e
[Discover][Context] Add functional test, convert to TypeScript (#108481) 2021-08-16 11:39:20 +02:00
Anton Dosov 72742d0d50
[FieldFormats] Example plugin (#108070) 2021-08-16 05:18:34 -04:00
Angela Chuang 7888c9cf5f
[Security Solution] Add reason field (#108449)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-15 22:27:54 -04:00
Michael Olorunnisola 565276a90d
[Security Solution][RAC] - Enable tGrid by default (#108632) 2021-08-15 22:10:24 -04:00
Patryk Kopyciński c347a7e5e5
[Osquery] Add ECS mapping editor (#107706) 2021-08-16 01:17:07 +02:00
Kevin Qualters 0828788b66
[RAC][Security Solution][Observability] Add the add to new case and add to existing case actions to observability alerts table (#108502)
* [RAC][Security Solution][Observability] Add the add to new case and add to existing case actions to observability alerts table

* Remove fake data and make features work with observability data format

* Remove console.log and unused translations

* Remove commented out code

* Remove unneeded copy pasta id, create initializeStore function in timelines

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-15 13:27:28 -04:00
Matthew Kime 2e68ce1a05
Developer guide - index patterns key concepts (#96017)
* add index patterns key concepts
2021-08-15 08:48:51 -05:00
Georgii Gorbachev 2ee11db1fa
[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation (#108115)
**Addresses:** https://github.com/elastic/kibana/issues/106421, https://github.com/elastic/kibana/issues/106428, https://github.com/elastic/kibana/issues/102089, https://github.com/elastic/kibana/issues/106433

## Summary

This PR focuses on consolidation of indexing implementations in `rule_registry` (https://github.com/elastic/kibana/issues/101016). It addresses some of the sub-tasks of the parent ticket.

- [x] Encapsulate index bootstrapping logic in a new improved API exposed by `RuleDataService`.
- [x] Enforce allowed values for the `datasetSuffix` on the API level.
- [x] Migrate plugins using the existing `RuleDataService` API to the improved one.
- [x] Make sure index names comply with design architecture.
    - https://github.com/elastic/kibana/issues/102089
- [x] Improve the API of `RuleDataClient`.
- [x] Enhance index bootstrapping: support custom ILM policy per index (`{registrationContext}.{datasetSuffix}`).
- [x] Enhance index bootstrapping: create index template per namespace and support rollovers properly
    - based on https://github.com/elastic/kibana/pull/107700
- [x] Enhance index bootstrapping: support secondary aliases
    - based on https://github.com/elastic/kibana/pull/107700
- [x] Remove `EventLogService` implementation
    - https://github.com/elastic/kibana/issues/106433

This will be addressed in follow-up PRs:

- [ ] Enhance index bootstrapping: implement suggestions for backwards compatibility (naming scheme for alias and backing indices; versioning).
- [ ] Enhance index bootstrapping: implement upgrades of existing index templates.
- [ ] Make index bootstrapping logic more robust. This _is partially addressed_ in this PR, but more improvements are needed.
- [ ] Change the way index prefix works.
- [ ] Add support for optional TS schema (static typing).
- [ ] Update `README` in `rule_registry`.

### Checklist

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-08-15 14:52:44 +02:00
liza-mae ee07d710d4
Update infra visual test (#108621) 2021-08-14 14:37:08 -06:00
Ece Özalp aa8ce39e03
[CTI] adds Range Picker to Threat Intel Tab (#107234) 2021-08-14 16:30:35 -04:00
Patryk Kopyciński 9d0a7b8394
[Osquery] Add Osquery tab to node details tabs (#104272) 2021-08-14 07:25:24 -04:00
Ryland Herrick a9844db461
[Security Solution][CTI] Update legacy CTI signals to latest ECS threat fields (#107988)
* WIP: Adding integration test

* Replace threat.indicator mappings with threat.enrichments mappings

The nested threat.indicator mappings were experimental, and replaced by
threat.enrichmentsin ECS 1.10. While these fields are also experimental,
they fix the conflict between CTI data's normal threat.indicator
mappings.

* Add threat.enrichments mappings to our signals template mappings

event.* is no longer nested within here; it was determined that event
fields were not relevant to enrichment. All relevant ECS fieldsets
(file, pe, etc) are now nested under threat.enrichments.

* Update snapshot with newest threat.enrichments mappings

This test is a snapshot of the actual mappings applied by our templates. Looks good to me!

* Update ECS types to match latest

We now have two threat fields we care about for CTI, for legacy and
official ECS.

* Add a basic test for behavior of legacy enriched signals.

They're still queryable by threat.indicator, meaning that any existing
dashboards will still work.

* WIP: First pass at a data migration for CTI signals

* Defines reindex script to move things around
* Adds integration tests to make sure the migration and new mappings
  work
* Need to test a few more things and verify corner cases
* Need to extract some helpers from tests

* Bump our template version to ensure devs roll over

Marshall bumped to 55, giving us 10 versions for 7.14.x updates.
However, devs would not otherwise roll over and get my mapping updates
without destroying their signals index and rebuilding (which is also not
the same thing, exactly), so this trades having one higher signals
version for a more streamlined dev workflow.

* More robust guard against data migration

We only attempt to migrate legacy enrichments if the document:

* is a signal from an indicator match rule
* has a `threat.indicator` field
* does not have a `threat.enrichments` field

* Minor reorder of operations to make logic clearer

* Add more assertions around our signals data migration

Tests a few more pieces of the resulting document, giving more
confidence that it's the correct transformation (and mappings).

This also modifies/anonymizes the data that was originally generated on
a work machine.

* Remove outdated note

This was for when these tests were driven via the UI; the API is more
responsive and now synchronization is currently needed here, beyond the
200 responses.

* Fix typo in comment

These fields are in ECS 1.11.

* Update snapshot test

We bumped the version previously, causing this test to become outdated.

* Update ECS typings in timelines plugin

These were copied from the security_solution plugin. I updated those,
but neglected to update these.

Until there's a better mechanism for deduplication here, I'm going to
kick the can and update both for now.

* Update enrichments logic to read/write from threat.enrichments

* indicator match rule logic
  * we now simply copy from the specified indicator path, and place that
    in `threat.enrichments.indicator`
* event enrichment API logic
  * We were previously returning fields from `indicator.*`, we now
    include the `indicator.*` suffix in order to be more consistent with
    the sibling `matched.*` fields
* row renderer logic
  * removal of dataset
  * updates relevant to API changes above

* Fix logical error in generating links from indicator fields

We want to link the reference field, not a `first_seen` field.

* Always include the indicator prefix in first-party indicator fields

Prior to this change we would display e.g. `threatintel.indicator.foo`
for investigation enrichment fields. Now that the structure has changed
slightly and we return both `indicator.*` and `matched.*` fields for
existing enrichents, we want to display investigation enrichment
similarly.

* Update indicator match rule integration tests

Now that we've updated our enrichment logic, we need to update our
enrichment tests.

* Remove unused translation

* Update example row renderer data for enriched alerts

* Update parallel CTI constants to get our CTI row renderer working

We were not requesting the necessary fields for our row renderer, since
these constants (specifically CTI_ROW_RENDERER_FIELDS) now exist in both
security_solution and the timelines plugin. I had updated the former,
but only the latter is actually used.

* Update CTI enrichment UI tests

* Update prepackaged threat timeline template with new threat fields

Also bumps the timelineTemplateVersion.

* Update Indicator Match rule tests

These needed three things:

* Update to timeline template (see previous commit)
* Changing expectations from `threat.indicator` to `threat.enrichments`
* Update row renderer expectation to exclude dataset

* Update mock data with newest CTI enrichment fields

* Fix assertion on our threat details

These fields are prefixed with `indicator` now because:

1. This data pertains to the indicator, not the match per se
2. The actual field is prefixed with indicator (or, it at least
   specifies an indicator in the case of a custom threat index (via
   threat_indicator_path))

* Update test data and tests for our field parsing helpers

* Update more event-parsing tests

Ths one involved updating a mock in another package.

* Modify our helper function to support old filebeat indicators

When we query indicators for enrichment matches, the current expectation
is that we'll be querying 7.14 filebeat modules, which have an indicator
path of 'threatintel.indicator'. The only place that matters on the UI
is on the threat intel panel, where these indicators come back with such
a prefix.

This change has one behavior: it brings back the `provider` field on the
Alert summary tab for queried enrichments from filebeat modules.

* Update variable and method names to be more consistent with internal terminology

Indicators come from a CTI index. Enrichments are the application of
indicator data to other documents, and contain both indicator fields and
matched context.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-14 01:57:23 -04:00
Davis Plumlee 5f947c2531
[Security Solution][Detections] Moves last updated info inline with status filter (#108096) 2021-08-14 01:24:00 -04:00
Yara Tercero 94d16f8882
[Security Solution] - remove alerts subfeature UI from behind feature flag (#108450)
### Summary
This removes the feature flag check for showing the alerts subfeature under security solution Kibana roles UI.
2021-08-14 00:12:31 -04:00
Sergi Massaneda a7661a553c
[TGrid] Alerts status update use RAC api (#108092)
Co-authored-by: Devin Hurley <devin.hurley@elastic.co>
2021-08-13 22:11:53 -04:00
liza-mae 78e7e40b77
Update map visual test (#108593)
* Update map visual test

* Up to latest
2021-08-13 19:35:01 -06:00
Caroline Horn e33cdc29c6
[Enterprise Search] Updated product_selector to match new No Data screens (#108592)
And updated product selector images to match new Kibana UI
2021-08-13 20:29:10 -04:00
CJ Cenizal bfea4a1c2b
Add EuiCodeEditor to ES UI Shared. (#108318)
* Export EuiCodeEditor from es_ui_shared and consume it in Grok Debugger. Remove warning from EuiCodeEditor.
* Lazy-load code editor so it doesn't bloat the EsUiShared plugin bundle.
* Refactor mocks into a shared jest_mock.tsx file.
2021-08-13 16:49:55 -07:00
Marshall Main 5ef1f95711
Add signal.original_event.reason to signal_extra_fields for insertion into old indices (#108594) 2021-08-13 18:51:23 -04:00
Ross Bell 1311fe38ae
Add Workplace Search sync controls UI (#108558)
* Wip

* Things are more broken, but closer to the end goal

* Get patch request working

* Update event type

* Other two toggles

* Force sync button

* Remove force sync button for now

* Disable the checkbox when globally disabled and introduce click to save

* Wip tests

* One test down

* Test for skipping name alert

* Linter

* Fix undefined check

* Prettier

* Apply suggestions from code review

Co-authored-by: Scotty Bollinger <scotty.bollinger@elastic.co>

* Refactor some structures into interfaces

* UI tweaks

Co-authored-by: Scotty Bollinger <scotty.bollinger@elastic.co>
2021-08-13 18:41:17 -04:00
Sébastien Loix b24d44d165
[Index pattern field editor] Add preview for runtime fields (#100198) 2021-08-13 18:27:23 -04:00
Clint Andrew Hall d78d66d424
[canvas] Create Labs Service; remove legacy service (#107354) 2021-08-13 17:31:37 -04:00
Ece Özalp f952643e54
[CTI] adds securitySoluion:defaultThreatIndex uiSetting (#108389) 2021-08-13 16:32:32 -04:00
Tim Sullivan 8ea29b6c5a
[Reporting] Implement a round of server code cleanup of types, unused files, and test mocks (#108480)
* rewrite createMockLevelLogger

* Remove redundant BaseParams references that were inlined

* remove unused files related to the 7.x compatibility shim

* remove unused legacy mock
2021-08-13 13:12:33 -07:00
Tim Sullivan 2fb785de64
[Reporting/Mgmt] Fix the missing deprecation warning under job status (#108484)
* [Reporting/Mgmt] Fix the missing deprecation warning under job status

* improve unit test

* add space before the text and update snapshots
2021-08-13 13:11:35 -07:00
Paul Tavares 3b31ffc5fd
[Security Solution][Endpoint] Improve logic for determining count of pending actions and reenable pending status on UI (#108114)
- Re-enable display of Pending isolation status on the UI along with a experimental feature flag to be able to turn it back off
- Improves the logic around determining if an isolation action has actually been processed by the Endpoint by looking for an endpoint metadata update whose `event.created` timestamp is more recent than the timestamp on the isolation Action Response. The goal is to minimize/avoid the UX around isolation where a user might not see the result from the Endpoint (isolated or released) after having seen the pending status on the UI.
- Add some protective code around our server side license watcher so that failures are not bubbled up and instead are logged to the kibana logs
- Added new `findHostMetadataForFleetAgents()` method to the `EndpointMetadataService`
- Added test mocks for `EndpointMetadataService` and tests for new method
2021-08-13 15:42:14 -04:00
Frank Hassanabad c922005644
Changes out cypress pipe (#108457)
## Summary

Reduces flake by changing out a Cypress pipe for a `cy.wait`. This UI element does unusual things that make it unfit for Cypress pipe such as multiple clicks against it will cause the component to have a dialog appear and disappear with transition effects which can make pipe not able to click once when the click handler is present.


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-08-13 13:41:17 -06:00
Nathan Reese 505043898e
[Maps] 'show this layer only' layer action (#107947)
* [Maps] 'show this layer only' layer action

* review feedback

* remove ts code from js file

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-13 13:38:57 -06:00
Apoorva Joshi c6dc6e207a
Adding host_risk_score_latest to the list of patterns to track for telemetry (#108547)
* Adding host_risk_score_latest to the list of patterns to track for telemetry

* Adding a test

* Removing extra spaces at end of line- should make the linter happy
2021-08-13 11:04:42 -07:00
Brandon Morelli 79eb426a8f
docs: Add anonymous auth to central config (#108285) 2021-08-13 10:59:01 -07:00
Tiago Costa 79f1e18686
chore(NA): moving @kbn/io-ts-utils to babel transpiler (#108517)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-13 18:39:35 +01:00
Tiago Costa 560bd0b57b
chore(NA): moving @kbn/es-archiver to babel transpiler (#108370)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-13 18:38:56 +01:00
Clint Andrew Hall 2d385b339d
[canvas] Fix setup server expressions cache; move to mount (#108473) 2021-08-13 13:19:16 -04:00
Clint Andrew Hall 44014c78b6
[canvas] Create Custom Elements Service (#107356)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-08-13 13:08:50 -04:00
Jason Stoltzfus f3e094c836
[App Search] Added a CrawlRulesTable to the Crawler view (#108458) 2021-08-13 13:08:46 -04:00
Joe Portner e35be9d87c
Fix flaky security/spaces tests (#108088) 2021-08-13 13:05:11 -04:00
Luke Elmers 57e395540d
[docs] Update Maps docs based on 7.14 UI. (#104762) 2021-08-13 10:51:38 -06:00
Marta Bondyra e33daccca3
[Lens] Disable the global timepicker for index pattern without primary timefield and visualizations without timefield (#108052) 2021-08-13 18:24:00 +02:00
Sandra G dd85150f73
[Monitoring] Convert elasticsearch_settings dir to typescript (#108112)
* convert elasticsearch_settings dir files to typescript

* fix type

* change tests to ts
2021-08-13 12:07:46 -04:00
Dima Arnautov fe3b7d61c8
[ML] Fix the Job audit messages service (#108526)
* [ML] refactor to ts

* [ML] fix types
2021-08-13 11:55:38 -04:00
Walter Rafelsberger fa3ae84b5b
[ML] APM Latency Correlations: Improve log log chart loading behavior and axis ticks. (#108211)
- Makes use of ChartContainer to improve the loading behavior of the log log chart to include a loading indicator.
- Improves y axis ticks for the log log chart. Will set the max y domain to the next rounded value with one more digit, for example, if the max y value is 4567, the y domain will be extended to 10000 and 10000 being the top tick. This makes sure we'll always have a top tick, fixes a bug where with low number <10 we'd end up with just a low 1 tick.
- Improves x axis ticks to support different time units.
2021-08-13 11:52:04 -04:00
Casper Hübertz 4a1366ca52
[APM] Change table SparkPlot content properties (#108516) 2021-08-13 17:01:56 +02:00
Tiago Costa 7dc24e65d6
chore(NA): upgrades bazel rules nodejs into v3.8.0 (#108471) 2021-08-13 15:24:51 +01:00
Brian Seeders 212b1898e6
Revert "[Enterprise Search] Set up basic scaffolding for Cypress tests in Kibana (#108309)" (#108541)
This reverts commit 4d7aa45e14.
2021-08-13 10:10:53 -04:00
Jason Stoltzfus 5bfba1b014
[App Search] Added a SitemapsTable to the Crawler view (#108405) 2021-08-13 10:00:48 -04:00
Ece Özalp a8b4433294
[CTI] Fixes AlienVaultOTX counts on the Overview page (#108448) 2021-08-13 09:50:43 -04:00
Dario Gieselaar a474a63a7f
[APM] Make environment & kuery required (#108338) 2021-08-13 09:50:14 -04:00
Tiago Costa 3c75b1faf5
skip flaky suite (#106660) 2021-08-13 14:15:45 +01:00
Kevin Qualters dcfb19fb53
Reposition the take action popover on scroll (#108475) 2021-08-13 08:51:52 -04:00
David Sánchez 37053e6a8d
[Security solution][Endpoint] Don't hide "add trusted app" button if we are checking if data exists and there was data before (#108373)
* Don't hide add button if we are checking if data exists and there was data before

* Moves duplicated code in a useCallback
2021-08-13 14:37:50 +02:00