* Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (#85778)""
This reverts commit f97958043f.
* Fix flaky saved objects management test #89953
* If a clone target exists, wait for yellow, not green, index status
* Fix test after master merge
* Fix types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Search Sessions: Unskip Flaky Functional Test
* Save all search sessions and then manage them based on their persisted state
* Get default search session expiration from config
* randomize sleep time
* fix test
* fix test
* Make sure we poll, and dont persist, searches not in the context of a session
* Added keepalive unit tests
* fix ts
* code review @lukasolson
* ts
* More tests, rename onScreenTimeout to completedTimeout
* lint
* lint
* Delete async seaches
* Support saved object pagination
Fix get search status tests
* better PersistedSearchSessionSavedObjectAttributes ts
* test titles
* Remove runAt from monitoring task
Increase testing trackingInterval (caused bug)
* support workload histograms that take into account overdue tasks
* Update touched when changing session status to complete \ error
* removed test
* Updated management test data
* Rename configs
* delete tap first
add comments
* Use sync config in data-enhanced plugin
* fix merge
* fix merge
* ts
* code review
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Anton Dosov <anton.dosov@elastic.co>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
* chore(NA): simple changes on bazelrc
* chore(NA): integrate bazel tools with BuildBuddy and remote cache service
* chore(NA) fix bazelrc line config
* chore(NA): move non auth settings out of bazelrc.auth
* chore(NA): output home dir
* chore(NA): load .bazelrc-ci.auth from /Users/tiagocosta dir
* chore(NA): remove bazelrc auth file and append directly into home bazelrc
* chore(NA): comment announce option
* chore(NA): integrate build buddy metadata
* chore(NA): update src/dev/ci_setup/.bazelrc-ci
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
* chore(NA): move build metadata integation to common confdig
* chore(NA): fix problem on bazel file location
* chore(NA): correct sh file permissions
* chore(NA): only get host on CI
* chore(NA): add cores into host info on CI
* chore(NA): sync with last settings to setup bazelisk tools on ci
* chore(NA): sync last changes on ci setup env
* chore(NA): sync settings on ci setup with the other PR
* chore(NA): remove yarn export
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
* Convert user values back to string after yaml template compilation if they were strings originally
* Add better test cases and adjust patch
* Fix when field is undefined
* Handle array of strings too
* chore(NA): build bazel projects all at once in the distributable build process
* chore(NA): make sure bazelisk is installed
* chore(NA): install bazelisk using npm
* chore(NA): remove extra spac
* chore(NA): test yarn path exports
* chore(NA): add direct global dir
* chore(NA): some more debug steps
* chore(NA): remove one statement
* chore(NA): comment one more line out for testing purposes
* chore(NA): export the correct yarn bin location into the PATH
* chore(NA): cleaning implementation
* chore(NA): move installation process of bazelisk into npm
* chore(NA): add missing type
## [Security Solution] [Timeline] Endpoint row renderers (1st batch)
This PR implements the 1st batch of Endpoint (`event.module: "endpoint"`) row renderers by updating and enhancing some of the existing "Endgame" (`event.module: "endgame"`) row renderers to use the latest [ECS fields](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html).
The following Endpoint events will be rendered via row renderers in Timeline:
| event.dataset | event.action |
|--------------------------|---------------------|
| endpoint.events.file | creation |
| endpoint.events.file | deletion |
| endpoint.events.process | start |
| endpoint.events.process | end |
| endpoint.events.network | lookup_requested |
| endpoint.events.network | lookup_result |
| endpoint.events.network | connection_accepted |
| endpoint.events.network | disconnect_received |
| endpoint.events.security | log_on |
| endpoint.events.security | log_off |
## File (FIM) Creation events
Endpoint File (FIM) Creation events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.file and event.action: creation
```
### Sample rendered File (FIM) Creation event
Each field with `this formatting` is draggable (to pivot a search) in the row-rendered event:
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` created a file `WimProvider.dll` in `C:\Windows\TEMP\F590BACBAE94\WimProvider.dll` via `MsMpEng.exe` `(2424)`
### Fields in a File (FIM) Creation event
`user.name` \ `user.domain` @ `host.name` created a file `file.name` in `file.path` via `process.name` `(process.pid)`
## File (FIM) Deletion events
Endpoint File (FIM) Deletion events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.file and event.action: deletion
```
### Sample rendered File (FIM) Deletion event
`SYSTEM` \ `NT AUTHORITY` @ `windows-endpoint-1` deleted a file `AM_Delta_Patch_1.329.2793.0.exe` in `C:\Windows\SoftwareDistribution\Download\Install\AM_Delta_Patch_1.329.2793.0.exe` via `svchost.exe` `(1728)`
### Fields in a File (FIM) Deletion event
`user.name` \ `user.domain` @ `host.name` deleted a file `file.name` in `file.path` via `process.name` `(process.pid)`
## Process Start events
Endpoint Process Start events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.process and event.action: start
```
### Sample rendered Process Start event
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` started process `conhost.exe` (`376`) `C:\Windows\system32\conhost.exe` `0xffffffff` `-ForceV1` via parent process `sshd.exe` (`6460`)
`sha256 697334c236cce7d4c9e223146ee683a1219adced9729d4ae771fd6a1502a6b63`
`sha1 e19da2c35ba1c38adf12d1a472c1fcf1f1a811a7`
`md5 1b0e9b5fcb62de0787235ecca560b610`
### Fields in a Process Start event
The following fields will be used to render a Process Start event:
`user.name` \ `user.domain` @ `host.name` started process `process.name` (`process.pid`) `process.args` via parent process `process.parent.name` (`process.parent.pid`)
`process.hash.sha256`
`process.hash.sha1`
`process.hash.md5`
## Process End events
Endpoint Process End events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.process and event.action: end
```
### Sample rendered Process End event
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` terminated process `svchost.exe` (`10392`) `C:\Windows\System32\svchost.exe` `-k` `netsvcs` `-p` `-s` `NetSetupSvc` with exit code `0` via parent process `services.exe` `(568)`
`7fd065bac18c5278777ae44908101cdfed72d26fa741367f0ad4d02020787ab6`
`a1385ce20ad79f55df235effd9780c31442aa234`
`8a0a29438052faed8a2532da50455756`
### Fields in a Process End event
The following fields will be used to render a Process End event:
`user.name` \ `user.domain` @ `host.name` terminated process `process.name` (`process.pid`) with exit code `process.exit_code` via parent process `process.parent.name` (`process.parent.pid`)
`process.hash.sha256`
`process.hash.sha1`
`process.hash.md5`
## Network (DNS) Lookup Requested events
Endpoint Network (DNS) Lookup Requested events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.network and event.action: lookup_requested
```
### Runtime matching criteria
All Network Lookup Requested events, including Endpoint and non-Endpoint DNS events matching the following criteria will be rendered:
```
dns.question.type: * and dns.question.name: *
```
### Sample rendered Network Lookup Requested event
`SYSTEM` \ `NT AUTHORITY` @ `windows-endpoint-1` asked for `logging.googleapis.com` with question type `A` via `google_osconfig_agent.exe` `(4064)` `dns`
### Fields in a Network Lookup Requested event
The following fields will be used to render a Network Lookup Request event:
`user.name` \ `user.domain` @ `host.name` asked for `dns.question.name` with question type `dns.question.type` via `process.name` `(process.pid)` `network.protocol`
## Network Lookup Result events
Endpoint Network (DNS) Lookup Result events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.network and event.action: lookup_result
```
### Runtime matching criteria
All Network Lookup Result events, including Endpoint and non-Endpoint DNS events matching the following criteria will be rendered:
```
dns.question.type: * and dns.question.name: *
```
### Sample rendered Network Lookup Result event
`SYSTEM` \ `NT AUTHORITY` @ `windows-endpoint-1` asked for `logging.googleapis.com` with question type `AAAA` via `GCEWindowsAgent.exe` `(684)` `dns`
### Fields in a Network Lookup Result event
The following fields will be used to render a Network Lookup Result event:
`user.name` \ `user.domain` @ `host.name` asked for `dns.question.name` with question type `dns.question.type` via `process.name` `(process.pid)` `network.protocol`
## Network Connection Accepted events
Endpoint Network Connection Accepted events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.network and event.action: connection_accepted
````
### Sample rendered Network Connection Accepted event
Network Connection Accepted events, like the one in the screenshot above, are also rendered by the _Netflow_ row renderer, which displays information that includes the directionality of the connection, protocol, and source / destination details.
`NETWORK SERVICE` \ `NT AUTHORITY` @ `windows-endpoint-1` accepted a connection via `svchost.exe` `(328)` with result `success`
### Fields in a Network Connection Accepted event
`user.name` \ `user.domain` @ `host.name` accepted a connection via `process.name` `(process.pid)` with result `event.outcome`
## Network Disconnect Received events
Endpoint Network Disconnect Received events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.network and event.action: disconnect_received
````
### Sample rendered Network Disconnect Received event
Network Disconnect Received events, like the one in the screenshot above, are also rendered by the _Netflow_ row renderer, which displays information that includes the directionality of the connection, protocol, and source / destination details.
`NETWORK SERVICE` \ `NT AUTHORITY` @ `windows-endpoint-1` disconnected via `svchost.exe` `(328)`
### Fields in a Network Disconnect Received event
`user.name` \ `user.domain` @ `host.name` disconnected via `process.name` `(process.pid)`
## Security Log On events
Endpoint Security Log On events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.security and event.action: log_on
```
### `event.outcome: "success"` vs `event.outcome: "failure"`
The row renderer for Security Log On events uses the `event.outcome` field to display different results for events matching:
```
event.dataset: endpoint.events.security and event.action: log_on and event.outcome: success
```
vs events matching:
```
event.dataset: endpoint.events.security and event.action: log_on and event.outcome: failure
```
### Sample rendered Security Log On / `event.outcome: "success"` event
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` successfully logged in via `C:\Program Files\OpenSSH-Win64\sshd.exe`
### Fields in an Security Log On / `event.outcome: "success"` event
`user.name` \ `user.domain` @ `host.name` successfully logged in via `process.name` (`process.pid`)
### Sample rendered Security Log On / `event.outcome: "failure"` event
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` failed to log in via `C:\Program Files\OpenSSH-Win64\sshd.exe`
### Fields in an Security Log On / `event.outcome: "failure"` event
`user.name` \ `user.domain` @ `host.name` failed to log in via `process.name` (`process.pid`)
## Security Log Off events
Endpoint Security Log Off events with the following `event.dataset` and `event.action` will be rendered in Timeline via row renderers:
```
event.dataset: endpoint.events.security and event.action: log_off
```
### Sample rendered Security Log Off event
`SYSTEM` \ `NT AUTHORITY` @ `win2019-endpoint` logged off via `C:\Program Files\OpenSSH-Win64\sshd.exe`
### Fields in a Security Log Off event
`user.name` \ `user.domain` @ `host.name` logged off via `process.name` (`process.pid`)
* Add eslint rule for linting unnecessary backticks
This needs to be below the Prettier overrides at the bottom of the file to override Prettier
* Run --fix
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Setup] Split rule that explicitly allows `any` in test/mock files into its own section
- so that the rules we're about to add apply correctly to all files
* Add react/jsx-boolean-value rule
* Run --fix
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Visualize] Removes the dashboard callout for users without permission
* Check if the user has the createNew permission
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* inital setup and experiments.
* Cast into ML job metric.
* Update mappings file.
* small refactor. add basic test to build on.
* mock out anomoly detector for testing from the usage collector.
* [PH JD] collect first set of ml job stats.
* Update telemetry schema.
* Include create and finished time.
* Cache datafeed calls and find / filter by naming convention.
* Fix jest test temp.
* [PH JD] Add datafeed to the usage collector payload.
* Get e2e test working.
* Update time complexity detail / df stats lookup. O(n) -> O(1)
* Update var names.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds e2e tests for https://github.com/elastic/kibana/pull/90326
* Adds e2 tests and backfills for updating actions and expected behaviors
* Adds two tests that would fail without the fix and if a regression happens this will trigger on the regression
* Adds two tests to the PATCH for exception lists even though there is no regression there. Reason is to prevent an accidental issue there.
* Adds tests to ensure the version number does not accidentally get bumped if PATCH or UPDATE is called on actions or exceptions for immutable rules.
* Adds utilities for cutting down noise.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Fixes regression: https://github.com/elastic/kibana/issues/90319 that has not been released where in some cases such as adding actions to a rule through an update we can and will update an immutable rule and do not expect the immutable to turn into a mutable through the tags.
Simple one-liner fix, I will update in a follow on PR with a regression test for this particular use case of actions but not with this one since we optimizing for speed of pull request to back-port.
Criticality is high and impact is high as this is data bug which can cause a lot of headaches and migrations if this goes out.
### Checklist
No unit test for this one, but a functional test will be added in a follow up
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* [maps] Top hits per entity--change to title to use recent, minor edits
* Updated TopHitsPerEntity title and description to use the term relevant
* updating top hits per entity topic to new title
Co-authored-by: Kent Marten <kmartastic@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduces the concept of a managed agent policy. Resolves most of the acceptance criteria from #76843. Remaining to be done in follow up PRs
- [x] Define hosted Agent Policy concept in Fleet.
- [x] Flag in policy? **_yes, added `is_managed: boolean`_ in agent policy SO**
- [x] Should not built only for cloud, an admin should be able to set theses restrictions.
- [x] We should have an API to configure it _**Can `POST` and `PUT` to `/api/fleet/agent_policies/{policy_id}`**_
- [x] Integration should be editable, we expect integration author to do the right thing and limit what can be edited.
- [x] Research if we can ensure the right behavior of Hosted Agent policy and restrict the super user.
- [ ] Capabilities restrictions
- [ ] An Agent enrolled in an Hosted Agent policy should not be able to be upgraded.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be unenrolled.
- [ ] No Agents cannot be enrolled into this policy by the user.
- Hide the enrollment key?
- Need to figure out the workflow.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be reassigned to a different configuration.
- [x] As a user I should be prevented to do theses action. _**No user-level checks. Only Agent Policy. No UI changes, but API errors are shown for failed actions like reassigning**_
- [x] As an API user I should receive error messages.
- [x] If making a single "flag" is easier/faster let's do it. _**Currently single `is_managed` property on agent policy SO.**_
Checks are implemented in service layer (is agent enrolled in a managed policy?)
No UI-specific changes added but UI is affected because HTTP requests (like `api/fleet/agents/{agentId}/reassign`) can fail. See screenshots below.
Tests at service (`yarn test:jest`) and http (`yarn test ftr`) layers for each of create policy, update policy, unenroll agent, and reassign agent
Bulk actions currently filter out restricted items. A follow-up PR will change them to throw an error and cause the request to fail.
## Managed Policy
Can create (`POST`) and update (`PUT`) an agent policy with an `is_managed` property. Each new saved object will have an `is_managed` property (default `false`)
<details><summary>HTTP commands</summary>
#### Create (`is_managed: false` by default)
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"edc236a0-5cbb-11eb-ab2c-0134aecb4ce8","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-01-22T14:12:58.250Z","updated_by":"elastic"}}
```
#### Create with `is_managed: true`
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-02-03T14:45:06.059Z","updated_by":"elastic"}}
```
#### Update with `is_managed: true`
```
curl --user elastic:changeme -X PUT -H 'Content-Type: application/json' -H 'kbn-xsrf: 1234' localhost:5601/api/fleet/agent_policies/67c785b0-662e-11eb-bf6b-4790dc0178c0 -d '{ "name":"User created policy","namespace":"default","is_managed":true }'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":true,"revision":2,"updated_at":"2021-02-03T14:47:28.471Z","updated_by":"elastic","package_policies":[]}}
```
</details>
## Enroll behavior
is not changed/addressed in this PR. Agents can still be enrolled in managed policies
## Unenroll Agent from managed policy behavior
#### Enrolled in managed agent policy, cannot be unenrolled
```
curl --user elastic:changeme -X POST http://localhost:5601/api/fleet/agents/441d4a40-6710-11eb-8f57-db14e8e41cff/unenroll -H 'kbn-xsrf: 1234' | jq
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot unenroll 441d4a40-6710-11eb-8f57-db14e8e41cff from a managed agent policy af9b4970-6701-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots for managed & unmanaged policies</summary>
#### Enrolled in managed agent policy, cannot be unenrolled
<img width="1931" alt="Screen Shot 2021-01-19 at 1 22 53 PM" src="https://user-images.githubusercontent.com/57655/105081614-67d05980-5a60-11eb-8faa-07e4e722a5b5.png">
<img width="1199" alt="Screen Shot 2021-01-19 at 1 30 26 PM" src="https://user-images.githubusercontent.com/57655/105081617-67d05980-5a60-11eb-9099-832dc6e04eca.png">
<img width="1971" alt="Screen Shot 2021-01-19 at 1 30 42 PM" src="https://user-images.githubusercontent.com/57655/105081618-67d05980-5a60-11eb-9a84-b80b6295ba19.png">
#### Enrolled agent policy is not managed, agent can be unenrolled<img width="1917" alt="Screen Shot 2021-01-19 at 1 44 12 PM" src="https://user-images.githubusercontent.com/57655/105081951-e3caa180-5a60-11eb-9308-7741b8986e8e.png">
<img width="2183" alt="Screen Shot 2021-01-19 at 1 44 19 PM" src="https://user-images.githubusercontent.com/57655/105081952-e3caa180-5a60-11eb-9833-1c721be0a107.png">
</details>
## Reassign agent
#### No agent can be reassigned to a managed policy
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent to managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1350" alt="Screen Shot 2021-02-04 at 2 14 51 PM" src="https://user-images.githubusercontent.com/57655/106943490-8044a300-66f3-11eb-9d2c-4b1ceef2e783.png">
</details>
#### Enrolled in managed agent policy, cannot be reassigned
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent from managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1364" alt="Screen Shot 2021-01-19 at 2 58 38 PM" src="https://user-images.githubusercontent.com/57655/105086737-72dab800-5a67-11eb-8f5e-93cd7768b914.png">
<img width="1367" alt="Screen Shot 2021-01-19 at 2 58 44 PM" src="https://user-images.githubusercontent.com/57655/105086740-73734e80-5a67-11eb-8ef9-9c7005a0a4ea.png">
<img width="623" alt="Screen Shot 2021-01-19 at 2 59 27 PM" src="https://user-images.githubusercontent.com/57655/105086741-740be500-5a67-11eb-8fc2-721f8b5d178a.png">
</details>
#### Enrolled agent policy is unmanaged, agent can be reassigned to another unmanaged policy
<details><summary>Screenshots</summary>
<img width="1368" alt="Screen Shot 2021-01-19 at 3 00 01 PM" src="https://user-images.githubusercontent.com/57655/105086754-78d09900-5a67-11eb-86a5-9e3ac02d6e1f.png">
<img width="1363" alt="Screen Shot 2021-01-19 at 3 00 08 PM" src="https://user-images.githubusercontent.com/57655/105086761-7a01c600-5a67-11eb-991d-acf994e2a393.png">
<img width="625" alt="Screen Shot 2021-01-19 at 3 00 46 PM" src="https://user-images.githubusercontent.com/57655/105086764-7a9a5c80-5a67-11eb-8290-e79648d01579.png">
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
