In 7.x, when saved objects point to an index and not an alias it will
assume it's pre-6.8 and perform a legacy migration. This causes issues
with the removal of oss builds. This update prevents that from happening
by using an alias.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
* updates the data used in the test
* adds matches test
* adds enrichment test
* improves speed and adds missing files
* fixes type check issue
* adds 'data-test-subj' for the json view tab
* refactor
* fixes typecheck issue
* updates tests with latest master changes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* update mocked index
* update mocked index
* update mocked index
* remove unused mock data
* update mock data
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
### Summary
This PR is a follow-up to #89066 - which fixed the same issue occurring with indicator match lists UI. The lack of stable ids for exception item entries resulted in some funky business by where invalid values could overwrite other values when deleting entries in the builder.
* update default index to match with description
* narrow down indices to match case description
* run specific tests
* run only security solution's tests
* debug users integration test
* wait_for_completion_timeout
* run only security_solution
* add back integration tests
* avoid multiple requests
* uncomment tests
* unskip tests
* add smaller mock data for users and uncommon_processes
* put test back
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes issues within Cypress whenever developers would re-run tests they could become blocked as some of the tests cause a migration/roll over of the signals.
To reproduce the error off of master:
Start Cypress
```ts
cd ./kibana/x-pack/plugins/security_solution
yarn cypress:open-as-ci
```
Then run the export test twice. The first time it will run ok. The second time the test will refuse to run.
<img width="585" alt="Screen Shot 2020-11-30 at 1 57 24 PM" src="https://user-images.githubusercontent.com/1151048/100683706-2233c200-3336-11eb-949f-48f86e884d8b.png">
The second time that you run the test you will get these errors from Cypress and will be blocked until you do workarounds such as logging into `https://localhost:5620` and manually changing the indexes manually or restarting Cypress altogether.
<img width="1067" alt="Screen Shot 2020-11-30 at 6 05 07 PM" src="https://user-images.githubusercontent.com/1151048/100683945-9a9a8300-3336-11eb-9d49-187dee4fc1dc.png">
What is going on with the errors is that a migration is occurring since the existing signals mapping for the tests are not the newer migrated version and these are creating a new index named `.siem-signals-default-0000002` like so:
<img width="1242" alt="Screen Shot 2020-11-30 at 1 57 12 PM" src="https://user-images.githubusercontent.com/1151048/100684145-fcf38380-3336-11eb-8fe1-b29d3462a1ae.png">
This index is not being cleaned up and when the next time we do an es_archive load we are marking two indexes as being writable
file: x-pack/test/security_solution_cypress/es_archives/alerts/mappings.json
```ts
".siem-signals-default": {
"is_write_index": true
}
},
"index": ".siem-signals-default-000001",
```
which leads to the stack trace and the Cypress errors on the front end:
```ts
MacBook-Pro.local] path: /.siem-signals-default-000001, params: {index=.siem-signals-default-000001}
│ java.lang.IllegalStateException: alias [.siem-signals-default] has more than one write index [.siem-signals-default-000002,.siem-signals-default-000001]
│ at org.elasticsearch.cluster.metadata.IndexAbstraction$Alias.computeAndValidateAliasProperties(IndexAbstraction.java:276) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
```
This fixes that by updating the mapping and migration number. Each time we migrate/change the signals mapping we will have to perform a PR like this to update each location.
At the moment this is 5 different locations we have to update with the latest mappings.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* improves 'Creates and activates a new custom rule' test
* fixes constant problem
* improves 'Creates and activates a new custom rule with override option' test
* improves 'Creates and activates a new threshold rule' test
* refactor
* fixes type check issue
* improves assertions
* removes unused code
* changes variables for constants
* improves 'waitForTheRuleToBeExecuted' test
* improves readability
* fixes jenkins error
* refactor
* blah
* more things
* finishes 'Creates an exception from rule details and deletes the excpetion' implementation
* implements 'Creates an exception from an alert and deletes the exception'
* updates VALUES_INPUT locator
* updates archiver
* refactor
* improves the code
* fixes CI error
* renames exceptions archive
* refactor
* fixes merge issue
* fixes CI issue
* debug
* refactor
* improves test data
* removes signals index after the execution
* removes unused line
* removes unused variable
* refactors 'numberOfauditbeatExceptionsAlerts' constant to camel case
* simplifies the archive
* waits for the rule to be executed after navigating to opened alerts tab
* cleaning data
* fixes tests flakiness
* cleans test data
* refactors code
* removes unsused archives
* cleans data
* simplifies data
* fixes CI issue
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fix threshold rule synthetic signal generation
* Use top_hits aggregation
* Find signals and aggregate over search terms
* Exclude dupes
* Fixes to algorithm
* Sync timestamps with events/signals
* Add timestampOverride
* Revert changes in signal creation
* Simplify query, return 10k buckets
* Account for when threshold.field is not supplied
* Ensure we're getting the last event when threshold.field is not provided
* Add missing import
* Handle case where threshold field not supplied
* Fix type errors
* Handle non-ECS fields
* Regorganize
* Address comments
* Fix type error
* Add unit test for buildBulkBody on threshold results
* Add threshold_count back to mapping (and deprecate)
* Timestamp fixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* shell scripts for creating roles + users for testing
* update readme's and updated privilege requirements based on testing with the users and inferring what the roles are supposed to do
* update role privileges based on feedback meeting yesterday
* updated scripts to accept filepath to role / user, added a test to ensure upload value list button is disabled
* updated role scripts to be parameterized
* adds login with role function and adds a sample test with a role to test that a t1 analyst user cannot upload a value list
* add object with corresponding roles
* fix spacing
* parameterize urls for basic auth with roles + users
* forgot to change the cy.visit string
* add KIBANA_URL env var for cli runner
* add env vars for curl script execution
* second script
* update readme's for each role and remove create_index from lists privilege for the soc manager role
* remove 'manage' cluster privilege for rule author
* remove 'create_index' privilege from soc_manager role since that is not parity with the security workflows spreadsheet
* update the login function logic with glo's feedback
* replace SIEM with Security Solution in markdown files
* make role param optional not just undefined
* remove unused file
* add copyright to scripts files
* update top-level README for roles scripts
* remove reference to internal spreadsheet and reference readme for this pr
* remove unnecessary -XPOST and remove verbose mode from post_detections_user script
* adds utils for running integration tests with other users and adds two sample tests showing example usage
* minor type updates and small refactor
* fix x-pack/test types
* use enum types instead of custom type
* fix path to json
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
* Adding alert.updatedAt field that only updates on user edit
* Updating unit tests
* Functional tests
* Updating alert attributes excluded from AAD
* Fixing test
* PR comments
* Unskipping tests and updating es archiver data
* Write failing status when executionStatus is in error
* adds unit test for error handling if rule status service throws an error
* adds success test for when executionStatus is failed
* moves logic for writing executionStatus failure to rule status saved object inside find rules status route, updates find rules route to display error if executionStatus is in error, but not be in charge of writing the status. That job belongs to the find rules status route.
* test if we are writing an error status when calls are made to find_rules_status_route and adds a test for general error checking
* adds JSDocs description for rules status route, updates findRules filter to append rule ids to the end of query, removes object.keys()
* don't write an error to our rule status in the route, only read from the executionStatus property and merge that result with our stored failures
* fixes tests
* move mock rule status service out of __mocks__ folder and remove unnecessary references to mock in tests
* fix type error
* updates json.gzip for cypress
* PR feedback
* fix timing issue with integration tests
* removes unzipped data.json
* Add cypress test around editing a detection rule
Right now this just navigates around and verifies that the form is
correctly repopulated; next step will be to modify/asset some changes.
* Add assertions for editing a rule
We already were asserting on the population of the Edit form after
creation; this additionally makes modifications, saves them, and asserts
the resulting values on the Rule Details page.
* Remove unused imports
* Inline our cypress expectations
So that expectation failures are less obfuscated, the decision was
previously made to abstract user navigation into functions, but to leave
expectations directly within the test body.
* Dynamically assert Rule Details based on titles
Rule Details are unfortunately unstructured: they're an array of <dt>s
and <dd>s without any hierarchy. To address this, tests
were previously hardcoding the order of these fields, and assertions
were performed by querying for all <dd>s and then indexing with the
hardcoded number (e.g. ABOUT_FALSE_POSITIVES).
However, in addition to being unstructured, these fields are also
_dynamic_, and will be present/absent depending on the data of the given
rule. Thus, we started needing multiple orderings for the different
combinations of rule fields/rule types.
In the absence of refactoring how we build rule details, I'm introducing
a simple helper function to fetch the relevant <dd> by the corresponding
<dt>s text. This should be more robust to change and more declarative.
* Fix bad merge conflict
Lots of these variables no longer exist upstream and this new test
needed to be refactored.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Initial pass at updating client routes, variables names, code comments, and UI copy
* Adjust server routes and param names, more var names and i18n fixes
* Fix test
* More var renaming
* Rest of server-side var renaming
* Rest of client side var renaming
* Rename agent SO attributes and add migrations
* Remove agent prefix from policy fields
* Rename agent policy SO attributes and add migrations
* Rename enrollment api key SO attributes and add migrations
* Rename package policy SO attributes and add migrations
* Rename agent event SO attributes and add migrations
* Rename subtype CONFIG to POLICY (I don't think this string is ever sent by agent, though)
* Update snapshot
* Remove unnecessary cloning in migrations
* Fix migration typos
* Update naming in tests and es archiver data
* Rename file names in /common
* Rename /server files
* Rename /public files
* Rename test file names
* Rename missed files
* Revert "Rename subtype CONFIG to POLICY (I don't think this string is ever sent by agent, though)"
This reverts commit 3c91e01ed9.
* Add migration version to updated es archiver data to fix tests
* removes not needed configuration
* fixes events columnts tests
* unksips persisten timeline test
* fixes failing test
* skips events test since need more time for investigation
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Rename `datasource` saved object to `package_config` (SO type `ingest-datasource` to `ingest-package-config`) and adjust mappings:
- Remove unused `processors` field on input & stream levels
- Remove unnecessary `enabled` field on package config & input levels
- Rename `agent_stream` field to `compiled_stream`
- Reorder other fields so that important fields are closer to top & similar fields are grouped together
Also,
- Remove 7.9.0 migrations as we are not supporting an upgrade path from experimental to beta release
* Pluralize `ingest-package-configs` for consistency
* Rename `Datasource`-related types to `PackageConfig`, update all references (does not include actual type definition changes yet)
* Rename `Datasource` schemas and rest spec typings to `PackageConfig` (does not include actual schema changes yet)
* Change `datasources` on agent config typings and schemas to `package_configs` and update all references
* Add back `enabled` field on package config and input levels. They are needed for current & future UI features. Also:
- Match types and schemas with saved object mappings (`agent_stream` to `compiled_stream`, removal of `processors`)
- Set `namespace` to be a required property on agent config and package config types, add validation support for it on UI
* Rename server-side datasource references in file names, variable names, and routes
* Update spec file and schema file
* Update doc wording
* Rename all instances of datasource in file paths and variable names on client sides, and for Endpoint too
* Minor copy adjustments, fix i18n check
* Replace datasource references in tests and fixtures; remove unused `ingest/policies` es archiver data
* Fix tests
* Fix test field name
* Fix test fixtures fields again
* Fix i18n
* rename siem to security_solution
* rename siem to security solution inside of code
* rename translation keys
* fix snapshot
* replace siem for security solution in tutorial
* missing translation to be renamed
* fix types for api test integration
* updates runner file to match the new path
* change category for kibana settings
* miss renaming in advance settings
* fixes cypress tests
* fix api integration test
* fix new translation
* fix unit test
* update translation i18n
* update translation i18n II
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
