## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* plugged Task Manager lifecycle into status reactively
* fixed tests
* Revert "fixed tests"
This reverts commit e9f2cd05bd.
* made action group fields optional
* revert deletion
* again
* extracted action type for mto its own component
* extracted more sections of the action form to their own components
* updated icon
* added docs
* fixed always firing alert
* fixed export of components
* fixed react warning
* Adding flag for notifying on state change
* Updating logic in task runner
* Starting to update tests
* Adding tests
* Fixing types check
* Tests and types
* Tests
* Tests
* Tests
* Tests
* Tests
* Renaming field to a more descriptive name. Adding migrations
* Renaming field to a more descriptive name. Adding migrations
* Fixing tests
* Type check and tests
* Moving schedule and notify interval to bottom of flyout. Implementing dropdown from mockup in new component
* Changing boolean flag to enum type and updating in triggers_actions_ui
* Changing boolean flag to enum type and updating in alerts plugin
* Fixing types check
* Fixing monitoring jest tests
* Changing last references to old variable names
* Moving form inputs back to the top
* Renaming to alert_notify_when
* Updating functional tests
* Adding new functional test for notifyWhen onActionGroupChange
* Updating wording
* Incorporating action subgroups into logic
* PR fixes
* Updating functional test
* Fixing types check
* Changing default throttle interval to hour
* Fixing types check
Co-authored-by: Gidi Meir Morris <github@gidi.io>
* Added accessibility test for ingest node pipelines empty state.
* Added accessibility test for detail view and list view and added helpers to keep the test class clean.
* Added the rest of the tests for a11y for ingest node pipelines and updated some test subjects to aid the testing.
* Fixed the spacing issues.
* Update config.ts
Uncommitted other test files.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Reporting/PDF] Custom layout option for Canvas
* fix snapshots
* --wip-- [skip ci]
* check pdf data
* add test
* functional tests work
* add fixme comment
* read strings from pdf for test
* Update reports.ts
* function name / comment improvment
* Add Canvas toggle to choose pdf layout type
* Fix Canvas pdf panel storybook test
* Update style for new Canvas report type switch
* Update canvas share menu snapshot
* Fix tests for validating Canvas PDF using inline snapshots
Run test server with:
node scripts/functional_tests_server.js --config x-pack/test/functional/config.js
Run test suite with:
node scripts/functional_test_runner.js --config x-pack/test/functional/config.js --grep 'Canvas PDF Report'
* Fix i18n and typo
* Add a test for removing borders
* Fix i18n
* Update snapshot
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle
* Add note
* Add version metadata to signals documents
* WIP: Starting over from the ground up
* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
that question
* Fleshing out upgrade of signals
* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
is finished
* Fleshing out more of the upgrade path
Still moving logic around a bunch.
* Pad the version number of our destination migration index
Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.
This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.
* Fleshing out more upgrade finalization
* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
determine that from the task itself.
* Ensure that new signals are generated with an appropriate schema_version
* Apply migration cleanup policy to obsolete signals indexes
After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.
This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.
* Move more logic into component functions
* Fix type errors
* Refactor to make things a little more organized
* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
limited utility elsewhere
* Add some JSDoc comments around our new functions
This is as much to get my thoughts in order as it is for posterity.
Next: tests!
* Adds integration tests around migration status route
* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index
* Adds API integration tests for our signals upgrade endpoint
* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
esArchive.load
* WIP: Fleshing out finalization tests
* Consolidate terminalogy around a migration
We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.
* Implement encoding of migration details
This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.
* Better transformation of errors thrown from the elasticsearch client
These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.
* Finishing integration tests around finalization endpoint
This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.
* Test an error case due to a reindexing failure
This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.
* Remove unnecessary version info from signals documents
We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.
When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.
The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.
* Migrate an index relative to the ACTUAL template version
This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.
If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.
* Enrich our migration_status endpoint with an is_outdated qualification
This can be determined programatically, but for users manually
interpreting this response, the qualification will help.
* Update migration scripts
* More uniform version checking
* getIndexVersion always returns a number
* version comparisons use isOutdated
* Fix signal generation unit tests
We now generate a version field to indicate the version under which the
signal was created/migrated.
* Support reindex options to be sent to create_migration endpoint
Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.
* Fix signal generation integration tests
These were failing on our new signal field.
* Add unit tests for getMigrationStatus
* Add a basic test for getSignalsIndicesInRange
Since this is ultimately just an aggregation query there's not much else
to test.
* Add unit test for the naming of our destination migration index
* Handle write indices in our migration logic
* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices
* Add original hot phase to migration cleanup policy
Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.
* Update old comment
The referenced field has changed.
* Delete task document as part of finalization
* Accurately report recoverable errors on create_signals_migration route
If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.
However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.
* initial commit
refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.
* Update ml_modules.tsx
added new module names to the list
* Update get_module.ts
added new module names
* Linter fixes
* Order matters
* manifest fixes
added colon char to the module name and shortened the description
* additon to description
after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."
* Adjust module recognizer test for auditbeat dataset
* influencers
changes to the metadata jobs to make influencers identical to the originals
* change for security app
changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
## Summary
- Move registry URL details from handler to Registry service
- rename `archive/save_to_es` to `archive/storage` to better reflect its role
- Move SO handling from http into new `getAsset` method in `archive/storage`
- Rearrange code in HTTP handler to be more easily moved to one or two service method(s) which deal with the logic for getting the asset.
### Checklist
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
1. cf0d567 - Export function which maps file path to saved object id
2. e33d7d4 - "make it work" ™️ quality version of "use cached or saved object or registry" file in HTTP handler. It's doing too much and should eventually move the logic to service method(s).
* Port @tsg's work on task manager.
Remove 2nd var to track telemetry opt in.
Add ES client to start querying index.
Use query to get docs from a dummy index.
Change how index is queried.
Get diagnostic alerts to send to staging cluster.
Record last timestamp.
PoC on telemetry opt in via 2 processes.
Revert to original solution
* Update on agreed method. Fixes race condition.
* Expand wildcards.
* stage.
* Add rule.ruleset collection.
* Update telemetry sender with correct query for loading diag alerts.
* Add similar task tests to endpont artifact work.
* Fix broken import statement.
* Create sender mocks.
* Update test to check for func call.
* Update unused reference.
* record last run.
* Update index.
* fix import
* Fix test.
* test fix.
* Pass unit to time diff calc.
* Tests should pass now hopefully.
* Add additional process fields to allowlist.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces a new concept of an _Action Subgroup_ (naming is open for discussion) which can be used by an Alert Type when scheduling actions.
An Action Subgroup can be dynamically specified, unlike Action Groups which have to be specified on the AlertType definition.
When scheduling actions, and AlertType can specify an _Action Subgroup_ along side the scheduled _Action Group_, which denotes that the alert instance falls into some kind of narrower grouping in the action group.
* moved error and loading notices for data allocation field to below description
* removed test code
* expect controls to be showing, only render notice after network request has finished
* added loading spinner for field inputs
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
