* Fixing resolver functional tests
* Import the animation constant
* Only check specific nodes instead of all the ones in view
* Removing check for link text
* updating test description
* Adding comments
* Adds new SO type for persisting our signals migrations
* WIP: Migration status SO client
Trying to forge a patterrn using io-ts to validate at runtime. I think
I've got it working but I want to refactor the pipeline out into a
reusable function(s).
* Implements our SavedObjects service for signals migrations
* Defines a simple client that delegates to the base SO client with
our SO type
* Defines a service that consumes the simpler client, adding validations
and data transforms on top.
* Refactoring migration code to work with saved objects
As opposed to the previous ephemeral, encoded tokens, we now retrieve migration
info from saved objects.
At the API level, this means that both the create and finalize endpoints
receive a list of concrete indices. No more passing around tokens.
As both endpoints are idempotent, users can hammer them as much as they
want with the same lists of indices. Redundant creates and finalizes
will be met with inline 400 messages, and as one continues to poll the
finalize endpoint they should see more and more indices respond with
"completed: true"
* Fixing integration tests first, and anything upstream breaking them
* Clean up API integration tests
* standardize assignment of responses (with types)
* deletes migration SOs as test cleanup
* Split API tests into separate files
This was getting big and unwieldy; this splits these into one file per
endpoint.
* Refactor: split existing migration service functionality into atomic functions
This will allow us to repurpose the service to compose more
functionality and be more specifically useful, while keeping the
component logic separate.
* WIP: moving logic into migrationService.create
* Splitting get_migration_status into component functions
getMigrationStatus was really two separate aggregations, so I split them
out and we recompose them in the necessary routes.
* Move finalization logic into function
* migrationService exposes this as .finalize()
* adds an error field to our migration SO
* We currently only have one error that we persist there, but it would
be very time-consuming to track down that information were it not
there.
* Adds function for migration "deletion" logic
* migrationService leverages this function
* adds new boolean to our savedObject
* deletes unused function (deleteMigrationSavedObject)
* Adds route for soft-deletion of migrations
* Updating tests related to migration status
* Adding/updating mocks/unit tests necessary to satisfy the things I
need to test
* I mainly wanted to test that the the status endpoint filtered out the
deleted migrations; this was accomplished with a unit test after
fleshing out some mocks/sample data.
* Move old migration service tests to the relevant function tests
This logic was previously moved out into component functions; this moves
the tests accordingly.
* Add some unit tests around our reindex call
* Fix create migration route tests
Mocks out our migration functions, rather than stubbing ES calls
directly.
* Updates finalize route unit tests
Addresses functionality that hasn't been moved to finalizeMigration()
* Unit tests our finalization logic
Fixes a bug where we weren't accounting for soft-deleted migrations.
ALso updates our test migration SO to have a status of 'pending' as
that's a more useful default.
* Fixes finalization integration tests
These were failing due:
* a change in the migration status API response
* a bug I introduced in the finalize route
* Adds tests for our migration deletion endpoint
* unit tests
* API integration tests
* Caught/fixed bug with deleting a successful migration
* Fixes types
Removes unused code.
* Prevent race condition due to template rollover during migration
If a user has an out of date index (v1) relative to the template (v2), but the
template itself is out of date (newest is v3), then it's possible that
the template is rolled over to v3 after the v1-v2 migration has been
created but before the new index has been created.
In such a case, the new index would receive the v3 mappings but would
incorrectl be marked as v2. This shouldn't necessarily be an issue, but
it's an unnecessary state that can easily be prevented with the guard
introduced here.
* Add real usernames to migration savedObjects
In addition to the SOs themselves giving us observability into what
migration actions were performed, this gives us the additional info of
_who_ performed the action.
* Index minimal migration SO fields needed for current functionality
* Add additional migration info to status endpoint
This will allow users to finalize a migration if they've lost the
response to their POST call.
* Finalize endpoint receives an array of migration IDs, not indices
This disambiguates _which_ migrations we were finalizing if you passed
an index (which was previously: the most recent migration).
* Fix type errors in tests after we threaded through username
* Update responsibilities of migration finalize/delete endpoints
Discussions with @marshallmain lead to the following refactor:
* finalize does not delete tasks
* finalize only applies cleanup policy to a failed migration
* delete takes an array of migration ids (like finalize)
* delete hard-deletes the SavedObject of a completed (failed or
successful) migration
This gives a bit more flexibility with the endpoints, as well as
disambiguates the semantics: it just deletes migrations!
* Fix tests that were broken during refactoring
* Fix type errors
I removed some logic here but forgot the imports :(
* Move outdated integration test
In the case of a successful migration, application of the cleanup policy
is done by the deletion endpoint. In the interest of data preservation,
we do not delete a sourceIndex unless it is explicitly deleted.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Adds context.alerts as available parameter for detection rule actions
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Alerts][License] Define minimum license required for each alert type (#84997)
* Define minimum license required for each alert type
* fixed typechecks
* fixed tests
* fixed tests
* fixed due to comments
* fixed due to comments
* removed file
* removed casting to LicenseType
* [Alerts][License] Add license checks to alerts HTTP APIs and execution (#85223)
* [Alerts][License] Add license checks to alerts HTTP APIs and execution
* fixed typechecks
* resolved conflicts
* resolved conflicts
* added router tests
* fixed typechecks
* added license check support for alert task running
* fixed typechecks
* added integration tests
* fixed due to comments
* fixed due to comments
* fixed tests
* fixed typechecks
* [Alerting UI][License] Disable alert types in UI when the license doesn't support it. (#85496)
* [Alerting UI][License] Disable alert types in UI when the license doesn't support it.
* fixed typechecks
* added licensing for alert list and details page
* fixed multy select menu
* fixed due to comments
* fixed due to comments
* fixed due to comments
* fixed typechecks
* fixed license error message
* fixed license error message
* fixed typechecks
* fixed license error message
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* started docs, removed kernel harden option
* Advanced policy field documentation
* consistent formatting
* consistent formatting
* drop unused fields
* grammar
* i18n first key
* i18n, change versions to 7.9, add some new fields, remove some that we don't want to expose
* Update x-pack/plugins/security_solution/public/management/pages/policy/models/advanced_policy_schema.ts
Co-authored-by: Daniel Ferullo <56368752+ferullo@users.noreply.github.com>
* Update x-pack/plugins/security_solution/public/management/pages/policy/models/advanced_policy_schema.ts
Co-authored-by: Daniel Ferullo <56368752+ferullo@users.noreply.github.com>
* Update x-pack/plugins/security_solution/public/management/pages/policy/models/advanced_policy_schema.ts
Co-authored-by: Daniel Ferullo <56368752+ferullo@users.noreply.github.com>
Co-authored-by: Daniel Ferullo <56368752+ferullo@users.noreply.github.com>
* fix clicking on host on netwrok detail page
* Fetch signal index at plugin level to avoid weird behavior
* bing back full screen timeline
* Show health check on timeline
* fix focus on modal of description and title
* fix focus on modal of description and title
* allow to know the next appId
* if user leave security solution and timeline has not been saved, ask them if they want to save it before leaving
* fix test + types
* Fix siem signal loading on plugin + UX on timeline with no data
* Add a callback to cleaner from solution + test
* fix bug + improve prompt leaving msg
* update note
* css improvements
* fix code to be true to our test
* miss one test
* update test
* fix unit test
* core review
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <yi-chun.chuang@elastic.co>
## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* plugged Task Manager lifecycle into status reactively
* fixed tests
* Revert "fixed tests"
This reverts commit e9f2cd05bd.
* made action group fields optional
* revert deletion
* again
* extracted action type for mto its own component
* extracted more sections of the action form to their own components
* updated icon
* added docs
* fixed always firing alert
* fixed export of components
* fixed react warning
* Adding flag for notifying on state change
* Updating logic in task runner
* Starting to update tests
* Adding tests
* Fixing types check
* Tests and types
* Tests
* Tests
* Tests
* Tests
* Tests
* Renaming field to a more descriptive name. Adding migrations
* Renaming field to a more descriptive name. Adding migrations
* Fixing tests
* Type check and tests
* Moving schedule and notify interval to bottom of flyout. Implementing dropdown from mockup in new component
* Changing boolean flag to enum type and updating in triggers_actions_ui
* Changing boolean flag to enum type and updating in alerts plugin
* Fixing types check
* Fixing monitoring jest tests
* Changing last references to old variable names
* Moving form inputs back to the top
* Renaming to alert_notify_when
* Updating functional tests
* Adding new functional test for notifyWhen onActionGroupChange
* Updating wording
* Incorporating action subgroups into logic
* PR fixes
* Updating functional test
* Fixing types check
* Changing default throttle interval to hour
* Fixing types check
Co-authored-by: Gidi Meir Morris <github@gidi.io>
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle
* Add note
* Add version metadata to signals documents
* WIP: Starting over from the ground up
* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
that question
* Fleshing out upgrade of signals
* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
is finished
* Fleshing out more of the upgrade path
Still moving logic around a bunch.
* Pad the version number of our destination migration index
Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.
This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.
* Fleshing out more upgrade finalization
* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
determine that from the task itself.
* Ensure that new signals are generated with an appropriate schema_version
* Apply migration cleanup policy to obsolete signals indexes
After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.
This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.
* Move more logic into component functions
* Fix type errors
* Refactor to make things a little more organized
* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
limited utility elsewhere
* Add some JSDoc comments around our new functions
This is as much to get my thoughts in order as it is for posterity.
Next: tests!
* Adds integration tests around migration status route
* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index
* Adds API integration tests for our signals upgrade endpoint
* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
esArchive.load
* WIP: Fleshing out finalization tests
* Consolidate terminalogy around a migration
We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.
* Implement encoding of migration details
This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.
* Better transformation of errors thrown from the elasticsearch client
These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.
* Finishing integration tests around finalization endpoint
This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.
* Test an error case due to a reindexing failure
This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.
* Remove unnecessary version info from signals documents
We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.
When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.
The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.
* Migrate an index relative to the ACTUAL template version
This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.
If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.
* Enrich our migration_status endpoint with an is_outdated qualification
This can be determined programatically, but for users manually
interpreting this response, the qualification will help.
* Update migration scripts
* More uniform version checking
* getIndexVersion always returns a number
* version comparisons use isOutdated
* Fix signal generation unit tests
We now generate a version field to indicate the version under which the
signal was created/migrated.
* Support reindex options to be sent to create_migration endpoint
Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.
* Fix signal generation integration tests
These were failing on our new signal field.
* Add unit tests for getMigrationStatus
* Add a basic test for getSignalsIndicesInRange
Since this is ultimately just an aggregation query there's not much else
to test.
* Add unit test for the naming of our destination migration index
* Handle write indices in our migration logic
* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices
* Add original hot phase to migration cleanup policy
Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.
* Update old comment
The referenced field has changed.
* Delete task document as part of finalization
* Accurately report recoverable errors on create_signals_migration route
If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.
However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.
* initial commit
refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.
* Update ml_modules.tsx
added new module names to the list
* Update get_module.ts
added new module names
* Linter fixes
* Order matters
* manifest fixes
added colon char to the module name and shortened the description
* additon to description
after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."
* Adjust module recognizer test for auditbeat dataset
* influencers
changes to the metadata jobs to make influencers identical to the originals
* change for security app
changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
* Port @tsg's work on task manager.
Remove 2nd var to track telemetry opt in.
Add ES client to start querying index.
Use query to get docs from a dummy index.
Change how index is queried.
Get diagnostic alerts to send to staging cluster.
Record last timestamp.
PoC on telemetry opt in via 2 processes.
Revert to original solution
* Update on agreed method. Fixes race condition.
* Expand wildcards.
* stage.
* Add rule.ruleset collection.
* Update telemetry sender with correct query for loading diag alerts.
* Add similar task tests to endpont artifact work.
* Fix broken import statement.
* Create sender mocks.
* Update test to check for func call.
* Update unused reference.
* record last run.
* Update index.
* fix import
* Fix test.
* test fix.
* Pass unit to time diff calc.
* Tests should pass now hopefully.
* Add additional process fields to allowlist.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Trying to flesh out new tree route
* Working on the descendants query
* Almost working descendants
* Possible solution for aggs
* Working aggregations extraction
* Working on the ancestry array for descendants
* Making changes to the unique id for ancestr
* Implementing ancestry funcitonality
* Deleting the multiple edges
* Fleshing out the descendants loop for levels
* Writing tests for ancestors and descendants
* Fixing type errors and writing more tests
* Renaming validation variable and deprecating old tree routes
* Renaming tree integration test file
* Adding some integration tests
* Fixing ancestry to handle multiple nodes in the request and writing more tests
* Adding more tests
* Renaming new tree to handler file
* Renaming new tree directory
* Adding more unit tests
* Using doc value fields and working on types
* Adding comments and more tests
* Fixing timestamp test issue
* Adding more comments
* Adding timerange and filters
* Updating schema
* Fixing timestamp test issue take 2
* Updating tests to use raw filter
* Adding time to generator
* Adding time filter and tests for retrieving lifecycles
* Removing min array size
* Updating the DAL
* Adding time range iso format
* Working on middleware
* Fleshing out middleware and actions
* Adding id, parent, and name fields to the top level response
* Adding logic for identifying when the view is moved
* WIP: updated data layers and selectors
* Switching to use isAnimating
* WIP: tree is displayed
* WIP: need events data for panel and fix tests
* Removing panning logic, adding comments and renaming things
* WIP: added name to graph
* Writing tests for the models
* Fixing generator start and end time generation
* Updating the mocks with the new interface
* Revert "Fixing generator start and end time generation"
This reverts commit c42ffd75c1.
* WIP: remove unnecessary front end data transformation
* Starting on loading state for nodes and details
* Find the terminated nodes in the middlewaree
* Fixing ingest tests
* Loading states seem to be working
* Removing some todos
* undo graphNodePositions naming
* Node loading state svg and pulse
* Fixing time range
* undo name changes, cleanup
* Creating mock that leverages the generator
* update tree generator
* log nested data in simulator.debugActions()
* change newResolverTree to resolverTree
* fix oneNodeWithPaginatedEvents mock and node events of type test
* Refactoring data reducer test and changing resolverTree DAL
* WIP: updating mocks
* remove deprecation tags
* Fixing the isometric tests
* Fixing process event tests
* updated resolver_tree mocks, update tests
* update additional tests
* fixing eslint
* fixing has more selectors
* update tests
* debugging click test
* Working node loading test
* Adding error cube and another test
* Adding a test for the error case
* use stored indices, update event api call for winlog, cleanup todos
* Adding more comments and restricting the analyze event to only endpoint and winlogbeat
* update to use schema provided by backend
* Fixing some type errors
* Fixing translation issue
* Fixing type errors
* Adding reload functionality
* Fixing translation issue
* Adding more tests for reload
* Cleaning comments up
* adding legend and schema info
* added legend and info popovers
* removed comment
* Adding comments and cleaning up stuff
* add schema and dataSource to mock actions
* Fixing some type errors and starting to address feedback
* Moving mock function
* Handling powershell events
* Adding test for winlogbeat schema
* remove cube loading className in favor of styledComponent
* fix closeAnalyzer jumping from middle of screen when resolver loads
* fix originID casing
* Cleaning up middleware and renaming time range
* Fixing node details test and some of the use selectors
* Fixing tests and types
* fix popover style, cube style, specific timestamp, some translations
* Fixed a test, and continuing to address feedback
* Addressing more feedback
* Refactoring the node data loading tests
* Adding selector for indices
* fix i18n, break apart graph controls, fix process event dot styles
* fix type error, styled description lists, nodeID
* style fix
* Removing unneeded test subjects
* recursion, recursion, recursion
* Calculating ancestors, descendants, generations once in factory and refactoring state
* Removing stringify replacer
* Adding default timerange to be beginning of epoch to max date in future
* refactoring winlog event query to use winlog record_id field
* fix popover toggle
* Fix type issue
* fix popover toggle
* add some tests
* fix types
* Adding link to time range comment
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Added default value for type parameter in ConditionEntry type.
* Added signer field UI. Flattened a bit component structure and reused some translations.
* Reverted the condition for signer option.
* Fixed the import.
* Removed unused translations.
* Fixed the test.
* Consolidated a bit the deletion and creation flows in redux.
