* [Maps] fix color-style disappears when mapping by percentiles when breaks are identical
* tslint
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds support for missing_bucket with group-by configurations in the advanced editor for pivot configurations.
Previously, the editor would allow you to add the setting, but it would be stripped from the config once the transform gets created.
* fix clicking on host on netwrok detail page
* Fetch signal index at plugin level to avoid weird behavior
* bing back full screen timeline
* Show health check on timeline
* fix focus on modal of description and title
* fix focus on modal of description and title
* allow to know the next appId
* if user leave security solution and timeline has not been saved, ask them if they want to save it before leaving
* fix test + types
* Fix siem signal loading on plugin + UX on timeline with no data
* Add a callback to cleaner from solution + test
* fix bug + improve prompt leaving msg
* update note
* css improvements
* fix code to be true to our test
* miss one test
* update test
* fix unit test
* core review
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <yi-chun.chuang@elastic.co>
They look for `.kbnLoadingIndicator` which is no longer there in the new loading indicator design. This changes it to look for an element that does exist and makes it a function in utils.
Change not.be.visible to not.exist in places where the element does not exist at in that state.
https://github.com/elastic/apm-agent-go/pull/859 adds
central config support for 'log_level' to the Go agent,
so we can now enable it in the UI too.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Some progress on making alerts better in the UI
* Handle edge case
* Updates
* More updates
* Show kibana instances alerts better
* Stop showing missing nodes and improve the detail alert UI
* WIP
* Fix the badge display
* Okay I think this is finally working
* Fix type issues
* Fix tests
* Fix tests
* Fix alert counts
* Fix setup mode listing
* Better detail page view of alerts
* Feedback
* Sorting
* Fix a couple small issues
* Start of unit tests
* I don't think we need this Mock type
* Fix types
* More tests
* Improve tests and fix sorting
* Make this test more resilient
* Updates after merging master
* Fix tests
* Fix types, and improve tests
* PR comments
* Remove nextStep logic
* PR feedback
* PR feedback
* Removing unnecessary changes
* Fixing bad merge issues
* Remove unused imports
* Add tooltip to alerts grouped by node
* Fix up stateFilter usage
* Code clean up
* PR feedback
* Fix state filtering in the category list
* Fix types
* Fix test
* Fix types
* Update snapshots
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* plugged Task Manager lifecycle into status reactively
* fixed tests
* Revert "fixed tests"
This reverts commit e9f2cd05bd.
* made action group fields optional
* revert deletion
* again
* extracted action type for mto its own component
* extracted more sections of the action form to their own components
* updated icon
* added docs
* fixed always firing alert
* fixed export of components
* fixed react warning
* Adding flag for notifying on state change
* Updating logic in task runner
* Starting to update tests
* Adding tests
* Fixing types check
* Tests and types
* Tests
* Tests
* Tests
* Tests
* Tests
* Renaming field to a more descriptive name. Adding migrations
* Renaming field to a more descriptive name. Adding migrations
* Fixing tests
* Type check and tests
* Moving schedule and notify interval to bottom of flyout. Implementing dropdown from mockup in new component
* Changing boolean flag to enum type and updating in triggers_actions_ui
* Changing boolean flag to enum type and updating in alerts plugin
* Fixing types check
* Fixing monitoring jest tests
* Changing last references to old variable names
* Moving form inputs back to the top
* Renaming to alert_notify_when
* Updating functional tests
* Adding new functional test for notifyWhen onActionGroupChange
* Updating wording
* Incorporating action subgroups into logic
* PR fixes
* Updating functional test
* Fixing types check
* Changing default throttle interval to hour
* Fixing types check
Co-authored-by: Gidi Meir Morris <github@gidi.io>
* Added accessibility test for ingest node pipelines empty state.
* Added accessibility test for detail view and list view and added helpers to keep the test class clean.
* Added the rest of the tests for a11y for ingest node pipelines and updated some test subjects to aid the testing.
* Fixed the spacing issues.
* Update config.ts
Uncommitted other test files.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Reporting/PDF] Custom layout option for Canvas
* fix snapshots
* --wip-- [skip ci]
* check pdf data
* add test
* functional tests work
* add fixme comment
* read strings from pdf for test
* Update reports.ts
* function name / comment improvment
* Add Canvas toggle to choose pdf layout type
* Fix Canvas pdf panel storybook test
* Update style for new Canvas report type switch
* Update canvas share menu snapshot
* Fix tests for validating Canvas PDF using inline snapshots
Run test server with:
node scripts/functional_tests_server.js --config x-pack/test/functional/config.js
Run test suite with:
node scripts/functional_test_runner.js --config x-pack/test/functional/config.js --grep 'Canvas PDF Report'
* Fix i18n and typo
* Add a test for removing borders
* Fix i18n
* Update snapshot
Co-authored-by: Timothy Sullivan <tsullivan@elastic.co>
Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle
* Add note
* Add version metadata to signals documents
* WIP: Starting over from the ground up
* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
that question
* Fleshing out upgrade of signals
* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
is finished
* Fleshing out more of the upgrade path
Still moving logic around a bunch.
* Pad the version number of our destination migration index
Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.
This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.
* Fleshing out more upgrade finalization
* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
determine that from the task itself.
* Ensure that new signals are generated with an appropriate schema_version
* Apply migration cleanup policy to obsolete signals indexes
After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.
This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.
* Move more logic into component functions
* Fix type errors
* Refactor to make things a little more organized
* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
limited utility elsewhere
* Add some JSDoc comments around our new functions
This is as much to get my thoughts in order as it is for posterity.
Next: tests!
* Adds integration tests around migration status route
* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index
* Adds API integration tests for our signals upgrade endpoint
* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
esArchive.load
* WIP: Fleshing out finalization tests
* Consolidate terminalogy around a migration
We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.
* Implement encoding of migration details
This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.
* Better transformation of errors thrown from the elasticsearch client
These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.
* Finishing integration tests around finalization endpoint
This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.
* Test an error case due to a reindexing failure
This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.
* Remove unnecessary version info from signals documents
We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.
When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.
The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.
* Migrate an index relative to the ACTUAL template version
This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.
If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.
* Enrich our migration_status endpoint with an is_outdated qualification
This can be determined programatically, but for users manually
interpreting this response, the qualification will help.
* Update migration scripts
* More uniform version checking
* getIndexVersion always returns a number
* version comparisons use isOutdated
* Fix signal generation unit tests
We now generate a version field to indicate the version under which the
signal was created/migrated.
* Support reindex options to be sent to create_migration endpoint
Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.
* Fix signal generation integration tests
These were failing on our new signal field.
* Add unit tests for getMigrationStatus
* Add a basic test for getSignalsIndicesInRange
Since this is ultimately just an aggregation query there's not much else
to test.
* Add unit test for the naming of our destination migration index
* Handle write indices in our migration logic
* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices
* Add original hot phase to migration cleanup policy
Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.
* Update old comment
The referenced field has changed.
* Delete task document as part of finalization
* Accurately report recoverable errors on create_signals_migration route
If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.
However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.
* initial commit
refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.
* Update ml_modules.tsx
added new module names to the list
* Update get_module.ts
added new module names
* Linter fixes
* Order matters
* manifest fixes
added colon char to the module name and shortened the description
* additon to description
after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."
* Adjust module recognizer test for auditbeat dataset
* influencers
changes to the metadata jobs to make influencers identical to the originals
* change for security app
changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.
Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co>
## Summary
- Move registry URL details from handler to Registry service
- rename `archive/save_to_es` to `archive/storage` to better reflect its role
- Move SO handling from http into new `getAsset` method in `archive/storage`
- Rearrange code in HTTP handler to be more easily moved to one or two service method(s) which deal with the logic for getting the asset.
### Checklist
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
1. cf0d567 - Export function which maps file path to saved object id
2. e33d7d4 - "make it work" ™️ quality version of "use cached or saved object or registry" file in HTTP handler. It's doing too much and should eventually move the logic to service method(s).
* Port @tsg's work on task manager.
Remove 2nd var to track telemetry opt in.
Add ES client to start querying index.
Use query to get docs from a dummy index.
Change how index is queried.
Get diagnostic alerts to send to staging cluster.
Record last timestamp.
PoC on telemetry opt in via 2 processes.
Revert to original solution
* Update on agreed method. Fixes race condition.
* Expand wildcards.
* stage.
* Add rule.ruleset collection.
* Update telemetry sender with correct query for loading diag alerts.
* Add similar task tests to endpont artifact work.
* Fix broken import statement.
* Create sender mocks.
* Update test to check for func call.
* Update unused reference.
* record last run.
* Update index.
* fix import
* Fix test.
* test fix.
* Pass unit to time diff calc.
* Tests should pass now hopefully.
* Add additional process fields to allowlist.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR introduces a new concept of an _Action Subgroup_ (naming is open for discussion) which can be used by an Alert Type when scheduling actions.
An Action Subgroup can be dynamically specified, unlike Action Groups which have to be specified on the AlertType definition.
When scheduling actions, and AlertType can specify an _Action Subgroup_ along side the scheduled _Action Group_, which denotes that the alert instance falls into some kind of narrower grouping in the action group.
* moved error and loading notices for data allocation field to below description
* removed test code
* expect controls to be showing, only render notice after network request has finished
* added loading spinner for field inputs
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* moved shrink field to shared_fields and added it to the hot phase
* updated test
* update legacy jest test
* removed configuration context for now
* remove unused i18n and remove duplicated isRolloverEnabled check;
* fixed shrink field to use new described field component
* added test for removing shrink field in serialization and re-ordered fields in hot phase advanced
* new UI route to show Edit Package Policy page
* Package policy List items point to new Integration Policy details page
* Refactor to use common service to generate pkgKey
* add breadcrumb for edit policy under integrations
* Trying to flesh out new tree route
* Working on the descendants query
* Almost working descendants
* Possible solution for aggs
* Working aggregations extraction
* Working on the ancestry array for descendants
* Making changes to the unique id for ancestr
* Implementing ancestry funcitonality
* Deleting the multiple edges
* Fleshing out the descendants loop for levels
* Writing tests for ancestors and descendants
* Fixing type errors and writing more tests
* Renaming validation variable and deprecating old tree routes
* Renaming tree integration test file
* Adding some integration tests
* Fixing ancestry to handle multiple nodes in the request and writing more tests
* Adding more tests
* Renaming new tree to handler file
* Renaming new tree directory
* Adding more unit tests
* Using doc value fields and working on types
* Adding comments and more tests
* Fixing timestamp test issue
* Adding more comments
* Adding timerange and filters
* Updating schema
* Fixing timestamp test issue take 2
* Updating tests to use raw filter
* Adding time to generator
* Adding time filter and tests for retrieving lifecycles
* Removing min array size
* Updating the DAL
* Adding time range iso format
* Working on middleware
* Fleshing out middleware and actions
* Adding id, parent, and name fields to the top level response
* Adding logic for identifying when the view is moved
* WIP: updated data layers and selectors
* Switching to use isAnimating
* WIP: tree is displayed
* WIP: need events data for panel and fix tests
* Removing panning logic, adding comments and renaming things
* WIP: added name to graph
* Writing tests for the models
* Fixing generator start and end time generation
* Updating the mocks with the new interface
* Revert "Fixing generator start and end time generation"
This reverts commit c42ffd75c1.
* WIP: remove unnecessary front end data transformation
* Starting on loading state for nodes and details
* Find the terminated nodes in the middlewaree
* Fixing ingest tests
* Loading states seem to be working
* Removing some todos
* undo graphNodePositions naming
* Node loading state svg and pulse
* Fixing time range
* undo name changes, cleanup
* Creating mock that leverages the generator
* update tree generator
* log nested data in simulator.debugActions()
* change newResolverTree to resolverTree
* fix oneNodeWithPaginatedEvents mock and node events of type test
* Refactoring data reducer test and changing resolverTree DAL
* WIP: updating mocks
* remove deprecation tags
* Fixing the isometric tests
* Fixing process event tests
* updated resolver_tree mocks, update tests
* update additional tests
* fixing eslint
* fixing has more selectors
* update tests
* debugging click test
* Working node loading test
* Adding error cube and another test
* Adding a test for the error case
* use stored indices, update event api call for winlog, cleanup todos
* Adding more comments and restricting the analyze event to only endpoint and winlogbeat
* update to use schema provided by backend
* Fixing some type errors
* Fixing translation issue
* Fixing type errors
* Adding reload functionality
* Fixing translation issue
* Adding more tests for reload
* Cleaning comments up
* adding legend and schema info
* added legend and info popovers
* removed comment
* Adding comments and cleaning up stuff
* add schema and dataSource to mock actions
* Fixing some type errors and starting to address feedback
* Moving mock function
* Handling powershell events
* Adding test for winlogbeat schema
* remove cube loading className in favor of styledComponent
* fix closeAnalyzer jumping from middle of screen when resolver loads
* fix originID casing
* Cleaning up middleware and renaming time range
* Fixing node details test and some of the use selectors
* Fixing tests and types
* fix popover style, cube style, specific timestamp, some translations
* Fixed a test, and continuing to address feedback
* Addressing more feedback
* Refactoring the node data loading tests
* Adding selector for indices
* fix i18n, break apart graph controls, fix process event dot styles
* fix type error, styled description lists, nodeID
* style fix
* Removing unneeded test subjects
* recursion, recursion, recursion
* Calculating ancestors, descendants, generations once in factory and refactoring state
* Removing stringify replacer
* Adding default timerange to be beginning of epoch to max date in future
* refactoring winlog event query to use winlog record_id field
* fix popover toggle
* Fix type issue
* fix popover toggle
* add some tests
* fix types
* Adding link to time range comment
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] use index-pattern field display name in UX
* use less complex solution for es agg label
* rename
* tslint
* one more tslint fix
* review feedback
* update field_editor snapshots
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add frequencyBlockList prop to CronEditor to allow selectively removing frequency options.
* Add unit test coverage for CronEditor component.
* Convert es_ui_shared cron editor component to TypeScript.
* [Maps] keep invalid metrics editor state local and only submit metrics state when valid
* review feedback
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
