* Stateless exception list translation with improved runtime checks
* use flatMap and reduce to simplify logic
* Update to new manifest format
* Fix test fixture SO data type
* Fix another test fixture data type
* Fix sha256 reference in artifact_client
* Refactor to remove usages of 'then' and tidy up a bit
* Zlib compression
* prefer byteLength to length
* Make ingestManager optional for security-solution startup
* Fix download functionality
* Use eql for deep equality check
* Fix base64 download bug
* Add test for artifact download
* Add more tests to ensure cached versions of artifacts are correct
* Convert to new format
* Deflate
* missed some refs
* partial fix to wrapper format
* update fixtures and integration test
* Fixing unit tests
* small bug fixes
* artifact and manifest versioning changes
* Remove access tag from download endpoint
* Adding decompression to integration test
* Removing tag from route
* add try/catch in ingest callback handler
* Fixing
* Removing last expect from unit test for tag
* type fixes
* Add compression type to manifest
* Reverting ingestManager back to being required for now
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* part I to rename alerts back to detections
* part 2 to rename alerts to detections
* update imports
* fix layout container event type
* Detection Alerts
* fix type/UT/cypress
### Summary
This PR fixes two bugs in the exceptions builder. The first was that it was not allowing you to select any of the "excluded" operators. The second was that it was not adding the "and" badge when it should on initial render. It also adds unit tests for the EntryItemComponent.
## Summary
* Changes the value list imports to use a streaming in model
* Adds a custom light hand spun multi-part parser for the incoming text
* Adds a buffer pause and resume which continues to buffer the incoming data if an async event such as creating a list from the attachment file needs to happen but does not emit the lines until the resume continues.
* Adds a data slicing if the buffer becomes larger than the maximum so that if we begin buffering too quickly within memory we don't blow up the limit of Elastic Search.
* Adds unit tests
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* add results_field input in form. handle in cloning and editor
* update regression evaluate metrics
* add additional reg evaluation stats to expanded row and result panel
* update jest test
* resultsField: check for only spaces string in editor and form
* update getValuesFromResult to be less repetitive
* update types
* fix type name typo
Makes two improvements to the urlState hook (also known as appState in some places):
- There was always a risk to run into a race condition because setUrlState could refer to a stale version of the state to act upon, for example if two calls were done in parallel. This is now fixed by using a local state copy of what we get from useLocation(). This allows us to use the callback version of useState's set function so we can make sure we always modify the latest state.
- Calls to history.push() are now gated by a check if the change actually referred to the corresponding instance of urlState (either _g or _a), this should reduce the updates resulting re-renders.
The two changes should make the use of setUrlState more safe against the pitfalls (race conditions/stale updates/lots of rerenders) we previously faced.
* Revert "temporarily disable firefox functional tests in PRs (#71116)"
This reverts commit 54bd07f81b.
* Revert "[savedObjects field count] run in baseline job (#70999)"
This reverts commit 53ee7a762d.
* Revert "[CI] Add pipeline task queue framework and merge workers into one (#64011)"
This reverts commit 465ed21194.
* Revert revert of change to jenkins_xpack_visual_regression.sh
Co-authored-by: spalger <spalger@users.noreply.github.com>
* creating overview page and menu
* styling the home page
* adjusting breadcrumb
* renaming isnt working
* renaming isnt working
* renaming isnt working
* fixing import
* fixing scroll when resize window
* fixing eslint errors
* prepending links
* adding target option
* refactoring
* adding dark mode support
* fixing prettier format
* fixing i18n
* reverting some unnecessary changes
* addressing PR comments
* fixing functional tests
* ordering observability menu
* fixing tests
* addressing PR comments
* fixing scroll
* addressing pr comments
* addressing pr comments
* creating overview page
* mocking data
* mocking data
* refactoring
* crearting apm chart
* adding overview page
* adding metric charts
* adding charts
* changing mock data location
* adding mock registry
* adding date picker
* adding route validation
* adding io-ts
* adding io-ts
* adding io-ts support
* fixing imports and mock data
* adding app folder
* creating a section for each plugin
* adding stats
* adding domain min max
* refactoring xcoordinaters
* fixing route
* adding bucket size
* adding group property on logs
* adding home page
* dont break page if location state is undefined
* each component fetches its own data
* Refactoring
* adding loading indicator to chart
* fixing uptime chart
* adding brush functionality to charts
* fixing refresh button and auto refresh function
* adding horizontal line to accordion section
* adding emptySection to dashboard page
* adding add data button
* adding resources section
* removing margins from horizontal rule
* changing min interval to 60s
* fixing empty section
* removing unnecessary code
* adding unit tests
* fixing imports
* adding initial story book for observability
* removeing uptime mock data
* fixing xDomain to show correct data on x-axis
* fixing empty state alignment
* adding story book and other improvements
* adding news component
* adding support to custom colors on EuiProgress and EuiStats
* removing infra mock data
* adding error message when api throwns an error
* adding alert section
* Adding alerts
* adding alert api call
* addressing PR comments
* adding storybook
* adding feedback button
* addressing PR comments
* chamging plugins return data
* fixing kibana app navigation
* fixing unit test
* fixing ts issues
* addressing PR comments
* using lodash truncate
* adding comment
* updating public documentation
* fixing alerts request
* fixing unit test
* fixing unit test
* aligin beta badge to the center
* adding moment duration to get the units as seconds
* addressing PR comments
* addressing PR comments
* Add beta and experimental badges to epm list and detail pages; clean up some epm components
* Clean up styled warnings
* Fix types
* Allow experimental query param to be passed through to registry /search
* Allow experimental query param to be passed through to registry /categories endpoint
* Fix buggy categories count (#64981)
* Always enable experimental packages and categories
* Handle long package names nicely; misc layout tweaks
* Move experimental=true flag to client side
* Prevent layout jumps even more
* Adjust beta/experimental badge tooltip copy
## Summary
Migrate tests from integration-test repo.
The integration-test repo's purpose is to smoke test the build artifacts of all the main products in the stack (the .deb, .rpm, .tar.gz, .zip files).
Currently Vagrant and VirtualBox are used to create VMs of the OSs appropriate for installing those build artifacts. These scripts are in the integration-test repo.
After the VMs are installed and running the stack, a small number of UI tests are run against Kibana to verify we have beats data, logstash data, etc.
Kibana-QA team also uses the various VMs for manual testing since manually setting up security across the full stack can be time consuming.
The new tests in this PR under x-pack/test/stack_functional_integration/ are NOT executed as part of Kibana CI process. They run from other periodic Jenkins jobs.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: LeeDr <lee.drengenberg@elastic.co>
* Multiline kql bar
* fix id
* use visibility rather than display to hide stuff, cross fingers for tests
* another vis trick for tests
* quasi fix tests, still some failures
* caroline feedback
* fun!
* fix for mouse
* fix test
* check api
* fix unit test on query_string_input
* Fix cypress test
* handle the resize of the height of the textarea when the window have been resize
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Liza K <liza.katz@elastic.co>
* add "@elastic/elasticsearch" to dependencies
* first POC of new client
* add logging
* add generation script for client facade API and implementation
* add back keepAlive
* add exports from client
* add new client mocks
* add some doc
* fix API usages
* rename legacy client to legacy in service
* rename currently unused config/client observable
* wire new client to service & update mocks
* fix mock type
* export client types
* add transport.request
* more doc
* migrate version_check to new client
* fix default port logic
* rename legacy client mocks
* move legacy client mocks to legacy folder
* start adding tests
* add configure_client tests
* add get_client_facade tests
* bump client to 7.8
* add cluster_client tests
* expose new client on internal contract only
* revert using the new client for es version check
* add service level test for new client
* update generated API
* Revert "rename legacy client mocks"
This reverts commit e48f3ad6
* address some review comments
* revert ts-expect-error from unowned files
* move response mocks to mocks.ts
* Remove generated facade, use ES Client directly
* log queries even in case of error
* nits
* use direct properties instead of accessors
* handle async closing of client
* review nits
* ElasticSearchClient -> ElasticsearchClient
* add test for encoded querystring
* adapt test file
* Add learn more links to data streams, indices, and index templates tabs.
* Add tooltips to detail panel.
* Unify data streams description text.
* Fix bug in which index tab showed an empty list, by clearing the filter state on unmount.
* Add indices count to data stream detail panel.
* Add usage collector for telemetry.
* Make minimal usage collector work.
* Add all fields to Usage and schema
* Type packages as array.
* Temporarily remove schema.
* Temporarily exclude our collector from schema checks.
* Add fleet telemetry.
* Remove events from agent stats.
* Add package telemetry.
* Use correct import.
* Add telemetry about enabled packages.
* Clean up comments.
* Update x-pack/plugins/ingest_manager/server/collectors/package_collectors.ts
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
* Update x-pack/plugins/ingest_manager/server/collectors/package_collectors.ts
Co-authored-by: Nicolas Chaulet <n.chaulet@gmail.com>
* Correctly check for element in array.
* Use a real SavedObjectsClient.
* Remove useless use of undefined.
* Use less deep path to import SavedObjectsClient.
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
Co-authored-by: Nicolas Chaulet <n.chaulet@gmail.com>
* Reactify visualize app
* Fix typescript failures after merging master
* Make sure refresh button works
* Subscribe filter manager fetches
* Use redirect to landing page
* Update savedSearch type
* Add check for TSVB is loaded
* Add unit tests for useSavedVisInstance effect
* Fix comments
* Fix uiState persistence on vis load
* Remove extra div around TableListView
* Update DTS selectors
* Add error handling for embeddable
* Add unit tests for createVisualizeAppState
* Add unit tests for useChromeVisibility
* Add filter_manager.mock
* Add unit tests for useVisualizeAppState
* Use app state stub
* Add unit tests for useLinkedSearchUpdates
* Add unit tests for useEditorUpdates
* Remove extra argument from useEditorUpdates effect
* Update comments, fix typos
* Remove extra div wrapper
* Apply design suggestions
* Revert accidental config changes
* Add unit tests for useEditorUpdates
* Use visualize services mock
* Add unit tests for getVisualizationInstance
* Fix eslint warnings
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Related to #70383 and #63455.
Refactors the action buttons of the transform and data frame analytics jobs list:
Previously custom actions included state and JSX for e.g. confirmation modals. Problem with that: If the actions list popover hides, the modal would unmount too. Since EUI's behaviour will change with the release/merge of #70383, we needed a refactor that solves that issue right now.
With this PR, state management for UI behaviour that follows after a button click like the confirmation modals was moved to a custom hook which is part of the outer level of the buttons itself. The modal now also gets mounted on the outer level. This way we won't loose the modals state and DOM rendering when the action button hides.
Note that this PR doesn't fix the nested buttons issue (#63455) yet. For that we need EUI issue #70383 to be in Kibana which will arrive with EUI v26.3.0 via #70243. So there will be one follow up to that which will focus on getting rid of the nested button structure.
* Fix base64 download bug
* Add test for artifact download
* Add more tests to ensure cached versions of artifacts are correct
* Convert to new format
* missed some refs
* partial fix to wrapper format
* update fixtures and integration test
* Fixing unit tests
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* adds 2 menu items to alert page, progress on exception modal
* adds enriching
* remove unused useExceptionList()
* implements some types
* move add exception modal files
* Exception builder changes to support latest schema
* Changes to lists plugin schemas and fix api bug
Needed to make the schemas more forgiving. Before this change they required name,
description, etc for creation and update.
The update item API was using the wrong url.
* Adding and editing exceptions working
- Modifies add_exception_modal component
- Creates edit_exception_modal component
- Creates shared comments component
- Creates use_add_exception api hook for adding or editing exceptions
- Updates viewer code to support adding and editing exceptions
- Updates alerts table code to use updated version of add_exception_modal
* fixes duplicate types
* updates os tag input
* fixes comment style
* removes checkbox programatically
* grahpql updates to expose exceptions_list
* Add fetch_or_create_exception_list hook
* fixes data population
* refactor use_add_exception hook, add tests
* fix rebase issues, pending updates to edit modal
* fix edit modal and default endpoint exceptions
* adds second checkbox
* adds signal index stuff
* switches boolean logic
* fix some type errors
* remove unnecesary code
* fixes checkbox logic in edit modal
* fixes recursive prop passing
* addresses comments/fixes types
* Revert schema type changes
* type fixes
* fixes regular exception modal
* fix more type errors, remove console log
* fix tests
* move add exception hook, lint
* close alert checkbox closes alert
* address PR comments
* add type to patch rule call, fix ts errors
* fix lint
* fix merge problems after conflict
* Address PR comments
* undo graphql type change
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
## Summary
Adds these data types to the value based lists end points from [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html):
Single value based list types:
* binary
* boolean
* byte
* date
* date_nanos
* date_range
* double
* float
* integer
* ip
* half_float
* keyword
* text
* long
* short
Range value based list types:
* double_range
* float_range
* integer_range
* ip_range
* long_range
Geo value based list types: (caveat is that you cannot query them using other geometry just yet ... you can only these and export them)
* geo_point
* geo_shape
* shape
For importing and exporting different values such as ranges, geo, or single values, this introduces a serialize and deserialize option for the endpoints.
For example if you want to serialize in an ip_range such as 192.168.0.1,192.168.0.3 which has a comma between the two would use the following:
```ts
POST /api/lists
{
"name": "List with an ip range",
"serializer": "(?<gte>.+),(?<lte>.+)",
"deserializer": "{{gte}},{{lte}}",
"description": "This list has ip ranges",
"type": "date_range"
}
```
If you want to serialize in keywords from a list that _only_ match a particular value you would use the following:
```ts
POST /api/lists
{
"id": "keyword_custom_format_list",
"name": "Simple list with a keyword using a custom format",
"description": "This parses the first found ipv4 only",
"serializer": "(?<value>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))",
"deserializer": "{{value}}",
"type": "keyword"
}
```
The serializer is a [named capturing group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) while the deserializer is using [MustacheJS](https://github.com/janl/mustache.js/). The range type, single value types, and geo types all have default captures for their serialize and default mustache templates if none are configured with an endpoint.
The default capture groups and mustache handles for each are:
* shape, geo_point, geo_shape: `(?<lat>.+),(?<lon>.+)`
* date_range: `(?<gte>.+),(?<lte>.+)|(?<value>.+)`
* other ranges are: `(?<gte>.+)-(?<lte>.+)|(?<value>.+)`
* all single data types: `(?<value>.+)`
For ranges you can use both `gte, lte`, and `value` together. If `gte` _and_ `lte` matches it will use that for the greater than, less than elastic range and ignore `value` even if `value` also matched. If _only_ `value` matches and `gte`, `lte` does not match then it will use `value` and put `value` as _both_ the `gte`, and `lte`.
For example, if you are serializing in a list of ip ranges as the list data type, `ip_range` and you have these 3 entries in the file:
```ts
127.0.0.1
127.0.0.2-5
```
The default `serializer` will use `(?<gte>.+)-(?<lte>.+)|(?<value>.+)` and you will get two elastic documents like so:
```ts
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.1",
"lte" : "127.0.0.1"
}
}
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.2",
"lte" : "127.0.0.5"
}
}
```
The default mustache handles for each are:
* shape, geo_point, geo_shape: `{{{lat}}},{{{lon}}}`
* date_range: `{{{gte}}},{{{lte}}}`
* other ranges are: `{{{gte}}}-{{{lte}}}`
* all values are: `{{{value}}}`
I use three instead of two handle bars (`{{{` vs.` {{`) so that HTML is not escaped for the lists. You can override and change it if you need or want the escaping.
If during the deserializer phase it detects that a `gte` and `lte` are exactly the same it will still output them as a two items and use the mustache deserialize value. Using the ip-range example above that will be outputted like so since it detects that the lte-gte are exactly the same value:
```ts
127.0.0.1-127.0.0.1
127.0.0.2-127.0.0.5
```
---
Interesting queries to run from the lists scripts folder for testing:
Load some small test files from `./lists/files` for example:
```ts
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_cidr.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range.txt
./import_list_items_by_filename.sh date ./lists/files/date.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_mixed.txt
...
```
Export them
```ts
./export_list_items.sh ip_range_cidr.txt
./export_list_items.sh ip_range.txt
./export_list_items.sh date.txt
./export_list_items.sh ip_range_mixed.txt
...
```
Find on them
```ts
./find_list_items.sh ip_range_cidr.txt
./find_list_items.sh ip_range.txt
./find_list_items.sh date.txt
./find_list_items.sh ip_range_mixed.txt
...
```
Find specific values such as:
```ts
./get_list_item_by_value.sh ip_range_mixed.txt 192.168.0.1
./get_list_item_by_value.sh date.txt 2020-08-25T17:57:01.978Z
...
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Removes the feature flag and turns on lists by default
* Applies to both exception lists and value lists
* Removes all scary messages about having it enabled
* Updates the unit tests to work with it on
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
