* Update the ems-client dependency
This PR adds the `appName` and `appVersion` parameters used by ems-client. The `appVersion` parameter replaces the now deprecated `kbnVersion` parameter in ems-client.
* Review feedback
* Fix borked merge
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* initial implementation of configurable test users
* user superuser by default to match master
* referenced the configs in reporting and api integration
* setting the minimum number of default roles
* looking for x-pack tests with users and roles
* add testUserService in dashboard mode tests
* running only ciGroup7
* uncommenting - addign visualization
* re-enabling all CI groups to run on CI
* reinstating Jenkinsfile
* disable Test user for OIDC config
* improved logging and added Roles for OSS tests to get better info on the runs.
* disable test_user for auth tests
* don't fetch enabledPlugins when testuser disabled
* fix es-lint
* running oss tests with x-pack enabled
* [revertme] build default dist for oss tests
* updating NOTICE.txt file as it complained in the kibana intake tests
* changed to pick OSS builds
* trying a license change to trial
* switch back to xpack builds
* created a new sample data role and used it in homepage tests
* revert test/scripts/jenkins_ci_group.sh
* only refresh browser and wait for chrome if we are already on Kibana page
* fix large_string test to use minimum set of roles and privileges
* fix for date nanos custom timestamp with a configured role
* changes to the files with addition of new roles for the test_user
* reverting to OSS changes and few additions to the time_zone test to run as a test_user
* changes to security
* changes to the x-pack test to use elastic superuser
* fix for chart_types test
* fixes to area chart , input control test
* fix for dashboard filtering test and a new config role
* changes to handle the x-pack tests
* additional role for date nanos mixed
* added the logstash role to the accessibility tests
* removed telemetry setting
* docs+few changes to the tests
* removed Page navigation
* removed pageNavigation which was unused
* test/accessibility/apps/management.ts
* update management.ts
* aria label, and other changes
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* accidentally checked in a piped file with results.
* reverted
* unloading of logstash data, fixing aria label
* aria-label
* added the required role
* fix for tsvb chart
* fix for sample data test reverted home_page pageobject file
* changes to sample data test and visualize index file to incorporate OSS changes
* changes to describe() and some more changes to incorporate in settings_page
* re-adding the after()
* removed unwanted roles
* replaced kibana_user with kibana_admin
* added the check of deprecated kibana_user
* testing with kibana_admin role
* fix for discover test
* incorporated the review comments
* incorporated the review comments
* incorporate review comments and added restoreDefaults()
* removed describe.only
* reverted the OSS logic change I had here- pulled into seperate PR
* incorporated the review comments
* incorporated review changes
* adding hidden=true to find hidden kibanaChrome
* change field.test.tsx to be same as that of master branch
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Move over to new plugin space, working implementation
* Fixing tests for report_listing snapshots
* WIP: Fixing react-component tests
* Fixing report_info_button tests
* Fixing download linksies
* WIP: Final working implementation
* Fixing attachAction API + API URLs
* Let the past die. Kill it if you have to. That’s the only way to become what you were meant to be.
* Fixing stream-client for new platform APIs
* Fixing types and tests
* Fix broken mock
* Adds back in warnings to report info button
* kibana.json line-breaks on required plugins
* Fixing broked snapshots
* Fix license checks in client-side components
* Adding back in warnings to report_listing component
* Fix danglig unused import
* Adds license checks for basic to our csv panel action
* Fixes issues from prior fork
* Move relative pathing to absolute
* Fix POST URL copying as we've moved from static methods
* Fix layoutId props
* Fixes types for layoutId
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Move redirect_when_missing to kibana utils
* Replace redirectWhenMissing in dashboard
* Replace redirectWhenMissing in discover
* Remove redirect in monitoring
* Remove extra import
* Move invalid vistype check into editor.js
* Mock the history folder
* Fix redirect when missing index or saved object
* Move history to discover services
* Use redirect to listing page
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [ML] Fixes to error handling for analytics jobs and file data viz
* [ML] Fix failing tests and address comments from review
* [ML] Add key prop to error messages map
* [ML] Add errors.ts
The current index threshold alert uses a `size` limit on term aggregation, when used, but does not sort the buckets, so it's just using descending count on the grouped buckets as the sort to determine what to return.
The watcher API for the index threshold notes this as "top N of", implying a sort.
This PR applies sorting when the using `groupBy: top`, and the `aggType != count`. For count, ES is already sorting the way we want.
The sort is calculated as a separate agg beside the date_range aggregation, which is the same metrics agg specified in the query - `aggType(aggField)`. This field is then referenced in a new `order` property in the terms agg, using 'asc' sorting for `min`, and `desc` sorting for `avg`, `max`, and `sum`.
This doesn't change the shape of the output at all, just changes which term buckets will be returned, if there are more term buckets than requested with the `termSize` parameter.
* Added variables button for text fields in Pagerduty component. Fixed bugs mentioned in https://github.com/elastic/kibana/issues/60067
* Fixed due to comments
* fixed language check issue
* Fixed tests
* Fixed due to comments
* Memo'ize some layout and EPM header components
* Split EPM home page into two tabs
* Clean up dead files and import paths
* Add empty state
* Use react routing for rendering tab content
* Fix import paths (again)
* [Maps] move MapSavedObject type out of telemetry
* move SavedObject from server to core/types
* review feedback
* results from check_published_api_changes
* test to verify alerts page header says alerts
* updating test with pr feedback
* updating test with pr feedback and better verbiage
* updating test with pr feedback for better test titling
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Move validated range files to new NP location
* Update refs in code
* Clean up old validated range files
* Change relative paths to 'kibana-react'. Some clean up
* Change to relative paths
* Fix i18n errors
* i18n clean up. Export module explicitly
* Change files over to TS to prevent build issue where validated range was missing
* Clean up TS conversion
* More clean up. Extend EuiRangeProps
* Remove unneeded ts-ignore
* Review feedback and test fixes
* Change double to single quotes
* min and max aren't always passed, make optional
* Type updates
* Review feedback. Set state to empty on init and add ignore comment
* Review feedback
* Add back in last 2 ts-ignores. Build fails without focusable attribute on EuiDualRange & No good alternatives for spread syntax in TS components
* Rollback change to state init. Initializing state to null actually triggers a react browser warning and complicates using 'prevState' in getDerivedStateFromProps
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Refactors signal rule alert type into a smaller executor
## Summary
* Breaks out the schema into its own file and function
* Breaks out the action group into its own file and function
* Moves misc types being added to this into the `./types` file
* Breaks out all the writing of errors and success into their own functions
* Uses destructuring to pull data out of some of the data types
* Tweaks the gap detection to accept a date instead of moment to ease "ergonomics"
* Updates unit tests for the gap detection
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* adds signals data
* adds 'closes and opens signals'
* refactors test
* adds extra check to see that the selected number of signals is correct
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [ML] Client side cut over
* updating paths and commented code
* changes based on review
* disabling telemetry tests
* fixing start job stylesheets
* fixing everything that is broken
* fixing types and ml icon order
* using icon constant
* Enable Switch for Collecting sys metrics on create form/flyout
* Add support for sys monitoring to `sendCreateAgentConfig()` (FE)
* Added `query` param to create Agent Config API schema and handler
* Create System datasource support in Create Agent config API handler
* Add . period to the isRegex variable L34
* Undo ID2 back to ID , done for testing
* Updated the Jest test message to include period
* Undo exception message to include periods
* Removed duplicate comments
* - adds primary color highlighting to connected edges of the focused service
- fixes z-indexing issues with overlapping edges when hovering on a service node
- always centers on the focused service node after layout reflows
* re-centers the already focused service node when the focus button is clicked
* remove environemnt filter when querying for sample trace ids
* - fixes missing query params in the generated hrefs for details and focused service navigation
- fix layout bug by passing undefined for roots (instead of []) in service maps layout when no roots are found
* Revert "remove environemnt filter when querying for sample trace ids"
This reverts commit d482aa124b1f2c47da01d4386c2b88108bc94275.
* Fixes extra prop from a merge conflict
* add global status count + add status count with find + rename state to status + allow batch update for cases
* fix all my bug and integrate API with UI
* add reporters functionality
* review II
* review III
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
resolves https://github.com/elastic/kibana/issues/58529
This PR extends alertType with an `actionVariables` property, which
describes the properties of the context object passed when scheduling
actions, and the current state. These property descriptions are used
by the web ui for the alert create and edit forms, to allow the properties
to be added to action parameters as mustache template variables.
* Fix Internal Server Error about mapping issue when creating a data source
* Fix double system datasource appearing in default config
* Ensure we can never double assign same datasource ID to agent config
Closes#59325.
It does sometimes cause the map to not be centered when switching, which will be addressed in a separate PR as part of #59164.
Also add a missing prop to the popover. It was causing the focus button there to never be disabled.
* [EPM] Documentation of HTTP routes & TS types for Ingest (#48798)
* Add beginning models and two routes for Ingest
* Update types & models per discussion w/Ruflin
Also reviewed data structures listed at https://docs.google.com/document/d/1IBR3f9dpHqJmXYEdg06WV34KSMd3g5k4aMGa4jde_Eg/edit#
* Update: /policies always returns array. /policy returns single policy
* Add pagination for /policy & /datasources. Uses per_page & page params
* Add API metadata. Standardize policy_id param name.
* Update descriptions to match Google Doc. Move use case to Policy.
Disabled the '@typescript-eslint/array-type' rule because it was going around in circles. It didn't like Datasource[] or Array<Datasource>
* Return to initial TS annotation for Arrays
Remove the line disabling @typescript-eslint/array-type now that it's behaving normally again 🤷
* [EPM] Add directory structure for server/lib. (#50469)
* Add directory structure for server/lib.
* 'tests' seems to be more common than 'test'
* Make CI happy
* [EPM] Add basic documentation directory (#50478)
* [EPM] Add basic documentation directory
Having the doc directory around allows us to easily add docs from here on to document how EPM works.
To run the docs build, use the following command from the kibana directory:
```
../docs/build_docs --doc docs/epm/index.asciidoc --open
```
The above assumes that docs (https://github.com/elastic/docs) are checked out in the same directory as Kibana.
With this change, the EPM docs build is not included yet in the overall docs build. For this adjustments to https://github.com/elastic/docs/blob/master/conf.yaml must be made.
* [EPM] Add basic index template (#50471)
This PR adds the very basic index template we will use for the packages. It contains all the basic settings and some examples. The examples will be remove as soon as we have an actual implementation with packages but for now is convenient to see if it is a valid package.
This code is put into the lib directory as it does not tie directly into any handlers.
It also adds an functional tests for loading a template. This means we have a way to check if a template is valid in Elasticsearch. Based on this we can check in the future all our generated templates for validity with Elasticsearch.
To run the functional test, go to the Kibana x-pack directory. Start the first command:
```
node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts
```
Keep the above running and switch to an other Terminal. Now run:
```
node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts
```
* 40752 rewrite ingest pipeline (#50627)
* Add directory structure for server/lib.
* 'tests' seems to be more common than 'test'
* Make CI happy
* Implement pipeline rewriting.
* Add more testcases
* For posterity (comment change)
* Allow beats-style template delimiters
* Be more succinct
* Document better
* Replace AssetType enum with union type (#50696)
See https://github.com/elastic/kibana/pull/50609#discussion_r346080439
Discussed in Slack and agree to revert for now. Can track down the issues & restore later
* Remove unnecessary await if we can return the promise (#50329)
* Fix whitespace per figma comments. Closes#47348 (#47350)
* Add fix & comment for TS 3.7.2 regression
* [EPM] cleanup assets, filter assets for those currently supported (#50609)
* cleanup assets, filter assets for those currently supported
* removed unused type
* fix type
* add comment for better type
* change type name to be more descriptive
* hardcode image width for ie11 (#49796)
* hardcode image width for ie11
* eslint
* improve comment
* add maxWidth
* [EPM] useKibana hook & render in plugin.start (#50110)
* plugin.start now does reactdom.render vs returning react element
export plugin function from public/index
* Move setClient call from plugin.start to plugin.setup
* Use `useUiSetting$` from `useKibana` hooks
* Fix broken app due to bad hooks usage
Can't use useKibana outside a React component.
Reverting to prior approach since it's still NP. Can revisit context usage in a followup PR
* [EPM] Install package from detail view on button click (#50735)
* Support basic "click button -> show spinner -> installed" install flow
* Remove incorrect comments. Add TS return types to data functions.
* [EPM] Use NP feature_catalogue.register (#50108)
* Use NP feature_catalogue.register
* Use type from NP plugin
* fix linting
* fix types
* fix headers in Fleet
* skipping test due to ES param change
* Revert "skipping test due to ES param change"
This reverts commit d05f20decf.
* remove type field
* remove unused import
* [EPM] Final(?) update from integrations_manager -> EPM (#50976)
* Update (all remaining?) references from integrations_manager to EPM
* Update path in i18n file
* [EPM] update compatibility section (#50975)
* change min max version to single range value
* add elastic stack icon and change text
* remove badge from version and use code font
* remove euistyled
* add back version component lost in merge
* remove euiStyled
* remove old file
* Restore RequirementVersionRange type
* Disable test for elasticsearch username.
Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.
* Revert "Disable test for elasticsearch username."
This reverts commit f1020e4eab.
* Disable test for elasticsearch username.
Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.
* Fix EPM typing issues in register feature
* Fix typings after master merge
* [EPM] CI fixes (#51284)
* Initialize es in test.
* Add it(), no es.init()
* Clean up.
* [EPM] add confirmation modal (#51172)
* add confirmation modal, move install state to Header
* update callout to use title
* move components only used in detail view to detail dir
* use better variable names
* update to more descriptive variable names
* Restore prior response vaulues for install & delete package (#51252)
Discussed this with @skh https://github.com/elastic/kibana/pull/51112#commitcomment-35961413 & https://github.com/elastic/kibana/pull/51112#commitcomment-35970664 as well as in a video call
Also added some TS type annotations for data fetching functions to make the contracts more explicit
* [EPM] /package API only lists installable assets. Restore enums. (#51414)
* API only shows installable assets. Server types to own file. Restore enums
* Fix type imports
* Only return installable asset types (kibana for now)
* server/types now only has code from hapi which shouldn't go to client
* Add more restricted TS types to DisplayAssets object
* Flip order of arguments to Extract
In these cases it still works the same, but looking at https://www.typescriptlang.org/docs/handbook/advanced-types.html the signature is
`Extract<T, U>` - Extract from `T` those types that are assignable to `U`
so the larger set should be first
* [Fleet] Enrollment api key UI (#51495)
* Make button pretty in dark mode as well. (#51610)
* [EPM] Add docs entry about registryUrl config (#51697)
This documentation is at the moment mainly for internal use. I found myself searching for this URL several times in the code or PRs so I thought I rather add it to the docs for now.
* [EPM] Remove encoding of Kibana objects as not needed anymore
* [Fleet] Move agent status server side and API to get aggregated status for a policy (#51673)
* [EPM] Add basic docs around install/delete API endpoint (#51728)
This is mainly for internal usage at the moment to look up.
* Ingest/policy (#51741)
* wip policy
* tweaks
* tweaks
* FIX TYPOS
* WIP move policy => agent config conversion to fleet, WIP policy changed method
* fix tests and bugs
* updates tests and snaps
* more fixes
* use AGENT_POLLING_INTERVAL
* cleanup and fix some formatting
* Update x-pack/legacy/plugins/ingest/server/libs/datasources.ts
Co-Authored-By: John Schulz <github.com@jfsiii.org>
* Update x-pack/legacy/plugins/ingest/server/libs/datasources.ts
Co-Authored-By: John Schulz <github.com@jfsiii.org>
* Update x-pack/legacy/plugins/ingest/server/libs/outputs.ts
Co-Authored-By: John Schulz <github.com@jfsiii.org>
* Update x-pack/legacy/plugins/fleet/server/libs/policy.ts
Co-Authored-By: John Schulz <github.com@jfsiii.org>
* fix things broken by PR review suggestions
* remove unused field
* fix types
* fix mappings
* add datasource mappings
* Fix mappings and remove get full policy from checkin
* Fix ingest api integration tests
* run es-lint to fix fleet
* Fix typescript issues
* [EPM] Track package install state and add toast notification (#51734)
* add notifications from core to plugin
* add package install state hook
* fix type error
* use toMountPoint helper to add jsx to notification
* add warning notification to failed install
* make notifications dependency explicit prop
* move PackageInstall provider lower
* add comment about InstallStatus type overlapping InstallationStatus
* use InstallStatus type in InstallationButton component
* fix type
* [Ingest] Adds support for a working default output (#51841)
* aadding config
* add working settings to the default output
* remove default username and password
* update libs
* [EPM] Add basics for creating the ILM setup (#50474)
This contains the basic objects to setup ILM
* Create index and alias with a write index
* Get the policy
The code does not contain any functional tests yet as it is still open on how to do it best. I suggest to get this in as a foundation and then iterate on top of it.
* [EPM] Add datasource (ingest pipeline) from package (#51851)
## Summary
This mixes a few concerns but I think it's worth it to show the parts working together.
Take a look at the individual commits for a better separation of features.
This adds
- the `/datasource/install/{pkgkey}` endpoint which installs ingest pipelines from a package into ES and saves a reference to them in the EPM state Saved Object
- Connects the "Add datasource" button in the successful installation Toast to the new API
- Adds a toast notification to inform the user the datasource was added correctly
- Adds a "Delete Package" button on the details page so we can uninstall a package while we're waiting for the separate view which allows deletes
- b99eda6 Pushes logic that was in the detail view into `InstallationButton`. This consolidates the logic in one component (or one component & the existing hook) and, iiic, means we can put `<InstallationButton package={...} />` on any view and get the same behavior
I'm marking this as a normal PR so people can merge if they wish
* [EPM] Add basic processing of fields.yml file (#51148)
The fields.yml is used to generate the Elasticsearch template and Kibana index pattern. This PR adds a very basic implementation of processing the fields.yml and then create an Elasticsearch template out of it. The only fields that are supported at the moment are keyword fields, more will be added as a follow up.
The testing was implemented with a golden file. The output from the method is compared to a json file. If the input is changed or the method is changed, it is possible to regenerate the files with the `-generate` flag as following:
```
node scripts/jest ./legacy/plugins/epm/server/lib/template/template.test.ts -generate
```
This will allow us to quickly test many inputs / outputs in the future, make adjustments to the existing files and generate the new outputs. We then can compare it in the diff it the changes make sense.
* [EPM] Create basic implementation to merge input template and dataset manifest (#51803)
* [EPM] Create basic implementation to merge input template and dataset manifest
With this code it is possible to take an input template for the agent and merge it with the config variables from the dataset manifest file. Currently only the name and the default value are merged. Later on we must implement to be able to pass user configured variables to it and make decision based on OS selection.
Closing https://github.com/elastic/kibana/issues/51794
* [EPM] Refactoring of lib structure (#51885)
This refactors the structure of lib. As so far all the lib parts are related to assets in the package, it is organised the same way with the same structure. Each directory has its own tests directory if it needs one. This makes it possible to (almost) not need relative paths for tests.
* [EPM] Allow to read files from fields directory (#51958)
This change allows to also extract files from the `fields` directory. Previously this was not possible because it always assumed a service must be there.
* [EPM] Install Elasticsearch Index Template for data source (#51878)
This installs the Elasticsearch index template for each dataset in a package. For now the names are hardcoded based on package key and dataset name but will be more dynamic later on when we pass the full dataset information.
The dataset extractions is a bit "hacky" at the moment and we should get a full implementation of dataset at a later stage and replace this code.
* [Fleet] Policy list, details, create, edit UIs (#51950)
* Set up simple policies list view
* Adjust spec to return single policy
* Set up simple policy details page
* Add demo stats/chart to policy details
* Add description string
* Initial setup of policy form and create policy UI
* Policy create/edit form; integrate policy list api
* Integrate create policy api
* Integrate policy detail, agent status, and policy edit APIs; adjust policy list api integration in agent enrollment
* Fix edit policy mock meta
* Fix policy list search bar
* PR and linting fixes; use typings from ingest plugin
* Fix i18n
* [EPM] Add datasource saved object type (#51871)
## Summary
This PR makes a few assumptions, and contains a lot of refactoring. It might be beneficial to look at the resulting directory structure under `server` first to get the (new) big picture.
Assumptions:
- our API deals with several concerns, for now these are packages and datasources
- we manage our own HTTP API endpoints for these concerns (in particular, don't use the ingest plugin for that)
- we manage (for now) the Kibana saved object in which datasources are saved. importing and calling methods from the ingest plugin to do that down the road will (hopefully) be a manageable change
This led to the following decisions:
- the code is separated into subdirectories by concern, containing all the route handlers and tightly coupled code
- for now, these directories are in `server/packages` and `server/datasources`. I'm tempted to move them into `server/api/{packages,datasources}` but wanted to limit the amount of refactoring in one PR
- shared code lives in `server/lib`
- some code from `server/packages` has been almost duplicated to handle saving to Datasource saved objects, some has been refactored and is used from both places. The deduplication needs further improvement
- maybe `server/registry` should also move under `server/lib` (but see above, I'm trying to not move everything around all at once)
Testing:
* Please note that this is a breaking change because the saved object type for package information
has also been renamed. You'll need to start with a fresh `.kibana-*` index. Restarting `yarn es snapshot` (withouth specifying a data directory) should do the trick.
* Package installation should still work, e.g. with a GET request to `http://localhost:5601/api/epm/package/coredns-1.0.1`. The saved objects for packages can be inspected with a GET request to `http://localhost:5601/api/saved_objects/epm-package/$PKG_KEY`, e.g. `http://localhost:5601/api/saved_objects/epm-package/coredns-1.0.1`
* Datasource creation should still work, e.g. with a GET request to `http://localhost:5601/api/epm/datasource/install/coredns-1.0.1`. The saved objects for datasources can be inspected with a GET to `http://localhost:5601/api/saved_objects/epm-datasource/$PKG_KEY`, e.g. `http://localhost:5601/api/saved_objects/epm-datasource/coredns-1.0.1`
* [Fleet] Expose policy during agent checkin (#51968)
* [EPM] Add /epr prefix to the tar.gz download path (#51881)
The registry slightly changed the .tar.gz path because of download stats reason. This adjusts for it. See https://github.com/elastic/package-registry/pull/169
* [EPM] Move template installation to lib and add asset helper (#52049)
* [EPM] Move template installation to lib and add asset helper
All the logic related to the installation of the templates for a package should be inside the template library folder. This moves the logic into this folder.
A few refactorings were made to simplify installation:
* Introduction of DataSet interface: This interface is needed to extract the data sets inside a package and install one template per data set.
* Pass package instead of package key to installation process: Passing the package instead of the package key means fetching of package information is decoupled from the installation process and abstracted. This separates the two concerns and should simplify testing.
* getAsssets method: The getAssets methods works on top of the package object to extract asset paths. It is inspired by get_objects methods but supports passing a package and a dataset.
Currently one problem with testing that exists is that to fetch the content of an asset is not decoupled yet.
* [EPM] Reduce data source to one type (#52061)
Between Fleet / Ingest / EPM there had been several interface definitions of Datasource and the related types. This reduces it to one place for the definition. The same applies to the policy definition.
The goal of this is that from now on we all rely on the same definition. If we make changes, we make them in all parts of the code.
In this PR is only the minimal change needed to get us all on one interface. Further changes will be needed that we all rely on the same saved objects etc.
* add export command
* revert 2 more files to rely on export
* revert imports
* Fix types for Datasource Saved Object
* merge in master
* fix type check
* Run VSCode's organize imports on EPM files (#52234)
Learned about it on Slack from https://twitter.com/ryanchenkie/status/1201883268527927301
Blog at https://code.visualstudio.com/updates/v1_23#_run-code-actions-on-save
Basically does the order we've been loosely following (3rd party, then relative) & alphabetic by location and variable name.
It's not customizable but it's reasonable and, afaict, consistent.
* [EPM] More realistic datasource SO. Error if package not installed. (#52229)
* Move cache 'hack' into getAssetsData
* p -> pkg. package is reserved. pkgkey is used in many places
* Remove unnecessary type cast
* Clarify reasons behind asset path manipulation
* Return the Datasource; not the Saved Object.
* Use real values from package in fake datasource SO
* Error if /datasource/install before /package/install
```
> curl --user elastic:changeme localhost:5601/api/epm/datasource/install/coredns-1.0.1
{
"statusCode": 403,
"error": "Forbidden",
"message": "coredns-1.0.1 is not installed"
}
> curl --user elastic:changeme localhost:5601/api/epm/install/coredns-1.0.1
[
{
"id": "53aa1f70-443e-11e9-8548-ab7fbe04f038",
"type": "dashboard"
},
{
"id": "Metricbeat-CoreDNS-Dashboard-ecs",
"type": "dashboard"
},
{
"id": "75743f70-443c-11e9-8548-ab7fbe04f038",
"type": "visualization"
},
{
"id": "36e08510-53c4-11e9-b466-9be470bbd327-ecs",
"type": "visualization"
},
{
"id": "277fc650-67a9-11e9-a534-715561d0bf42",
"type": "visualization"
},
{
"id": "cfde7fb0-443d-11e9-8548-ab7fbe04f038",
"type": "visualization"
},
{
"id": "a19df590-53c4-11e9-b466-9be470bbd327-ecs",
"type": "visualization"
},
{
"id": "a58345f0-7298-11e9-b0d0-414c3011ddbb",
"type": "visualization"
},
{
"id": "9dc640e0-4432-11e9-8548-ab7fbe04f038",
"type": "visualization"
},
{
"id": "3ad75810-4429-11e9-8548-ab7fbe04f038",
"type": "visualization"
},
{
"id": "57c74300-7308-11e9-b0d0-414c3011ddbb",
"type": "visualization"
},
{
"id": "27da53f0-53d5-11e9-b466-9be470bbd327-ecs",
"type": "visualization"
},
{
"id": "86177430-728d-11e9-b0d0-414c3011ddbb",
"type": "visualization"
},
{
"id": "4804eaa0-7315-11e9-b0d0-414c3011ddbb",
"type": "visualization"
}
]
> curl --user elastic:changeme localhost:5601/api/epm/datasource/install/coredns-1.0.1
[
{
"id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_plaintext_json",
"type": "ingest-pipeline"
},
{
"id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_json_json",
"type": "ingest-pipeline"
},
{
"id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_entry_json",
"type": "ingest-pipeline"
}
]
```
* fix duplicated imports
* [EPM] Move golden files generation over to jest snapshot (#52203)
* [EPM] Move golden files generation over to jest snapshot
I initially used my own implementation to write the generated files. It runs out jest has a feature to write snapshots which simplifies the code a lot.
I added a loop with an additional test file so in the future we can just keep adding test files without having to modify the test code.
To updated the snapshots, the param `-u` has to be used:
```
node scripts/jest legacy/plugins/epm/server/lib/fields/field.test.ts -u
```
* [EPM] Create metrics-* and logs-* Kibana index pattern (#52277)
This creates the very basic Kibana index patterns metrics-* and logs-* for Kibana. At the moment it is overwritten every time. We need to change this in the future to take the fields from all installed data sources and regenerate it.
* [EPM] Create helper for elasticsearch asset names (#52265)
Most of the Elasticsearch assets have the same base name. This creates a helper to get the base name for the assets. In case we decide to change the base name in the future, we can change it in one place.
* fix tests and destructing
* [EPM] Update Registry types. Prevent errors installing certain datasources. (#52285)
* Update RegistryPackage type
* Use download key from EPR to fetch archive
* Fix errors caused by correcting the Registry types.
The issues were largely that some Registry types like `title, `datasets` and `assets` where marked as required, but are actually optional. This highlighted area in the code were we relied on them always being present.
We added to the issue by wrapping Registry types in `Required` which made those items which were correctly listed as optional, required for EPM code. Updated EPM types to reflect the largely pass-through nature of the EPM types.
There are two properties which we ensure are in every EPM response, those were put into their own (unexported) type.
Confirm by trying to add a datasource to a package which has no datasources, like apache-1.0.1 or system-2.0.1. In `feature-ingest` and the earlier version of this PR, the `/datasource/install` call returns a 500. In this PR it succeeds.
* [EPM] Add setup of default ILM policies (#52272)
This creates two ILM policies: logs-default and metrics-default. These are the default ILM policies used. Currently the policy content is hardcoded in the code but should be fetched from the base package in the future. The setup happens as part of the datasource installation. When a data source is installed it is a good time to check if the assets are there but we might extract this to a better place in the future.
* [EPM] 52075 add data source first page (#52320)
* Update RegistryPackage type
* add first page of add data source
* fix for ie11 flex min width bug
* remove toDetailViewRelative
* remove unneeded spread
* Update TS type names for EPR search results (#52512)
* `RegistryList -> RegistrySearchResults`
* `RegistryListItem -> RegistrySearchResult`
* Restore import sort order from #52234 (#52548)
Many of the changes from #52234 were lost. Presumably due to PR(s) merging which were based on branches which had the previous unsorted order.
* [EPM] Replace wildcard export (#52554)
* PackageNotInstalledError -> packages/index.ts
* pkgToPkgKey -> registry/index.ts (will convert existing `${name}-${version}` instances later)
* Replace export * from packages.
There's an argument that the import sites should be updated to import from `packages/get`, `packages/install`, etc but that can wait for a later PR.
* [EPM] Reduce usage of epm-package SavedObject (#52576)
* Delete existing Installation type. Rename InstallationAttributes to Installation
* Reduce usage of EPM SO. Add getInstallation().
Replaced two calls of getInstallationObject() with getInstallation().
Two less places with knowledge of SO internals.
Lots of potential improvements for EPM TS types remain (refactoring/removing Installable, etc), but this is a good incremental step, IMO
* [EPM] Fix missing export link (#52628)
Without it, things break. I am surprised CI did not catch this.
* [EPM] Cleanup ILM loading (#52632)
Before the check for the ILM policy to exist triggered an exception. With this change it is a normal response also if the policy does not exist yet.
A follow up issue will be created in Elasticsearch to get a HEAD request for this available.
* [EPM] Switch to staging URL for registry (#52626)
The old cluster with the registry will be removed as soon as this is merged.
* [EPM] Use Dataset interface to generate template (#52255)
This will make sure we have to pass much feature params and can fully rely on the datasource object to create names for assets.
* [Fleet] Use agent events to compute agent health (#52513)
* [EPM] Data source integration tests (#52542)
* Add fixtures for data source integration test.
* Move test setup to beforeEach
* Add test for datasource creation
* Handle pipelines in yml format.
* Make integration test for adding a data source pass.
* Use EPR staging URL with CDN. (#52776)
See https://github.com/elastic/kibana/pull/52626#pullrequestreview-330622868
* [EPM] Add Data Source page updates (#52705)
* remove dupe type RegistryPackage
* change switches to checkboxes, use datasets to create checkboxes, add some local form state
* update types
* [EPM] redirect after package install (#52771)
* add callback after successful installation and redirect
* add temp data sources tab content to access add data source page
* remove assets tab for mvp
* hide data sources link and redirect from data sources tab if package not installed
* change callback name
* remove commented out assets logic
* add redirect to hook
* fix type
* Use ingest datasource api (#52964)
Incremental change. Uses HTTP API for datasource creation. Will do follow-up PR which uses JS function instead
* Remove duplicate fetchInfo & installTemplates
I think this was from a bad merge, but pretty sure we don't want these functions called twice in the same function
* WIP. Pushing so others can see
* Improve correctness/flexibility of absolute URL
* Disable datasource test & template installation
* [Ingest] Data source APIs (#52448)
* Clean up ingest imports and remove unneeded mock_spec files
* Initial pass at datasources lib and API endpoints
* Add add/remove datasource to/from policy API endpoints
* Add datasource contract tests and related policy contract tests; update snapshots
* Fix tests
* Fix tests again
* Fix tests 3
* Adjust routes, PR feedback
* modify epm createDatasource endpoint to use user data (#52971)
* change epm/datasource/install/{pkg} to POST, send user data to endpoint, install pipelines and templates based on user selected datasets
* change test to post for installing a datasource
* change some names and types around
* delete request.headers['transfer-encoding'] being passed through from epm request
* [EPM] Don't share CreateFakeDatasource type (#53068)
It's not shared between client & server so it doesn't need to be in common. Also, it imports server code which would try to bring server types to the client. It's types so they're compiled away but it's important to keep common to what's truly common. Breaking this separation is why we thought enums broke the client. A lint rule just landed in master to prevent this.
* [EPM] Index template generation fixes (#53104)
* Only add keyword type field to mappings.
* Index template installation
* Handle empty fields definition files
* Re-enable index template installation
* [Fleet] Assign/Unassign data source from policy UI (#53058)
* Add index files to export various modules; normalize imports
* Clean up unused files; extract datasources table component from policy details page
* Expose http client to frontend libs; remove unused types; import ES UI's useRequest lib
* Adjust shape of rest api adapter interface to better match with rest of kibana; remove unused node adapter; change per_page param to perPage in agent events route
* Initial pass at assign data sources flyout
* Initial pass at unassigning data sources from policy
* Make data sources table searchable by package values
* Fix enrollment key lib for rest adapter param changes
* Fix imports and types
* `yarn.lock` changes after bootstrapping
* [EPM] Implement getConfig for dataset (#53261)
* [EPM] Implement getConfig for dataset
* Implements a getConfig method on a dataset object.
* Build the configuration for each dataset in a package.
* construct and save streams into datasource saved object
* [EPM] Fix template installation (#53272)
As dataset.package was not set, the installed templates contained undefined in the template name. This changes fixes this.
* [EM] Refactor ingest pipeline installation (#53309)
* Refactor ingest pipeline installation
* Only install index templates for requested datasets
* Add index.default_pipeline to index template
* Hook up pipeline rewriting
* Add correct types.
* change POST create datasources path (#53165)
* change POST create datasources path
* remove pkgkey from params
* Fix creation of a data source with a custom ID (#53537)
* [Ingest] Return associated policy IDs in data source info (#53350)
* Return number of policies from data source, surface in assign data source UI
* Update snapshots
* [EPM]: Assign data source to policy in UI (#53597)
* Let ES generate source ids. Refactor along the way.
* Datasource.id isn't optional. It's just missing before we send to Ingest
* Delete EPM's mapping of datasources saved object. Ingest handles that.
* Keep datasource object-related work in constructDatasource
* Move asset installation into own function. Keep entry point high-level.
* More descriptive (less ambiguous) names for these two functions
* Use enum values from Ingest instead of plain strings
* Limit the 'type' key of references to known asset types.
* Update variable names to clarify that we're merging arrays of references
* Use [].flat instead .reduce + .concat to avoid error on empty arrays.
* Pass PackageInfo value directly to component vs pulling off n properties
* Name handlers/options based on the data, not the UI element
* Populate policy combo box based on values from Ingest policy API
* Mark Dataset.vars as optional.
* Add TODOs
* Add commands to run API tests to README (#53847)
* Limit functions to 3 params max. Update those which used more (#53848)
* [EPM] Code in 'common' directories shouldn't import server code (#53854)
* [Fleet] Code in 'common' directories shouldn't import server code (#53938)
* [Fleet] Remove server code from common folder in fleet
* [Fleet] Fix typescript issues after master merge
* [EPM] Fix typescript issues after master merge
* Fix eslint issues
* Fix typescript issues after merge
* Fix merge master missing line
* Fix merge conflict
* [Fleet] Fix registration of Ingest management section (#54065)
* Fix registration of Ingest management section
* Fix i18n key
* [Fleet] Remove server code from common folder in ingest (#53969)
* [Fleet] Connect fleet to policy change update (#53201)
* [Fleet] Send created event when a policy is created
* [Fleet] updated created event when a policy is created
* [Fleet] Send deleted event when a policy is deleted
* [Fleet] Rename output.url => output.hosts (#54258)
* [Ingest] Remove policies UI (#54308)
* Remove meta field UI from policy add/edit form
* Initial pass at policy bulk+single delete UI and API
* Adjust policy links from agent list and detail pages so that links are only active if policy exists
* Add delete policy UI to policy detail page
* Disable policy delete button for default policy
* Commit updated kbn-pm artifact.
CI is failing with messages like
14:52:28 ERROR: 'yarn kbn run build -i @kbn/pm' caused changes to the following files:
14:52:28
14:52:28 packages/kbn-pm/dist/index.js
Following advice from https://elastic.slack.com/archives/C0D8P2XK5/p1570032166063400 and running/committing build
* Update kbn/pm package
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
* [EPM] create logs metrics index patterns (#54037)
* fixes bug in for loop returning too early and not looping through all yaml files creating incomplete index template, move loading yaml files to own function, other cleanup
* use reduce in place of for loop
* basic functionality for creating index patterns
* separate logs and metrics index patterns
* dedupe fields
* adjust flattenFields to rename nested fields with parent name path
* some tests
* use yml files for tests
* add awaits as part of installing the package
* optimize loading of yaml files
* fix typo
* change type packageName to package
* update tests to use all files from the beginning
* fix type errors
* fix test
* Use dataset.package from registry
https://github.com/elastic/kibana/pull/54037#pullrequestreview-340362812
* Form validation on add datasource page. (#53920)
* [Ingest] Add support for policy `label` field (#54413)
* Allow `label` field in policy APIs, update UIs to support `label` field
* PR review changes, typo fixes, update tests
* [Fleet] Fix api key creation (#54498)
* [Fleet] Move agent acks to his own endpoint (#54401)
* [Ingest] Fix MaxListenersExceededWarning during kibana boot (#54745)
* [Fleet] Create a default api key for the default output (#54658)
* [EPM] update index pattern fields (#54862)
* add DatasetType type
* move loadFieldsFromYaml to fields
* add logic to determine field values
* update tests
* add back accidentally removed readFromDocValues
* remove DatasetType and add IndexPatternType
* rename dedup to dedupe
* group tests
* use null coalescing operator
* Fix typing issues
* [Fleet] Support agent event subtype STOPPING (#55231)
* [EPM] Prevent double submit when creating data sources
Update installationRequested state after submit (#55100)
* [EPM] handle alias fields when creating kibana index pattern (#55254)
* add alias support, update flattenFields to handle alias copying, update and add tests
* update snapshot
* update findFieldByPath to return undefined if not leaf node
* remove temporary alias type from map
* [EPM] handle multi fields when creating index patterns (#55554)
* handle mult_fields
* move nested function out
* [Fleet] Fleet/spec docs (#55619)
* [EPM] Start to document definitions (#55361)
This is a first stab at creating a place where we define the terms we use across ingest management. This is not complete but defines a place where we can add all the future defintions.
* [EPM] Indexing strategy docs (#55301)
* [EPM] Indexing strategy docs
This documentation is to start documenting our new indexing strategy in a public place and have it versioned. This will allow us to share the current state of the indexing strategy more easily in a single place, track it when updated and also already have it ready for our future users to look it up and understand the benefits of it.
* update typos
* fix one more typo
* apply review feedback
* skip fields that are disabled (#55735)
* [Fleet] Fix fleet typing issues after merging master
* [EPM] Create fieldFormatMap in kibana index pattern (#55892)
* add support for fieldFormatMap
* output params must be camel case
* fix case
* add url_template param
* [Fleet] Use user from saved object to create apiKeys (#55458)
* [EPM] Document package upgrade behaviour (#56138)
This PR adds more detailed documentation on what should happen when a package is upgraded.
* Remove some explicit typing to pass type checks.
This abstraction will be removed in the single plugin going into master.
* remove readFromDocValues (#56227)
* Add symlink to yarn.lock to fix(?) CI
https://github.com/elastic/kibana/pull/56443/checks?check_run_id=418123781 failed saying EPM directory "MUST have a 'yarn.lock' symlink"
Seems to have originated with https://github.com/elastic/kibana/pull/55440
Following example from other legacy/plugins/* in that PR
* Like 441d9ed, but correct
* [Ingest] Convert `ingest` plugin to new platform `ingest_manager` plugin (#56262)
* Seed Ingest Manager as a new NP plugin
* Add contexts for core, deps, and config. Begin routing and nav UI
* Export NP ready request from top-level es_ui_shared/public/
* Add nav styling w/ theming, add useRequest hook
* Set up license and config server-side services; add test routes
* Move most types and constants into /common
* Initial pass at:
* data stream and agent config models
* data stream routes and schemas
* Initial pass at agent config api route handlers
* Change plugin id to camel case
* Fix circular schema dependency, add security as optional plugin
* Create appContext service, use request user info in agent config routes + libs
* Create default agent config
* Add default output host config, output typings, and create default output and its api key
* Move saved object mapping to new plugin
* Change data streams -> datasources
* Add legacy plugin to bootstrap mappings
* Adjust fleet's ingest dependencies
* Disable policies UI in Fleet
* Adjust EPM's ingest dependencies
* Adjust ingest manager base API route
* Adjust fleet's client side ingest dependencies
* Remove more ingest dependencies from fleet
* REMOVE MOST OF LEGACY INGEST PLUGIN
* Add section for agent configs in UI nav
* Allow useRequest and sendRequest consumers to specify typing for response
* Initial pass at porting over agent config list UI
* Port over agent config creation
* Port over delete agent config functionality
* Fix app routing
* Port over fleet setup routes
* Adjust fleet's ingest dependencies
* Make fleet happy path work, skip some tests (MESSY! :))
* Remove policy list UI code from fleet
* Change useRequestResponse error type
* Add missing agent config schemas and hooks
* Fix type check issues
* Register IM under management
* Fix type issues as a result of changes to use/sendRequest interfaces
* Make all ingest saved objects *not* space-aware
* Fix i18n path
* Fix app categories import
* Fix datasource package assets schema (array of asset objects)
* Seed Ingest Manager privileges to fix tests
* Change `features` to optional plugin instead of required
* Fix security privileges tests
* Fix feature test
* Fix duplicate enrollment key created for default agent config
* Fix fleet agent enrollment by catching agent config 404
* PR feedback
* [Fleet] Detailed docs of fleet <-> agents interactions (#56212)
* [EPM] update index patterns on install/uninstall of package (#56591)
* create index patterns functionality on install/uninstall of package
* update snapshots
* [Fleet] Generate an ES api key per agent per output (#56637)
* [Fleet] Remove unused enrollement rules (#56753)
* remove files related to creating data source (#56745)
* Fix typing issues after mergin master
* Fix api key authentication after master merge
* [EPM] NP Migration: Move server files and route handlers to ingest_manager (#56854)
* initial pass moving registry and categories endpoint
* moves all needed server files and gets list endpoint working
* add route list validation schema
* remove epm config
* use config to get the registryUrl
* clean up registry url
* gets file endpoint working
* add info endpoint
* get install package endpoint working
* support uninstall package endpoint
* add API response types
* move epm types to models
* move AssetType to IngestAsset type from ingest plugin
* remove redundant export
* update epm_api_integration tests to new endpoint paths in ingest manager
* fix imports
* [Ingest Management] Change indexing from {type}-{namespace}-{dataset} to {type}-{dataset}-{namespace} (#56132)
Currently we have the indexing strategy defined as `{type}-{namespace}-{dataset}`. In this PR I propose to change this to `{type}-{dataset}-{namespace}`. As all 3 fields are constant keyword fields, the orders does not matter in most cases.
The reason I propose this change is to better align the name of indices with the name of the other assets:
* ingest pipeline: {type}-{dataset}
* index template: {type}-{dataset}
* Index pattern: {type}-{dataset}-*
* alias name: {type}-{dataset}-{namespace}
This makes it easier to remember the asset names conventions (at least for me).
It makes a difference when specifying security per namespace: To lock down security, previously it is `/(logs|metrics)-prod-$/` and becomes `/(logs|metrics)-[^-]+-prod-$/`. In any case, we should help / assist the user to get this right.
* [EPM] Remove epm plugin and directory (#57309)
* delete server files, move over epm saved object schemas and mapping, stop epm plugin loading in xpack
* updated yarn
* change to updated name
* remove epm dir, copy readme to ingest_manager
* move package.json over and update yarn.lock
* update package name
* add yarn.lock symlink
* fix yarn.lock symlink
* remove epm from security privileges map
* remove epm from feature endpoint and i18nrc
* [Fleet] Move fleet to the new platform and to ingest_manager plugin (#56803)
* [Fleet] NP migration public part (#57567)
* [EPM] EPM to new plugin, UI part (#56882)
* Move EPM home / list view over to ingest-manager
* Use react-router-dom in epm section.
* WIP: add package detail view.
* Use correct route.
* Only import needed types to public
* Remove obsolete file.
* Import type correctly
* Revert "Remove obsolete file."
This reverts commit 4b061102ebc62b49e7d1291060405ea8d23a3a8a.
* Routes are still needed, fix them.
* Import types correctly
* More type import fixes.
* update get categories hook
* remove no longer used getCategories function
* get list packages hook working
* delete routes.tsx, cleanup links
* add the usePackageInstall hook
* replace rest of api calls with use/send request
* remove tmp_routes
* bring over breadcrumbs
* remove comments and get styles working
* get ride side col loading
* temp type fix
* remove useCore
* add assets
* remove comment
* add public directory to legacy ingest_manager and update asset path
* Fix PackageInfo type. Use for API & UI vs a saved object type.
The `as PackageInfo` cast was required because the pipeline was typed as returning `Installed | NotInstalled` which are saved object response.
Updating that to PackageInfo allows the `as` to be removed but revealed an incompatibility between the `assets` properties of RegistryPackage and PackageInfo
```
Types of property 'assets' are incompatible.
Type 'Record<"kibana", Record<KibanaAssetType, KibanaAssetParts[]>>' is missing the following properties from type 'string[]': length, pop, push, concat, and 28 more.
```
It seems the `RegistryPackage & PackageAdditions` didn't cause the PackageAdditions.assets to replace the RegistryPackage.assets property.
I changed the definition of PackageInfo to do what I initially thought it was doing. See comments in models/epm.ts for more about how the new type is constructed.
* remove comment
* fix paths
* fix public paths
* fix path
* remove ui types file
* fix types
Co-authored-by: Sandra Gonzales <neptunian@users.noreply.github.com>
Co-authored-by: John Schulz <github.com@jfsiii.org>
* [Fleet] AgentEvent change agent_id and remove data (#57818)
* Remove legacy `ingest` plugin completely (#58056)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* WIP. 1 type error (but >1 bugs) remaining.
* Add `callCluster` accessors for using ES vs appContext.getClusterClient()
* Undo (?) changes to kbn-pm/dist/index.js
* Run scripts/build_renovate_config for @types/tar
* Replace data w/ agent_id in server schema
* Different way to declare a saved object type
* Use a more specific path to the agent script
* Replace data with agent_id for agent checkin
* Restore internalSavedObjectsClient in app context
* Use project & HTTP TS types in scripts/dev_agent
* Remove ingestManager from FTR features
The plugin is disabled by default and not currently running the FTR tests. I believe we'll add this back when we restore the EPM integration tests.
* Move more variables to common/{constants,types}
* Remove ingestManager from default expected features
* Enable conditional routes. Adjust integration tests
EPM routes currently return a 500 for these tests. For now they `.expect(500)` when enabled and `.expect(404)` when disabled. We can look into the issue and get them to `.expect(200)` in later tests.
* Replace React.FC with React.FunctionComponent following new repo pattern
* Enable Fleet & Ingest FTR tests
* Remove duplicate *Response entries from server/types
* Update README instructions for CLI flags. Rearrange sections (#58363)
* Add instructions for CLI flags. Rearrange sections
Also added some more information about the plugin behavior with links to relevant code
* Add instructions for Ingest & Fleet FTR tests
* Restore search to EPM list page
* [Fleet] Allow to configure CA sha for kibana and elasticsearch (#58186)
* [Ingest] Adjust saved object mappings and rename policy -> config (#58670)
* Replace all reference to (agent) "policy"/"policies" with "config"/"configs"
* Adjust output and datasource saved object mappings
* Update schemas and types to match SO changes
* Fix type check
* Adjust default output object
* Fix property names in tests
* Move installing of index templates and ingest pipelines to package installation (#58619)
* update template and pipeline asset names, install on package install
* fix package install error handling messaging
* save references to installations to package saved object
* add epm.enabled flag for epm functional test runner
* don't add suffix to pipeline entry
* [Fleet] Add a schema of all of our saved objects (#58769)
* [Fleet] Do not use default id for saved objects that need to be encrypted (#57876)
* Adding events for index pattern generation (#58908)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fix yarn.lock after merge
* [Fleet] create agent config before enabling fleet user (#59166)
* support for top level elasticsearch assets installation (#58869)
* add support for ilm policy installation
* check if ilm policy exists
* handle prebuilt index templates, update tests
* cleanup
* update type in install
* fix installing index templates to create multiple ones for inheritance
* [Fleet] Create default output while creating default config (#59223)
* Use the new definitions from package-registry#176 (#59311)
* [Fleet] Remove our custom API key authentication (#59212)
* [Ingest] Full agent config schema & API (#59262)
* Add schema for full agent config and business logic to convert SO agent config -> full agent config
* Adjust output properties in full agent config
* Whitelist full agent config output fields
* [Fleet] UI Agent enrollment flyout (#58524)
* Use POST vs GET for EPM install & remove (#59367)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* install default packages during setup (#59330)
* install default packages during setup
* check if package is already installed
* [Ingest] Updates to Agent Configuration List UI (#59374)
- Added search bar
- Sync of columns to design
- Actions per row in popup menu
- Connect pagination and per-page count to API request
- Support for `kuery` url param
* Update docs from /api/:section to /api/ingest_manager/:section (#59422)
* [EPM] Add Streams TS type to mirror EPR's (#59446)
* Add Streams TS type in EPM to mirror EPR's
Follows the changes add in https://github.com/elastic/package-registry/pull/230/files#diff-7dea786222588c32c19238bffffee9c2
* Add RegistryPackage.datasources
* Add more detailed shape for Registry Vars
* Don't code in modify src/plugins/management
See PR convo at c47975535f (r389042975)
* [Ingest] Add agent counts to each agent config output of `/agent_configs` API (#59552)
* Added agent counts to getAgentConfigsHandler
* Show agent counts on Agent Config List
* [Fleet] Agent list header (#59487)
* [EPM] Use /packages & /packages/{pkgkey} (#59550)
* Use /packages & /packages/{pkgkey}
* Update paths in skipped tests
* Docs use /packages vs /package/{install,delete}
Some copy & code sample changes.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [Ingest] Design sync for Agent Configuration Create Flyout (#59479)
* Match form to design
* Support URI route param to open flyout
* [Ingest] Create data source step 1 & 2 (#59590)
* Add Error and PackageIcon components
* Add create datasource layout+navigation, and select package step
* Add description field to datasource, remove assets and description from datasource package
* Add temporary datasources typing for EPM package info
* Initial pass at configure datasources step. Stream vars only (no input vars yet)
* First pass at input vars config; separate components
* Fix issue with adding more than one datasource to a config
* Add shell review step; save datasouce
* Remove assign/unassign datasource from agent config details UI, replace with add datasource buttons
* Remove actions column from datasource table
* Initial pass at create datasource from package
* Move package to config service to /common, add tests
* Rename VarsEntry to RegistryVarsEntry, add datasets and datasources to RegistrySearchResult definition
* Add typings to create datasource flow
* Add real count of agents to select agent config list
* Ensure the necessary package is installed at time of datasource creation
* Use lowercase pkgkey for consistency
* Update EPM file path to use /packages (#59693)
* fixed header padding (#59711)
* [EPM] Use icons from packages, if present (#59765)
* [Ingest] Create data source step 3 (#59822)
* Make app setup loading state prettier
* Add review step of datasource wizard
* Change name to ID in agent config datasources field
* Fix types
* Add stored datasource to agent datasource unit tests
* Adjustment of registry typings and which registry copy fields to show to sync with elastic/package-registry#242
* Fix `multi` vars not populating with array: elastic/kibana#59724
* Account for if a stream is enabled by default from registry package definition: elastic/kibana#59724
* Adjust tests to account for last two commits
* Fix review page back link
* Fix d'oh typo
* [Ingest] Agent Config Details header and sub tabs navigation (#59783)
- Syncs Agent Config Details header to design
- Includes sub navigation tabs connected to route URL
- Agent Config List Create data source row action enabled
* [Fleet] update agent list UI (#59685)
* [Fleet] ensure default packages are added to the default config (#59759)
* [Fleet] fix output rename api_token => api_key (#60001)
* [Ingest] Address #59376 feedback (#59961)
* Disable create/destroy CTAs if no write capability
Use `core.application.capabilities.ingestManager.write` to test user permissions
* Add -all & -read tags for HTTP routes
* Update test .expect() to match description
* Add useCapabilities hook. Fix two issues with hiding/disabling CTA.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Missed one in e12a8ad8a4
* Use package icon as default when no other can be found (#60025)
* Remove duplicate xpack setting
FTR tests failing to start ES with error
```
ERROR ERROR: setting [xpack.security.authc.api_key.enabled] already set, saw [true] and [true]
```
https://github.com/elastic/kibana/pull/59376/checks?check_run_id=503930031 & https://github.com/elastic/kibana/pull/59376/checks?check_run_id=503975576 etc
It appears the xpack.security.authc.api_key.enabled flag was recently added to master in another part, so remove our instance of the setting to prevent the error
* Update EPM file tests to use /packages/{pkgkey}
These should have failed when the routes were changed. Will go back and see what happened.
* [Ingest] Add `revision` to agent configs & data sources (#59848)
* Add revision to agent config and datasource saved objects, add delete datasource service and datasource
* Add revision to full agent config output
* PR feedback
Co-authored-by: Sonja Krause-Harder <sonja.krause-harder@elastic.co>
Co-authored-by: Nicolas Ruflin <spam@ruflin.com>
Co-authored-by: Sandra Gonzales <neptunian@users.noreply.github.com>
Co-authored-by: Matt Apperson <me@mattapperson.com>
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Nicolas Chaulet <n.chaulet@gmail.com>
Co-authored-by: Matt Apperson <matthew.apperson@elastic.co>
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: neptunian <sandra.gonzales@elastic.co>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>
Co-authored-by: Henry <henry.harding@elastic.co>
This moves the navigation bar and the source configuration loading indicator such that the bar is immediately visible and usable even before the source configuration has finished loading.
closes#56728
* Add SearchBar UI. Query, filter, and dateRange work
* Sync url with search bar, fix excluded filters on BE
* fix welcome title
* Use KibanaReactContext, fix indexPattern fetch
* Mock data plugin, api and ui tests pass
* Add view and functional tests
* use observables to handle filter updates
* address comments
* Aggregate queue types being used by Beats
* Making tests pass again
* Updating fixture + tests for queue type
* Adding queue field to BeatsBaseStats type
* Add undefined guard
* Adding queue field to source fields list
* Fixing type error
Will need to be touching some of these soon as part of elastic/observability-dev#532 so I'm converting to TypeScript.
Get rid of default exports on all of them as well.
* Revert "Skip all logstash pipeline tests"
This reverts commit d8aa1fb2da.
* Reenable tests and fix them based on recent changes
* Fix more tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Plugin file and services in place. Some redux logic ported
* Port and update index pattern util
* Move reducers over to NP. Update refs in legacy
* Port inspector to NP
* Move some kibana services init to NP. Some cleaning
* Clean up work not related to reducers/store
* Ignore temp imports from NP. Clean up of changes unrelated to this PR
* More cleanup. Check injected vars avab. before calling to handle dashboard case
* Bind embeddables services the same way Maps app services bound. Create function for eventual init in NP
* Call binding from constructor. Fix npStart plugins arg
* Adapt changes from master
* Register inspector views for embeddable. Add NP folder to i18n
* Clean up. Add comments. Move inspector map view registration to NP
* Remove unused inspector files in legacy
* Move full screen action to legacy
* Add in missing tooltip updates
* Review feedback. Update constants and i18n_getters to latest in NP
* Review feedback. Add redundancy comments to common files redundant in legacy and NP
* Remove unneeded copy of parse xml string test in legacy
* Review feedback. Remove redundant portions. Export from NP where possible. General clean up
* Remove remaining refernce and case for 'TOUCH_LAYER'. It's never used
* rebalance x-pack groups
* fix typo in ciGroup10
* move from group 9 to group 5
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: spalger <spalger@users.noreply.github.com>
- Replaces the custom KqlFilterBar with Kibana's QueryStringInput. This means the wizard now supports both lucene and kuery input.
- Using this component we no longer need to do cross-imports from the ML plugin. The use of setDependencyCache is no longer necessary.
- Replaces the custom AppDependencies provider code with Kibana's KibanaContextProvider.
* [ML] Typescripting client side endpoint functions
* type clean up
* cleaning up http requests
* remove http generics
* better use of generics and type clean up
* removes some generics
* update comment
* updating data frame analytics types
* fixing type errors
* Refactor: Move checking of closed index to single point
We should rather only check if an index is currently closed the
moment before starting to reindex. We still store a flag to
indicate that we opened an index that was closed, but this
should not be set from the reindex handlers because the reindex
task may only start some time later in which case the closed
index could have been opened and our reindex job will open it
and close it again.
* Added debug log
* Added comment
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [ML] Adding shared services to ml setup contract
* adding data recognizer
* typescripting js client
* adding results service
* code clean up
* adding generic ml index search
* making cloud optional
* Added app load test for API Keys app
* Fixed linting issues.
* Fixed typecheck issue, edited config to enable API Keys app, and fixed typo in method name.
* Config fix.
* Fixed assertion for API Keys header.
* Added changes per nits on PR.
* Removed API Keys server config from the siem_cypress config since it's defined at the top level. Left a comment saying so.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Old search strategy cleanup
* restore rollup strategy
* Remove hasSearchStategyForIndexPattern
* ts
* fix jest tests
* cleanup exports
* pass index pattern type to server for rollups
* merge fix
* Fix types
* ts fixes
* oss strategy error handling
* update translations
* jest test fix
* Use indexType instead of index
* code review 1
* updated docs
* jest test
* jest snapshot
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [Metrics Alerts] Fix error when a metric reports no data
* Clarify no data case handler, add separate error state
* Throw error state when callCluster fails
Before EuiInMemoryTable had TypeScript support we used our own typings and some state & CSS fixes. This is now all solved by the original EUI component.
- Use EuiInMemoryTable instead of custom typed table.
- Deletes some legacy leftover files.
The Alert Instances list calculates duration on each page load, which makes it hard for the test runner to know what the correct value should be.
In the PR we expose the epoch used by the duration calculation in such a way that the test runner can read it and asses the duration correctly.
This is a pre-cursor to https://github.com/elastic/kibana/issues/58529
I realized a bit ago that we weren't making quite enough info available
in the action parameter templating that happens when alerts schedule
actions to execute. Missing were alert name, tags, and spaceId.
For the index threshold alert, I had added them to it's context, but
then every other action would have to do the same if they also
wanted those values.
So I added these as additional top-level variables that can be
used in templates, along with the alert id, alert instance id,
context, and state. The other bits in RawAlert didn't seem
that interesting, to be used as an action parameter.
* scaffolding and notes.md
* add skeleton event generator to kibana
* add optional entityID param to generateEvent
* add tree generation
* add tests
* working tests
* fix up tests
* fix linting
* fix event types
* make process parent types consistent
* make generator match types
* move test resolver node out of common types
* fix random string generation
* fix typecheck errors
* remove extraneous stuff
* address PR comments
* add test for full resolver tree
* cleanup
* make tests clearer
* add seedrandom to endpoint plugin. contains DONOTMERGE example code
* remove robs test
* start replacing random with seedrandom
* use seeded random for uuidv4
* separate out IP randomization
* typecheck fixes
Co-authored-by: oatkiller <robert.austin@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Convert our manual throwing of TypeError to a custom Error
Throwing a TypeError meant that our manual errors were indistinguishable
from, say, trying to invoke a method on undefined. This adds a custom
error, BadRequestError, that disambiguates that situation.
* Present API Error messages to the user
With Core's new HTTP client, an unsuccessful API call will raise an
error containing the body of the response it received. In the case of
SIEM endpoints, this will include a useful error message with
potentially more specificity than e.g. 'Internal Server Error'.
This adds a type predicate to check for such errors, and adds a handling
case in our errorToToaster handler.
If the error does not contain our SIEM-specific message, it will fall
through as normal and the general error.message will be displayed in the
toaster.
* Remove unnecessary use of throwIfNotOk in our client API calls
The new HTTP client raises an error on a 4xx or 5xx response, so there
should not be a case where throwIfNotOk is actually going to throw an
error.
The established pattern on the frontend is to catch errors at the call
site and handle them appropriately, so I'm mainly just verifying that
these are caught where they're used, now.
* Move errorToToaster and ToasterError to general location
These were living in ML since that's where they originated. However, we
have need of it (and already use it) elsewhere.
The basic pattern for error handling on the frontend is:
1) API call throws error
2) caller catches error and dispatches a toast
throwIfNotOk is meant to convert the error into a useful message in 1).
We currently use both errorToToaster and displayErrorToast to display
that in a toaster in 2)
Now that errorToToaster handles a few different types of errors, and
throwIfNotOk is going to be bypassed due to the new client behavior of
throwing on error, we're going to start consolidating on:
1) Api call throws error
2) caller catches error and passes it to errorToToaster
* Refactor Rules API functions to not use throwIfNotOk
* Ensures that all callers of these methods properly catch errors
* Updates error toasterification to use errorToToaster
* Simplifies tests now that we mainly just invoke the http client and
return the result.
throwIfNotOk is not being used in the majority of cases, as the client
raises an error and bypasses that call.
The few cases this might break are where we return a 200 but have errors
within the response. Whether throwIfNotOk handled this or not, I'll need
a simpler helper to accomplish the same behavior.
* Define a type for our BulkRule responses
These can be an array of errors OR rules; typing it as such forces
downstream to deal with both. enableRules was being handled correctly
with the bucketing helper, and TS has confirmed the rest are as well.
This obviates the need to raise from our API calls, as bulk errors are
recoverable and we want to both a) continue on with any successful rules
and b) handle the errors as necessary. This is highly dependent on the
caller and so we can't/shouldn't handle it here.
* Address case where bulk rules errors were not handled
I'm not sure that we're ever using this non-dispatch version, but it was
throwing a type error. Will bring it up in review.
* Remove more throwIfNotOk uses from API calls
These are unneeded as an error response will already throw an error to
be handled at the call site.
* Display an error toaster on newsfeed fetch failure
* Remove dead code
This was left over as a result of #56261
* Remove throwIfNotOk from case API calls
Again, not needed because the client already throws.
* Update use_get_tags for NP
* Gets rid of throwIfNotOK usage
* uses core http fetch
* Remove throwIfNotOk from signals API
* Remove throwIfNotOk
This served the same purpose as errorToToaster, but in a less robust
way. All usages have been replaced, so now we say goodbye.
* Remove custom errors in favor of KibanaApiError and isApiError type predicate
There was no functional difference between these two code paths, and
removing these custom errors allowed us to delete a bunch of associated
code as well..
* Fix test failures
These were mainly related to my swapping any remaining fetch calls with
the core router as good kibana denizens should :salute:
* Replace use of core mocks with our simpler local ones
This is enough to get our tests to pass. We can't use the core mocks for
now since there are circular dependencies there, which breaks our build.
* add signal api unit tests
* privilege unit test api
* Add unit tests on the signals container
* Refactor signals API tests to use core mocks
* Simplifies our mocking verbosity by leveraging core mocks
* Simplifies test setup by isolating a reference to our fetch mock
* Abstracts response structure to pure helper functions
The try/catch tests had some false positives in that nothing would be
asserted if the code did not throw an error. These proved to be masking
a gap in coverage for our get/create signal index requests, which do not
leverage `throwIfNotOk` but instead rely on the fetch to throw an error;
once that behavior is verified we can update those tests to have our
fetchMock throw errors, and we should be all set.
* Simplify signals API tests now that the subjects do less
We no longer re-throw errors, or parse the response, we just return the
result of the client call. Simple!
* Simplify API functions to use implict returns
When possible. Also adds missing error-throwing documentation where
necessary.
* Revert "Display an error toaster on newsfeed fetch failure"
This reverts commit 64213221f5.
* Error property is readonly
* Pull uuid generation into default argument value
* Fix type predicate isApiError
Uses has to properly inspect our errorish object. Turns out we have a
'message' property, not an 'error' property.
* Fix test setup following modification of type predicate
We need a message (via new Error), a body.message, and a
body.status_code to satisfy isApiError.
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add a hook for seamlessly handling onClick and href props of links, buttons etc
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Felix Stürmer <weltenwort@users.noreply.github.com>
* add management section to SavedObjectsType
* adapt import/export routes to get types accessor
* add documentation
* update generated doc
* update migration guide
* use request context to access exportable types
* update generated doc
* adapt SavedObjectsManagement to use the registry
* stop magical tricks about the config type, register it as any other so type.
* fix FTR assertions
* fix so_mixin tests
* register the `config` type from the uiSettings service
* nits and comments
* update generated doc
* remove true from dynamic property definition, use force-cast back for config type
* remove obsolete test comment
in cytoscape graph. Also selects root nodes with no incoming edges
rather than just unconnected nodes.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Prior to this PR, the alerting UI used two HTTP endpoints provided by the
Kibana watcher plugin, to list index and field names. There are now two HTTP
endpoints in the alerting_builtins plugin which will be used instead.
The code for the new endpoints was largely copied from the existing watcher
endpoints, and the HTTP request/response bodies kept pretty much the same.
resolves https://github.com/elastic/kibana/issues/53041
* wip
* wip
* wip
* will this work?
* wip but it works
* pedro
* remove thing
* remove TODOs
* fix type issue
* add tests to check that alert index api works
* Revert "add tests to check that alert index api works"
This reverts commit 5d40ca18337cf8deb63a0291150780ec094db016.
* Moved schema
* undoing my evils
* fix comments. fix incorrect import
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Exclude disallowed, private setting at index creation
* Remove intl from tabs component
* Added logic for checking the current index status
* Added ES contract integration test
Using _cluster/state is considered internal. This adds an integration
test for checking the contract in CI.
* Add the client side notification for closed indices
* First version of end-to-end functionality working
* Clean up unused, incorrect type information
* Fix type issues and added a comment about the new reindex options
* Fixed server side tests, added comments and logic updates
Updated the handling of reindexOptions to make it more
backwards compatible (treat it as if it could be undefined).
Also update the logic for checking for open or closed indices.
No optional chaining! It should break if the response does not
exactly match.
* Clean up unused code
* Improved idempotency of test and check explicitly for "close".
Rather check for the specific value we want, as this is what is
also gauranteed by the tests. In this way, the information we
send back to the client is also more accurate regarding the index
status. If, in future, more index states are introduced this will
need to be revisited if it affects the ability for an index to be
re-indexed.
* Update client-side tests
* Fix types
* Handle a case where the index name provided may be an alias
* Fix types
* merge-conflict: finish merge conflict resolution
* Update x-pack/plugins/upgrade_assistant/public/application/components/tabs/checkup/deprecations/reindex/closed_warning_icon.tsx
Co-Authored-By: Alison Goryachev <alisonmllr20@gmail.com>
* merge-conflict: Remove duplicate import
VSCode does not auto-save as expected :sigh:
* ui: Revisit the UI
Moved the warning icon to inside of the button and tooltip to
on the button.
Added a callout to the reindex flyout for when an index is closed.
* logic: slight update to when the index closed callout is shown
We only show the index closed callout in the flyout when the
reindex operation is not considered "completed"
* tests: fix jest tests
* refactor: remove "openAndClose" from reindex endpoints
"openAndClose" should just happen automatically. The user should
not have to pass the flag in, that would be a weird API. We just
need to warn the user about that reindexing a closed index will
take more resources
* test: update upgrade assistant integration test
* fix: types
* copy: use sentence case
* refactor: use the in scope declaration of reindex op
* test: Clean up tests
Reindexing test was generating index name, could just get it from
server response. Also removed openAndClose from all integration
tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
When only one node is displayed, show an empty message.
Also:
* Start adding a basic Jest test for the ServiceMap component
* Fix bug where EuiDocsLink was rendering "children" instead of the actual children
Closes#59326.
Closes#59128.
