* provide packageInfo to server plugins
* provide env to kbn server
* provide env to client plugins
* update server mocks and tests
* update client mocks and tests
* add integration test
* regen docs
* remove weird file
* export types for the client from special folder
* env should not affect test ourcome
* [APM] Replace ui/kfetch with core.http
Closes#46548.
* Remove kfetch mocks in tests
* Expose HttpFetchError from src/core/public/index
* Make HttpFetchError public
* Simplify tests for ServiceOverview
* Add button with link to license management in ML popup
* Corrections to button styling
* Updated button icon
* Update snapshots
* Correcting json translation files
* Fix zh_CN json file
* And now fixing the translation file properly
* [ML] Adding datafeed validation to advanced wizard
* shared invalid time format message
* updating translations
* adding time format validation function
* Add API Keys app to Management > Security.
- For admins, list all API Keys created by the user: Name, Date Created, Expiration Date, Status, User, and Realm.
- For non-admins, list own API keys: Name, Date Created, Expiration Date, and Status.
- Surface admin status above table.
- Ability to search by Name and Revoke (invalidate) API keys, and filter by User and Realm.
- Surface feedback when API keys are disabled on Elasticsearch or when user lacks required permissions.
* Add `SectionLoading` component to `es_ui_shared` plugin.
* Fail file-protocol requests and bogon IP
* Revert "Fail file-protocol requests and bogon IP"
This reverts commit 8a1ff56179.
* Ensuring all URLs from PDF and PNG reports are relative
* Missing dep, saw it in a prior yarn.lock
* Tighenting our URL checks
* More edgecases that can be triggered with a window.goto type behavior
* Javascript URLs
* Tightening implementation
* New networkPolicy that allows for setting allow/deny list for chromium requests
* Fixing tests, always fail file:// URLs
* Never allow file responses
* Make sure we test other protocols in deny list
* Don't allow `file:` protocols in the allow-list
* Expanding upon network policy to match ufw-style patterns
* Applying network policies to outbound and inbound requests
* Fixing gateway logic on network-policy
* My network-policy opus
* Moving to more explicit ufw format
* Updating snapshots
* Default reject requests when enabled and no rule matches
* [Maps] clean features in locked tooltip after re-fetch
* fetch geometry from FEATURE_ID_PROPERTY_NAME instead of _id
* set FEATURE_ID_PROPERTY_NAME for pew pew source
* do not update tooltip state if no features were removed
* set FEATURE_ID_PROPERTY_NAME for EMS_file source and kibana_regionmap source
* avoid adding lodash to map actions
* use if else instead of early return
## Summary
* Adds a Read, Update, Delete, and Find API endpoints.
* Adds several scripts to exercise the endpoints as well as improves the helper scripts
* Fixes a bad assumption with the way alert params works with `null`
* Fixes a bug where I was using an array instead of a number of `max_signals`
* Fixes a bug with the log level since it upgraded recently and requires a log level
IMPORTANT NOTE:
---
This still uses auto-generated GUID's and not the alert id so there are not stable id's just yet. However, either we will add that capability through alert params or the alerting team will add the capability to do a POST of the {id} into the create endpoints.
Testing:
---
Follow the `README.md` for initial setup of our temporary environment variables. Use the scripts and post a signal after a hard reset like so:
```ts
./hard_reset.sh
./post_signal.sh
```
Then run the following scripts to test each piece:
```sh
# Creates a new signal
./post_signal.sh
{
"id": "908a6af1-ac63-4d52-a856-fc635a00db0f",
"alertTypeId": "siem.signals",
"interval": "5m",
"actions": [
{
"group": "default",
"params": {
"message": "SIEM Alert Fired"
},
"id": "7edd7e98-9286-4fdb-a5c5-16de776bc7c7"
}
],
"alertTypeParams": {},
"enabled": true,
"throttle": null,
"createdBy": "elastic",
"updatedBy": "elastic",
"apiKeyOwner": "elastic",
"scheduledTaskId": "4f401ca0-e402-11e9-94ed-051d758a6c79"
}
# Read a signal that is from the result
./read_signal.sh 908a6af1-ac63-4d52-a856-fc635a00db0f
# Edit the file `vim signals/temp_update_1.json` (manually) and add the ID into
# it like so since we don't have stable alert ID's just yet
{
"id": "908a6af1-ac63-4d52-a856-fc635a00db0f",
"description": "Only watch winlogbeat users",
"index": ["winlogbeat-*"],
"interval": "9m",
"name": "Just watch other winlogbeat users",
"severity": 500,
"enabled": false,
"type": "filter",
"from": "now-5d",
"to": "now-1d",
"kql": "user.name: something_else"
}
# Then update it
./update_signal.sh
# Then run a find to see all signals
./find_signals.sh
# Delete the signal
./delete_signal.sh 908a6af1-ac63-4d52-a856-fc635a00db0f
```
Take a look at the arguments and play around with removing fields from the update document as well as using different features of the API to see if something is broken for the initial roll out.
### Checklist
Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.
~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~
~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~
~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~
~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~
~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~
### For maintainers
~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
When running Jest with `--coverage`, modules using idx fail with the error shown in facebookincubator/idx#19.
Change the babel configuration so that is `NODE_ENV=test`, the idx plugin is not loaded.
* [ML] Adding all ML detector functions to advanced wizard
* updating stubbed endpoint responses
* updating tests
* changes based on review
* small refactor
* working commit with no re-index.. very messy working draft
* Add flag for using GET/PUT instead of reindex, remove painless script from GET/PUT, clean up code
* unset alerting and actions features since feature flags are not present
* fix leftover merge conflicts from rebase
* Defaults to scroll and bulk index combination instead of reindex, updates docs, updates erroneous log.info to log.error
* Refactors bulkIndex method to remove useage of let and increase readability
* Replaces env variable, remove scroll and bulk index logic from signals_alert_type.ts and updates logger.info -> logger.error in catch clause
* add ternary for scroll lock parameter
* minor cleanup from merge with master
* removes class definition and exports separate functions for scroll and bulk functionality, replaces logger.info with a logger.error, adds another signal for testing zero documents upon initial search query
* Fix time filter woes and add tests
* Fix up tests and fix loading states for non ssp tables
* Only add onChange if searching is enabled
* Replace missing function
Summary
Adds pagination to the Uptime app's monitor overview page. Also cleans up/removes a lot of unused code that laid the foundation for the states index approach we opted not to take.
This PR is somewhat complex due to our need to use composite aggregations with searchAfter style pagination.
Understanding our new query approach
How the schema works
At its simplest, we run a single heartbeat from a single location checking a single endpoint with a single IP.In this case, everytime we run a check we create a new document.
------ time --------------->
[D] [D] [D] [D] [D] [D] [D]
We can also check from multiple geo locations
Location | ---------- time ----------->
US-E ` [D] [D] [D] [D] [D] [D] [D]
US-W ` [D] [D] [D] [D] [D] [D] [D]
We can also check multiple ip addresses per check if there are multiple DNS entries. If the endpoint we're checking has 3 DNS entries we will create 3 documents per check. The final document has an extra summary field with two integer values: summary.up and summary.down summarizing how many documents were up vs down across all three documents in the check. All documents also contain a monitor.check_group UUID string field that has a shared value across all three documents.
Note the partially written group at the end
--------------- time --------------------->
Docs | [D] [D] [S] [D] [D] [S] [D]
Groups | |--UUID1--| |--UUID2--| |-UUID3-
Putting it all together, we can have multiple geo locations with multiple IPs checked within each location.
Location | ------------------time-------------------------->
`
US-W ` [D] [D] [S] [D] [D] [S] [D]
` |--UUID1--| |--UUID2--| |-UUID3-
`
US-E ` [D] [D] [S] [D] [D] [S] [D] [D] [S]
` |--UUIDA--| |--UUIDB--| |--UUIDC--|
The Query
Goals
Logically, what we want to do when searching is to:
Match the most recent complete check group from each location and return all documents from that check group.
For any specific fields (say error.message or monitor.ip) consider the monitor matched if any documents in any location have that value.
For status filtering consider the monitor up if all documents within the latest check groups for each location are up.
Additional properties we'd like this query to have:
Be paginatable
Be fast
Be in a consistent order (required for pagination)
At a High Level
There are three phases, described in detail below, but the TL;DR is:
queryPotentialMatches() Find monitor.id,mostrecent(check_group) tuples that match the query and all filters except status. We eagerly fetch 500 results here knowing that we may discard some of these results in the next phase.
refinePotentialMatches() Perform an additional query that pulls in all the most recent check summaries from all locations for the monitor IDs from the last phase. We compare the matched check groups from the last phase to see if what matched previously was out of date. We can now apply the status filter as well.
enrich At this point we already have the matching monitor IDs and their status, but we haven't brought in most of their fields. Here we re-query to get all docs from the matching monitor IDs most recent checks, then format them for return via API.
queryPotentialMatches()
This query returns via a composite agg all monitor IDs matching all query. It also returns the latest matching monitor.check group of all monitor IDs with the correct monitor.status value. Note that the returned monitor.check_group value will only be the most recent value matching the query terms. It may be an old check.
So, the output of the check is all monitor.id values that have ever had a document that matched the query. However, these matches may be old.
TODO: The PR as it stands doesn't work exactly this way, I added in an optimization for status filtering that actually is incorrect. We can exclude matches that are down if we're looking for things that are up, but nothing more than that.
refinePotentialMatches()
In this phase we take the monitor.id values from the preceeding phase and query for all the latest check groups per geo-location for each via a terms query plus terms aggs, using a top hit to get the most recent (top hit size 1 is more efficient than a terms agg on a high cardinality string field sorted , I believe due to global ordinals). Then, in Javascript, we can further exclude monitors based on the monitor.status fiter. If the status filter is up we exclude any monitors that have any down checks using the summary fields. By using the summary fields we additionally ensure that we only include the most recent complete check groups, instead of partial ones.
We then combine the output of the preceding phase, kicking out results from the initial phase that didn't have their latest complete check groups match.
Enrichment
We already have the correct monitor IDs, but we don't have all the data from all matching documents inside the check groups. We only got the summary documents (and only returned the monitor.id and check group) in the preceding phase. In this straightforward phase we get all the documents for the latest check groups for each monitor and combine them into the final response. Additionally adding histogram data via an auxiliary query.
Pagination
The query phases described above omit discussion of how pagination works. We don't want to retrieve all matching monitor.ids, if a user has 30k monitors that can be expensive. Our default page only shows 10 in this patch. By limiting results we increase speed. Additionally, by using "after" style pagination as composite aggregations use, each page load is as fast as the first, we don't have the performance issues traditional page number style pagination has as you go deep into the result set.
Internally the code has an Iterator style paginator object covering all phases except the final enrichment phase that has the following functions:
next() Gets the next monitor matching the query advances the internal cursor.
peek() Gets the next monitor matching the query, does not advance the internal cursor.
current() Gets the monitor at the current cursor
paginationAfterCurrent() Checks via peek to see if there is more data, and if so returns pagination data to be sent on the next request to get the next page. This powers the 'next' button.
paginationBeforeCurrent() Same as preceeding, but gets the pagination info for the page before the current item. TODO add this method instead of the current hacky way with reverse()
Internally we overfetch for the initial filter query to provide a buffer for the paginator; we don't want a query per next() invocation. That's currently set to 500 items at the moment, which is pretty snappy.
Fixeselastic/uptime#63
Alternative approaches considered
Using data frames: We rejected this approach for two reasons: 1.) It's one more thing that a user can accidentally break 2.) Worry over the cost of constantly re-indexing every document with low latency. We'd need to reindex everything once every 5-10s to prevent excessive lag.
Kibana Background jobs and a high water mark. This would involve having a background job in Kibana that would compute the most recent documents for each monitor at a given time and then mark them with a high water mark using an update by query. This has the same issues as the data frames in terms of excessive doc writes and scalability. However, there's less to break in a sense in that there's no extra indices. OTOH, users have configure kibana to let it write to uptime indices.
