Search Sessions will enable Kibana applications and solutions to start a group of related search requests (such as those coming from a single load of a dashboard or SIEM timeline), navigate away or close the browser, then retrieve the results when they have completed.
// If this is a new session, the `search_interceptor` will associate and keep track of the async search ID with the session ID.
// If this is a restored session, the server will immediately return saved results.
// In the case where there is no saved result for a given request, or if the results have expired, `search` will throw an error with a meaningful error code.
Kibana is great at providing fast results from large sets of "hot" data. However, there is an increasing number of use cases where users want to analyze large amounts of "colder" data (such as year-over-year reports, historical or audit data, batch queries, etc.).
For these cases, users run into two limitations:
1. Kibana has a default timeout of 30s per search. This is controlled by the `elasticsearch.requestTimeout` setting (originally intended to protect clusters from unintentional overload by a single query).
2. Kibana cancels queries upon navigating away from an application, once again, as means of protecting clusters and reducing unnecessary load.
In 7.7, with the introduction of the `_async_search` API in Elasticsearch, we provided Kibana users a way to bypass the timeout, but users still need to remain on-screen for the entire duration of the search requests.
The primary motivation of this RFC is to enable users to do the following without needing to keep Kibana open, or while moving onto other work inside Kibana:
- Run long search requests (beyond 30 seconds)
- View their status (complete/incomplete)
- Cancel incomplete search requests
- Retrieve completed search request results
# Detailed design
Because a single view (such as a dashboard with multiple visualizations) can initiate multiple search requests, we need a way to associate the search requests together in a single entity.
We call this entity a `session`, and when a user decides that they want to continue running the search requests while moving onto other work, we will create a saved object corresponding with that specific `session`, persisting the *sessionId* along with a mapping of each *request's hash* to the *async ID* returned by Elasticsearch.
## High Level Flow Charts
### Client side search
This diagram matches any case where `data.search` is called from the front end:
long as a user continues to extend the expiration.
## Frontend Services
Most sessions will probably not be saved. Therefore, to avoid creating unnecessary saved objects, the browser will keep track of requests and their respective search IDs, until the user chooses to store the session. Once a session is stored, any additional searches will be immediately saved on the server side.
### New Session Service
We will expose a new frontend `session` service on the `data` plugin `search` service.
The service will expose the following APIs:
```ts
interface ISessionService {
/**
* Returns the current session ID
*/
getActiveSessionId: () => string;
/**
* Sets the current session
*@param sessionId: The ID of the session to set
*@param isRestored: Whether or not the session is being restored
There are cases where search requests are issued by the server (Like TSVB).
We can simplify this flow by introducing a mechanism, similar to the frontend one, tracking the information in memory and polling for a saved object with a corresponding sessionId to store the ids into it.
```ts
interface SearchService {
/**
* The search API will accept the option `trackId`, which will track the search ID, if available, on the server, until a corresponding saved object is created.
**/
search: async (
context: RequestHandlerContext,
request: IEnhancedEsSearchRequest,
options?: ISearchOptions
) => ISearchResponse<Payload=any>
}
```
### Server Routes
Each route exposes the corresponding method from the Session Service (used only by the client-side service, not meant to be used directly by any consumers):
If the `EnhancedEsSearchStrategy` receives a `restore` option, it will attempt reloading data using the Search Session saved object matching the provided `sessionId`. If there are any errors during that process, the strategy will return an error response and *not attempt to re-run the request.
It will query the list of all incomplete sessions, and check the status of each search that is executing. If the search requests are all complete, it will update the corresponding saved object to have a `status` of `complete`. If any of the searches return an error, it will update the saved object to an `error` state. If the search requests have expired, it will update the saved object to an `expired` state. Expired sessions will be purged once they are older than the time definedby the `EXPIRED_SESSION_TTL` advanced setting.
Once there's a notification area in Kibana, we may use that mechanism to push completion \ error notifications to the client.
## Miscellaneous
#### Relative dates and restore URLs
Restoring a sessionId depends on each request's `sha-256` hash matching exactly to the ones saved, requiring special attention to relative date ranges, as having these might yield ambiguous results.
There are two potential scenarios:
- A relative date (for example `now-1d`) is being used in query DSL - In this case any future hash will match, but the returned data *won't match the displayed timeframe*. For example, a report might state that it shows data from yesterday, but actually show data from a week ago.
- A relative date is being translated by the application before being set to the query DSL - In this case a different date will be sent and the hash will never match, resulting in an error restoring the dashboard.
Both scenarios require careful attention during the UI design and implementation.
The former can be resolved by clearly displaying the creation time of the restored Search Session. We could also attempt translating relative dates to absolute one's, but this might be challenging as relative dates may appear deeply nested within the DSL.
The latter case happens at the moment for the timepicker only: The relative date is being translated each time into an absolute one, before being sent to Elasticsearch. In order to avoid issues, we'll have to make sure that restore URLs are generated with an absolute date, to make sure they are restored correctly.
If you have restored a Search Session, making any type of change to it (time range, filters, etc.) will trigger new (potentially long) searches. There should be a clear indication in the UI that the data is no longer stored. A user then may choose to send it to background, resulting in a new Search Session being saved.
When trying to restore a Search Session, if any of the requests hashes don't match the ones saved, or if any of the saved async search IDs are expired, a meaningful error code will be returned by the server **by those requests**. It is each application's responsibility to handle these errors appropriately.
Sessions are given an expiration date defined in an advanced setting (5 days by default). This expiration date is measured from the time the Search Session is saved, and it includes the time it takes to generate the results.