kibana/rfcs/text/0019_lifecycle_preboot.md

• Start Date: 2020-06-04
• RFC PR: (leave this empty)
• Kibana Issue: https://github.com/elastic/kibana/issues/89287

---

• 1. Summary
• 2. Motivation
• 3. Detailed design
  • 3.1 Core client-side changes
  • 3.2 Core server-side changes
    • 3.2.1 Plugins service
    • 3.2.2 HTTP service
    • 3.2.3 Elasticsearch service
    • 3.2.4 UI Settings service
    • 3.2.5 Rendering service
    • 3.2.6 I18n service
    • 3.2.7 Environment service
    • 3.2.8 Core app service
    • 3.2.9 Preboot service
    • 3.2.10 Bootstrap
• 4. Drawbacks
• 5. Alternatives
• 6. Adoption strategy
• 7. How we teach this
• 8. Unresolved questions
  • 8.1 Lifecycle stage name
  • 8.2 Development mode and basepath proxy
• 9. Resolved questions
  • 9.1 Core client-side changes

1. Summary

The preboot (see unresolved question 1) is the Kibana initial lifecycle stage at which it only initializes a bare minimum of the core services and a limited set of special-purpose plugins. It's assumed that Kibana can change and reload its own configuration at this stage and may require administrator involvement before it can proceed to the setup and start stages.

2. Motivation

The preboot lifecycle stage is a prerequisite for the Kibana interactive setup mode. This is the mode Kibana enters to on the first launch if it detects that user hasn't explicitly configured their own connection to Elasticsearch. In this mode, Kibana will present an interface to the user that would allow them to provide Elasticsearch connection information and potentially any other configuration information. Once the information is verified, Kibana will write it to the disk and allow the rest of Kibana to start.

The interactive setup mode will be provided through a dedicated userSetup plugin that will be initialized at the preboot stage.

3. Detailed design

The central part of the preboot stage is a dedicated HTTP server instance formerly known as Not Ready server. Kibana starts this server at the preboot stage and shuts it down as soon as the main HTTP server is ready to start, as illustrated at the following diagram:

Currently, preboot HTTP server only exposes a status endpoint and renders a static Kibana server is not ready yet string whenever users try to access Kibana before it's completely initialized. The changes proposed in this RFC should allow special-purpose plugins to define custom HTTP endpoints, and serve interactive client-side applications on this server, and hence make Kibana interactive setup mode possible.

3.1 Core client-side changes

The RFC aims to limit the changes to only those that are absolutely required and doesn't assume any modifications in the client-side part of the Kibana Core at the moment. This may introduce a certain level of inconsistency in the client-side codebase, but we consider it insignificant. See resolved question 1 for more details.

3.2 Core server-side changes

We'll update only several Core server-side services to support the new preboot lifecycle stage and preboot plugins.

Once none of the preboot plugins holds the setup anymore, Kibana might need to reload the configuration before it can finally proceed to setup. This doesn't require any special care from the existing plugin developers since Kibana would instantiate plugins only after it reloads the config. We'll also make sure that neither of the Core services relies on the stale configuration it may have acquired during the preboot stage.

3.2.1 Plugins service

First of all, we'll introduce a new type of special-purpose plugins: preboot plugins, in contrast to standard plugins. Kibana will initialize preboot plugins at the preboot stage, before even instantiating standard plugins.

Preboot plugins have only setup and stop methods, and can only depend on other preboot plugins. Standard plugins cannot depend on the preboot plugins since Kibana will stop them before starting the standard plugins:

export interface PrebootPlugin<TSetup = void, TPluginsSetup extends object = object> {
  setup(core: CorePrebootSetup, plugins: TPluginsSetup): TSetup;
  stop?(): void;
}

To differentiate preboot and standard plugins we'll introduce a new optional type property in the plugin manifest. The property can have only two possible values: preboot for preboot plugins and standard for the standard ones. If type is omitted, the standard value will be assumed.

// NOTE(azasypkin): all other existing properties have been omitted for brevity.
{
  "type": "preboot", // 'preboot' | 'standard' | undefined
}

The Plugins service will split plugins into two separate groups during discovery to use them separately at the preboot, setup, and start stages. The Core contract that preboot plugins will receive during their setup will be different from the one standard plugins receive, and will only include the functionality that is currently required for the interactive setup mode. We'll discuss this functionality in details in the following sections:

export interface CorePrebootSetup {
  elasticsearch: ElasticsearchServicePrebootSetup;
  http: HttpServicePrebootSetup;
  preboot: PrebootServiceSetup;
}

3.2.2 HTTP service

We'll change HTTP service to initialize and start preboot HTTP server (formerly known as Not Ready server) in the new preboot method instead of setup. The returned InternalHttpServicePrebootSetup contract will presumably be very similar to the existing InternalHttpServiceSetup contract, but will only include APIs we currently need to support interactive setup mode:

// NOTE(azasypkin): some existing properties have been omitted for brevity.
export interface InternalHttpServicePrebootSetup
  extends Pick<HttpServiceSetup, 'auth' | 'csp' | 'basePath' | 'getServerInfo'> {
  server: HttpServerSetup['server'];
  externalUrl: ExternalUrlConfig;
  registerRoutes(path: string, callback: (router: IRouter) => void): void;
}

The only part of this contract that will be available to the preboot plugins via CorePrebootSetup is the API to register HTTP routes on the already running preboot HTTP server:

export interface HttpServicePrebootSetup {
  registerRoutes(path: string, callback: (router: IRouter) => void): void;
}

The Core HTTP context available to handlers of the routes registered on the preboot HTTP server will only expose the uiSettings service. As explained in the UI Settings service section, this service will only give access to the default Core UI settings and their overrides set through Kibana configuration, if any.

// NOTE(azasypkin): the fact that the client is lazily initialized has been omitted for brevity.
export interface PrebootCoreRouteHandlerContext {
  readonly uiSettings: { client: IUiSettingsClient };
}

The authentication and authorization components are not available at the preboot stage, and hence all preboot HTTP server routes can be freely accessed by anyone with access to the network Kibana is exposed to.

Just as today, Kibana will shut the preboot HTTP server down as soon as it's ready to start the main HTTP server.

3.2.3 Elasticsearch service

As mentioned in the Motivation section, the main goal of the interactive setup mode is to give the user a hassle-free way to configure Kibana connection to an Elasticsearch cluster. That means that users might provide certain connection information, and Kibana preboot plugins should be able to construct a new Elasticsearch client using this information to verify it and potentially call Elasticsearch APIs.

To support this use case we'll add a new preboot method to the Elasticsearch service that will return the following contract, and make it available to the preboot plugins via CorePrebootSetup:

export interface ElasticsearchServicePrebootSetup {
  readonly createClient: (
    type: string,
    clientConfig?: Partial<ElasticsearchClientConfig>
  ) => ICustomClusterClient;
}

The Elasticsearch clients created with createClient rely on the default Kibana Elasticsearch configuration and any configuration overrides specified by the consumer.

NOTE: We may need to expose a full or portion of Elasticsearch config to the preboot plugins for them to check if the user has already configured Elasticsearch connection. There are other ways to check that without direct access to the configuration though.

3.2.4 UI Settings service

We'll introduce a new preboot method in the UI Settings service that will produce a UI Settings client instance. Since during the preboot stage Kibana can access neither user information nor Saved Objects, this client will only give access to the default Core UI settings and their overrides set through Kibana configuration, if any:

export interface InternalUiSettingsServicePrebootSetup {
  defaultsClient(): IUiSettingsClient;
}

UI Settings service isn't strictly necessary during the preboot stage, but many Kibana Core components rely on it explicitly and implicitly, which justifies this simple change.

3.2.5 Rendering service

We'll introduce a new preboot method in the Rendering service that will register Kibana main UI bootstrap template route on the preboot HTTP server as it does for the main HTTP server today. The main difference is that bootstrap UI will only reference bundles of the preboot plugins and will rely on the default UI settings.

3.2.6 I18n service

We'll introduce a new preboot method in the I18n service to only include translations for the Core itself and preboot plugins in the translations bundle loaded with the preboot UI bootstrap template. This would potentially allow us to switch locale during interactive setup mode if there is such a need in the future.

3.2.7 Environment service

There are no changes required in the Environment service itself, but we'll expose one additional property from its setup contract to the plugins: the paths to the known configuration files. The interactive setup mode should be able to figure out to which configuration file Kibana should save any changes users might need to make.

3.2.8 Core app service

We'll introduce a new preboot method in the Core app service to register routes on the preboot HTTP server necessary for the rendering of the Kibana preboot applications. Most of the routes will be the same as for the main HTTP server, but there are three notable exceptions:

1. JS bundles routes will only include those exposed by the preboot plugins

2. Default route for the preboot HTTP server will be hardcoded to the root path (/) since we cannot rely on the default value of the defaultRoute UI setting (/app/home)

3. Main application route (/app/{id}/{any*}) will be replaced with the catch-all route (/{path*}). The reason is that if the user tries to access Kibana with a legit standard application URL (e.g. /app/discover/?parameters) while Kibana is still at the preboot stage, they will end up with Application is not found error. Instead, with the catch-all route, Kibana will capture the original URL in the next query string parameter and redirect the user to the root (e.g. /?next=%2Fapp%2Fdiscover%2F%3Fparameters). This will allow us to automatically redirect the user back to the original URL as soon as Kibana is ready. The main drawback and limitation of this approach are that there can be only one root-level preboot application. We can lift this limitation in the future if we have to though, for example, to support post-preboot Saved Objects migration UI or something similar.

Serving a proper Kibana application on the root route of the preboot HTTP server implies that we'll also have a chance to replace the static Kibana server is not ready yet string with a more helpful and user-friendly application. Such application may potentially display a certain set of Kibana status information.

3.2.9 Preboot service

To support interactive applications at the preboot stage we should allow preboot plugins to pause Kibana startup sequence. This functionality will be exposed by the new Preboot service, and will be available to the preboot plugins via CorePrebootSetup. Preboot plugins will be able to provide a promise to hold setup and/or start for as long as needed, and also let Kibana know if it has to reload configuration before it enters the setup stage.

export interface PrebootServiceSetup {
  readonly isSetupOnHold: () => boolean;
  readonly holdSetupUntilResolved: (
    reason: string,
    promise: Promise<{ shouldReloadConfig: boolean } | void>
  ) => void;
  readonly isStartOnHold: () => boolean;
  readonly holdStartUntilResolved: (
    reason: string,
    promise: Promise<void>
  ) => void
}

Preboot service will provide a pair of helper isSetupOnHold and isStartOnHold methods that would allow consumers to check if setup or start are on hold before they are blocked on waiting.

Internal Preboot service contract will also expose waitUntilCanSetup and waitUntilCanStart methods that bootstrap process can use to know when it can proceed to setup and start stages. If any of these methods returns a Promise that is rejected, Kibana will shut down.

// NOTE(azasypkin): some existing properties have been omitted for brevity.
export interface InternalPrebootServiceSetup {
  readonly waitUntilCanSetup: () => Promise<{ shouldReloadConfig: boolean } | void>;
  readonly waitUntilCanStart: () => Promise<void>;
}

3.2.10 Bootstrap

We'll update Kibana bootstrap sequence to include preboot stage and to conditionally reload configuration before proceeding to setup and start stages:

// NOTE(azasypkin): some functionality and checks have been omitted for brevity.
const { preboot } = await root.preboot();

const { shouldReloadConfig } = await preboot.waitUntilCanSetup();
if (shouldReloadConfig) {
  await reloadConfiguration('pre-boot request');
}
await root.setup();

await preboot.waitUntilCanStart();
await root.start();

It's not yet clear if we need to adjust the base path proxy to account for this new lifecycle stage (see unresolved question 2).

4. Drawbacks

The main drawback is that proposed changes affect quite a few Kibana Core services that may impose a risk of breaking something in the critical parts of Kibana.

5. Alternatives

The most viable alternative to support interactive setup mode for Kibana was a standalone application that would be completely separated from Kibana. We ruled out this option since we won't be able to leverage existing and battle-tested Core services, UI components, and development tools. This would make the long-term maintenance burden unreasonably high.

6. Adoption strategy

The new preboot stage doesn't need an adoption strategy since it's intended for internal platform use only.

7. How we teach this

The new preboot stage shouldn't need much knowledge sharing since it's intended for internal platform use only and doesn't affect the standard plugins. All new services, methods, and contracts will be sufficiently documented in the code.

8. Unresolved questions

8.1 Lifecycle stage name

Is preboot the right name for this new lifecycle stage? Do we have a better alternative?

8.2 Development mode and basepath proxy

Currently, the base path proxy blocks any requests to Kibana until it receives SERVER_LISTENING message. Kibana's main process sends this message only after start, but we should change that to support interactive preboot applications. It's not yet clear how big the impact of this change will be.

9. Resolved questions

9.1 Core client-side changes

The server-side part of the preboot plugins will follow a new PrebootPlugin interface that doesn't have a start method, but the client-side part will stay the same as for standard plugins. This significantly simplifies implementation and doesn't introduce any known technical issues, but, unfortunately, brings some inconsistency to the codebase. We agreed that it's tolerable assuming we define a dedicated client-side PrebootPlugin interface that would hide from CoreStart all services that are unavailable to the preboot plugins (e.g., Saved Objects service).