* [Reporting] Screenshot Service RFC
* rewrite summary
* simplify design
* Update 0009_screenshot_mode_service.md
* mention the 3 screenshot report apps
* try not to say this is a high-level service
* clarify that print media css is just ok
* clarify the intent
* drop the `app`
* add the possibility to test screenshot mode through a URL parameter
* keep it more low-level
* keep the discussion high level
* move a sectioin of text
* Rename alerts plugin to alerting
* Deprecate old config values
* Few more renames
* Update plugin list
* Rename xpack.alerts -> xpack.alerting
* Fix some ESLint rules
* Fix typecheck
* Fix some test failures
* Some more renames
* Fix ESLint
* Fix some test failures
* Fix failing jest test
* Undo exclusive test
* Fix APM deps
* Fix docs
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Instead of cloning, reindex legacy index
* Reindex for every v2 migration
* Use _reindex?require_alias=true and a write block toggle to prevent lost deletes
* Use a ..._reindex_in_progress alias so that waiting for and preventing other reindex operations is idempotent
The first version of the reindex block had only the instance which was able to mark the migration as complete
set and remove the write block. This means other instances couldn't know if any reindex operaitons were in
progress if the migration was already marked as complete. It also meant that a failure in this critical step
could result in a permanent write block.
* Revert "Use a ..._reindex_in_progress alias so that waiting for and preventing other reindex operations is idempotent"
This reverts commit 8baf9b13db.
* Revert "Use _reindex?require_alias=true and a write block toggle to prevent lost deletes"
This reverts commit d7237ca42c.
* Use reindex + clone as a way to prevent lost deletes
* Fix numbering and ignore index_not_found_exceptionfor temporary index
* Apply suggestions from code review
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Rename Background Sessions to Search Sessions (with a send to background action)
* doc
* doc
* jest fun
* rename rfc
* translations
* update so name in features
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Initial structure of migration state-action machine
* Fix type import
* Retries with exponential back off
* Use discriminated union for state type
* Either type for actions
* Test exponential retries
* TaskEither types for actions
* Fetch indices instead of aliases so we can collect all index state in one request
* Log document id if transform fails
* WIP: Legacy pre-migrations
* UPDATE_TARGET_MAPPINGS
* WIP OUTDATED_DOCUMENTS_TRANSFORM
* Narrow res types depending on control state
* OUTDATED_DOCUMENTS_TRANSFORM
* Use .kibana instead of .kibana_current
* rename control states TARGET_DOCUMENTS* -> OUTDATED_DOCUMENTS*
* WIP MARK_VERSION_INDEX_READY
* Fix and expand INIT -> * transition tests
* Add alias/index name helper functions
* Add feature flag for enabling v2 migrations
* split state_action_machine, reindex legacy indices
* Don't use a scroll search for migrating outdated documents
* model: test control state progressions
* Action integration tests
* Fix existing tests and type errors
* snapshot_in_progress_exception can only happen when closing/deleting an index
* Retry steps up to 10 times
* Update api.md documentation files
* Further actions integration tests
* Action unit tests
* Fix actions integration tests
* Rename actions to be more domain-specific
* Apply suggestions from code review
Co-authored-by: Josh Dover <me@joshdover.com>
* Review feedback: polish and flesh out inline comments
* Fix unhandled rejections in actions unit tests
* model: only delay retryable_es_client_error, reset for other left responses
* Actions unit tests
* More inline comments
* Actions: Group index settings under 'index' key
* bulkIndex -> bulkOverwriteTransformedDocuments to be more domain specific
* state_action_machine tests, fix and add additional tests
* Action integration tests: updateAndPickupMappings, searchForOutdatedDocuments
* oops: uncomment commented out code
* actions integration tests: rejection for createIndex
* update state properties: clearer names, mark all as readonly
* add state properties currentAlias, versionAlias, legacyIndex and test for invalid version scheme in index names
* Use CONSTANTS for constants :D
* Actions: Clarify behaviour and impact of acknowledged: false responses
* Use consistent vocabulary for action responses
* KibanaMigrator test for migrationsV2
* KibanaMigrator test for FATAL state and action exceptions in v2 migrations
* Fix ts error in test
* Refactor: split index file up into a file per model, next, types
* next: use partial application so we don't generate a nextActionMap on every call
* move logic from index.ts to migrations_state_action_machine.ts and test
* add test
* use `Root` to allow specifying oss mode
* Add fix and todo tests for reindexing with preMigrationScript
* Dump execution log of state transitions and responses if we hit FATAL
* add 7.3 xpack tests
* add 100k test data
* Reindex instead of cloning for migrations
* Skip 100k x-pack integration test
* MARK_VERSION_INDEX_READY_CONFLICT for dealing with different versions migrating in parallel
* Track elapsed time
* Fix tests
* Model: make exhaustiveness checks more explicit
* actions integration tests: add additional tests from CR
* migrations_state_action_machine fix flaky test
* Fix flaky integration test
* Reserve FATAL termination only for situations which we never can recover from such as later version already migrated the index
* Handle incompatible_mapping_exception caused by another instance
* Cleanup logging
* Fix/stabilize integration tests
* Add REINDEX_SOURCE_TO_TARGET_VERIFY step
* Strip tests archives of */.DS_Store and __MAC_OSX
* Task manager migrations: remove invalid kibana property when converting legacy indices
* Add disabled mappings for removed field in map saved object type
* verifyReindex action: use count API
* REINDEX_BLOCK_* to prevent lost deletes (needs tests)
* Split out 100k docs integration test so that it has it's own kibana process
* REINDEX_BLOCK_* action tests
* REINDEX_BLOCK_* model tests
* Include original error message when migration_state_machine throws
* Address some CR nits
* Fix TS errors
* Fix bugs
* Reindex then clone to prevent lost deletes
* Fix tests
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add steps to migrate from a legacy kibana index
* Clarify data loss from legacy index to alias with same name
* Use aliases api to safely add a .kibana alias to a legacy index
* RFC: Improve saved object migrations
* Added rolling upgrades as alternative
* Clarify advantage of re-using index for avoiding max shards open errors
* Add more detail about how dry run migrations and invalid document exports will be exposed to users
* Add separate section to highlight assumptions and tradeoffs
* Apply nits from code review
Co-authored-by: Josh Dover <me@joshdover.com>
* Move 'Single node migrations coordinated through a lease/lock' under alternatives
* Fix 5.2.2 document lock algorithm step 3.2
* Minor clarificiations
* Minimizing data loss with mixed Kibana versions: move 7.x algorithm into alternatives and clarify 8.x algorithm
* Adds impact idempotent restrictions have on upcoming/requested migration features
* nit: clarify 4.3.1.2.3
Clarify that if documents already exist, they aren't overwritten.
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
* Update 4.5.2: we need only one version-specific alias, not separate read/write aliases
* Add 4.0.4 the assumption that a saved object type will only ever be registered by one plugin
* Updates 4.3.1.2: Adds mappings compatibility check, adds soft deletes to prevent lost deletes, add note about data loss with mixed plugins on the same kibana version
* Don't try to minimize data loss for unsupported configurations, update index mappings during 8.x migration
- Move "4.5.2 Minimizing data loss with mixed Kibana versions (9.0)" to
alternatives section. Expand this section with a more detailed algorithm as
well as enumerating some data loss scenarios that could still occur.
- Add the supported upgrade procedure which guarantees no undetected data loss.
- Update "4.5.1 Migration algorithm (8.0)"
- Added a step to update index mappings
- If there's update version mismatch errors, complete all batches before
restarting the migration to improve performance.
* Flesh out upgrade and rollback procedure with gracefull shutdown
* New migration algorithm, upgrade procedure based on index cloning and new ES features
* Minor: clarify that index cloning algorithm solves (2.6) but (2.7) is out of scope
* Move 'Tag objects as invalid if their transformation fails' to the alternatives section
* Don't migrate when there's orphan documents so that rollback is always possible. Updates: 4.2.1.2 Migration algorithm: Cloned index per version & 4.2.1.4 Handling documents that belong to a disabled plugin
* Link to new Elasticsearch features required for the implementation
* 4.2.1.2 Clarify the purpose of the target index postfix
* Add open questions around mappings growing over time. Fix rollback index example
* minor: fix formatting
* Update assumptions and tradeoffs to match latest algorithm
* Review feedback
- Remove 4.0.2 since the latest algorithm resolves the tradeoff
- 4.2.1.2 should handle v6 `.kibana` indexes
* More alternatives for 4.2.1.4 Handling documents that belong to a disabled plugin. Don't introduce breaking changes in 7.x
* Handle new ES cluster without any indices. Fix clone API parameters
* Update 4.2.1.2: transform outdated documents on every startup, only fail for unknown types in 8.x
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
* add skeleton for global_search plugin
* base implementation of the server-side service
* add utils tests
* add server-side mocks
* move take_in_array to common folder
* implements base of client-side plugin
* add tests for server-side service
* fix server plugin tests
* implement `navigateToUrl` core API
* extract processResults for the client-side
* fetch server results from the client side
* factorize process_results
* fix plugin start params
* move things around
* move all server types to single file
* fix types imports
* add basic FTR tests
* add client-side service tests
* add tests for addNavigate
* add getDefaultPreference & tests
* use optional for RequestHandlerContext
* add registerRoutes test
* add base test for context
* resolve TODO
* common nits/doc
* common nits/doc on public
* update CODEOWNERS
* add import for declare statement
* add license check on the server-side
* add license check on the client-side
* eslint
* address some review comments
* use properly typed errors for obs
* add integration tests for the find endpoint
* fix unit tests
* use licensing start contract
* translate the error message
* fix eslint rule for test_utils
* fix test_utils imports
* remove NavigableGlobalSearchResult, use `application.navigateToUrl` instead.
* use coreProvider plugin in FTR tests
* nits
* fix service start params
* fix service start params, bis
* I really need to fix this typecheck oom error
* add README, update missing jsdoc
* nits on doc
* initial file import
* update types
* update RFC PR number
* first draft complete
* draft v1.1.0
* initial self review
* nits and comments
* add 'preference' option
* change meta type to Record<string, Serializable>
* specify cancellation
* remove generic for GlobalSearchResponse, use distinct type on client and server
* use plain string instead of enum for GlobalSearchResult.type
* remove terminology from unresolved questions
* update pros/cons of core vs plugin
* GS is exposed from a plugin instead of core
* remove the server.publicAddress proposal, return mixed abs url / rel paths instead
* distinguish result type between GS api and result providers
* return batched results, no longer sort
* remove request from GlobalSearchProviderContext
* add maxResults to GlobalSearchProviderFindOptions
* nit/typo
* [RFC][skip-ci] Pulse
* Add drawback
* Add Opt-In/Out endpoint
* Add clarification about synched local internal indices
* Update rfcs/text/0008_pulse.md
Co-Authored-By: Josh Dover <me@joshdover.com>
* Add Phased implementation intentions, Security and Integrity challenges and example of use
* Refer to a follow up RFC to talk about security in the future
* Fix wording + add Legacy behaviour
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Josh Dover <me@joshdover.com>
* Draft RFC for unblocking plugin lifecycle methods
* Draft RFC for unblocking kibana startup
* Rename rfc from 0006 to 0007
* Add references to TC39 top-level await
* Update with review suggestion
Co-Authored-By: Court Ewing <court@epixa.com>
* Update RFC#0007
* Apply suggestions from code review
Co-Authored-By: Aleh Zasypkin <aleh.zasypkin@gmail.com>
* Address review comments from @joshdover and @azasypkin
1. Fleshed out motivation, this RFC doesn't prevent errors, but isolates the
impact on the rest of Kibana.
2. Added a footnote explaining that sync lifecycles can still block on sync
for loops, so it's not a perfect guarantee (from @azasypkin).
3. Updated IContextProvider type signature in (2) to match latest master
4. Dynamically reloading configuration changes should be limited to a
whitelist, risky changes like the Elasticsearch host should still require a
complete restart. Added to (3) based on
https://github.com/elastic/kibana/pull/45796#discussion_r331277153
5. Added Section 5, "Core should expose a status signal for Core services &
plugins" (from @joshdover)
6. Added the drawback that incorrect, but valid config would not block Kibana,
and might only be surfaced when the associted API/UI gets used (from
@azasypkin)
* Formatting: number ordered list instead of letter for github rendering
* Apply suggestions from code review
Co-Authored-By: Josh Dover <me@joshdover.com>
* Update rfcs/text/0007_lifecycle_unblocked.md
Co-Authored-By: Josh Dover <me@joshdover.com>
* Example of plugin exposing API dependent on internal async operation
* Clarify that context providers won't block kibana, just their handlers
* Update adoption strategy as per latest discussion
* Fix formatting
* General structure of Public w/ legacy brace + autocomplete 🤔
Refactor Resizer functionality (panel component)
Refactor Play Button
Refactor Auto-completion
Refactor Docs opener
Refactor Storage
* First refactor of kbn ace keyboard mode to TS+React
* clean up unused props
* console_menu.js -> console_menu.tsx
* Remove unused file from quarantine and added fixed ui ace keyboard mode react hook
* - Refactored history and storage to app-wide services
- Pre-emptive changes to tests
- sense-history -> HistoryList
- removed unused kbn top nav v2 component
* A lot of cleanup, re-introduced editor resize checker, re-introduced history viewer as TS+React. `history` still needs refactoring.
* First iteration of tap nav menu, with history toggle working
* Lots of fixes
Also moved over and integrated remaining three react components
* Moved a lot of files around again, tidied up NP set up
* Replace angular directive
* Remove used code
* Re-order imports and move all ace dependencies to same location
* Remove more unused code
* Revise quarantined setup mocks
* Don't suggest 'undefined' or other null-like values in autocomplete
* Clean up api_server folder
* Re-add missing style
* Updated karma spec mock
* Fix editors cutting of at bottom of screen
* Refactor console editors into single components
Refactor a lot of business logic to main.tsx container
Minor renaming of variables for better readability
* Updated use of contexts with better error message
Fixed broken render sync cycles (using useCallback)
Fixed Main container render cycle (added missing deps to useEffect)
Fixed default input and removed auto indent from being called on init for already formatted text
* Updated test mocks
* Update to be more in line with NP conventions https://github.com/elastic/kibana/blob/master/src/core/CONVENTIONS.md
* Update console history when making new requests
Fixed spacing between editor and console history
Moved registration of keyboard commands to TS
Fixed setup_mocks.js after renaming app to application
* Clean up git merge conflict artifact
* Use updated NP interfaces
* More typings fixed after updating local project dependencies
* Removing some dependencies on KUI and font awesome from legacy editor
* Fix clear history not re-rendering
Refactor prop name to be more descriptive
* Simplify split_panel and add tests
* Fix accessibility tabbing behaviour for ace editor
* Refactor ConsoleEditor into two separate components
Remove unused changeCursor code
Remove unused textArea ref
Use default lodash debounce (remove unnecessary arg)
* Major a11y fix when tabbing
Major fix for ace in IE11 and Edge browsers
* Update comment
* Add core-only bundle
* Add ApplicationService mounting
* Add LegacyCore{Setup,Start}
* Fix PR comments
* Add functional tests
* Fix PR comments
* Fix PR comments
* Remove other usages of rootRoute
* Use state field notation
* Add support for open in new tab
* Fix PR comments
* Fix pesky await from the dead
* Update docs
* Bump @types/history
The `setup` lifecycle function for core and plugins will be for
one-time setup and configuration logic that should be completed
in a finite amount of time rather than be available throughout
the runtime of the service.
The existing `start` lifecycle function will continue to serve
only the purpose of longer running code that intentionally only
executes when `setup` is finished.
