* Core task manager (#23555)
* Initial stab at core task manager logic
* Update task_manager readme
* Update task manager readme
* Add cancelable helper package, OSS, and general purpose, but
will be useful for writing cancelable x-pack tasks.
* Make the cancellable package promise-compliant
* Update task manager readme w/ reference to cancellable
* Change pool from lazy to eager, add support for sub-pools per task type.
* Move cancellable to task_manager, and typescriptify it.
* Working proof of concept for task manager. Still have lots to do:
clean up, tests, comments, validations, assertions, etc.
* Add pagination support to task manager fetch
* Move task manager to OSS
* Remove task manager reference from x-pack
* Make task_manager a valid core plugin
* Modify how task resource allocation works
* Remove the special case 'max' workers occupied value
* Remove x-pack package.json changes
* Make taskDefinitions a part of uiExports
* Make task docs saved-object compliant.
* Add kbnServer to the task context.
* Allow tasks to have a void / undefined return type
* revert x-pack change
* move cancellable to src/utils
* move to src/server
* use afterPluginsInit hook
* task_manager.ts rename
* add a wrapper with a setClient method
* Add tests for task runner
* Break task_pool into smaller, testable pieces
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix misidentified field name
* fix rest args warning
* flatten task_pool
* remove cancellable
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix rest args warning
* typescript fixes
* roll back setClient takes a callback
* createTaskRunner returns an object with run/cancel functions
* Test task store, tweak task poller tests
* Rename num_workers to max_workers
* Tweak task pool method names
* Fix cancellable test, and ts errors
* Rename doc to _doc
* Fix the doc type in the task manager index mappings
* Make task costs configurable in kibana.yml
* fix a tslint warning
* TaskManager.afterPluginsInit replaces circuitous stuff
* addMiddleware, implement beforeSave
* wip
* comment
* run context stuff
* pretty fix
* comments
* lint/pretty fixes
* safeties in case they don't define both beforeRun and beforeSave
* task runner test constructor opts fix
* Add task_manager integration tests
* FIx readme mistakes, fix schedule signature
* Fix integration tests
* Add blurb about middleware
* make a libs directory for the small things
* test for extractTaskDefinitions
* unit test for middleware
* Comment, refactor, and test task_manager
* Touch up comments
* Make task store throw error if given an unknown task type
* Fix sample task definition in readme
* Make task manager toggle on / off based on Elasticsearch connectivity
* Prevent task manager from crashing Kibana if
elasticsearch plugin is disabled
* Fix task manager signature
* Move the task manager into the task manager mixin,
fix tests. It's an uber integration object anyway, so
it seemed to make sense to keep it in one place.
* Fix task manager integration tests
* Update the task manager index creation to use auto_expand_replicas
* Fix task manager integration tests
* Fix task manager integration tests
* Fix the overzealous deletion of .kibana during tests
* Core task manager
* Allow hardcoded ID passed in
* comments
* don't deconstruct type and task for payload
* remove uiExport support
* move task_manager to x-pack/plugins
* expose as client on the plugin namespace
* fix tests
* move plugin api integration tests
* roll back readme sample task definition example
* fix sample plugin link
* server.taskManager
* sanitizeTaskDefinitions
* fix integration tests
* sanitize rename
* assert unintialized and check for duplicates in registerTaskDefinitions
* Remove unnecessary defaults from task_manager.test.ts
* Remove task manager from OSS
* Remove task manager from src
* Tidy up task_manager unit tests around elasticsearch status changes
* Integration test for specifying ID in scheduling
* Task_manager init -> preInit to allow other plugins to consume
taskManager in their init.
* Remove task manager integration tests from OSS
* spelling
* readme fix
* fix test code impacted by hapi upgrade
* Task Manager index creation changes (#24542)
* Uses putTemplate for task manager mappings and index settings.
* Removes create option.
* Opposite day, the test catches up to the code changes.
* Ignores index unavailable during searches.
* Improve taskRunner's processResult (#24880)
* Set task state to idle after run, and add failed state
* fix tests
* Test alerting demo (#25136)
* Ensures that task store init gets called before schedule.
* Removes unused option for debugging purposes.
* Fix unit tests because a second callCluster was made.
* Task manager starts sanely.
Does not preInit
Will not allow definitions after initialization
Creates store immediately.
Modifies store after all plugins have initialized
Adds static tasks that will be defined by plugins.
* Task manager should not allow operations before initialization.
* Attempts to fix runner tests.
* Fixes unit test contract with APIs.
* Removes unused type definitions.
* Removes unused package json.
* Removes unused import type.
* Removes unnecessary async applied to a function.
* Returns diferently if task store has already initialized.
* Explains how to add static task instances to task definitions. (#25880)
* Tasks cannot be scheduled until task manager is initialized.
* Adds task manager api integration tests to ci group4.
* Context of describe test must be the test framework object.
* Update src/es_archiver/lib/indices/kibana_index.js
regex update to actually disallow non-kibana system indices
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* verify fillPool exception passing
* readme update about max_workers/numWorkers
* change forEach to reduce
* use public interfaces in internal method
* replace getMaxAttempts with public readonly maxAttempts
* Update x-pack/plugins/task_manager/task_store.ts
`throw new Error` and initializattion spell fix
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* min = 1 for max_workers
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* timeOut => timeout
* min 1
* scope as an array
* no retries
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ConcreteTaskInstance is a TaskInstance
* remove guard per joi logic
* more return types for functions
* more comments around incremental back-off
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* throw error instead of return undefined
* poll_interval min 1000
* avoid handling err.stack directly
* break up processResult
* fix a few runtime issues
* only fetch idle tasks
* remove check for status idle
* always return a state, and when there is an error return the state we had at beforeRun
* check isStarted before attemptWork
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ts fix
* Make saved object client error while Kibana index is migrating
* Tidy up a bit, and refactor the way the `isMigrated` check is accessed
* Remove unused interface declaration
* Remove default migrator from saved objects repository constructor
* Fix repository migrator isComplete check
* Wrap callCluster and delay it until migrations have completed...
* Fix inaccurate comment
* Ensure migrations wait for elasticsearch to go green prior to running
* Add tests for callCluster being wrapped in the repository, fix
the es_archiver's call to migrate index.
* Fix es_archiver's kbnServer mock
fixes#20741
Rather than just generate random words with the default length of 2, generate a random 5 letter word for the type to filter by and 10 letter words for the other types, so there will never be more than 3 records with `type1`.
* [esArchiver/deleteIndex] wait and retry if snapshot in progress
* [esArchiver/deleteIndex] use recursion for retry
* [esArchiver/waitForSnapshot] invert status check
* [esArchiver] share delete-with-retry with create stream
* [esArchiver/stats] include index name in message
* [esArchiver/indexDelete] wait for snapshot completion up to three times
* [esArchiver] log status of snapshot during checks
* [utils/jsonParseStream] ensure only callback once
* [utils/streams] add util for piping many streams in series
* [esArchiver/load] combine record streams to fix skipExisting functionality
(cherry picked from commit 21d6dd2cba)
* [npm] upgrade elasticsearch/elasticsearch-browser packages
* [es/patchKibanaIndex] update tests to expect indices.getMapping
* [npm] use ^ version range for elasticsearch(-browser) packages
* [es/patchKibanaIndex/tests] expand assertion
* [precommitHook] move to dev script
* [eslint] fix lint error
* [dev/eslint] do simply boolean checks first
* [dev] use shared REPO_ROOT constant
* [dev/File] precompute relative/ext of path
* [dev/eslint] fix relative import
* [dev/eslint] fix typos
* [grunt] remove unused run:eslintStaged config
* [dev/run] only create log if we are going to use it
* [dev/run/isFailError] ensure return value is Boolean
* [dev/precommitHook] use less intermediate vars
* [functional_test_runner] replace functional testing tools with custom/pluggable solution
* [functional_test_runner] Convert unit tests to commonjs format
* [functional_test_runner] Fix dashboard test in wrong mode
* [functional_test_runner] Add dashboardLandingPage test subject
* [functional_test_runner] Get Visualize page object
* [functional_test_runner] Fix outdated references
* [functional_test_runner] Fix more outdated refs
* [functional_test_runner] Remove duplicate tests
* [functional_test_runner] Improve test readability
* [functional_test_runner] 😞 So many duplicate methods
* [functional_test_runner] Move mgmt `before` outside toplevel describe
* [functional_test_runner] Settings page obj missing methods
* [functional_test_runner] Add improvements from @gammon
* [functional_test_runner] Fix return statements in async funcs
* [functional_test_runner] Move before() to correct scope
* [functional_test_runner] Add after() hooks to remove index patterns
* [functional_test_runner] Attempt to fix vertical bar chart tests
* [functional_test_runner] Clean up
* [functional_test_runner] Reinstate unit tests
* [functional_test_runner] Set default loglevel back to info
* [functional_test_runner] Replace `context`s with `describe`s
* [functional_test_runner] Better error handling
* [functional_test_runner] Add in new Tile Map tests
* Incorporate changes from master
* [functional_test_runner] validate that every test file has a single top-level suite
* Update contributing doc with link to full doc
* [docs] Spelling and grammar fixes
* docs: writing and running functional tests
* [docs] Move plugin doc to plugin area
* [docs] Housekeeping. Doc in wrong place
* [docs] Remove dup doc file
* [grunt] Only run mocha_setup when running tests, not every grunt task
* As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces.
Differences:
- Streaming implementation allows for much larger archives
- CLI for creating and using archives
- Configurable archive location
- Stores the data in gzipped files (better for source control, searching, large archives)
- Automatically identifies and upgrades Kibana config documents
Methods:
- `#load(name)`: import an archive
- `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index
- `#unload(name)`: delete the indexes stored in an archive
CLI operations:
- `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive
- `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive
* [functional_tests/common/nagivate] check for statusPage
* [es_archiver] move bins into new scripts dir
* [functional_tests/apps/context] use esArchiver
* [esArchiver] general improvements after showing to a few folks
- remove auto-upgrading config doc logic (until we have better access to kibana version info)
- export unload command
- remove preemptive checks in favor of reacting to errors
- use type "doc" vs "hit" for doc records (consistency)
- wrote a bunch of pending tests to think though and plan
* [esArchiver] make log a stream that writes to itself
* [esArchiver] fill in stats and archive format tests
* [esArchiver] splitup action logic
* [esArchiver/cli] fix cli --help output and comment
* [esArchiver] remove type-based param coercion
* [esArchiver/log] use strings for log levels
* [esArchvier] remove unused var
* [esArchiver/indexDocRecordsStream] add tests
* [esArchive] fill in remaining tests
* [esArchiver] fix dem tests
* [eslint] remove unused vars
* [esArchiver/loadIfNeeded] fix call to load()
* [esArchiver] remove loadDumpData helpers
