* Adds new SO type for persisting our signals migrations * WIP: Migration status SO client Trying to forge a patterrn using io-ts to validate at runtime. I think I've got it working but I want to refactor the pipeline out into a reusable function(s). * Implements our SavedObjects service for signals migrations * Defines a simple client that delegates to the base SO client with our SO type * Defines a service that consumes the simpler client, adding validations and data transforms on top. * Refactoring migration code to work with saved objects As opposed to the previous ephemeral, encoded tokens, we now retrieve migration info from saved objects. At the API level, this means that both the create and finalize endpoints receive a list of concrete indices. No more passing around tokens. As both endpoints are idempotent, users can hammer them as much as they want with the same lists of indices. Redundant creates and finalizes will be met with inline 400 messages, and as one continues to poll the finalize endpoint they should see more and more indices respond with "completed: true" * Fixing integration tests first, and anything upstream breaking them * Clean up API integration tests * standardize assignment of responses (with types) * deletes migration SOs as test cleanup * Split API tests into separate files This was getting big and unwieldy; this splits these into one file per endpoint. * Refactor: split existing migration service functionality into atomic functions This will allow us to repurpose the service to compose more functionality and be more specifically useful, while keeping the component logic separate. * WIP: moving logic into migrationService.create * Splitting get_migration_status into component functions getMigrationStatus was really two separate aggregations, so I split them out and we recompose them in the necessary routes. * Move finalization logic into function * migrationService exposes this as .finalize() * adds an error field to our migration SO * We currently only have one error that we persist there, but it would be very time-consuming to track down that information were it not there. * Adds function for migration "deletion" logic * migrationService leverages this function * adds new boolean to our savedObject * deletes unused function (deleteMigrationSavedObject) * Adds route for soft-deletion of migrations * Updating tests related to migration status * Adding/updating mocks/unit tests necessary to satisfy the things I need to test * I mainly wanted to test that the the status endpoint filtered out the deleted migrations; this was accomplished with a unit test after fleshing out some mocks/sample data. * Move old migration service tests to the relevant function tests This logic was previously moved out into component functions; this moves the tests accordingly. * Add some unit tests around our reindex call * Fix create migration route tests Mocks out our migration functions, rather than stubbing ES calls directly. * Updates finalize route unit tests Addresses functionality that hasn't been moved to finalizeMigration() * Unit tests our finalization logic Fixes a bug where we weren't accounting for soft-deleted migrations. ALso updates our test migration SO to have a status of 'pending' as that's a more useful default. * Fixes finalization integration tests These were failing due: * a change in the migration status API response * a bug I introduced in the finalize route * Adds tests for our migration deletion endpoint * unit tests * API integration tests * Caught/fixed bug with deleting a successful migration * Fixes types Removes unused code. * Prevent race condition due to template rollover during migration If a user has an out of date index (v1) relative to the template (v2), but the template itself is out of date (newest is v3), then it's possible that the template is rolled over to v3 after the v1-v2 migration has been created but before the new index has been created. In such a case, the new index would receive the v3 mappings but would incorrectl be marked as v2. This shouldn't necessarily be an issue, but it's an unnecessary state that can easily be prevented with the guard introduced here. * Add real usernames to migration savedObjects In addition to the SOs themselves giving us observability into what migration actions were performed, this gives us the additional info of _who_ performed the action. * Index minimal migration SO fields needed for current functionality * Add additional migration info to status endpoint This will allow users to finalize a migration if they've lost the response to their POST call. * Finalize endpoint receives an array of migration IDs, not indices This disambiguates _which_ migrations we were finalizing if you passed an index (which was previously: the most recent migration). * Fix type errors in tests after we threaded through username * Update responsibilities of migration finalize/delete endpoints Discussions with @marshallmain lead to the following refactor: * finalize does not delete tasks * finalize only applies cleanup policy to a failed migration * delete takes an array of migration ids (like finalize) * delete hard-deletes the SavedObject of a completed (failed or successful) migration This gives a bit more flexibility with the endpoints, as well as disambiguates the semantics: it just deletes migrations! * Fix tests that were broken during refactoring * Fix type errors I removed some logic here but forgot the imports :( * Move outdated integration test In the case of a successful migration, application of the cleanup policy is done by the deletion endpoint. In the interest of data preservation, we do not delete a sourceIndex unless it is explicitly deleted. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> |
||
---|---|---|
.. | ||
common | ||
public | ||
server | ||
jest.config.js | ||
kibana.json | ||
README.md |
README.md for developers working on the backend lists on how to get started using the CURL scripts in the scripts folder.
The scripts rely on CURL and jq:
Install curl and jq (mac instructions)
brew update
brew install curl
brew install jq
Open $HOME/.zshrc
or ${HOME}.bashrc
depending on your SHELL output from echo $SHELL
and add these environment variables:
export ELASTICSEARCH_USERNAME=${user}
export ELASTICSEARCH_PASSWORD=${password}
export ELASTICSEARCH_URL=https://${ip}:9200
export KIBANA_URL=http://localhost:5601
export TASK_MANAGER_INDEX=.kibana-task-manager-${your user id}
export KIBANA_INDEX=.kibana-${your user id}
source $HOME/.zshrc
or ${HOME}.bashrc
to ensure variables are set:
source ~/.zshrc
Open your kibana.dev.yml
file and add these lines with your name:
xpack.lists.listIndex: '.lists-your-name'
xpack.lists.listItemIndex: '.items-your-name'
Restart Kibana and ensure that you are using --no-base-path
as changing the base path is a feature but will
get in the way of the CURL scripts written as is.
Go to the scripts folder cd kibana/x-pack/plugins/lists/server/scripts
and run:
./hard_reset.sh
./post_list.sh
which will:
- Delete any existing lists you have
- Delete any existing list items you have
- Delete any existing exception lists you have
- Delete any existing exception list items you have
- Delete any existing mapping, policies, and templates, you might have previously had.
- Add the latest list and list item index and its mappings using your settings from
kibana.dev.yml
environment variable ofxpack.lists.listIndex
andxpack.lists.listItemIndex
. - Posts the sample list from
./lists/new/ip_list.json
Now you can run
./post_list.sh
You should see the new list created like so:
{
"id": "ip_list",
"created_at": "2020-05-28T19:15:22.344Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"name": "Simple list with an ip",
"tie_breaker_id": "c57efbc4-4977-4a32-995f-cfd296bed521",
"type": "ip",
"updated_at": "2020-05-28T19:15:22.344Z",
"updated_by": "yo"
}
You can add a list item like so:
./post_list_item.sh
You should see the new list item created and attached to the above list like so:
{
"id": "hand_inserted_item_id",
"type": "ip",
"value": "127.0.0.1",
"created_at": "2020-05-28T19:15:49.790Z",
"created_by": "yo",
"list_id": "ip_list",
"tie_breaker_id": "a881bf2e-1e17-4592-bba8-d567cb07d234",
"updated_at": "2020-05-28T19:15:49.790Z",
"updated_by": "yo"
}
If you want to post an exception list it would be like so:
./post_exception_list.sh
You should see the new exception list created like so:
{
"created_at": "2020-05-28T19:16:31.052Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "bcb94680-a117-11ea-ad9d-c71f4820e65b",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "86e08c8c-c970-4b08-a6e2-cdba7bb4e023",
"type": "endpoint",
"updated_at": "2020-05-28T19:16:31.080Z",
"updated_by": "yo"
}
And you can attach exception list items like so:
{
"comments": [],
"created_at": "2020-05-28T19:17:21.099Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "included",
"type": "match",
"value": "Elastic, N.V."
},
{
"field": "event.category",
"operator": "included",
"type": "match_any",
"value": [
"process",
"malware"
]
}
],
"id": "da8d3b30-a117-11ea-ad9d-c71f4820e65b",
"item_id": "endpoint_list_item",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"os_types": ["linux"],
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "21f84703-9476-4af8-a212-aad31e18dcb9",
"type": "simple",
"updated_at": "2020-05-28T19:17:21.123Z",
"updated_by": "yo"
}
You can then do find for each one like so:
./find_lists.sh
{
"cursor": "WzIwLFsiYzU3ZWZiYzQtNDk3Ny00YTMyLTk5NWYtY2ZkMjk2YmVkNTIxIl1d",
"data": [
{
"id": "ip_list",
"created_at": "2020-05-28T19:15:22.344Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"name": "Simple list with an ip",
"tie_breaker_id": "c57efbc4-4977-4a32-995f-cfd296bed521",
"type": "ip",
"updated_at": "2020-05-28T19:15:22.344Z",
"updated_by": "yo"
}
],
"page": 1,
"per_page": 20,
"total": 1
}
or for finding exception lists:
./find_exception_lists.sh
{
"data": [
{
"created_at": "2020-05-28T19:16:31.052Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "bcb94680-a117-11ea-ad9d-c71f4820e65b",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"os_types": ["linux"],
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "86e08c8c-c970-4b08-a6e2-cdba7bb4e023",
"type": "endpoint",
"updated_at": "2020-05-28T19:16:31.080Z",
"updated_by": "yo"
}
],
"page": 1,
"per_page": 20,
"total": 1
}
See the full scripts folder for all the capabilities.