No description
fbe48221ae
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle * Add note * Add version metadata to signals documents * WIP: Starting over from the ground up * Removes obsolete endpoints/functions * Adds endpoint for checking the migration status of signals indices * Adds helper functions to represent the logical pieces of answering that question * Fleshing out upgrade of signals * triggers reindex for each index * starts implementing followup endpoint to "finalize" after reindexing is finished * Fleshing out more of the upgrade path Still moving logic around a bunch. * Pad the version number of our destination migration index Instead of e.g. `.siem-signals-default-000001-r5`, this will generate `.siem-signals-default-000001-r000005`. This shouldn't matter much, but it may make it easier for users at a glance to see the story of each index. * Fleshing out more upgrade finalization * Verifies that task matches the specified parameters * Verifies that document counts are the same * updates aliases * finalization endpoint requires both source/dest indexes since we can't determine that from the task itself. * Ensure that new signals are generated with an appropriate schema_version * Apply migration cleanup policy to obsolete signals indexes After upgrading a particular signals index, we're left with both the old and new copies of the index. While the former is unlinked, it's still taking up disk space; this ensures that it will eventually be deleted, but gives users enough time to recover data if necessary. This also ensures that, as with the normal signals ILM policy, it is present during our normal sanity checks. * Move more logic into component functions * Fix type errors * Refactor to make things a little more organized * Moves migration-related routes under signals/ to match their routing * Generalizes migration-agnostic helpers, moves them to appropriate folders (namely index/) * Inlined getMigrationStatusInRange, a hyper-specific function with limited utility elsewhere * Add some JSDoc comments around our new functions This is as much to get my thoughts in order as it is for posterity. Next: tests! * Adds integration tests around migration status route * Adds io-ts schema for route params * Adds es_archiver data to represent an outdated signals index * Adds API integration tests for our signals upgrade endpoint * Adds io-ts schema for route params * Adds second signals index archive, updates docs * Adds test helper to wait for a given index to have documents * Adds test helper to retrieve the relevant index name from a call to esArchive.load * WIP: Fleshing out finalization tests * Consolidate terminalogy around a migration We're no longer making a distinction between an upgrade vs. an update vs. a migration vs. a reindex: a migration is the concept that encompasses this work. Both an index and individual documents can require a migration, but both follow the same code path to migrate. * Implement encoding of migration details This will be a slightly better API: rather than having to pass all three fields to finalize the migration, API users can instead send the token. * Better transformation of errors thrown from the elasticsearch client These often contain detailed information that we were previously dropping. This will give better info on the migration finalization endpoint, but should give more information across all detection_engine endpoints in the case of an es client error. * Finishing integration tests around finalization endpoint This lead to a few changes in the responses from our different endpoints; mainly, we pass both the migration token AND its constituent parts to aid in debugging. * Test an error case due to a reindexing failure This would be really hard to reproduce with an integration test since we'd need to generate a specific reindex failure. Much easier to stub some ES calls to exercise that code in a unit test. * Remove unnecessary version info from signals documents We now record a single document-level version field. This represents the version of the document's _source, which is generated by our rule execution. When either a mapping _or_ a transformation is added, this version will be bumped such that new signals will contain the newest version, while the index itself may still contain the old mappings. The transformation pipeline will use the signal version to short-circuit unnecessary transformations. * Migrate an index relative to the ACTUAL template version This handles the case where a user is attempting to migrate, but has not yet rolled over to the newest template. Running rules may insert "new" signals into an "old" index, but from the perspective of the app no migration is necessary in that case. If/when they roll over, the aforementioned index (and possibly older ones) will be qualified as outdated, and can be migrated. * Enrich our migration_status endpoint with an is_outdated qualification This can be determined programatically, but for users manually interpreting this response, the qualification will help. * Update migration scripts * More uniform version checking * getIndexVersion always returns a number * version comparisons use isOutdated * Fix signal generation unit tests We now generate a version field to indicate the version under which the signal was created/migrated. * Support reindex options to be sent to create_migration endpoint Rather than having to perform a manual reindex, this should give API users some control over the performance of their automated migration. * Fix signal generation integration tests These were failing on our new signal field. * Add unit tests for getMigrationStatus * Add a basic test for getSignalsIndicesInRange Since this is ultimately just an aggregation query there's not much else to test. * Add unit test for the naming of our destination migration index * Handle write indices in our migration logic * Treat write indices as any other index in migration status endpoint * Migration API rejects requests containing write indices * Migration API rejects requests containing unknown/non-signals indices * Add original hot phase to migration cleanup policy Without this phase, ILM gets confused as it tries to move to the delete phase and fails. * Update old comment The referenced field has changed. * Delete task document as part of finalization * Accurately report recoverable errors on create_signals_migration route If we have a recoverable error: e.g. the destination index already exists, or a specified index is a write index, we now report those errors as part of the normal 200 response as these do not preclude other specified indices from being migrated. However, if non-signals indices are specified, we do continue to reject the entire request, as that's indicative of misuse of the endpoint. |
||
|---|---|---|
| .ci | ||
| .github | ||
| .teamcity | ||
| common/graphql | ||
| config | ||
| data | ||
| docs | ||
| examples | ||
| licenses | ||
| packages | ||
| plugins | ||
| rfcs | ||
| scripts | ||
| src | ||
| tasks | ||
| test | ||
| typings | ||
| utilities | ||
| vars | ||
| x-pack | ||
| .backportrc.json | ||
| .browserslistrc | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc.js | ||
| .fossa.yml | ||
| .gitattributes | ||
| .gitignore | ||
| .i18nrc.json | ||
| .node-version | ||
| .nvmrc | ||
| .prettierrc | ||
| .sass-lint.yml | ||
| .telemetryrc.json | ||
| .yarnrc | ||
| api-documenter.json | ||
| CONTRIBUTING.md | ||
| FAQ.md | ||
| github_checks_reporter.json | ||
| Gruntfile.js | ||
| Jenkinsfile | ||
| jest.config.integration.js | ||
| jest.config.js | ||
| jest.config.oss.js | ||
| kibana.d.ts | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| package.json | ||
| preinstall_check.js | ||
| README.md | ||
| renovate.json5 | ||
| SECURITY.md | ||
| STYLEGUIDE.md | ||
| tsconfig.base.json | ||
| tsconfig.browser.json | ||
| tsconfig.json | ||
| tsconfig.refs.json | ||
| tsconfig.types.json | ||
| TYPESCRIPT.md | ||
| yarn.lock | ||
Kibana
Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.
- Getting Started
- Documentation
- Version Compatibility with Elasticsearch
- Questions? Problems? Suggestions?
Getting Started
If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.
If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.
Using a Kibana Release
If you want to use a Kibana release in production, give it a test run, or just play around:
- Download the latest version on the Kibana Download Page.
- Learn more about Kibana's features and capabilities on the Kibana Product Page.
- We also offer a hosted version of Kibana on our Cloud Service.
Building and Running Kibana, and/or Contributing Code
You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:
- CONTRIBUTING.md will help you get Kibana up and running.
- If you would like to contribute code, please follow our STYLEGUIDE.md.
- For all other questions, check out the FAQ.md and wiki.
Documentation
Visit Elastic.co for the full Kibana documentation.
For information about building the documentation, see the README in elastic/docs.
Version Compatibility with Elasticsearch
Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.
Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.
| Situation | Example Kibana version | Example ES version | Outcome |
|---|---|---|---|
| Versions are the same. | 5.1.2 | 5.1.2 | 💚 OK |
| ES patch number is newer. | 5.1.2 | 5.1.5 | ⚠️ Logged warning |
| ES minor number is newer. | 5.1.2 | 5.5.0 | ⚠️ Logged warning |
| ES major number is newer. | 5.1.2 | 6.0.0 | 🚫 Fatal error |
| ES patch number is older. | 5.1.2 | 5.1.0 | ⚠️ Logged warning |
| ES minor number is older. | 5.1.2 | 5.0.0 | 🚫 Fatal error |
| ES major number is older. | 5.1.2 | 4.0.0 | 🚫 Fatal error |
Questions? Problems? Suggestions?
- If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
- Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.