## [SIEM] Overview Page "1.5"
A redesigned SIEM Overview page that includes `Recent timelines`, a `Security news` feed, visualizations, and rolled-up event counts
### Overview enhancements
- Added the global Search bar and Date picker to the Overview page
- New `Recent timelines` widget affords quick access to favorite and recently modified timelines
- New `Security news` widget
- New Kibana advanced settings (toggle switch) for enabling or disabling the news widget and configuring the news URL
- New `Events count by dataset` widget
- Updated the `Host Events` and `Network Events` widgets to integrate with the Search bar and date picker input
- Enhanced the `Host Events` and `Network Events` widgets to use an accordion paradigm that summarizes stats by source (e.g. `Auditbeat`, `Endgame`)
- Enhanced the `Host Events` and `Network Events` widgets to visualize relative percentages of events collected as progress bars
- New `Alerts count by category` widget
- New `Signals count by MITRE ATT&CK™ category` widget
- New `View events`, `View alerts`, and `View signals` navigation buttons for their respective visualizations
### FTUE enhancements
- FTUE "no data" view design refresh
- When the FTUE "no data" page is displayed, hide all global navigation links (i.e. `Hosts`, `Network`, `Detection engine`), such that only `Overview` appears in the global nav
- App Help popover design refresh
- Removed the `Beta` badge and `Security Information & Event Management with the Elastic Stack` from the Overview header
- Tested in Chrome `79.0.3945.117`, Firefox `72.0.1`, and Safari `13.0.4`
## Known issues
- The `siem:newsFeedUrl` advanced setting is defaulted to `https://feeds.elastic.co/kibana`
- The `Signals count by MITRE ATT&CK™ category` visualization does not display all categories
- The `Signals count by MITRE ATT&CK™ category` visualization may require a different index pattern
- `EuiButtonGroup` throwing a `Can't perform a React state update on an unmounted component` warning when switching from the Overview tab
https://github.com/elastic/siem-team/issues/484
* [DOCS] Moves index pattern doc to Discover
* [DOCS] Improves intro to index patterns doc
* [DOCS] Edits index patterns doc
* [DOCS] Incorporates comments into index patterns doc
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Asciidoctor sees `,` as the edge of parameters and chokes on one of our
deprecation warnings, rendering funny looking garbage. This wraps the
whole parameter in `"` which makes it look good. It *does* add `"`s
around the result in AsciiDoc, but we plan to migrate from AsciiDoc
"real soon now".
* [DOCS] Updates Index Management doc to include index templates
* [DOCS] Added example of creating a template
* [DOCS] Incorporates review comments
* [DOCS] Fixes typo
* Added an inclusion of, "This setting is off by default..."
* Update docs/management/advanced-options.asciidoc
Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com>
Discover currently executes a search as soon as it loads. For some users this is useful. But for others it may return worthless results at the expense of extra load on their ES cluster and increased page load times, making it harder to get to the data they actually want. This PR adds an advanced setting allowing users to turn off the "search on page load" functionality in Discover.
* [DOCS] Updates index patterns doc
* [DOCS] Incorporates review comments in index patterns doc
* [DOCS] More review comments on index patterns doc
* [DOCS] Fixed section on managing index patterns
* Adding read-only mode section to Discover
* No more "mode" or "badge" terminology
* Putting newline back
* Adding "Saving" section for visualize
* Adding dashboard read only access section
* Timelion gets read-only documentation
* Adding read only access section to index patterns and advanced settings
* Adding graph read only access section
* Allow select settings to specify labels for their values
* Rename kuery setting to KQL
* Change docs for KQL setting
* Add warnings for unused options
* Address review
* Remove chinese translation for modified string
* Fix translations again (... should have pulled first)
* Remove old chinese translation
* [dashboard+gis] remove dark mode options
* [reporting/extract] restore fixtures
* remove mentions of old `.theme-dark` class
* import panel styles from panel/_index.scss
* [DOCS] Adds documentation for index lifecycle policies
* [DOCS] Updated image for policy options to show all menu items
* Update create-policy.asciidoc
* [DOCS] Incorporated review comments on hot and warm phase
* [DOCS] Additional changes to warm phase
* [DOCS] Removed the word open in the warm phase
Allows Kibana users to configure the max_concurrent_shard_requests param used by Kibana when sending _msearch requests. Exposes the config as an advanced setting. By default we won't send the param at all, relying on the ES default instead.
* Beginning to work on the role management APIs. Added docs for GET
* Adding PUT docs
* Adding PUT details
* Adding delete docs
* Fixing linking
* Adding Kibana privileges section
* Fixing dashboard only mode docs
* Fixing a few more references to managing roles
* Beginning to work on authorization docs, might be moving some to
stack-docs
* Collapsing authorization description in the kibana privileges page
* Adding audit logging section
* Revising the language on the Kibana role management section
* Splitting back out the auth/privileges and adding legacy fallback
details
* Revising language around impact of disabling security
* Changing Kibana to {kib} and Elasticsearch to {es}
* Beginning to work on developer centric docs
* Fixing some formatting, adding some diagrams
* Adding note about the role management APIs
* Adding overview, fixing small syntax issues
* Fixing chunk name for transitioning to application privileges
* Adjusting tone for the authorization introduction
* Changing the tone and structure of the RBAC docs
* Deleting blog stuff after refactoring
* Addressing first round of peer review comments
* Fixing endpoints links
* Peer review suggested edits
* Addressing other PR feedback
* enhance index pattern delete documenation
* add line about breaking saved objects that still reference index pattern
* indices spelling
* better wording from gchaps
Makes our language updates more visible to users and removes mentions of Kuery as a separate language. Users still get the old lucene experience by default, but have the option to opt-in to "experimental query features" directly in the query bar. Goal is to get more feedback by making these new features more prominent and less of a jump from lucene.
* First stab at refactoring typeahead
* Don't double submit on enter
* Add item templating
* Introduce simple kuery language
* Rename to kql and add modules
* Update KQL syntax
* Automatically insert matching pairs
* Don't match quotes after alphanumeric chars
* Get field and value suggestions
* Remove accidental changes
* Remove unnecessary test
* Don't submit on enter
* Fix typeahead
* Suggest matching recent searches
* Suggest operators
* Suggest conjunctions
* Use template, separate suggestions into separate modules
* Whoops, add this module back
* Add clarifying comment
* Fix history log key
* Don't update suggestions on every key press
* Fix key handling
* Update terminology to be clearer
* Fix typo
* Simplify building of nodes
* Don't always hide on submit
* Check items exists
* Add icon directive which wraps EuiIcon.
* kql design start
* remove comment bits
* Simplify select next/prev and reset selected on hide/backspace
* Add test
* Put persistedLog on scope so it can be tested
* Fix typos
* Build up AST for sublist by returning functions that take a field name
* Remove single quoted strings and add double quote to special characters
* Build nodes with arg nodes instead of args themselves
* Add support for exact phrase search for quoted values
* Update typeahead items when language changes
* Finish that work I didn't do in the last commit
* This commit makes Bargs very happy cuz it does a lot
* Add wildcard field support to range query
* Remove range support for wildcard values
* Remove KQL as a separate language
Updates kuery to use KQL's grammar. This will lead to a smoother
transition for both us and Kuery users. We mainly added KQL as a
separate language so that we could notify Kuery users that the syntax
had changed. I realized we could do the same by trying to parse their
query strings with the old grammar if the new grammar fails, and if the
old grammar parses successfully we can display an error message with a
link to the docs describing the syntax changes. Since Kuery now uses the
more simple KQL syntax, I've also re-enabled the filter bar when Kuery
is selected.
* Fix typeahead behavior
* Update conjunctions
* Use scope apply
* Suggest conjunctions after ranges
* Support strings in wildcard node constructor and add tests for wildcard node
* test updates
* Removing unused serializeStyle and toKueryExpression, updating tests
* Fix functional test
* Fix typo
* Show fields that match in any part of the name
* Alter order of operators
* Preserve focus after selecting by click
* Ports tests for `fromKueryExpression` to `fromKqlExpression`
* More KQL syntax tests
* Suggest conjunctions after ranges
* Fix suggestions inside parens
* fromLiteralExpression tests
* remove serializeStyle arguments which no longer exist in the function definition
* tests for getFields
* Case insensitive search for field names
* update tests for is.js
* add wildcard fieldname test for range.js
* Fix removed div
* Fix line spacing for autocomplete suggestions
* Fix conjunction suggestions with escaped preceding literals
* Escape special characters in fields and values
* Don't suggest the value that's already selected
* Update icons
* Sort prefix first
* Simplify cursor detection and suggest booleans
* Use quotes for suggested values
* get rid of references to KQL
* Don't show errors from parsing
* That didn't even exist
* Use config to determine if values are suggested
* Update suggestions on home/end
* remove hack
* Update reference to kql
* Suggestions for quoted values
* Clean up grammar
* Better support for cursor inside spaces
* Create grunt task to generate parsers from peg files
* Simplify wildcard handling
* Don't filter out the exact fields/values
* Update parsing modules
* Fix peg task
* Make operator syntax more visible
* Update OR verbiage
* Simplify and improve match pairs
* Revert "Simplify wildcard handling"
This reverts commit 915861beab.
* Support escaped backslashes inside quoted strings
* Support escaped keywords
* Remove lodash dependency cuz w33ble
* Escape user input and fix conjunction description
* Clear suggestions after submitting
* Fix insertion of suggestion to account for selection
* Remove unnecessary?
* Remove extraneous file
* Better name for method
* Move functions out of event handler
* Don't wrap result in promise
* Don't show kuery suggestions for lucene
* some cleanup and polish for kql autocomplete
* Omit description completely for values
* Don't suggest and/or for quoted strings that end in spaces
* Submit recent search suggestions on select
* Scroll selected suggestion into view
* Better handling of key events and hiding typeahead
* Update suggestions to work in other apps with multiple index patterns
* Only update suggestions if not submitting
* Hide suggestions on focus
* Simplify wildcard (again)
* Fix console error
* Remove references to kql
* Fix match pairs so that suggestions occur with cursor in correct place
* Memoize value suggestions
* Debounce model updates
* Add tests for suggestion providers
* Add setting and docs
* Add custom error, helper for detecting leading wildcards, and check for
leading wildcards in the Value rule of the grammar.
* Better handling of suggestion clicks
* Dedup suggestions
* Sort keywords first
* Fix value suggestions memoizing
* Check if query exists
* Reduce size of dialog and fix ranges
* Create grunt task to generate parsers from peg files
* Lazy load typeahead items
* Fix wildcard tests
* Fix value suggestion tests
* Fix typeahead tests
* Fix value suggestion memoize resolver
* Leave comment
* Add a ttl for the value suggestion resolver
* Move grunt config to config/
* Bargs can suck it
* Fix more tests that bargs broke (and one I did too)
* Fix tests
* Check sort:options for Discover default sort order
I went back and forth on a lot of different solutions for this.
Initially I thought it would make sense to just allow users to set a
default saved search in Discover. There were some problems with that
approach though. It would change the default workflow in Discover.
Instead of starting with an unsaved search, users would be editing a
saved search by default. I could see this leading to a lot of
unintentional changes to the default. The settings from the default
saved search also wouldn't carry over to new searches, which I think
would be desirable most of the time.
I also considered adding a new advanced setting for specifying a default
sort field/direction. This kind of setting would make more sense at the
index pattern level though. One field may not be valid across all index
patterns.
So I ended up going with the simplest solution. It solves the issue
identified by the author of the linked issue and nothing more. If a sort
order is specified in the existing sort:options advanced setting, we'll
use that direction when sorting on the index pattern's timestamp field
by default.
Fixes https://github.com/elastic/kibana/issues/5164
* Create a new advanced setting instead of re-using sort:options
* Just pass a default order
* Re-enable filter editor suggestions
* Use search instead of include
* Escape query
* Show spinner
* Use include rather than search
* Add additional regex and explanation for parameters
* Add suggestions API test
* Make sure test actually runs
* Use send instead of query
* Fix suggestions API test
* Change use of all_fields in highlight_query to default_field
* Remove highlight query and option
* Fix tests
* Remove unused setting
* Remove lingering references to all_fields
Using fields with docvalues (like `_doc`) for tie-breaking yields
significantly better performance than using `_uid`, which lacks
docvalues at the moment. The downside is that sorting by `_doc` by
default is not stable under all conditions, but better than no
tie-breaking at all.
The new setting `context:tieBreakingFields` enables the user to
customize the list of fields Kibana attempts to use for tie-breaking.
The first field from that list, that is sortable in the current index
pattern, will be used. It defaults to `_doc`, which should change to
`_seq_no` from version 6.0 on.
In addition to just showing a notification, errors that occur while
loading documents from the database will be stored as part of the
`loadingStatus` along with a reason code (if known). This is used to
display more nuanced and helpful error messages to the user.
The first such error message indicates a missing or invalid tiebreaker
field required for sorting the context.
- Users can now create choropleth maps. This requires configuring an inner join between the results of a term-aggregation and a reference vector layer. This vector layer needs to be in the GeoJson format. By default, Kibana uses vector layers serverd by a data service hosted by Elastic. Users can also bring in their own layers by adding configuration entries in the kibana.yml. These need to point to a CORS-enabled data service that accepts requests from the Kibana application.
- For clarity, the tilemap is renamed to Coordinate Map.
- A new manifest is published by Elastic. this includes metadata for the available tilemap services, as well as metadata for the available vector data layers.
The discover:aggs:terms:size config value was never added to the default values, so when the trash-button is clicked it removes the config completely and causes discover to send an empty size param to visualize.
* Fixes#11231 - Adds check for max_bucket size for Time Series Visual Builder
* Added Math.floor() to bucket calculation; changed error message; changed option description
This adds a link to the detail view of discover rows to switch to a view of the documents immediately before and after the selected document. Since that view uses the timestamp field of the index pattern, it is only available for time-based indices.
See #9198 for detailed screenshots.
* Add all_fields to highlight query to improve highlighting
* Refactor highlighting and move out of _flatten
* Make changes as per @bargs' requests
* Add documentation about highlightAll setting
* Fix docs typo
* Remove unused function
* Remove unused code
* Update managing-fields.asciidoc
* Remove references to Groovy
* Add information about Painless
* Link to blog for additional examples
* Update managing-fields.asciidoc
Make editorial changes per @debadair's feedback.
This overhaul of the docs structure puts Kibana's documentation more
inline with the structure that is used in Elasticsearch. This will help
us better organize the docs going forward as more docs are added.
This also includes a few necessary content changes for 5.0.
