* [data views] clarify field subtype typescript types (#112499)
* separate out multi and nested subTypes
* separate out multi and nested subTypes
* add undefined checks
* remove expect error statements
* use helper functions in es-query
* simplify changes with helper functions
* checking existence instead of getting value x2
* simplify types and revert discover changes
* update discover sidebar with helper methods
* try helpers with group_fields file
* try different helper with group_fields file
* revert group field changes, try nested field helpers
* revert nested field changes, try field_name.tsx helpers
* fix maps jest test
* use helpers in discover instead of setting types
* fix field_name.tsx
* Update index_pattern_util.test.ts
* lint fix
* fix common exports
* reduce data_views plugin bundle size
* reduce data_views plugin bundle size
* remove discover reliance on es-query package
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
# packages/kbn-es-query/src/es_query/handle_nested_filter.ts
* fix test
* first pass at renaming exports
* type fixes
* fix jest test
* look for correct error type
* remove transitional error
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
# src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts
* Update dependency @elastic/elasticsearch to ^8.0.0-canary.17 (#107536)
* Update dependency @elastic/elasticsearch to ^8.0.0-canary.15
* update tests for new error message building mechanism
* fix integration tests
* fix functional test
* mute new type errors
* fix new type errors
* bump es client to canaary.16
* fix integration test
* fix type errors in infra plugin
* mute type error in ml plugin
* fix type errors in monitoring plugin
* fix and mute errors in security solution plugin
* bump version to canary.18
* remove an unnecessary change
Co-authored-by: Renovate Bot <bot@renovateapp.com>
Co-authored-by: restrry <restrry@gmail.com>
* bump to canry3
* fix another conflict
* fix another conflict in test file
* fix test. use assetion against stable parts
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Renovate Bot <bot@renovateapp.com>
* es-query types
* jest and lint
* cc
* options
* type
* types for kuery FUNCTIONS
* doc
* sec fixes
* typey type
* test typescript
* test
* fixes
* test
* cr
* cleanup a bit more
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Liza Katz <lizka.k@gmail.com>
* Add ability to generate KQL filters in the "must" clause
Also defaults search source to generate filters in the must clause if _score is one of the sort fields
* Update docs
* Review feedback
* Fix tests
* update tests
* Fix merge error
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [build_ts_refs] improve caches, allow building a subset of projects
* cleanup project def script and update refs in type check script
* rename browser_bazel config to avoid kebab-case
* remove execInProjects() helper
* list references for tsconfig.types.json for api-extractor workload
* disable composite features of tsconfig.types.json for api-extractor
* set declaration: true to avoid weird debug error
* fix jest tests
Co-authored-by: spalger <spalger@users.noreply.github.com>
# Conflicts:
# .gitignore
# examples/hello_world/tsconfig.json
# tsconfig.json
# tsconfig.refs.json
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Move @kbn/es-query into data plugin - es-query folder (#50182)
* Move @kbn/es-query into data plugin - es-query
* fix eslint issues
* Fix PR comments
* fix CI
* fix Ci
* remove extra ts-ignore
* fix imports
* fix imports
* Test importing from data/public and casting to ES Field Types.
* Test importing from data/public and casting to ES Field Types.
# Conflicts:
# src/legacy/core_plugins/data/public/index_patterns/index_patterns/index_pattern.tsx
# x-pack/legacy/plugins/siem/public/components/timeline/helpers.test.tsx
# x-pack/legacy/plugins/siem/public/components/timeline/helpers.tsx
# x-pack/legacy/plugins/siem/public/lib/keury/index.ts
# x-pack/legacy/plugins/siem/public/pages/hosts/details/index.tsx
# x-pack/legacy/plugins/siem/public/pages/hosts/hosts.tsx
# x-pack/legacy/plugins/siem/public/pages/network/ip_details/index.tsx
* fix merge conflicts
This PR adds a new syntax to KQL for querying nested fields.
Nested fields can be queried in two different ways:
Parts of the query may only match a single nested doc (bool inside nested). This is what most people want when querying on a nested field.
Parts of the query may match different nested docs (nested inside bool). This is how a regular object field works but nested fields can be queried in the same way. Although generally less useful, there are occasions where one might want to query a nested field in this way.
The new KQL syntax supports both.
* Update moment related packages
* unify moment version in yarn.lock
* adapt ro pluralization fixes
* update some bad pluralization in fixtures/snapshots
* Update dependency del to v5
* unify del/rimraf usage with del v5
* update yarn.lock
* update kbn-pm distributable
* remove poorly transformed code
* force some deletes from tmp
* mock less of the fs module
* force tmp deletion in a few more suites
* please make this the last force
# Conflicts:
# packages/kbn-plugin-helpers/package.json
We've migrated the old syntax at query time for a long time, but when creating a phrase filter we still used the old (now invalid) syntax. This could be confusing to users since the raw query DSL is viewable in the filter editor. Also, since map_phrase.ts only expected the old syntax, it wouldn't detect a match_phrase filter if the user entered the new syntax into the raw DSL editor.
This PR updates all of the filter generation code that I know of to use the new syntax, and it updates the map_phrase.ts file to accept both syntaxes.
* Expose Saved Objects client in request context
* API Integration test for savedobjects in req context
* SavedObjectsClient docs
* SavedObjectsClient#find remove dependency on indexPatterns
And use the saved objects mappings instead
* Review comments
* Review comments, fixes and tests
* Use correct type for KQL syntax check
* Get rid of addFiltersAndChangeTimeFilter
* ts fix
* remove timefilter dependency from filter manager
* code review change
* Fixed bug in tests
* changeTimeFilter
* Refactored mappers and filter service to have no dependency on indexPatterns by generating the filter disaplyName in the relevant components.
* Fix map and flatten test
* Fixed filter state manager test
* Remove async from addFIlters and setFilters
* Fixed saved objects test - removed (display)value from url
* Make removeAll sync
* defer setFilters and removeAll in dashboard controller - temp hack
* fixed translation in filter view
* update strings
* Fixed range rendering
* map range converter
* [lens] Use top nav in Lens app
* Add tests for saved query, pass filters around more places
* Fix filter passing
* Add unit test for field popover making correct queries
* Respond to review feedback
* Fix type errors
* Respond to all review comments
* Remove commented code
* Top nav should be compatible as angular directive
* Fix rendering issue with filter updates
* Respond to review comments and add onChange test
* Add specific test for the index pattern bug from Tina
* Add KQL functionality in the find function of the saved objects
wip
rename variable from KQL to filter, fix unit test + add new ones
miss security pluggins
review I
fix api changes
refactor after reviewing with Rudolf
fix type
review III
review IV
for security put back allowed logic back to return empty results
remove StaticIndexPattern
review V
fix core_api_changes
fix type
* validate filter to match requirement type.attributes.key or type.savedObjectKey
* Fix types
* fix a bug + add more api integration test
* fix types in test until we create package @kbn/types
* fix type issue
* fix api integration test
* export nodeTypes from packages @kbn/es-query instead of the function buildNodeKuery
* throw 400- bad request when validation error in find
* fix type issue
* accept api change
* renove _ to represent private
* fix unit test + add doc
* add comment to explain why we removed the private
* Update babel related packages (#43595)
* upgrade all babel related modules, and bump lodash minors
* update kbn-pm dist
* update fetch-mock, necessary to use core-js 3
* use regenerator transform in jest tests, as required by EUI
* disable useBuiltIns rather than using regenerator plugin
* remove extra regenerator-runtime import
* Update package.json
* update yarn.lock
Filters were never really meant to take part in scoring in the first place. This PR puts filters back in the filter context where they were in 4.x so they can benefit from the filter cache.
* Updating to @elastic/lodash to 3.10.1-npm-kibana2
* Updating lodash to 4.17.13
There's a stray dependency to lodash 4.17.11 via cypress which doesn't
use "^". This is a dev dependency, so we can ignore it for a bit while
they update their package.json
* Updating some forgotten references to @elastic/lodash@3.10.1-kibana1
* Updating lodash-es to 4.17.13
* Updating to @elastic/lodash to 3.10.1-npm-kibana3
* Using the root resolutions to update cypress's version of lodash.
Thanks @spalger!!!
* [ts] upgrade to 3.5.3 (#40228)
* [ts] upgrade to 3.5.2
* [ts] run from cwd so that relative paths are correct
* move eslint-disable-line comment into jsx attribute
* autofix eslint violations
* avoid generic type, it's not necessary and problematic
* make elasticsearch.cluster optional, many instances don't have them
* remove invalid prop
* expand AllowUnknownProperties to cleanly handle arrays
* esfilter values can sometimes be an array of strings
* allow exception objects to have unknown properties
* define accumulator as a boolean
* fix return type
* return a 404 if beat isn't found after update
* use Object.values for better types
* define return type of get() call
* define value type for Set
* define return value of get()
* define State property type
* use less get(), so ts can infer types
* define Set item type
* map state type
* make default_operator optional, since it's not always defined
* remove seemingly unused prop
* define return type of get() fn
* define inner type for state
* don't define base types are objects with index signatues
* regenerate public api docs
* make indices privileges optional too
* remove unnecessary index-signature from Exception interface
* use variadic _arg instead
* [core/plugin] use Record<string, any> rather than {}
* replace a couple more instances of {}
* revert some unnecessary changes
* remove unused types
* [reporting] output, payload, and meta are required properties
* bump to latest patch version
# Conflicts:
# x-pack/legacy/plugins/canvas/server/usage/custom_element_collector.ts
# x-pack/legacy/plugins/canvas/server/usage/workpad_collector.ts
* define inner type for Set
* default `config.get()` to any instead of unknown
* Enable use of KQL and autocomplete in filters agg editor (#37287)
This PR updates the filters agg editor to use the full QueryBar component, enabling use of KQL and autocomplete inside the editor for this aggregation in Visualize.
* remove unused translation
* Pin dependencies (#37406)
* Pin dependencies
* include core-js in babel packages
* specify core-js version when using `useBuiltIns`
* dedupe @babel/types to avoid "range of null" problem
* chore(deps): update dependency del to v4 (#37466)
* chore(deps): update dependency del to v4
* remove @types/del since they ship with the package now
(cherry picked from commit c60e3491a2)
* Update gulp related packages (#37456)
(cherry picked from commit 62893ec0bc)
* Update babel related packages (#37464)
(cherry picked from commit 830bd1f0ca)
Attempts to make KQL syntax errors more sensical to the average user.
I initially tried to use a similar solution to the one we used for detecting usage of old lucene syntax. In other words, I tried to create rules in the grammar that would match strings containing common mistakes the user might make and throw custom error messages for each situation. This proved to be more difficult for detecting errors in the regular language. While the Lucene rules could be completely separated from the main grammar, the KQL error rules had to be mixed into the main grammar which made it much more complex and had a lot of unintended side effects.
So instead I decided to lean more heavily on PEG's built in error reporting. Giving certain rules human readable names allows the parser to use those names in the error reporting instead of auto generating a long list of possible characters that might be expected based on the matching rules. Since the PEG errors contain location information I was also able to add ascii art that points the user to exactly where the error occurred in their query string. While this approach is not quite as nice as bespoke error messages that tell the user exactly what is wrong in plain English, it's much more maintainable and I think it still results in much better error messages compared to what we have today.
I've also removed the old original kuery grammar (for queries like is(response, 200)). We were only using it to display an error if I user was still using the old syntax. This version of kuery hasn't existed since 6.3 and we've had error messages telling users this since then. I think it's safe to remove the legacy parser at this point, which greatly reduces the complexity of our error reporting.
* Adds time_zone to query
* Adds dateFormatTZ to kuery query
Removes comments
* Adds defaults for the dateFormatTZ to the function signatures
* Adds tests for date match in kuery
Modifies test
* Adds a test for get_es_query_config
* Adds test for get timezone from settings utility method
* Adds tests for modified range method
Adds config param test to node_types/functions
code clean up
* resolves initial PR comments
* Refactors build_es_query test
* Refactors get_time_zone_from_settings test
* Uses spys to test that the config is passed down to children in ast toElasticsearchQuery
* removes default config nulls
* Deletes sinon.spy tests in kuery
* removes moment.setDefault from __tests__/get_timezone_from_settings.js
