synapse/tests/storage
James Salter d902181de9
Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635)
Support a unified search query syntax which leverages more of the full-text
search of each database supported by Synapse.

Supports, with the same syntax across Postgresql 11+ and Sqlite:

- quoted "search terms"
- `AND`, `OR`, `-` (negation) operators
- Matching words based on their stem, e.g. searches for "dog" matches
  documents containing "dogs". 

This is achieved by 

- If on postgresql 11+, pass the user input to `websearch_to_tsquery`
- If on sqlite, manually parse the query and transform it into the sqlite-specific
  query syntax.

Note that postgresql 10, which is close to end-of-life, falls back to using
`phraseto_tsquery`, which only supports a subset of the features.

Multiple terms separated by a space are implicitly ANDed.

Note that:

1. There is no escaping of full-text syntax that might be supported by the database;
  e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this
  as accidental functionality and depend on something we don't guarantee.
2. English text is assumed for stemming. To support other languages, either the target
  language needs to be known at the time of indexing the message (via room metadata,
  or otherwise), or a separate index for each language supported could be created.

Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries
Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
2022-10-25 14:05:22 -04:00
..
databases Remove _get_events_cache check optimisation from _have_seen_events_dict (#14161) 2022-10-18 10:33:21 +01:00
util Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
__init__.py Reference Matrix Home Server 2014-08-12 15:10:52 +01:00
test__base.py Optimise _update_client_ips_batch_txn to batch together database operations. (#12252) 2022-04-08 15:29:13 +01:00
test_account_data.py Use the ignored_users table to test event visibility & sync. (#12225) 2022-03-15 14:06:05 -04:00
test_appservice.py Remove code which updates application_services_state.last_txn (#12680) 2022-05-17 11:07:18 +01:00
test_background_update.py Add type hints to tests files. (#12256) 2022-03-21 09:43:16 -04:00
test_base.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
test_cleanup_extrems.py Default to private room visibility rather than public when a client does not specify one, according to spec. (#12350) 2022-04-01 15:55:09 +01:00
test_client_ips.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_database.py Handle cancellation in DatabasePool.runInteraction() (#12199) 2022-03-16 15:07:41 +00:00
test_devices.py Additional constants for EDU types. (#12884) 2022-05-27 07:14:36 -04:00
test_directory.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_e2e_room_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_end_to_end_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_event_chain.py Persist CreateRoom events to DB in a batch (#13800) 2022-09-28 10:11:48 +00:00
test_event_federation.py Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
test_event_metrics.py Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
test_event_push_actions.py Use threaded receipts when fetching events for push. (#13878) 2022-10-04 12:07:02 -04:00
test_events.py Faster room joins: avoid blocking when pulling events with missing prevs (#13355) 2022-07-26 12:39:23 +01:00
test_id_generators.py Remove redundant get_success calls in test code (#12346) 2022-04-01 16:10:31 +01:00
test_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_main.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_monthly_active_users.py Update mypy and mypy-zope, attempt 3 (#13993) 2022-09-30 17:36:28 +01:00
test_profile.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_purge.py Async get event cache prep (#13242) 2022-07-15 09:30:46 +00:00
test_receipts.py Clarify that a method returns only unthreaded receipts. (#13937) 2022-09-29 07:07:31 -04:00
test_redaction.py Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
test_registration.py Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
test_relations.py Accept threaded receipts for events related to the root event. (#14174) 2022-10-14 18:05:25 +00:00
test_rollback_worker.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_room.py Remove obsolete RoomEventsStoreTestCase (#13200) 2022-07-07 13:47:26 +01:00
test_room_search.py Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635) 2022-10-25 14:05:22 -04:00
test_roommember.py Fix that user cannot /forget rooms after the last member has left (#13546) 2022-08-30 09:58:38 +00:00
test_state.py Make DictionaryCache have better expiry properties (#13292) 2022-07-21 17:13:44 +01:00
test_stream.py Support filtering the /messages API by relation type (MSC3874). (#14148) 2022-10-17 11:32:11 -04:00
test_transactions.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_txn_limit.py Require direct references to configuration variables. (#10985) 2021-10-06 10:47:41 -04:00
test_unsafe_locale.py Refuse to start if DB has an unsafe locale (#12262) 2022-03-23 10:23:05 -07:00
test_user_directory.py Remove remaining bits of groups code. (#12936) 2022-06-01 09:41:25 -04:00