Improved sync speed by resolving the N+1 query issues.
Solves #1402 and Solves #1453
With this change there is just one query done to retreive all the
important data, and matching is done in-code/memory.
With a very large database the sync time went down about 3 times.
Also updated misc crates and Github Actions versions.
This is a rather large PR which updates the async branch to have all the
database methods as an async fn.
Some iter/map logic needed to be changed to a stream::iter().then(), but
besides that most changes were just adding async/await where needed.
Updated several json response models.
Also fixed a few small bugs.
ciphers.rs:
- post_ciphers_create:
* Prevent cipher creation to organization without a collection.
- update_cipher_from_data:
* ~~Fixed removal of user_uuid which prevent user-owned shared-cipher to be not editable anymore when set to read-only.~~
* Cleanup the json_data by removing the `Response` key/values from several objects.
- delete_all:
* Do not delete all Collections during the Purge of an Organization (same as upstream).
cipher.rs:
- Cipher::to_json:
* Updated json response to match upstream.
* Return empty json object if there is no type_data instead of values which should not be set for the type_data.
organizations.rs:
* Added two new endpoints to prevent Javascript errors regarding tax
organization.rs:
- Organization::to_json:
* Updated response model to match upstream
- UserOrganization::to_json:
* Updated response model to match upstream
collection.rs:
- Collection::{to_json, to_json_details}:
* Updated the json response model, and added a detailed version used during the sync
- hide_passwords_for_user:
* Added this function to return if the passwords should be hidden or not for the user at the specific collection (used by `to_json_details`)
Update 1: Some small changes after comments from @jjlin.
Update 2: Fixed vault purge by user to make sure the cipher is not part of an organization.
Resolves#971Closes#990, Closes#991
When using MariaDB v10.5+ Foreign-Key errors were popping up because of
some changes in that version. To mitigate this on MariaDB and other
MySQL forks those errors are now catched, and instead of a replace_into
an update will happen. I have tested this as thorough as possible with
MariaDB 10.5, 10.4, 10.3 and the default MySQL on Ubuntu Focal. And
tested it again using sqlite, all seems to be ok on all tables.
resolves#1081. resolves#1065, resolves#1050
If org owners/admins set their org access to only include selected
collections, then ciphers from non-selected collections shouldn't
appear in "My Vault". This matches the upstream behavior.
Diesel requires the following changes:
- Separate connection and pool types per connection, the generate_connections! macro generates an enum with a variant per db type
- Separate migrations and schemas, these were always imported as one type depending on db feature, now they are all imported under different module names
- Separate model objects per connection, the db_object! macro generates one object for each connection with the diesel macros, a generic object, and methods to convert between the connection-specific and the generic ones
- Separate connection queries, the db_run! macro allows writing only one that gets compiled for all databases or multiple ones
This includes migrations as well as Dockerfile's for amd64.
The biggest change is that replace_into isn't supported by Diesel for the
PostgreSQL backend, instead requiring the use of on_conflict. This
unfortunately requires a branch for save() on all of the models currently
using replace_into.
