forked from MirrorHub/synapse
Add an admin API for users' media statistics (#8700)
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users. Related to #6094 It is the first API for statistics. Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats) Signed-off-by: Dirk Klimpel dirk@klimpel.org
This commit is contained in:
Dirk Klimpel
GitHub
parent
e4676bd877
commit
c3119d1536
6 changed files with 820 additions and 0 deletions
1
changelog.d/8700.feature
Normal file
1
changelog.d/8700.feature
Normal file
|
|
@ -0,0 +1 @@
|
|||
Add an admin API for local user media statistics. Contributed by @dklimpel.
|
||||
83
docs/admin_api/statistics.md
Normal file
83
docs/admin_api/statistics.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# Users' media usage statistics
|
||||
|
||||
Returns information about all local media usage of users. Gives the
|
||||
possibility to filter them by time and user.
|
||||
|
||||
The API is:
|
||||
|
||||
```
|
||||
GET /_synapse/admin/v1/statistics/users/media
|
||||
```
|
||||
|
||||
To use it, you will need to authenticate by providing an `access_token`
|
||||
for a server admin: see [README.rst](README.rst).
|
||||
|
||||
A response body like the following is returned:
|
||||
|
||||
```json
|
||||
{
|
||||
"users": [
|
||||
{
|
||||
"displayname": "foo_user_0",
|
||||
"media_count": 2,
|
||||
"media_length": 134,
|
||||
"user_id": "@foo_user_0:test"
|
||||
},
|
||||
{
|
||||
"displayname": "foo_user_1",
|
||||
"media_count": 2,
|
||||
"media_length": 134,
|
||||
"user_id": "@foo_user_1:test"
|
||||
}
|
||||
],
|
||||
"next_token": 3,
|
||||
"total": 10
|
||||
}
|
||||
```
|
||||
|
||||
To paginate, check for `next_token` and if present, call the endpoint
|
||||
again with `from` set to the value of `next_token`. This will return a new page.
|
||||
|
||||
If the endpoint does not return a `next_token` then there are no more
|
||||
reports to paginate through.
|
||||
|
||||
**Parameters**
|
||||
|
||||
The following parameters should be set in the URL:
|
||||
|
||||
* `limit`: string representing a positive integer - Is optional but is
|
||||
used for pagination, denoting the maximum number of items to return
|
||||
in this call. Defaults to `100`.
|
||||
* `from`: string representing a positive integer - Is optional but used for pagination,
|
||||
denoting the offset in the returned results. This should be treated as an opaque value
|
||||
and not explicitly set to anything other than the return value of `next_token` from a
|
||||
previous call. Defaults to `0`.
|
||||
* `order_by` - string - The method in which to sort the returned list of users. Valid values are:
|
||||
- `user_id` - Users are ordered alphabetically by `user_id`. This is the default.
|
||||
- `displayname` - Users are ordered alphabetically by `displayname`.
|
||||
- `media_length` - Users are ordered by the total size of uploaded media in bytes.
|
||||
Smallest to largest.
|
||||
- `media_count` - Users are ordered by number of uploaded media. Smallest to largest.
|
||||
* `from_ts` - string representing a positive integer - Considers only
|
||||
files created at this timestamp or later. Unix timestamp in ms.
|
||||
* `until_ts` - string representing a positive integer - Considers only
|
||||
files created at this timestamp or earlier. Unix timestamp in ms.
|
||||
* `search_term` - string - Filter users by their user ID localpart **or** displayname.
|
||||
The search term can be found in any part of the string.
|
||||
Defaults to no filtering.
|
||||
* `dir` - string - Direction of order. Either `f` for forwards or `b` for backwards.
|
||||
Setting this value to `b` will reverse the above sort order. Defaults to `f`.
|
||||
|
||||
|
||||
**Response**
|
||||
|
||||
The following fields are returned in the JSON response body:
|
||||
|
||||
* `users` - An array of objects, each containing information
|
||||
about the user and their local media. Objects contain the following fields:
|
||||
- `displayname` - string - Displayname of this user.
|
||||
- `media_count` - integer - Number of uploaded media by this user.
|
||||
- `media_length` - integer - Size of uploaded media in bytes by this user.
|
||||
- `user_id` - string - Fully-qualified user ID (ex. `@user:server.com`).
|
||||
* `next_token` - integer - Opaque value used for pagination. See above.
|
||||
* `total` - integer - Total number of users after filtering.
|
||||
|
|
@ -47,6 +47,7 @@ from synapse.rest.admin.rooms import (
|
|||
ShutdownRoomRestServlet,
|
||||
)
|
||||
from synapse.rest.admin.server_notice_servlet import SendServerNoticeServlet
|
||||
from synapse.rest.admin.statistics import UserMediaStatisticsRestServlet
|
||||
from synapse.rest.admin.users import (
|
||||
AccountValidityRenewServlet,
|
||||
DeactivateAccountRestServlet,
|
||||
|
|
@ -227,6 +228,7 @@ def register_servlets(hs, http_server):
|
|||
DeviceRestServlet(hs).register(http_server)
|
||||
DevicesRestServlet(hs).register(http_server)
|
||||
DeleteDevicesRestServlet(hs).register(http_server)
|
||||
UserMediaStatisticsRestServlet(hs).register(http_server)
|
||||
EventReportDetailRestServlet(hs).register(http_server)
|
||||
EventReportsRestServlet(hs).register(http_server)
|
||||
PushersRestServlet(hs).register(http_server)
|
||||
|
|
|
|||
122
synapse/rest/admin/statistics.py
Normal file
122
synapse/rest/admin/statistics.py
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 Dirk Klimpel
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Tuple
|
||||
|
||||
from synapse.api.errors import Codes, SynapseError
|
||||
from synapse.http.servlet import RestServlet, parse_integer, parse_string
|
||||
from synapse.http.site import SynapseRequest
|
||||
from synapse.rest.admin._base import admin_patterns, assert_requester_is_admin
|
||||
from synapse.storage.databases.main.stats import UserSortOrder
|
||||
from synapse.types import JsonDict
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from synapse.server import HomeServer
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class UserMediaStatisticsRestServlet(RestServlet):
|
||||
"""
|
||||
Get statistics about uploaded media by users.
|
||||
"""
|
||||
|
||||
PATTERNS = admin_patterns("/statistics/users/media$")
|
||||
|
||||
def __init__(self, hs: "HomeServer"):
|
||||
self.hs = hs
|
||||
self.auth = hs.get_auth()
|
||||
self.store = hs.get_datastore()
|
||||
|
||||
async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
|
||||
await assert_requester_is_admin(self.auth, request)
|
||||
|
||||
order_by = parse_string(
|
||||
request, "order_by", default=UserSortOrder.USER_ID.value
|
||||
)
|
||||
if order_by not in (
|
||||
UserSortOrder.MEDIA_LENGTH.value,
|
||||
UserSortOrder.MEDIA_COUNT.value,
|
||||
UserSortOrder.USER_ID.value,
|
||||
UserSortOrder.DISPLAYNAME.value,
|
||||
):
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Unknown value for order_by: %s" % (order_by,),
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
start = parse_integer(request, "from", default=0)
|
||||
if start < 0:
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter from must be a string representing a positive integer.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
limit = parse_integer(request, "limit", default=100)
|
||||
if limit < 0:
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter limit must be a string representing a positive integer.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
from_ts = parse_integer(request, "from_ts", default=0)
|
||||
if from_ts < 0:
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter from_ts must be a string representing a positive integer.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
until_ts = parse_integer(request, "until_ts")
|
||||
if until_ts is not None:
|
||||
if until_ts < 0:
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter until_ts must be a string representing a positive integer.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
if until_ts <= from_ts:
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter until_ts must be greater than from_ts.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
search_term = parse_string(request, "search_term")
|
||||
if search_term == "":
|
||||
raise SynapseError(
|
||||
400,
|
||||
"Query parameter search_term cannot be an empty string.",
|
||||
errcode=Codes.INVALID_PARAM,
|
||||
)
|
||||
|
||||
direction = parse_string(request, "dir", default="f")
|
||||
if direction not in ("f", "b"):
|
||||
raise SynapseError(
|
||||
400, "Unknown direction: %s" % (direction,), errcode=Codes.INVALID_PARAM
|
||||
)
|
||||
|
||||
users_media, total = await self.store.get_users_media_usage_paginate(
|
||||
start, limit, from_ts, until_ts, order_by, direction, search_term
|
||||
)
|
||||
ret = {"users": users_media, "total": total}
|
||||
if (start + limit) < total:
|
||||
ret["next_token"] = start + len(users_media)
|
||||
|
||||
return 200, ret
|
||||
|
|
@ -16,15 +16,18 @@
|
|||
|
||||
import logging
|
||||
from collections import Counter
|
||||
from enum import Enum
|
||||
from itertools import chain
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from twisted.internet.defer import DeferredLock
|
||||
|
||||
from synapse.api.constants import EventTypes, Membership
|
||||
from synapse.api.errors import StoreError
|
||||
from synapse.storage.database import DatabasePool
|
||||
from synapse.storage.databases.main.state_deltas import StateDeltasStore
|
||||
from synapse.storage.engines import PostgresEngine
|
||||
from synapse.types import JsonDict
|
||||
from synapse.util.caches.descriptors import cached
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -59,6 +62,23 @@ TYPE_TO_TABLE = {"room": ("room_stats", "room_id"), "user": ("user_stats", "user
|
|||
TYPE_TO_ORIGIN_TABLE = {"room": ("rooms", "room_id"), "user": ("users", "name")}
|
||||
|
||||
|
||||
class UserSortOrder(Enum):
|
||||
"""
|
||||
Enum to define the sorting method used when returning users
|
||||
with get_users_media_usage_paginate
|
||||
|
||||
MEDIA_LENGTH = ordered by size of uploaded media. Smallest to largest.
|
||||
MEDIA_COUNT = ordered by number of uploaded media. Smallest to largest.
|
||||
USER_ID = ordered alphabetically by `user_id`.
|
||||
DISPLAYNAME = ordered alphabetically by `displayname`
|
||||
"""
|
||||
|
||||
MEDIA_LENGTH = "media_length"
|
||||
MEDIA_COUNT = "media_count"
|
||||
USER_ID = "user_id"
|
||||
DISPLAYNAME = "displayname"
|
||||
|
||||
|
||||
class StatsStore(StateDeltasStore):
|
||||
def __init__(self, database: DatabasePool, db_conn, hs):
|
||||
super().__init__(database, db_conn, hs)
|
||||
|
|
@ -882,3 +902,110 @@ class StatsStore(StateDeltasStore):
|
|||
complete_with_stream_id=pos,
|
||||
absolute_field_overrides={"joined_rooms": joined_rooms},
|
||||
)
|
||||
|
||||
async def get_users_media_usage_paginate(
|
||||
self,
|
||||
start: int,
|
||||
limit: int,
|
||||
from_ts: Optional[int] = None,
|
||||
until_ts: Optional[int] = None,
|
||||
order_by: Optional[UserSortOrder] = UserSortOrder.USER_ID.value,
|
||||
direction: Optional[str] = "f",
|
||||
search_term: Optional[str] = None,
|
||||
) -> Tuple[List[JsonDict], Dict[str, int]]:
|
||||
"""Function to retrieve a paginated list of users and their uploaded local media
|
||||
(size and number). This will return a json list of users and the
|
||||
total number of users matching the filter criteria.
|
||||
|
||||
Args:
|
||||
start: offset to begin the query from
|
||||
limit: number of rows to retrieve
|
||||
from_ts: request only media that are created later than this timestamp (ms)
|
||||
until_ts: request only media that are created earlier than this timestamp (ms)
|
||||
order_by: the sort order of the returned list
|
||||
direction: sort ascending or descending
|
||||
search_term: a string to filter user names by
|
||||
Returns:
|
||||
A list of user dicts and an integer representing the total number of
|
||||
users that exist given this query
|
||||
"""
|
||||
|
||||
def get_users_media_usage_paginate_txn(txn):
|
||||
filters = []
|
||||
args = [self.hs.config.server_name]
|
||||
|
||||
if search_term:
|
||||
filters.append("(lmr.user_id LIKE ? OR displayname LIKE ?)")
|
||||
args.extend(["@%" + search_term + "%:%", "%" + search_term + "%"])
|
||||
|
||||
if from_ts:
|
||||
filters.append("created_ts >= ?")
|
||||
args.extend([from_ts])
|
||||
if until_ts:
|
||||
filters.append("created_ts <= ?")
|
||||
args.extend([until_ts])
|
||||
|
||||
# Set ordering
|
||||
if UserSortOrder(order_by) == UserSortOrder.MEDIA_LENGTH:
|
||||
order_by_column = "media_length"
|
||||
elif UserSortOrder(order_by) == UserSortOrder.MEDIA_COUNT:
|
||||
order_by_column = "media_count"
|
||||
elif UserSortOrder(order_by) == UserSortOrder.USER_ID:
|
||||
order_by_column = "lmr.user_id"
|
||||
elif UserSortOrder(order_by) == UserSortOrder.DISPLAYNAME:
|
||||
order_by_column = "displayname"
|
||||
else:
|
||||
raise StoreError(
|
||||
500, "Incorrect value for order_by provided: %s" % order_by
|
||||
)
|
||||
|
||||
if direction == "b":
|
||||
order = "DESC"
|
||||
else:
|
||||
order = "ASC"
|
||||
|
||||
where_clause = "WHERE " + " AND ".join(filters) if len(filters) > 0 else ""
|
||||
|
||||
sql_base = """
|
||||
FROM local_media_repository as lmr
|
||||
LEFT JOIN profiles AS p ON lmr.user_id = '@' || p.user_id || ':' || ?
|
||||
{}
|
||||
GROUP BY lmr.user_id, displayname
|
||||
""".format(
|
||||
where_clause
|
||||
)
|
||||
|
||||
# SQLite does not support SELECT COUNT(*) OVER()
|
||||
sql = """
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT lmr.user_id
|
||||
{sql_base}
|
||||
) AS count_user_ids
|
||||
""".format(
|
||||
sql_base=sql_base,
|
||||
)
|
||||
txn.execute(sql, args)
|
||||
count = txn.fetchone()[0]
|
||||
|
||||
sql = """
|
||||
SELECT
|
||||
lmr.user_id,
|
||||
displayname,
|
||||
COUNT(lmr.user_id) as media_count,
|
||||
SUM(media_length) as media_length
|
||||
{sql_base}
|
||||
ORDER BY {order_by_column} {order}
|
||||
LIMIT ? OFFSET ?
|
||||
""".format(
|
||||
sql_base=sql_base, order_by_column=order_by_column, order=order,
|
||||
)
|
||||
|
||||
args += [limit, start]
|
||||
txn.execute(sql, args)
|
||||
users = self.db_pool.cursor_to_dict(txn)
|
||||
|
||||
return users, count
|
||||
|
||||
return await self.db_pool.runInteraction(
|
||||
"get_users_media_usage_paginate_txn", get_users_media_usage_paginate_txn
|
||||
)
|
||||
|
|
|
|||
485
tests/rest/admin/test_statistics.py
Normal file
485
tests/rest/admin/test_statistics.py
Normal file
|
|
@ -0,0 +1,485 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2020 Dirk Klimpel
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from binascii import unhexlify
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import synapse.rest.admin
|
||||
from synapse.api.errors import Codes
|
||||
from synapse.rest.client.v1 import login
|
||||
|
||||
from tests import unittest
|
||||
|
||||
|
||||
class UserMediaStatisticsTestCase(unittest.HomeserverTestCase):
|
||||
servlets = [
|
||||
synapse.rest.admin.register_servlets,
|
||||
login.register_servlets,
|
||||
]
|
||||
|
||||
def prepare(self, reactor, clock, hs):
|
||||
self.store = hs.get_datastore()
|
||||
self.media_repo = hs.get_media_repository_resource()
|
||||
|
||||
self.admin_user = self.register_user("admin", "pass", admin=True)
|
||||
self.admin_user_tok = self.login("admin", "pass")
|
||||
|
||||
self.other_user = self.register_user("user", "pass")
|
||||
self.other_user_tok = self.login("user", "pass")
|
||||
|
||||
self.url = "/_synapse/admin/v1/statistics/users/media"
|
||||
|
||||
def test_no_auth(self):
|
||||
"""
|
||||
Try to list users without authentication.
|
||||
"""
|
||||
request, channel = self.make_request("GET", self.url, b"{}")
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.MISSING_TOKEN, channel.json_body["errcode"])
|
||||
|
||||
def test_requester_is_no_admin(self):
|
||||
"""
|
||||
If the user is not a server admin, an error 403 is returned.
|
||||
"""
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url, json.dumps({}), access_token=self.other_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(403, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.FORBIDDEN, channel.json_body["errcode"])
|
||||
|
||||
def test_invalid_parameter(self):
|
||||
"""
|
||||
If parameters are invalid, an error is returned.
|
||||
"""
|
||||
# unkown order_by
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?order_by=bar", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# negative from
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from=-5", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# negative limit
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?limit=-5", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# negative from_ts
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from_ts=-1234", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# negative until_ts
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?until_ts=-1234", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# until_ts smaller from_ts
|
||||
request, channel = self.make_request(
|
||||
"GET",
|
||||
self.url + "?from_ts=10&until_ts=5",
|
||||
access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# empty search term
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?search_term=", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
# invalid search order
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?dir=bar", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(400, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(Codes.INVALID_PARAM, channel.json_body["errcode"])
|
||||
|
||||
def test_limit(self):
|
||||
"""
|
||||
Testing list of media with limit
|
||||
"""
|
||||
self._create_users_with_media(10, 2)
|
||||
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?limit=5", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 10)
|
||||
self.assertEqual(len(channel.json_body["users"]), 5)
|
||||
self.assertEqual(channel.json_body["next_token"], 5)
|
||||
self._check_fields(channel.json_body["users"])
|
||||
|
||||
def test_from(self):
|
||||
"""
|
||||
Testing list of media with a defined starting point (from)
|
||||
"""
|
||||
self._create_users_with_media(20, 2)
|
||||
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from=5", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 20)
|
||||
self.assertEqual(len(channel.json_body["users"]), 15)
|
||||
self.assertNotIn("next_token", channel.json_body)
|
||||
self._check_fields(channel.json_body["users"])
|
||||
|
||||
def test_limit_and_from(self):
|
||||
"""
|
||||
Testing list of media with a defined starting point and limit
|
||||
"""
|
||||
self._create_users_with_media(20, 2)
|
||||
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from=5&limit=10", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 20)
|
||||
self.assertEqual(channel.json_body["next_token"], 15)
|
||||
self.assertEqual(len(channel.json_body["users"]), 10)
|
||||
self._check_fields(channel.json_body["users"])
|
||||
|
||||
def test_next_token(self):
|
||||
"""
|
||||
Testing that `next_token` appears at the right place
|
||||
"""
|
||||
|
||||
number_users = 20
|
||||
self._create_users_with_media(number_users, 3)
|
||||
|
||||
# `next_token` does not appear
|
||||
# Number of results is the number of entries
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?limit=20", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], number_users)
|
||||
self.assertEqual(len(channel.json_body["users"]), number_users)
|
||||
self.assertNotIn("next_token", channel.json_body)
|
||||
|
||||
# `next_token` does not appear
|
||||
# Number of max results is larger than the number of entries
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?limit=21", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], number_users)
|
||||
self.assertEqual(len(channel.json_body["users"]), number_users)
|
||||
self.assertNotIn("next_token", channel.json_body)
|
||||
|
||||
# `next_token` does appear
|
||||
# Number of max results is smaller than the number of entries
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?limit=19", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], number_users)
|
||||
self.assertEqual(len(channel.json_body["users"]), 19)
|
||||
self.assertEqual(channel.json_body["next_token"], 19)
|
||||
|
||||
# Set `from` to value of `next_token` for request remaining entries
|
||||
# Check `next_token` does not appear
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from=19", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], number_users)
|
||||
self.assertEqual(len(channel.json_body["users"]), 1)
|
||||
self.assertNotIn("next_token", channel.json_body)
|
||||
|
||||
def test_no_media(self):
|
||||
"""
|
||||
Tests that a normal lookup for statistics is successfully
|
||||
if users have no media created
|
||||
"""
|
||||
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url, access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
|
||||
self.assertEqual(200, channel.code, msg=channel.json_body)
|
||||
self.assertEqual(0, channel.json_body["total"])
|
||||
self.assertEqual(0, len(channel.json_body["users"]))
|
||||
|
||||
def test_order_by(self):
|
||||
"""
|
||||
Testing order list with parameter `order_by`
|
||||
"""
|
||||
|
||||
# create users
|
||||
self.register_user("user_a", "pass", displayname="UserZ")
|
||||
userA_tok = self.login("user_a", "pass")
|
||||
self._create_media(userA_tok, 1)
|
||||
|
||||
self.register_user("user_b", "pass", displayname="UserY")
|
||||
userB_tok = self.login("user_b", "pass")
|
||||
self._create_media(userB_tok, 3)
|
||||
|
||||
self.register_user("user_c", "pass", displayname="UserX")
|
||||
userC_tok = self.login("user_c", "pass")
|
||||
self._create_media(userC_tok, 2)
|
||||
|
||||
# order by user_id
|
||||
self._order_test("user_id", ["@user_a:test", "@user_b:test", "@user_c:test"])
|
||||
self._order_test(
|
||||
"user_id", ["@user_a:test", "@user_b:test", "@user_c:test"], "f",
|
||||
)
|
||||
self._order_test(
|
||||
"user_id", ["@user_c:test", "@user_b:test", "@user_a:test"], "b",
|
||||
)
|
||||
|
||||
# order by displayname
|
||||
self._order_test(
|
||||
"displayname", ["@user_c:test", "@user_b:test", "@user_a:test"]
|
||||
)
|
||||
self._order_test(
|
||||
"displayname", ["@user_c:test", "@user_b:test", "@user_a:test"], "f",
|
||||
)
|
||||
self._order_test(
|
||||
"displayname", ["@user_a:test", "@user_b:test", "@user_c:test"], "b",
|
||||
)
|
||||
|
||||
# order by media_length
|
||||
self._order_test(
|
||||
"media_length", ["@user_a:test", "@user_c:test", "@user_b:test"],
|
||||
)
|
||||
self._order_test(
|
||||
"media_length", ["@user_a:test", "@user_c:test", "@user_b:test"], "f",
|
||||
)
|
||||
self._order_test(
|
||||
"media_length", ["@user_b:test", "@user_c:test", "@user_a:test"], "b",
|
||||
)
|
||||
|
||||
# order by media_count
|
||||
self._order_test(
|
||||
"media_count", ["@user_a:test", "@user_c:test", "@user_b:test"],
|
||||
)
|
||||
self._order_test(
|
||||
"media_count", ["@user_a:test", "@user_c:test", "@user_b:test"], "f",
|
||||
)
|
||||
self._order_test(
|
||||
"media_count", ["@user_b:test", "@user_c:test", "@user_a:test"], "b",
|
||||
)
|
||||
|
||||
def test_from_until_ts(self):
|
||||
"""
|
||||
Testing filter by time with parameters `from_ts` and `until_ts`
|
||||
"""
|
||||
# create media earlier than `ts1` to ensure that `from_ts` is working
|
||||
self._create_media(self.other_user_tok, 3)
|
||||
self.pump(1)
|
||||
ts1 = self.clock.time_msec()
|
||||
|
||||
# list all media when filter is not set
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url, access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["users"][0]["media_count"], 3)
|
||||
|
||||
# filter media starting at `ts1` after creating first media
|
||||
# result is 0
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?from_ts=%s" % (ts1,), access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 0)
|
||||
|
||||
self._create_media(self.other_user_tok, 3)
|
||||
self.pump(1)
|
||||
ts2 = self.clock.time_msec()
|
||||
# create media after `ts2` to ensure that `until_ts` is working
|
||||
self._create_media(self.other_user_tok, 3)
|
||||
|
||||
# filter media between `ts1` and `ts2`
|
||||
request, channel = self.make_request(
|
||||
"GET",
|
||||
self.url + "?from_ts=%s&until_ts=%s" % (ts1, ts2),
|
||||
access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["users"][0]["media_count"], 3)
|
||||
|
||||
# filter media until `ts2` and earlier
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?until_ts=%s" % (ts2,), access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["users"][0]["media_count"], 6)
|
||||
|
||||
def test_search_term(self):
|
||||
self._create_users_with_media(20, 1)
|
||||
|
||||
# check without filter get all users
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url, access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 20)
|
||||
|
||||
# filter user 1 and 10-19 by `user_id`
|
||||
request, channel = self.make_request(
|
||||
"GET",
|
||||
self.url + "?search_term=foo_user_1",
|
||||
access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 11)
|
||||
|
||||
# filter on this user in `displayname`
|
||||
request, channel = self.make_request(
|
||||
"GET",
|
||||
self.url + "?search_term=bar_user_10",
|
||||
access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["users"][0]["displayname"], "bar_user_10")
|
||||
self.assertEqual(channel.json_body["total"], 1)
|
||||
|
||||
# filter and get empty result
|
||||
request, channel = self.make_request(
|
||||
"GET", self.url + "?search_term=foobar", access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
|
||||
self.assertEqual(channel.json_body["total"], 0)
|
||||
|
||||
def _create_users_with_media(self, number_users: int, media_per_user: int):
|
||||
"""
|
||||
Create a number of users with a number of media
|
||||
Args:
|
||||
number_users: Number of users to be created
|
||||
media_per_user: Number of media to be created for each user
|
||||
"""
|
||||
for i in range(number_users):
|
||||
self.register_user("foo_user_%s" % i, "pass", displayname="bar_user_%s" % i)
|
||||
user_tok = self.login("foo_user_%s" % i, "pass")
|
||||
self._create_media(user_tok, media_per_user)
|
||||
|
||||
def _create_media(self, user_token: str, number_media: int):
|
||||
"""
|
||||
Create a number of media for a specific user
|
||||
Args:
|
||||
user_token: Access token of the user
|
||||
number_media: Number of media to be created for the user
|
||||
"""
|
||||
upload_resource = self.media_repo.children[b"upload"]
|
||||
for i in range(number_media):
|
||||
# file size is 67 Byte
|
||||
image_data = unhexlify(
|
||||
b"89504e470d0a1a0a0000000d4948445200000001000000010806"
|
||||
b"0000001f15c4890000000a49444154789c63000100000500010d"
|
||||
b"0a2db40000000049454e44ae426082"
|
||||
)
|
||||
|
||||
# Upload some media into the room
|
||||
self.helper.upload_media(
|
||||
upload_resource, image_data, tok=user_token, expect_code=200
|
||||
)
|
||||
|
||||
def _check_fields(self, content: List[Dict[str, Any]]):
|
||||
"""Checks that all attributes are present in content
|
||||
Args:
|
||||
content: List that is checked for content
|
||||
"""
|
||||
for c in content:
|
||||
self.assertIn("user_id", c)
|
||||
self.assertIn("displayname", c)
|
||||
self.assertIn("media_count", c)
|
||||
self.assertIn("media_length", c)
|
||||
|
||||
def _order_test(
|
||||
self, order_type: str, expected_user_list: List[str], dir: Optional[str] = None
|
||||
):
|
||||
"""Request the list of users in a certain order. Assert that order is what
|
||||
we expect
|
||||
Args:
|
||||
order_type: The type of ordering to give the server
|
||||
expected_user_list: The list of user_ids in the order we expect to get
|
||||
back from the server
|
||||
dir: The direction of ordering to give the server
|
||||
"""
|
||||
|
||||
url = self.url + "?order_by=%s" % (order_type,)
|
||||
if dir is not None and dir in ("b", "f"):
|
||||
url += "&dir=%s" % (dir,)
|
||||
request, channel = self.make_request(
|
||||
"GET", url.encode("ascii"), access_token=self.admin_user_tok,
|
||||
)
|
||||
self.render(request)
|
||||
self.assertEqual(200, channel.code, msg=channel.json_body)
|
||||
self.assertEqual(channel.json_body["total"], len(expected_user_list))
|
||||
|
||||
returned_order = [row["user_id"] for row in channel.json_body["users"]]
|
||||
self.assertListEqual(expected_user_list, returned_order)
|
||||
self._check_fields(channel.json_body["users"])
|
||||
Loading…
Reference in a new issue