maxmustermann/synapse
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge branch 'develop' into matrix-org-hotfixes

Browse source
This commit is contained in:
Matthew Hodgson 2018-09-26 11:00:26 +01:00
parent 128043072b f65163627f
commit 2e0d2879d0
72 changed files with 642 additions and 357 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

74
CHANGES.md
View file

@ -1,3 +1,77 @@
Synapse 0.33.5.1 (2018-09-25)
=============================
Internal Changes
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko! ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
Synapse 0.33.5 (2018-09-24)
===========================
No significant changes.
Synapse 0.33.5rc1 (2018-09-17)
==============================
Features
--------
- Python 3.5 and 3.6 support is now in beta. ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
- Implement `event_format` filter param in `/sync` ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates a large number of new series. ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
- guest users should not be part of mau total ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
- Fix existing room tags not coming down sync when joining a room ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
- Fix jwt import check ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
- fix VOIP crashes under Python 3 (#3821) ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
- Fix manhole so that it works with latest openssh clients ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
- Fix outbound requests occasionally wedging, which can result in federation breaking between servers. ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
- Show heroes if room name/canonical alias has been deleted ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
- Fix handling of redacted events from federation ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
- Mitigate outbound federation randomly becoming wedged ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR. ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
- http/ is now ported to Python 3. ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
- Improve human readable error messages for threepid registration/account update ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
- Make /sync slightly faster by avoiding needless copies ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
- handlers/ is now ported to Python 3. ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
- Limit the number of PDUs/EDUs per federation transaction ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
- Only start postgres instance for postgres tests on Travis CI ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
- tests/ is now ported to Python 3. ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
- crypto/ is now ported to Python 3. ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
- rest/ is now ported to Python 3. ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
- add some logging for the keyring queue ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
- speed up lazy loading by 2-3x ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
- Improved Dockerfile to remove build requirements after building reducing the image size. ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
- Disable lazy loading for incremental syncs for now ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
- federation/ is now ported to Python 3. ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
- Log when we retry outbound requests ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
- Removed some excess logging messages. ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
- Speed up purge history for rooms that have been previously purged ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
- Refactor some HTTP timeout code. ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
- Fix running merged builds on CircleCI ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
- Fix typo in replication stream exception. ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
- Add in flight real time metrics for Measure blocks ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
- Disable buffering and automatic retrying in treq requests to prevent timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
- mention jemalloc in the README ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
- Remove unmaintained "nuke-room-from-db.sh" script ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
Synapse 0.33.4 (2018-09-07)
===========================

1
changelog.d/3576.feature
View file

@ -1 +0,0 @@
Python 3.5+ is now supported.

1
changelog.d/3578.bugfix Normal file
View file

@ -0,0 +1 @@
Fix problem when playing media from Chrome using direct URL (thanks @remjey!)

1
changelog.d/3704.misc
View file

@ -1 +0,0 @@
CircleCI tests now run on the potential merge of a PR.

1
changelog.d/3771.misc
View file

@ -1 +0,0 @@
http/ is now ported to Python 3.

1
changelog.d/3788.bugfix
View file

@ -1 +0,0 @@
Remove connection ID for replication prometheus metrics, as it creates a large number of new series.

1
changelog.d/3789.misc
View file

@ -1 +0,0 @@
Improve human readable error messages for threepid registration/account update

1
changelog.d/3790.feature
View file

@ -1 +0,0 @@
Implement `event_format` filter param in `/sync`

1
changelog.d/3795.misc
View file

@ -1 +0,0 @@
Make /sync slightly faster by avoiding needless copies

1
changelog.d/3800.bugfix
View file

@ -1 +0,0 @@
guest users should not be part of mau total

1
changelog.d/3803.misc
View file

@ -1 +0,0 @@
handlers/ is now ported to Python 3.

1
changelog.d/3804.bugfix
View file

@ -1 +0,0 @@
Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.

1
changelog.d/3805.misc
View file

@ -1 +0,0 @@
Limit the number of PDUs/EDUs per federation transaction

1
changelog.d/3806.misc
View file

@ -1 +0,0 @@
Only start postgres instance for postgres tests on Travis CI

1
changelog.d/3808.misc
View file

@ -1 +0,0 @@
tests/ is now ported to Python 3.

1
changelog.d/3810.bugfix
View file

@ -1 +0,0 @@
Fix existing room tags not coming down sync when joining a room

1
changelog.d/3822.misc
View file

@ -1 +0,0 @@
crypto/ is now ported to Python 3.

1
changelog.d/3823.misc
View file

@ -1 +0,0 @@
rest/ is now ported to Python 3.

1
changelog.d/3824.bugfix
View file

@ -1 +0,0 @@
Fix jwt import check

1
changelog.d/3826.misc
View file

@ -1 +0,0 @@
add some logging for the keyring queue

1
changelog.d/3827.misc
View file

@ -1 +0,0 @@
speed up lazy loading by 2-3x

1
changelog.d/3834.misc
View file

@ -1 +0,0 @@
Improved Dockerfile to remove build requirements after building reducing the image size.

1
changelog.d/3835.bugfix
View file

@ -1 +0,0 @@
fix VOIP crashes under Python 3 (#3821)

1
changelog.d/3840.misc
View file

@ -1 +0,0 @@
Disable lazy loading for incremental syncs for now

1
changelog.d/3841.bugfix
View file

@ -1 +0,0 @@
Fix manhole so that it works with latest openssh clients

1
changelog.d/3845.bugfix
View file

@ -1 +0,0 @@
Fix outbound requests occasionally wedging, which can result in federation breaking between servers.

1
changelog.d/3846.feature
View file

@ -1 +0,0 @@
Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users

1
changelog.d/3847.misc
View file

@ -1 +0,0 @@
federation/ is now ported to Python 3.

1
changelog.d/3851.bugfix
View file

@ -1 +0,0 @@
Show heroes if room name/canonical alias has been deleted

1
changelog.d/3853.misc
View file

@ -1 +0,0 @@
Log when we retry outbound requests

1
changelog.d/3855.misc
View file

@ -1 +0,0 @@
Removed some excess logging messages.

1
changelog.d/3856.misc
View file

@ -1 +0,0 @@
Speed up purge history for rooms that have been previously purged

1
changelog.d/3857.misc
View file

@ -1 +0,0 @@
Refactor some HTTP timeout code.

1
changelog.d/3858.misc
View file

@ -1 +0,0 @@
Fix running merged builds on CircleCI

1
changelog.d/3859.bugfix
View file

@ -1 +0,0 @@
Fix handling of redacted events from federation

1
changelog.d/3860.misc
View file

@ -1 +0,0 @@
Fix typo in replication stream exception.

1
changelog.d/3871.misc
View file

@ -1 +0,0 @@
Add in flight real time metrics for Measure blocks

1
changelog.d/3872.misc
View file

@ -1 +0,0 @@
Disable buffering and automatic retrying in treq requests to prevent timeouts.

0
changelog.d/3874.bugfix
View file

1
changelog.d/3875.bugfix
View file

@ -1 +0,0 @@
Mitigate outbound federation randomly becoming wedged

1
changelog.d/3877.misc
View file

@ -1 +0,0 @@
mention jemalloc in the README

1
changelog.d/3888.misc
View file

@ -1 +0,0 @@
Remove unmaintained "nuke-room-from-db.sh" script

1
changelog.d/3911.misc Normal file
View file

@ -0,0 +1 @@
Fix the docker image building on python 3

1
changelog.d/3925.misc Normal file
View file

@ -0,0 +1 @@
Fix spurious exceptions when remote http client closes conncetion

1
changelog.d/3932.bugfix Normal file
View file

@ -0,0 +1 @@
Fix some instances of ExpiringCache not expiring cache items

1
changelog.d/3936.bugfix Normal file
View file

@ -0,0 +1 @@
Fix out-of-bounds error when LLing yourself

1
changelog.d/3947.misc Normal file
View file

@ -0,0 +1 @@
Require attrs 16.0.0 or later

750
contrib/grafana/synapse.json
View file

File diff suppressed because it is too large Load diff

3
docker/Dockerfile
View file

@ -1,4 +1,5 @@
FROM docker.io/python:2-alpine3.8
ARG PYTHON_VERSION=2
FROM docker.io/python:${PYTHON_VERSION}-alpine3.8
COPY . /synapse

3
docker/start.py
View file

@ -5,6 +5,7 @@ import os
import sys
import subprocess
import glob
import codecs
# Utility functions
convert = lambda src, dst, environ: open(dst, "w").write(jinja2.Template(open(src).read()).render(**environ))
@ -23,7 +24,7 @@ def generate_secrets(environ, secrets):
with open(filename) as handle: value = handle.read()
else:
print("Generating a random secret for {}".format(name))
value = os.urandom(32).encode("hex")
value = codecs.encode(os.urandom(32), "hex").decode()
with open(filename, "w") as handle: handle.write(value)
environ[secret] = value

2
synapse/__init__.py
View file

@ -27,4 +27,4 @@ try:
except ImportError:
pass
__version__ = "0.33.4"
__version__ = "0.33.5.1"

1
synapse/app/appservice.py
View file

@ -172,7 +172,6 @@ def start(config_options):
def start():
ps.get_datastore().start_profiling()
ps.get_state_handler().start_caching()
reactor.callWhenRunning(start)

1
synapse/app/client_reader.py
View file

@ -181,7 +181,6 @@ def start(config_options):
ss.start_listening(config.worker_listeners)
def start():
ss.get_state_handler().start_caching()
ss.get_datastore().start_profiling()
reactor.callWhenRunning(start)

1
synapse/app/event_creator.py
View file

@ -199,7 +199,6 @@ def start(config_options):
ss.start_listening(config.worker_listeners)
def start():
ss.get_state_handler().start_caching()
ss.get_datastore().start_profiling()
reactor.callWhenRunning(start)

1
synapse/app/federation_reader.py
View file

@ -168,7 +168,6 @@ def start(config_options):
ss.start_listening(config.worker_listeners)
def start():
ss.get_state_handler().start_caching()
ss.get_datastore().start_profiling()
reactor.callWhenRunning(start)

1
synapse/app/federation_sender.py
View file

@ -201,7 +201,6 @@ def start(config_options):
def start():
ps.get_datastore().start_profiling()
ps.get_state_handler().start_caching()
reactor.callWhenRunning(start)
_base.start_worker_reactor("synapse-federation-sender", config)

1
synapse/app/frontend_proxy.py
View file

@ -258,7 +258,6 @@ def start(config_options):
ss.start_listening(config.worker_listeners)
def start():
ss.get_state_handler().start_caching()
ss.get_datastore().start_profiling()
reactor.callWhenRunning(start)

1
synapse/app/homeserver.py
View file

@ -384,7 +384,6 @@ def setup(config_options):
def start():
hs.get_pusherpool().start()
hs.get_state_handler().start_caching()
hs.get_datastore().start_profiling()
hs.get_datastore().start_doing_background_updates()
hs.get_federation_client().start_get_pdu_cache()

1
synapse/app/media_repository.py
View file

@ -168,7 +168,6 @@ def start(config_options):
ss.start_listening(config.worker_listeners)
def start():
ss.get_state_handler().start_caching()
ss.get_datastore().start_profiling()
reactor.callWhenRunning(start)

1
synapse/app/pusher.py
View file

@ -228,7 +228,6 @@ def start(config_options):
def start():
ps.get_pusherpool().start()
ps.get_datastore().start_profiling()
ps.get_state_handler().start_caching()
reactor.callWhenRunning(start)

1
synapse/app/synchrotron.py
View file

@ -435,7 +435,6 @@ def start(config_options):
def start():
ss.get_datastore().start_profiling()
ss.get_state_handler().start_caching()
reactor.callWhenRunning(start)

1
synapse/app/user_dir.py
View file

@ -229,7 +229,6 @@ def start(config_options):
def start():
ps.get_datastore().start_profiling()
ps.get_state_handler().start_caching()
reactor.callWhenRunning(start)

28
synapse/federation/federation_client.py
View file

@ -66,6 +66,14 @@ class FederationClient(FederationBase):
self.state = hs.get_state_handler()
self.transport_layer = hs.get_federation_transport_client()
self._get_pdu_cache = ExpiringCache(
cache_name="get_pdu_cache",
clock=self._clock,
max_len=1000,
expiry_ms=120 * 1000,
reset_expiry_on_get=False,
)
def _clear_tried_cache(self):
"""Clear pdu_destination_tried cache"""
now = self._clock.time_msec()
@ -82,17 +90,6 @@ class FederationClient(FederationBase):
if destination_dict:
self.pdu_destination_tried[event_id] = destination_dict
def start_get_pdu_cache(self):
self._get_pdu_cache = ExpiringCache(
cache_name="get_pdu_cache",
clock=self._clock,
max_len=1000,
expiry_ms=120 * 1000,
reset_expiry_on_get=False,
)
self._get_pdu_cache.start()
@log_function
def make_query(self, destination, query_type, args,
retry_on_dns_fail=False, ignore_backoff=False):
@ -229,10 +226,9 @@ class FederationClient(FederationBase):
# TODO: Rate limit the number of times we try and get the same event.
if self._get_pdu_cache:
ev = self._get_pdu_cache.get(event_id)
if ev:
defer.returnValue(ev)
ev = self._get_pdu_cache.get(event_id)
if ev:
defer.returnValue(ev)
pdu_attempts = self.pdu_destination_tried.setdefault(event_id, {})
@ -285,7 +281,7 @@ class FederationClient(FederationBase):
)
continue
if self._get_pdu_cache is not None and signed_pdu:
if signed_pdu:
self._get_pdu_cache[event_id] = signed_pdu
defer.returnValue(signed_pdu)

13
synapse/handlers/sync.py
View file

@ -717,10 +717,6 @@ class SyncHandler(object):
)
]
# always make sure we LL ourselves so we know we're in the room
# (if we are), to fix https://github.com/vector-im/riot-web/issues/7209
types.append((EventTypes.Member, sync_config.user.to_string()))
# only apply the filtering to room members
filtered_types = [EventTypes.Member]
@ -730,6 +726,13 @@ class SyncHandler(object):
}
if full_state:
if lazy_load_members:
# always make sure we LL ourselves so we know we're in the room
# (if we are) to fix https://github.com/vector-im/riot-web/issues/7209
# We only need apply this on full state syncs given we disabled
# LL for incr syncs in #3840.
types.append((EventTypes.Member, sync_config.user.to_string()))
if batch:
current_state_ids = yield self.store.get_state_ids_for_event(
batch.events[-1].event_id, types=types,
@ -798,7 +801,7 @@ class SyncHandler(object):
else:
state_ids = {}
if lazy_load_members:
if types:
if types and batch.events:
# We're returning an incremental sync, with no
# "gap" since the previous sync, so normally there would be
# no state to return.

49
synapse/http/server.py
View file

@ -84,10 +84,21 @@ def wrap_json_request_handler(h):
logger.info(
"%s SynapseError: %s - %s", request, code, e.msg
)
respond_with_json(
request, code, e.error_dict(), send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
# Only respond with an error response if we haven't already started
# writing, otherwise lets just kill the connection
if request.startedWriting:
if request.transport:
try:
request.transport.abortConnection()
except Exception:
# abortConnection throws if the connection is already closed
pass
else:
respond_with_json(
request, code, e.error_dict(), send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
except Exception:
# failure.Failure() fishes the original Failure out
@ -100,16 +111,26 @@ def wrap_json_request_handler(h):
request,
f.getTraceback().rstrip(),
)
respond_with_json(
request,
500,
{
"error": "Internal server error",
"errcode": Codes.UNKNOWN,
},
send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
# Only respond with an error response if we haven't already started
# writing, otherwise lets just kill the connection
if request.startedWriting:
if request.transport:
try:
request.transport.abortConnection()
except Exception:
# abortConnection throws if the connection is already closed
pass
else:
respond_with_json(
request,
500,
{
"error": "Internal server error",
"errcode": Codes.UNKNOWN,
},
send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
return wrap_async_request_handler(wrapped_request_handler)

4
synapse/python_dependencies.py
View file

@ -58,7 +58,9 @@ REQUIREMENTS = {
"phonenumbers>=8.2.0": ["phonenumbers"],
"six": ["six"],
"prometheus_client": ["prometheus_client"],
"attrs": ["attr"],
# we use attr.s(slots), which arrived in 16.0.0
"attrs>=16.0.0": ["attr>=16.0.0"],
"netaddr>=0.7.18": ["netaddr"],
}

1
synapse/rest/media/v1/download_resource.py
View file

@ -52,6 +52,7 @@ class DownloadResource(Resource):
b" script-src 'none';"
b" plugin-types application/pdf;"
b" style-src 'unsafe-inline';"
b" media-src 'self';"
b" object-src 'self';"
)
server_name, media_id, name = parse_media_id(request)

1
synapse/rest/media/v1/preview_url_resource.py
View file

@ -79,7 +79,6 @@ class PreviewUrlResource(Resource):
# don't spider URLs more often than once an hour
expiry_ms=60 * 60 * 1000,
)
self._cache.start()
self._cleaner_loop = self.clock.looping_call(
self._start_expire_url_cache_data, 10 * 1000,

9
synapse/state/__init__.py
View file

@ -95,10 +95,6 @@ class StateHandler(object):
self.hs = hs
self._state_resolution_handler = hs.get_state_resolution_handler()
def start_caching(self):
# TODO: remove this shim
self._state_resolution_handler.start_caching()
@defer.inlineCallbacks
def get_current_state(self, room_id, event_type=None, state_key="",
latest_event_ids=None):
@ -428,9 +424,6 @@ class StateResolutionHandler(object):
self._state_cache = None
self.resolve_linearizer = Linearizer(name="state_resolve_lock")
def start_caching(self):
logger.debug("start_caching")
self._state_cache = ExpiringCache(
cache_name="state_cache",
clock=self.clock,
@ -440,8 +433,6 @@ class StateResolutionHandler(object):
reset_expiry_on_get=True,
)
self._state_cache.start()
@defer.inlineCallbacks
@log_function
def resolve_state_groups(

1
synapse/util/caches/expiringcache.py
View file

@ -58,7 +58,6 @@ class ExpiringCache(object):
self.metrics = register_cache("expiring", cache_name, self)
def start(self):
if not self._expiry_ms:
# Don't bother starting the loop if things never expire
return

4
tests/http/test_fedclient.py
View file

@ -54,7 +54,7 @@ class FederationClientTests(HomeserverTestCase):
def test_client_never_connect(self):
"""
If the HTTP request is not connected and is timed out, it'll give a
ConnectingCancelledError.
ConnectingCancelledError or TimeoutError.
"""
d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
@ -76,7 +76,7 @@ class FederationClientTests(HomeserverTestCase):
self.reactor.advance(10.5)
f = self.failureResultOf(d)
self.assertIsInstance(f.value, ConnectingCancelledError)
self.assertIsInstance(f.value, (ConnectingCancelledError, TimeoutError))
def test_client_connect_no_response(self):
"""

1
tests/util/test_expiring_cache.py
View file

@ -65,7 +65,6 @@ class ExpiringCacheTestCase(unittest.TestCase):
def test_time_eviction(self):
clock = MockClock()
cache = ExpiringCache("test", clock, expiry_ms=1000)
cache.start()
cache["key"] = 1
clock.advance_time(0.5)