Merge branch 'rav/url_preview_limit_title_2' into matrix-org-hotfixes
This commit is contained in:
commit
7fa4586e36
41
AUTHORS.rst
41
AUTHORS.rst
|
@ -1,34 +1,8 @@
|
|||
Erik Johnston <erik at matrix.org>
|
||||
* HS core
|
||||
* Federation API impl
|
||||
The following is an incomplete list of people outside the core team who have
|
||||
contributed to Synapse. It is no longer maintained: more recent contributions
|
||||
are listed in the `changelog <CHANGES.md>`_.
|
||||
|
||||
Mark Haines <mark at matrix.org>
|
||||
* HS core
|
||||
* Crypto
|
||||
* Content repository
|
||||
* CS v2 API impl
|
||||
|
||||
Kegan Dougal <kegan at matrix.org>
|
||||
* HS core
|
||||
* CS v1 API impl
|
||||
* AS API impl
|
||||
|
||||
Paul "LeoNerd" Evans <paul at matrix.org>
|
||||
* HS core
|
||||
* Presence
|
||||
* Typing Notifications
|
||||
* Performance metrics and caching layer
|
||||
|
||||
Dave Baker <dave at matrix.org>
|
||||
* Push notifications
|
||||
* Auth CS v2 impl
|
||||
|
||||
Matthew Hodgson <matthew at matrix.org>
|
||||
* General doc & housekeeping
|
||||
* Vertobot/vertobridge matrix<->verto PoC
|
||||
|
||||
Emmanuel Rohee <manu at matrix.org>
|
||||
* Supporting iOS clients (testability and fallback registration)
|
||||
----
|
||||
|
||||
Turned to Dust <dwinslow86 at gmail.com>
|
||||
* ArchLinux installation instructions
|
||||
|
@ -62,16 +36,13 @@ Christoph Witzany <christoph at web.crofting.com>
|
|||
* Add LDAP support for authentication
|
||||
|
||||
Pierre Jaury <pierre at jaury.eu>
|
||||
* Docker packaging
|
||||
* Docker packaging
|
||||
|
||||
Serban Constantin <serban.constantin at gmail dot com>
|
||||
* Small bug fix
|
||||
|
||||
Jason Robinson <jasonr at matrix.org>
|
||||
* Minor fixes
|
||||
|
||||
Joseph Weston <joseph at weston.cloud>
|
||||
+ Add admin API for querying HS version
|
||||
* Add admin API for querying HS version
|
||||
|
||||
Benjamin Saunders <ben.e.saunders at gmail dot com>
|
||||
* Documentation improvements
|
||||
|
|
25
CHANGES.md
25
CHANGES.md
|
@ -1,3 +1,17 @@
|
|||
Synapse 1.5.0 (2019-10-29)
|
||||
==========================
|
||||
|
||||
Security updates
|
||||
----------------
|
||||
|
||||
This release includes a security fix ([\#6262](https://github.com/matrix-org/synapse/issues/6262), below). Administrators are encouraged to upgrade as soon as possible.
|
||||
|
||||
Bugfixes
|
||||
--------
|
||||
|
||||
- Fix bug where room directory search was case sensitive. ([\#6268](https://github.com/matrix-org/synapse/issues/6268))
|
||||
|
||||
|
||||
Synapse 1.5.0rc2 (2019-10-28)
|
||||
=============================
|
||||
|
||||
|
@ -19,13 +33,6 @@ Internal Changes
|
|||
Synapse 1.5.0rc1 (2019-10-24)
|
||||
==========================
|
||||
|
||||
This release includes a database migration step **which may take a long time to complete**:
|
||||
|
||||
- Allow devices to be marked as hidden, for use by features such as cross-signing.
|
||||
This adds a new field with a default value to the devices field in the database,
|
||||
and so the database upgrade may take a long time depending on how many devices
|
||||
are in the database. ([\#5759](https://github.com/matrix-org/synapse/issues/5759))
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
|
@ -69,6 +76,10 @@ Internal Changes
|
|||
----------------
|
||||
|
||||
- Update `user_filters` table to have a unique index, and non-null columns. Thanks to @pik for contributing this. ([\#1172](https://github.com/matrix-org/synapse/issues/1172), [\#6175](https://github.com/matrix-org/synapse/issues/6175), [\#6184](https://github.com/matrix-org/synapse/issues/6184))
|
||||
- Allow devices to be marked as hidden, for use by features such as cross-signing.
|
||||
This adds a new field with a default value to the devices field in the database,
|
||||
and so the database upgrade may take a long time depending on how many devices
|
||||
are in the database. ([\#5759](https://github.com/matrix-org/synapse/issues/5759))
|
||||
- Move lookup-related functions from RoomMemberHandler to IdentityHandler. ([\#5978](https://github.com/matrix-org/synapse/issues/5978))
|
||||
- Improve performance of the public room list directory. ([\#6019](https://github.com/matrix-org/synapse/issues/6019), [\#6152](https://github.com/matrix-org/synapse/issues/6152), [\#6153](https://github.com/matrix-org/synapse/issues/6153), [\#6154](https://github.com/matrix-org/synapse/issues/6154))
|
||||
- Edit header dicts docstrings in `SimpleHttpClient` to note that `str` or `bytes` can be passed as header keys. ([\#6077](https://github.com/matrix-org/synapse/issues/6077))
|
||||
|
|
|
@ -114,17 +114,6 @@ directory, you will need both a regular newsfragment *and* an entry in the
|
|||
debian changelog. (Though typically such changes should be submitted as two
|
||||
separate pull requests.)
|
||||
|
||||
Attribution
|
||||
~~~~~~~~~~~
|
||||
|
||||
Everyone who contributes anything to Matrix is welcome to be listed in the
|
||||
AUTHORS.rst file for the project in question. Please feel free to include a
|
||||
change to AUTHORS.rst in your pull request to list yourself and a short
|
||||
description of the area(s) you've worked on. Also, we sometimes have swag to
|
||||
give away to contributors - if you feel that Matrix-branded apparel is missing
|
||||
from your life, please mail us your shipping address to matrix at matrix.org and
|
||||
we'll try to fix it :)
|
||||
|
||||
Sign off
|
||||
~~~~~~~~
|
||||
|
||||
|
|
14
UPGRADE.rst
14
UPGRADE.rst
|
@ -2,7 +2,7 @@ Upgrading Synapse
|
|||
=================
|
||||
|
||||
Before upgrading check if any special steps are required to upgrade from the
|
||||
what you currently have installed to current version of Synapse. The extra
|
||||
version you currently have installed to the current version of Synapse. The extra
|
||||
instructions that may be required are listed later in this document.
|
||||
|
||||
* If Synapse was installed using `prebuilt packages
|
||||
|
@ -29,7 +29,7 @@ instructions that may be required are listed later in this document.
|
|||
running:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
|
||||
git pull
|
||||
pip install --upgrade .
|
||||
|
||||
|
@ -75,6 +75,16 @@ for example:
|
|||
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
|
||||
|
||||
|
||||
Upgrading to v1.5.0
|
||||
===================
|
||||
|
||||
This release includes a database migration which may take several minutes to
|
||||
complete if there are a large number (more than a million or so) of entries in
|
||||
the ``devices`` table. This is only likely to a be a problem on very large
|
||||
installations.
|
||||
|
||||
|
||||
Upgrading to v1.4.0
|
||||
===================
|
||||
|
||||
|
|
|
@ -1 +0,0 @@
|
|||
Fix bug where room directory search was case sensitive.
|
1
changelog.d/6331.feature
Normal file
1
changelog.d/6331.feature
Normal file
|
@ -0,0 +1 @@
|
|||
Limit the length of data returned by url previews, to prevent DoS attacks.
|
1
changelog.d/6334.feature
Normal file
1
changelog.d/6334.feature
Normal file
|
@ -0,0 +1 @@
|
|||
Limit the length of data returned by url previews, to prevent DoS attacks.
|
6
debian/changelog
vendored
6
debian/changelog
vendored
|
@ -1,3 +1,9 @@
|
|||
matrix-synapse-py3 (1.5.0) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.5.0.
|
||||
|
||||
-- Synapse Packaging team <packages@matrix.org> Tue, 29 Oct 2019 14:28:41 +0000
|
||||
|
||||
matrix-synapse-py3 (1.4.1) stable; urgency=medium
|
||||
|
||||
* New synapse release 1.4.1.
|
||||
|
|
|
@ -36,7 +36,7 @@ try:
|
|||
except ImportError:
|
||||
pass
|
||||
|
||||
__version__ = "1.5.0rc2"
|
||||
__version__ = "1.5.0"
|
||||
|
||||
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
|
||||
# We import here so that we don't have to install a bunch of deps when
|
||||
|
|
|
@ -56,6 +56,9 @@ logger = logging.getLogger(__name__)
|
|||
_charset_match = re.compile(br"<\s*meta[^>]*charset\s*=\s*([a-z0-9-]+)", flags=re.I)
|
||||
_content_type_match = re.compile(r'.*; *charset="?(.*?)"?(;|$)', flags=re.I)
|
||||
|
||||
OG_TAG_NAME_MAXLEN = 50
|
||||
OG_TAG_VALUE_MAXLEN = 1000
|
||||
|
||||
|
||||
class PreviewUrlResource(DirectServeResource):
|
||||
isLeaf = True
|
||||
|
@ -169,7 +172,7 @@ class PreviewUrlResource(DirectServeResource):
|
|||
ts (int):
|
||||
|
||||
Returns:
|
||||
Deferred[str]: json-encoded og data
|
||||
Deferred[bytes]: json-encoded og data
|
||||
"""
|
||||
# check the URL cache in the DB (which will also provide us with
|
||||
# historical previews, if we have any)
|
||||
|
@ -270,6 +273,18 @@ class PreviewUrlResource(DirectServeResource):
|
|||
logger.warn("Failed to find any OG data in %s", url)
|
||||
og = {}
|
||||
|
||||
# filter out any stupidly long values
|
||||
keys_to_remove = []
|
||||
for k, v in og.items():
|
||||
# values can be numeric as well as strings, hence the cast to str
|
||||
if len(k) > OG_TAG_NAME_MAXLEN or len(str(v)) > OG_TAG_VALUE_MAXLEN:
|
||||
logger.warning(
|
||||
"Pruning overlong tag %s from OG data", k[:OG_TAG_NAME_MAXLEN]
|
||||
)
|
||||
keys_to_remove.append(k)
|
||||
for k in keys_to_remove:
|
||||
del og[k]
|
||||
|
||||
logger.debug("Calculated OG for %s as %s", url, og)
|
||||
|
||||
jsonog = json.dumps(og)
|
||||
|
@ -504,6 +519,10 @@ def _calc_og(tree, media_uri):
|
|||
og = {}
|
||||
for tag in tree.xpath("//*/meta[starts-with(@property, 'og:')]"):
|
||||
if "content" in tag.attrib:
|
||||
# if we've got more than 50 tags, someone is taking the piss
|
||||
if len(og) >= 50:
|
||||
logger.warning("Skipping OG for page with too many 'og:' tags")
|
||||
return {}
|
||||
og[tag.attrib["property"]] = tag.attrib["content"]
|
||||
|
||||
# TODO: grab article: meta tags too, e.g.:
|
||||
|
|
|
@ -247,6 +247,41 @@ class URLPreviewTests(unittest.HomeserverTestCase):
|
|||
self.assertEqual(channel.code, 200)
|
||||
self.assertEqual(channel.json_body["og:title"], "\u0434\u043a\u0430")
|
||||
|
||||
def test_overlong_title(self):
|
||||
self.lookups["matrix.org"] = [(IPv4Address, "8.8.8.8")]
|
||||
|
||||
end_content = (
|
||||
b"<html><head>"
|
||||
b"<title>" + b"x" * 2000 + b"</title>"
|
||||
b'<meta property="og:description" content="hi" />'
|
||||
b"</head></html>"
|
||||
)
|
||||
|
||||
request, channel = self.make_request(
|
||||
"GET", "url_preview?url=http://matrix.org", shorthand=False
|
||||
)
|
||||
request.render(self.preview_url)
|
||||
self.pump()
|
||||
|
||||
client = self.reactor.tcpClients[0][2].buildProtocol(None)
|
||||
server = AccumulatingProtocol()
|
||||
server.makeConnection(FakeTransport(client, self.reactor))
|
||||
client.makeConnection(FakeTransport(server, self.reactor))
|
||||
client.dataReceived(
|
||||
(
|
||||
b"HTTP/1.0 200 OK\r\nContent-Length: %d\r\n"
|
||||
b'Content-Type: text/html; charset="windows-1251"\r\n\r\n'
|
||||
)
|
||||
% (len(end_content),)
|
||||
+ end_content
|
||||
)
|
||||
|
||||
self.pump()
|
||||
self.assertEqual(channel.code, 200)
|
||||
res = channel.json_body
|
||||
# We should only see the `og:description` field, as `title` is too long and should be stripped out
|
||||
self.assertCountEqual(["og:description"], res.keys())
|
||||
|
||||
def test_ipaddr(self):
|
||||
"""
|
||||
IP addresses can be previewed directly.
|
||||
|
|
Loading…
Reference in a new issue