Require poetry>=1.3.2 (#14860)

* Upgrade to new lockfile format

Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write.

Cheat sheet:

```
poetry --version
poetry show > scratch/before
pipx upgrade poetry
poetry --version
poetry show > scratch/after
diff scratch{before,after} && echo "no change!"
```

* Use Poetry 1.3.2 when reading or writing lockfile

* Remove unneeded(?) poetry dep for cibuildwheel

* Update docs

* Remove redundant call to setup-python

* Remove outdated comments related to Poetry 1.x

* Remove outdated docs line

was fixed in #13082

* Minor improvements to poetry cheat sheet

* Invoke setup-python-poetry with explicit version

Not sure about this. It's hardcoding versions everywhere.

* Changelog

* Check the lockfile is version 2.0

Might one day incorporate other checks like #14742

* Typo fixes, thanks Sean

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
This commit is contained in:
David Robertson 2023-01-17 18:04:44 +00:00 committed by GitHub
parent f820740b7d
commit b88cfe6d41
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 1739 additions and 1684 deletions

23
.ci/scripts/check_lockfile.py Executable file
View file

@ -0,0 +1,23 @@
#! /usr/bin/env python
import sys
if sys.version_info < (3, 11):
raise RuntimeError("Requires at least Python 3.11, to import tomllib")
import tomllib
with open("poetry.lock", "rb") as f:
lockfile = tomllib.load(f)
try:
lock_version = lockfile["metadata"]["lock-version"]
assert lock_version == "2.0"
except Exception:
print(
"""\
Lockfile is not version 2.0. You probably need to upgrade poetry on your local box
and re-run `poetry lock --no-update`. See the Poetry cheat sheet at
https://matrix-org.github.io/synapse/develop/development/dependencies.html
"""
)
raise

View file

@ -53,7 +53,7 @@ with open('pyproject.toml', 'w') as f:
" "
python3 -c "$REMOVE_DEV_DEPENDENCIES" python3 -c "$REMOVE_DEV_DEPENDENCIES"
pip install poetry==1.2.0 pip install poetry==1.3.2
poetry lock poetry lock
echo "::group::Patched pyproject.toml" echo "::group::Patched pyproject.toml"

View file

@ -37,7 +37,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: "3.x" python-version: "3.x"
poetry-version: "1.2.0" poetry-version: "1.3.2"
extras: "all" extras: "all"
# Dump installed versions for debugging. # Dump installed versions for debugging.
- run: poetry run pip list > before.txt - run: poetry run pip list > before.txt

View file

@ -127,7 +127,7 @@ jobs:
python-version: "3.x" python-version: "3.x"
- name: Install cibuildwheel - name: Install cibuildwheel
run: python -m pip install cibuildwheel==2.9.0 poetry==1.2.0 run: python -m pip install cibuildwheel==2.9.0
- name: Set up QEMU to emulate aarch64 - name: Set up QEMU to emulate aarch64
if: matrix.arch == 'aarch64' if: matrix.arch == 'aarch64'

View file

@ -33,11 +33,10 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: "3.x"
poetry-version: "1.3.2"
extras: "all" extras: "all"
- run: poetry run scripts-dev/generate_sample_config.sh --check - run: poetry run scripts-dev/generate_sample_config.sh --check
- run: poetry run scripts-dev/config-lint.sh - run: poetry run scripts-dev/config-lint.sh
@ -52,6 +51,15 @@ jobs:
- run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'" - run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'"
- run: scripts-dev/check_schema_delta.py --force-colors - run: scripts-dev/check_schema_delta.py --force-colors
check-lockfile:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- run: .ci/scripts/check_lockfile.py
lint: lint:
uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v2" uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v2"
with: with:
@ -88,6 +96,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }} ref: ${{ github.event.pull_request.head.sha }}
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
poetry-version: "1.3.2"
extras: "all" extras: "all"
- run: poetry run scripts-dev/check_pydantic_models.py - run: poetry run scripts-dev/check_pydantic_models.py
@ -163,6 +172,7 @@ jobs:
- lint-pydantic - lint-pydantic
- check-sampleconfig - check-sampleconfig
- check-schema-delta - check-schema-delta
- check-lockfile
- lint-clippy - lint-clippy
- lint-rustfmt - lint-rustfmt
runs-on: ubuntu-latest runs-on: ubuntu-latest
@ -219,6 +229,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: ${{ matrix.job.python-version }} python-version: ${{ matrix.job.python-version }}
poetry-version: "1.3.2"
extras: ${{ matrix.job.extras }} extras: ${{ matrix.job.extras }}
- name: Await PostgreSQL - name: Await PostgreSQL
if: ${{ matrix.job.postgres-version }} if: ${{ matrix.job.postgres-version }}
@ -294,6 +305,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: '3.7' python-version: '3.7'
poetry-version: "1.3.2"
extras: "all test" extras: "all test"
- run: poetry run trial -j6 tests - run: poetry run trial -j6 tests
@ -328,6 +340,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
poetry-version: "1.3.2"
extras: ${{ matrix.extras }} extras: ${{ matrix.extras }}
- run: poetry run trial --jobs=2 tests - run: poetry run trial --jobs=2 tests
- name: Dump logs - name: Dump logs
@ -419,6 +432,7 @@ jobs:
- run: sudo apt-get -qq install xmlsec1 postgresql-client - run: sudo apt-get -qq install xmlsec1 postgresql-client
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
poetry-version: "1.3.2"
extras: "postgres" extras: "postgres"
- run: .ci/scripts/test_export_data_command.sh - run: .ci/scripts/test_export_data_command.sh
env: env:
@ -470,6 +484,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1 - uses: matrix-org/setup-python-poetry@v1
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
poetry-version: "1.3.2"
extras: "postgres" extras: "postgres"
- run: .ci/scripts/test_synapse_port_db.sh - run: .ci/scripts/test_synapse_port_db.sh
id: run_tester_script id: run_tester_script

View file

@ -148,7 +148,7 @@ jobs:
run: | run: |
set -x set -x
DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx
pipx install poetry==1.2.0 pipx install poetry==1.3.2
poetry remove -n twisted poetry remove -n twisted
poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk

View file

@ -0,0 +1 @@
Poetry 1.3.2 or higher is now required when `poetry install`ing from source.

View file

@ -31,12 +31,11 @@ case $(dpkg-architecture -q DEB_HOST_ARCH) in
esac esac
# Manually install Poetry and export a pip-compatible `requirements.txt` # Manually install Poetry and export a pip-compatible `requirements.txt`
# We need a Poetry pre-release as the export command is buggy in < 1.2
TEMP_VENV="$(mktemp -d)" TEMP_VENV="$(mktemp -d)"
python3 -m venv "$TEMP_VENV" python3 -m venv "$TEMP_VENV"
source "$TEMP_VENV/bin/activate" source "$TEMP_VENV/bin/activate"
pip install -U pip pip install -U pip
pip install poetry==1.2.0 pip install poetry==1.3.2
poetry export \ poetry export \
--extras all \ --extras all \
--extras test \ --extras test \

6
debian/changelog vendored
View file

@ -1,3 +1,9 @@
matrix-synapse-py3 (1.75.1) UNRELEASED; urgency=medium
* Use Poetry 1.3.2 to manage the bundled virtualenv included with this package.
-- Synapse Packaging team <packages@matrix.org> Tue, 17 Jan 2023 15:08:00 +0000
matrix-synapse-py3 (1.75.0) stable; urgency=medium matrix-synapse-py3 (1.75.0) stable; urgency=medium
* New Synapse release 1.75.0. * New Synapse release 1.75.0.

View file

@ -17,14 +17,8 @@
# Irritatingly, there is no blessed guide on how to distribute an application with its # Irritatingly, there is no blessed guide on how to distribute an application with its
# poetry-managed environment in a docker image. We have opted for # poetry-managed environment in a docker image. We have opted for
# `poetry export | pip install -r /dev/stdin`, but there are known bugs in # `poetry export | pip install -r /dev/stdin`, but beware: we have experienced bugs in
# in `poetry export` whose fixes (scheduled for poetry 1.2) have yet to be released. # in `poetry export` in the past.
# In case we get bitten by those bugs in the future, the recommendations here might
# be useful:
# https://github.com/python-poetry/poetry/discussions/1879#discussioncomment-216865
# https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker?answertab=scoredesc
ARG PYTHON_VERSION=3.9 ARG PYTHON_VERSION=3.9
@ -49,7 +43,7 @@ RUN \
# We install poetry in its own build stage to avoid its dependencies conflicting with # We install poetry in its own build stage to avoid its dependencies conflicting with
# synapse's dependencies. # synapse's dependencies.
RUN --mount=type=cache,target=/root/.cache/pip \ RUN --mount=type=cache,target=/root/.cache/pip \
pip install --user "poetry==1.2.0" pip install --user "poetry==1.3.2"
WORKDIR /synapse WORKDIR /synapse

View file

@ -67,7 +67,7 @@ pipx install poetry
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation) but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
for other installation methods. for other installation methods.
Synapse requires Poetry version 1.2.0 or later. Developing Synapse requires Poetry version 1.3.2 or later.
Next, open a terminal and install dependencies as follows: Next, open a terminal and install dependencies as follows:

View file

@ -2,6 +2,13 @@
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/). This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/).
# Installing
See the [contributing guide](contributing_guide.md#4-install-the-dependencies).
Developers should use Poetry 1.3.2 or higher. If you encounter problems related
to poetry, please [double-check your poetry version](#check-the-version-of-poetry-with-poetry---version).
# Background # Background
Synapse uses a variety of third-party Python packages to function as a homeserver. Synapse uses a variety of third-party Python packages to function as a homeserver.
@ -123,7 +130,7 @@ context of poetry's venv, without having to run `poetry shell` beforehand.
## ...reset my venv to the locked environment? ## ...reset my venv to the locked environment?
```shell ```shell
poetry install --extras all --remove-untracked poetry install --all-extras --sync
``` ```
## ...delete everything and start over from scratch? ## ...delete everything and start over from scratch?
@ -183,7 +190,6 @@ Either:
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else - manually update `pyproject.toml`; then `poetry lock --no-update`; or else
- `poetry add packagename`. See `poetry add --help`; note the `--dev`, - `poetry add packagename`. See `poetry add --help`; note the `--dev`,
`--extras` and `--optional` flags in particular. `--extras` and `--optional` flags in particular.
- **NB**: this specifies the new package with a version given by a "caret bound". This won't get forced to its lowest version in the old deps CI job: see [this TODO](https://github.com/matrix-org/synapse/blob/4e1374373857f2f7a911a31c50476342d9070681/.ci/scripts/test_old_deps.sh#L35-L39).
Include the updated `pyproject.toml` and `poetry.lock` files in your commit. Include the updated `pyproject.toml` and `poetry.lock` files in your commit.
@ -196,7 +202,7 @@ poetry remove packagename
``` ```
ought to do the trick. Alternatively, manually update `pyproject.toml` and ought to do the trick. Alternatively, manually update `pyproject.toml` and
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock` `poetry lock --no-update`. Include the updated `pyproject.toml` and `poetry.lock`
files in your commit. files in your commit.
## ...update the version range for an existing dependency? ## ...update the version range for an existing dependency?
@ -240,9 +246,6 @@ poetry export --extras all
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`. Be wary of bugs in `poetry export` and `pip install -r requirements.txt`.
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may
be required.
## ...build a test wheel? ## ...build a test wheel?
I usually use I usually use
@ -260,7 +263,7 @@ doesn't require poetry. (It's what we use in CI too). However, you could try
## Check the version of poetry with `poetry --version`. ## Check the version of poetry with `poetry --version`.
The minimum version of poetry supported by Synapse is 1.2. The minimum version of poetry supported by Synapse is 1.3.2.
It can also be useful to check the version of `poetry-core` in use. If you've It can also be useful to check the version of `poetry-core` in use. If you've
installed `poetry` with `pipx`, try `pipx runpip poetry list | grep installed `poetry` with `pipx`, try `pipx runpip poetry list | grep

View file

@ -100,6 +100,13 @@ and vice versa.
Once all workers are upgraded to v1.76 (or downgraded to v1.75), account data Once all workers are upgraded to v1.76 (or downgraded to v1.75), account data
and device replication will resume as normal. and device replication will resume as normal.
## Minimum version of Poetry is now 1.3.2
The minimum supported version of Poetry is now 1.3.2 (previously 1.2.0, [since
Synapse 1.67](#upgrading-to-v1670)). If you have used `poetry install` to
install Synapse from a source checkout, you should upgrade poetry: see its
[installation instructions](https://python-poetry.org/docs/#installation).
For all other installation methods, no acction is required.
# Upgrading to v1.74.0 # Upgrading to v1.74.0

3323
poetry.lock generated

File diff suppressed because it is too large Load diff