Require poetry>=1.3.2 (#14860)

* Upgrade to new lockfile format

Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write.

Cheat sheet:

```
poetry --version
poetry show > scratch/before
pipx upgrade poetry
poetry --version
poetry show > scratch/after
diff scratch{before,after} && echo "no change!"
```

* Use Poetry 1.3.2 when reading or writing lockfile

* Remove unneeded(?) poetry dep for cibuildwheel

* Update docs

* Remove redundant call to setup-python

* Remove outdated comments related to Poetry 1.x

* Remove outdated docs line

was fixed in #13082

* Minor improvements to poetry cheat sheet

* Invoke setup-python-poetry with explicit version

Not sure about this. It's hardcoding versions everywhere.

* Changelog

* Check the lockfile is version 2.0

Might one day incorporate other checks like #14742

* Typo fixes, thanks Sean

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
This commit is contained in:
David Robertson 2023-01-17 18:04:44 +00:00 committed by GitHub
parent f820740b7d
commit b88cfe6d41
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 1739 additions and 1684 deletions

23
.ci/scripts/check_lockfile.py Executable file
View file

@ -0,0 +1,23 @@
#! /usr/bin/env python
import sys
if sys.version_info < (3, 11):
raise RuntimeError("Requires at least Python 3.11, to import tomllib")
import tomllib
with open("poetry.lock", "rb") as f:
lockfile = tomllib.load(f)
try:
lock_version = lockfile["metadata"]["lock-version"]
assert lock_version == "2.0"
except Exception:
print(
"""\
Lockfile is not version 2.0. You probably need to upgrade poetry on your local box
and re-run `poetry lock --no-update`. See the Poetry cheat sheet at
https://matrix-org.github.io/synapse/develop/development/dependencies.html
"""
)
raise

View file

@ -53,7 +53,7 @@ with open('pyproject.toml', 'w') as f:
"
python3 -c "$REMOVE_DEV_DEPENDENCIES"
pip install poetry==1.2.0
pip install poetry==1.3.2
poetry lock
echo "::group::Patched pyproject.toml"

View file

@ -37,7 +37,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: "3.x"
poetry-version: "1.2.0"
poetry-version: "1.3.2"
extras: "all"
# Dump installed versions for debugging.
- run: poetry run pip list > before.txt

View file

@ -127,7 +127,7 @@ jobs:
python-version: "3.x"
- name: Install cibuildwheel
run: python -m pip install cibuildwheel==2.9.0 poetry==1.2.0
run: python -m pip install cibuildwheel==2.9.0
- name: Set up QEMU to emulate aarch64
if: matrix.arch == 'aarch64'

View file

@ -33,11 +33,10 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: "3.x"
poetry-version: "1.3.2"
extras: "all"
- run: poetry run scripts-dev/generate_sample_config.sh --check
- run: poetry run scripts-dev/config-lint.sh
@ -52,6 +51,15 @@ jobs:
- run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'"
- run: scripts-dev/check_schema_delta.py --force-colors
check-lockfile:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- run: .ci/scripts/check_lockfile.py
lint:
uses: "matrix-org/backend-meta/.github/workflows/python-poetry-ci.yml@v2"
with:
@ -88,6 +96,7 @@ jobs:
ref: ${{ github.event.pull_request.head.sha }}
- uses: matrix-org/setup-python-poetry@v1
with:
poetry-version: "1.3.2"
extras: "all"
- run: poetry run scripts-dev/check_pydantic_models.py
@ -163,6 +172,7 @@ jobs:
- lint-pydantic
- check-sampleconfig
- check-schema-delta
- check-lockfile
- lint-clippy
- lint-rustfmt
runs-on: ubuntu-latest
@ -219,6 +229,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: ${{ matrix.job.python-version }}
poetry-version: "1.3.2"
extras: ${{ matrix.job.extras }}
- name: Await PostgreSQL
if: ${{ matrix.job.postgres-version }}
@ -294,6 +305,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: '3.7'
poetry-version: "1.3.2"
extras: "all test"
- run: poetry run trial -j6 tests
@ -328,6 +340,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: ${{ matrix.python-version }}
poetry-version: "1.3.2"
extras: ${{ matrix.extras }}
- run: poetry run trial --jobs=2 tests
- name: Dump logs
@ -419,6 +432,7 @@ jobs:
- run: sudo apt-get -qq install xmlsec1 postgresql-client
- uses: matrix-org/setup-python-poetry@v1
with:
poetry-version: "1.3.2"
extras: "postgres"
- run: .ci/scripts/test_export_data_command.sh
env:
@ -470,6 +484,7 @@ jobs:
- uses: matrix-org/setup-python-poetry@v1
with:
python-version: ${{ matrix.python-version }}
poetry-version: "1.3.2"
extras: "postgres"
- run: .ci/scripts/test_synapse_port_db.sh
id: run_tester_script

View file

@ -148,7 +148,7 @@ jobs:
run: |
set -x
DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqq python3 pipx
pipx install poetry==1.2.0
pipx install poetry==1.3.2
poetry remove -n twisted
poetry add -n --extras tls git+https://github.com/twisted/twisted.git#trunk

View file

@ -0,0 +1 @@
Poetry 1.3.2 or higher is now required when `poetry install`ing from source.

View file

@ -31,12 +31,11 @@ case $(dpkg-architecture -q DEB_HOST_ARCH) in
esac
# Manually install Poetry and export a pip-compatible `requirements.txt`
# We need a Poetry pre-release as the export command is buggy in < 1.2
TEMP_VENV="$(mktemp -d)"
python3 -m venv "$TEMP_VENV"
source "$TEMP_VENV/bin/activate"
pip install -U pip
pip install poetry==1.2.0
pip install poetry==1.3.2
poetry export \
--extras all \
--extras test \

6
debian/changelog vendored
View file

@ -1,3 +1,9 @@
matrix-synapse-py3 (1.75.1) UNRELEASED; urgency=medium
* Use Poetry 1.3.2 to manage the bundled virtualenv included with this package.
-- Synapse Packaging team <packages@matrix.org> Tue, 17 Jan 2023 15:08:00 +0000
matrix-synapse-py3 (1.75.0) stable; urgency=medium
* New Synapse release 1.75.0.

View file

@ -17,14 +17,8 @@
# Irritatingly, there is no blessed guide on how to distribute an application with its
# poetry-managed environment in a docker image. We have opted for
# `poetry export | pip install -r /dev/stdin`, but there are known bugs in
# in `poetry export` whose fixes (scheduled for poetry 1.2) have yet to be released.
# In case we get bitten by those bugs in the future, the recommendations here might
# be useful:
# https://github.com/python-poetry/poetry/discussions/1879#discussioncomment-216865
# https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker?answertab=scoredesc
# `poetry export | pip install -r /dev/stdin`, but beware: we have experienced bugs in
# in `poetry export` in the past.
ARG PYTHON_VERSION=3.9
@ -49,7 +43,7 @@ RUN \
# We install poetry in its own build stage to avoid its dependencies conflicting with
# synapse's dependencies.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --user "poetry==1.2.0"
pip install --user "poetry==1.3.2"
WORKDIR /synapse

View file

@ -67,7 +67,7 @@ pipx install poetry
but see poetry's [installation instructions](https://python-poetry.org/docs/#installation)
for other installation methods.
Synapse requires Poetry version 1.2.0 or later.
Developing Synapse requires Poetry version 1.3.2 or later.
Next, open a terminal and install dependencies as follows:

View file

@ -2,6 +2,13 @@
This is a quick cheat sheet for developers on how to use [`poetry`](https://python-poetry.org/).
# Installing
See the [contributing guide](contributing_guide.md#4-install-the-dependencies).
Developers should use Poetry 1.3.2 or higher. If you encounter problems related
to poetry, please [double-check your poetry version](#check-the-version-of-poetry-with-poetry---version).
# Background
Synapse uses a variety of third-party Python packages to function as a homeserver.
@ -123,7 +130,7 @@ context of poetry's venv, without having to run `poetry shell` beforehand.
## ...reset my venv to the locked environment?
```shell
poetry install --extras all --remove-untracked
poetry install --all-extras --sync
```
## ...delete everything and start over from scratch?
@ -183,7 +190,6 @@ Either:
- manually update `pyproject.toml`; then `poetry lock --no-update`; or else
- `poetry add packagename`. See `poetry add --help`; note the `--dev`,
`--extras` and `--optional` flags in particular.
- **NB**: this specifies the new package with a version given by a "caret bound". This won't get forced to its lowest version in the old deps CI job: see [this TODO](https://github.com/matrix-org/synapse/blob/4e1374373857f2f7a911a31c50476342d9070681/.ci/scripts/test_old_deps.sh#L35-L39).
Include the updated `pyproject.toml` and `poetry.lock` files in your commit.
@ -196,7 +202,7 @@ poetry remove packagename
```
ought to do the trick. Alternatively, manually update `pyproject.toml` and
`poetry lock --no-update`. Include the updated `pyproject.toml` and poetry.lock`
`poetry lock --no-update`. Include the updated `pyproject.toml` and `poetry.lock`
files in your commit.
## ...update the version range for an existing dependency?
@ -240,9 +246,6 @@ poetry export --extras all
Be wary of bugs in `poetry export` and `pip install -r requirements.txt`.
Note: `poetry export` will be made a plugin in Poetry 1.2. Additional config may
be required.
## ...build a test wheel?
I usually use
@ -260,7 +263,7 @@ doesn't require poetry. (It's what we use in CI too). However, you could try
## Check the version of poetry with `poetry --version`.
The minimum version of poetry supported by Synapse is 1.2.
The minimum version of poetry supported by Synapse is 1.3.2.
It can also be useful to check the version of `poetry-core` in use. If you've
installed `poetry` with `pipx`, try `pipx runpip poetry list | grep

View file

@ -100,6 +100,13 @@ and vice versa.
Once all workers are upgraded to v1.76 (or downgraded to v1.75), account data
and device replication will resume as normal.
## Minimum version of Poetry is now 1.3.2
The minimum supported version of Poetry is now 1.3.2 (previously 1.2.0, [since
Synapse 1.67](#upgrading-to-v1670)). If you have used `poetry install` to
install Synapse from a source checkout, you should upgrade poetry: see its
[installation instructions](https://python-poetry.org/docs/#installation).
For all other installation methods, no acction is required.
# Upgrading to v1.74.0

3323
poetry.lock generated

File diff suppressed because it is too large Load diff