Follow-up to #17652, https://github.com/element-hq/synapse/pull/17641,
https://github.com/element-hq/synapse/pull/17634,
https://github.com/element-hq/synapse/pull/17631 and
https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17512

This allows us to skip checking the database a lot of the time.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

When using timeline limit of 1 we end up fetching 2 events from the DB
purely to tell if the response was "limited" or not. Lets not do that.

Sliding Sync: Add comment to explain extra case where you can be invited -> banned -> unbanned (#17654)

Add comment to explain extra case where you can be
invited -> banned -> unbanned and we want to be able
to find the invite event.

Follow-up to https://github.com/element-hq/synapse/pull/17636#discussion_r1738993330

Get `bump_stamp` from [new sliding sync
tables](https://github.com/element-hq/synapse/pull/17512) which should
be faster (performance) than flipping through the latest events in the
room.

This reverts commit a3c49565.

This allows us to skip checking the database a lot of the time.

Speed up incremental sync by avoiding extra work. We first look at the
state delta changes and only fetch and calculate further derived things
if they have changed.

We should only look for bump stamps in joined rooms, otherwise we should
just use the membership stream ordering.

A couple of small performance improvements for sliding sync.

Instead of having a large cache of `room_id -> bool` about whether a
room is partially stated, replace with a "fetch rooms the user is which
are partially-stated". This is a lot faster as the set of partially
stated rooms at any point across the whole server is small, and so such
a query is fast.

The main issue with the bulk cache lookup is the CPU time looking all
the rooms up in the cache.

We ended up spending ~10% CPU creating a new dictionary and
`_RoomMembershipForUser`, so let's avoid creating new dicts and copying
by returning `newly_joined`, `newly_left` and `is_dm` as sets directly.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

This reverts commit f73c8444.

We should only look for bump stamps in joined rooms, otherwise we should
just use the membership stream ordering.

Follow-up to #17634, https://github.com/element-hq/synapse/pull/17631
and https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17512

Prevent duplicate tags being added to traces.

Noticed because we see these warnings in Jaeger:

<img width="462" alt="Screenshot 2024-09-03 at 2 34 05 PM"
src="https://github.com/user-attachments/assets/6fac12ed-0074-435b-9451-eccde7e7012a">

Broke in #17630

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

I thought ruff check would also format, but it doesn't.

This runs ruff format in CI and dev scripts. The first commit is just a
run of `ruff format .` in the root directory.

Based on https://github.com/element-hq/synapse/pull/17629

Utilizing the new sliding sync tables added in
https://github.com/element-hq/synapse/pull/17512

 for fast acquisition of
rooms for the user and filtering/sorting.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

This is to make it easier to reuse the logic when adding support for the
new tables

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

This reverts commit b4d0356e.

Follows on from #17512, other fixes include: #17633, #17634, #17635