Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)

), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at.

---------

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>

Revert "Federation outbound proxy (#15773)"

This reverts commit b07b14b4.

Co-authored-by: reivilibre <oliverw@matrix.org>

Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.

The original code is from @erikjohnston's branches which I've gotten in-shape to merge.

Allow for the configuration of max request retries and min/max retry delays in the matrix federation client (#15783)

Revert "Allow for the configuration of max request retries and min/max retry delays in the matrix federation client (#12504)"

This reverts commit d84e6614.

Allow for the configuration of max request retries and min/max retry delays in the matrix federation client (#12504)


Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>

Implement stable support for MSC3882 to allow an existing device/session to generate a login token for use on a new device/session (#15388)

Implements stable support for MSC3882; this involves updating Synapse's support to
match the MSC / the spec says.

Continue to support the unstable version to allow clients to transition.

Adds a new configuration setting to connect to Redis via a Unix
socket instead of over TCP. Disabled by default.

* Add SSL options to redis config

* fix lint issues

* Add documentation and changelog file

* add missing . at the end of the changelog

* Move client context factory to new file

* Rename ssl to tls and fix typo

* fix lint issues

* Added when redis attributes were added

* Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master.

* Fix typo in drive by.

* Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place)

* Several updates:

1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future.
2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map.
3. Clean up old comments/commented out code.
4. Adjust unit tests to match with new code.
5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template.

* Initial Docs upload

* Changelog

* Missed some commented out code that can go now

* Remove TODO comment that no longer holds true.

* Fix links in docs

* More docs

* Remove debug logging

* Apply suggestions from code review

Co-authored-by: reivilibre <olivier@librepush.net>

* Apply suggestions from code review

Co-authored-by: reivilibre <olivier@librepush.net>

* Update version to latest, include completeish before/after examples in upgrade notes.

* Fix up and docs too

---------

Co-authored-by: reivilibre <olivier@librepush.net>

This stops media (and thumbnails) from being accessed from the
listed domains. It does not delete any already locally cached media,
but will prevent accessing it.

Note that admin APIs are unaffected by this change.

This is largely based off the stats and user directory updater code.

Signed-off-by: Sean Quah <seanq@matrix.org>

This is to discourage timing based profiling on the push gateways.

Add a note to the config documentation that the 'delete_stale_devices_after' job always runs on the main process (#15452)

Signed-off-by: Warren Bailey <warren@warrenbailey.net>

* doc: fix account login requests ratelimit defaults typo

Signed-off-by: td <nirvejayesh@gmail.com>

* chore: changelog.d file

---------

Signed-off-by: td <nirvejayesh@gmail.com>

* Correct documentation about registration_shared_secret_path

* Create 15168.doc

* Update changelog.d/15168.doc

---------

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* fix a typo in registration_shared_secret_path docs
Signed-off-by: Filip Rutar <filip.rutar@gmail.com>

* changelog

Note that this is only used for key-value store (cached values)
and not for the pub/sub replication used by Synapse.

* Change Documentation to have v10 as default room version

* Change Default Room version to 10

* Add changelog entry for default room version swap

* Add changelog entry for v10 default room version in docs

* Clarify doc changelog entry

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Improve Documentation changes.

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Changelog entry to have correct format

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Spec Version to 1.5

* Only need 1 changelog.

* Fix test.

* Update "Changed in" line

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>

Fixes #13852

Closes: #13643

Fixes: #8780

* Add `tag` to `listeners` documentation

* newsfile

* Add missing worker settings to shared configuration

* newsfile

* update docs after review

* more update for doc

* This -> These

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Fix stale external links

* Fix some internal links

* Fix URLs without trailing / where needed

* Fix more links

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Reapply docs/openid.md fix after conflict

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

PKCE can protect against certain attacks and is enabled by default. Support
can be controlled manually by setting the pkce_method of each oidc_providers
entry to 'auto' (default), 'always', or 'never'.

This is required by Twitter OAuth 2.0 support.

OpenID specifies the format of the user info endpoint and some
OAuth 2.0 IdPs do not follow it, e.g. NextCloud and Twitter.

This adds subject_template and picture_template options to the
default mapping provider for more flexibility in matching those user
info responses.

* Move `email` to server in config file documentation

* changelog

Fix `target_memory_usage` being used in the description for the actual `cache_autotune` sub-option `target_cache_memory_usage`.

Signed-off-by: Jeremy Kescher <jeremy@kescher.at>

Signed-off-by: Jeremy Kescher <jeremy@kescher.at>

* Declare new config

* Parse new config

* Read new config

* Don't use trial/our TestCase where it's not needed

Before:

```
$ time trial tests/events/test_utils.py > /dev/null

real	0m2.277s
user	0m2.186s
sys	0m0.083s
```

After:
```
$ time trial tests/events/test_utils.py > /dev/null

real	0m0.566s
user	0m0.508s
sys	0m0.056s
```

* Helper to upsert to event fields

without exceeding size limits.

* Use helper when adding invite/knock state

Now that we allow admins to include events in prejoin room state with
arbitrary state keys, be a good Matrix citizen and ensure they don't
accidentally create an oversized event.

* Changelog

* Move StateFilter tests

should have done this in #14668

* Add extra methods to StateFilter

* Use StateFilter

* Ensure test file enforces typed defs; alphabetise

* Workaround surprising get_current_state_ids

* Whoops, fix mypy

* Fix push.enabled config documentation

* Create 14619.doc

* Update 14619.doc

* Fix one typo on line 3700(and apparently do something to other lines, no idea)

* Update config_documentation.md with more information about how federation_senders and pushers settings can be handled.

Specifically, that the instance map style of config does not require the special other variables that enable and disable functionality and that a single worker CAN be added to the map not only just two or more.

* Extra line here for consistency and appearance.

* Add link to sygnal repo.

* Add deprecation notice to workers.md and point to the newer alternative method of defining this functionality.

* Changelog

* Correct version number of Synapse the deprecation is happening in.

* Update quiet deprecation with simple notice and suggestion.

* Add initial option

* changelog

* Some more linting

This commit adds support for handling a provided avatar picture URL
when logging in via SSO.

Signed-off-by: Ashish Kumar <ashfame@users.noreply.github.com>

Fixes #9357.

Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)

* Fix missing SSL support in worker endpoints.

* Add changelog

* SSL for Replication endpoint

* Remove unit test change

* Refactor listener creation to reduce duplicated code

* Fix the logger message

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add config documentation for new TLS option

Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>