Newer
Older
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
### Mandatory Configuration Change
Datastores are no longer managed by matrix-media-repo internally, meaning you MUST specify a datastore ID on each of your
configured datastores. If you're setting up matrix-media-repo for the first time then you can use whatever you want for
a datastore ID (though it's recommended to stick to alphanumeric strings). If you're *upgrading* to this version however,
you will need to pull the datastore IDs out of the matrix-media-repo and add them to your configuration.
**For safety, the datastores table is *not* deleted from the database in this upgrade. A future version may drop the table,
however.**
#### Getting existing datastore IDs
**Before upgrading**, you can get your datastore IDs fairly easily. The best way might be to look at the startup log of
your media repo:
```text
INFO[2023-05-21 20:58:45.116 Z] Datastores:
INFO[2023-05-21 20:58:45.116 Z] file (e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336): /mnt/mmr-store/location4
INFO[2023-05-21 20:58:45.117 Z] s3 (7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992): s3://redacted.r2.cloudflarestorage.com/redacted
```
This way, you're able to correlate locations to IDs. For example, the `file` datastore configured to put media at
`/mnt/mmr-store/location4` has ID `e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336`. Add this as
`id: "e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336"` in your media repo config file.
Alternatively, you can use the admin API to get your datastores:
```text
curl -s -X GET -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://example.org/_matrix/media/unstable/admin/datastores
{
"e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336": {
"type": "file",
"uri": "/mnt/mmr-store/location4"
},
"7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992": {
"type": "s3",
"uri": "s3://redacted.r2.cloudflarestorage.com/redacted"
}
}
```
The returned object is keyed by ID over the API.
In either case, take the ID and add it to the associated datastore in your config, similar to the following:
```yaml
# Your specific configuration may be different
datastores:
- type: file
id: "e9ce13bbb062383ce1bcee76414058668877f2d51635810652335374336" ## ADD THIS
enabled: true
forKinds: ["archives"]
opts:
path: "/mnt/mmr-store/location4"
- type: s3
id: "7669e2fb8ccaa0801e4255a417ad20884f76b8611659655069202644992" ## ADD THIS
enabled: true
forKinds: ["all"]
opts:
ssl: true
tempPath: "/mnt/mmr-store/s3-staging"
endpoint: redacted.r2.cloudflarestorage.com
accessKeyId: "redacted"
accessSecret: "redacted"
bucketName: "redacted"
```
**Note**: If matrix-media-repo detects that a datastore ID is used but not referenced in the config then it will refuse
to start.
This new configuration style additionally allows for out-of-band datastore transfers. If you move all your data to a new
path/server, for example, then you can simply update the path in the config for that datastore.
### Deprecations
* The `GET /_matrix/media/unstable/local_copy/:server/:mediaId` (and `unstable/io.t2bot.media` variant) endpoint is deprecated and scheduled for removal. If you are using this endpoint, please comment on [this issue](https://github.com/turt2live/matrix-media-repo/issues/422) to explain your use case.
### Added
* Added a `federation.ignoredHosts` config option to block media from individual homeservers.
* Support for [MSC2246](https://github.com/matrix-org/matrix-spec-proposals/pull/2246) (async uploads) is added, with per-user quota limiting options.
* Support for [MSC4034](https://github.com/matrix-org/matrix-spec-proposals/pull/4034) (self-serve usage information) is added, alongside a new "maximum file count" quota limit.
### Removed
* IPFS support has been removed due to maintenance burden.
* **Mandatory configuration change**: You must add datastore IDs to your datastore configuration, as matrix-media-repo will no longer manage datastores for you.
* Datastores no longer use the `enabled` flag set on them. Use `forKinds: []` instead.
* Some admin endpoints for purging media, quarantining media, and background task information now require additional path components. See [docs/admin.md](./docs/admin.md) for more information.
Travis Ralston
committed
* Per-user upload quotas now do not allow users to exceed the maximum values, even by 1 byte. Previously, users could exceed the limits by a little bit.
* Connected homeservers must support at least Matrix 1.1 on the Client-Server API.
### Fixed
* URL previews now follow redirects properly.
* Overall memory usage is improved, particularly during media uploads.
* Note: If you use plugins then memory usage will still be somewhat high due to temporary caching of uploads.
* In version 1.3.0, IPFS will no longer be supported as a datastore. Please migrate your data if you are using the IPFS support.
### Added
* Added the `Cross-Origin-Resource-Policy: cross-origin` header to all downloads, as per [MSC3828](https://github.com/matrix-org/matrix-spec-proposals/pull/3828).
* Added metrics for tracking which S3 operations are performed against datastores.
### Changed
* Swap out the HEIF library for better support towards [ARM64 Docker Images](https://github.com/turt2live/matrix-media-repo/issues/365).
* The development environment now uses Synapse as a homeserver. Test accounts will need recreating.
* Improved error message when thumbnailer cannot determine image dimensions.
### Fixed
* Return default media attributes if none have been explicitly set.
## [1.2.12] - March 31, 2022
### Fixed
* Fixed a permissions check issue on the new statistics endpoint released in v1.2.11
### Added
* New config option to set user agent when requesting URL previews.
* Built-in early support for content ranges (being able to skip around in audio and video). This is only available if
caching is enabled.
* New config option for changing the log level.
* New (currently undocumented) binary `s3_consistency_check` to find objects in S3 which *might* not be referenced by
the media repo database. Note that this can include uploads in progress.
* Admin endpoint to GET users' usage statistics for a server.
### Removed
* Support for the in-memory cache has been removed. Redis or having no cache are now the only options.
* Support for the Redis config under `features` has been removed. It is now only available at the top level of the
config. See the sample config for more details.
### Fixed
* Fixed media being permanently lost when transferring to an (effectively) readonly S3 datastore.
* Fixed HEIF/HEIC thumbnailing. Note that this thumbnail type might cause increased memory usage.
* Ensure endpoints register in a stable way, making them predictably available.
* Reduced download hits to datastores when using Redis cache.
* Updated support for post-[MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) homeservers.
## [1.2.10] - December 23rd, 2021
### Deprecation notices
In a future version (likely the next), the in-memory cache support will be removed. Instead, please use the Redis
caching that is now supported properly by this release, or disable caching if not applicable for your deployment.
### Added
* Added support for setting the Redis database number.
### Fixed
* Fixed an issue with the Redis config not being recognized at the root level.
### Deprecation notices
In a future version (likely the next), the in-memory cache support will be removed. Instead, please use the Redis
caching that is now supported properly by this release, or disable caching if not applicable for your deployment.
### Added
* Added support for `HEAD` at the `/healthz` endpoint.
* Added `X-Content-Security-Policy: sandbox` in contexts where the normal CSP
header would be served. This is a limited, pre-standard form of CSP supported
by IE11, in order to have at least some mitigation of XSS attacks.
* Added support for the `org.matrix.msc2705.animated` query parameter.
* Added support for S3 storage classes (optional).
* Added support for listening on Matrix 1.1 endpoints (`/_matrix/media/v3/*`).
### Changed
* Support the Redis config at the root level of the config, promoting it to a proper feature.
### Fixed
* Improved performance of datastore selection when only one datastore is eligible to contain media.
## [1.2.8] - April 30th, 2021
### Fixed
* Fixed crashes when internal workers encounter panics.
This release includes a fix for [CVE-2021-29453](https://github.com/turt2live/matrix-media-repo/security/advisories/GHSA-j889-h476-hh9h).
Server administrators are recommended to upgrade as soon as possible. This issue is considered to be exploited in the wild
due to some deployments being affected unexpectedly.
### Added
* Added support for structured logging (JSON).
### Changed
* Turned color-coded logs off by default. This can be changed in the config.
### Fixed
* Fixed memory exhaustion when thumbnailing maliciously crafted images.
### Added
* Added ffmpeg and ImageMagick to Docker image to support specialized thumbnail types.
### Fixed
* Handle guest accounts properly. Previously they were still declined, though by coincidence.
### Added
* Added a `-verify` mode to imports to determine if large imports were successful.
* Added optional support for [Sentry](https://sentry.io/) (error reporting).
### Changed
* `Content-Disposition` of plain text files now defaults to `inline`.
### Fixed
* Fixed rich oEmbed URL previews (Twitter).
* Fixed orientation parsing for some thumbnails.
* Fixed file name being incorrect on the first download from remote servers.
* Fixed a download inefficiency where remote downloads could use extra bandwidth.
* Fixed a problem where secondary imports can never finish.
* Fixed imports not handling duplicate media IDs.
* Fixed some database connection errors not being handled correctly.
## [1.2.4] - March 5th, 2021
### Fixed
* Fixed build error for modern versions of Go, improving IPFS implementation.
### Added
* Introduced early plugin support (only for antispam for now).
* Includes a simple OCR plugin to help mitigate text-based image spam.
* Added an `X-Robots-Tag` header to help prevent indexing. Thanks @jellykells!
* Fixed crash when generating some thumbnails of audio.
* Fixed various artifact problems with APNG and GIF thumbnails. Thanks @Sorunome!
* Fixed a missing "unlimited size" check for thumbnails. Thanks @Sorunome!
### Fixed
* Generate JPEG thumbnails for JPEG for reduced file size. Thanks @Sorunome!
* Strip `charset` parameter off binary media for better compatibility with other homeservers.
### Added
* Added a new tool, `export_synapse_for_import`, which can be used to do an offline import from Synapse.
* After running this tool, use the `gdpr_import` tool to bring the export into the media repo.
* Added thumbnailing support for some audio waveforms (MP3, WAV, OGG, and FLAC).
* Added audio metadata (duration, etc) to the unstable `/info` endpoint. Aligns with [MSC2380](https://github.com/matrix-org/matrix-doc/pull/2380).
* Added simple thumbnailing for MP4 videos.
* Added an `asAttachment` query parameter to download requests per [MSC2702](https://github.com/matrix-org/matrix-doc/pull/2702).
### Fixed
* Fixed thumbnails for invalid JPEGs.
* Fixed incorrect metrics being published when using the Redis cache.
* Fixed errors generating thumbnails when bad EXIF headers were provided.
* Use `r0` instead of `v1` for federation requests. No changes should be needed to configurations or routing - it'll just work.
### Upgrade notes
**This release contains a database change which might take a while.** In order to support quotas, this
release tracks how much a user has uploaded, which might take a while to initially calculate. If you have
a large database (more than about 100k uploaded files), run the following steps before upgrading:
1. The PostgreSQL script described [here](https://github.com/turt2live/matrix-media-repo/blob/a8951b0562debb9f8ae3b6e517bfc3a84d2e627a/migrations/17_add_user_stats_table_up.sql).
This can be run while the server is running.
2. If you have no intention of using stats or quotas, you're done (the stats table will be inaccurate). If
you do plan on using either, run `INSERT INTO user_stats SELECT user_id, SUM(size_bytes) FROM media GROUP BY user_id;`
which may take a while.
3. Change the owner of the table and function to your media repo's postgresql user. For example, if your postgres
user is `media`, then run:
```sql
ALTER TABLE user_stats OWNER TO media;
ALTER FUNCTION track_update_user_media() OWNER TO media;
```
### Added
* Add webp image support. Thanks @Sorunome!
* Experimental support for Redis as a cache (in preparation for proper load balancing/HA support).
* Added oEmbed URL preview support.
* Added support for dynamic thumbnails.
* Added a way to prevent certain media from being quarantined (attributes API).
### Changed
* Remove deprecated support for restricting uploads to certain mime types.
* Remove deprecated support for `forUploads`.
* Clarified what `uploads.minBytes` is intended to be used for.
### Fixed
* GIFs now thumbnail correctly. Thanks @Sorunome!
* Fixed empty Content-Type header on retrieved remote media. Thanks @silkeh!
* Fixed various issues with IPv6 handling. Thanks @silkeh!
* Fixed high database usage for uploads when only one datastore is present.
* Fixed incorrect HTTP status codes for bad thumbnail requests.
* Fixed dimension checking on thumbnails.
* Fixed handling of URL previews for some encodings.
* Fixed `Cache-Control` headers being present on errors.
### Added
* Added options to cache access tokens for users. This prevents excessive calls to `/account/whoami` on your homeserver, particularly for appservices.
* [Documentation](https://github.com/turt2live/matrix-media-repo/blob/master/docs/contrib/delegation.md) on how to set up delegation with the media repo and Traefik. Thanks @derEisele!
### Changed
* Deprecated support for restricting uploads to certain mime types, due to inability to make it work correctly with encrypted media.
* Removed deprecated `storagePaths` config option. Please use datastores.
### Fixed
* Fixed federation with some homeserver setups (delegation with ports). Thanks @MatMaul!
* Fixed the Synapse import script to not skip duplicated media. Thanks @jaywink!
* Removed excessive calls to the database during upload.
### Fixed
* Fixed templates being corrupt in the Docker image.
* Fixed `REPO_CONFIG` environment variable not being respected for auxiliary binaries in the Docker image.
### Changed
* The Docker image now uses the migrations packed into the binary instead of the in-image ones.
* Reduced log spam when someone views an export.
## [1.1.1] - March 26th, 2020
### Added
* Added pprof endpoints for debugging performance. Only enabled with a `MEDIA_PPROF_SECRET_KEY` environment variable.
### Fixed
* Fixed a few very slow memory leaks when using S3 datastores.
* Added support for [MSC2448](https://github.com/matrix-org/matrix-doc/pull/2448).
* Added support for specifying a `region` to the S3 provider.
* Pass-through the `Accept-Language` header for URL previews, with options to set a default.
* Consistent inclusion of a charset for certain text `Content-Type`s.
* New metrics for the cache composition reality (`media_cache_num_live_bytes_used` and `media_cache_num_live_items`).
### Fixed
* Fixed thumbnails producing the wrong result.
* Fixed `expireAfterDays` for thumbnails potentially deleting media under some conditions.
* Fixed a bug where items could be double-counted (but not double-stored) in the cache.
* Fixed the cache metrics reporting inaccurate values.
* Fixed a general memory leak in the cache due to inaccurate counting of items in the cache.
* Updated the Grafana dashboard and moved it in-tree.
### Added
* Added support for a `forKinds: ["all"]` option on datastores.
### Fixed
* Fixed a bug with the cache where it would never expire old entries unless it was pressed for space.
* Fixed a bug with the cache where the minimum cache time trigger would not work.
* Fix a memory leak within the cache layers.
### Added
* Compile assets (templates and migrations) into the binary for ease of deployment.
* Added binaries to make exports and imports easier.
### Fixed
* Fix error message when an invalid access token is provided.
* Fixed imports not starting in 1.0.0-rc.2.
### Fixed
* Fixed exports not starting in 1.0.0-rc.1.
### Added
* First ever release of matrix-media-repo.
* Deduplicate media from all sources.
* Support downloads, thumbnails, URL previews, identicons.
* Support for GDPR-style media exports.
* Support for importing from a previous export (for transferring data between repos).
* Admin utilities for clearing up space and undesirable content.
* Built-in S3 (and S3-like) support.
* Animated thumbnail generation.
* Importing media from an existing Synapse homeserver.
* Support for multiple datastores/locations to store different kinds of media.
* Federation for acquiring remote media.
* Media identification ([MSC2380](https://github.com/matrix-org/matrix-doc/pull/2380)).
* Support for cloning media to the local homeserver.
* Various other features that would be expected like maximum/minimum size controls, rate limiting, etc. Check out the
sample config for a better idea of what else is possible.
[unreleased]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.13...HEAD
[1.2.13]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.12...v1.2.13
[1.2.12]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.11...v1.2.12
[1.2.11]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.10...v1.2.11
[1.2.10]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.9...v1.2.10
[1.2.9]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.8...v1.2.9
[1.2.8]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.7...v1.2.8
[1.2.6]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.6...v1.2.7
[1.2.6]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.5...v1.2.6
[1.2.5]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.4...v1.2.5
[1.2.4]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.3...v1.2.4
[1.2.3]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.2...v1.2.3
[1.2.2]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.1...v1.2.2
[1.2.1]: https://github.com/turt2live/matrix-media-repo/compare/v1.2.0...v1.2.1
[1.2.0]: https://github.com/turt2live/matrix-media-repo/compare/v1.1.3...v1.2.0
[1.1.3]: https://github.com/turt2live/matrix-media-repo/compare/v1.1.2...v1.1.3
[1.1.2]: https://github.com/turt2live/matrix-media-repo/compare/v1.1.1...v1.1.2
[1.1.1]: https://github.com/turt2live/matrix-media-repo/compare/v1.1.0...v1.1.1
[1.1.0]: https://github.com/turt2live/matrix-media-repo/compare/v1.0.2...v1.1.0
[1.0.2]: https://github.com/turt2live/matrix-media-repo/compare/v1.0.1...v1.0.2
[1.0.1]: https://github.com/turt2live/matrix-media-repo/compare/v1.0.0...v1.0.1
[1.0.0]: https://github.com/turt2live/matrix-media-repo/compare/v1.0.0-rc.2...v1.0.0
[1.0.0-rc.2]: https://github.com/turt2live/matrix-media-repo/compare/v1.0.0-rc.1...v1.0.0-rc.2
[1.0.0-rc.1]: https://github.com/turt2live/matrix-media-repo/releases/tag/v1.0.0-rc.1