matrix-media-repo

Designed for environments with multiple homeservers, matrix-media-repo de-duplicates all media automatically, including remote content. Environments with only one homeserver can still make use of the de-duplication and performance of matrix-media-repo.

Installing

Assuming Go 1.12+ is already installed on your PATH:

# Get it
git clone https://github.com/turt2live/matrix-media-repo
cd matrix-media-repo

# Build it
./build.sh

# Configure it (edit media-repo.yaml to meet your needs)
cp config.sample.yaml media-repo.yaml

# Run it
bin/media_repo

Deployment

This is intended to run behind a load balancer and beside your homeserver deployments. Assuming your load balancer handles SSL termination, a sample nginx config would be:

# Federation / Client-server API
# Both need to be reverse proxied, so if your federation and client-server API endpoints are on
# different `server` blocks, you will need to configure that.
server {
  listen 443 ssl;
  listen [::]:443 ssl;

  # SSL options not shown - ensure the certificates are valid for your homeserver deployment.

  # Redirect all traffic by default to the homeserver
  location /_matrix {
      proxy_read_timeout 60s;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_pass http://localhost:8008; # Point this towards your homeserver
  }
  
  # Redirect all media endpoints to the media-repo
  location /_matrix/media {
      proxy_read_timeout 60s;
      proxy_set_header Host $host; # Make sure this matches your homeserver in media-repo.yaml
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_pass http://localhost:8000; # Point this towards media-repo
  }
}

Your synapse listener configuration would look something like this:

listeners:
  - port: 8008
    bind_addresses: ['127.0.0.1']
    type: http
    tls: false
    x_forwarded: true
    resources:
      - names: [client]
        compress: true
      - names: [federation]
        compress: false

After importing your media, setting enable_media_repo: false in your Synapse configuration will disable the media repository.

Importing media from synapse

Media is imported by connecting to your synapse database and downloading all the content from the homeserver. This is so you have a backup of the media repository still with synapse. Do not point traffic at the media repo until after the import is complete.

Note: the database options provided on the command line are for the Synapse database. The media repo will use the connection string in the media-repo.yaml config when trying to store the Synapse media.

1. Build the media repo (as stated above)
2. Configure the media-repo.yaml
3. Run bin/import_synapse. The usage is below.

   Usage of ./bin/import_synapse:
     -baseUrl string
           The base URL to access your homeserver with (default "http://localhost:8008")
     -dbHost string
           The IP or hostname of the postgresql server with the synapse database (default "localhost")
     -dbName string
           The name of the synapse database (default "synapse")
     -dbPassword string
           The password to authorize the postgres user. Can be omitted to be prompted when run
     -dbPort int
           The port to access postgres on (default 5432)
     -dbUsername string
           The username to access postgres with (default "synapse")
     -serverName string
           The name of your homeserver (eg: matrix.org) (default "localhost")

   Assuming the media repository, postgres database, and synapse are all on the same host, the command to run would look something like: bin/import_synapse -serverName myserver.com -dbUsername my_database_user -dbName synapse
4. Wait for the import to complete. The script will automatically deduplicate media.
5. Point traffic to the media repository