Commits · 5561b94fefef1f87e6ecb1de10399c48631430db · TeDomum / matrix-media-repo

Mar 13, 2021
- Add a verify mode to imports · 5561b94f
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/276
  5561b94f
- Improve GDPR imports: end when finished, handle duplicate media · 367ed7e0
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/274 Fixes https://github.com/turt2live/matrix-media-repo/issues/273
  367ed7e0
Aug 12, 2020

Use a slim memory profile for imports done with gdpr_import · c260888d

Travis Ralston authored 4 years ago

Instead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information.

On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.

c260888d

Aug 06, 2020
- Add an offline import tool for Synapse · 97e36a23
  Travis Ralston authored 4 years ago
  
  97e36a23
Apr 24, 2020

Allow skipping user check in upload_controller.StoreDirect · 908c42bd

Jason Robinson authored 4 years ago

The new parameter `filterUserDuplicates` is set to true
everywhere else except in the import_synapse call. This
ensures direct duplicates from the same user will get
stored in an import from Synapse situation when migrating
to the mediarepo.

Closes #241

908c42bd

Mar 26, 2020
- Move stop and import channel update closures into more relevant places · f110f849
  Travis Ralston authored 5 years ago
  
  f110f849
- Dump and close streams; Close channels · 0ad91835
  Travis Ralston authored 5 years ago
  
  This should help memory some more. See https://github.com/turt2live/matrix-media-repo/issues/237 Theory is that by dumping the streams we'll relieve any bytes held on by the S3 libraries.
  0ad91835
Mar 07, 2020
- Experimental support for an IPFS datastore · 783c42ed
  Travis Ralston authored 5 years ago
  
  783c42ed
Jan 05, 2020
- Add binaries for easier exports/imports · 213675f5
  Travis Ralston authored 5 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/215
  213675f5
Jan 03, 2020
- Fix contexts in the import controller as well · e3165605
  Travis Ralston authored 5 years ago
  
  See d0c83eb4
  e3165605
Dec 28, 2019
- Switch to using the contextual config everywhere · 6dffc90d
  Travis Ralston authored 5 years ago
  
  Part of https://github.com/turt2live/matrix-media-repo/issues/209
  6dffc90d
- Replace entire context flow through app · fb83c63c
  Travis Ralston authored 5 years ago
  
  Part of https://github.com/turt2live/matrix-media-repo/issues/209
  fb83c63c
Dec 27, 2019
- Track the user ID for exports/imports · e39b865c
  Travis Ralston authored 5 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/212
  e39b865c
Dec 25, 2019
- Support importing from a previously exported dataset · ff7f22a1
  Travis Ralston authored 5 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/208
  ff7f22a1