Commits · cda49dbfccb9fa5859056d07f104466e3d786d71 · TeDomum / matrix-media-repo

Aug 02, 2023
- Fix stall on shutdown when the config was reloaded 2+ times · 902eaa35
  Travis Ralston authored 1 year ago
  
  902eaa35
- Centralize image encoding for future experiments with thumbnails · b9ffed61
  Travis Ralston authored 1 year ago
  
  b9ffed61
Aug 01, 2023
- Remove more dead code to make it easier to see changes · b5e4c88a
  Travis Ralston authored 1 year ago
  
  b5e4c88a
- Fix unfinished task scheduling · ddbdd424
  Travis Ralston authored 1 year ago
  
  ddbdd424
Jul 17, 2023
- Small misc cleanup · 65308fa7
  Travis Ralston authored 1 year ago
  
  65308fa7
Jul 16, 2023
- Improve asset file compression/on-disk size · a75aab5c
  Travis Ralston authored 1 year ago
  
  a75aab5c
- Clean up assets a bit better · ea8c1fe6
  Travis Ralston authored 1 year ago
  
  ea8c1fe6
- Small script updates · 27de3a27
  Travis Ralston authored 1 year ago
  
  27de3a27
- Make s3 consistency check better · f23d6b2c
  Travis Ralston authored 1 year ago
  
  f23d6b2c
- Use new archival system in gdpr_import · 6ac4aee4
  Travis Ralston authored 1 year ago
  
  6ac4aee4
Jul 15, 2023
- Use new archival system in gdpr_export · 6933fc44
  Travis Ralston authored 1 year ago
  
  6933fc44
- Rewrite archiver for export_synapse_for_import · 24730ec1
  Travis Ralston authored 1 year ago
  
  24730ec1
Jun 12, 2023
- Remove excessive ID generator checks · a1f23e68
  Travis Ralston authored 1 year ago
  
  a1f23e68
- Use new upload pipeline in synapse import · 892f4ba2
  Travis Ralston authored 1 year ago
  
  For https://github.com/turt2live/matrix-media-repo/issues/415
  892f4ba2
Jun 04, 2023

Move downloads over to download pipeline · 90abe06e

Travis Ralston authored 1 year ago

For https://github.com/turt2live/matrix-media-repo/issues/409
For https://github.com/turt2live/matrix-media-repo/issues/412

90abe06e

May 28, 2023
- Re-route direct uploads through new pipeline · 987f9cc0
  Travis Ralston authored 1 year ago
  
  For https://github.com/turt2live/matrix-media-repo/issues/411
  987f9cc0
May 22, 2023
- Start cutover to new datastores structure · a73876dc
  Travis Ralston authored 1 year ago
  
  a73876dc
May 13, 2023
- Move snowflake stuff in-process · c60269a5
  Travis Ralston authored 1 year ago
  
  c60269a5
Mar 10, 2023
- Readd io import in plugin_antispam_ocr · 4728b8b0
  Charlotte authored 2 years ago
  
  Fixes broken build
  Unverified
  
  4728b8b0
Mar 06, 2023
- Replace deprecated ioutil usage · f32bbbf6
  Travis Ralston authored 2 years ago
  
  f32bbbf6
Feb 15, 2023
- Replace http routing library & update endpoints to match · ab8b9ace
  Travis Ralston authored 2 years ago
  
  ab8b9ace
Feb 12, 2023
- Use unique ID generation *everywhere* · 2a2acea5
  Travis Ralston authored 3 years ago
  
  2a2acea5
- Initial clustering support: ID generation service · b22b9e22
  Travis Ralston authored 2 years ago
  
  b22b9e22
- Remove IPFS · 33aa8cad
  Travis Ralston authored 2 years ago
  
  33aa8cad
- Actually, just remove broken Complement testing · 910b7157
  Travis Ralston authored 2 years ago
  
  910b7157
- Remove unused performance testing packages · e121f22e
  Travis Ralston authored 2 years ago
  
  e121f22e
Mar 20, 2022
- Increase efficiency of consistency check · fc70e690
  Travis Ralston authored 3 years ago
  
  fc70e690
Mar 19, 2022
- Add an "s3 consistency check" runner to find excess uploads · 15b983a7
  Travis Ralston authored 3 years ago
  
  15b983a7
Mar 11, 2022
- Add support for changing the log level · 4f005d73
  Travis Ralston authored 3 years ago
  
  4f005d73
Feb 11, 2022
- Complement fix round 1 · a40ddbf4
  Travis Ralston authored 3 years ago
  
  a40ddbf4
Apr 16, 2021
- Add structured logging support · d45d8b1f
  Travis Ralston authored 3 years ago
  
  d45d8b1f
- Disable log colours by default · e9f82230
  Travis Ralston authored 3 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/323
  e9f82230
Mar 18, 2021
- Add optional Sentry support for error reporting · a040a3e5
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/312
  a040a3e5
Mar 13, 2021
- Add a verify mode to imports · 5561b94f
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/276
  5561b94f
- Improve GDPR imports: end when finished, handle duplicate media · 367ed7e0
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/274 Fixes https://github.com/turt2live/matrix-media-repo/issues/273
  367ed7e0
Jan 03, 2021
- Add antispam plugin support + OCR example · d38f198b
  Travis Ralston authored 4 years ago
  
  d38f198b
Aug 22, 2020
- Initial creation of a common export interface · d57593d3
  Travis Ralston authored 4 years ago
  
  d57593d3
Aug 17, 2020
- Add a flag to skip missing media · afc1ac8c
  Travis Ralston authored 4 years ago
  
  afc1ac8c
- Pull URL preview images for Synapse from the url cache · 32f5657b
  Travis Ralston authored 4 years ago
  
  Fixes https://github.com/turt2live/matrix-media-repo/issues/272
  32f5657b
Aug 12, 2020

Use a slim memory profile for imports done with gdpr_import · c260888d

Travis Ralston authored 4 years ago

Instead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information.

On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.

c260888d