Skip to content
Snippets Groups Projects
  • Travis Ralston's avatar
    c260888d
    Use a slim memory profile for imports done with gdpr_import · c260888d
    Travis Ralston authored
    Instead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information.
    
    On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.
    c260888d
    History
    Use a slim memory profile for imports done with gdpr_import
    Travis Ralston authored
    Instead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information.
    
    On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.