-
Travis Ralston authored
Instead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information. On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.
Travis Ralston authoredInstead of buffering all the export files into memory (which could be hundreds of gb) we buffer them in one at a time and hope that they don't contain extra information. On a 171mb export, this change used 45mb of memory at most. Previously the task would have required about 1gb. For larger (200gb+ exports) this is obviously better.