FSA download checkpoints copy the whole file every 512 MiB (quadratic disk I/O on large files)

jhagberg commented

2026-09-01 14:56:07 +00:00

(Migrated from github.com)

Describe the bug

Follow-up to #173 (resume FSA downloads). checkpointDownload in frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx closes the writable stream every 512 MiB (FILE_SYSTEM_DOWNLOAD_CHECKPOINT_BYTES) and reopens it with createWritable({ keepExistingData: true }).

According to MDN and the File System Standard, keepExistingData: true means the existing file is first copied into the temporary swap file (.crswap) before writing continues. On macOS the swap file can be created copy-on-write, but on Windows and Linux ext4 it is a real copy (see this chromium-dev thread).

The disk work per file therefore grows with the square of the file size. With checkpoint size C and file size N, the total copying is roughly N² / (2C):

10 GiB file: about 100 GiB copied in total
50 GiB file: about 2.5 TiB copied in total

With FILE_SYSTEM_BATCH_CONCURRENCY = 2, two files can be doing this at the same time. The last chunk can also push bytesSinceCheckpoint over the limit just before the final close(), so a finished file is copied one extra time.

Measured

I ran a standalone page that repeats the same loop as checkpointDownload (write 64 MiB, close(), createWritable({ keepExistingData: true }), seek()) on Chrome 152, Ubuntu 24.04, ext4, local SSD, with a 1 GiB file and 15 checkpoints:

chunk	file size after write	write	`close()`	reopen + `seek()`
1	64 MiB	127 ms	478 ms	132 ms
4	256 MiB	187 ms	1387 ms	565 ms
8	512 MiB	202 ms	2360 ms	1207 ms
12	768 MiB	292 ms	3299 ms	1799 ms
15	960 MiB	247 ms	4363 ms	1950 ms
16	1024 MiB	249 ms	4253 ms	(final close)

The run took 61.0 s in total: 3.2 s writing data, 40.4 s in close() and 17.4 s reopening. /proc/diskstats counted 8351 MiB written for a 1024 MiB file. Writing the same 1 GiB through a single writable with one close() took 9.2 s (3.3 s of writes, 5.8 s for that one close()) and 1044 MiB of disk writes. Watching the folder during the run, the .crswap file grows from 0 to the full current size at every reopen. That growth is the copy.

On this machine both steps scale with the current file size: close() at about 4 ms per MiB, which fits Chrome reading the whole swap file for its after-write check before moving it into place, and reopen at about 2 ms per MiB for the copy. Extrapolated to the 512 MiB interval in the PR, a 10 GiB file spends roughly 11 minutes in checkpoints and writes about 105 GiB to disk. A 50 GiB file spends roughly 4.5 hours and writes about 2.5 TiB.

Steps to reproduce

Use Chrome on Linux (ext4) or Windows and a dataset with a large file (several GiB or more).
Start a "Download selected files to folder" download.
Watch disk I/O (for example iotop or Task Manager) and the size of the .crswap file in the target folder.
Every 512 MiB the .crswap file is recreated with the full size of the file so far, and download throughput drops as the file grows.

The numbers above come from a small HTML page (about 60 lines) that only uses the File System Access API. I can attach it if that helps.

Environment

Chrome or Chromium, Linux or Windows. Introduced by #173 (branch 149-resume-FSA-file-downloads).

Expected behavior

Total disk writes for one file stay proportional to the file size (for example bounded by about twice the file size).
Checkpoints still limit how much has to be re-downloaded after a crash, with a documented trade-off.
No checkpoint is taken when nextWriteOffset >= totalBytes.
Tests verifying the fix are added

Additional context

Possible approaches, from smallest change to largest:

Double the checkpoint interval after each checkpoint. Total copying is then bounded by about twice the file size, at the price of re-downloading at most half a file after a crash.
Stage the download in OPFS with a FileSystemSyncAccessHandle in a worker (in-place writes, cheap flush()), and copy into the chosen folder once when the file is complete. This needs quota handling and a worker, so it is a bigger change.

Cancelling with the button already commits the partial file through close(), so checkpoints only matter for crashes and tab closes.

## Describe the bug Follow-up to #173 (resume FSA downloads). `checkpointDownload` in `frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx` closes the writable stream every 512 MiB (`FILE_SYSTEM_DOWNLOAD_CHECKPOINT_BYTES`) and reopens it with `createWritable({ keepExistingData: true })`. According to [MDN](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createWritable) and the [File System Standard](https://fs.spec.whatwg.org/), `keepExistingData: true` means the existing file is first copied into the temporary swap file (`.crswap`) before writing continues. On macOS the swap file can be created copy-on-write, but on Windows and Linux ext4 it is a real copy (see [this chromium-dev thread](https://groups.google.com/a/chromium.org/g/chromium-dev/c/UyU-U-V0Jo4)). The disk work per file therefore grows with the square of the file size. With checkpoint size C and file size N, the total copying is roughly N² / (2C): - 10 GiB file: about 100 GiB copied in total - 50 GiB file: about 2.5 TiB copied in total With `FILE_SYSTEM_BATCH_CONCURRENCY = 2`, two files can be doing this at the same time. The last chunk can also push `bytesSinceCheckpoint` over the limit just before the final `close()`, so a finished file is copied one extra time. ### Measured I ran a standalone page that repeats the same loop as `checkpointDownload` (write 64 MiB, `close()`, `createWritable({ keepExistingData: true })`, `seek()`) on Chrome 152, Ubuntu 24.04, ext4, local SSD, with a 1 GiB file and 15 checkpoints: | chunk | file size after write | write | `close()` | reopen + `seek()` | |---|---|---|---|---| | 1 | 64 MiB | 127 ms | 478 ms | 132 ms | | 4 | 256 MiB | 187 ms | 1387 ms | 565 ms | | 8 | 512 MiB | 202 ms | 2360 ms | 1207 ms | | 12 | 768 MiB | 292 ms | 3299 ms | 1799 ms | | 15 | 960 MiB | 247 ms | 4363 ms | 1950 ms | | 16 | 1024 MiB | 249 ms | 4253 ms | (final close) | The run took 61.0 s in total: 3.2 s writing data, 40.4 s in `close()` and 17.4 s reopening. `/proc/diskstats` counted 8351 MiB written for a 1024 MiB file. Writing the same 1 GiB through a single writable with one `close()` took 9.2 s (3.3 s of writes, 5.8 s for that one `close()`) and 1044 MiB of disk writes. Watching the folder during the run, the `.crswap` file grows from 0 to the full current size at every reopen. That growth is the copy. On this machine both steps scale with the current file size: `close()` at about 4 ms per MiB, which fits Chrome reading the whole swap file for its after-write check before moving it into place, and reopen at about 2 ms per MiB for the copy. Extrapolated to the 512 MiB interval in the PR, a 10 GiB file spends roughly 11 minutes in checkpoints and writes about 105 GiB to disk. A 50 GiB file spends roughly 4.5 hours and writes about 2.5 TiB. ## Steps to reproduce 1. Use Chrome on Linux (ext4) or Windows and a dataset with a large file (several GiB or more). 2. Start a "Download selected files to folder" download. 3. Watch disk I/O (for example `iotop` or Task Manager) and the size of the `.crswap` file in the target folder. 4. Every 512 MiB the `.crswap` file is recreated with the full size of the file so far, and download throughput drops as the file grows. The numbers above come from a small HTML page (about 60 lines) that only uses the File System Access API. I can attach it if that helps. ### Environment Chrome or Chromium, Linux or Windows. Introduced by #173 (branch `149-resume-FSA-file-downloads`). ## Expected behavior - [ ] Total disk writes for one file stay proportional to the file size (for example bounded by about twice the file size). - [ ] Checkpoints still limit how much has to be re-downloaded after a crash, with a documented trade-off. - [ ] No checkpoint is taken when `nextWriteOffset >= totalBytes`. - [ ] Tests verifying the fix are added ## Additional context Possible approaches, from smallest change to largest: - Double the checkpoint interval after each checkpoint. Total copying is then bounded by about twice the file size, at the price of re-downloading at most half a file after a crash. - Stage the download in OPFS with a `FileSystemSyncAccessHandle` in a worker (in-place writes, cheap `flush()`), and copy into the chosen folder once when the file is complete. This needs quota handling and a worker, so it is a bigger change. Cancelling with the button already commits the partial file through `close()`, so checkpoints only matter for crashes and tab closes.

aaperis commented

2026-09-01 15:10:40 +00:00

(Migrated from github.com)

Thanks @jhagberg. Related to and will be closed together with #182.

jhagberg commented

2026-09-02 06:26:41 +00:00

(Migrated from github.com)

Thanks! I had a look at #182 and I am not sure it closes this one as written. If the time-based flush uses the same close() + createWritable({ keepExistingData: true }) mechanism, every flush copies the whole file so far into a new swap file and writes it back to disk, so on a 10 GiB file one flush would take about a minute on my machine. A time interval would then make this worse rather than better.

Would it be worth adding "total disk writes stay proportional to the file size" to the AC of #182, or is the plan to change the flush mechanism there? One option might be one part file per chunk (each closed once) that are concatenated into the final file at the end, which does not need move(). Staging in OPFS, where flush() is cheap, would be another.

Thanks! I had a look at #182 and I am not sure it closes this one as written. If the time-based flush uses the same `close()` + `createWritable({ keepExistingData: true })` mechanism, every flush copies the whole file so far into a new swap file and writes it back to disk, so on a 10 GiB file one flush would take about a minute on my machine. A time interval would then make this worse rather than better. Would it be worth adding "total disk writes stay proportional to the file size" to the AC of #182, or is the plan to change the flush mechanism there? One option might be one part file per chunk (each closed once) that are concatenated into the final file at the end, which does not need `move()`. Staging in OPFS, where `flush()` is cheap, would be another.

aaperis commented

2026-09-02 08:52:06 +00:00

(Migrated from github.com)

Thanks! I had a look at #182 and I am not sure it closes this one as written. If the time-based flush uses the same close() + createWritable({ keepExistingData: true }) mechanism, every flush copies the whole file so far into a new swap file and writes it back to disk, so on a 10 GiB file one flush would take about a minute on my machine. A time interval would then make this worse rather than better.

Would it be worth adding "total disk writes stay proportional to the file size" to the AC of #182, or is the plan to change the flush mechanism there? One option might be one part file per chunk (each closed once) that are concatenated into the final file at the end, which does not need move(). Staging in OPFS, where flush() is cheap, would be another.

Thanks for the input. The plan was to refine the issue and add what needs to be done, i.e. OPFS.

EDIT: The biggest practical bottleneck here is not the GB written but the close time that needs to go through the whole file. This and the fact that we would like to account for unstable network so flush frequently, makes OPFS the best option for fixing #182 and concomitantly this behavior, too.

> Thanks! I had a look at [#182](https://github.com/NBISweden/sda-download-ui/issues/182) and I am not sure it closes this one as written. If the time-based flush uses the same `close()` + `createWritable({ keepExistingData: true })` mechanism, every flush copies the whole file so far into a new swap file and writes it back to disk, so on a 10 GiB file one flush would take about a minute on my machine. A time interval would then make this worse rather than better. > > Would it be worth adding "total disk writes stay proportional to the file size" to the AC of [#182](https://github.com/NBISweden/sda-download-ui/issues/182), or is the plan to change the flush mechanism there? One option might be one part file per chunk (each closed once) that are concatenated into the final file at the end, which does not need `move()`. Staging in OPFS, where `flush()` is cheap, would be another. Thanks for the input. The plan was to refine the issue and add what needs to be done, i.e. OPFS. EDIT: The biggest practical bottleneck here is not the GB written but the close time that needs to go through the whole file. This and the fact that we would like to account for unstable network so flush frequently, makes OPFS the best option for fixing #182 and concomitantly this behavior, too.

aaperis commented

2026-09-02 12:28:26 +00:00

(Migrated from github.com)

Suggested best-effort fix to reduce open/close times was added in #149. Follow-up for implementing OPFS is left for #182.

Rows
Columns

FSA download checkpoints copy the whole file every 512 MiB (quadratic disk I/O on large files) #193

Describe the bug

Measured

Steps to reproduce

Environment

Expected behavior

Additional context