Resume FSA file downloads #173

2026-08-25T06:46:47Z

aaperis commented

2026-08-25 06:46:47 +00:00

(Migrated from github.com)

This PR closes #149 .

It improves the File System Access (FSA) batch downloads by adding persistent download metadata, partial-file resume support, periodical streamed file commits (saves byte chunks to disk), and user-visible download progress. It enables interrupted or canceled FSA downloads to be resumed by selecting the same destination folder again, while preserving local files and avoiding unsafe metadata updates.

It also adds a non-dismissible in-page progress modal for current page-local FSA downloads, including rough byte-based progress based on selected file sizes. The modal was needed because navigating away from the dataset's page stops the download. A later improvement could be to make the download persistent in the background but this is out of scope for now.

Type of change

New feature (non-breaking change which adds functionality)

List of changes made

FSA metadata persistence

Adds a .download-metadata file in the selected download directory.
Tracks per-file download metadata, including: file ID, file path, status, ETag, downloaded byte count, total byte count
File format is compact JSON.

Saving to a local file instead of e.g. the browser's indexDB has the benefit that if the session is cleared when logging out, the local file is not.

Resume and partial download handling

Supports resuming canceled FSA downloads and partial local files.
Skips files that are already completely downloaded.
Detects unsafe resume situations and either restarts or cancels the download. For example if:
- missing metadata for an existing local file
- local file larger than expected
- invalid or missing Content-Range
- resume offset mismatch
- final downloaded size mismatch

Durable checkpoint commits

This was needed because FSA's partial files are not durable on disk, i.e. there is no mechanism in FSA that persists a partial file by default. Without this, whenever a download was cancelled or interrupted the file size on disk remained 0.

Writes streamed content to local disk every 512MB so that if a download needs to be resumed, we need to re-download a maximum 512MB.
Periodically closes the writable stream to commit staged writes to disk. This creates some overhead but is needed for the above functionality.

Download result and progress reporting

Refactors per-file download result reporting to include: resumed, skipped, restarted files.
Adds rough byte-based progress tracking using selected files’ known sizes.

Adds a FSA download progress component that shows current FSA download progress in a Bootstrap modal.
Modal is intentionally non-dismissible because downloads are currently page-local and navigation may interrupt them.
Provides a “Cancel downloads” action.
Includes an estimated progress bar based on downloaded bytes vs total selected file sizes.

Testing

Instructions on how to test

Use Chrome or chromium based browser.
Use a dataset with multiple downloadable files and/or at least one large file.

docker compose -f sda-dev-stack.yml --profile sda-auth down
docker compose -f sda-dev-stack.yml --profile sda-auth --profile seed-bp-mock-dataset --profile seed-large-file up

should do the trick.

Start a download of selected files and hit CANCEL or close the tab before it finishes, confirm that:
- A non-dismissible download progress modal is shown.
- The progress bar advances during the download.
- Files are written to the selected folder.
- A .download-metadata file is created in the selected folder.
Repeat the same download to the same folder, observe the status in the modal. Make sure that:
- The modal closes or updates when the download completes.
- The completed files have the expected sizes.
Repeat the same download and make sure that nothing happens since all files are already downloaded, i.e. completed files are skipped.
Missing metadata. Delete the .download-metadatafile and try to repeat the download. Confirm that the download fails with an error message.

Definition of Done checklist

I have made an effort making the commit history understandable
I have performed a self-review of my own code and commented any hard-to-understand areas
Existing tests are passing
My changes generate no new warnings
Make sure the layout and text follow design sketches, if there are any
All Acceptance Criteria (AC) Fulfilled and checked in the Related issue

## Related issue(s) and PR(s) This PR closes #149 . It improves the File System Access (FSA) batch downloads by adding persistent download metadata, partial-file resume support, periodical streamed file commits (saves byte chunks to disk), and user-visible download progress. It enables interrupted or canceled FSA downloads to be resumed by selecting the same destination folder again, while preserving local files and avoiding unsafe metadata updates. It also adds a non-dismissible in-page progress modal for current page-local FSA downloads, including rough byte-based progress based on selected file sizes. The modal was needed because navigating away from the dataset's page stops the download. A later improvement could be to make the download persistent in the background but this is out of scope for now. ## Type of change - [x] New feature (non-breaking change which adds functionality) ## List of changes made ### FSA metadata persistence - Adds a `.download-metadata` file in the selected download directory. - Tracks per-file download metadata, including: file ID, file path, status, ETag, downloaded byte count, total byte count - File format is compact JSON. Saving to a local file instead of e.g. the browser's indexDB has the benefit that if the session is cleared when logging out, the local file is not. ### Resume and partial download handling - Supports resuming canceled FSA downloads and partial local files. - Skips files that are already completely downloaded. - Detects unsafe resume situations and either restarts or cancels the download. For example if: - missing metadata for an existing local file - local file larger than expected - invalid or missing `Content-Range` - resume offset mismatch - final downloaded size mismatch ### Durable checkpoint commits This was needed because FSA's partial files are not durable on disk, i.e. there is no mechanism in FSA that persists a partial file by default. Without this, whenever a download was cancelled or interrupted the file size on disk remained 0. - Writes streamed content to local disk every 512MB so that if a download needs to be resumed, we need to re-download a maximum 512MB. - Periodically closes the writable stream to commit staged writes to disk. This creates some overhead but is needed for the above functionality. ### Download result and progress reporting - Refactors per-file download result reporting to include: `resumed`, `skipped`, `restarted` files. - Adds rough byte-based progress tracking using selected files’ known sizes. ### Progress modal - Adds a FSA download progress component that shows current FSA download progress in a Bootstrap modal. - Modal is intentionally non-dismissible because downloads are currently page-local and navigation may interrupt them. - Provides a “Cancel downloads” action. - Includes an estimated progress bar based on downloaded bytes vs total selected file sizes. ## Testing ### Instructions on how to test - Use Chrome or chromium based browser. - Use a dataset with multiple downloadable files and/or at least one large file. ``` docker compose -f sda-dev-stack.yml --profile sda-auth down docker compose -f sda-dev-stack.yml --profile sda-auth --profile seed-bp-mock-dataset --profile seed-large-file up ``` should do the trick. - Start a download of selected files and hit CANCEL or close the tab before it finishes, confirm that: - A non-dismissible download progress modal is shown. - The progress bar advances during the download. - Files are written to the selected folder. - A `.download-metadata` file is created in the selected folder. - Repeat the same download to the same folder, observe the status in the modal. Make sure that: - The modal closes or updates when the download completes. - The completed files have the expected sizes. - Repeat the same download and make sure that nothing happens since all files are already downloaded, i.e. completed files are skipped. - Missing metadata. Delete the `.download-metadata`file and try to repeat the download. Confirm that the download fails with an error message. ## Definition of Done checklist - [x] I have made an effort making the commit history understandable - [x] I have performed a self-review of my own code and commented any hard-to-understand areas - [x] Existing tests are passing - [x] My changes generate no new warnings - [x] Make sure the layout and text follow design sketches, if there are any - [x] All Acceptance Criteria (AC) Fulfilled and checked in the Related issue

AnnikaSonnek (Migrated from github.com) reviewed

2026-08-25 06:46:47 +00:00

AnnikaSonnek commented

2026-08-26 14:40:06 +00:00

(Migrated from github.com)

I started to test this and most of it works as described but the things that are not fully working for me is:

I can't see that the .download-metadata file is added to the folder I've selected for the download (the files are downloaded though). EDIT: Okay, I now see it! :)
When I check the size for individual files it looks correct. But if I download all files for a dataset (via selection) and compare the total size of them to the size of the dataset that is show in the "card" for the dataset in the page the numbers don't always match. But I might be looking at the wrong thing though. EDIT: It's the size of the individual files that should be checked. They seem correct when I tested it!

I started to test this and most of it works as described but the things that are not fully working for me is: - I can't see that the `.download-metadata` file is added to the folder I've selected for the download (the files are downloaded though). EDIT: Okay, I now see it! :) - When I check the size for individual files it looks correct. But if I download all files for a dataset (via selection) and compare the total size of them to the size of the dataset that is show in the "card" for the dataset in the page the numbers don't always match. But I might be looking at the wrong thing though. EDIT: It's the size of the individual files that should be checked. They seem correct when I tested it!

AnnikaSonnek (Migrated from github.com) reviewed

2026-08-28 12:53:30 +00:00

AnnikaSonnek (Migrated from github.com) left a comment

No description provided.

I tested this and it works as expected. I looked through the code and just found one thing that I thought could be changed. (I saw later that this was already flagged as a warning by the linting 😅) I think it could be good if someone else also takes a look since the PR is quite big and I'm not so experienced with these kind of functions.

HenrikeW commented

2026-08-31 12:18:20 +00:00

(Migrated from github.com)

I'm starting a review now...

HenrikeW (Migrated from github.com) requested changes

2026-08-31 14:29:22 +00:00

HenrikeW (Migrated from github.com) left a comment

I've tested this and works as expected! Great job! Also, thank you for adding many explaining comments, a detailed PR description and most of all detailed testing instructions 💯

Most of the UI looks good as well but I think that the error messages need a little more attention, see my comments. I also saw that you need to rebase on develop as there were some layout changes to the files page that are not part of your code yet.
The comment I made on the ModalDialog is optional, feel free to dismiss it if you think it's too little benefit.

I also noticed that two of the smaller files I tested with didn't have exactly the same size as shown in the UI. On my computer, they had 1,176 bytes while the UI said 1.02 kB. I don't know if that is expected? It's a small difference but still a difference.

I've tested this and works as expected! Great job! Also, thank you for adding many explaining comments, a detailed PR description and most of all detailed testing instructions 💯 Most of the UI looks good as well but I think that the error messages need a little more attention, see my comments. I also saw that you need to rebase on develop as there were some layout changes to the files page that are not part of your code yet. The comment I made on the ModalDialog is optional, feel free to dismiss it if you think it's too little benefit. I also noticed that two of the smaller files I tested with didn't have exactly the same size as shown in the UI. On my computer, they had 1,176 bytes while the UI said 1.02 kB. I don't know if that is expected? It's a small difference but still a difference.

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

The error messages here are pushing the button around in the UI in a way that is hard to predict and can result in weird layouts, e.g on small screens or with bigger text in the browser.
I would suggest showing a dismissable modal with the error message instead.

The error messages here are pushing the button around in the UI in a way that is hard to predict and can result in weird layouts, e.g on small screens or with bigger text in the browser. I would suggest showing a dismissable modal with the error message instead.

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

The code comment mentions a "changed c4gh key" while the error message to the user states that "download metadata is missing". Is the error message misleading?

I'm even wondering if we should mention the download metadata for the user at all. They don't necessarily are aware of that hidden file and the term could be confusing. I think it would be enough to say "Existing local file ... has been partially downloaded but cannot be safely resumed. Delete or move the local file before retrying."

The code comment mentions a "changed c4gh key" while the error message to the user states that "download metadata is missing". Is the error message misleading? I'm even wondering if we should mention the download metadata for the user at all. They don't necessarily are aware of that hidden file and the term could be confusing. I think it would be enough to say "Existing local file ... has been partially downloaded but cannot be safely resumed. Delete or move the local file before retrying."

frontend/src/app/components/FileSystemDownloadProgressModal.tsx

					
				@ -0,0 +14,4 @@

				  onCancel: () => void;

				};

				export function FileSystemDownloadProgressModal({

It kind of bothers me that we are not reusing the existing ModalDialog component here but I suppose the reason is that the modal in this case must not be dismiss-able. Was that your reasoning as well?
I wonder if it would reduce code if we instead add a boolean prop dismissable to the ModalDialog component so that you can reuse it here without showing the "Close"-button. But I'll leave that up to you.

It kind of bothers me that we are not reusing the existing ModalDialog component here but I suppose the reason is that the modal in this case must not be dismiss-able. Was that your reasoning as well? I wonder if it would reduce code if we instead add a boolean prop `dismissable` to the ModalDialog component so that you can reuse it here without showing the "Close"-button. But I'll leave that up to you.

aaperis (Migrated from github.com) reviewed

2026-09-01 11:45:03 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

That's a great catch!
Actually, I forgot to use this in the logic below. I now pushed the fix in commit 1473bbc.

Background: The problem here is that the file size reported by the sda-download backend service is not the same with the file size written on disk. The actual download size will be slightly larger due to the recipient-specific Crypt4GH header.
So, the estimateFilesize variable is used as a way to get a more precise progress estimate. Imo precise accuracy for the progress estimate is not needed but this is a cheap way to improve things.

That's a great catch! Actually, I forgot to use this in the logic below. I now pushed the fix in commit [1473bbc](https://github.com/NBISweden/sda-download-ui/commit/1473bbcb49e2e5b1185930e37bab8390bd3f3366). Background: The problem here is that the file size reported by the sda-download backend service is not the same with the file size written on disk. The actual download size will be slightly larger due to the recipient-specific Crypt4GH header. So, the `estimateFilesize` variable is used as a way to get a more precise progress estimate. Imo precise accuracy for the progress estimate is not needed but this is a cheap way to improve things.

aaperis (Migrated from github.com) reviewed

2026-09-01 12:45:23 +00:00

frontend/src/app/components/FileSystemDownloadProgressModal.tsx

					
				@ -0,0 +14,4 @@

				  onCancel: () => void;

				};

				export function FileSystemDownloadProgressModal({

Yes, the reasoning is that the modal needs to be non-dismissible.I agree that it would be better practice to reuse existing components if the boolean mod is small. I opened #191 for this, please feel free to edit.

aaperis commented

2026-09-01 12:51:34 +00:00

(Migrated from github.com)

I also noticed that two of the smaller files I tested with didn't have exactly the same size as shown in the UI. On my computer, they had 1,176 bytes while the UI said 1.02 kB. I don't know if that is expected? It's a small difference but still a difference.

Yes, that is expected, because the sda-download backend API reports file sizes without including the header size as FileInfo, see here. This is the simplest estimate we can do without doing elaborate work trying to calculate the header size and summing both sizes. Even if we do, this will probably mean doing extra head requests to the backend API for every file with little gain.

We could raise the issue at a sprint review and maybe add clarifying text as a hover over maybe.

> I also noticed that two of the smaller files I tested with didn't have exactly the same size as shown in the UI. On my computer, they had 1,176 bytes while the UI said 1.02 kB. I don't know if that is expected? It's a small difference but still a difference. Yes, that is expected, because the sda-download backend API reports file sizes without including the header size as FileInfo, see [here](https://github.com/neicnordic/sensitive-data-archive/blob/9cce45411467305a27ce01c5c20422f40b66fe4b/sda/cmd/download/swagger_v2.yml#L866). This is the simplest estimate we can do without doing elaborate work trying to calculate the header size and summing both sizes. Even if we do, this will probably mean doing extra head requests to the backend API for every file with little gain. We could raise the issue at a sprint review and maybe add clarifying text as a hover over maybe.

👍 1

aaperis (Migrated from github.com) reviewed

2026-09-01 16:16:59 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

Great comment, thanks for pointing out. I have now made these appear as modals as you suggested.

aaperis (Migrated from github.com) reviewed

2026-09-01 16:18:17 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

This was a leftover comment. Great catch. I have fixed the error message and comment to be more user friendly and reflect the actual behavior.

aaperis commented

2026-09-01 16:20:11 +00:00

(Migrated from github.com)

Thanks @AnnikaSonnek and @HenrikeW for the nice feedback! Please take a look again since approvals have been dismissed by recent pushes and comments have been addressed.

jhagberg (Migrated from github.com) reviewed

2026-09-02 06:50:21 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

					
				@ -123,3 +220,1 @@

				        </button>

				        {isDownloading && (

				    <>

Since resuming depends on the user picking the same folder again, would it be worth passing an id to showDirectoryPicker, for example showDirectoryPicker({ mode: "readwrite", id: "sda-batch-download" })? Chrome then remembers the last folder used for that id and opens the picker there, which makes the "select the same folder" step from the modal text a bit easier to get right.

Since resuming depends on the user picking the same folder again, would it be worth passing an `id` to `showDirectoryPicker`, for example `showDirectoryPicker({ mode: "readwrite", id: "sda-batch-download" })`? Chrome then remembers the last folder used for that id and opens the picker there, which makes the "select the same folder" step from the modal text a bit easier to get right.

jhagberg commented

2026-09-02 06:54:51 +00:00

(Migrated from github.com)

Sorry for one more comment on a PR I am not reviewing, feel free to ignore it. I might have missed something, but I could not find any new test files in the diff, while the DoD checklist has the unit test item ticked. Is that a leftover from the template?

HenrikeW (Migrated from github.com) reviewed

2026-09-02 08:33:12 +00:00

frontend/src/app/components/FileSystemDownloadProgressModal.tsx

					
				@ -0,0 +14,4 @@

				  onCancel: () => void;

				};

				export function FileSystemDownloadProgressModal({

That's awesome! Good idea to turn it into a refactoring issue.

HenrikeW (Migrated from github.com) requested changes

2026-09-02 11:44:58 +00:00

HenrikeW (Migrated from github.com) left a comment

This now looks much better than before, great! I would still vote for a small change to the modal shown when the download is cancelled manually. I made a comment for the modal title.
I also would suggest turning the error message into a proper sentence, like "The download process was cancelled." That way I think the modal will look a bit more complete and there's a better balance between length of the heading and length of the body, like this:

Otherwise this PR looks great now.

Thanks also for the explanation regarding the file sizes. I think it's fine if it differs a little but it probably makes sense to raise the issue during a sprint review or refinement.

This now looks much better than before, great! I would still vote for a small change to the modal shown when the download is cancelled manually. I made a comment for the modal title. I also would suggest turning the error message into a proper sentence, like "The download process was cancelled." That way I think the modal will look a bit more complete and there's a better balance between length of the heading and length of the body, like this: <img width="731" height="295" alt="Image" src="https://github.com/user-attachments/assets/eb22082d-2f31-4390-a4a6-14ceb686adb8" /> Otherwise this PR looks great now. Thanks also for the explanation regarding the file sizes. I think it's fine if it differs a little but it probably makes sense to raise the issue during a sprint review or refinement.

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

This modal is shown when the user manually interrupts the download, either by clicking "Cancel download" or just closing the file explorer window. So I think the term "error" is too strong here, I would suggest using "Information" or something equally neutral.

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

Small thing: If you start the sentence with "The file .....", the text will start with a capital letter instead of a file name and (I think) will look a bit more tidy and complete.

aaperis (Migrated from github.com) reviewed

2026-09-02 12:12:03 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

Good point, changed it to "Notice".

aaperis (Migrated from github.com) reviewed

2026-09-02 12:12:24 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

Yes, this reads better. Now changed.

aaperis commented

2026-09-02 12:19:06 +00:00

(Migrated from github.com)

Sorry for one more comment on a PR I am not reviewing, feel free to ignore it. I might have missed something, but I could not find any new test files in the diff, while the DoD checklist has the unit test item ticked. Is that a leftover from the template?

Yes.

> Sorry for one more comment on a PR I am not reviewing, feel free to ignore it. I might have missed something, but I could not find any new test files in the diff, while the DoD checklist has the unit test item ticked. Is that a leftover from the template? Yes.

aaperis (Migrated from github.com) reviewed

2026-09-02 12:19:41 +00:00

frontend/src/app/components/FileSystemAccessBatchDownloadActions.tsx

					
				@ -123,3 +220,1 @@

				        </button>

				        {isDownloading && (

				    <>

/na

aaperis commented

2026-09-02 12:22:11 +00:00

(Migrated from github.com)

I also added a commit to somewhat progressively reduce the times a file is closed by FSA as mentioned in #193 since this creates the bottlenecks in the download progress. It will improve slightly the UX until there is a better mechanism in place for the resuming mechanism.