1848542 - Backgroundtask_removeDirectory fails to remove cache2 directory

Status: RESOLVED FIXED

(Core :: Networking: Cache, defect, P2)

Description

[avec]

It seems that bug still not fixed. I have the same problem with cache2.*****.purge.bg_rm directories in my C:\Users\Username\AppData\Roaming\Mozilla\Firefox\Profiles\ProfileName directory.

Windows 10 x64 with latest updates, Firefox 116.0.2. The same had happened with 114.0.2 after clean install Firefox 114.0.2 on 'old' installation (the same distributive of OS and Firefox) doesn't seem to have this bug.

+++ This bug was initially created as a clone of Bug #1818714 +++

There's a public report that the directory gets cache2.(timestamp).purge.bg-rm-cachePurge-(HASH) files when shutting Firefox down:

1. Open Firefox Settings (about:preferences) page.
2. Click on “Privacy & Security” section in left-side pane.
3. In right-side pane, scroll down to History section and enable “Clear history when Firefox closes” option.
   PS: The same clear history when Firefox closes option can be enabled using about:config page. Just set privacy.sanitize.sanitizeOnShutdown preference/flag to True.
4. Again click on “Settings” button given next to the clear history option and enable “Cache” option only as shown in following screenshot:

After enabling the option, I closed Firefox and I checked the “C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38” folder again. I was surprised to see a new file “Cache2.2023-xx.purge.bg_rm-cachePurge-308046B0AF4A39CB” created in the folder. The file size was 0 bytes and Windows was unable to detect its file type. Windows was showing it as an unknown “.bg_rm-cachePurge-308046B0AF4A39CB file” as the extension was automatically taken from the file name.

That should be being created by CachePurgeLock calling MultiInstanceLock.

The similar UpdateLock- has a removal logic, maybe the cache purger needs an equivalent?

Comment 1

[Valentin Gosu [:valentin] (he/him)]

Hi, thank you for reporting this.
It seems odd that bug 1818714 didn't fix the issue for you.
I have a few questions that might help track down the issue:

1. First, delete all the older bg_rm-cachePurge files.
2. Then , we shoud figure out at whether your cache folder actually gets deleted at shutdown:

• Go to about:cache - Look at the disk section, and go to the folder listed in Storage disk location
• Close Firefox and check if a new .bg_rm-cachePurge file is added and if the cache2 folder is removed.

If the cache2 folder is still there, that means maybe there's something preventing it from being deleted, which is also preventing the lock from being removed. See if you can manually remove it, and repeat the steps above to see if it still happens.

Thanks!

Comment 2

[avec]

Hi, Valentin, and thank you for your care!

1, 2 - Done.
Files/folder doesn't deleted automatically. Manually deleted w\o any problem (no lock prompt, etc.) After close/launch/close again Firefox the situation repeats: cache2 folder recreated and exists, another one .bg_rm-cachePurge file is created and exists wit preious one(s).

Comment 3

[Valentin Gosu [:valentin] (he/him)]

Thanks for checking. It seems that "something" is preventing the folder from being deleted.
Could you also check if setting network.cache.shutdown_purge_in_background_task to false and restarting works?
Does the cache folder get deleted?
Does it block shutdown? (can you open Firefox after 10 seconds?)

Comment 4

[avec]

Could you also check if setting network.cache.shutdown_purge_in_background_task to false and restarting works?
Does the cache folder get deleted?

Yes, setting it to false works! No more .purge.bg_rm directories after closing Firefox.

Does it block shutdown? (can you open Firefox after 10 seconds?)

I may reopen it within 1 sec but I have a fast SSD.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

(In reply to avec from comment #4)

Could you also check if setting network.cache.shutdown_purge_in_background_task to false and restarting works?
Does the cache folder get deleted?

Yes, setting it to false works! No more .purge.bg_rm directories after closing Firefox.

Setting the pref to false would avoid the code that creates the empty lock file.
The question is does the cache2 folder also get deleted, or is it still there?

Comment 6

[avec]

The question is does the cache2 folder also get deleted, or is it still there?
Oh, my fault. It still recreates, but with 3 files only (no extensions):
ce_Kg==
ce_T151c2VyQ29udGV4dElkPTUs
ce_T151c2VyQ29udGV4dElkPTUsYSw=
no doomed and entries subdirs in it.

Comment 7

[Valentin Gosu [:valentin] (he/him)]

Sorry to keep needinfo-ing you all the time.
It's good that the folder gets recreated after startup, but after you shut down the browser, is the cache2 directory still there or was it deleted?

Comment 8

[avec]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #7)

Sorry to keep needinfo-ing you all the time.
It's good that the folder gets recreated after startup, but after you shut down the browser, is the cache2 directory still there or was it deleted?
Yes, it created after browser startup and remains after shutdown. I can delete it manualy but it would recreated after next startup and remains after next shutdown.

Comment 9

[Valentin Gosu [:valentin] (he/him)]

(In reply to avec from comment #8)

Yes, it created after browser startup and remains after shutdown. I can delete it manualy but it would recreated after next startup and remains after next shutdown.

Good, that means there's something on your system preventing the deletion.
Could you capture some logs with network.cache.shutdown_purge_in_background_task set to false?
Instructions here: https://firefox-source-docs.mozilla.org/networking/http/logging.html

Go to about:logging, log to file, start logging, shutdown firefox, save the logs (before opening a new firefox instance), then upload the logs either here or to [e-mail]
That should show us why the folder doesn't get removed.

Thanks!

Comment 10

[avec]

Could you capture some logs with network.cache.shutdown_purge_in_background_task set to false?

Sent by e-mail, hope I had all done right. BTW: I can manualy delete cache2 folder with all it content while Firefox running, no problem with locks etc. After that Firefox haven't create cache2 folder again until I'm to close/restart it. Further, it doesn't create those 3 files inside cache2 folder if I had delete it (possible it create them somewhere else).

Comment 11

[Valentin Gosu [:valentin] (he/him)]

Thank you for the logs.
It seems all of the files appear to be successfully deleted by the main process, it's the background task that has some issues.

If you would be able to help with some more debug steps, I'd really appreciate it:
Please set the network.cache.shutdown_purge_in_background_task pref back to true. Then shut down Firefox.
Then run the following in the Command Prompt (press Windows key, type cmd, then press Enter to open Command prompt)
C:\Windows\System32\cmd.exe /c "SET GLEAN_DEBUG_VIEW_TAG=valentinbugtest && START /D ^"C:\Program Files\Mozilla Firefox^" firefox.exe"
If Firefox is installed to a different folder, make sure to change the command above.
Then close Firefox and let me know here. We should receive a telemetry ping with this specific tag, allowing me to see if the background task is having issues.

Let me know when you were able to complete these steps. Thanks!

Comment 12

[avec]

It's done right now, but I'm not sure you've got telemetry ping while telemetry was disabled in settings, in my firewall, hosts file, and somewhere else ;) So I need full prerequirements list for sending telemetry or some way to check if telemetry have fail/success in real-time and on my side, could you suggest something?

Comment 13

[Valentin Gosu [:valentin] (he/him)]

Unfortunately the probe doesn't seem to have gone through. 🙁
I'm not sure what to suggest for the ping to go through. If you've disabled telemetry or have anything blocking it, that wouldn't work.

The gist of this bug is that removing the cache2/ dir doesn't work in the background task, but does work when run from the main Firefox process.

Another question, when using a background task, after shutting down Firefox, is the cache folder still called cache2?
That would indicate that it's renaming it during shutdown which failed.

Could you also attach the contents of about:support to the bug?
Thanks!

Comment 14

[avec]

Another question, when using a background task, after shutting down Firefox, is the cache folder still called cache2?
cache2 after launch and renamed to cache2.%datetime%.purge.bg_rm after shutdown, cache2 again after next launch (cache2.%datetime%.purge.bg_rm still exists) and cache2 disappears after next shutdown but onother one cache2.%datetime%.purge.bg_rm appears
All of the above is true for network.cache.shutdown_purge_in_background_task pref set to true

Could you also attach the contents of about:support to the bug?
Sent to [e-mail]

Comment 15

[Valentin Gosu [:valentin] (he/him)]

(In reply to avec from comment #14)

cache2 after launch and renamed to cache2.%datetime%.purge.bg_rm after shutdown, cache2 again after next launch (cache2.%datetime%.purge.bg_rm still exists) and cache2 disappears after next shutdown but onother one cache2.%datetime%.purge.bg_rm appears
All of the above is true for network.cache.shutdown_purge_in_background_task pref set to true

Thanks for the info.
If it were only about the lock file staying around it wouldn't have been so bad, but this seems more problematic.

Comment 16

[Manuel Bucher [:manuel]]

It's done right now, but I'm not sure you've got telemetry ping while telemetry was disabled in settings, in my firewall, hosts file, and somewhere else ;) So I need full prerequirements list for sending telemetry or some way to check if telemetry have fail/success in real-time and on my side, could you suggest something?

You can also go to about:telemetry, download the raw json and send via Email to [e-mail]. I assume this is still helpful.

Comment 17

[avec]

(In reply to Manuel Bucher [:manuel] from comment #16)

It's done right now, but I'm not sure you've got telemetry ping while telemetry was disabled in settings, in my firewall, hosts file, and somewhere else ;) So I need full prerequirements list for sending telemetry or some way to check if telemetry have fail/success in real-time and on my side, could you suggest something?

You can also go to about:telemetry, download the raw json and send via Email to [e-mail]. I assume this is still helpful.

<quote>Blocked Page
Your organization has blocked access to this page or website.</quote>
It's not a problem to unblock it, but I didn't remember how since I'd blocked it too long-long time ago... ;)

Comment 18

[Manuel Bucher [:manuel]]

Huh, yes that is elaborate disabling of telemetry ^^. Probably something with enterprise policy. You could try to enable telemetry for the bug report if you have policies specified in https://github.com/mozilla/policy-templates/blob/469ddf78e2c888d1d32725a3ec09be6a13103921/README.md

Comment 19

[avec]

I had reproduce this behavior on new profile: you need to check 'Clear history when Firefox closes' and further check 'Cache' in 'Settings...' Voila, we get endless cache2.%datetime%.purge.bg_rm

Comment 20

[Valentin Gosu [:valentin] (he/him)]

Hi,

I have a few more questions.
Are you using an anti-virus? If yes, which one? Could you check if disabling the AV fixes this issue?
Is the Firefox profile folder that contains cache2 being synced/backed up? Using OneDrive maybe?

Otherwise I'll try to create a custom build for you with extra logging for the background task so we can figure out why removing the folder is failing.

Comment 21

[avec]

Hi, Valentin!

1. No, no AV installed, no even Windows Defender.
2. Yes, synced manually by FreeFileSync. So it's not a sync program lock.
   Lets do it! ;)

Comment 22

[Valentin Gosu [:valentin] (he/him)]

I'll try to provide you with a set of instructions to reliably capture logs from the background task.

Comment 23

[abap.weasel]

Hi, I just want to report that issue regarding bg_rm-cachePurge still happens for me in v119 64x.
Also folder C:\Users\Username\AppData\Roaming\Mozilla\Firefox\Profiles\ProfileName\storage\to-be-removed is probably affected by the issue.
Oldest folder in there for me is 02.01.2023 which is exactly same date as oldest bg_rm-cachePurge folder.
That to-be-removed folder should also cleanup itself with clean-cache-on-exit, right? Are there other cache folders that need to be checked?
-> deleted about 35 GB of Firefox cache stuff from my machine today.

Comment 24

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1848542 - Do a cache cleanup when cache purging is enabled r=#necko

It seems that sometimes the cache purge task may fail to remove the
cache folder. When that happens, we might end up with multiple cache
folders in the profile folder. Since the task is unable to remove it,
we should remove the folders on a background thread (check once a day)

Comment 25

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1848542 - Add telemetry for residual cache folder removal or failure r=#necko

Depends on D194051

Comment 26

[Valentin Gosu [:valentin] (he/him)]

Attachment: data-review.md

Comment 27

[[e-mail]]

Comment on attachment 9364747 [details]
data-review.md

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes.
​
Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
If the request is for permanent data collection, is there someone who will monitor the data over time?
​
No. This collection will expire 136
​
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 1, Technical.
​
Is the data collection request for default-on or default-off?
​
Default on for all channels.
​
Does the instrumentation include the addition of any new identifiers?
​
No.
​
Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
Does the data collection use a third-party collection tool?
​
No.
​

Result: datareview+

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd904f455af0
Do a cache cleanup when cache purging is enabled r=necko-reviewers,kershaw
https://hg.mozilla.org/integration/autoland/rev/9069ce37de4f
Add telemetry for residual cache folder removal or failure r=necko-reviewers,kershaw

Comment 29

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/bd904f455af0
https://hg.mozilla.org/mozilla-central/rev/9069ce37de4f

Comment 30

[avec]

(In reply to Cristian Tuns from comment #29)

https://hg.mozilla.org/mozilla-central/rev/bd904f455af0
https://hg.mozilla.org/mozilla-central/rev/9069ce37de4f

Unfortunately not completely fixed in FF 122.0 Win10 x64 :( I've got:

• cache2.(timestamp).purge.bg-rm-cachePurge-(HASH) *
  with 2 empty subfolders:
• doomed
  entries *
  and 3 zero size files:
• ce_Kg==
  ce_T151c2VyQ29udGV4dElkPTUs
  ce_T151c2VyQ29udGV4dElkPTUsYSw= *

Comment 31

[Donal Meehan [:dmeehan - PTO - Back Oct 13]]

Setting the flags back to the expected state. Any follow-ups should be tracked under separate bugs.