Farmer suddenly shows “Have multiple copies of the plot…” on Chia V1.2.6 and 1.2.7

could you add the debug.log wiith these dupe messages ?

could you add the debug.log wiith these dupe messages ?

I could, but I dont like to disclose all the information about my plots ;)

fair enough, you could contact @pyl on our keybase https://keybase.io/team/chia_network.public
or email it to rc 'at' chia.net ?

Done - thanks for looking into that! @loppefaaret

oh wow - that just is completely random, and so odd... could you do me a favor, if you can catch it in this "Multiple copies" state during today, run a chia plots show command, and triple check that is does not output the same plots dir more than once ?
on windows, that would be under cd ~\AppData\Local\chia-blockchain\app-1.2.7\resources\app.asar.unpacked\daemon\ folder, where you can run .\chia.exe plots show

oh wow - that just is completely random, and so odd... could you do me a favor, if you can catch it in this "Multiple copies" state during today, run a chia plots show command, and triple check that is does not output the same plots dir more than once ?

Yes - thats the thing. There is no pattern. Sure...I forward you the according logs when they come up.

This code area was change/refactored in PR #8380 - so it seems a bug was introduced at that time.

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

Hi there, I'm working on loppefaaret on this. Right now, I think this log entry is spurious, in that it is being printed in error, but should not negatively affect any farming. What I do see is some flaky connection to your NAS, which may be more responsible for your issues. For example:
Very long lookup times (almost 30 seconds in some cases)
Plots being removed (I assume you aren't deleting any - but in this case during the refresh, the harvester code could not detect the file (python pathlib exists() failed)
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 219, processed_plots 0, remaining_plots 0, duration: 3.68 seconds
Those plots then come back later on
refresh_batch: loaded_plots 158, loaded_size 15.64 TiB, removed_plots 0, processed_plots 158, remaining_plots 0, duration: 12.78 seconds

We continue to look into this

I have reproduced one scenario where this could happen - and it does result in those plots getting ignored until the harvester is restarted.
The problem I can reproduce deals with file modification times - the harvester checks the file modification time to see if the file has been changed since it was last seen. I believe in some cases (like maybe NAS or USB) the modification time doesn't match anymore and this causes the plot to get marked as a duplicate and not loaded.
I can simulate by using touch -m to manually change the modification time of the plot file

I just mailed out all the logs from yesterday; concerning to the GUI and the corresponding logs I "lost" about 1000 plots over the day.

yup, got it, thanks

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

Hey - I just read your answer. I can surely do this over the weekend. Will mail it to rc afterwards.

Cheers

@derscheff Can you try the build here: https://github.com/Chia-Network/chia-blockchain/actions/runs/1271511224

With that build I'm hoping the following:

1. No Have multiple copies warnings
2. replaced with Ignoring modification time mismatch
3. Plots not being "lost"

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

Doing this now. Gonna send the "window report" out tomorrow...

Hi there, I'm working on loppefaaret on this. Right now, I think this log entry is spurious, in that it is being printed in error, but should not negatively affect any farming. What I do see is some flaky connection to your NAS, which may be more responsible for your issues. For example:
Very long lookup times (almost 30 seconds in some cases)
Plots being removed (I assume you aren't deleting any - but in this case during the refresh, the harvester code could not detect the file (python pathlib exists() failed)
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 219, processed_plots 0, remaining_plots 0, duration: 3.68 seconds
Those plots then come back later on
refresh_batch: loaded_plots 158, loaded_size 15.64 TiB, removed_plots 0, processed_plots 158, remaining_plots 0, duration: 12.78 seconds

We continue to look into this

Could be a hint, but never was a problem in months of farming though

@derscheff Can you try the build here: https://github.com/Chia-Network/chia-blockchain/actions/runs/1271511224

With that build I'm hoping the following:

1. No Have multiple copies warnings
2. replaced with Ignoring modification time mismatch
3. Plots not being "lost"

Happy to do so after the 24h peroid of window logging of @loppefaaret ;)

Powershell report just sent.

You guys want me to try the updated chia version of @emlowe now?

yes, that should hopefully fix the issue, if you dont mind running it