Farmer suddenly shows “Have multiple copies of the plot…” on Chia V1.2.6 and 1.2.7

This issue has been tracked since 2021-09-22.

Hey there,

strange things started happening after installing V1.2.6:

“Have multiple copies of the plot …”

which is a total b/s since there are zero duplicates of my blocks. My farmer never had this behavior before from April 2021 until I installed V1.2.6.

And no, its not just the GUI, its in the logs too, see screenshot:

Bildschirmfoto 2021-09-22 um 10 55 57

Its quite annoying since the plots “degrade” to 50% over 24h, so I have to restart chia at least every 4 hours to have all my 2800 plots eligible for farming. All plots on randomly stated harddisks are all shown as multiple copies of the plot.

Checked and verified several times, there are no dupes at all. BUG seems to hit other users too, see my post @https://chiaforum.com/t/farmer-suddenly-shows-have-multiple-copies-of-the-plot-on-chia-v1-2-6-and-1-2-7/13129/11

OS is Windows Server 2019, Chia V1.2.7, Plots are hosted on several Synology NAS via 10Gbit network.

Cheers

loppefaaret wrote this answer on 2021-09-22

could you add the debug.log wiith these dupe messages ?

derscheff wrote this answer on 2021-09-22

could you add the debug.log wiith these dupe messages ?

I could, but I dont like to disclose all the information about my plots ;)

loppefaaret wrote this answer on 2021-09-22

fair enough, you could contact @pyl on our keybase https://keybase.io/team/chia_network.public
or email it to rc 'at' chia.net ?

derscheff wrote this answer on 2021-09-23

Done - thanks for looking into that! @loppefaaret

loppefaaret wrote this answer on 2021-09-23

oh wow - that just is completely random, and so odd... could you do me a favor, if you can catch it in this "Multiple copies" state during today, run a chia plots show command, and triple check that is does not output the same plots dir more than once ?
on windows, that would be under cd ~\AppData\Local\chia-blockchain\app-1.2.7\resources\app.asar.unpacked\daemon\ folder, where you can run .\chia.exe plots show

derscheff wrote this answer on 2021-09-23

oh wow - that just is completely random, and so odd... could you do me a favor, if you can catch it in this "Multiple copies" state during today, run a chia plots show command, and triple check that is does not output the same plots dir more than once ?

Yes - thats the thing. There is no pattern. Sure...I forward you the according logs when they come up.

emlowe wrote this answer on 2021-09-23

This code area was change/refactored in PR #8380 - so it seems a bug was introduced at that time.

loppefaaret wrote this answer on 2021-09-23

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

emlowe wrote this answer on 2021-09-23

Hi there, I'm working on loppefaaret on this. Right now, I think this log entry is spurious, in that it is being printed in error, but should not negatively affect any farming. What I do see is some flaky connection to your NAS, which may be more responsible for your issues. For example:
Very long lookup times (almost 30 seconds in some cases)
Plots being removed (I assume you aren't deleting any - but in this case during the refresh, the harvester code could not detect the file (python pathlib exists() failed)
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 219, processed_plots 0, remaining_plots 0, duration: 3.68 seconds
Those plots then come back later on
refresh_batch: loaded_plots 158, loaded_size 15.64 TiB, removed_plots 0, processed_plots 158, remaining_plots 0, duration: 12.78 seconds

We continue to look into this

emlowe wrote this answer on 2021-09-23

I have reproduced one scenario where this could happen - and it does result in those plots getting ignored until the harvester is restarted.
The problem I can reproduce deals with file modification times - the harvester checks the file modification time to see if the file has been changed since it was last seen. I believe in some cases (like maybe NAS or USB) the modification time doesn't match anymore and this causes the plot to get marked as a duplicate and not loaded.
I can simulate by using touch -m to manually change the modification time of the plot file

derscheff wrote this answer on 2021-09-24

I just mailed out all the logs from yesterday; concerning to the GUI and the corresponding logs I "lost" about 1000 plots over the day.

loppefaaret wrote this answer on 2021-09-24

yup, got it, thanks

derscheff wrote this answer on 2021-09-24

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

Hey - I just read your answer. I can surely do this over the weekend. Will mail it to rc afterwards.

Cheers

emlowe wrote this answer on 2021-09-24

@derscheff Can you try the build here: https://github.com/Chia-Network/chia-blockchain/actions/runs/1271511224

With that build I'm hoping the following:

  1. No Have multiple copies warnings
  2. replaced with Ignoring modification time mismatch
  3. Plots not being "lost"
derscheff wrote this answer on 2021-09-26

@derscheff if we could get you into DEBUG logging, and get you to send additional logs to me, that could help us narrowing down this bug...
either edit config.yaml and set

  log_level: DEBUG

under the farmer: section
or through the CLI with .\chia.exe configure --log-level DEBUG
and then restart Chia services (.\chia.exe start -r farmer)

on windows, you can open a PowerShell window, and do:

Get-Content -Wait ~\.chia\mainnet\log\debug.log | Select-String -Pattern 'multiple'

this will start scanning the debug.log file for the word "multiple" - as soon as it happens, it will be listed in this window, and you can send the log to me again
Thank you

Doing this now. Gonna send the "window report" out tomorrow...

derscheff wrote this answer on 2021-09-26

Hi there, I'm working on loppefaaret on this. Right now, I think this log entry is spurious, in that it is being printed in error, but should not negatively affect any farming. What I do see is some flaky connection to your NAS, which may be more responsible for your issues. For example:
Very long lookup times (almost 30 seconds in some cases)
Plots being removed (I assume you aren't deleting any - but in this case during the refresh, the harvester code could not detect the file (python pathlib exists() failed)
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 219, processed_plots 0, remaining_plots 0, duration: 3.68 seconds
Those plots then come back later on
refresh_batch: loaded_plots 158, loaded_size 15.64 TiB, removed_plots 0, processed_plots 158, remaining_plots 0, duration: 12.78 seconds

We continue to look into this

Could be a hint, but never was a problem in months of farming though

derscheff wrote this answer on 2021-09-26

@derscheff Can you try the build here: https://github.com/Chia-Network/chia-blockchain/actions/runs/1271511224

With that build I'm hoping the following:

  1. No Have multiple copies warnings
  2. replaced with Ignoring modification time mismatch
  3. Plots not being "lost"

Happy to do so after the 24h peroid of window logging of @loppefaaret ;)

derscheff wrote this answer on 2021-09-27

Powershell report just sent.

You guys want me to try the updated chia version of @emlowe now?

loppefaaret wrote this answer on 2021-09-27

yes, that should hopefully fix the issue, if you dont mind running it

More Details About Repo
Owner Name Chia-Network
Repo Name chia-blockchain
Full Name Chia-Network/chia-blockchain
Language Python
Created Date 2019-07-16
Updated Date 2022-08-13
Star Count 10914
Watcher Count 340
Fork Count 1913
Issue Count 343

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date