GW runs out of memory with extreme retries

[akusto]

Hi!

I am using version 1.5 and have noticed that some of my old Amiga disks could be converted to adf by using insane amount of retries. However, if I set retry to for example 5000 I run to flux overflow:

T7.0: AmigaDOS (8/11 sectors) from Raw Flux (400957564 flux in 1917035.65ms) (Retry #1.2763)
Command Failed: GetFluxStatus: Flux Overflow
.
Is it possible to change the command so that it starts flux counting from zero after each successful track? Then I could leave gw to work overnight and perhaps have entire disk read in the morning.

[keirf]

Wow, that's a lot of retries. The error is not actually an OOM, but I would have to guess that something has slowed down host side for the data flow from Greaseweazle to slow down enough to get the buffer overflow.

Accumulating the raw flux is completely pointless if --raw is not specified to gw read, as the raw flux gets thrown away in this case. So, I can fix that and then you can try again.

[keirf]

Please can you try the following fixed version. You will need to be logged in to GitHub to see the Artifacts link to the Windows prebuilt binaries: https://github.com/keirf/greaseweazle/actions/runs/10989783265

Or if you are running on Linux or Mac you can install the latest tip of master something like:

pipx install https://github.com/keirf/greaseweazle@master

You can tell if you are running with the fix because during retries you should not see the flux count accumulating across retries.

Please do give it a try on a bad disk and confirm that the overflow is gone.

[akusto]

Well, I celebrated too early. I left one of my problematic disks run over night, and it took on worst tracks more than 20k retries until it found all 11 sectors. However, it matches them too easily as OK, because even if it stated at end: "Found 1760 sectors of 1760 (100%)" there was numerous checksum errors on the disk when tried the result with Amiga emulator. I don't know how gw determines sector to be correct from flux, but somehow it seems it can make mistake every now and then if disk is in really bad condition.

[keirf]

Very easily done because the Amiga checksum is very poor indeed. It would I guess help if the amigados analyser could also check data block checksums where available. Amiga OFS disks have these extra checksums. Amiga FFS doesn't. But I think most floppy disks are OFS.

That said it might not help as I think both checksums are merely XOR so they're not exactly independent of each other.

I suppose it would have helped here because clearly the OFS checksums are failing (else the Amiga couldn't have detected the corruption!).

So... Could be something else for me to add?

[keirf]

Another thought is that bad disk areas can look like random noise. If you do enough reads of random noise you may satisfy a 16-bit checksum eventually within some thousands of attempts, whether the checksum algorithm is good or not! Have you looked at some of your bad disks as raw flux dumps in a visualiser such as HxC tools? Could you share one?

[akusto]

Yes, that is fact about checksums. Amount of retries (70) correlates somehow with invalid checksums when trying to open adf-file with unadf (102). Visually comparing numbers (not perfect method, but seems correlating): if amount of retries exceeds roughly 1024 times, it contains a bad sector for sure. The more read attempts after that, the more bad sectors per track. I think I just have to take it as granted, that old disks, which are stored tens of years here and there, are most likely completely dead. I opened the disk lid and rotated the disk with fingers, then looked at the surface against light, and the disk surface seems clearly malformed by time.

[keirf]

Oh well, the original fix to not accumulate raw flux was still a good one :)