HFEv3 hard sector support

[teiram]

After reading this thread I got the understanding that HFEv3 should support hard-sectored formats, seems that allowing to control directly the index signal.
Is this feature supported currently in flashfloppy? I think it would be interesting since I've seen there are other ongoing efforts related to hard-sectored formats.

Regards

[keirf]

Yes that is supported.

[keirf]

Actually no, the index opcode is currently ignored... Easy to fix though!

[teiram]

That would be something really nice to have!

[keirf]

Are there any example HFEv3 hard sector images?

[teiram]

I think the ones here should be

[ejona86]

If you need a "clean" HFE file with hard sectors, you can use this script. It is configured for Micropolis, but would be fairly easy to adapt to other formats. The result is unformatted, so it really only cares about track size, bitrate, number of tracks, and number of hard sectors.

#!/bin/env python3
import itertools
import math
import sys


def pad(bs):
    return bs + b'\xFF' * (512-len(bs))


track_data_len = 100000//8  # bitcell data, in bytes
sector_count = 16
track_count = 77
f = sys.stdout.buffer

header = (
    b'HXCHFEV3'  # magic
    b'\x00'      # formatrevision
    + bytes([track_count]) +  # nr_tracks
    b'\x02'      # nr_sides
    b'\x00'      # track_encoding
    b'\xFA\x00'  # bitrate in kB/s
    b'\x00\x00'  # rpm (unused)
    b'\x07'      # interface_mode
    b'\x01'      # rsvd
    b'\x01\x00'  # track_list_offset
)
f.write(pad(header))

tracklist = bytearray()
offset = 2
length = track_data_len + sector_count + int(sector_count > 1)
for track in range(track_count):
    tracklist.append(offset & 0xFF)
    tracklist.append((offset >> 8) & 0xFF)
    tracklist.append(math.ceil(length*2) & 0xFF)
    tracklist.append((math.ceil(length*2) >> 8) & 0xFF)
    offset += math.ceil(math.ceil(length) / 256)
f.write(pad(tracklist))

# Single side of track
track_side = bytearray(itertools.repeat(0x01, track_data_len))
sector_len = track_data_len/sector_count
track_side.insert(track_data_len - int(sector_len)//2, 0x8F)
for sector in range(sector_count-1, -1, -1):
    track_side.insert(int(sector_len * sector), 0x8F)
assert len(track_side) == length
track_side.extend(b'\x0F' * (256 - (len(track_side) % 256)))

# Interleave sides
track = bytearray()
for pos in range(0, len(track_side), 256):
    track.extend(track_side[pos:pos+256])
    track.extend(track_side[pos:pos+256])

for track_num in range(track_count):
    f.write(track)
f.flush()

[ejona86]

I've rebased #412 on top of master with the async I/O changes: https://github.com/ejona86/FlashFloppy/compare/master...ejona86:hfe-hard-sectors?expand=1

@keirf, given the below, should I move forward with the index_pulses array approach? Do you want to review the changes, or the approach sounds fine and I just push directly? I'll need to resolve the FIXMEs for fake_fired; probably will disable fake_fired based on track-change = realtime/write-drain = realtime (instant as well?).

I now have access to a Vector 4 and normal access is reliable with the branch (I didn't bother testing it with #412). It appears the Vector's behavior matches the suggested behavior of the FDD board manual and it does verify every write, so there is guaranteed to be 200 ms of slack time after writes. Larger writes only write every 5th sector, requiring 5 rotations to write a full track, and the verification takes the same amount of time so large writes have over a second of slack for writes to drain.

IDX cutting out is the Vector swapping to read from the HDD. It uses a combined HDD+FDD circuit board and many of the signals are shared, so the first STP in the screenshot is for the HDD (this was a file copy).

Formatting is not so easy. It writes every other sector with no pause between sides and provides 50 ms head settle time. It writes the entire disk (outside in) and then verifies the entire disk (inside out). The 4k write_bc provides 8 ms buffer, so writes to flash have 58 ms to complete when changing tracks. Still, I had reasonable success with a (newish) SD card and also the Raspberry Pi Zero.

Luckily, writes write the entire sector, starting at the sector pulse, so we only really need the directory blocks to be written correctly which is just two tracks. We can just ignore most formatting errors. (One manual suggested reading the sector id in the sector prior to the one you want to write; I don't know if that simply isn't being done, but I've written large portions of the disk afterward without issue.)

[keirf]

I've rebased #412 on top of master with the async I/O changes: https://github.com/ejona86/FlashFloppy/compare/master...ejona86:hfe-hard-sectors?expand=1

@keirf, given the below, should I move forward with the index_pulses array approach? Do you want to review the changes, or the approach sounds fine and I just push directly? I'll need to resolve the FIXMEs for fake_fired; probably will disable fake_fired based on track-change = realtime/write-drain = realtime (instant as well?).

You can go ahead. Is it worth doing 4.0a beforehand or wait until afterwards? Some kind of benchmark pre-release is needed I think, for gathering broader testing feedback.

[ejona86]

Is it worth doing 4.0a beforehand or wait until afterwards?

Seems fine either way. DSK is still TODO for #426; I have looked into it and have a good idea of what's necessary but had limited time for a period and now I have access to the Vector. If you're fine with lack of DSK for the pre-release, I'd just assume doing it sooner. If we think hard sectors is neat enough that we'd want to do a pre-release, we can wait for it. I sort of expect those of us doing hard sector would be fine for the while using builds from GitHub Actions.

[keirf]

Is it worth doing 4.0a beforehand or wait until afterwards?

Seems fine either way. DSK is still TODO for #426; I have looked into it and have a good idea of what's necessary but had limited time for a period and now I have access to the Vector. If you're fine with lack of DSK for the pre-release, I'd just assume doing it sooner. If we think hard sectors is neat enough that we'd want to do a pre-release, we can wait for it. I sort of expect those of us doing hard sector would be fine for the while using builds from GitHub Actions.

Maybe we do it sooner then. It really is just a selected point pre-release for testing, not a release candidate in any way, and we can pick another after hard sectoring is in. It just makes sense to have a few of these at chosen points imo so we can point users at them for testing, get feedback, and track the effects of the big changes.

[ejona86]

HFEv3 hard sector support is now on master. For the while you can use this recent build: https://github.com/keirf/FlashFloppy/suites/3098062873/artifacts/70800000

The HFE image really needs its tracks to be aligned, although limited skew should still work.

[keirf]

@ejona86 It's been a while since I checked in here. Are we ready for a v4.0a pre-release would you say?

[ejona86]

@keirf, it seems a good time for a v4 alpha.

[keirf]

I have some ED fixes to do for #496 it will make sense to pull through into master, so I will do that first as ED support is something that may be obviously improved in v4.

[keirf]

@ejona86 I have pushed out v4.0a experimental pre-release. I hope that the very brief release notes are acceptable. Please feel free to edit RELEASE_NOTES if not!

[ejona86]

I just pushed an update to scripts/mk_hfe.py for hard-sector support. When using hard-sectors, you should include in your FF.CFG:

index-suppression = no
track-change = realtime
write-drain = realtime

I looked to add that to the Wiki, but it seemed likely to just confuse most users. This will be good until there are dozens of us using hard sectors.

Hard sector support should be pretty complete at this point. If hard sectors don't work for you, let's work through it on a new issue as it could be other things like flash timing and HFE track misalignment.

[snhirsch]

Not clear what the status and/or availability is for hard-sector floppy support. Does it support only HFEv3 or can I emulate from a dsk image? I have both Heath and Northstar systems that use hard-sector (FM and MFM respectively) floppy formats. Jeff (HxC) started working on support for this a few years ago, but ended up being unable to support Northstar at all due to timing constraints and implemented read-only on Heath.

[ejona86]

@snhirsch, HFEv3 is required for hard sectors with the current support. The other existing image formats are only applicable for soft sectors. The support is only present in the v4 release series (v4.0a currently). Make sure to use the FF.CFG snippet I mention in my last comment.

The work has only been demonstrated to work on my Vector 4 using a blank image and formatting it with the Vector 4. You can use master's scripts/mk_hfe.py to generate an image of your own with the --hard-sectotrs N flag. There were some errors during formatting, but ignoring them worked fine.

There are some test images at https://hxc2001.com/vrac/HS/ and #412 (comment), but I have noticed some images have poor inter-track alignment which may or may not cause problems. There is some code in place to handle small levels of misalignment. But based on an earlier comment of mine, I think there's little hope for the heath_kf.hfe image without modifying the file to manually align the tracks.

Northstar was tested in #412 and the experience drove me to the async work (#426; done except for DSK and QuickDisk, and hopefully seeing wider testing with the 4.0a release). I'd love further testing, and the current code has much stronger pulse timings than in #412 so I am hopeful. I'd suggest trying to boot with some of the existing Northstar images and seeing where it goes.

[snhirsch]

Thanks for getting back. Yes, all those images that Jeff posted were produced during the period of time when he was trying to get HS support working. He took raw sector images I sent and generated those HFE files from them.

DREM is able to fully emulate both NS and Heath formats - even supporting low-level format! But, it's an expensive solution.

I'll try your code and see how it goes.

[ejona86]

@snhirsch, also, when you report back, it'd be helpful to know if your Gotek is STM or Artery (https://github.com/keirf/FlashFloppy/wiki/Gotek-Compatibility). Based on #351, it seems you may have a STM which would be preferred for this because of its greater RAM.

It would be possible to support image formats other than HFE, but each hard sectored system is its own special snowflake, so it is more convenient to leverage a general-purpose format like HFE and rely on PC software to convert as necessary.

[snhirsch]

I'll let you know what I find. I have only STM based hardware at this point. Genuine Gotek and a white Simulant board.

[teiram]

Sorry I didn't have much time lately to test.
I had the chance to upgrade the Gotek clon (STM32 based) on my Northstar Advantage to 4.0a and I think I'm still experimenting similar problems with writes as I used to have with @ejona86's fork:
I created a blank HFE with the updated mk_hfe.py script (35 tracks, 10 hard sectors). Tried to format it with the Advantage CP/M utilities and got lots of low level errors from the advantage. Similar results using one of the HFE images we used during previous testing (also with the ones that were cleaned up with the script prepared by @ejona86)
Just mention that I included a FF.CFG file in the pendrive with the recommended options.

I will try to wire analyzer and serial port to check logs and the relevant signals, in case it can be helpful.

[snhirsch]

The Northstar has extremely tight timing requirements that have been hashed out in some depth on the HxC forums. I'm also able to capture activity on a logic analyzer (Hantek 4032 with Sigrok software) if that helps.

[ejona86]

Poking around, it seems you saw Heath working a while ago. https://torlus.com/floppy/forum/viewtopic.php?p=18213#p18213 . In that case, maybe that is a better starting point if the Northstar is known to be particular. I'll continue thumbing through the thread and see what useful details y'all discovered in the past. Seems some of that previous work was with NSI, not HFE, which might have made it a bit more particular.

Seems we're all discussing here now, so reopening.

Looking with a logic analyzer would be helpful. Sharing a recording is maybe best, but also screenshots of particular parts you notice something fishy is great too. I typically use Sigrok myself. I may have a read through the Heath and Northstar manuals; I think I've already found some useful manuals in the past for them.

[ejona86]

Tried to format it with the Advantage CP/M utilities and got lots of low level errors from the advantage

@teiram, you might see if the disk still works okay, even if there are formatting errors. You might also formatting twice. I expect there'd still be formatting errors the second time, but the disk may be more usable. (I think on my device it may only look at the sector number in the sector header when writing, and wouldn't care about wrong checksums from lost writes during the format process.)

That said, I am disappointed to hear it only sorta works for reads from those images. Flash read latency may be coming into play. That could maybe be improved by delaying reading from flash for a few ms after swapping tracks to make sure another step isn't coming. We may be able to notice this being a problem via the logic analyzer.

[ejona86]

@teiram, I wonder if that was just a fluke (no need to test formatting more). I didn't expect head-settle-ms = 20 to change formatting compatibility. I had mentioned it for PIP compatibility.

[teiram]

@teiram, I wonder if that was just a fluke (no need to test formatting more). I didn't expect head-settle-ms = 20 to change formatting compatibility. I had mentioned it for PIP compatibility.

Oh, maybe placebo effect then :-)
I retried with PIP and I would say the error took more time to show up. Anyways once the error happens the image gets corrupted: No longer boots CPM properly and the error that happened during the copy operation shows now during boot.

[ejona86]

@teiram, I've been unable to reproduce the formatting issue. I re-created host timings with FF_TestBed and the index pulses were rock solid. I did initially think I had reproduced it, but I had a wrong FF.CFG. Maybe try resetting old FF.CFG settings? But that is a shot-in-the-dark.

I was able to reproduce the PIP issue and I've fixed it on master. A build is available at https://github.com/keirf/FlashFloppy/suites/3627370546/artifacts/87757013

[teiram]

@ejona86 weird. I flashed your new version, reset settings and tried PIP with same result. Then tried too with just:
index-suppression = false
in FF.CFG after resetting settings again and pretty much the same: failures during PIP and then the disc is rendered unusable. No longer boots into CP/M.
What is curious is that with that same FF.CFG FORMAT.COM runs without apparent issue. I was able to format 70 tracks at once without failure but then the disc is unreadable anyways. :(

Tell me what dumps would be useful for you and I will try to provide them

[ejona86]

@teiram, let's have another recording of PIP doing its thing and hitting a failure. Use the normal index-suppression = false+realtime FF.CFG.

[teiram]

@ejona86 , here you are:
pip-write-error-2.zip

[ejona86]

@teiram, can you try a different thumb drive? Another PIP recording would be useful, for me to see how random this failure is.

pip-write-error-2 is quite a riddle. First, the good. Index pulses are perfect. There was a place I thought was a retry, but 1) the sector was written perfectly and 2) the second write is actually different. So that was just a red herring and the host decided to write the same sector twice in a row.

The error appears to be a full bit being inserted (a 2us gap) which throws out the phase of data vs clock bits of MFM. But that requires a bit insertion within the HFE stream and requires shifting all following bytes. The full flow is:

1. a sector is written
2. the sector looks good the following rotation
3. the track is changed and other writes are made
4. the track is changed back to the earlier track
5. the sector has a bit inserted in the middle, and is the same every rotation

So (2) means that the write was successfully decoded by the Gotek and the update was properly made into the in-memory HFE track. The only difference between (2) and (5) should be that (3) will cause the track cache to be cleared and we'll have to read from flash for (5).

I'll note that the index pulse is still good, so that means that the bit shift did not extend into the block with the index opcode (even though that block was updated by the write). Even more strangely, just after the check byte (the last real byte of the sector) there is 1.5 ms of a pattern of one flux pulse every 8ms (HFE data 0x11) which should be random. The time between the injected bit and the end of the pattern is 4 ms, which would be two HFE blocks.

[teiram]

Hi @ejona86, sure.
I've tried same operation with a different drive. Apparently it goes the same way at least from user perspective:

• I run the command: PIP STAT2.COM=STAT.com
• Operation starts, takes some time, I can see attempts to write in the OLED screen (W shows)
• After a while, I get an error on screen ( T 2 D 1 S 0020 )
• I press enter to retry, same error comes quite fast: T 2 D 1 S 0020

pip-write-error-other-device.zip

[ejona86]

@teiram, what HFE disk image are you using for that PIP test?

Initial analysis for pip-write-error-other-device: the same sector is getting corrupted, but not as severely. Just the CRC and the following "random" flux to the end of the sector appears corrupt, or 2.4ms. The CRC is corrupted differently from the "random" flux. The corruption ends exactly at the index pulse. This should help quite a bit to find the culprit.

There's a "track and sector" byte in each sector and I've known the track ID didn't make sense to me. But I really don't see how "track 2" and "0x80" relate. Not important though. "Track 2" does explain why booting failed. Unclear what the 0020 means, since sector 0 sees the corruption. Even in the format output it was a bit unclear what the extra zeros are for. But none of that is probably important.

[teiram]

Attaching the disk image @ejona86 :
cpm.zip

[ejona86]

@teiram, try this new build: https://github.com/keirf/FlashFloppy/suites/3688369881/artifacts/89885255

@keirf, 0669a4c fixes a data loss bug. I've update the v4.0a release notes to discourage using the release.

I noticed that when things broke the check byte didn't actually get written and the pattern was pre-existing as well. So the problem was part of the write not getting written to flash. I reproduced the issue with the same HFE image and fixed the bug (0669a4c). The bug fix is on master.

[teiram]

Just tried the new build @ejona86,
seems now the pip operations finishes without errors!! But then the newly created STAT2 executable doesn't work and throws one of those cryptical errors, in this case: T 2 D 1 S 0099

I tried to restart with the so modified HFE and booting and some of the existing programs on the disk seems to keep running okey, only STAT2 fails consistently with the same error.

In case it's useful, I'm attaching the modified HFE here.

cpm-after-pip-stat2.zip

Oh, I also tried FORMAT with the new build and worked flawlessly.

[teiram]

I also experimented a weird behaviour with some HFE images. Sometimes after a read operation that failed, the gotek seems to hang. The buttons (quad encoder in my device) doesn't change the selected image anymore, neither is any reaction on pressing the encoder button. Only way I found to go out of this situation is to power cycle.
I will try to connect the serial port with a debug version to see if there is anything that could provide any lead.

[ejona86]

I'm glad it is working much better now. Thank you so much for the logic analyzer recordings. They helped a lot.

I looked at the cpm-after-pip-stat2 hfe and it seems to be a lost block write. I haven't been able to reproduce it. I've not yet found the bug.

Interestingly, I did find a different issue, which looks benign. For cpm.hfe specifically, the first index pulse is 1ms into the start of the HFE track which causes part of the last sector to be at the start of the HFE track. The HFE code doesn't support wrapping-around during writing, so those writes are thrown away. But the last 1ms of a sector isn't used so this doesn't cause a visible problem.

The lockups don't sound good. It doesn't sound like a crash. Maybe there's an accounting bug and we get permanently hug on I/O. Could be something else. If the serial debugging doesn't show anything, I could probably put in an assertion when syncing and the writing takes longer than, say, 5 seconds have it log and crash. That'd at least narrow it down.

[teiram]

Hi @ejona86,
unfortunately the serial debug shows nothing relevant. This is the sequence I followed:

• Booted with a CP/M boot disk (image named: CPM_Master_NSI.hfe).
• Switched to the offending image: Transfer Utils_NSI.hfe
• DIR worked okey
• Tried to execute a program there: PCGET and I get immediately error T4 D1 S 0322, as if it didn't even tried to read anything.
• From here on, the unit doesn't respond and needs a power cycle to work again.

[ejona86]

I think I've reproduced it. Maybe. I made a stress test and things were solid. By happenstance, I varied the write size by 6 bytes and then I saw some bizarre behavior. The drive doesn't fully-hang for me. I noticed the drive light turn off, for example. I have noticed hanging periods, but it eventually processes my input. But it's also not entirely consistent.

Something is clearly wrong, so I have something to investigate.

[teiram]

Hey good news. Do you think it could be useful to have the HFE image I experiment the failure with?

[ejona86]

Huh. I thought I had posted an update, but I guess not.

I don't need the HFE image, unless the image is "tainted" such that the image alone is enough to reproduce the hang. I made a Transfer Utils_NSI.hfe of my own and it seemed to work fine.

I root caused the issue I was seeing. By failing to initialize some bytes I was accidentally writing (the equivalent of) FM instead of MFM. HFEv3 then treated some of that data as opcodes. Based on the earlier logic recordings, I don't think that is the hang experienced so I've been trying out some stress tests. So far, nothing jumps out.

That opcode issue is particularly subtle, so I have gone ahead and taught the HFE code to avoid it by forcing a bit flip to 0. https://github.com/keirf/FlashFloppy/suites/3879621509/artifacts/96579204 has the fix. @teiram, assuming that doesn't fix the issue, then I think I need to see logic analyzer recordings when this happens.

[teiram]

Sorry for the delay.
Just tested with the fixed version but I'm afraid the issue is still there.
The unit gets frozen as before on trying to access the same file.
I will try to setup the logic analyzer again and provide some recordings

[ejona86]

I found a fixed another write-loss bug. Fluxengine now does verifying reads after writes, which let me easily detect lost writes since following read passes should then be fully reliable. I then noticed the bug with a throughput-oriented flash stick that has fast median writes but long-tail 780ms writes. The fast median writes was important to detect the bug. The fix is available in https://github.com/keirf/FlashFloppy/suites/4008179986/artifacts/101169672

That shouldn't help any hang, though.

[geneb]

This might be useful in your testing - I've put a number of NSI images up here: http://annex.retroarchive.org/disks/northstar/ - these were generated using an FC5025. Does FlashFloppy have the ability to natively use NSI images or do they need to be converted to HFE first?

[ejona86]

@geneb, no, FlashFloppy doesn't support NSI. So yes, they need to be converted to HFEv3 first, using HxCFloppyEmulator or similar.

[geneb]

Rats. Thanks!

[ejona86]

I discovered that HxCFloppyEmulator may not be converting NSI to HFE correctly. It could have decreased the reliability of reading from a converted image. I've made a nsi2hfe.py script and discuss some at: https://forum.vcfed.org/index.php?threads/flashfloppy-hard-sector-testing.1242608/#post-1319264 . There's not yet been an A/B test to confirm the padding my script uses produces better results.

That thread also mentions a working read-only usage from NSI with the Northstar Advantage. Limited usage; just booting. At least it seems the nsi2hfe.py script isn't completely busted.

[teiram]

Hi @ejona86. Thanks for posting about your conversion script.
I've just tried it on my Northstar Advantage, got a wordstar nsi disk image (probably the one everyone has) and converted it to hfe with your script just providing input and output file names. Worked amazingly well, I was able to:

• Boot from the generated image (CP/M 2.2)
• Execute wordstar and mbasic without any error whatsoever
• Erase wordstar files (to get some room available) without errors, STAT and DIR showing consistent info after removal
• Execute PIP MB2.COM=MBASIC.COM without errors too, STAT and DIR showing all good after
• Restart the computer using this same disk image as boot disk, all good
• Execute MB2.COM, entered BASIC without problems.

So I would say that at least for this image, all worked perfect, reading and writing and I didn´t get a single disk access error.
Good job! Thanks a lot :)

[snhirsch]

Are you planning to send Jeff at HxC a bug report? Nice work, BTW.

[geneb]

@teiram If you're interested, I've got a lot of NSI images available for download here: http://annex.retroarchive.org/disks/northstar/ - they were created a few years ago using an AppleSauce. The ones I spot checked booted on a Northstar Emulator.

[snhirsch]

I'd like to give FF a shot on my Advantage, but am not clear on which revision of the firmware has this support. The artifact links above are broken. Is this now in the release code?

[ejona86]

@teiram, wow, you tested that quickly. That's great news! I'm so very glad the riddle is solved. Thank you so very much for all your testing.

@snhirsch, yeah, I was going to post on Jeff's forum. I've just been waiting on strong confirmation. I might get to it tomorrow.

To run this yourself, you can use the FlashFloppy 4.7a release. STM32 and AT32F435 Goteks are expected to work. AT32F415 will be highly dependent on your USB stick. Since now multiple people have seen this running (other than just on the Vector Dual-Mode controller), I'll probably create a wiki page for it. You need to use the config posted earlier in this thread: #400 (comment) . You might use the first post of https://forum.vcfed.org/index.php?threads/flashfloppy-hard-sector-testing.1242608 for reference.

[ejona86]

I shared my NSI index finding with Jeff, and he's posted a new version of HxCFloppyEmulator with fixed index pulses.