Autotuning - Higher TPI, more cylinders

[auron-muc]

I would like to start by saying thank you for a fantastic project, and making it open sopurce.
I started toying around with increasing the TPI for a 5.25" floppy some time ago, and recently revived that project. Initially I had tried to use microstepping, but switched to changing the stepper motor recently.

My goals with this are

• Auto-tuning for reading mis-aligned disks, saving data that previously could not be read
• Ability to read/write 100 TPI disks
• Possibly creating/cloning aligment disks

My current setup has "1270 TPI" ( somewhat limited repeatability due to cheap stepper and mechanical play in components), and already works remarkably well.
To support the dramatically increased cylinders, I modified my greaseweazle firmware and the python tool, only to find my changes were (almost 1:1) already implemented in the latest development version of greaseweazle-fw.

Is anyone else tinkering with a setup like mine, or is this due to something else?

I am aware the usability will be a niche within the niche of retro-computing, but I hope some modifications will be able to flow back into the main gw repo.

[keirf]

There was recent groundwork to support MFM (and early RLL) HDDs. These can have up to 1024 cylinders, hence the seek command was extended. There are still a cylinder limit in the firmware, for how hard to try to get to cylinder 0. That was either going to be increased in "HDD mode" or changed to a timeout.

Interestingly the data rate is far higher than Full Speed USB, so the data cannot be streamed in real time as with floppy disks, but the actual data per track is similar to a HD disk. So Greaseweazles with enough RAM (mostly V4 with an AT32F403A MCU) can buffer a whole track of flux.

[keirf]

More than the hard limits, I'm concerned about how to treat seek-to-cylinder-0, which currently just steps 256 times. I was thinking about making this a timeout, and getting rid of the +100 check altogether. It's not correct for any regular 40-cyl or 80-cyl drive anyway!
The timeout could be two seconds, which might be enough for you anyway. And if not we could add it to PARAMS_DELAYS (which ought to be renamed to PARAMS_TIMINGS or similar, with no change to the ABI).

[keirf]

That said your approach isn't bad either.... And max_cylinder could be default 256 and applied to seek-to-0 too. Do you actually really care about min_cylinder right now? Are you microstepping flippy-modded drives?

[keirf]

So basically if we can decide whether or not flippy min_track is needed, I will take your idea and clean it up for master. It seems silly for you to require private firmware patches for your work.

[auron-muc]

Well yes, I am, sort of. 100 TPI disks (CBM 8250 / SFD1001) do have an offset for the back side, though it seems to be something else than the -8 tracks used for c64 disks, and they're not flippy. But you're right, I possibly didn't think this through enough.
I currently just dump all tracks and subtracks in raw format, then renumber and sort them into 1/nth track offset "disks", because all tools just assume a certain number of tracks - obviously.
If autotuning works, I would not need to do that at all, except for high-res dumps of alignment disks.

[keirf]

All double sided disks have an offset between back and front sides. I suppose it makes sense it could be different at 100tpi though. Usually it's invisible because drive heads on a double-sided drive are also offset, and it only becomes visible if you read a flippy disk in a double-sided drive.

[auron-muc]

So basically if we can decide whether or not flippy min_track is needed, I will take your idea and clean it up for master. It seems silly for you to require private firmware patches for your work.

Thanks, that sounds promising. But hold back for now - I realized I would have to implement something for the non-flippy negative cylinders for the SFD1001 drives as well.
Let me test how that goes, then I'll gladly clean up to standards you would be expecing.

[keirf]

Regarding negative domain, if you do need it, I don't know how far you've investigated but for most drives you would need "flippy mod" anyway: Most drives seek to 0 on power on, and will not seek out beyond 0. Hence the need for a sensor defeat to fool the drive into stepping further out. These features of the trk0 sensor apply to most drives.

[auron-muc]

Regarding negative domain, if you do need it, I don't know how far you've investigated but for most drives you would need "flippy mod" anyway

I'm aware, thanks. That flippy modification was sort of the entry point into modding the drive further.

Another option on a modded drive would be to move the track-0 sensor back as far as mechanically reasonable.

Yes, I did consider that, not sure if I could move it back all the way on my panasonic drive though - and it very precisely aligns with the real 0-track for my normal 96 TPI operation right now, not that it matters much.
I was not planning to auto-tune accross several tracks either, it's a possibility though.

All double sided disks have an offset between back and front sides.

Depends on how you see it ;-) Flippy disks will actually not have any physical offset. Back and front will be identical on disk.
On the other hand, on a standard pc floppy drive back and front will not have an offset when reading pc disks. Seek to track 4 and you will get track 4 on head 0 and head 1. But there's a physical offset on disk, because the read/write heads have a spacing of 8 tracks.

Yeah, I've investigated a small bit :)

[keirf]

Okay, then it seems that a maximum_cylinder and flippy_minimum_cylinder are both necessary and sufficient?

[auron-muc]

Yes, for high-resolution dumps, that would be necessary and sufficient. For autotuning it's not strictly necessary.

Maybe let me quickly outline what I have done:

1. I flippy modded my panasonic ju-475-4
2. I built a small adapter board with my own stepper driver and an Arduino nano, that will sample the output from the panasonic floppy motor connector to detect steps.
3. The Arduino can be interfaced via serial port, and I can change the TPI or the offset on the fly.
4. I replaced the stepper motor in my floppy with a 0.4mm advance/full turn, instead of the original 2.65mm advance

In the end it would have been easier to just tap into the shugart interface, but I wanted it to be a complete drop-in solution initially. (using microstepping, not actually replacing parts of the drive) Also, the panasonic board will not work if the seek-to-track-0 on power on does not happen, but that could be driven externally as well, of course.

The Idea with the autotuning is, to go to the regular center-track, as usual, and collect one or two revolutions of data, then do a histogram (or heatmap) of the flux data to see how narrow the bands are. Like, if you look at the bitmap output from kryoflux dtc -pg2 option, basically count how many white pixels there are. The more white, the narrower the bands with actual data are, and we can assume we're better on-track.
Then go 1/4 track back and sample/evaluate again, etc. Where the best signal is, do the real sampling.
Then remember the offset for the next track, but keep an eye on the histogram, if that get's significantly worse, do another fine-tuning.

(off-track to on-track with subtrack-stepping)

The python CLI tool will need modification to talk to the motor controller arduino (to do the offset stepping), as well as the algorithm as described.
Now, theoretically, the cli tool could also be rewritten to just assume a 96*8=768 TPI drive, and just to skip several tracks ( I think it can do that already), but if the offset changes things will get messy with the counting. As I'm not sure how many people would reproduce my hardware mod I kind of doubt it's worth the additional complexity in the mainstream greaseweazle tool.

If everyone thinks it's great, we can of course build a greaseweazle++, and get rid of the arduino, just add a stepper driver and some IO for the track-0 detection.

[auron-muc]

And a third option - If the CLI tool can individually access all subtracks, I would not need to add the algorithm to the tool at all. It could also all be an external wrapper, calling the CLI multiple times. Less performant though probably.