ibm.py read address mark failure needs accurate diagnostic data

[crestr1]

Have an assortment of disks that have different side 1 sector id's the IBM std ones are from 27. I have non std disks would like to know what is expected from the printout which may be expecting interleaved numbers just looking at the data printout.
It reads standard ibm ones without error side 1 sectors are 27 up
but this failing disk reads perfectly in ibm.scan mode

greaseweazle/src/greaseweazle/codec/ibm/ibm.py

Line 658 in b0af2e4

def decode_flux(self, track: HasFlux, pll: Optional[PLL]=None) -> None:

line 658 to print statement
Format dec.rx01
T0.0: IBM FM (26/26 sectors) from Raw Flux (222540 flux in 674.79ms) : Is ok side 0
T0.1: Ignoring unexpected sector C:0 H:0 R:22 N:0: now errors from side 1 follow
T0.1: Ignoring unexpected sector C:0 H:0 R:13 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:24 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:4 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:26 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:15 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:6 N:0
T0.1: Ignoring unexpected sector C:0 H:0 R:17 N:0

my diskdef entry
disk dec.rx01
cyls = 77
heads = 2
tracks 0-76.0 ibm.fm
secs = 26
bps = 128
gap3 = 26
rate = 250
rpm = 360
end
tracks 0-76.1 ibm.fm
secs = 26
bps = 128
gap3 = 26
rate = 250
rpm = 360
id = 27 # normal for ibm side 1 sector nos?
h = 0
end
end

[crestr1]

dump of end of failing read.txt

gw read myimage.img --diskdefs /home/ubman/Greaseweazel/greaseweazle-1.19/src/greaseweazle/data/diskdefs.cfg --format dec.rx01 --revs 4

[crestr1]

More data from my tests - simplified my layout - data attached in zip
cmd: "gw read motst.img --diskdefs /home/ubman/Greaseweazel/diskdefs.cfg --format mot.mdos3 --revs 4 --adjust-speed 360 --retries 0"
Recovery screen.zip
It is going after more sectors than it should: see attached.
it has totally lost the plot for specs of side 1: should be 26 128 byte sectors/ track, 77 tracks.It seems to have got into some interleaved mode for 52 sectors/tk, none of which it can read.
The good thing it shows the error marking on dud sectors works nicely in the .img file and it is bad-marking the correct no of bytes (0xD00) for the track it is not able to read.
Doing all this under Mint Linux now.

[crestr1]

While the ibm.scan mode works and makes usable .img files with perfect disks. I has no way of handling defective sectors, it merely omits their length from the final .img.
The important thing to me is needing to make complete intact .img files that just have a dummy value placed in bad sectors so the images are full sized and usable under an operating system's tool chains, the potential offsets to data pointed to by directory entries needs to be always preserved in the .img size structure.

[keirf]

Yes you will need to create a diskdef format for each disk format in your collection so that your reads can be checked against the format. In the original post here looks like perhaps your dec.rx01 format is not being picked up. Did you specify diskdefs=path/to/your/diskdef/file?

[crestr1]

Thanks Kier,
The attached recovery_screen .zip contains the latest results cmd file and diskdefs that all play together. I have narrowed my collection down to two disk types the ibm ones that recover perfectly if ok (S1 starts at sector 27) and the others that are all the same but i have no way of describing.
Is there a way I can set things to ignore the sector numbers and just recover the side 1 data sectors sequentially. this is I assume is what the ibm.scan mode does. but it still needs to know the size of the image it needs to create or it will screw up on bad sectors.
Can you suggest where i can put print statements in the code that actually executes.
the messages "T0.1: Ignoring unexpected sector C:0 H:1 R:10 N:0" is totally uninformative, what is R (expected or read from disk) and what is N which is always 0 ?

[keirf]

ibm.scan scrapes all the valid sectors it can find and writes them to an IMG file in ascending order of sector id.

The message comes from https://github.com/keirf/greaseweazle/blob/master/src/greaseweazle/codec/ibm/ibm.py#L681

You can actually run Greaseweazle out of the source directory by installing it with the -e option, like this, from the Greaseweazle source repository (eg. gw clone https://github.com/keirf/greaseweazle && cd greaseweazle):

python -m pip install -e .

With the above, you can run gw as normal but Python will be executed out of the source tree.

In this message, R is the sector id, N is the sector size (size is 128<<N, so N=0 means size 128 bytes).

[crestr1]

Hi Kier,
Is there a way in the diskdefs I can set things to ignore the sector numbers and just recover the side 1 data sectors sequentially.
Could you answer this please I'm having trouble following the manifestation of the variables:
the messages "T0.1: Ignoring unexpected sector C:0 H:1 R:10 N:0" : what is R (expected or read from disk, you don't say) and what is N's expected derived from which is always 0 should this be so ?
I sent in the zip the results I've gotten that suggest something goes wrong when one tacks the side 1 control on the end of the diskdef entry. I put 27 in id and it only works on IBM disks but I cannot find a number that works on at least one s1 sector of my other disks. This is strange.
every sector number is ignored :

T0.0: IBM FM (26/26 sectors) from Raw Flux (216219 flux in 667.16ms)
T0.1: Ignoring unexpected sector C:0 H:1 R:10 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:19 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:1 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:12 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:23 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:14 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:25 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:16 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:5 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:3 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:7 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:18 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:9 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:20 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:11 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:22 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:2 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:13 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:24 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:4 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:15 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:26 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:6 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:17 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:8 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:21 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:10 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:19 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:1 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:12 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:23 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:14 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:25 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:16 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:5 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:3 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:7 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:18 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:9 N:0
T0.1: Ignoring unexpected sector C:0 H:1 R:20 N:0
etc it's all in the zip i sent and its repeated twice even though the retry is off in the CMD
could be that the derivation of what is expected gets screwed up by what is on the disk which almost seems to be interleaved by 11 or in some strange way if it is its sector numbers being shown. This then said, why does the ibm.scan recover the data perfectly to an .img

[keirf]

CHRN are the sector header bytes. C = Cylinder, H = Head, R = Sector Number, N = Sector Size (128<<N).

In this case the correct diskdef would be the standard single-sided dec.rx01 as shipped with Greaseweazle, except with heads = 2.

[keirf]

If you just want to recover any sectors found, use format ibm.scan.

[keirf]

Apologies. Please share a raw dump of an offending disk (eg. some variant of gw read name.scp) and your attempted diskdef configuration, and I can advise how to fix up the diskdef configuration. Perhaps in future I can also add a command to automatically generate a diskdef from a good raw dump, but that doesn't exist right now.

[crestr1]

Your initial advise was along the lines of yesterdays:
"In this case the correct diskdef would be the standard single-sided dec.rx01 as shipped with Greaseweazle, except with heads = 2."
It did not work and you sent me a suggested fix that added a side1 tail to the diskdef to try, this made the only the genuine ibm disks recoverable when we gave it start sector number of 27, for the other disks it still failed. Here the long nightmare started.

What was also missing was an * --- " tracks 0-76.* ibm.fm" apply to any heads

This diskdef seems to work properly:
disk mot.mdos3
cyls = 77
heads = 2
tracks 0-76.* ibm.fm
secs = 26
bps = 128
gap3 = 26
rate = 250
rpm = 360
end
end

Need do some error management tests as final tests

[keirf]

FYI You should also be able to simplify the tracks line to tracks * ibm.fm

Everything else looks good. You don't necessarily need to specify gap3 but the inherited value from the default dec.rx01 format looks good. If you omitted it, Greaseweazle would pick a sensible value itself when writing back to disk.

[crestr1]

Hi Kier
it is still not working out scp, diskdefs, cmds etc in attached zip
"gw read tony.img --format ibm.scan" reads everything ok what i'm trying to get to to have bad sectors flagged in the .img's i create from this type of file
problem recoveries.zip
there are two different diskdef approaches there, neither work, just need to get one diskdef that works. side 1 becomes the problem
Would really appreciate some help please.

[keirf]

The attached SCP image files are not raw but re-generated via Greaseweazle using the format you specified to gw read. Therefore they look perfect. To supply proper raw flux dumps, do not specify --diskdefs or --format to gw read. Then given such a raw dump I can look at how to fix up your diskdefs, by passing your raw dumps to gw convert.

[crestr1]

Sorry Kier,
hope this is as required
[raw scp.zip](https://github.com/user-attachments/files/17054633/raw.scp.zip

[keirf]

Attached the required format added to your diskdefs.cfg as mot.mdosx. This format has sectors marked as h=0 on the second side, but sectors are still numbered from 1, rather than consecutively following the first side. It's a bit like two single-sided disks!

diskdefs.cfg.zip

[crestr1]

Really appreciate that Keir, at least it is making .img's now, will get them tested for
recovery accuracy of file structure tonite by feeding some to the mdos os disk utilities.
Thanks

[crestr1]

Recovery error indication performance test:
reading a defective disk: terminal screen and .img file attached
Recovery error performance.zip

[crestr1]

Hi Keir
Things are working really well now, really appreciate the help you've been.
Is it possible to set the greaseweazle up so that it behaves as a floppy disk to a floppy disk reader. I'm presently using USB HXC hardware to supply floppy HFE images to a device that once had a floppy disk. Every time it accesses drive 0 it reads whatever .HFE image is selected there. Can my greaseweazle's do this somehow ? If so I'll buy lots more of them.
The problem with the USB HXC is that it is read only access, be really great if something was available that gave both reading and writing. If it did this using disk images it would be really wonderful

[keirf]

One of my other projects, FlashFloppy, is an (almost) universal floppy-drive replacement.

You can put HFE image files on a USB drive, insert it into a Gotek programmed with FlashFloppy, and read/write those images on a host machine as if a normal floppy drive.

Be sure to get the latest Gotek model with AT32F435 MCU. This sort (my listing, but these are fairly widely available): https://www.ebay.co.uk/itm/126371834387

[crestr1]

Thanks Keir,
will buy two immediately for evaluation,
how many steps per rev is the rotary ?
have put some questions on eBay link above, am in Australia

[keirf]

Hi,

If you do wish to buy from me, contact via email (my address is in my GitHub profile). Or you may find other sellers selling the same as me you prefer to use. The listing can then be used to know what you're looking for.

The encoder has twenty detents per revolution. In addition, the firmware accelerates through the available images if you spin the rotary fast, so in practice you can zoom through far more than twenty images per revolution. It works much more conveniently than the original Gotek buttons, if the rotary is mechanically and aesthetically suitable for your system.

[crestr1]

Hi Keir,
Floppy Disk drive alignment uses of Greaseweazle.
Do you have any utilities for GW to support the disk head alignment adjusting. I have alignment disks from various makers that have track pairs adjacent to tk40 or 44 these are normally used looking at an oscilloscope but because the L and R patterns are sine-wave based i thought you my have some clever stuff that can work out the incoming wave peaks in a GW and report when the L&R alternate peaks are the same level for alignment purposes.
There are also other tests on alignment disks for track zero etc, If you have code that supports any of this I would greatly appreciate being able to use it..

[crestr1]

Hi,

If you do wish to buy from me, contact via email (my address is in my GitHub profile). Or you may find other sellers selling the same as me you prefer to use. The listing can then be used to know what you're looking for.

The encoder has twenty detents per revolution. In addition, the firmware accelerates through the available images if you spin the rotary fast, so in practice you can zoom through far more than twenty images per revolution. It works much more conveniently than the original Gotek buttons, if the rotary is mechanically and aesthetically suitable for your system.

Thanks, i went in via ebay and the transaction was refused. it offered me an alternative in USA supplier and a excellent price and fast delivery