Gaps in Software IEC

Happy New Year Commodore 64 lovers, welcome to 2023.

C64 OS is selling well. I couldn't be happier about it. I'm currently restocking for the 4th stock run and hope to be back ready to take orders again around the end of January.

Lots of stuff is going on. We're just doing some beta testing on the v1.02 and v1.03 updates to make sure these work without any issues, and then they'll be released for download from the Software Updates page.

v1.02 is very minor; It fixes a bug in the Installer Utility which is important for making sure that subsequent updates install properly. At the same time v1.03 is being released with several new features I've been working on. More details on this, when it's released, will come in another weblog post.

I thought I'd share these pictures; A day in the life of an OpCoders Inc. grunt worker, assembling C64 OS bundles and making them ready for shipment.

Many people have asked me if it is possible to use C64 OS with a 1541 Ultimate II+ or Ultimate64. This usually comes in the form of a question, "Is C64 OS compatible with the 1541 Ultimate II+ or Ultimate 64?"

This is a tough question to answer, because these devices are not just one thing with one feature. The Ultimate64 is a complete modern C64 replacement, so, can C64 OS be used on an Ultimate64? Of course! My last two World of Commodore demos have been delivered using an Ultimate64, along with both of my demos for Commodore Users Europe.

And a 1541 Ultimate II+ (its functionality is also built into Ultimate64) can be so many things: a 17xx REU, a GEORAM, a multi-SID chip emulator, a KERNAL ROM replacement, a speed loader cartridge, a source for Realtime Clock, an Ultimate Audio Module (for digital audio) and much more. So, is C64 OS compatible with a 1541 Ultimate II+? Of course! You can definitely plug a 1541UII+ into your C64 and also use C64 OS, and C64 OS can benefit from and use many of these features.

But that's not really what most people are asking. What they really want to know is, can C64 OS be installed on and booted directly from a 1541UII+ or Ultimate64? The short answer is, No. The longer answer is because these devices primarily provide a highly compatible modern implementation of a pair of 1541 disk drives (or now, 1571 or 1581 disk drives), which backend on disk image files rather than using physical floppy disk media. C64 OS cannot be installed on 1541, 1571 or 1581 disks because A) they are too low capacity, and B) they don't support subdirectories. All of this is outlined in some detail in the User's Guide, Chapter 2: Installation and in the Appendix: FAQ.

Inevitably, the curious person then asks me, "what about installing it on Software IEC?" and the explanation being, "because it supports subdirectories and access to the native USB device's file system." This is the point where it starts to become difficult to explain to people, in a nutshell, why Software IEC (at this point in time) is not sufficiently compatible with the other supported device families to be able to run C64 OS. The other device families are: CMD HD, RamLink, IDE64 and SD2IEC. All of which are remarkably DOS compatible with one another.

I sat down one night to explain in detail, via email to a C64 OS user (and who is also interested in developing software for C64 OS,) what exactly the gaps are between the current state of Software IEC and the other supported device families. This blog post is that detailed email, which I have reproduced here after clearing with him that he's okay with me publishing it. (Since it began as private correspondence.)

There is interest from the developers of the 1541UII+ and Ultimate64 firmware to try to bring Software IEC more into line with the other device families, but I'm not sure how long that will take. So I can't make any promises. But, at the very least, this blog post will serve as the canonical explanation for why C64 OS cannot at this time be installed on and booted from the Software IEC feature.

Hi!

I just spent maybe an hour simply playing around to try to refamiliarize myself with the scope of the difference in functionality.

First, I'm using an Ultimate64, and when I enter the menus the top bar says:

Ultimate 64 V1.41 - 3.10a

I'm assuming this is a version number and a build number. But you should be able to interpret it. Hopefully I'm not running some ancient version, and what I'm reporting to you isn't vastly out of date.

I'll try to make a list of individual incompatibilities. Though honestly, the delta is so large that it would make more sense to take a top down approach to describing how it should behave. But first, the issue of "should behave." Ultimate64/1541UII+ developers have to make a decision; Either they want to just do whatever they want, in which case it's fine as it is, but it's very different from everything else and I'll probably never add support for it in C64 OS. Or, they are interested in attaining a high degree of compatibility with all the previous mass-storage device families, in which case, we need to talk about this in "should behave" language.

Since I have just published C64 OS v1.0, and I've taken strides to support the DOS commands and path structure used in common by several other devices, I am advocating that you aim for the latter of the two choices in the paragraph above. And everything else in this email assumes you are on board.

It's important to note, we're not looking for "compatibility with SD2IEC", as though SD2IEC is some sort of gold standard. But rather to see previous device families in their relationship to one another.

Creative Micro Designs extended CBM DOS, (maintaining many clever similarities with CBM DOS,) to support multiple partitions of different types, subdirectories within native-mode partitions, and a partition and path syntax that can be used within commands to rename files, copy files, scratch files, open files, save files, and load files. It also included date/time stamps on files, and related options and RTC commands. This DOS was first released on the CMD HD in 1990.

They released the CMD FD-2000 and the CMD RamLink within the next 2 years. Despite differences in the hardware, from a DOS perspective all three are all virtually identical.

But the mid-late 90s the first version of IDE64 was released, followed by several new versions through the 2000s. Although IDE64 has some unique features, which are accessed with new commands that are only available on it, the core DOS commands, partition and path syntax were slavishly copied from CMD HD/FD/RamLink in order to maintain as much compatibility as possible. Including all the time stamping and RTC commands.

In the mid-late 2000s the first SD2IEC was released (or perhaps its ancestor, I'm hazy on this history as I was on hiatus from the Commodore world at this point.) SD2IEC slavishly copies the CMD HD DOS commands, partition and path syntax, including the time stamping and RTC commands, if an RTC chip is included in the hardware. Unlike the CMD devices and the IDE64 devices which have a custom file system designed for the needs of a C64, and which uses media which is not interchangeable with a PC/Mac, SD2IEC is special in that it backends on a FAT16/32 file system. This leads to some complication, and different solutions trading off different compromises is necessary. I have documented the various modes and the tradeoffs and compromises, in great detail, here:

https://c64os.com/post/sd2iecfilenames

What IS Ultimate 64/1541UII+ ?

When I first returned to the scene, I was totally confused by the name "1541 Ultimate II+". It took me way longer than it should have to figure out what this device actually is and what it can do.

It's hard to capture in a word what it is, because it's a fully independent, high powered computer, packaged into a compact cartridge. As such it can do a huge and growing number of things, most of which have absolutely nothing to do with the "1541" in its name.

(The Ultimate 64, in my opinion, has a much more accurate name.)

In its capacity as a 1541, (and more recently a 1571 and 1581) it is fantastic. In its capacity as an REU, and a KERNAL ROM replacer, and a freezer cartridge, and so on, it is exemplary. But what about as a mass storage device? If we take this to mean, what about as a replacement for a CMD HD, or a RamLink, or an IDE64, or an SD2IEC? The answer is, it is not even close to being capable of replacing one of these devices. But, with the right software design, could it be? Maybe. Maybe. And if it were, and you could simply use, say, an Ultimate 64 alone and get sufficient functionality that you'd be willing to leave your CMD HD or your IDE64 or your SD2IEC behind, would that be good? Yes, it would be fucking amazing!! That's my personal opinion.

At the moment though, sticking a tiny cableless SD2IEC into the back of an Ultimate64 is the closest we can get.

Devices and device mapping

In the screenshot below, these are, for lack of a better word, devices. One is a built-in flash drive, one is a build-in ram drive. One is a USB KEY stuck in port 1, one is a USB dongle stuck in port 0. These are different devices. Some are logical divisions of the built-in hardware (Flash and Temp), and some are actual physically separate devices, USB storage devices.

The C64 also has a devices concept. It has a bus that can support multiple devices, plus it too has some devices that are logical divisions of built-in hardware, such as RS-232, keyboard and screen. I think the logical thing to do here is to allow the devices in the list below (except for Net0) to be individually mapped to different C64 device numbers.

I could set Temp to device #11, Usb1 to device #12, and Usb0 to device #8, say.

Why? What's wrong with how it's done currently?

The way it's done now, all the devices are listed as though they are directories (file type DIR) in the root directory of a single device. And the problem with this, is that this root directory is not actually a directory at all. It doesn't behave like a directory. On every other mass-storage device you can put files in the root directory. C64 OS, as one example, requires that its system directory be installed in the root directory. But no one can put anything in this root directory.

It's clever that the directory lists as 0 blocks free. That explains to the user that "here is not a place where files can be put. Look, there isn't any space for a file." But this introduces just more inconsistency. There are no other C64 storage devices in which a directory listing shows several DIR type files, and shows a blocks free count, in which, if you use the CD command to navigate into one of those directories, the blocks free count will be different in the subdirectory. But that's exactly what happens here. When you change into a subdirectory of the root directory, you are in fact changing to a different device, with a different file system, and a different blocks free count.

Another problem is that, even when you have 0 blocks free, on all other devices you can rename a directory. But you can't rename one of these directories, because they're not directories, they're devices. And when you do try to rename one of these directories you get a 69, filesystem error. That's weird.

I'll get to partitions later. But for now, it's enough to point out an inconsistency. There are no other C64 storage devices where the first digit on the first directory line is a 0 at the same time that there are DIR type files in the directory. Why? Because on a 1541, 1571, or 1581, that number is 0 (for the zeroth physical drive unit) but these storage devices don't support subdirectories. On all the mass-storage devices, this number is the partition number. But on all devices with partitions, the zeroth partition is the system partition. You can't change to the system partition, so the first usable partition is 1. That means, partitions that contain DIR type files are never partition 0!

Another reason why zero is not usable to represent some arbitrary partition, is because in command syntax 0 is reserved to refer to the current partition. This is to accommodate software that uses zero while expecting to be running from a 1541.

Path syntax and the CD command

It's very important that path syntax, i.e., how paths are structured and how paths are embedded within commands, be compatible with the standards used by all the other devices.

On all the other devices, a path is structured like this:

The root directory is specified with two slashes. Subdirectories are added to this path with the name of the subdirectory plus a trailing slash (as shown above.) There are actually some inconsistencies between the devices, but there is at least a common structure that works across them all that can be used programmatically even if the user typing takes shortcuts.

On all the devices, it's acceptable to leave off the trailing slash if and only if nothing follows it in the command. If a full colon is used after which a filename is specified, then on CMD devices (and probably IDE64, not certain) the trailing slash on the last path component before the colon is required. Otherwise you get a syntax error. On SD2IEC, the trailing slash on the path is optional, even when you use a :filename after it. It would be bad to program with this SD2IEC shortcut, because you'd break compatibility with the other devices. But SD2IEC also supports having the final slash there, so, programmatically you're safe to always include the final slash.

The current directory, at the start of a path, is specified by a single front slash, rather than two front slashes. Therefore, if we want to move into a subdirectory of the directory we're in, on all other devices you use one slash, like this:

cd/audio

But on SoftIEC a single front slash is used to mean the root directory, so this doesn't work in the example below. The first single front slash should mean current directory, but instead it's looking for audio in the root directory, doesn't find it and so no path change occurs.

SoftIEC uses ./ (dot slash) to represent the current directory. I don't think this causes any problems, as this doesn't collide with anything on the other devices. As long as you can also use / (slash) as the current directory too, for compatibility.

An alternative way to change into a directory from the current directory is to put the directory name after the colon as though it were a filename. However, on all the other devices, only a single directory name can follow the the colon. On SoftIEC we can issue this,

cd:audio/tuneful 8

And that changes into both directories. This is inconsistent with the others.

To go up a parent directory, all the other devices use the back arrow.

cd←

However, on SoftIEC the back arrow seems to be used to take you to the "default directory" as specified in the settings. I think it's a neat idea to have a command that takes you to a defined default directory. It's a nice little extra that SoftIEC can offer, but it should use a command that is not the command used on all the other devices to go up one parent directory.

SoftIEC can use two dots in path to go up to the parent, and then following a slash go back into a child directory of that parent. This is actually something that none of the other devices can do! So, it is a nice capability (more modern) of SoftIEC. The other devices which use backarrow to go up, cannot use the backarrow in a relative path to go up and then go back down into a different directory.

The only problem with using .. (dot dot) is that on the CMD devices (and IDE64 I think) there is no problem with creating a directory name whose name is ".." And why not? It's just a name with two characters. Thus, on a CMD HD you can issue a command like this:

cd/testdir/../test/

And the .. is actually the name of a directory that's in testdir, and it navigates into that directory. There is an inconsistency here with SD2IEC because the underlying file system reserves two dots. This creates the weird effect that, if you try to create a directory named with two dots, you can get a 63,file exists. But when you list the directory it doesn't show. If you try to scratch or remove that directory, you get a 62,file not found. It's probably best to avoid directories named ".." on CMD devices for this reason.

It is however possible to create a file named ".." on an SD2IEC, but only when in C64-compatibility mode (e02) because it creates the file with a .P00/.S00 wrapper, which makes it acceptable even on the underlying FAT FS.

Additionally, it should be possible to use these path components in commands, but that doesn't seem to work. I've tried using the relative path "../" in the copy command, but that didn't work.

Then I realized, that the copy command isn't working properly at all, it seems. As you can see in the next screenshot, rather than create a copy of kernal.bin in the parent directory, it has created a copy in the current directory, but it's called kernal.bin.bin !! I have no idea why!

Maybe the copy command is just broken. Even just copying a file within the current directory, as shown below...

I end up with a new file called kernal2.bin.bin in the directory. This seems to be a problem with the copy command, and therefore maybe doesn't belong in this section about path syntax, but I noticed it while trying to copy a file to an alternative path.

Next I tried to copy a file to an absolute path, (using SoftIEC's single front slash for root, of course) and that didn't work either. It didn't make a copy of any kind.

c/usb0/:kernal.bin=kernal.bin

It's almost as if the path is just being ignored completely. I don't know, all of these commands should work.

Pattern matching in directory listings

Pattern matching on filenames seems to work quite well. For example, * can be used to represent 0 or more characters of any kind. ? can be used to substitute a single character. SoftIEC even supports more than one * which the other devices do not. I don't think this inconsistency will lead to any problems, it's just a nice extra on SoftIEC.

However, matching on CBM file type seems not to work at all.

That should be listing the SEQ files.

It doesn't seem to work at all. That should be showing the PRG files, and then B is traditionally used for "branch" which works on all the devices, but doesn't work here at all. D is used as an alternative to B on SD2IEC, for DIR, but that's not working.

Directory headers

On CMD devices and IDE64, each directory can have a header name that is independent of the name of the directory in its parent directory. This is because, like the 1541/71/81, each directory has a special directory header block that stores the directory's name.

On CMD and IDE64 devices, a new subdirectory's header is by default set to the name of the directory in its parent, but the R-H command can be used later to rename the header. The R-H command isn't supported by SD2IEC because the underlying file system doesn't support a custom directory name. Instead, SD2IEC always shows the directory header name as the name of the directory in its parent. This is okay, because it is at least the default behavior on CMD and IDE64 if you don't use R-H.

But on SoftIEC it tries to show you the current path. This is inconsistent for the sake of doing something clever, but it breaks the expectation. C64 OS in particular uses the directory header when creating a favorite, for example. The name of the favorite is taken from the directory header name.

At the very least SoftIEC limits the header length to 16 characters, unlike VICE FS, which does something much worse, it puts the full path in there even when it blows out way past 16 characters. But, the downside on SoftIEC is that as soon as the path is longer than 16 chars, it's only showing you some partial path anyway. I would change this behavior to SD2IEC's way, in the name of compatibility.

Partitions

This is a big subject.

Partitions are handled very poorly by SoftIEC. Let's start with the partition directory.

On all the other mass-storage devices, there is the ability to list a partition directory, with the command $=p.

The first number before the partition directory's header line is a count of how many "user" partitions there are. In this case 2, because the 0 System partition of type sys (as mentioned earlier) is not user accessible. Following system is the list of partitions, by name, with a type of "nat" if that partition type supports subdirectories. The typical block size of a directory entry is used for the partition #. The partition directory does not list a blocks free count.

This behavior is the same across all the other device families. But is totally unsupported by SoftIEC. There is also the ability to filter the partition directory by partition types, but since SoftIEC doesn't support the partition directory, that's a moot point.

Next, all the other devices support a current partition, in the same way each partition supports a current directory. You can change the current partition with two commands cp and cP. The former is followed by a PETSCII number (1 to 3 chars from 1 to 255), and is designed to be easy for the user to type. The latter is used with a single binary byte (int value 1 to 255), and is designed to be easier to use programmatically.

Also as mentioned earlier, in a normal directory listing, regardless of subdirectory, the first number before the directory header is the partition on which this directory is found.

As you can see in the screenshot above. The first directory listing is on SD2IEC, the number 1 before the header is because this is on partition 1. On SoftIEC, the second directory listed above, the "partition" number is always listed as 0.

On all the other devices, the partition number can be used (optionally) as part of the path. This is a direct analog to the dual disk drives from CBM that use the number 0 or 1 to reference the different drive units. On all the mass storage devices this number can be 1 to 255 for the partition (or 0 for the current partition). Each partition is like a drive unit.

If a command omits a partition number the current partition number is used. If a command uses the partition number but omits the path, the current path for that partition is used. For example:

c2:myfile=3:myfile

That copies myfile from the current directory on partition 3 to the current directory on partition 2. This syntax can be used with the scratch command and the rename command too.

s2//some/path/from/root/:sratchme

s3/relative/path/:scratchme

s4:scratchme

The first scratches a file on partition 2 with an absolute path. The second on partition 3 with a relative path from the current path of partition 3. The last example scratches a file in the current directory of partition 4.

None of these work on SoftIEC because partitions are just not handled in a compatible way at all.

So, how are partitions handled by SoftIEC?

I've formatted my usb key in port 1 with 2 partitions. And the usb key in port 0 with 1 partition. On the device with only 1 partition, it hides the partition from the path altogether. The path thus becomes /usb0/audio/tuneful 8 where "audio" is a directory in the root directory of the first partition. As you see from that path, no partition is referenced at all, probably to shorten the path. Why make the user include a partition in the path when there is only one?

But as shown in the screenshot above, when listing the directory from /usb1/ it shows another directory with 0 blocks free! Oh No! Now we've got two levels of directory into which files cannot be written. It lists the two partitions as though they are part of the path. All the problems of listing the devices in the root directory can be restated here.

Next, the partition names are not even used. I named these C64OS and IMAGES when formatting them, but they get rendered as part0 and part1. No idea why.

They also get listed as DIR type, which they aren't. This is all downstream of just not handling partitions the same way as all the other devices. So partition syntax is just not supported. Instead SoftIEC sort of kinda hacks partitions into the directory structure.

Filenames

Filenames are a mess.

IDE64 and CMD devices don't have to worry about this, because their media is not interchangeable with a PC/Mac. The filenames are always in PETSCII, and never need to worry about crossing the names back and forth. SD2IEC on the other hand uses FAT FS and has all the complications of moving files between the two worlds, Commodore and PC.

I cannot get into the details here of how SD2IEC handles all of this complication, but as linked above, I wrote a huge and detailed blog post outlining the different modes, and the trade offs of each mode in the blog post, Understanding SD2IEC Filenaming.

In short, SoftIEC doesn't seem to do anything clever at all. The names entered on the C64 are just stored in PETSCII, but when viewed from a PC/Mac, they can't interpret PETSCII so the characters are just whacky and broken and uneditable.

I suggest deeply reviewing the solutions provided by SD2IEC and aping them. They are very well thought out. Poorly documented!! But very well implemented. And hopefully with third-party documentation (such as my blog post above) that solves the problem of being poorly documented.

Attention to how the CBM file types are stored, and the various means by which they are stored, on SD2IEC is important.

Timestamps

I know that Ultimate 64 exposes the RTC via the UCI interface. And that's fine. C64 OS has a driver for the UCI RTC.

However, all the other devices support the family of RTC commands via the device's DOS over IEC. I think you should review the SD2IEC user's manual here:

https://c64os.com/post/sd2iecdocumentation#realtimeclock

This family of commands is optional, of course. BUT, all the other mass storage devices support these commands. So, it would be nice.

Further, in the directories section of the SD2IEC user's manual:

https://c64os.com/post/sd2iecdocumentation#directories

Besides more information on how the partition number and path syntax work, which you can review, it also has a section titled "Time and Date Stamped Directory Listings". This shows all the commands for listing the date and time stamps in different formats for directory listings. And also the date and time stamp filtering options.

These are all used by C64 OS (and probably other software). And these are all 100% consistently implemented, including the format of the date and time output and the input options for filtering, on all of the other mass storage devices. For any sort of advanced software, (for example, I include with C64 OS a recursive incremental backup program, run from the READY prompt,) proper support for the date and time stamping is essential. And it is a core part of all the other mass storage devices.

Miscellany

• Seeking within files. This is not supported by CMD devices, but it is supported by IDE64 and SD2IEC. This is described under the heading "Positioning (seeking) Within a File" in the SD2IEC User's Manual.

https://c64os.com/post/sd2iecdocumentation#filemanagement

• Changing the device address with u0> command. This is supported on all devices going back to ancient times, before the 1541. This produces a syntax error in SoftIEC.

• Warm reset seems to work nicely, "ui", that's great! Good identifier is returned too. Easy to identify the device and its DOS version. However, cold reset and hard reset "uj" and "uJ" seem to lock up the bus. Needs a STOP+RESTORE to recover.

• Direct access commands. I haven't and don't have time to test these. But I'm 99% sure none of this is supported. I suggest reviewing that section of the SD2IEC user's manual under the title "Reading and Writing Data with Direct Access".

• Fastloaders. SoftIEC does not seem to be accelerated by JiffyDOS. Loading data off the SoftIEC is excruciatingly slow. IDE64 and RamLink have their own means of being very fast as they communicate over the cartridge port. But all the other devices have support for JiffyDOS over the IEC bus.

Wrap up

In short, I started by saying that it would be easier to start with a top down approach rather than trying to state every individual thing that needs changing to come up to what the other devices support.

The main issue is with how "devices" are treated, how "partitions" are treated within devices, and then the path syntax with partition numbers and so on. Until these are more in line with the standards used across all the other devices, it's not worth looking into supporting SoftIEC. It's just too radically different.

After that, it's innumerable little things. Directory filtering, timestamps, filenaming compatibility between C64 and PCs, the broken copy command, and then we get into the less well known things, seeking, direct access, buffer pointer, etc.

I hope this gives you an idea of the delta. And, to whichever developer you're going to send this on to: I know it's a lot of work. I don't know if you've got it in you to do all of this work, but, as I said, if SoftIEC was so good that we could just flip it on and all of a sudden Ultimate64 or 1541UII+ could replace a CMD HD, IDE64 or SD2IEC, that would be just manna from heaven.

Do you like what you see?

You've just read one of my high-quality, long-form, weblog posts, for free! First, thank you for your interest, it makes producing this content feel worthwhile. I love to hear your input and feedback in the forums below. And I do my best to answer every question.

I'm creating C64 OS and documenting my progress along the way, to give something to you and contribute to the Commodore community. Please consider purchasing one of the items I am currently offering or making a small donation, to help me continue to bring you updates, in-depth technical discussions and programming reference. Your generous support is greatly appreciated.

Greg Naçu — C64OS.com

Want to support my work?

C64 OS version 1.04 is now available

The best way to help C64OS.com grow and continue is to purchase a commercial copy of C64 OS version 1.04.

Join the growing C64 OS user community. Take advantage of the powerful and useful features available now, and get access to run new and future software made for C64 OS.

Learn More Order Now

C64OS.com is hosted on a Digital Ocean droplet.

Digital Ocean is a cloud hosting platform, built for developers. They make it simple to get started. They have intuitive administrative tools and reasonable prices. And they can scale as your needs grow.

If you sign up with this referral link, you'll get a $100 credit, and after you've spent $25, I'll get $25. If you are looking for a cloud platform to host your website or web app, this is win-win.

Sign up for Digital Ocean with this refereral link.
Save money and help C64OS.com at the same time.

No, thank you.

Want to support my hard work? Here's how!