NEWS, EDITORIALS, REFERENCE

Menu/Status Bar: File and Memory

I'm feeling good that I'm back on pace with my posting rhythm. This is my third post in August. Today I just wanted to show off a system feature that I built in last month or so. It's the system status bar, as well as new global, configurable keyboard shortcuts. And this also includes some work on the File and Memory modules to support the status bar.

Let's just dig right in. I was anticipating that this would be a briefer post than some of my others, but it's me after all. I'm very wordy and detailed. So, this may turn out to be longer than I'd originally expected.

Drive Status

You know, it's funny, I've been using a C64 since I was 9 years old, and it never fully dawned on me until just recently the importance of reading the drive's error channel.

As I discussed at some length in the post from earlier this year, KERNAL, File Refs and Services, the C64 itself really doesn't know anything about files or file systems, or anything else about disk management. I argued then that this is a strength, when you consider the nature of the system. The machine is small, with limited RAM and even less ROM. And yet, it can easily make use of modern storage media like CD roms, SD Cards, CF Cards and hard drives. This is because the DOS is entirely offloaded into the device.

The C64 merely opens a connection over the serial bus, and sends commands over the command channel. This allows the C64 to read and write files, and issue commands to the device's DOS to let it handle the nitty details of manipulating the data on the storage medium.

Consequently, though, the C64 itself doesn't know very much about error states in the devices it talks to. There are a couple of conditions it handles. For example, if you try to access a device number that doesn't exist, well, the C64 knows about that. And if you're reading in bytes and you hit the end of the file, the C64 knows about that too. But, imagine that what you really want to do is change a directory, then create and open a file, write some data to it, and close it. To do this, the C64 must send the change directory command, over the command channel, prior to trying to open the file. If all goes well, the file will end up being created inside the subdirectory.

But what happens if things don't go as planned? What happens if, for some reason, the directory doesn't exist? What will happen is that the C64 will issue the change directory command. That will produce an error internal to the drive, about which the C64 knows absolutely nothing. If the C64 then simply proceeds to open/create the file, it will be created in the wrong place. Even worse, if you open with an overwrite flag, who knows, you might accidentally overwrite a file with the same name that's not even supposed to be there, because you're not where you think you are.

There is only one way to deal with this: Read the error channel, religiously.

It's not like this hasn't been known for 40 years, it's right there in the User's Guide. But, when you actually get into the details of programming, somehow things that have always been there but that you've never thought much about suddenly pop up become noticeable.

A quick aside. Here's a neat thing I learned. The structure of the data provided by the device over the error channel, is intentionally formatted to work with the BASIC interpreter. That makes sense, but it's kind of a cool observation.

So, what do I mean by that? If you have some variables, some strings and some numbers, and you open a file and PRINT# those variables to disk, they end up structured much like data statements. The numbers get converted into strings of PETSCII. And the data fields are separated by a comma and a space. You can open the sequential file with a text editor to see that this is the case. They're written to disk that way, because when you read data in using INPUT#, it expects the data in this format, and parses them back into BASIC number variables and strings.

All of a sudden it makes sense why the error channel is structured like this: "00, OK, 00, 00". And those zeros are actually PETSCII zeros, byte values of $30. It's structured just like any other BASIC data. This actually makes it slightly less convenient when you're coding in assembly, but at least it makes sense why they designed it this way. The drive's DOS produces structured data that fits hand-in-glove with how the BASIC rom works. It's very streamlined.

The Error Channel

Here's another interesting tidbit I learned, just in the last couple of months, while reading the 1541-II User's Guide, and comparing it with the CMD HD User's Guide, to see how they differ and how they are the same.

Each drive supports opening some combination of files depending on the file type, and how much RAM is available in the drive. The 1541, for example, has only 2 kilobytes of RAM. Those are addresses from $0000 to $07FF. A very small range. Surely this was because in the early 80s memory was expensive. This provides a grand total of just 8 pages of memory, where a page is 256 bytes big. The zeroth page is its 6502's zero page, used for workspace memory, just like in the C64. The first page is the 6502's hardware stack. After that is one more page for workspace, where speed is less important than whatever is being stored in zero-page. You can read all about the memory map of the 1541 here.

That leaves 5 more 256-byte pages, $03, $04, $05, $06 and $07. One of these is used to hold the BAM, the Block Allocation Map. Leaving 4 buffers left over for files. The block size of a 1541 disk is 256 bytes. The way it works, in a tiny nutshell, is that one of those 4 buffers is assigned to an open file. When you want to read a file, the DOS controls the mechanism and loads a full block off the disk and into one of those buffers. Workspace memory then maintains a read offset into that buffer. As you read characters one at a time over the serial bus, the drive moves the cursor through the buffer and sends you the next character. After you read the last character it loads the next block off the disk into that same buffer, and so on. Similarly with writing. When you write a file, you actually write into a buffer one character at a time. When the buffer fills up, the whole buffer is written to a sector on disk, and the buffer is emptied for you to write more data to it. The BAM is in memory, and it is used to find free sectors on the disk. At a certain point, the in-memory version of the BAM gets written to the sector on disk that permenantly stores the BAM.

Different file types, however, require differing numbers of buffers. For instance, if you open a REL file, it instantly consumes all of the buffers, and you can't have any other files open, for read or write, at the same time. In the best case scenario though, reading a SEQ file only uses one of the 4 buffers. The DOS allows you to have open 4 different SEQ files for read all at the same time.

This is, of course, another way for an unexpected error to occur that your software needs to be able to handle. What happens if the OS, or a Utility, or something, already has open a couple of files, and the application wants to open a file or two. It is easy to imagine that the application could unexpectedly get an error when trying to open a file, because there just isn't enough free RAM in the device to handle another open file.

But, look at it the other way, you may in fact have more than one file open. Maybe a Utility needs to read one file while writing data out to another file. Simultaneously the application has a data file open. That's okay, we can do this.

We still need to be able to read the error channel though. Here's another interesting thing I learned just recently. When you close the error channel, the DOS automatically closes all of the currently open files!! It does this on purpose, of course. Partially because if something goes south with your software, and you lose track of the open channels, you would also lose the ability to close those files. For example, in your basic program, if you open some files, but then you crash the computer and you end up at the ready prompt again, the green active light may still be on on the drive, but your program is ended. You cannot necessarily issue close statements. The recommended procedure is to close a logical file number, in case it's in use. Then open the command channel with that same logical file number, and then close that logical file number again. This will guarantee that the command channel gets closed, and the DOS will automatically close all the other files, and the green drive light will turn off. This should prevent the formation of splat files, files that were improperly closed.

This behaviour is essentially the same on the 1571 and 1581, as well as the CMD drives. The CMD HD is structured very much like a beefed up Commodore 15x1 drive.

This may be fine for a BASIC program that has unexpectedly ended. But it's not great for an operating system that is expected to have different and unrelated bodies of code all working with files: System code, Utility, Application. If the Application decides to close the error channel, it could unexpectedly be closing files that the Utility or System code have open and are still actively working with. That's bad.

The programming information found in the User's Guide is heavily oriented around the casual user, and thus, around use in a BASIC program. The nature of your BASIC program and that environment is that you and your code are the only thing that will ever have a file open, guaranteed. The KERNAL will never just spontaneously open a file, such as to write a log, or read some configuration, or write to a data cache. It's just not that sophisticated. Interestingly though, your BASIC program itself may become quite sophisticated, and may itself use multiple open files at the same time. Therefore, the recommended procedure, in the User's Guide, is to open the error channel at the start of your program and leave it open until the very end. That's very interesting advice.

C64 OS is, in a certain sense, like one enormously large, super-complex, program. And so that's what I do. The booter runs the drive detection routine. This builds a table of drive type codes at indexes in the table that correspond to their device number. So, a 41 at index 11 means there is a 1541 on dev 11. And so on. Devices that support subdirectories use drive type codes with bit7 set. There are only 30 device numbers, and storage devices only 23, from 8 to 30. C64 OS reserves dev 30 for device number juggling (a trick I picked up from Wheels), so that's just 22 devices. The C64 KERNAL (which C64 OS backends on) has a limit of 10 logical files open at the same time. And Logical file numbers can run from 1 to 127. (above 127 weird things start happening, extra carriage returns start getting sent. I think that was related to supporting something special about printers.)

What C64 OS does, is, after detecting devices, for each recognized device type, it opens an error channel where the logical file number is 96 plus the device number. In practice, storage devices don't have a number less than 8, so the first logical file number for error channels starts at 104. Using 103 is safe, although, you don't need to worry about that, because C64 OS also dynamically tracks and allocates unused logical file numbers for you. The last device number, 30, would be LFN 96+30 or 126. That's because 127 is reserved as a temporary LFN for loading binaries.

Here's a table of how that works out:

Special Dev Numbers / Logical File Numbers

Thus, if you are reading from device #8, Logical File Number 104 is the open error channel for that device. However, you only need to know any of this if you happen to be doing manual or low-level disk access. In practice, if you use C64 OS file references, and you use finit, fopen, fclose, fread and fwrite, then after every operation that could theoretcially produce an error, that device's error channel is automatically read. Super cool.

Besides the table of auto-detected devices and types, logical file number allocation, pre-opened error channels, and automatic status reading, there is also a drive status buffer. It holds the device number of the last read status, plus the standard status code, human readable message, track and sector codes. Additionally, the status code is automatically parsed from its PETSCII default into a 1-byte status int. Everything above 20 is considered a genuine error, and causes the standard file routines, (fopen, etc. mentioned above,) to return with the carry set.

The Status Bar

I'm feeling pretty good about the File module in C64 OS. It does a LOT of work for you. As long as your program passes around pointers to C64 OS file references, the system makes it easy for programs to access files on storage devices, across file systems, partitions and subdirectories. But it goes a lot further. You never need to juggle logical file numbers, they're assigned and stored in the file references automatically, and you don't need to read the error channel or parse its results manually.

The next problem is, though, what to do when an error occurs. I thought about displaying a sort of pop over that would only appear when a true error occurs, a status code greater than 20. But then, that's not a great solution. Many applications might just be testing the drive and the fact it generates an error might be expected behavior. You certainly wouldn't want the OS to interject. And the flip side, what would you do with all the feedback that is less than 20, not an error, but you might still want the user to see.

Principally, I don't want every program to have to handle displaying disk status its own way, because that's a ticket to generate totally inconsistent results. Apps should feel like they belong to the same coherent system. The solution I arrived at is to have a system controlled status bar.

The status bar runs the bottom of the screen, opposite the menu bar. It is controlled and drawn by the menu module, and thus shares a screen layer with the menu system. They draw above everything. So if the application draws something into the bottom row, it will get covered over by the status bar. Although the application can check the system draw flags to see if the status bar is on and adjust itself. More on this below. Here's an interesting trick that the menu and status bars do: When dragging a utility by its title bar, if you drag the mouse over the menu bar, pixel rows 0 through 7, the menu layer's mouse event handler rewrites the event to have a vertical position of 8, and then allows the event to propagate. This causes the Utility to remain responsive to horizontal mouse movements while the mouse is over the menu bar, but stops it from being dragged under the menu bar. Try it in macOS, it works exactly the same way.

The same happens with the status bar now too. If you drag a utility down and your mouse overlaps the status bar, the title bar of the utility stays stuck above the status bar rather than getting lost underneath it. It feels very natural in practice.

A drive error message can be quite lengthy, up to 25+ characters ("SELECTED PARTITION ILLEGAL"), plus the device number, status code and track and sector. You need almost the whole width of the screen to be able to show the drive status properly.¹ Whenever the drive status changes, which is managed by the file module automatically reading the error channel, a system service call is made to notify the system that the status is out of date. I'll return to this below.

Available Memory

Once a system wide status bar was available, I thought it made sense to also have it display available memory. The page allocation map is easy to read, and there was actually already a routine in the memory module that returns a count of available pages.

The memory module has been updated such that calls to allocate or free pages now make the same system service call as the File module, to notify the system that the status is out of date.

Drawing the Menu bar and Status bar

I've shown before, in the post Menu Systems, a comparison, that the menu bar can have its visibility programmatically toggled. This could be very useful in a number of different situations. Maybe you want to create a presentation app, and the screen will be projected large for a crowd to see. You'd want to hide the menu bar so they are just seeing the content of the presentation. Or maybe you've got a game that wants to use the full screen for visual effect. Or, more likely, you just want to recover screen real estate for your own productivity. For all these reasons and more the menu bar can be hidden.

So, how does it do that?

The system keeps track of a number of drawing related states via a bitfield called DrawFlags. There are flags for the visibility of: menu bar, status bar, clock, and CPU busy. As well as a modal flag for the top layer.

When the file or memory modules make a service call to notify a change of status, the status bar draw flag is checked. If the status bar is off, the notification is ignored. If the status bar is on, the layer is marked for redraw, forcing the status bar to be redrawn during the next redraw phase at the end of the main event loop. I should point out again that flagging for redraw is much more efficient than actually triggering a redraw when, say, a memory allocation is made. A routine could make many page allocation calls in one iteration of the main event loop, and the redraw will only happen once at the end, and in the correct order relative to the other screen layers.

Keyboard Shortcuts

The screen module is where the main event loop is implemented. It checks for events and distributes them via the dispatch tables of the screen layers. This process has been outlined in much more detail in a post from last month, Command and Control.

I've added a system service call into the main event loop to process system-level (aka Global) keyboard shortcuts. The service module loads the keyboard shortcuts from the main config file at boot up, so these keyboard commands can be configured.

At the time of this writing, there are only two shortcuts.² One which toggles the visibility of the menu bar, one which toggles the status bar. These are very simple routines, they just EOR the draw flag and then call MarkRedraw. These keyboard shortcuts are intercepted before key events are handled by any screen layer, so they can't be programmatically blocked. The upshot is that the user is always in control of whether the menu or status bars are visible.

There could be an application that programmatically hides these bars when it initializes, but then does not check their status after that and just assumes they're hidden. The user could manually show one or both bars again. The application doesn't really need to care about this. It means, while they're visible, they may be covering over part of the application's UI. But the behavior of the app's UI is not otherwise affected. The user might do this, for example, to see the time, or to open a utility. After which, the user can just use the keyboard shortcut again to hide the bar and see the app's full UI again.

When an application is quit, or when the OS is freshly booted, the visibility of the menu bar and status bars is automatically restored. It does this to prevent a situation where the application turns one of the bars off, and the user doesn't know how to get it back.

Extending the Status Bar

A couple of built-in features of the menu bar and status bar include its ability to launch some standard Utilities. Double clicking on available memory opens the Memory utility. Double clicking on the clock will open the Today utility.³

Single clicking on the left half of the status bar toggles it between drive status, open file name, and application custom status. When the status bar is in application mode, and it needs to redraw, it makes a call to the application's msgcmd dispatch routine with a msg code requesting a pointer to a status string.

The application can maintain status either as a set of null-terminated strings, and return the correct string pointer upon request. Or, alternatively, the application could maintain a status buffer which it programmatically updates, and on each request returns a pointer to that one buffer. I can see the practicality of either approach.

If the application does not implement the custom status msg command, the standard way to indicate that a msg command is unrecognized is to return with the carry set. At this point, the other registers don't mean anything, and the status bar falls back on displaying the drive status. Otherwise, if the application supports the msg command, it retuns with the carry clear, and a regptr (X/Y lo/hi pointer) to a null-terminated PETSCII string.

The status bar converts that string to screen codes, pads it out if it is too short, and/or truncates it if it is too long. There are a few reasons for not allowing the application to freely draw its status to screen.

• It would violate layer ownership
• It would let the app draw outside the status bar
• It would allow the app to do unexpected things, such as change the status bar color

The whole point of screen layers is that each draws on top of the previous layer. No matter what the application draws on its own layer, it can never cover over part of the utility, because the utility draws after and over top of the app layer. And similarly with the menu/status layer. It always draws above the app and utility, and thus, won't ever end up with application artifacts over top of it.

If the application were given the go ahead to draw, first, it could decide to draw wherever it wanted rather than only in the status bar. Second, it could do unexpected or unintended things, such as change the color of the status bar, which would decrease the UI consistency from app to app. I think it's important to have consistency, and the way to do that is to enforce certain sensible limitations.

There is no protected memory. An app can always try to draw whenever it wants. For example, when the msgcmd comes in asking for a status string, you know the menu layer is getting ready to draw the status bar. So, why not draw something directly to screen, say in the row above the status bar? That would give you a fully custom second status bar that draws on the menu/status layer. You could do that, but it's a hack, and there is a good chance that something, at some point in time would break, or overwrite part of your hacked status bar. So, I recommend not doing that. The system should provide enough resources that it is easier to work with the system, stay within the limits, and also be able to accomplish most of anything you'd creatively want.

What about making the status bar interactive? Shouldn't the app be able to put a button in the status bar, and have code execute when the user clicks it? I have thought of this, but it is a matter of priority.

I've got a list of features, including full applications, and numerous utilities, that I still need write before reaching a viable v1.0 release. So, I have to draw a line on functionality somewhere. And, if there is room in memory, perhaps in C64 OS v2.0, something advanced like interactive controls in the status bar can be added.

1. Could be up to 38 characters. However, in practice, the longest status messages get truncated so there is always enough room left over to show the available memory. [↩]
2. The mechanism is in place to be able to add more. Triggering screen captures, for example, is on my radar. [↩]
3. The Today utility is like a basic calendar. It lists the current time, date and day of week, and it shows a simple calendar view of the current month with today's date highlighted. [↩]