NEWS, EDITORIALS, REFERENCE
Utilities and the Utilities Menu
Last year I made a nice post introducing C64 OS Utilities. In which I discuss their advantages over GEOS Desk Accessories, and go into some detail about what they can do and how they will be used to augment the functionality of the system and its applications. That post is mostly still accurate, and is well worth a read. A few key points will be repeated in this post, but it should be considered a continuation of the earlier introduction.
Utilities are useful small apps that can be loaded, one at a time, concurrently with the main application. By relying on multiple smaller Utilities to do a chunk of work for your application you get several advantages:
- Your application's main code can be made smaller
- More main RAM can be freed up for use by your app's data
- Your application will launch faster
- The utilities you create can be used to enhance other applications
- Using your application will be more consistent with the rest of the system
C64 OS's system folder is quite structured. Applications go in the applications folder. Drivers go in the drivers folder. And Utilities go in the utilities folder. The downstream benefits of even a simple level of organization like this are bigger than you would expect. Now to be fair, the opportunity to organize the many files of an OS like this often comes on the heels of hardware that enables it.
Let's recap GEOS for a moment. GEOS ran from a single side of a 1541 floppy disk. That's a 170K disk with a formatted capacity of just 166K. That's only 2.5X the total capacity of the C64's RAM. If you put the essential system files and the deskTop on a disk, you've already used just under half of that capacity. Then you add one application, maybe geoWrite, one document file and a couple of fonts, and the disk is full. Another thing to consider is that 1541 disks don't support subdirectories. But frankly, there's not much point in supporting subdirectories when the disk is full with just 10 or so of these files.
4 System files, no applications, no desk accessories, no documents... and only 94 kilobytes free.
It's hard to be idealistic about how to structure the file system, where OS components should go, and how resources should be divided up, when the entire storage medium is at capacity with just 10 files. There is only room for one application and one or two documents, plus a few supporting files per disk. The most important consideration in an environment of such unimaginably constrained resource is how the system should gracefully handle an essential component not being available. In GEOS that means loading as much into memory at a time as necessary for basic functionality, and then every subsequent disk access has to support asking the user to swap in a disk containing the needed file.
The Commodore 64 was birthed into a world where storage was nearly non-existent. In the early days, most users outside North America didn't even have a floppy drive. But C64 OS is being birthed into a world where storage is so copious that it is effectively without end. The decisions outline below were made with these considerations in mind.
Utilities as UI Components
The entire operating system, including all components, applications and utilities, is contained in a single folder. That means everything is in one partition, on one storage medium, in one device. Not including documents and media files, they can be wherever you want to put them, spread across different devices and storage media.
Within the system folder there are numerous top level folders. Applications, utilities, drivers, services, desktop, etc. All installed utilities are in the utilities folder. To be installed a utility merely needs to be moved or copied into this folder. The system routine to launch a utility does not take a file reference. It takes only a string that is matched against the filename in the utilities folder. Applications can programmatically launch utilities by calling the system routine loadutil and passing it a pointer to a string containing the name of the utility. If a different utility is already open, the system orders the current utility to quit, giving it a chance to save its data and state. Then it loads and opens the new one in its place.
Storage is so abundant that there is never a reason for every utility you can find or write to be installed at the same time. Applications can come to depend on a standard set of utilities being available. 1 And common user interface components can be built into utilities. For example, if you need a date picker, you can open the date picker utility, and then support the messages that it sends back to your application.
Support for this can be done in just a few lines of code. But implementing support for displaying the months of the year, and counting days per month, and calculating which year is a leap year, and having buttons with callbacks, etc. is a surprising amount of work. Furthermore, if this date picking code is not in a utility, it means it probably needs to be in the application's main code, and then it takes up main memory, and has to be loaded at application launch time. It's much better to defer loading this functionality until when the user actually needs to pick a date.
Now in this kind of example, it makes sense for the application to programmatically open the date picker. The user clicks a field that takes a date. In response, the app invokes the date picker utility. It floats above the app, the user selects the date using its select boxes and buttons, then the picker closes and the date is sent back to the app, and the app fills it into the field. Another example is the color picker. In an application like Chess, maybe there is a menu option that says "Board Color..." You choose that, and up pops the color picker utility. You pick a color, close the utility and now the Chess board is blue instead of brown.
Wanna know how easy it is to open, say, the date picker? Here it is:
That's it. That's everything. Now, to receive a date from it you have to support the message command sent from the date picker. But believe me, it's not much harder than the above, which is a lot easier than implementing a date picker into your app from scratch.
There is a different class of utility though. I haven't come up with the right naming convention yet for exactly how the different styles of utilites are categorized, but if the above were UI components, then there is also a set of service utilities.
Let's say you're in the kind of application that edits a file. The canonical example is a text editor, but it could be a drawing program, or a music tracker, or what have you. Regardless of the specific application, there is a file being edited. In another example of structure leading to flexibility, C64 OS applications that are editing a file, refer to that file using a standard file reference, indicated by the Open File Reference pointer (opnfileref).
When the user double clicks a file in the file manager, an application is opened to handle that file. The file reference is passed to the application by means of the open file reference pointer. When the application starts up it uses this file reference to know what file to open, and later, when saving changes, that's the file it will save the changes to. Besides the application, the system-controlled status bar has an open file mode, which shows the device, partition, filename and path to the currently open file, which it gets from this standard pointer.
Can an application buck this trend and ignore the open file reference? Yeah, it can, but the coherency of the system suffers as a consequence. Flexibility and power is lost if an app decides to go off on its own and do something non-standard. By adhering to the structure, we get the additional and perhaps unexpected flexibility of a service utility such as the one shown below. I'm tentatively calling it the File utility. (File deals with just the single currently open file. Not to be mistaken for the Files utility, plural, designed to let you navigate the file system and see all your files.)
While working in your text editor, you might want to rename the file you're working on. Or make a backup copy, better yet. Or perhaps you want to see how big the file currently is and/or when it was last saved?
This is too much to build into the status bar. It shouldn't be taking up memory as part of the core system, especially considering all those apps that don't even edit files. You wouldn't want to write this into every application, either. Some apps would end up offering some of its features, others not so much. Every application would implement this sort of thing differently. Besides, it's not really an application feature, per se. It's more of an ability of the operating system to offer you some management of your computer while an application is running.
Forget about how cruftily or inconsistently different applications would implement equivalent features, in my opinion you cannot even rely on an application to provide a means to open a utility like this. Maybe the app developer just doesn't care, or doesn't think this is very important. Or maybe a utility like this doesn't even exist, at the time the application is written. It is precisely for these kinds of utilities that the system needs a utilities menu.
The Utilities Menu
As soon as I decided that C64 OS would support utilities, I knew there would be a system-wide utilities menu. But for a long time I put off writing the code to make it.
The regular menu module was already quite large. It has support for hierarchically nested menus, with keyboard shortcuts that use modifer-key combinations. The menu options can be enabled or disabled and active or inactive, using the delegation pattern. They support mouse roll-over highlighting, and pop open and closed as the mouse passes from menu to menu. But one of the best features, which I'm most pleased with, is that they aren't statically coded into the application's binary.
Menu data in C64 OS is stored in a resource file, menus.m, in the application's bundle but external to the main binary. The format is human readable and editable in any text editor.2 The data format is read recursively, and efficiently translated into linked memory structures that can be rapidly searched for matching keyboard shortcuts. And further, the width of the menus, where exactly they display on screen, is all dynamic. And unlike GEOS menus, they composite in realtime above an application that continues to run and actively update its UI. Many other nice little features exist, such as blinking the root menu item that contains a menu item that was activated with a keyboard shortcut. Spacers can be left to help group menu items for greater clarity. And menu action codes are sent to the application, as message commands, keeping the menus and the application neatly decoupled. The post Menu Systems, a comparison goes into all the details on everything that the menu system in C64 OS can do.
Needless to say, adding in a whole new type of menu, that is generated and maintained by the system rather than the application, was daunting. The menu code was quite complex and hooking into it was not something I relished doing. The menu module also implements the system status bar, which only adds to the code size. I simply cannot continue to balloon the size of the core modules with wreckless abandon. So I put the utilities menu on the back burner and focused on other more pressing needs.
Now that most of the essential functionality of the C64 OS KERNAL is done, I'm in a new phase of feature development. For months I've remained focused on how to implement features without increasing the system's memory footprint.
Utilities themselves are a way to do just that. As mentioned earlier, I can't simply build the features of the File utility directly into the status bar (further expanding the menu module.) This is simply not a sustainable way to implement new features. But, by implementing those features as a utility, the doors are blown open. An application can have access to a vaste assortment of features simply by parceling the features out into utilities. And the ability to open them programmatically means the user doesn't necessarily feel as though they have opened a utility, but rather a small tool palette just appears on the screen in response to interacting naturally with the application's main UI.
Simple block diagram of memory layout. Utilities run beneath the KERNAL ROM, and can be loaded and unloaded during the runtime of the application.
Another beneficiary of thinking about how to conserve memory is the recent work on relocatable 6502 code. Some people were making noises about wanting a joystick input driver, rather than needing to buy a mouse or mouse adapter. But with the mouse code built into the KERNAL, how could a joystick driver be added? Should I just add that right into the KERNAL side-by-side with the mouse driver? That sounds wasteful. So with relocatable code, the mouse and joystick drivers can be removed from the KERNAL and converted into loadable and unloadable drivers. That was a big win. I've been playing around with my Koala Pad and Animation Station, and hope to make drivers for them. I also had an idea for an input driver to use the numeric keypad on a C128 (which can be read even in C64 mode).
All of these ideas for more features are downstream possibilities born of initial thinking about conserving memory. Strange how that works.
But what about the Utilities menu? I mean, it's gotta be built into the system somehow right? Somewhere along the way, I realized that there is a bunch of work that has be performed to clean up after an app quits, and prepare for a new app to launch. Like what you say? Well, if the application crashes due to an unhandled exception, for example, you have to clean up the exceptions table. You also have to automatically free all memory pages that had been allocated by the last app. There are also a handful of things I do to make sure transitions go smoothly:
- the mouse is reactivated
- the menu and status bars are unhidden
- the border and background colors are restored to your preferred defaults
- any running utility is quit
- any image data in high memory is expunged
- screen layers are popped
- split screen is closed, and so on.
There's quite a bit to do.
All of this work was being done by the service module. As part of this process the service module would also make calls into the menu module. One call was to free the memory pages being used for the existing menu structures. And another was to load in the menu definitions file for the app being launched, and parse the data into new structures in freshly allocated memory.
It occurred to me though, that all of this work is happening only during the transition between two applications. One app is quiting, and a bunch of clean up has to be done. And the next app is launching and it needs a bunch of setup work. But while the application is running none of that code is really useful for anything. So I pulled most of that code out of the service and menu modules and put it all into a program called the loader. Up until recently almost all of C64 OS code was in either a KERNAL module, or an application, or a utility, with a couple of very small exceptions, such as the booter. But more recently some code is now found in the relocatable drivers, and some code is in what I'm calling components. Here is some functionality found in components:
- Drive Detection
- Exception Handling
- Memory Mapping3
- Reading Realtime Clock
- And now, the Loader
Because these non-KERNAL based components don't always stay resident, they actually have a substantial amount of freedom in how big they can be. Drive detection, for example, is close to two kilobytes of code but only leaves a 22-byte table resident in workspace memory. Reading the realtime clock only results in about 10 bytes of permanently resident workspace memory.
By moving code out and into the loader, hundreds of bytes were trimmed off the size of the KERNAL. It doesn't sound like much by modern standards, but it represents between 2 and 3 percent of the total KERNAL size. There is a small downside. In the post I linked to above I wrote this about the menu system:
It's even cooler than that. Reading in and parsing the menus is part of the standard procedure of loading and running an app, but it's also just a standard system call. The app itself can programmatically ask the OS to load and initialize an alternative menu data file, at runtime. So, if your app has multiple modes, when the user switches modes, the app could load in a different set of menus custom for each mode. That is really friggen cool. Greg Nacu — Menu Systems, a comparison — June 2018
You know what? That is friggen cool. And unfortunately we are going to lose this ability because the menu data loading/parsing code will no longer be available at the whim of a system call. But to trim this code out of the KERNAL still feels like the right thing to do. If an application some day really needs the ability to swap menus at runtime, perhaps we can figure out a way to gain access to the feature as an optional, temporary, OS component.
There are upsides though. The loader now displays a sort of splash screen when loading an app, that says:
READY. LOADING... //os/applications/chess
Or whatever the name of the app is. It makes the experience feel more polished. And it adds some nice flavor, a blend of the old and the new, that I wouldn't have wanted to waste the space on when it was built into the KERNAL.
UPDATE: October 9, 2019
Writing this post inspired me. Thinking of it as a "splash screen" made me realize that the application's standard 3x3 icon (9 custom characters) should be read in by the loader and displayed on the screen while the app is launching. I got home last night after our meeting of the Kingston Retro Computer Club, sat down and banged it out, and I got it working on the first try. Awesome!
But of course the biggest upside is that I'm freed to expand it to support building the utilities menu.
Linking Two Sources
If you want to get into the weeds—I actually find that a lot of fun—about how the menu memory structures are created and linked up to each other, I wrote a short but technical post about it back in 2017, Pointers in Practice, Menus.
There are three types of structures, for three types of nodes found in the menus:
Menu headers have a title and link to two other nodes, their first child and optionally a next sibling. Headers have no action code and no keyboard shortcuts. Menu items are what you actually select with the mouse to perform an action. They have a title, an action code and the option for a keyboard shortcut. Menu items link to their next sibling, but they can't link to a child node. Lastly, menu spacer nodes are the simplest. They have no title, no action code and no option for a keyboard shortcut. Spacer's link only to their next sibling. Their presence in the node tree informs the renderer about where to leave an organizational spacer in the menu.
The three types of menu nodes, their links and properties.
There is a single byte in workspace memory called Root Menu Page. When the menu parsing code reads in the menu data and constructs the node tree it allocates new pages as it needs them. The root node, the first sibling of the chain of menu headers that constitute the menubar, is always found at the very beginning of the first page allocated. This page byte is written into the Root Menu Page workspace variable. And it is used by the menu system to find the start of the menu node tree. Any other allocated pages are found implicitly by following the pointers.
A typical C64 OS menubar then consists of the root menu header pointing to a next sibling and that to another sibling and so on. The utilities menu, though, only ever has one top level header node. It would be the leftmost character cell of the menu bar. I put the hamburger menu character there as a placeholder, and when the CPU is busy it gets temporarily replaced by the CPU busy character animation. But, if there is only one header node, (and it's easy to find, it's the first one in the page,) then that header node's next sibling pointer could simply be pointed at the first header node of another independently parsed and allocated node tree.
The solution ended up being pretty easy. The system settings folder now has a menu definitions file called utilities.m. It is structured exactly the same way as the menu definitions file of any application, with one exception. The utilties menu must have only one root level header. utilities.m has its own memory allocated and is parsed into a node tree just as an application's menus are. And its root level menu is assigned to the workspace variable Root Menu Page. (This node tree is shown in the green box below.) The loader, when launching a new application, reads in the app's menu definitions file, allocates new memory for it and builds its node tree completely independently. (The app's menu node tree is shown in the blue box below.) But then, rather than pointing the Root Menu Page at the first application menu header, it points the utilities menu's root menu header's next sibling to the first menu header of the application. (Phew, that sounded complicated. But it's really just the link formed by the black arrow shown below.)
Two totally independent menu node trees, linked together.
When the previous application's menus are being freed, rather than starting from the Root Menu Page and freeing everything from there (like it used to,) it finds the start of the application's menus from the link pointer from the root header of the utilities menu. Deallocation of the menu tree then proceeds exactly as before, but it only deallocates the application's portion of the menus (the blue box) leaving the utilities menu node tree intact (the green box).
When the menu system draws the menus, it has no clue that part of the tree is generated by the system and part of the tree is generated by the application. Because they are all linked together as a single node tree. No custom code needed to be written in the menu system to support a special utilities menu. And all of the features and benefits of the regular menu system are extended into the utilities menu.
- Hierarchical nesting of menus
- Keyboard shortcuts to launch utilities
- Spacers for organization
- Dynamic layout
- Menu item rollovers
- Root menu blinks when a keyboard shortcut is used
- Human readable/editable definitions file
- Roll back and forth between the utilities menu and app menus
The net size of the menu module went down by a couple of hundred bytes, while adding support for the utilities menu. And the utilities menu is a full class citizen. Solid win.
Some additional support
Some additions were necessary in the menu module, but the net size went down, because of such a huge savings from pulling the main parsing and freeing code out and into non-KERNAL components.
Typically the menu bar automatically inserts a space between every root menu title. So, "File", "Edit", "View" etc. in the menu definitions file appear in the menubar as "File Edit View" not "FileEditView". But the utilities menu's root entry is just the hamburger icon and I didn't want a space to follow it before the start of the application's first menu title.
Also, usually a menu item has an associated action code. When the item is triggered (either by mouse or by keyboard shortcut) the action code is sent to the application in a menu message command. In order to support opening utilities, there is now a reserved action code for that. Rather than sending that action code to the application, the menu system interprets it as the code to open a utility. It copies a pointer to the menu item's title text, and passes that to the system's loadutil routine. So, the title that appears on the menu item merely has to match the name of an installed utility.
There is an extra little side benefit to this. Use of the reserved action code isn't confined to the utilities menu. The menu system itself doesn't distinguish between the utilities menu and the application's menus. They're all the same. What this means is that you can use that reserved action code in your application's menus, and your application can thus allow the user to open a utility and you don't have to have any support in your application binary at all! For example, the App Launcher lets you customize the color of the aliases on the desktop. You merely open the color picker utility, select one or more aliases, and click a color. The App Launcher thus wants you to always have access to the color picker, regardless of whether you choose to include it in the customizable utilities menu. So under the "view" menu there is an entry for the color picker, but the action code is the reserved code for launching utilities. When the user picks that menu item, no message need be sent to the app, no support in the app is necessary to call loadutil. Unexpected bonus.
There is another side effect of all this, although I'm more doubtful of the usefulness of this one. You can create an entry in the utilities menu that has a regular action code, and a title that is not the name of a utility. Such an action code will be interpreted as normal and sent to the application via message command. The only thing is, it would always be the same action code sent to whatever application happens to be running. Applications ignore action codes they don't recognize, so it won't hurt, but unless the application has support for it, it wouldn't do anything useful either.
There could be something to this though. Maybe the command to quit the application and return to the homebase app could be standardized at the bottom of the Utilities menu, after a spacer perhaps. This would obviate the need for every single application to include that in its menu definitions file. I don't know, I'll have to think about this some more.
A wrap up comparison
The GEOS menubar is kind of an odd duck. Every menu and every menu's position on the screen is hardcoded into data tables in the application binary.
The menubar only stretches as far across the screen as there are menus to fill the space. This could actually be kind of cool. I generally like ideas that save screen real estate. But the weirdness doesn't end there. The date and time, which are available from the deskTop, are not there when you launch into an application. Apps like geoWrite and geoPaint for example show the title of the current document, in what looks like a file folder tab, at the top right of the screen. My guess is that the date and time are a feature of the deskTop program itself.
In the deskTop, as well as in the main apps like geoWrite, geoPaint, geoCalc and geoPublish, the leftmost menu is the "geos" menu (all top level menus in GEOS appear to be in lower case letters only.) The GEOS menu contains a mixture of things. From the deskTop it has entries for "GEOS info" and "deskTop info", both of which pop up a dialog with some version, copyright and author credits, for GEOS as a whole and for the deskTop application respectively. Then there are two entries to choose drivers: "select printer" and "select input". Select input is the only one that has a keyboard shortcut. COMMODORE-I. This is because when you boot up GEOS it goes straight into the deskTop. But if it has the mouse driver installed and you have a joystick hooked up, you can press COMMODORE-I. This opens the dialog to list the available input drivers on disk. And, interestingly enough, only when this particular dialog is open, you can use the keyboard cursor keys and RETURN to move the mouse pointer and make selections. This allows you to change to the appropriate input driver for your hardware and then go from there.
Below that, there could be listed up to 8 desk accessories. The first 8 found in the disk directory. They have no keyboard shortcuts. Their menu labels appear to be exactly the same as their name on disk. And they are listed in the menu in the same order as they are found on the disk.
This set of entries in the GEOS menu isn't the same in geoWrite, geoPaint, et al. In these applications the top so many entries are different, followed by the 8 desk accessories. But then in some applications there is no GEOS menu at all. I don't know this for certain, but it feels like each application is required to build its GEOS menu from scratch. And quite possibly the code used by the deskTop to read in the first 8 desk accessories and add them to the bottom of the GEOS menu is just the same piece of code that has been incorporated into each of the flagship applications.
We don't have to get into all the ways the C64 OS menubar is faster, more advanced and more consistently available than the GEOS menubar. I've already documented that in detail. But let's just talk about the utilities menu.
Launching a new application leaves the existing utilities menu data structures in memory. So, every C64 OS application will have access to the utilities menu, not just in those apps that deign to build support for it by hand.
There may be tens or more of utilities installed in the system utilities folder at the same time. But the utilities that appear in the menu are not just the first ones found in the directory. That's good too, because it is not so easy to rearrange directory entries on an SD2IEC as it was on a 1541. Instead the utilities you want in the menu can be added to the utilities.m definitions file, and they can be sorted in any order.
Unlike in the GEOS menu, the utilities can be assigned keyboard shortcuts. And those shortcuts can use multiple keyboard modifier keys to prevent them from colliding with more common shortcuts that may be used in the application's menus.
Spacers can be added where desired to provide visual groupings.
In the first pull down menu, there can be up to 24 entries, including spacers. The 24th item in the menu will be drawn above the status bar, if the status bar is visible.
But if 24 isn't enough, submenus are supported, easily allowing up to 100+ utilities to be organized beneath that single root menu header. The menu system permits access to more than the amount of memory most users will be willing to dedicate to them.
In GEOS the deskTop has the menu option "GEOS info", which opens a static panel built into the deskTop itself. In C64 OS there is the "About C64 OS" utility. It can be opened from within any application, if added to the utilities menu.
In GEOS each application, if it supports such a feature, may have an entry to show information about that application. On the deskTop it is the "deskTop info" menu item. This too is statically built into the deskTop application. In C64 OS, there is an "About This App" utility. When About This App is opened, it uses the standard appfileref pointer to locate the application's bundle folder. It reads out of that folder the application's icon, name, copyright and credits information and displays it to the user in a consistent format for every app. Add this to the utilities menu, and every application has the perfect place to list its copyright info and author credits. Freeing each application developer from having to invent a new way to cram that onto the screen somewhere.
GEOS doesn't have a consistent way to display help. The deskTop (in GEOS 2.0) has a hardcoded panel that can be called up to show a list of the keyboard shortcuts that aren't found on menu items. Other than that, there is nothing that's standardly available. In C64 OS, simply add the the "Help" utility to the utilities menu. Just like "About This App", the help utility uses the appfileref pointer to find the application's bundle, and opens the standard help text file the app provides. But, none of that is in memory until the user requests it. So an application can provide up to nearly 7K of help text (that's more text than is in this weblog post). And you can just stick that in the utilities menu. You can even assign it a standard shortcut, like COMMODORE-CTRL-H for Help. And every application can have its help opened with that command. This is a thing of beauty.
Here's a short video showing the utilities menu persisting between the App Launcher and the TestGround apps. Shows keyboard shortcuts, spacers, nesting and launching utilities.
Thanks for reading, C64 fans. On the list I've got posts planned for raster interrupts and split screen, toolkit class structure and related updates, the user port real time clock project, building C64 OS Utilities, and more. Stay tuned.
- You can think about a standardized set of C64 OS Utilities like the Unix/BSD subsystem. An assembly of standard programs that other programs depend on to fulfill their complete functionality. [↩]
- My favorite text editor for the C64 is Novatext, by the way. It comes bundled with Novaterm 9.6 but can be used as a stand-alone program. It's got a built-in help screen, it's very intuitive and easy to use. And it has block copy and move of text. It's a great little text editor. The only problem is, it can't read or write files from or to an SD2IEC. Even to a mounted disk image. This is quite disappointing. [↩]
- Not to be confused with the memory KERNAL module. The memory mapping component is installed to a fixed address range in low memory, and is used for setting the PLA's memory mapping of I/O, KERNAL and BASIC. [↩]
Do you like what you see?
You've just read one of my high-quality, long-form, weblog posts, for free! First, thank you for your interest, it makes producing this content feel worthwhile. I love to hear your input and feedback in the forums below. And I do my best to answer every question.
I'm creating C64 OS and documenting my progress along the way, to give something to you and contribute to the Commodore community. Please consider purchasing one of the items I am currently offering or making a small donation, to help me continue to bring you updates, in-depth technical discussions and programming reference. Your generous support is greatly appreciated.
Greg Naçu — C64OS.com