NEWS, EDITORIALS, REFERENCE

Application Loading

In February I had a post about memory managed loaders. The general idea is that when you load some content into memory from disk you have to take into consideration the memory manager. The memory manager is there to organize memory availability so you can make best use of it. And so that that two things that need memory don't clobber each other. If you need 2 kilobytes of workspace for something, you can ask for 8 consecutive pages from the memory manager, and you'll get them. And you don't have to worry about interfering with the memory space needed by the TCP/IP stack, or the arbitrary length of text that a user may be editing in the application.

When the application is already running but needs some space, that's easy, it requests the space from the memory manager and uses what it gets. But what about when you need to load some content into memory from disk? As I discussed in the previous post, there are really two types of content that can be loaded in. Fixed address and relocatable. Under ordinary circumstances assembled code cannot be arbitrarily relocated. It is possible to write some short routines that are naturally relocatable, but any code, not just JMPs and JSRs, but LDAs and STAs or any other instruction that uses an absolute addressing mode will likely fail to relocate properly. It is also possible to have the code be rewritten at load time so that it functions at a different load offset than the one it was originally assembled to, but so far that is beyond the scope of C64 OS. Sometimes also there will be data that can be loaded to anywhere in memory, because it doesn't execute. The first example I had in mind for this is an application's menu definitions file. It is just text, so it can be loaded anywhere. The menu parsing routines expect it to be page aligned and this works just fine with the memory manager's page allocator.

These loaders are primatives. They bring data in from the disk and put it in memory, and they make sure the memory manager is informed of where the data went (in the former fixed loader case), or they ask the memory manager to reserve them some space a head of time (in the latter, relocatable case.) But they have to be called by something. Something needs to find and load the main executable of an application. Something needs to find and load the application's menu definitions file. I realized that I am going to need more than just routines for loading executable and sequential data, what I really need is a way to load and launch an application that conforms to a standard structure. Here I'll talk about what I'm planning for that and where I'm up to in development.

The Core OS

The C64 OS Kernal consists of the booter, which initializes the memory map, sets a few critical drawing flag defaults, goes through a drive and device detection routine constructing a map of the hardware configuration and then loads into place all of the statically assembled kernal modules. I wanted each module to have a single keyword that defines what it is, so they are:

• File (deals with drives and file access)
• Input (keyboard and mouse)
• Memory (memory management)
• Menu (primary UI, menu system)
• Petscii (text and character conversion)
• Screen (screen compositing, and event loop and dispatching)
• Service (IRQ handler, jiffy clock and timers) and
• String (string manipulation)

These are all loaded high into memory just before and partially overlapping I/O at $DXXX.

The booter, the kernal modules as well as a handful of other things, such as the mouse pointer sprites, system config file, background image, etc., are stored inside a main system folder. My guess, at this point, is that C64 OS will not be runable from a 1541/71/81 alone. But, this is 2017, and the uIEC and IDE64 among other devices are readily available for low prices. There really is no reason why someone who wants an OS on their C64 (like me) shouldn't just go get one of these wonderful mass storage devices.

After reading all about implementing "Present Working Directory" or PWD, that is compatible across a wide variety of Commodore drives, CMD HD/RL/RD/FD, SD2IEC, IDE64, etc, it comes to my attention that trying to figure out where you are on the disk, according to the internal pointers of the drive's DOS, is complete madness. So, how will C64 OS handle this somewhat vexing problem of simply knowing where you are?

Knowing the current device is not hard, figuring out the current partition is also not hard. The main system folder will have a config file which simply declares what the system folder's path is. The device, partition number and path are used to construct a file reference object, which I'll discuss in more detail in a future post. References to files are constructed with file reference structures. The File module has a routine called prepdisk. It takes a pointer to a file reference and changes the default device, changes that device's default partition and path before loading a file or opening a file for read or write. The load or open should occur immediately after the call to prepdisk, before another call to prepdisk can be made that will change those defaults. The actual load or open is used using just the filename, which is also stored in the file reference structure and all other details are relative to the last prepdisk.

Loads happen immediately and then the file is closed. If a file is opened, the KERNAL's logical file number is also stored into the file reference structure. Once a file is open, the DOS in the drive continues to know how to read and write to that file, even if the default partition and/or path are changed. I'm explaining this here, because it gives some context to how the C64 OS kernal will load applications.

When C64 OS boots up, it will create one file reference struct that remains in memory for the duration of the OS's run time. It is populated by the default device, aka the device that was booted from, that device's current partition, and the path to the system folder that is retrieved by loading and parsing the system config file. It will therefore be necessary to switch to the boot partition and change into the system folder before booting C64 OS. I tested out what happens with Wheels if you are currently in partition 5, but Wheels is in partition 1, and you load it like this:

	LOAD "1//:STARTER",8

What happens is that you get the funky hires monochrome image of a three-spoked wheel, and as soon as the "starter" (what Wheels calls its booter) tries to load the next file, the CMD HD's error light starts blinking and the boot does not proceed. So, it looks like Wheels makes similar assumptions to the ones I'm making.

The system's file reference struct is used as the starting point from which to build file references that are relative to the system directory that are used to access files and folders in known system defined locations. So, what are some of those locations? Inside the system folder there is an "Applications" folder and a "Services" folder. "Applications" is for non-system related applications, I have several in mind that I intend to write when I've got at least a v0.9 of C64 OS ready to go. The Services folder is for programs and applications that are core to the Operating System. Chunks of code that are not always memory resident, like the KERNAL is, but without which you can't really do much.

File System Structure

Let me take a short detour here to talk about how files and data are stored on disk. I think there is an interesting parallel here between Commodore and Macintosh. In the original Macintosh File System (MFS) there were no directories, no nested folders. Every file lived at the root level of the disk. This is pretty much identical to Commodore's own 1541/71 and later the 1581 3.5" disk drive. Although the 1581 had some sort of sub-partitioning that I never understood and have never used. The Mac's MFS instead had a pretty novel feature in "resource forks." Basically, one file reference in the disk's root directory could internally reference more than one stream of data. This allowed an application to have code in one stream, and a number of non-code resources like images, sounds, text blocks, icons and more that were all wrapped up in one file "bundle". When GEOS came out for the C64c in 1986, they clearly thought MFS's resource forks were a good idea.

GEOS introduced a new File Format called VLIR, Variable Length Indexed Records. These files were used on 1541 disks which didn't have directories. The VLIR file consisted of a first block, 254 bytes of usable data, that was dedicated to 127 2-byte pointers. This initial block is an index to a set of 127 records. Each record is like an ordinary sequential file, a chain of blocks where each block points to the next, all the way to the end. The benefits of a VLIR file are quite similar to that of an MFS file with a resource fork. The records can be used to bundle together different types of resources along with an application's code. So, an application's code only needs to know about the VLIR file it comes from, and then it can use the GEOS kernal to load in the contents of individual records during run time. This is a great solution for a disk format that doesn't support folders. Each VLIR file is like a little folder unto itself.

The problem with VLIR files is also pretty much the same as the problem with MFS files with resource forks. On a C64, other file copying utilities, anything that isn't GEOS and VLIR aware, have no idea how to copy a VLIR file. Copying and file management of GEOS files pretty much has to be handled by GEOS itself, or GEOS files have to be encoded in a more standard sequential format first. This is similar to the problem facing the Mac, except that a Mac only ever ran the Mac OS, so you didn't have to worry about dropping to basic, loading up a non-Mac OS based copy utility that knows nothing of resource forks! While, that problem certainly did exist on the C64 where GEOS was only one of many types of environments the user might be in for file management. On the Mac the problem really arose when networking with non Macs became common. Sending Mac files to a Unix computer became problematic because Unix file systems were not as "clever" as MFS (and later HFS and HFS+). FTP programs on the Mac would often hexcode Mac files in realtime upon upload and could unhexcode them on the other side upon download. It was okay, but not ideal.

Eventually when Mac OS X came along, and the internet was in full swing, Apple made the painful decision to ax the concept of resource forks. They stuck around for a while, but it was pretty obvious they were on the way out. Instead, Applications and even fancy document files became "folder"-based bundles. An app in the modern macOS is a folder with the file extension .app. Inside the folder is a standard layout of code and resources each in separate files. I suggested to Jolz Maginnis when we were working on WiNGs that we should do the same thing there. He liked the idea and implemented it. Now, I fully intend to continue this migration with C64 OS. We live in a world of CMD HD, uIEC, and IDE64, where file system folders are easy to come by. So each Application will live in its own folder which the high level file managers and launchers of the OS will treat as though they are one "thing." But under the hood the code and resources of a bundle are each stored in their own file.

The Application Loader

Enough of the history lesson. Let me say that Bundles are great. One thing I particularly like about them is their hackability. You can easily navigate into the bundle, see all the individual files and modify them. This sort of thing wasn't possible with VLIR files and required special tools like ResEdit on early Macs. I'll come back to the hackability aspect in another post.

Directly within the Services folder and the Applications folder there will only be folders. Each of these folders has the name of an application, and is a bundle with a standard internal arrangement. The app bundles that are inside Services are required and will simply be expected to be found by the OS. If they're not found bad things will happen. In the Applications folder, these are launched by the actions of the user so none of them are required. C64 OS has a concept called "Homebase." This is the application that is launched when the OS is first booted. It allows the user to launch other applications, and it is returned to when a user-launched application is quit.

The OS tracks what the Homebase application is, by name. So, there is a 16 byte string somewhere in memory that holds the name of the Homease app and it is assumed to always be in the Services folder. When C64 OS boots up, it loads in a config file which contains the initial Homebase app's name. After loading all the OS components, the booter calls the OS's loadapp routine with a pointer to a file reference struct. The file reference struct is assembled from the OS's root file reference struct, with "/services/XYZ" added to the path, where XYZ is the Homebase app's name. The loadapp routine calls prepdisk on that file reference to set the drive's default partition and path to the root of the Homebase app. Next it loads in the menu definitions file. It does this by using the loadseq primitive with just the filename "menu.m". This file is found at the relative path and partition that drive has already been prepped for. Once it's loaded in loadapp calls mnuinit and passes the page number and page count of the data. This frees old menu data and parses and initializes the new menu data.

Loadapp next loads "main", which is non-relocatable, executable code loaded to its original assembled location. This should be low in memory, preferably page aligned, and in fact preferably loaded to $0800. The memory manager flags as unavailable the pages that main loaded into. Loadapp then JSRs to the start of main, so let's call that $0800. This is the initialization routine of the app. It's job is to push the first screen layer. This screen layer has the pointers for mouse and keyboard event handlers, as well as the redraw routine. The initalization of an app could also optionally set up timers, or register network listeners, or load in its own config file or other resource data.

The screen layer is marked as dirty and execution returns to the main event loop. The event loop draws the screen layers as described in other posts. This causes the menus to be rendered, and gives the app's first screen layer's draw routine its first shot at drawing the screen. And there you have it, an application is running and events are being passed to it.

I'll follow up shortly with another post about the Homebase apps and continue to go into more details about quitting apps, and loading apps with a default file to open.