Networking in C64 OS

For the last few months I have been super hard at work on the next major update to C64 OS, version 1.07. It will be released as a free update to all licensed users of C64 OS v1.x, and will be made available for download from the C64 OS System Updates page.

The big new technology coming to C64 OS is networking. It's been a long time coming, but up until this point there have been many more basic features that had to be completed first. And now I'm very excited to finally be at the point where I can work on the networking layer, which I think will really bust open the doors of what we can do with C64 OS.

Along the way I've been implementing a number of smaller items on the list and fixing a bunch of annoying bugs. Here's a short list of some of the updates and improvements that are coming in v1.07:

• Improved NMI handling and behavior of STOP+RESTORE
• New Network Utility for configuring C64 OS networking
• Settings Utility now offers to open the Network Utility
• TKInput fields gain support for "secure" mode
• Fixed a bug rotating through status bar modes
• COMMODORE+O in Copy and Move Utilities toggles overwrite checkbox
• Network status is now shown in status bar along with drive status
• Fixed a memory leak in the Clipboard Utility
• Hold COMMODORE key when mounting disk images to auto-mount
• Hold CONTROL when selecting mouse pointers in Mouse Utility for large pointer
• Fixed a bug in Installer with 16-character directory names
• Booter now supports "optional" run components
• CONTROL-click refresh button in Utilities Utility to rebuild Utilities menu
• New 32-bit division support in the KERNAL
• Fixed a bug when dragging a scroll bar a very long distance
• New Help Utility; each Application can provide built-in help content
• C64 Archiver can produce .CAR files with scratch file support
• Fixed a bug renaming items on SD2IEC that only change by case
• Fixed many bugs rendering MText in the TKText class
• Added support for hyperlink clicking and callbacks from TKText
• Added link support in the TextView Utility
• Keyboard driver is no longer part of the KERNAL

And much more.

Now let's dig into the meat of this post; All things networking.

Early History of Telecommunications

A number of years ago, I got to thinking about the history of telecommunications. And I knew something about the history of computers, but what I was curious to know was, after the invention of computers, how long did it take before telecommunications became a thing?

I was shocked to discover that I had it all backwards. Analog, long-distance, communications by means of script on parchment with some sort of security seal and a trusted courier, of course, goes back centuries and perhaps even millennia. When electricity and electrical apparatuses were first being experimented with in the 17th century, the idea of applying electricity to communicating over long distances was already in mind. By the early-mid 19th century, almost a century before the very first, most primitive, digital computer, sophisticated telegraph machines were already being worked on. I had the question completely backwards.

Here's another fun question you can put to the average person; how long ago did humans discover fire and start using it for things like cooking? If you haven't read up on this, the answer is quite fascinating. Fire use actually began with our hominid ancestors around a million years ago. But our species, Homo sapiens, anatomically modern humans, only started to emerge around 300,000 years ago. Therefore, amazingly, our species didn't discover the use of fire at all. In fact, there has never been a time when we weren't fire users. What's more, at least one of the arguments for why our species developed as it did was precisely because our hominid ancestors started to use fire.

We didn't tame fire... fire tamed us. Someone cool probably said this.

Perhaps not a perfect analogy, but a titillating comparison nonetheless. Telecommunications wasn't something that we eventually discovered was possible after we'd invented computers. Telecommunications predates computers. The work that Claude Shannon was doing for Bell labs during the wartime of the early 20th century—he was trying to filter noise from signal in electronic telecommunications systems—directly led to the birth of information theory. And that became one of the cornerstones of digital computers as we know them today. Computers didn't lead to the development of telecommunications... telecommunications led to the development of computers.

Brief History of Modem Networking

Let's talk briefly about how the telephone system works and how modems first piggybacked on that pre-existing communications network. In the original telephone networks, starting in the late 1800s and on up into the 1950s, your call was routed by manually connecting the path way at one or more switching stations. You'd pick up the phone in your house and there was no input control of any kind, you were immediately speaking to a human switchboard operator. Today, my kids have probably never even heard the word "operator," and people of my generation have a sort of vague notion that if you dial zero you can connect to the operator, but exactly why you would need to do that is unclear. In the olden days, you'd pick up the phone and tell the operator who you wanted to be connected to. She (by the later period switchboard operators were usually women) would then shunt some cables around to route your call through to the next switching station.

There were also limited numbers of copper wires running between stations, far fewer than the number of possible pairs of phones that could be interconnected. This is why long distance calls used to cost money. The further your call was routed, the more human operators were needed to intervene to do the switching, and the more copper wire you occupied. Charging to use those lines gave you a monetary incentive to get your business done quickly and get off the lines to free them up for other people. It's the same reason why parking downtown costs you money for every 15-minutes, with a maximum stay of 2 hours (or some such variation.) The space is limited and the cost encourages most people to move on, freeing up the space for others.

This also explains why a kid living in Ontario couldn't call Grandpa in Saskatchewan on Christmas morning. Everyone was trying to call their family on Christmas morning (to thank them for that new Commodore 64 they just received, obviously,) and there just wasn't enough copper for that many simultaneous connections. In my era, you would instead get a pre-recorded message saying, "Your call cannot be completed at this time." But we're getting a bit ahead of ourselves.

I'm not 100% sure when these different technologies overlapped, but this is more of a thought experiment than an accurate history lesson. Imagine a telegraph machine connected to an early phone network. The first thing you'd have to do is route the connection by manual switching. Once the switching has been established, one machine is connected directly to another by a pair of copper wires that goes out the building, across the poles and into the switching station, then out another set of poles to another switching station, and then out another set of poles and into another building and into another machine. Encoded pulses sent down that loop travel at more or less the speed of light directly from one machine to the other.

Eventually the manual switchboard operators got replaced, at a great savings in cost, by electro-mechanical relay switching. Then when you picked up the phone, you no longer talked to a person, but you got a dial tone. You would then physically rotate a dial by placing your finger in the corresponding hole, turn the dial to its limit position and release it. By a spring, the dial would tick back to its original position, sending a fixed number of pulses at a specific rate down the line. These pulses electromechanically actuated the relay switches that would do the call routing automatically. This was a great improvement. Again, what the kids don't know, is that our word "dialing" (still used with smartphones today) comes from the fact that there was an actual dial, a rotating circular apparatus.

Some Telex machines had a rotary dial built into them, such as the one above, from 1966. To do the routing, therefore, you held a number in your head and used your finger to rotate the dial. The dial sent the pulses that actuated the switching relays to electrically connect the two machines. Then the two machines could proceed to send encoded information on the copper line, and bingo-bango they are telecommunicating, i.e., communicating at a distance.

The first computer modems used an acoustic coupler. You would lift the telephone handset off the hook and fit it snuggly into the coupler. The coupler was connected to the computer via RS-232, a bi-directional, one-to-one, asynchronous digital communications protocol for exchanging one data word at a time.

Eventually telephone switching systems were upgraded to allow for push button keypads that generated different tone combinations to actuate the switching. To do the routing, however, you still used your brain and your finger to dial the number by pushing the buttons on the phone.

Originally, the computer's RS-232 connection to the modem transmitted data only. Between the modem and the wider world was the telephone, which was used to issue the routing information before the computer became involved. The routing, therefore, was kind of oddly partly in-band and partly out of band. From the phone network's perspective the routing was in-band. But from the computer's perspective the routing is out of band.

Once the telephone switching system became faster and more automatic and computers became a bit more sophisticated, the obvious next step is to have the modem handle part of the routing by doing the dialing, and to have the computer send the routing information (i.e., the phone number) to the modem. There is a problem though. The computer is only connected to the modem via RS-232, and the computer now has to send information to the modem to tell the modem what to do, for example, to tell it to dial a number. But once the modem has completed the phone switching stage, information sent over RS-232 is no longer destined for the modem but for the machine on the far end of the connection.

This problem was "solved" by introducing to the modem different modes of operation: Command mode and Data mode. I put solved in scare-quotes because it is so clearly a hack. It is clear that in the original conception, and given the limitations of the technology at the time, the routing was done by a person using the phone, and so the RS-232 connection to the modem had only one purpose: exchange data with the remote end. But gradually, as the backend network technologies improved and computers got more capable, one wanted to use the computer to communicate with two targets: the remote end AND the local modem hardware itself. But there was only one communications channel, so they introduced modes.

The Complications of RS-232 and Modems

This state of affairs, two targets one channel, led to complications. For example, once the modems are connected, they switch into data mode so that any byte sent from one computer over RS-232 is modulated and transmitted over the telephone line to the remote end. There it is demodulated back into a byte and delivered to the remote computer as data. How then does the computer signal to the modem that it should hang up the connection?

Somehow the computer has to get the modem back into command mode. How does it do this? The standard that was developed was to send an uncommon or unlikely series of bytes; three pluses ("+++") in a row which the modem sees and interprets as a special sequence and switches back into command mode. This has its own problems of course. What happens if, by chance, the data stream includes "+++" ? The modem would accidentally interpret that as the special sequence, switch back into command mode, and the following data would erroneously be delivered to the wrong target.

To solve this problem, a slight delay was included after the "+++". So only, "+++" followed by 1 second of no data would trigger the mode change. If the pluses came in a long sequence of data the character sent immediately after the third plus would prevent the mode from switching. Nonetheless, it is still a total hack. What happens if the three pluses just happen to come at the end of a burst of data? Unless something comes next, that modem is switching into command mode.

It's a hack for other reasons too. Consider that when the computer wants to switch the modem to command mode it has to send "+++"{1-second-delay}. To do this, it starts by sending "+", at this moment that looks like a byte of data. The modem cannot foresee the future (unfortunately) and so it must send the "+" to the remote end. Then the computer sends the second "+", and once again the modem sends it to the remote end. The computer then sends the third "+". Now, in theory the modem could hold that in a special buffer, wait to see if there will be a delay and if there is it could not send that third "+" and just switch into command mode. However, in my experience, modems generally don't do this. Thus, the remote computer receives the data string "+++", every time, just before the local computer is able to send a command to its own modem.

How could this not be considered a horribly ugly hack? For one, it is impossible for the transmitting computer to send a command to the modem in the middle of the receiving end expecting a string of data. The "+++" would end up in the middle of that data stream and there is nothing that the transmitting end can do about that, because the only way to get the modem into command mode is to send "+++" to the remote end!! That feels crazy to me.

A further complication of having the modem handle the dialing is that the command mode has to have a command format. Initially different modems had different command syntax. This was fortunately resolved by having one syntax, "Hayes," become the defacto standard that every other modem copied. All of those "AT"-prefixed commands are part of the Hayes standard. It was not a perfect solution though, because different modems support different features, and so exactly which Hayes commands work on which modem is inconsistent.

The other problem with RS-232 is that it has no facility for auto-detection of its various options. The options are: baud rate, word length, parity and stop bits.

Baud Rate

Data is sent over RS-232 by first initiating the transmission of a single word with a start signal, pulling the transmit line low. The receiving end then begins sampling the line at a fixed rate, while the transmitting end raises and lowers the line at a corresponding fixed rate. If those rates don't match, then the receiving end is sampling nonsense. Too fast and it's reading the same bit twice, two slow and it's missing a bit, with some risk that it's sampling during a bit transition.

The two sides have to be configured the same, but there is no automatic way for the computer to know the speed of the modem. Typical speeds are 300, 1200, 2400, etc. baud. Baud meaning the rate of symbols per second. Symbols per second is different from data-bits per second because the start signal, for instance, is a symbol that has to be transmitted too.

Word length

ASCII began as a telegraph code before its adoption by computers. True ASCII is only a 7-bit code. Using RS-232 to transmit ASCII, therefore, meant a data word only needs to be 7 bits long. When RS-232 was adapted to computers that use 8-bit data words a new complication was added to RS-232 configuration.

The receiving end gets a start signal and it starts sampling the data bits. How many data bits should it expect to receive? 7 or 8? (Or something else?) If the equipment sends and receives only 7-bit words then the computer needs to be configured for 7-bit words too, or very bad things would happen.

Fortunately, for us at least, our Commodore 64 is an 8-bit computer and our modems always deal with 8-bit data, and so we can safely assume 8-bit words and just ignore the history of the protocol's support for differently sized words.

Parity Bit

Early computers didn't have either screens or keyboards. Sounds crazy, right? But instead a teletype machine with a big roll of paper could be connected to the computer via RS-232. The asynchronous bi-directional nature of RS-232 is designed and well-suited for this situation. When the user presses a key on the teletype machine, it is encoded as 1 byte and transmitted to the computer. And inversely, when the computer wants to output a character, it transmits it to the Teletype machine which prints it out on the roll of paper.

The bytes are coming one at a time. The teletype machine is also one-to-one coupled with the computer, serving as its main input device and its main output device. The sampled nature of RS-232, however, is a bit unreliable. Therefore a method for detecting errors (a questionable method) was introduced: the parity bit. The parity bit can configured either as even or odd. If you add up all the data bits you'll get either an odd number or an even number. When the parity bit is added to the data bits the result should either be odd or even depending on how parity is configured. If the expected parity is not what is received, that indicates that an error occurred in the transmission of that word.

There are three major problems with the parity bit. First, it's not very reliable. It's easy for two errors to result in a correct parity, thus the error goes unnoticed. Second, it adds an extra symbol to every word of data. With one start bit, one stop bit, and 8 data bits, there is already 20% overhead on the transmission of every data byte. If a parity bit is included, the ratio becomes 3 to 8, or ~27% overhead, so a parity bit dramatically decreases the data transmission rate for little benefit.

The third problem is rarely mentioned in discussions of RS-232, but we'll talk about it a lot more later. The problem is that the protocol has no way to cope with an error. If the receiving end detects an error in a received byte, there is nothing it can do. It reports the error to the receiving controller, but if the error occurred while the sender is in the process of sending, say, 100 bytes, there is no way for the receiver to inform the sender that one particular byte was bad, and could it please resend that byte.

Given these three problems, the use of the parity bit is essentially obsolete. This is convenient for us, of course, because, like the 7-bit word length, we can relegate it to the history books and forget that parity was once a thing.

Stop Bits

The way that RS-232 start bits work is that the transmitter must pull the transmit line low. It is the transition to low that triggers the start of the next frame. However, if the final data bit from the previous frame (or parity bit if one were using parity) happened already to be low, then the start bit wouldn't transition; it would already be low.

The stop bit gives the transmitter time to raise the line. RS-232 can be configured for either 1 stop bit or 2... or even 1.5! You say, wait, how can there be 1.5 bits? This only means that the line is expected to remain high for 1, 1.5 or 2 times the symbol transmission time, according to the baud rate. So, at 300 baud, there are 300 symbols per second, or ~3.33ms per symbol. If the stop bits were 1.5 then the "stop period" or the wait period between the last data bit and the next start bit would be 1.5 * 3.33 or 5ms.

As mentioned during the discussion of parity, any unnecessary overhead per data word is detrimental to the overall data transmission rate. Therefore, once again, with our modern use of modems, anything other than 1 stop bit is essentially obsolete.

In summary, modems over RS-232 have their issues. Features offered by the modems, command sets, and command syntax are inconsistent. Moving between data mode and command mode necessitates transmitting nonsense data ("+++") to the remote end. RS-232 configuration is largely a manual process, but in the modern age, we can take some of the complication off the table. In C64 OS, all RS-232 drivers for network hardware is hardcoded for 8-bit data words, no parity, and 1 stop bit. Detection of baud rate is not part of the standard either, but we'll return to this issue to see what we can do about that.

Modems and RS-232 on Commodore 8-Bits

Mainframes and minicomputers were invented during the two decades before the arrival of microprocessors and home computers. These big computers were already in the full swing of telecommunications before anyone had a computer in the home. Although the Commodore 64 was early, it was not the first generation of home computer. Focusing just on Commodore, their first home computer was the PET. Released in 1977, it already included an RS-232 port. Although the RS-232 on Commodore computers was non-standard (an edge connector with TTL voltage levels, rather than DB25 with some higher positive/negative voltage swing like ±9V,) it was included primarily for interfacing with telecommunications hardware.

The VIC-20 was a lower-end but much more affordable home computer, released world-wide in 1981. Its branding was used for the VICModem (Model 1600) which was released in 1982 for a mere $100. The Commodore 64 was released in mid-1982 and was much more powerful than the VIC-20. For their 8-bit line of computers Commodore went on to release the modem models 1650, 1660, and finally the 1670, although they dropped the "VIC" branding. The VICModem (the 1600) was not able to do the dialing; it was a very simple device. You performed the dialing with the telephone as discussed earlier, and when the routing was complete and you a heard high pitched noise from the modem on the other end, you unplugged the cable from the handset and plugged it into the modem.

The Model 1650, which came out the same year, 1982, was branded as an AUTOMODEM, because it was capable of doing the pulse dialing. It did not however use Hayes commands. The Model 1660 came in 1985, with support for tone dialing. And finally in 1987, the internal guts of the Model 1670 became much more sophisticated and gained support for the Hayes command set.

Let's talk briefly about how the Commodore 64 communicates with these modems.

They plug into the C64's user port and the user port is primarily driven by one of the two CIA chips. The other CIA chip handles the keyboard and the joystick controller ports. Going back all the way to the 1960s there existed UART chips, (Universal Asynchronous Receiver/Transmitter,) which carry most of the heavy burden of RS-232 communications. However, the CIA (Complex Interface Adapter) is not a UART; it is a general purpose I/O controller with a few bells and whistles.

Commodore opted not to include a UART in the Commodore 64 to save costs. Instead, the KERNAL ROM includes software that allows the CPU and the general-purpose CIA, together, to emulate the behavior of a UART chip. The user port CIA chip has an IRQ pin which is connected to the CPU's NMI pin. The CIA also has timers that can generate an interrupt when they expire. And lastly there is a special /FLAG pin, which, when it transitions from high to low, also triggers an interrupt. These few abilities allow the CIA to manage RS-232 communications. The RS-232 receive line is connected simultaneously to the /FLAG pin and one of the CIA's data port pins. When the remote device begins a transmission, according to the RS-232 standard the start bit pulls the line low. This triggers the /FLAG interrupt in the CIA, which triggers an NMI in the CPU. The CPU then sets and begins one of the CIA's timers and masks the /FLAG interrupt. The remote end, according to the RS-232 standard, begins changing the state of the line. Because the /FLAG pin's interrupt has been masked, the changing state of the line for the data bits is not triggering an NMI. Instead, the CIA's timer is triggering the NMI at regular intervals that fall close to the middle of each bit to be received. Each time the NMI is handled the CPU reads the current state of the receive line and rolls the bit onto the incoming byte. That's RS-232 in a nutshell.

The RS-232 routines built into the KERNAL ROM have support for 5, 6, 7 or 8-bit data words, for 1 or 2 stop bits, odd or even parity, or no parity. The routines can handle 8 baud rates from 50 to 1200.

The problem with the KERNAL ROM routines is that they are very slow and inefficient. Every bit that arrives requires an NMI which comes with its own non-trivial CPU overhead. The handling of each bit has to go through all the checks for the various modes that are supported, even though most of these modes are rarely used. A received byte is put into a circular 256-byte receive buffer and the main loop of your program must poll the buffer by attempting to read a byte from it. The RS-232 input buffer is handled like any other device; the screen, keyboard, cassette drive, or IEC serial bus device. This means a chkin with a logical file number for the opened RS-232 device and a chrin to read from the buffer. Taken together, this is all very inefficient, which is why it is limited to just 1200 baud. At 2400 baud, even when the program is written in assembly, the KERNAL ROM's RS-232 routines become unreliable.

Terminals, Time-sharing, RS-232, Modems, and BBSes

To understand the development of telecommunications we have to understand how the technologies unfolded.

The early mainframes and minicomputers didn't have a screen or even a keyboard. This is almost unimaginable when we think about computers today. But it's true, they did not have a screen. As discussed earlier, one way they performed their input/output was via RS-232. Telegraph machines already existed as dumb machines capable of sending textual data using a keyboard and capable of receiving textual data and printing it to paper. By marrying a telegraph machine to a computer, you type on the telegraph keyboard and each keystroke sends one byte at a time into the computer via RS-232. If the computer program wants you to see what you're typing then it echoes your input back as output and the telegraph machine prints the character to paper.

This goes a long way towards understanding why modern command line interfaces in UNIX descendents, such as Linux and macOS, have a single line input at the bottom of the terminal screen. When you type "ls" to list the directory, the directory lists one line at a time, the screen scrolls from the bottom up towards the top, and you are left with a new single line input at the bottom. This is how paper works. That directory listing is meant to be printed to a roll of paper, and it is the paper that is scrolled up and out the top.

Eventually, paper-based telegraph terminal machines got replaced by video terminals. They still connect over RS-232 and they work in a similar way but instead of wasting all that paper the textual output is held and displayed from a single screen buffer. As new lines of content are sent from the computer the video terminal scrolls the screen up; the topmost lines are lost forever and new lines are added to the bottom of the screen.

Mainframes and minicomputers were very expensive, too expensive for individuals to own. Instead governments, banks, the military and universities and other research institutions owned computers. In order to lower costs of computing, time-sharing via remote terminals allowed perhaps 100 users to use the same computer at the same time. Although there was more than one way to connect a terminal to a minicomputer, one way was to use a video terminal connected to a modem via RS-232. The video terminal has an RS-232 port and the computer has an RS-232 port, but by putting modems between them they could be physically separated by a long distance.

Use of the phone lines might have cost money, but use of the computer time was even more important. Computer resources were unbelievably scarce and their usage was billed in multiple ways; billed by the hour for being connected, billed by the second for CPU usage, and billed by the month for every kilobyte of data stored. Modems made video terminals blissfully unaware of the distance between them and the actual computer. Users who worked on the video terminal felt as though they had their own computer, even though the terminal itself had no computational abilities.

This was "telecommunication." But the terminal was connected to just a single remote machine. Every keystroke was an independent packet of information with only a single target. The routing for the entire communications session precedes any data transmission. And so every packet of data, an 8-bit word with a parity bit, say, was routed by means of the phone network's switchboards.

Not too long after video terminals became available, home computers became available. The microprocessor in a home computer was not very powerful, but you got to own the whole thing.

Now consider, if you have a microcomputer with a modem then you no longer need a video terminal for connecting to a mainframe or a university minicomputer. All you need is a software package that runs on your microcomputer that emulates the behavior of a video terminal, and bingo-bango your microcomputer can now do telecommunications in precisely the same way as the video terminal. You can log into the time-sharing system of a big but remote computer.

Suddenly, you could play video games on your home computer and do professional work on a powerful mainframe running Unix simply by dialing in with your modem and terminal emulation software. I'm only 42 years old, and, believe it or not, this is precisely how my telecommunications history began. I owned a Commodore 64 and a 1200 baud Model 1670 modem. I used my C64 for games and small projects like school papers, but with my modem and Novaterm I could shell into a much more powerful computer running FreeBSD in the city that was about 35KM from where I lived. At the time that I was doing it though, that FreeBSD machine probably wasn't a minicomputer anymore, but a workstation with a powerful microprocessor.

Logging into remote workstations, minicomputers or mainframes to do real work might have been a powerful option, but most home computer users had no need to do this kind of work. As home computers with modems proliferated, a whole new opportunity sprang up. Instead of dumb terminals connecting to super-computers, why not home computers connecting to other home computers? A home computer might not be able to support 100 simultaneous users, but there's no reason why it couldn't support 100 users or a 1000 users if they connected sporadically, one at a time, for short periods of time. And thus was born the BBS or Bulletin Board System.

In principle, the BBS works the same way as remote dial-in time-sharing on a big computer. The major difference is that the host is another home computer dedicated to the task of running a bulletin board system. Many different BBS software packages sprang up. Users who dialed in with their terminal emulation software were not presented with a cold and daunting UNIX command line prompt, but an interactive menu system. Instead of the text-only video terminals of yesteryear, new terminal emulators were developed that could take advantage of the abilities of home computers. For example, the Commodore Color Graphics could take advantage of the PETSCII graphical character set and the full 16-colors of a Commodore 64 or Commodore 128 to draw expressive and impressive images and menus.

With the availability of BBSes, the excitement of telecommunication for home computer users grew. Since it was so grassroots, connecting to most BBSes was free, although your session time might be limited to only 20 or so minutes per day. One user could log in, leave a message in a forum or a private message for another user, and the next user could log in later to see and respond to the messages.

There were other online services that home users could pay for and access too, such as GEnie and CompuServe. I never used these, but in the early days they sort of functioned like big multi-user BBSes.

Despite it all, these services were not like the internet we have today. They were much closer in spirit to the telegraph machine connected to a mainframe, sending and receiving single characters at a time, between a single host and a single client, and rendering the characters that arrived according to simple commands that positioned the cursor or changed display properties like color.

While this was all going on for home users, the packet-switching internet was being developed by ARPA.

Packet switching and the internet

Packet switched networks began at ARPA and the creation of ARPANet, the Advanced Research Projects Agency Network. Packet-switched networks started being developed in the very late 60s and throughout the 70s. A number of early protocols were used, like the Network Control Protocol, which gradually evolved into TCP/IP. TCP/IP was adopted by the Department of Defense as their standard computer networking communication protocol in 1980. ARPNet switched to TCP/IP in 1982 and other protocols were then gradually phased out in favor of TCP/IP.

TCP/IP (Transmission Control Protocol and Internet Protocol, used together) is now the ubiquitous implementation of packet-switched networks. The basic concept is that a chunk of data, typically from just a single byte up to several tens of kilobytes, is packaged together with one or more headers. I am not going to go into the details here on how TCP and IP divide up the work, instead, for simplicity, treating them as one big header. The header is metadata with a well defined structured, describing properties of the payload data; its size, an error correction checksum, its time to live in network hops, its priority, its sequence number, its source address and port number, and its destination and port number, among other things.

The header and the payload data together comprise a packet. The bytes of the packet are always kept together when transmitted by the lower-level network hardware, giving robust structure to that overall sequence of bytes. The originating computer composes a packet with information about its own address and the destination address for the remote computer—these are more or less unique addresses on the whole global internet—and then delivers the packet to the nearest routing device. Complex routing tables are maintained for determining where the packet should be sent next. Each transmission of the packet decrements the packet's time to live preventing a packet from getting into an infinite transit loop. A typical packet will make 10 to 30 hops (but sometimes more) on its journey from the source to the destination.

When the initial packet arrives at the destination computer, that computer sends a reply packet, and there is an exchange protocol for establishing a socket connection; a virtual two-way data pipeline from one address:port combination to another address:port combination. The important difference between packet-switching and a dialup connection joining one computer to another, is that each packet is treated as a momentary unit of data; it's received, its headers are analyzed and modified slightly, it's then passed to the next machine in the routing and forgotten about. The computer or routing hardware that receives a packet handles it for a tiny fraction of a second before being freed up to handle the next packet. But the next packet may be coming from and going to a completely different origin and destination than the previous one.

The connection between two computers is virtualized across many small pieces, rather than one physical and continuous connection. It is much more like the postal system than like the original copper-wire phone system. Recall how on a busy day, like Christmas morning, it was sometimes impossible to make a phone connection to a family member because at that moment in time all the lines were occupied by continuous physical connections. This situation cannot ever happen with the postal system. You write a short piece of information, put it in an envelope and put a to-address and a return address on the envelope. You then deliver the envelope to your nearest post office or drop box. If the postal system happens to be very busy your letter still gets delivered, it just takes longer. Before you get a reply, you could send out another letter, and then another, and another. If you wrote a sequence number on each letter, eventually, when the receiver gets each letter, they could put the contents of the envelopes together in the sequence order you specified to reconstitute the full set of information. However, in the middle, you have no idea of the precise physical path that each letter is making, and it's quite possible that each letter takes a slightly different path through the overall system.

Packet switching vs. a dialup connection

Packet switching networks are an amazing invention that have unlocked the awesome power and interconnectivity of the internet, but they are also wildly more complex and have insanely more overhead than a computer with a modem dialing up a BBS.

In a packet switched network connection, every piece of data that travels through the network must be encapsulated in a packet. Every packet contains routing information which must be processed by 10, 20, 30 or more intermediate computational routing devices. Let's compare the journey of a single byte from a source application from one computer to the destination application of another computer over a dialup modem connection versus the same journey from an application on one computer to the application of another over a TCP/IP socket connection over the internet.

Let's say we're in our terminal emulation software, like Novaterm 9.6 for the Commodore 64. And we're already dialed up and our modem is connected to a remote computer through the old-school copper-wire phone network. You press the letter "A" on the keyboard. The terminal software writes that one byte into the modem driver's transmit buffer. The driver uses the RS-232 protocol to transmit that one byte to the modem hardware. The modem modulates the byte into frequencies that can be carried by the phone line network. That information journeys down the dedicated copper wire, through the switching station, possibly through mulitple switching stations that were switched (originally by human hands plugging and unplugging sets of cables) during the dialing/connecting stage, and into the modem on the other end. The modem demodulates the sound waves into a byte and sends the byte into its computer over RS-232. The BBS software on that computer does something with it, like perhaps treats it as input. If the BBS software wants that "A" to appear to the end user it echoes it back and the exact reverse process happens. When the original computer receives the "A" back again it prints it to the screen and the complete journey is over. That entire process will be repeated with each individual character that the user types, and so it goes.

Now let's see what happens with a TCP/IP socket connection. Assuming it's still a kind of terminal emulator, but connected over an internet socket, the user presses "A". The byte is written to the socket. The networking stack sees that the stream of data, at this time, only consists of one byte. It wraps that byte in a TCP header, computing all of the metadata for it. The TCP header is 20 bytes, including a checksum on the data. The TCP packet is then given an IP packet header, which is 20 more bytes. The total packet header size is 40 bytes with just one data byte. The lower level networking hardware then needs to transmit the packet to the local router. If this is done over ethernet, the TCP/IP packet is wrapped in an ethernet packet, with its own checksum, targeting the MAC address of the router. This is another 18 bytes. Ethernet has its own protocol for dealing with collisions and pullback timeouts and retries, until the ethernet packet is successfully transmitted to the local router. The router then unwraps the ethernet packet, and from there no one has any idea how it transmits it to the internet service provider because it depends on the underlying technology: cable modem, DSL, fiber optic, cellular, satellite, etc. Somehow the TCP/IP packet is routed to the next major hub which analyzes the packet and passes it to the next major hub until eventually the packet makes it to the ISP that owns the IP address block of the destination computer. The packet comes into the modem in the home of the destination computer, then into a router, which, by the way, might be doing network address translation to support multiple local computers on a single IP address for the entire house. The router then wraps the packet up with an ethernet packet to transmit to the ethernet port of the computer, which unwraps the ethernet, and unwraps the IP packet, and unwraps the TCP packet, validating and verifying the checksums along the way, and then buffers the data into the socket, and signals that the socket has data. The Application then reads the byte from the socket's buffer and does something with it. If it's a TCP/IP-based BBS, and it wants to echo the "A" back to the user, everything then happens in reverse. In fact, even if it doesn't want to send any data back, TCP/IP requires that the packet be acknowledged, so the network stack would automatically return an acknowledgement packet which undergoes the same wild journey in reverse.

Now, this is a bit dramatic, because when you're downloading a file, like a graphic or a chunk of HTML, etc., a packet can carry much more than a single byte of payload. Additionally, the routing hardware along the way is unspeakably fast. In the length of time it takes to transmit a single byte over RS-232 at 1200 baud, a standard, inexpensive, home internet connection of 150 Mbps can transfer 156,250 bytes. So just a standard home connection is more than 156 thousand times or 5 orders of magnitude faster. Routers used in the internet backbone process data at around 100Gbps. These can move data at a rate of around 13,107,200,000 bytes per second, or around 110 million times faster than an RS-232 connection at 1200 baud. This is why even when a one-byte TCP/IP payload has anywhere from 40 to 60 bytes of headers, making it ~97 to ~99 percent overhead, its voyage through the internet is effectively instantaneous. It has to be this fast, when you consider that voice over IP or video conference calls over the internet are realtime with basically no latency.

Therefore, even when two dialup modems are connected to each other by a long pair of dedicated copper wires and there is zero switching overhead per byte of data transmitted, the modern packet switched internet is so much faster that any inefficiencies due to overhead are overshadowed by about a million fold or more. The internet is so fast and can handle so much data that even traditional analog phone calls are no longer actually carried by a copper-wire phone network. They may still be carried by copper wire to the local telephone switching station, but from there they are digitized and bridged onto the internet, and then converted back to analog for the last stretch of the journey if the destination is another traditional landline. What exactly you're paying for today when you pay for a long-distance phone call is kind of a farce. That's why you can place a FaceTime or a Discord video call, or whatever, to anyone on the planet and it costs nothing above and beyond your internet service plan.

RS-232 bridged TCP/IP Wifi Modems

For the Commodore 64 and other retro computers, doing TCP/IP is not impossible, but when you compare the journey of a byte over a dial-up modem connection to the journey of a byte through the internet, the complication of journeying through the internet has to be borne at one end by the C64.

It really is the case that when a C64 sends one character it must be wrapped in a TCP packet, wrapped in an IP packet, and wrapped in an ethernet packet, adding around 58 bytes of headers to that one byte of data. And while it may be true that the internet backbone can move those packets with phenomenal speed, overshadowing any loss of efficiency due to overhead, the C64 itself is still saddled with managing those packets and computing the checksums, etc. This is a lot to ask from a computer that moves 8-bits at a time, runs at 1MHz, and has access to only 64 kilobytes of memory. It's clearly not impossible, but it is a lot to ask. And the more memory you use implementing TCP/IP the less memory you have for doing everything else.

Along came TCP/IP modems, over Wifi, via the ESP32. They do all of the TCP/IP work in the modem and present to the C64 just a raw stream of data that is reconstructed from the TCP/IP packets. On the one hand, it has been an incredible boon. For example, with a simple ATD command, followed by a hostname and port number rather than a telephone number, the modem opens a TCP/IP socket connection, and from the perspective of the computer it is no different than being dialed up to a remote computer via classic modems. The communication between the computer and the modem is still over RS-232.

For BBSes it's a boon. But, networking today, especially TCP/IP networking, is capable of so much more than what BBSes can offer. Remember, being connected to a BBS via terminal emulator software is inspired from the days when a telegraph machine sent individual characters directly into the mainframe to which it was directly connected. A BBS effectively makes your Commodore 64 a remote keyboard and screen for a text-based computer interface running on a host computer somewhere far away. But TCP/IP lets us connect to anything. We could connect directly to a webserver, send that webserver a GET command and some headers, and it would return to us the HTML content of a webpage.

As powerful as that sounds, this is also where the problem begins with bridging TCP/IP to RS-232. On the one hand it relieves the C64 of all the burden of TCP/IP and Ethernet/WiFi networking overhead. But on the other hand we also lose all of the benefits of TCP/IP; all the abilities afforded by all of the features built into those many headers are suddenly unavailable to us and in fact are completely hidden from us.

For example, let's say we're going to download a .D64 directly from a webpage. First the C64 issues a HAYES-style command to the modem to open a network socket to a webserver. The socket is open and the RS-232 data received and transmitted are now coming from and going to that remote webserver. The C64 sends an HTTP GET command to specify the .D64 file it wants to download. The webserver replies and starts dumping the file down the socket connection. The modem is receiving the TCP/IP packets; the modem is computing the checksums; the modem is ordering any out-of-order packets and buffering the data; the modem is handling the flow-control from the remote webserver by sending the acknowledgement packets, and receiving resent packets if its own buffer fills up before the C64 can read the data in over RS-232. All of this is being conveniently and opaquely handled for us. All we have to do is receive bytes of data that are coming in over RS-232.

But what happens if we miss one of those bytes because our own buffer overflows? What if we misread one of those bytes because an unexpected NMI (like pressing the RESTORE key by mistake) interferes with the careful bit-banged timing? The question is, what happens? Do we even know that we've got an error in the data? If we misread a byte, and somehow we know it, how do we tell the modem to resend that data? Even if we could ask it to resend, how would we specify how much to resend? That last point is moot anyway, because we cannot ask the modem to resend. If somehow we knew that we had misread a byte but we can't ask the modem to resend, what are we even supposed to do? Ride it out to the end? Try to interrupt (by sending "+++" and a 1 second pause) and then drop the connection? Then what, try to download the same file again from the start and hope it doesn't go bad this time? This is moot too because we have no way of knowing if we've got an error. So what happens in practice? We download the entire 664 block .D64 file and then we're confused later when it doesn't mount because it's not precisely the right size? What if it is the right size but it's got an error somewhere in the middle of it?

Here's the point. TCP/IP is complicated but the complication exists for a reason. Letting the modem handle all the TCP/IP means that we have a robust, error-corrected, flow-controlled and packet-switched connection from the remote computer all the way to our modem. But from the modem to our Application's code we have no error correction, no structured units of data, no ability to rerequest a chunk if an error were detected, no ability to interleave data from more than one source, and flow-control that is all-or-nothing with typical RS-232 complications. A chain is as weak as its weakest link and the last 10cm of the data's journey is very weak indeed.

Modem Feature Variability, and Minimum Viable Feature Set

One more complication that we haven't really discussed is that not every TCP/IP (i.e., Wifi) modem is the same as every other. There are several different ESP32 firmwares. The most robust, in my experience, is ZiModem developed by Bo Zimmerman. But there are others, such as by Paul Rickards, and the SwiftLink emulator in the Ultimate II+ and Ultimate64. These firmwares also have different versions. Between the firmware types and their versions, the set of configurable options varies widely. The exact HAYES commands and command syntax used to set these options varies considerably.

The modems also have different hardware options available to them. The C64Net, user port, has full support for hardware flow control and for carrier detect via physical signal lines. Whereas the user port modem sold by Shareware Plus, running the Paul Rickards firmware, not only has a different firmware and command set, but the modem is missing certain signal lines. For example, there is no carrier detect signal line. And I mean, it's not just a firmware issue, the User Port pin used for carrier detect is not soldered to anything on the ESP32. (The hardware is literally missing that trace.) The C64Net modem also has jumpers for configuring the modem for UP9600, which uses a radically different technique for transferring data. UP9600 uses the serial port in CIA2 to help move the data at a much faster rate. What other modems support that, which don't? Which modems support hardware flow control, which don't? ZiModem firwmare also supports a limited packet-based receipt of data, allowing for more than one socket to be open at a time. But if we were to use that feature—or if we depended on that feature—how would we also support other modems that lack this feature?

What we want from a modem is a minimum viable feature set. This means a feature set that:

• Can be reasonably expected to be available from a wide variety of hardware options, and
• Offers some minimum capability that can actually be worked with.

Variation in the command set handled by the firmware is expected and is acceptable as long as a driver can handle the differences up to the point of being able to make an API that abstracts those differences to something common, which the next level of the networking stack can work with.

What this means in practice is:

• The modems have to be able to connect to a TCP/IP socket and asynchronously stream data in both directions. More than one socket at a time, at the hardware level, is not a requirement because that would eliminate the largest swath of hardware options available.
• The modems have to support hardware flow control. Although some modems do not support this feature, it is a common feature of RS-232 modems going back a very long time. In C64 OS, where the REU needs to halt the CPU to refresh the screen, having control to momentarily pause the flow is essential to not losing data.
• The modems have to support hardware carrier detect. As discussed earlier, all but the oldest dial-up modems mix command mode and data mode over the same RS-232 data channel. C64 OS needs a reliable programmatic way to determine when data coming from the RS-232 connection is coming from the modem itself or coming from the remote end. Hardware carrier detect is a separate signal line that is asserted while the modem has a remote connection open.

As of this writing, these constitute the minimum viable feature set required from a modem to be usable with C64 OS. The actual firmware command set, the supported baud rates, the communications mechanism (UP2400, UP9600, UART chip like SwiftLink or Turbo232, or some other technique like WIC64 uses), are handled by different drivers. Extended feature options, such as PETSCII translation in command mode, commands to fetch a list of SSIDs, and commands to join a WiFi hotspot, are optional and may be taken advantage of by drivers but are not required.

C64 OS Networking Stack

Having read everything above and understanding the lay of the land, we now turn to how C64 OS's network stack addresses and solves these issues.

The networking in C64 OS consists of an ensemble of parts which work together to provide Applications and Utilities with an asynchronous network interface that is quite straightforward to program and provides multiple reliable data channels, while fully abstracting the underlying hardware.

The parts consist of:

• Network Hardware Drivers
  (//os/drivers/:nhd.*)
• Commodore Network Protocol Library
  (//os/library/:cnp.lib)
• Coordinating Network Library
  (//os/library/:network.lib)
• Programming headers for sockets and drivers
  (//os/h/:cnp.h,network.h,nhd.h, //os/s/:cnp.s,network.s,nhd.s)
• Network configuration Utility
  (//os/Utilities/:Network)
• Network settings files
  (//os/settings/:network.t)
• Network status flags and system messages
  (netstat, mc_ntwrk)
• Network status in the status bar

When an Application (or a Utility) wants access to the internet it loads the coordinating network library, network.lib. This is a shared library, so if the underlying Application has already loaded this in and a Utility is opened that also wants to access the internet, the network library and other components are only in memory once.

The Application makes a call to the network library to tell it to fully boot the network. This happens automatically upon launching the network-using Application. For example, after a fresh boot of the OS, you find yourself in App Launcher. The network stack has not been initialized yet in any way. You double click the Wikipedia App (included in v1.07) and it launches. It loads network.lib and tells it to fully boot the network. If all goes well, you go online which is indicated in the status bar, and without any further intervention you can start making searches and getting results.

The network library is called a coordinating library because its job is to pull together all the other parts and link them to one another. The network library loads settings from //os/settings/:network.t which holds three distinct categories of settings:

• The network hardware driver and its initial and preferred speed.
• The Wifi SSID and password.
• The CNP server's hostname and port and your username and password.

If any of this information is not available, (for example, the first time you try to use C64 OS networking and you've never configured it before,) the network library loads the Network Utility. If you have already manually opened the Network Utility and configured all the network settings, then those settings are already available and the network library's full network boot will simply proceed using those settings. If something goes wrong, such as, the Wifi SSID or Password doesn't work, or the CNP server can't be connected to, or your CNP authentication fails, or anything like that, the network library handles this by opening the Network Utility so you can see the details and correct the problem by updating the configuration.

You can manually open the Network Utility to adjust settings and manage the network configuration at any time. Choose Settings from the Utilities menu. In previous versions of C64 OS a placeholder button called "Wifi" was greyed out under the hardware section. Starting in C64 OS v1.07 that button is enabled and has been renamed to "Network".

Network Utility

The Network Utility itself also tries to load the network library. If the Network Utility is opened automatically because an App was trying to boot the network but couldn't, then it uses and accesses the same network.lib that the App loaded in because it's a shared lib.

The Network Utility has 3 tabs that correspond to the 3 areas of configuration: Driver, Wifi, and CNP server.

Network Hardware Driver

You pick the network hardware driver with a cycle button that corresponds to the combination of hardware and firmware that you're using. There are several modems that connect to the user port that run different firmwares or offer options for different hardware modes.

For example, if you have a C64Net user port modem, it uses the ZiModem firmware. It can also be in either 2400-baud mode (physical jumpers set for RS-232 to be fully bit-banged) or in 9600-baud mode (physical jumpers set to use different CIA lines for the special UP9600 protocol.) Let's say your C64Net is configured for 2400-baud mode; you would choose the driver "up24.zi" which stands for "UserPort2400" "ZiModem". If your C64Net were configured for the 9600 baud protocol, you would choose the driver "up96.zi".

There are similar drivers for other combinations of hardware and firmware. There are other firmwares for modems on the user port and there are also modems that connect via an ACIA/UART chip such as Swiftlink, Turbo232, or IDE64's DUART. Each supported "hardware + firmware" combination gets its own driver and new drivers can be written to support new hardware or hardware not yet supported.

After choosing a driver, you choose an initial baud rate and a preferred or maximum baud rate. These values range from 300, 1200, 2400, 4800, 9600, and so on up to 230kbps. If you pick an initial or maximum value that is out of range, the driver should adjust these to be in range.

You may be asking yourself, why would you ever choose some value slower than what is possible? Why would a modem ever be set to 2400 when it could be set to 9600? Why would a modem ever have an initial value of 300 or 1200 baud, when it can be bumped up to 2400? Why not just configure your modem for 2400 (or 9600) once and leave it like that? There could be several reasons. For example, the UP9600 protocol works on a C64 but not on a C128 in C64 mode. Another reason might be that your favorite terminal software for BBSing doesn't support the UP9600 protocol. C64 OS aims to be unintrusive in the way you use your hardware with other non-C64-OS software. Why leave the modem configured for 300 or 1200 baud on power up, only to step it up to 2400 in C64 OS when the network boots? One reason might be that BASIC can only handle up to 1200 baud. If you want to access your modem from BASIC, that's a reason to leave its default configuration (in its own firmware settings) at 300 or 1200 even though it will get stepped up to 2400, temporarily, when it's used by C64 OS.

As discussed earlier in this post, RS-232 is an old protocol with lots of options, none of which were designed to be auto-discoverable. But some of these options in a modern context no longer make sense to use. By streamlining the drivers for specific settings, the drivers can be made faster, more efficient and more reliable.

With certain options fixed (8-bit words, 1 stop bit, no parity,) testing for a matching baud rate becomes a tractable problem. Your modem has a baud rate stored in its firmware, often this is 300 or 1200 baud. In the Network Utility, you'll probably want to bump this up to 2400. During driver initialization the driver has to be able to speak to the modem, at the speed it's currently at, in order to send it the initialization string and then to send the command to change its speed. Say the driver starts communicating with the modem at 1200 baud and sends the command to change speed to 2400 baud, then the driver changes its own speed to 2400 baud to match.

After a fresh power up, this is quite straightforward. But things can get complicated quickly. If you use C64 OS to launch a non-C64-OS terminal software package, like Novaterm, you could use that to change the baud rate of the modem. When you boot back into C64 OS the modem is no longer in its firmware default state. You could power-cycle the computer in order to reset the modem, but then you'd lose the contents of the REU; C64 OS's fast app switching and fast reboot would get reset too. You could push a button on the modem to reset just the modem, but only some modems have such a feature. Instead, the C64 OS network hardware drivers perform baud rate testing. The driver starts by communicating at the initial speed and sends a test command to the modem. If the communications speed doesn't match, the modem fails to respond in the expected way. After a failed test the driver steps up its own speed and tries the test again. If the test passes, all is good and that phase of initialization passes.

It's not 100% foolproof. If you use other software to change the modem's speed to something other than the initial or maximum speeds configured in C64 OS, the tests will still fail and you'll need to take some manual steps to get them talking to each other. On the other hand, it's pretty good, because there is a more common situation that this handles very well: You do a fresh power up, boot C64 OS, and C64 OS steps up the modem successfully. Then you quit to BASIC and don't run anything else that touches the modem, but you either don't have an REU or the contents of the REU get replaced by something else and you have to do a fresh boot of C64 OS. In this situation the modem is not in its firmware defaults but it's because it has been changed by C64 OS itself! The testing and step-up procedure handles this situation like a champ.

This may sound like it's complicated under the hood. That's because it is complicated. But the goal is to make it as seamless as possible for the user.

Joining a Wifi Router³

The second tab lets you configure which Wifi router to join. It has fields for an SSID and a password. The password field uses the new secure flag on the TKInput object to display asterisks instead of the password.

Although joining the Wifi router can be saved to the firmware settings of the modem, C64 OS maintains its own configuration for the router and can join a different router than the firmware settings. From the Network Utility you can click join to join the router or click the refresh button to recall from the modem which SSID it's currently connected to.

The network hardware driver supports the ability to join a router or to read from the hardware which SSID it's currently connected to, but the driver still depends on the "hardware + firmware" to have this low-level ability. For example, on the Ultimate64 and Ultimate II+ you cannot configure which SSID to join using commands sent to the modem. You have to set this through the backend menu system. In this case, there is nothing the driver can do except provided some sensible default feedback. This is similar to how the IDE64 does not have a DOS command to set its realtime clock. In that case, there is nothing the RTC driver can do to set the time; it just has to be set through the IDE64's CMOS Setup Utility.

Joining a CNP Server

The third and final tab in the Network Utility is to give you the ability to join a CNP server. Much more detail about what CNP (Commodore Network Protocol) is can be found below. The CNP tab provides 4 fields: Host, Port, Username and Password. Password is a secure field again. Plus two buttons, stop and start.

The CNP server is a packet routing proxy server with support for up to 8 open sockets per Application bank with up to 32 Application banks for a total of 256 open sockets. In practice Fast App Switching only supports 30 Apps open at the same time, but the CNP networking still supports 8 open sockets in each Application bank. The current Application, any Utility open with it, plus any system-level process using networking in that same App bank, share that pool of 8 sockets.

The CNP server is (will be) open source—it's written in NodeJS—and can be installed on any modern hardware: Linux, macOS, Windows, or anything that can run NodeJS. There is an official CNP server hosted at services.c64os.com, but I am still considering whether I will need to charge a nominal fee to help me offset server costs. I already pay monthly for my server space and bandwidth and as I start hosting proxy servers my costs are certain to go up. I'm trying to figure out the most reasonable way to keep the services perpetually available, and the only way to do that is to not have them constantly lose money.

However, I don't want even a nominal fee to prevent C64 OS users from taking advantage of the networking, which is why I am open-sourcing the CNP server and other NodeJS-based proxy servers. If you're a hacker who enjoys configuring servers and who doesn't want to pay anything, you're welcome to configure your own proxies. If you want to sell access to the proxies running on your own infrastructure, the license will probably allow for that too. But this is still something I'm thinking on.

When you join the CNP server, the other options in the Network Utility are greyed out. You have to be offline, i.e., not connected to the CNP server, in order to change SSIDs for example.

Saving Settings

Lastly, at the bottom of the Network Utility, not within any of the 3 tabs, are reload and save buttons. Clicking save preserves any changes that were made in any of the 3 tabs. The settings are written to //os/settings/:network.t. Passwords in that file are in plain text for now.¹

The reload button allows you to recall the current settings from disk, overwriting any unsaved changes in the fields. For example, you could temporarily change the SSID and Password for the Wifi router and click join to join that new router. But then click reload and it will pull in your previously saved values for SSID and password. Click join again and you're back on your typical router.

Commodore Network Protocol (CNP)

Applications and Utilities in C64 OS that use the internet do not have direct and unfettered access to the network hardware. Although people have made some noise on Twitter/X, YouTube comments, and elsewhere, that I should just let people use the hardware directly, these comments and suggestions are shortsighted.

Multiple Applications and Utilities need to be able to share access to the network in a reliable and hardware-independent manner. And after a lot of thought, I believe the networking stack that I've been developing over the last few months is the best way to do that.

Full Network Boot

The Network Utility is used to test and configure the individual stages of getting online. It lets you set up the driver and test that the driver can talk to the hardware and set its communications speed. It lets you set up the Wifi network and test that you have indeed joined a router. And it lets you set up the CNP connection and authentication credentials and to start and stop the CNP connection.

However, once you've used the Network Utility to configure and test everything, those settings are saved. An Application or Utility, that wants to go online, makes an API call to the network library to perform a full network boot.

The only thing an Application need do to go online is to load the network library (network.lib) and then that does most of the work for you. The App tells the network library to perform a full network boot; it loads the CNP library; it loads the settings from //os/settings/:network.t and uses those to load the correct network hardware driver; it coordinates with the driver to initialize the network hardware and set its speed; it makes certain the network hardware is connected to a Wifi router; and then it starts the connection to the CNP server. The network library links the CNP library to the network hardware driver. It hands control over to the CNP library with a structure containing your credentials and the CNP library authenticates you with the CNP server. All of this happens, without user intervention, when the Application starts up.

Since v1.0 the C64 OS system status bar has had a drive status mode. Starting in v1.07 this has been extended to be a status mode. It shows the network status, the latest drive status, and the total free memory. The network status only shows "Online" or "Offline."² The Online status means that, in this particular App bank, all of the required steps are active:

• Network hardware driver is installed
• Network hardware is configured (computer can communicate with network hardware)
• Network hardware is configured with router credentials
• Network hardware is connected to router
• CNP server address and credentials are configured
• CNP server connection is up

Those are 6 different steps which correspond to 6 bit flags in a new workspace byte for network status.

When you switch from an Application that uses networking to an Application that does not, all of the network status bits are maintained, except for the installation of the network hardware driver. In other words, one of those bits (bit 0) goes low when you switch from an App that has the full network stack booted up to an Application that doesn't use networking at all.

If you then launch another Application (or Utility) that wants to use networking, it loads the network library and tells it to do a full network boot. Using the network status flags, it can see that portions of the network are already up. It loads in the CNP library and the network hardware driver, but because the status bits indicate the state of the network, this instance of the driver has its speed configured to match the modem without attempting to initialize or adjust the modem's speed because that's already been done.

Similarly, joining the wifi router doesn't need to be done, and as long as the hardware's carrier detect is still asserted then the CNP server connection is still up. So the full network boot takes only those steps necessary to get the current App bank ready to use the network.

The Application developer doesn't have to worry about any of this. You just load in network.lib, tell it to do a full network boot, and then you're ready to make CNP library calls to manage socket connections.

The network library sends out a new message type to both Utilities and Applications, mc_ntwrk, whenever there is a change to network status. This allows the Application to become aware of the operating system going online or offline. During a full network boot, certain error conditions, such as a network hardware driver not being configured, or the driver's inability to establish communications with the network hardware, result in the Network Utility being opened automatically. This allows the user to see the details, set things up for the first time, or change settings as required (such as if your wifi router's password changes, or your CNP server credentials change, etc.) The Application need not be involved at that level. It just pays attention to the network status messages and waits for C64 OS to go online.

CNP Sockets

Finally we come to a description of how the Commodore Network Protocol works.

The network library opens the connection to the CNP server by making a request to the network hardware driver to open a TCP/IP socket and passing in the host and port of the CNP server. When that connection is up, control over the modem's incoming and outgoing data buffers is transferred to the CNP library. The first thing the CNP library does upon gaining control is send the authentication, the username and password. If the authentication fails, the CNP server simply closes the TCP/IP socket and the network library retakes control over the modem's data buffers. If the CNP library doesn't transmit the authentication within 10 seconds, the CNP server terminates the connection. After sending the authentication, for as long as that TCP/IP connection remains open, the CNP library has control over the network hardware's data buffers and considers itself active.

The CNP library configures a keep alive timer to transmit a single byte keep alive command once per minute of no other activity. Whenever socket activity occurs the keep alive timer is reset, deferring it from sending a keep alive until a minute of inactivity. The CNP server is configured to keep the TCP/IP connection open for 10 minutes of inactivity. If the CNP server connection is up, and you switch to an Application that does not have the network stack loaded in, or you drop to BASIC to do other things, the keep alives are no longer being sent. After 10 minutes, the CNP server will drop the connection and any open sockets are closed and expunged from the server side. The exact interval is something we can play with, and could be different on different instances of the CNP server.

The CNP library has an open sockets table, capable of holding 8 socket pointers. The table of 8 sockets is different for each instance of the CNP library in different App banks. A CNP socket consists of a structure that the Application (or Utility) creates. In most cases an App will statically allocate a socket or two or three, but could also dynamically create a socket structure, if it needed to.

Opening a Socket

The Application opens a socket by calling opencnps in the CNP library and passing a pointer to the socket structure. The CNP library writes a pointer to this socket struct into a vacant slot in its socket table. The socket itself has a pointer to a string containing the hostname and port to open.

The CNP library flags the socket as opening (sets the csf_opng bit on the csk_flgs property.) It then sends a socket open-type packet to the CNP server, with the "hostname:port" as the packet's data payload. This packet is structured like any other CNP packet, with a port number, data size, data checksum, etc.

CNP Packet Structure

Remember how a TCP header is 20 bytes, and that gets wrapped with an IP header that's 20 more bytes, and that gets wrapped in an ethernet or a wifi packet which adds another 10 to 20 bytes? That's a huge amount of overhead, especially if the payload is small, like 1 byte. The difference is, a TCP/IP packet has to carry a segment of a data stream all the way from one process on one computer to a partner process on a different computer, somewhere else on the internet, anywhere in the entire world.

Whereas, a CNP packet has to route a segment of a data stream from one process in C64 OS (one socket, owned by one Application, running in a particular Fast App Switch bank) to the CNP server. And that's it, because from there, the CNP server bridges that data to the internet over TCP/IP. Therefore, the overhead of a CNP packet can be very small. How small? I got it down to just 4 bytes. From ~50 bytes down to 4 bytes makes a big difference when data is being exchanged between the computer and the network hardware at just 2400 baud. Meanwhile those 4 bytes allow us to maintain many of the advantages of packet switching that we lose when we let the modem do everything for us.

A CNP packet consists of a 1-byte type, which comes first, followed by a 1-byte port number that identifies the socket that the packet belongs to. Packet sizes vary by type, which is why the type byte comes first.

Packet Types

As you can see in the packet types table above, a keep alive packet is a well-defined packet type. The first byte sent is the packet type. If that is either a "+" or "-" then it's a keep alive packet. This is defined as having a header size of 1 and a data size of 0, so the packet consists of just the type byte.

The reason it is "+" or "-" is so that if the network hardware driver needs to switch the modem (over RS-232) from data mode to command mode, it can issue "+++" (followed by 1 second with no data). The pluses are sent to the CNP server, but it interprets them as keep alive packets. The "-" is typically sent for keep alive, so as to prevent the modem from mistaking 3 keep alive packets in a row with the sequence to enter command mode.

Continuing with the description of opening a socket now, a new socket is opened by sending an open socket-type packet. The first byte is pt_open. All of the packet types are human readable (and typable) characters. Most have some mnemonic value, "o" for open, "c" for close, "d" for data, etc. Most packet types (except keep alive and server messages) belong to a socket and identify that socket by using a port byte.

Port Format

A CNP port is similar but different from a TCP/IP port. A CNP port is basically a socket identifier. When the CNP Server receives a packet (from one of the many C64 OS computers connected to it) the TCP/IP socket the data comes in on identifies which CNP connection it's for. The port number in the CNP packet header is used to figure out which bridged socket this packet belongs to. And in reverse, when data from a bridged socket arrives the CNP server specifies the port in the packet it sends to the C64 OS computer, so that when the packet is received by the C64, it knows which socket that packet belongs to.

A port is just one byte, which allows for up to 256 simultaneous sockets to be uniquely identified. One problem that had to be addressed is that an App with an open socket could be Fast App Switched into the background, while another App with its own sockets gets Fast App Switched into the foreground. (Complications like this are why I implemented Fast App Switching before tackling networking.) A port can't just be an index into a socket table, because the table of sockets changes when an App is switched.

A port, therefore, is structured like this:

The upper 5 bits hold the Fast App Switch bank number. When an Application (or a Utility) loads the network library, and it loads the CNP library, the CNP library reads the current App bank shifts it left 3 times and saves that as a port mask.

The lower 3 bits are an index into the socket table belonging to just one App bank. 5 bits for App bank allows for up to 32 Apps, which is just slightly more the Fast App Switching limit. And 3 bits for the socket table index allows for up to 8 sockets per App bank.

Back to opening a socket again; when an App tries to open a socket, the socket pointer is put into an empty slot in the socket table. A packet is then assembled which consists of the packet type byte pt_open, followed by a port number. The port number being the socket index assigned to this socket when it was slotted into the table, logically OR'd with the port mask to give it the current App bank.

The socket structure has a pointer, csk_name, to the "hostname:port" string. This string's length becomes the packet's data size value. Next the bytes of the string are XOR'd together for a fast and simple error-detection algorithm and that byte is used as the packet's data checksum. This 4-byte header is then sent, immediately followed by the packet's data, which is the "hostname:port" string. The socket's status flag csf_opng is set.

Opening a CNP socket, like all other socket operations, is asynchronous. Your App code opens the socket by calling opencnps and passing a pointer to the socket struct. Everything else described above is what the CNP library does. It writes that packet out to the network hardware driver and immediately returns control to your App.

Handling various open responses

Your Application is doing nothing except waiting, but its user interface is fully active. The driver is asynchronously spooling the packet from its outgoing buffer to the network hardware. The CNP server is receiving the data, one or a few bytes at a time.

The CNP server receives the first byte, it's a pt_open byte so it knows its receiving an open-socket-type packet. Then it receives the port and creates a bridge socket object that it identifies by this port number. All of this, of course, is managed separately for each C64 OS client. The CNP server will typically have many C64 OS clients connected at the same time. It knows which C64 OS client it receives a packet from because the data comes in on a TCP/IP socket for a particular C64.

The CNP server receives the data size and the checksum byte, and it then expects to receive a number of bytes equal to the data size. There should be no unreasonable delay in this data arriving because it all belongs to a single CNP packet. In other words, it is definitely not the case that the C64 itself could still be waiting on something, like, reading from storage or getting input from the user, etc. Once the packet starts to be sent from the C64, the checksum has already been computed, all the data for that packet is dumped to the network hardware device, and there should be only normal TCP/IP packet delays in getting the full CNP packet from the C64 to the CNP Server.

The CNP server computes the checksum on the incoming data. If the CNP server fails to receive the full packet within a short period of time, or if the checksum fails, the CNP server sends a pt_nak packet in reply. This is only two bytes. It sends the pt_nak type byte followed by the port number. The C64's CNP library gets the NAK packet and connects it to the correct socket via the port byte.

But, what does a NAK mean in this case? The CNP library uses the status flags on the socket to give that NAK context. Since the csf_opng flag is set, it knows that it got a NAK during the opening phase. This means either: the checksum failed, or the packet in its entirety failed to be received. Without even informing your Application, the CNP library immediately retransmits the socket-opening packet. It can do this without intervention from the App because it has everything it needs from the socket structure.

When the CNP server gets the full packet and the checksum passes, which is what normally happens, it opens a new remote TCP/IP socket using the "hostname:port" open-packet data for where to open that remote socket. Note that, C64 OS doesn't deal with name resolution or caching of names to IP addresses. It could send the "hostname:port" string as an "IPaddress:port" though. Opening the remote TCP/IP socket is also asynchronous for the CNP server.

If the remote TCP/IP socket eventually times out, or the name resolution fails, or it cannot establish the connection for any reason, the CNP server replies to the C64 with a timeout, pt_time, packet. The CNP server destroys the bridge socket object that was created. The CNP library on the C64 removes this socket from the socket table, clears the socket status flags (it's no longer marked as opening, csf_opng) and it sends a message to the process that owns this socket (it could be an App, or a Utility, or some other process) by calling the socket's status callback, to which it passes a socket state change notification.

Socket State Change Notifications

When an Application creates a socket, the socket must have a pointer to a status callback. The CNP library informs the process of changes to the socket's status by calling this callback and passing in one of the above status change notifications.

While opening a socket, if the CNP library gets a NAK packet, it doesn't notify the App, it just retransmits the packet. After several such failed attempts, or if the CNP Server's attempt to open the remote TCP/IP socket fails or times out, then a timeout, pt_time packet is sent back. Only then does the CNP library remove the socket from its socket table, and call the status change callback with a csc_fail notification.

The Application can respond to that however it sees fit. Maybe it displays a message to the user that the connection could not be established.

If all goes well though, the CNP server receives the open-type packet. The checksum passes, it tries to open the remote TCP/IP socket, and it opens. Finally, the remote socket is associated with the bridge socket object, and an ACK packet, pt_ack is sent back to the C64. Again, what does ACK mean in this context? The CNP library sees that the socket's csf_opng flag is set, so the ACK means that the socket opened.

It clears the csf_opng flag and sets the csf_open flag instead. Then it calls the socket's status change callback and passes it the csc_open notification. Now the Application is ready to receive data (asynchronously, at any time) or to transmit data on that socket.

Receiving Socket Data

Let's start with how data gets received on a socket and how the Application handles this.

We can imagine that we've opened a socket connection to a service that immediately sends some information. Like, maybe it sends the top 10 stocks on the NYSE and their current trading value. Shortly after the remote TCP/IP socket is opened, that socket receives data from the stock info service.

The CNP server packages it up in a new CNP packet to send to the C64. Packet type is pt_data, the port is whatever port was assigned to this bridge socket object when it was created from the open-type packet. The maximum amount of data in the CNP packet is 256 bytes, so if the TCP/IP socket receives more then 256 bytes, the CNP server chops it into 256 byte chunks. It computes the XOR checksum and the data size on the first packet it will send. It sends the 4-byte header followed by data of length specified by that header. A data-type packet is never transmitted with zero bytes of payload data. Therefore, 1 to 255 represents a data length of 1 to 255 bytes and data length of 0 represents 256 bytes.

The CNP library on the C64 receives the packet type byte first. In this case it's pt_data so it knows to expect a data packet and it continues by reading the next three bytes into the incoming packet header. It now knows how much data to expect. The network hardware driver is responding to NMIs and pulling the data in from the hardware and putting it in its own incoming buffer. The network hardware driver has no concept of the structure of the data. When the main event loop loops, the CNP library pulls data out of the network hardware driver's buffer and interprets the data as packets.

In the waiting time between packets, the CNP library is expecting that the very next byte to arrive is the start of the next packet. It reads the first few bytes directly into structures for the current incoming packet. If the packet is the kind that has data, then the packet header has a data size, so it then knows how many of the next bytes constitute this packet's data.

As soon as the port byte has been read in, the CNP library uses it to look up the corresponding socket structure. The socket structure has a page byte to the receive buffer just for this socket. If that byte is zero, then its not currently allocated, the CNP library will allocate a 1-page buffer for it automatically. As the CNP library reads in the packet's data, it computes the XOR checksum and writes the data into the socket's receive buffer.

This clears out the network hardware driver's buffer freeing it up for the immediate arrival of a packet that belongs to a different socket. If the XOR checksum doesn't match, the CNP library tosses the packet and sends a pt_nak packet. If the CNP server gets a NAK packet, it immediately sends the same packet again. However, from the C64's side, it is totally possible that between getting a bad packet for one socket, sending a NAK and getting the same packet sent again, it could receive and process a totally unrelated packet for some other socket.

Assuming that the XOR checksum passes, the data size and checksum are transferred to the socket structure properties, csk_rsiz and csk_rsum, so that the CNP library's own packet header structure is free to receive and process the next packet. The CNP library then calls the socket's status callback, passing the csc_data notification.

The Application doesn't have to act on that immediately. It can wait if it's busy with something else first. The packet data will hang out in its socket's receive buffer, and other packets for other sockets are still free to come in.

When the Application is ready to process the data, it does so somewhat like it reads in data from an IEC storage device. To read data from a file, you must checkin with the logical file number of the channel to read from. Then you can make repeated chrin calls until the status indicates that you've read the last byte, then you clear that channel. From the KERNAL ROM's perspective, the whole process of checkin, repeated chrin's, clear channel, is atomic. For example, you cannot checkin, read a couple of bytes and return to the main event loop, and then have some other process checkin while the first is still checked in. It just doesn't work this way.

Reading data from the socket is an atomic process too. You can read the current size of data waiting on the socket, from the socket's csk_rsiz property first, if that helps you decide what you need to do. When the App is ready to read the data it calls cnpsin on the CNP library, passing a RegPtr to the socket. This is the equivalent of chkin. You then call cnpsget, which is the equivalent of chrin, and the next byte is returned in the accumulator. The carry is returned clear when there is more data to be read, the carry is returned set when the byte you just read is the last byte. Finally, you call cnpsack which is the equivalent of clrchn. It allows another process to cnpsin, and it also acknowledges that this packet has been received. The CNP library sends an ACK packet back to the CNP server so that it can send the next packet on this socket.

One thing to note is that, after calling cnpsin, you have to read every byte from the socket that is available before calling cnpsack. After calling cnpsack the socket's receive buffer is probably going to get overwritten by the next packet.

Eventually, after some number of packets and retrieving all of the data available, if the remote TCP/IP socket has closed the connection on the bridge socket object in the server, then the server sends a close-type packet. When the CNP library receives the close packet, pt_close, it removes the socket pointer from its table, it clears the socket's status flag csf_open, and it sends a csc_clos notification to the App by calling the socket's status change callback.

Sending Socket Data

Let's imagine now that we open a socket and we want to send data on it. In testing the C64 OS networking stack, I wrote a simple service to request moon phase information for any date, past, present or future. You open the socket to the hostname and port of the service and send a date in the format YYYY-MM-DD followed by a carriage return. It sends back some lines of numbers with a carriage return at the end of each line and then closes the socket.

In order to use this service, we have to send data first. So let's see how we do this.

Opening the socket is performed the same way. The opening procedure happens, the CNP server creates the bridge socket object, opens the remote TCP/IP connection, sends the pt_ack packet indicating that the socket is open. We're ready to go. Except, the remote end is not sending any data, so we're not about to receive any data packets.

Just as how reading socket data parallels reading data from a file, there is a parallel between writing data to file and writing data to a socket. To write data to a file you have to call chkout on the logical file number, followed by a series of calls to chrout, and finally clrchn to end that atomic write session. To write data to a socket you call cnpsout in the CNP library, passing a RegPtr to the open socket. At this point, if the socket does not have an allocated transmit buffer, the CNP library automatically allocates one. It also initializes its internal structure for an outgoing packet. The size is initialized to zero, the checksum initialized to zero. You then make repeated calls to cnpsput (the equivalent of chrout when writing to a file) and it writes that byte into the socket's transmit buffer. It calculates the running checksum and increments the count of bytes in the buffer.

The transmit buffer can only take a maximum of 256 bytes. After each call to cnpsput the carry is returned clear if the socket can take more data, or set if the socket is now full. If your app is finished writing all that it has to write, or if the buffer is full, you call cnpsclr to end the write session. The CNP library sets the socket status flag csf_txng to indicate that the socket is transmitting. Then it dumps the whole packet, with type of pt_data and the data size and checksum in the header, to the network hardware driver. The packet is sent asynchronously. While that packet is going out, a packet on the same socket could be coming in, or a packet on a different socket could be coming in.

The CNP server receives the packet and validates the checksum. If the checksum doesn't match, it sends a NAK packet, pt_nak. The CNP library connects the NAK packet to its socket via the port, and how does it interpret the NAK this time? It's not opening, it's not closing, it's transmitting. So it uses the csf_txng flag to interpret the NAK as a bad packet. The last outgoing packet data, as well as its size and checksum, are still being held by the socket structure and its transmit buffer. The CNP library immediately sends the packet again without informing the Application.

When the CNP server receives the data-type packet and the checksum passes, it writes the data out to the remote TCP/IP socket. Then it sends back to the C64 an ACK packet, pt_ack. The CNP library again interprets the ACK in light of the socket's status flags. Its csf_txng flag is set, so the transmitted packet was acknowledged. The CNP library clears the csf_txng flag, and sends a csc_tclr notification to the App via the socket's status callback.

The Application gets the clear to transmit notification and uses that to perform another atomic write session of up to 256 bytes.

In our example of the moon phase service, we only have to send 11 bytes. "YYYY-MM-DD" plus the carriage return to end the line. That can easily be sent in a single packet. Although our Application receives the clear to transmit notification, we don't have any more data to send, so we just wait until the data received, csc_data, notification comes in. Then we go through the process of reading data from a socket and do something with the data, like render it.

Closing a long-lived socket

The moon phase service closes the socket after transmitting the response, so we'll also get a socket closed notification, csc_clos. If the Application doesn't care about the socket closing, it can just ignore the notification.

However, there might be services that allow multiple exchanges without closing automatically. We send a request, it sends a response, the socket stays open. We send another request, it sends another response, the socket stays open. The C64 OS networking stack is happy to leave that socket open forever, until it times out or one end or the other explicitly closes it. Socket-level keep-alives must be handled at the level of the protocol being used on that socket. The CNP packet protocol does not attempt to prevent a bridged TCP/IP socket from timing out.

With the socket left open, our Application could receive data pushed down to us at any time. This could be very useful for a chat client or instant messenger, or turn-based game, etc. But it does lead to some complications that need to be discussed.

First let's handle the easy situation. We know what happens when the remote end closes the socket, but our end can close the socket too. The Application calls closcnps in the CNP library and passes a RegPtr to an open socket. The CNP library sets the csf_clng status flag to indicate that the socket is closing. It then sends a close-type packet, pt_close, for this socket's port to the CNP server. There is no data, no data size, and no checksum. The CNP server replies with an ACK packet. And again, this time, because the status flag indicates the socket is closing, the ACK is interpreted as acknowledgment of the request to close. The CNP server destroys the bridge socket object and closes the remote TCP/IP socket. The CNP library unsets the csf_clng flag and also unsets the csf_open flag. Then it sends a csc_clos notification to the Application via the socket's status callback. And everything is squared away. The Application can respond to that if it needs to.

The complicated part is, what happens when the Application has a socket open but the user Fast App Switches to a different Application, or even more extreme, what happens if the user temporarily drops to BASIC? What happens if a packet arrives, but the Application whither it's destined is frozen in a background App bank of the REU?

There are several situations addressed in different ways.

While you're in an App bank in which nothing (not the Application, nor any Utility open with it) is using networking, then the network library is not loaded into memory. That means the CNP library isn't loaded into memory either, and there is no network hardware driver installed. But, the network hardware device, the modem, is still connected to the CNP server and it could still receive a packet.

If this happened the modem would try to transmit the data to the C64 over RS-232 (for a user port modem), but there is no network hardware driver even installed! The NMI might fire (depending on how CIA 2's interrupt mask is set) but the data would not be received, it would just go into oblivion. This wouldn't be ideal. What actually happens when Fast App Switching is, if there is a network hardware driver installed in the App that's about to be frozen, that driver is told to pause network activity. This is a standard API for network hardware drivers. In practice, the driver applies hardware flow control to the modem. A test procedure has it wait a progressive-minimum necessary time to allow some buffered data to continue to come in before flow stops.

This is very useful for the situation where, you're in an App that has networking enabled. Data is coming and going on open sockets. But you want to drop back to File Manager to create a directory, or make a new file, write some quick notes, or something like that. When you leave the App, the network is paused. While you're in File Manager, which has no networking stack installed, data is being held in the modem. When you return to the App, because this App bank has a network hardware driver, the network is resumed automatically. Packets held in the modem suddenly come in, but the Application is now ready to process them and everything is good.

A similar situation is if you actually drop to BASIC. Now you're no longer in C64 OS at all.

From any App, if you use the Switcher to drop to the READY prompt, it is mostly the same process as switching to a different App. The current App is frozen to the REU, the network is told to pause before the freezing happens, then you get dropped to BASIC. It's the same situation as above, incoming packets are held in the modem until you Fast Reboot. The network is then unpaused and the App is able to process the waiting packets.

A major difference is, from the READY prompt, you could load and run software designed to talk to the modem. There is nothing that C64 OS can do about this. If you launch some terminal software package and it hangs up the modem, then, when you return to C64 OS, it has to deal with that situation as though the CNP server disconnected.

This is one variation of a more general problem. Whenever you're not in C64 OS or not in an Application with the network stack installed, keep alive packets are not being sent. After 10 minutes (by default) of inactivity the CNP server will drop the connection. Any open sockets are closed when the bridge socket objects are destroyed. When you return to C64 OS, or return to an Application with networking, it may need to deal with the sudden closure of sockets.

There is another situation that has to be considered.

You leave one App that has networking installed with open sockets, and switch into another Application that also has the network stack up and running. When you leave the first App the network is paused. But when you go into the next App, the network stack is either booted up, or if the new App was frozen, it thaws and the network is resumed.

The good news is, network activity in the new App, including keep alives packets, will prevent the CNP server connection from timing out. The tricky part is that packets destined for the other App might come in and be received by this App. This is the reason why the port format includes the App bank number.

When a packet arrives, the CNP library looks up what socket it belongs to. The first step is to read the port, clear the low 3 bits, and compare the upper 5 bits to the port mask (which was derived from the App bank number when this instance of the CNP library was initialized.) If the port mask doesn't match, then this packet belongs to a different App. The CNP library can just ignore this packet. It doesn't send an ACK, so the CNP server doesn't just continue to dump data to this socket. After a period of time, the CNP server will attempt to resend the unacknowledged packet.

If you switch back to the original Application, and then an unacknowledged packet is resent, the correct Application will then receive it, acknowledge it, and nothing has been lost.

There are complicated situations that could arise here though, and I admit that some strange edge cases need to be improved. For example, if a packet is partially received when you Fast App Switch to a different Application with networking installed, the network will be paused (hardware flow control asserted) but when it's resumed, the second half of the packet will be received by the networking stack in the other App. In theory, the networking stacks in both Apps should recover, due to timeouts and improperly structured packets. But there will probably be some bugs in this area in the first networking release (v1.07).

Another complicated situation arises when the network is paused, because you've switched to an App that doesn't have the network stack installed, or you've dropped to the READY prompt, and a packet comes in. Yes, it gets held by the modem, but it also doesn't get acknowledged. The CNP server might then, after a brief delay, resend that unacknowledged packet. The modem could now be buffering two packets (or more) back-to-back, that are the same packet. When the App resumes the network it would then receive them both. In this case, however, if a packet was received for a socket that had not yet acknowledged the previous packet, it could be assumed to be a resend. Nonetheless, there are probably some bugs in this area of the code, and this might also lead to an overrun of the network hardware driver's receive buffer, since there is no socket buffer available into which to dump this packet.

It's complicated. But, I'm confident that the overall structure is a huge leap up from just having the modem dump completely unstructured data at the computer. The Commodore Network Protocol already allows two different processes, an Application like Wikipedia and a Utility like MoonPhase, to be running at the same time, and while a Wikipedia article is being downloaded the MoonPhase Utility can make a request to a different server, and get and render the response all while the Wikipedia article is still downloading! It's incredibly impressive to see this happen on our beloved brown 8-bit computer, with just 64K of RAM.

Last but not least, video 7 of 7 about new C64 OS networking tech. #C64 This video shows how the Commodore Network Protocol allows for different processes to send and receive packets interleaved on different sockets. It’s pretty cool! More about this in an upcoming blog post. pic.twitter.com/IiZsV7Fk1b

— Gregorio Naçu (@gregnacu) September 18, 2024

Final Thoughts

Before I started writing C64 OS, I wanted to write network-based Apps for my Commodore 64. I wanted to write a web browser (by proxy server), and a chat client, and an email client, and an FTP client, etc. But when I got back into the C64, and started learning again about how it works and what the KERNAL and BASIC ROMs provide, I was shell-shocked by how anything gets done at all.

Building a robust user interface was such a massive pain. And getting a mouse-driven UI was a whole other level. I thought that it just didn't make sense to spend time working on a web browser, or an email client, if the user interface was going to be either A) abysmal, or B) massively difficult to write. So instead, I wrote C64 OS. But everything I've written for C64 OS has been with an eye towards how to support network-based Applications. C64 OS was not intended to be a games platform, and it wasn't meant to be for desktop publishing. It was meant to be a way to easily snap together object-oriented user interface elements, to build what feels like a modern computer UI, with menus and split panels, and tabs, and buttons, lists, tables with resizable columns, text input fields and areas and a universal clipboard, that you can control with a mouse and keyboard, so that when the network stack gets written and you want to write an email client, you can focus on the business logic of an email client, and not be bogged down trying to build out everything else first.

An example of this original dream coming to life is in this Wikipedia client for C64 OS.

The actual amount of code that is specific to the Wikipedia App in C64 OS is stunningly little. It's around 3KB. And yet, you get a search bar that you can show and hide, you can type a long search term and edit the content in that field, and make selections and use cut/copy/paste in that field. You get a split panel to change how much screen real estate is available for the table of contents and for the article. The table of contents is in a scrollable list with a proportional scroll bar. Lots of options are available from the menu bar with the mouse and keyboard shortcuts. The text of the article does natural wrapping and has clickable links. And then of course, there is all the networking code that undergirds the ability for the App to open a socket to the Wikipedia proxy server, make a request and get a response, and then there is all the memory management to handle where everything gets stored. And on top of that, you can open Utilities and use them at the same time.

There is an absolutely epic amount of code, design, planning, and structure that underlies an Application like Wikipedia for C64 OS that makes it possible. And yet, writing the Wikipedia App itself was surprisingly easy. I'm super excited for all the network-based Applications that are now possible to write for C64 OS. And I hope that you are excited by the possibilities too.

C64 OS v1.07 is just entering the initial beta testing stage as I get some of this into other people's hands. And I've got more drivers to write to support more networking hardware.

1. I'm looking for a simple solution to obscure this somehow. ROT13 would be better than nothing, but maybe there is something just as easy and slightly less obvious.
2. Subject to revision during feedback from beta testing.
3. A point of clarity. Wherever I say a "wifi router" this should technically be a "wifi access point." As, not all wifi access points are internet routers.

Do you like what you see?

You've just read one of my high-quality, long-form, weblog posts, for free! First, thank you for your interest, it makes producing this content feel worthwhile. I love to hear your input and feedback in the forums below. And I do my best to answer every question.

I'm creating C64 OS and documenting my progress along the way, to give something to you and contribute to the Commodore community. Please consider purchasing one of the items I am currently offering or making a small donation, to help me continue to bring you updates, in-depth technical discussions and programming reference. Your generous support is greatly appreciated.

Greg Naçu — C64OS.com

Want to support my work?

C64 OS version 1.04 is now available

The best way to help C64OS.com grow and continue is to purchase a commercial copy of C64 OS version 1.04.

Join the growing C64 OS user community. Take advantage of the powerful and useful features available now, and get access to run new and future software made for C64 OS.

Learn More Order Now

C64OS.com is hosted on a Digital Ocean droplet.

Digital Ocean is a cloud hosting platform, built for developers. They make it simple to get started. They have intuitive administrative tools and reasonable prices. And they can scale as your needs grow.

If you sign up with this referral link, you'll get a $100 credit, and after you've spent $25, I'll get $25. If you are looking for a cloud platform to host your website or web app, this is win-win.

Sign up for Digital Ocean with this refereral link.
Save money and help C64OS.com at the same time.

No, thank you.

Want to support my hard work? Here's how!