Frequent BMOW readers have doubtless noticed the lack of project progress  lately. I could claim I’ve been trapped under a piece of heavy furniture, but the truth is I’ve been procrastinating while trying to set a new course for my professional future. That’s a fancy way of saying I’m busy looking for a job. But I can’t post an update here without at least some topical content, so here are the two editor’s choice blue ribbons that BMOW won at the 2009 Bay Area Maker Faire. O’Reilly Media just sent them yesterday, and posted a complete list of winning projects, even though the event was almost three years ago. Their “to do” backlog must be as long as mine!

Despite the contents of the BMOW web site, I’ve never had a professional job in electronics or computer hardware. My career has been in software development, with most of it working in the video game industry in engineering leadership and management roles. Most recently I was involved with the launch of RIFT, a sprawling fantasy MMO game that took five years to develop. I was one of the first employees, and I led the engineering team that developed all the client, server, tools, and other technologies from the ground up. It was an incredible experience, and the game has raked in money since its launch, but I’ll never again run a project whose development lasts half a decade.

For the past several months I’ve had no job, by choice. This has been hard on my finances, but it’s provided me an opportunity to consider lots of interesting and unconventional ideas for the future.

Make BMOW a career.

My first instinct was to turn the BMOW projects into a self-financing operation, creating a micro-business from the sales of project hardware and advertising revenue from the web site’s content. You’re probably already familiar with many such operations, like Evil Mad Science, Dangerous Prototypes, and Modern Device. I may still give this a shot, but my guess is that BMOW projects are best off remaining as a hobby. I’d hate to be forced to make all my projects “useful” in a commercial sense. I doubt there’s much market for hand-made 8-bit computers, for example, even if designed as an educational kit for nerds. And I don’t like the idea of filling the web site with a bunch of advertising crap.

Find an electronics job.

If turning BMOW into a business isn’t the answer, then maybe an engineering job at a hardware or electronics company is a better solution. The San Francisco Bay Area is a pretty good place to find such companies, after all. I looked into the options in this space, but came away disappointed. Without an electrical engineering degree or any past professional experience working in electronics, I would have to argue my qualifications based solely on my hobby project experience. Some companies might consider that favorably, but most would not. And even if I could land an EE job, a hands-on hardware engineer role would be something of a step backwards on the career ladder for me. Better would be a technical management job at a company making hardware or software/hardware products, but I don’t think anyone would hire me to do that without past domain-specific professional experience.

Continue on the software or gaming path.

The most obvious route is to continue on my current path, and seek an interesting technical leadership role at a game developer or web-based business. In fact this is what I’ll most likely end up doing. I’m very fortunate that there are many good options for me in this area, and I have plenty of personal contacts at local companies, so it’s more a question of finding the best fit than of finding any job at all. My LinkedIn profile says I’m seeking opportunities that combine technical leadership with wider product development and business responsibilities. Translation: a software technology-oriented role that isn’t exclusively about engineering, but more about the whole product. If you’re in the San Francisco Bay Area and have any leads to share about such positions, let me know.

Bootstrap a software business.

Another option I’ve considered is building a niche software product by myself, and turning it into a small business. With low development costs (primarily just the cost of my own time), the business wouldn’t need a tremendous amount of revenue in order to be successful. I have a few ideas in particular involving casual strategy games for kids, and I may pursue one of these even while I continue to examine my other career options. Bootstrapping isn’t my preference, though. I prefer working with teams to working alone, and the quality level I could expect to hit would certainly be lower for a solo project than for something developed by a team of experienced developers. I’m also acutely aware that having a couple of game concepts doesn’t constitute a meaningful business plan.

Launch an investment-backed startup.

The startup company concept is deeply embedded in Silicon Valley’s culture, yet it was only recently that I began seriously considering it myself. Having now been an early employee (non-founder) at three startups, and having lots of friends and colleagues who’ve done it successfully themselves, I’ve slowly realized that successful founders are just smart, motivated people not very different from myself. I have a few friends at venture capital and investment banking firms to whom I could bring ideas, and many more well-placed friends-of-friends I could meet with an introduction. From a practical standpoint, then, getting my foot in the door of the startup dating game wouldn’t be difficult. Investment backing would enable hiring an experienced team to build a high-quality product, and would also bring referrals to potential cofounders with the operations, financial, and marketing skills that I lack. What’s missing is a compelling product idea, and perhaps another cofounder or two with a complementary background to my own. I’ll be working on both of those needs over the next few weeks.

Been there, done that?

Why am I analyzing my professional future here, as if it were an interesting circuit to debug? My reasons are selfish: I’m hoping for your advice. Have you been in a similar situation with your own career? Ever tried to turn a hobby into a vocation? Ever bootstrapped a product, or launched or startup? How did it go, and what did you learn from the experience?

After months of procrastination, I finally assembled the Floppy Emu board and began work on the firmware modifications this week. So far, so good! Despite being out of practice with soldering, the assembly went smoothly, and the board checked out fine electrically. The firmware has now been partially converted to the new microcontroller and pin arrangement, and I’m able to read the SD card and write to the LCD screen without problems.

For those who may have missed the earlier progress updates, Floppy Emu is a floppy disk drive emulator for classic Macintosh computers like the Mac Plus. It plugs into the Mac’s DB-19 port, and behaves exactly like an external Sony 3.5″ disk drive would, so no special system software or other modifications are required. Floppy disk images are stored on a standard SD memory card, and a microcontroller (Atmel ATEMGA1284P) and CPLD (Xilinx XC9572XL) are used to read/write the floppy data. The data is converted into the GCR-encoded serial pulse stream that the Mac expects, exactly like the signal from a magnetic read head flying across a track on a real floppy disk.

The Floppy Emu prototype was constructed on a breadboard, using whatever parts were on hand. The prototype demonstrated 100% successful read emulation of an 800K floppy disk, and partially successful write emulation, depending on the type of write operation and the specific SD memory card used. The new Floppy Emu board shown here uses a more powerful microcontroller and different type of CPLD, and combines everything onto a single custom-made circuit board that fits right into the back of the Mac at the external floppy port. Power is provided by the Mac too, so there’s nothing to do but connect it and go.

These photos show how small the Floppy Emu board is: about 1.75 inches wide and 4.5 inches long, including the DB-19 connector. The SD memory card extends an additional 0.5 inch beyond the end of the board. A roll of Scotch tape is also shown as a size reference. The Floppy Emu board is purple, but in most of the photos you’ll also see an LCD display on a red daughterboard. The LCD daughterboard is socketed, and can be connected and disconnected as needed. It’s the same Nokia 5110 LCD board sold by SparkFun and other several other vendors.

Thanks to its small size, the board fits nicely at the rear of the Mac, right between the mouse and the SCSI connector (or mouse and serial port on older Macs without SCSI).

In addition to plugging straight into the external DB-19 floppy port, Floppy Emu can also be connected using a rectangular 20-pin IDC connector. This is the same connector found on the Mac motherboard, so a standard IDC cable can be used to connect Floppy Emu internally instead of at the external floppy connector. A DB-19 to IDC-20 adapter cable can also be used, such as the Apple II cable from IEC shown below. The cable enables Floppy Emu to connect to the external floppy port at the Mac’s rear, but positioned in the front of the Mac where it’s easier to use.

Everything is looking good so far. The next step is to program the CPLD, so communication with the Mac can be performed. The Floppy Emu board has a Xilinx JTAG connector at the upper-right of the LCD daughterboard, but it’s not populated and I’m hoping not to use it. Instead, my plan is save the CPLD configuration file on the SD memory card, and then use the microcontroller to configure the CPLD using a bit-bang JTAG method described in Xilinx app note XAPP058. Once that’s done, the final step will be to use the more powerful microcontroller on this board (as compared with the prototype) to experiment with new write emulation methods, and hopefully achieve 100% success for emulated floppy disk writes as well as reads.

Last October, Plus Too first booted successfully into the Macintosh Finder. Ever since then, it’s exhibited an intermittent mouse freezing bug. The FPGA Mac replica runs normally for a few minutes, during which the mouse works normally, and it’s possible to exercise menus, run programs, and do everything else you’d expect from a working Mac. But somewhere after a minute or two of activity, the mouse pointer invariably freezes in one spot, and the computer seems to halt. The bug appears to be related to mouse movements, and faster, more frequent mouse movements cause the problem to appear sooner. If the mouse remains unmoved, then Plus Too will happily run for hours without problems.

In October I was already tired from the work needed to get Plus Too to that point, and had no desire to chase the mouse freeze problem further at that time. The project sat idle while I turned my attention to other things, and saw now further progress until this week. That’s when I decided it was finally time to track down the cause of the mouse freeze bug.

Mouse Interrupts

Macintosh mouse handling requires two different interrupts in order to work correctly. When the user moves the mouse, the SCC triggers a level 2 CPU interrupt to read the new position data. The interrupt handler adjusts a low memory global variable called MTemp to set the new on-screen mouse pointer position. Then every 1/60th of a second during the VBLANK interval (video retrace), the VIA triggers a level 1 CPU interrupt. The VBLANK interrupt handler erases the on-screen mouse pointer from its old position, and redraws it at the new position indicated by MTemp.

When the Plus Too mouse froze, I found that the level 2 SCC interrupt was still getting called normally, and MTemp was being adjusted correctly. However, the level 1 cursor VBL task was not getting called, so the mouse pointer was never redrawn at the new position. Further investigation showed that no other VBL tasks were getting called either. In fact, no level 1 VIA interrupts of any type were being processed. At first I thought this might be a problem with the Verilog code that implements my VIA replica, but I found that the VIA was still asserting its IRQ line, but the CPU was just ignoring it. Why?

According to the CPU status register, when a mouse freeze occurs, the CPU is permanently stuck with its current interrupt priority level at 1, instead of its normal value of 0. Because interrupts equal to or below the current IPL will be ignored, no VIA interrupts are ever processed, so the mouse VBL task never gets called. Level 2 SCC interrupts can still pre-empt the CPU, so MTemp gets updated correctly, but when the level 2 interrupt handler completes it returns to whatever the CPU was previously doing at level 1.

Stuck at Interrupt Priority Level 1

So how might the CPU get stuck at IPL 1? How does it get to IPL 1 in the first place? The normal way IPL 1 is reached is during a level 1 VIA interrupt handler, when the CPU sets the IPL automatically. These handlers normally do some processing and then return, which automatically restores the IPL to 0. This means one way the CPU could get stuck at IPL 1 would be if a level 1 interrupt handler went into an infinite loop and never returned. Looking at the level 1 interrupt handlers in the Mac Plus ROM, there are:

• One Second timer – From inspecting the code, this is a trivial handler, and will always return.
• VBLANK – This handler explicitly sets the IPL back to 0, so it can be pre-empted by other level 1 interrupts.
• Timer 1 and keyboard – I haven’t implemented these interrupts in the VIA yet, so they can never occur.
• Timer 2 – This is the only VIA interrupt yet implemented whose ROM handler might conceivably fail to return.
• System handlers – After booting the Mac, the system software might install new VIA interrupt handlers or patch the ones in ROM, creating additional opportunities for handlers that don’t return. Unfortunately I have no good way to test that further.

In addition to a non-returning level 1 interrupt handler, the other way the CPU could get stuck at IPL 1 is if some code explicitly sets the IPL to 1. From looking at a disassembly of the ROM code, several routines definitely do this when modifying global lists: VInstall, PostEvent, OSEventAvail, FlushEvents. The Sony floppy driver also explicitly sets the IPL to 1 in at least two cases. There are also many examples in the ROM code where the IPL is set using a value passed in a register or on the stack, where I can’t say for certain what value it’s being set to. And as before, the system software loaded from disk might contain additional code that directly manipulates the IPL, which I wouldn’t see in the ROM disassembly.

The best way to determine what’s happening would be to wait until the mouse freezes, then pause the CPU when it’s stuck at IPL 1, and look at what code it’s executing. I’ve attempted to do just that, but I lack good tools for software debugging (as opposed to debugging the Verilog hardware model), and I haven’t been able to learn anything very useful. Whenever I interrupt the CPU, it’s either executing some system code in RAM that was loaded from disk, or some fairly innocuous piece of ROM code like the trap dispatcher. I’ve been able to determine any higher level purpose to the code that suggests what it’s trying to do or why it never exits IPL 1.

Finding a Fix

One path might be to add MacsBug to my system disk image, then invoke it when the mouse freeze occurrs, and examine the stack trace and disassembly in an attempt to learn more. MacsBug requires the use of a keyboard, though, and I haven’t yet implemented the keyboard hardware. Even if the keyboard worked, I’m reluctant to start into debugging random pieces of system software that I know nothing about, but maybe that’s unavoidable.

Another possibility is to determine what was the most recent time the IPL was changed from 0 to 1. That might not be enough information to solve the problem, but it would be a start. I might be able to find that info using Altera’s Signal Tap logic analyzer, or maybe I could modify the Verilog machine model to keep track of the IPL changes for me.

My hunch is that some piece of code is going into an infinite loop while trying to access a piece of hardware I haven’t yet implemented, like VIA timer 1, the keyboard, serial port, sound hardware, or PRAM. If all else fails, I could just keep adding more hardware to my Verilog model, and see if the mouse freeze problem disappears at some point. One intriguing clue is that the mouse problem is much more difficult to reproduce when the General control panel is in the foreground. This control panel sets the date and time, sound volume, and other settings that are stored in PRAM. With PRAM not yet implemented, the control panel behaves oddly, and the system time never advances beyond 12:00:00 AM. Perhaps the General control panel is constantly attempting to read or write PRAM, which somehow affects the likelihood of the mouse freeze bug occurring? It’s little more than a wild guess, but PRAM is as good a place as any to start implementing more hardware.

One thing I can’t explain is why frequent rapid mouse movements appear to cause the freeze problem, since my investigations suggest the frozen mouse pointer is merely a symptom of VIA interrupts not getting processed, rather than a cause of anything. Since mouse movements generate a level 2 SCC interrupt, maybe there’s a bug in my design that occurs when a level 2 interrupt pre-empts a level 1 interrupt under certain conditions, or when both interrupts are triggered at the same time. There are some bugs in my mouse implementation as well, which appear to cause a backlog of mouse updates under some situations. I’d assumed these were unrelated to the freezing problem, but maybe I should try getting to the problem of that first. I wish I had a clearer idea of how to proceed, instead of just clutching at straws!

Mark McDougall (tcdev) discovered what looks like a serious bug in the way Plus Too handles interrupts. It appears my design causes the 68000 CPU to use the wrong interrupt handler vector! How it could work at all under those circumstances isn’t clear, since I would expect it to crash the moment an interrupt is first triggered, but I fixed the bug anyway. I had hoped it would eliminate the mysterious freeze-ups I’ve been getting with Plus Too after a few minutes of active mouse movements in the Finder, but sadly it didn’t appear to make any difference.

Vectored Interrupts

Here’s what’s happening. Plus Too (and the Macintosh it replicates) use vectored interrupts. When an interrupt is triggered, the 68000 responds with an interrupt acknowledge cycle. It sets the 24-bit address bus to all 1’s, except for A3-A1, which are set to the level of the interrupt being acknowledged. There is no A0 output from the CPU, since it uses upper/lower byte strobes instead. So to acknowledge a level 1 interrupt (the VIA), the CPU would set the address bus to:

1111 1111 1111 1111 1111 001x

with X being the invisible A0 bit. The memory interface (in this case, my Plus Too design) is supposed to respond by placing an interrupt vector number on the 16-bit data bus. The CPU then multiplies the vector number by 4 internally, in order to get the memory address of the interrupt vector. It then uses that vector to find the location of the interrupt handler routine to execute.

In the case of the Macintosh, the external interrupt handlers begin with vector number $18, which when multiplied by 4 is the vector found at memory address $60. The level 1 VIA interrupt vector is number $19 found at $64, and so on. So to select the proper vector, the memory interface should respond to interrupt acknowledge cycles with $18 + the interrupt level.

A Missing Bit

That’s what I intended to do, but somewhere during development, my Plus Too code lost an address bit. The relevant piece of Verilog code looked something like:

input [1:0] addrLo; // A2-A1
output [15:0] dataOut;

...

assign dataOut = { 13'h3, addrLo }; // use A3-A1 to construct an interrupt number offset from $18

Oops. That code doesn’t do what the comment says. It ignores A3, meaning that interrupt levels 4-7 will never be handled properly. These correspond to the programmer’s debug switch on the Mac. Worse, it generates interrupt numbers that are offset from $C, not $18. So for interrupt level 1 (the VIA), it will generate a response of interrupt number $D, which is at memory address $38.

According to my docs, $38 is an unassigned/reserved vector. In fact, all the vectors from $30 to $5C are reserved or unassigned. So how does that work at all? Why doesn’t it crash the moment a VIA interrupt is first triggered? Is it possible that the reserved vector entry just happens to contain the right value somehow? That seems very unlikely.

Fixed?

The fix is pretty simple: addrLo should be three bits instead of two, and contain A3-A1. I made this change, and Plus Too behaves no differently than before as far as I can tell. It still kind of mostly works, but exhibits frequent freeze-ups after a few minutes of use, that seem to be related to mouse movements somehow. Maybe the two problems are totally unrelated, but I’d hoped the interrupt vector problem might explain the freeze-ups.

I still can’t explain how Plus Too ever worked before, with external interrupt numbers given the wrong offset.

It took longer than I’d expected, but all the parts needed to build three Floppy Emus have finally arrived! That’s three custom-made circuit boards from Dorkbot PDX, plus the SD card sockets, CPLDs, AVRs, and a whole pile of buttons, LEDs, and other components. Now it needs some soldering love.

The boards from Dorkbot are as high-quality as ever: well-centered drills, crisp silkscreen, gold-plated pads, proper via tenting, and everything else. Unfortunately my last few Dorkbot PCB orders have been slower than normal to arrive. Even when I submit my design on the night before the deadline, the total end-to-end time from submission to boards in my mailbox is no faster than from a Chinese PCB maker delivering via China post. It’s about 17 calendar days in either case. With manufacturing done in the USA, one of Dorkbot’s big advantages is supposed to be turn-around time, but in practice it’s more a question of a higher price for a higher quality board than any question of time. I don’t mean to harp on Dorkbot– the guy who runs the PCB order is very nice, and I’m pretty sure he sinks a ton of his time into it while earning almost zero profit. I only wish it could be a little faster.

Now for the bad news: my enthusiasm for putting this board together seems to have evaporated. My hobby efforts come in waves, and I was very busy on this project in November and December, but now I can’t quite get excited about it. It’s not simply a question of soldering on the parts– I also need to modify the AVR firmware to use the ATMEGA1284 instead of the ‘328 from the prototype, implement indirect CPLD programming, and then implement the write buffering mechanism I keep talking about. I’m sure I’ll get to it soon enough, but right now when I look at the bare board, my mind wants to go read a book or go for a hike rather than jump into assembly. I’ve learned to listen to those voices, else a hobby can turn into a chore and cease to be any fun.

Today I ordered the parts needed to build three Floppy Emu boards, the Macintosh floppy disk drive emulator. Everything should be here by next week, so I can start building! In these small quantities, the total cost for the parts is about $47 per board, which is a lot more than I’d hoped. If I ever sell assembled Floppy Emus, they’ll probably need to have a retail price over $100 to cover the cost of assembly, testing, packaging, and a small profit to make it worth the time required.

In addition to being a “real” device instead of a hand-wired breadboard prototype, the new board will also use different parts than the original. The biggest change is the switch to an ATMEGA1284P AVR microcontroller, which has 16K of internal RAM to allow experimentation with different write buffering strategies. The CPLD is different too, with a Xilinx XC9572XL replacing the Altera-powered module that I scavenged from Tiny CPU for prototyping use.

Although the board has a footprint for a Xilinx JTAG header, I didn’t actually order the header, nor do I own a Xilinx JTAG programmer. I’m counting on programming the CPLD indirectly via the microcontroller, using the method described in Xilinx app note 058. If I can’t get that to work, I’ll have to go back and buy the header and a Xilinx programmer, which will mean more delays and more money. I’m keeping my fingers crossed that the indirect programming method turns out OK.