BMOW title
Floppy Emu banner

Yellowstone Firmware 221024, Total Replay and Reset

Firmware version 221024 is now available for the Yellowstone Universal Disk Controller for Apple II. This is the first new firmware since Yellowstone was released, and it fixes two unrelated problems with reset handling and with the NMOS version of the 6502 CPU. The reset handling issue could cause disk I/O to stop working after pressing Control-Reset (not Control-Apple-Reset like is frequently used to reboot an Apple II). The Total Replay collection of games also exposed a subtle problem with Yellowstone on the NMOS 6502, which is used in the unenhanced Apple IIe and the Apple II+, causing Total Replay to freeze during startup on these computers. Firmware 221024 resolves both issues. You can download the latest Yellowstone firmware from the project home page.

Yellowstone is a universal disk controller card for Apple II computers. It supports nearly every type of Apple disk drive ever made, including standard 3.5 inch drives, 5.25 inch drives, smart drives like the Unidisk 3.5 and the BMOW Floppy Emu’s smartport hard disk, and even Macintosh 3.5 inch drives. Yellowstone combines the power of an Apple 3.5 Disk Controller Card, a standard 5.25 inch (Disk II) controller card, the Apple Liron controller, and more, all in a single card. Get yours now from the BMOW Store.

Be the first to comment! 

NMOS 6502 Phantom Reads, Odd Yellowstone Bugs

History repeats itself. I’ve been bitten by an obscure Yellowstone bug related to 6502 phantom reads, and it’s exactly the same problem that I struggled with 13 years ago during the design of my BMOW 1 homebrew wire-wrapped CPU. In both cases, I designed some hardware where CPU reads from a specific address would have side-effects that changed the machine state. And in both cases, I later encountered baffling bugs where the CPU would perform unintentional reads from those special addresses, thanks to the CPU’s implementation details, even though the program instructions never specified a read there. It’s an especially sneaky bug when examining the source code listing reveals nothing, and you need to break the abstraction barrier and look at how the CPU actually implements each instruction.

Back in 2009 the problem was BMOW 1’s audio system, and this time it was the Yellowstone Apple II disk controller‘s Smartport hard disk I/O.

$CFF8: A Special-Purpose Disk Read Register

STA ($4B),Y

With a traditional Apple II disk controller, in order to read the data bitstream from the disk, the program must continuously poll a shift register and check if the MSB is 1. This checking and polling loop eats CPU time, creating an upper bound on the fastest bit rate that a 1 MHz Apple II can process reliably. Yellowstone takes a different approach. When the program reads from the special address $CFF8, the Yellowstone hardware halts the CPU by deasserting the 6502 RDY signal until the shift register is filled. When the CPU resumes and the program reaches the next instruction, it’s guaranteed to have a good value with an MSB of 1, so checking the value isn’t needed. This provides just enough time savings in the inner loop for a 1 MHz Apple II to handle the faster bit rate of 3.5 inch disks, which is twice as fast as 5.25 inch disks. Pretty neat!

If the program code ever accidentally read from $CFF8 when it didn’t intend to fetch disk data, that would create a big problem by halting the CPU and desynchronizing the disk bitstream. But accidental reads should be trivial to avoid – just don’t use $CFF8 for any other purpose. If the code doesn’t reference $CFF8, then $CFF8 won’t be read… right? RIGHT? Wrong. It turns out that was a safe assumption for the 65C02, the second-generation version of the CPU that included various improvements and new instructions, and that’s present in the enhanced Apple IIe and Apple IIc. But for the original NMOS 6502, as found in the Apple II+ and unenhanced Apple IIe, it’s a different story.

Total Replay, Yellowstone, and the Unenhanced IIe

I first became aware of a possible problem after a few Yellowstone owners reported that Total Replay was freezing during startup, before ever reaching the main menu. Putting all the reports together, it emerged that everyone with this problem was running on an unenhanced Apple IIe system. For quite a while I assumed this was probably a Total Replay bug. Maybe it used one of the instructions only available on the 65C02, or relied on some ROM code only present in the enhanced IIe ROM.

Recently with the help of Peter Ferrie, I began digging into the mystery in more detail. Peter shared that Total Replay was running without trouble on unenhanced IIe systems with other disk controllers, so this was apparently a Yellowstone-specific problem. Where could I begin troubleshooting? I don’t own an unenhanced IIe, but I have an enhanced Apple IIe and some ROMs and a spare NMOS 6502, so I could make an unenhnaced IIe. I could even create a Frankenstein hybrid system with a 65C02 but the unenhanced ROMs, or a plain 6502 and the enhanced ROMs. My experiments found that the problem was caused by the NMOS 6502 CPU, not by any difference in the ROM. But why?

STA (indirect),Y

Deep in the Yellowstone firmware for Smartport hard disk reads, you’ll find the instruction STA ($4B),Y. This instruction stores one byte from the disk into a user-supplied buffer, using a 16-bit buffer pointer stored at $4B-$4C plus an 8-bit offset from the Y register. For reasons of program efficiency, and using the Y register for two purposes at once, the code subtracts a fixed value from the $4B-$4C pointer and adds the same value to Y in order to compensate. For example if the buffer begins at $D000, then to store a byte at $D0F1, the code might use a Y value of $F9 and a $4B-$4C value of $CFF8. Queue the spooky foreboding music here.

On the 65C02, STA (indirect),Y requires five clock cycles to execute: two cycles to fetch the opcode and operand, two cycles to read two bytes in the 16-bit pointer, and one cycle to perform the write to the calculated address including the Y offset. All good. But for the NMOS 6502 the same instruction requires six clock cycles. During the extra clock cycle, which occurs after reading the 16-bit pointer but before doing the write, the CPU performs a useless and supposedly-harmless read from the unadjusted pointer address: $CFF8. This is simply a side-effect of how the CPU works, an implementation detail footnote, because this older version of the 6502 can’t read the pointer and apply the Y offset all in a single clock cycle. But it means that when Yellowstone reads a Smartport disk block into address $D000 using an NMOS 6502, it triggers a phantom read from $CFF8 and halts the CPU. That’s what was happening to Total Replay on the unenhanced Apple IIe.

Now What?

Peter suggested it would be possible to modify Total Replay to avoid loading blocks directly to $D000. I hope to save him that effort, and fix the root problem in the Yellowstone firmware instead, because other software may have the same problem when running on the NMOS 6502. I haven’t found another example yet, and standard software like DOS 3.3 and ProDOS appear to work OK. But there are probably other examples lurking out there somewhere.

How can the Yellowstone hardware tell the difference between an intentional read of $CFF8 and a phantom read? It can’t, not really. It just sees that $CFF8 appears on the address bus, the proper address decoding signals are asserted, and that’s the end of the story.

Fortunately Yellowstone is built around an FPGA, which creates the possibility for more complex address decoding behavior than would be possible with just a couple of discrete logic chips. My plan is to save information about what happened on the preceding bus cycles, and when $CFF8 appears on the address bus, use that extra information to help decide whether it was an intentional read. The details may be tricky but I think this approach should work. Meanwhile I’ll have more respect for the unintended consequences of phantom reads, and hopefully avoid making the same mistake again 13 years from now. Check back in 2035.

Read 10 comments and join the conversation 

ADB-USB Wombat Firmware 0.3.8 – Caps Lock Mod

Firmware version 0.3.8 is now available for the Wombat ADB-to-USB input converter. This version fixes a minor problem with the latching Caps Lock key on ADB keyboards in ADB-to-USB translation mode. If you accidentally bumped the Caps Lock key without depressing it fully, you could sometimes end up with Caps Lock enabled even though the key wasn’t latched down. Firmware 0.3.8 resolves this little annoyance. You can download the latest Wombat firmware from the project home page.

Are you new to the Wombat? It’s a bidirectional ADB-to-USB and USB-to-ADB converter for keyboards and mice. With the Wombat you can connect modern USB keyboards and mice to a vintage computer like an ADB-based Macintosh, Apple IIgs, or NeXT. Or you can do the reverse, and hook up vintage ADB keyboards and mice to a modern USB-based computer running Windows, OSX, or Linux. Want to rock an AEK II on your new M2 MacBook Air? The Wombat is your solution. Get yours now from the BMOW Store.

Be the first to comment! 

Extraterritorial Compliance and Small Business

I’m a tiny business selling physical products through a web store to a global audience. I face a pile of business-related regulatory compliance requirements from my local city, county, state, and national government. A recent trend has seen the unwelcome addition of new compliance requirements from states and countries where I don’t live and don’t have any physical business assets – extraterritorial laws and regulations designed to reach beyond the borders of the governments that created them. While most of these regulations are well-meaning, there’s a very large combined burden from regulations from dozens of states and countries around the world where customers exist. For a small business, it means ever more time spent pushing paper, and less time focused on the core business and product development. Ultimately it could force the difficult choice to stop serving customers outside the home area.

Extraterritorial Regulations Background

For many US-based businesses, the European GDPR regulations were their first experience with an extraterritorial law. This EU regulation is mostly responsible for the proliferation of “cookie warnings” we see on virtually every web site today, regardless of whether the site’s owner is located in the EU. In this case, the EU asserts jurisdiction over companies anywhere that may interact with EU citizens on the web. Unless the web site uses geolocation to block visitors from Europe, this means GDPR is effectively a global regulation.

How are these extraterritorial laws enforced? The EU doesn’t have any legal jurisdiction over people outside its borders, and no standard way to enforce its extraterritorial regulations. Enforcement relies on the high amount of international business interconnections in today’s world. Non-EU companies that violate GDPR may have their EU-based shipments and assets seized, or they may be ostracized by forbidding EU companies from doing business with them. Only the most insular companies can afford to ignore such threats.

Just in case it’s not clear, I’m not a huge fan of extraterritorial regulations. They not only make BMOW’s operations more difficult, but they subject people to regulations where they had no representation in the decision making. But I do acknowledge something is needed to craft sensible laws in a cloud-based world where location can be a fuzzy concept.

EU Requirements

From BMOW’s viewpoint, many of these externally-imposed regulations are coming from the EU.

LUCID is an obligatory packaging requirement. As best as I can understand, it exists in many EU countries but the enforcement effort is from Germany. All companies, wherever they’re located, are required to register themselves with the LUCID authorities and declare their annual weight of shipped packages. Then they must purchase a packaging license from a third party. The cost of this license is intended to offset the cost of disposing of or recycling the packaging from shipments, and it depends on the type of packaging used (for example paper versus plastic) as well as the total amount. The goal is to encourage companies to use more environmentally-friendly packaging options, which is laudable. But for businesses outside the EU, the implementation is onerous and it creates an extra hoop that must be jumped through for every international sale.

Language and product support requirements for other countries have also been a challenge and a source of confusion. I once had a shipment rejected by the German customs authority for reasons that weren’t immediately clear. After speaking with the customer, who spoke with the customs inspector, I was informed that I needed to provide German language product documentation and an in-country representative to handle product concerns. The customer didn’t get their package, and I had to apologize and give a refund. I’d never heard of that requirement before, nor has it come up again since then.

Another potential source of externally-imposed regulation is the GDPR. Beyond the actual privacy requirements that were already mentioned, businesses that do not have a physical presence in any EU country may need to have a physical representative located inside the region to comply with the GDPR.

Sales Tax Collection

Tax collection is the most visible area where other states and countries impose requirements on businesses outside their borders. For internet-based sellers in the USA, it used to be common practice to collect sales tax from customers located in the same state as the business, and everyone else got tax-free purchases. If a customer in New York bought something from an internet merchant in California, the customer was supposed to file and pay a New York use tax return, but in reality zero people did this and New York would just lose the tax revenue. In recent years the states have been much more aggressive about pursuing sales tax from out-of-state business, based on confusing requirements about economic nexus. For sales within the USA, businesses may have to collect sales (at varying location-based rates) and submit separate sales tax returns to up to 45 different states, individually.

Most states now apply a sales threshold beyond which out-of-state sellers are required to collect and report sales tax, even if they don’t have a physical presence in the state. Some e-commerce platforms like Shopify will track these thresholds for you. Here’s some of mine:

Unfortunately the automated tracking of a sales threshold is a completely different animal from actually registering with a state’s tax authority and filing a quarterly sales tax return. California’s sales tax return already requires several hours of my time four times per year, and I don’t relish the thought of adding returns in New York and Illinois and 42 other states. I’m undecided about what I’ll do when that New York threshold reaches 100 percent. Cutting off sales to customers in New York isn’t a fun prospect, but neither is adding more compliance paperwork to an already full pile.

VAT Collection

Just as New York and Illinois are eager to get sales tax when their residents buy something from my California business, France and Germany are eager to get VAT when their residents buy something from my USA business. Now it’s called VAT instead of sales tax, but it’s the same concept and it extends the potential tax collection and reporting burden from 45 US states to every nation in the world.

Currently it’s the EU and Canada at the forefront of international VAT collection efforts. And unlike for out-of-state US sales tax, there’s generally no sales threshold and no minimum purchase amount below which small businesses can take shelter. EU countries require that non-EU businesses collect and submit VAT beginning with their very first sale, for any amount of dollars or euros. It’s totally understandable that these countries want to recoup the lost VAT revenue, but it puts a huge amount of extra compliance burden on small businesses. The idea of registering with tax authorities in Belgium, collecting per-customer VAT payments, and submitting periodic European tax returns and reports… for me, as a company with no EU business presence, that’s simply a bridge too far.

Fortunately there’s an alternative that works for now. BMOW can send shipments to the EU and Canada without any VAT, and the authorities in those countries will collect the VAT directly from the customer before delivering the shipment. It sounds great, except it creates some extra delays and costs for customers in those locations. Even when everything goes as expected, it’s a cumbersome process where the package gets stored in a warehouse while the shipper sends a notice to the customer, asking for the payment of VAT plus an additional brokerage fee. The customer may be able to pay online, or may need to physically visit a post office or shipping office. Once the VAT is paid, the package gets released for final delivery to the customer’s door. But if there’s a hiccup in the process, the customer never gets notified that their package is awaiting VAT payment. It languishes in a warehouse for weeks until it’s finally returned to sender by the slowest possible method, arriving back at my address six months or a year later.

So What?

There’s a lot of regulation that impacts small businesses, and much of it from external authorities, but so what? I expect this essay may attract some negative comments and a story on Hacker News titled Small Business Snowflake Thinks Laws Shouldn’t Apply to Him. They’ll say “If you don’t like the laws in (wherever), then don’t serve customers there!” Quite right. And that’s exactly the decision I’m facing now.

I’m realistic. The tax man needs to get paid, and can’t allow buyers to escape paying tax by purchasing everything out-of-state or out-of-country. Governments need to be able to craft product environmental and safety laws that won’t be trivially circumvented by products coming from outside their borders. These are real problems.

But where does this leave a small business with a global reach, a 1.5-person show selling retrocomputer gadgets that were designed for fun? Every day I’m looking at the mounting pile of compliance requirements and paperwork coming from every corner of the globe, and wondering how I’ll ever find time for business and product development amid everything else. My local and state compliance requirements already include the city business license, state reseller permit and sales tax returns, income tax returns, liability insurance, worker’s comp, employment regulations, and more. It’s already a mountain of red tape, even before external jurisdictions begin to impose their own requirements on me.

Ultimately my hope is to pay somebody to handle most of this stuff, if it’s not prohibitively expensive. I don’t spend time stressing about payroll filings, because I use a payroll service. Maybe an accountant to handle the 50 different sales tax and VAT reports? For several of these requirements like LUCID and EU VAT, there already exist cottage industries of in-country agents that will assist foreign businesses. Shopify has some support for these things too, although they tend to be focused on collecting information and stop short of actually fulfilling the compliance obligations. When I researched them in the past, my eyes quickly glazed over with legalese and I failed to grasp the big picture of what exactly I’ll need to do. But with a bit of luck, I hope to reach the summit of this regulatory compliance mountain.

In my dream world, the US would have a single national sales tax system instead of countless separate state, county, and city tax fiefdoms. Then governments wouldn’t worry about losing revenue on out-of-state sales, while businesses would only have to cope with a single collection and reporting system, and everyone would benefit. In this dream world, countries would also work together at the UN or WTO to draft product requirements that applied consistently across the globe, instead of a hodge-podge of different requirements in every different country. Compliance and reporting requirements would also come with intelligent thresholds based on sales volume or business size, to ensure that the compliance burden was appropriate for the business size, and nobody was driven out of business due to overwhelming red tape. I can dream.

Read 6 comments and join the conversation 

Semiconductor Shortage and Business Threat

I’ve written several times about the global semiconductor shortage and the difficulty of getting parts. This has been a problem for everyone since COVID hit, but a year ago I’d hoped things would be improving by fall 2022. Unfortunately it’s worse than ever. It’s now looking increasingly likely that I’ll need to cease most production sometime in 2023, sell off the remaining stock of hardware, shut down the BMOW business, and ride off into the sunset to do something else.

Is it really that bad? In the pre-COVID days, I could just buy thousands of parts from Digikey or Mouser whenever I needed them. More recently I’ve had to place orders for chips 3-4 months in advance, which requires more upfront planning but is still doable. But now several key chips for Floppy Emu and Yellowstone just aren’t available with any realistic lead-time. Suppliers are quoting dates in 2024, or just declining to give any dates at all. Yellowstone has been in this zone for a while, but now Floppy Emu is falling into the same trouble, and without those two products I really don’t have a business.

Yellowstone uses a Lattice FPGA that was widely available when I designed the board, but has since become unobtanium. Officially they’re still quoting delivery dates in late 2023, but the dates keep moving back. I strongly suspect that the dates are based on nothing more than hopes and prayers. A few days ago I received an unexpected email from Mouser, telling me that the FPGA was now back in stock and available to order. But I checked the site, and it still showed zero stock and a year+ lead time. A Mouser rep confirmed this was an error. Ugh.

For the Floppy Emu, microcontrollers from Atmel (now Microchip) have been harder and harder to find. But at least the lead times are somewhat manageable, currently about 4 months, and so far they seem to be hitting those dates. The biggest challenge is the CPLD from Xilinx (now AMD), which isn’t available from any official supplier and where reliable delivery dates are scarce. I had some Xilinx parts on order from Mouser with an expected delivery date of January 2023, then I received notice today that the delivery date has been pushed out to January 2024. Say what?

Some of these parts may be available from third-party suppliers, but I hate dealing with these guys. The whole surplus parts industry seems built on a foundation of deception and lies. Some of them will give an upfront quote if you ask nicely, but many use a retroactive repricing approach that drives me bananas. They might advertise having 37957 parts in stock with a price of $6.02 each, but the fine print says “Due to the shortage of the global supplier, some products may rise in price and out of stock. We will manually confirm after placing the order.” So I place an order and pay for it, then 24 hours I get an email saying sorry we only have 233 parts and the price is $25.44 each. If I decline, then they (eventually) refund me, but it’s a huge waste of time and a terrible way to do business. Even if I can get the parts, I’ll pay through the nose and my retail product prices will surely need to increase.

I’m afraid it’s a gloomy forecast. I’ll keep scrounging for parts as long as I can, and hope that this shortage eases soon.

Read 11 comments and join the conversation 

Yellowstone Future Forecast Update

Back in April I wrote about the future of Yellowstone, and due to the global semiconductor shortage, the future was not looking bright. The Lattice FPGA at the heart of Yellowstone was out of stock everywhere, with an estimated factory lead time on new chips of an incredible 67 weeks. I predicted that BMOW would exhaust the existing stock of Yellowstone hardware sometime in August or September, and that would be the end.

Since then I’ve been watching Lattice’s estimated lead times, hoping that maybe they’d drop to something half-way normal, but unfortunately it’s gotten even worse. It’s been 19 weeks since that April update, so theoretically the 67 week lead time should now be down to 48 weeks. But as of today the lead time on that chip has actually increased to 79 weeks! Ugh.

The best case is a different Lattice FPGA package variant that currently lists a 53 week lead time, but I no longer trust these numbers at all. I strongly suspect that Lattice’s numbers for this chip family are somewhere between wild-ass guesses and complete fiction. They have no idea when or even if they’ll ever have these chips available again. They’re clearly having major difficulty sorting out their manufacturing capacity and meeting all the demand, so an older inexpensive chip that earns little profit for Lattice is probably at the bottom of their priority list. After two years of unavailability and a prediction of at least one year more, I wouldn’t be surprised if this chip were never available again. They’ll continue to push back the estimated lead times for another year or so, before finally admitting reality and giving the chip End of Life status.

This is all just a long-winded way of saying that the future availability of Yellowstone looks even more doubtful now than it did back in April. The only bright spot (for buyers, not for me) is that sales have been slower than expected, so I expect the existing stock will last longer than my original estimate of August-September. At the present rate, the current stock should last until sometime in the spring of 2023. We’ll keep our fingers crossed that Lattice will have sorted out its manufacturing problems by then.

I’m disappointed of course, but that’s life in this industry. There’s no guarantee that parts will continue to be available forever. Usually you get more of a grace period, and a part is eventually designated Not For New Designs, and finally you get a “last-time buy” notification to stock up before the chip is gone for good. The only answer is to factor the likely future availability of parts into your designs, and be prepared to redesign the product around new parts when existing parts reach end-of-life. That would be theoretically possible here, but it would be a big task and I’m not sure I have the appetite for it. Stay tuned for 2023 and beyond.

Read 4 comments and join the conversation 

« Newer PostsOlder Posts »