BMOW title
Floppy Emu banner

Archive for the 'Business' Category

Global Chip Shortage Hits Home

Discouraging news from the BMOW manufacturing front: the global chip shortage seems to be growing worse, and is now causing major problems for some of my vintage computer products. This is a change from a year or two ago, when the shortage mostly affected new high-end semiconductors that BMOW doesn’t use anyway. Since then I’d noticed that prices were rising on lower-end semiconductors, with some parts shortages, but I was mostly able to absorb the costs or find alternative parts. In the past three to six months, the landscape has changed. Many of my essential parts have become difficult or impossible to find. This jeopardizes the continued viability of several BMOW products, including Yellowstone and the Floppy Emu.

When searching for semiconductor parts using a specialized search engine like Octopart, at first it may look like the major suppliers are sold out but secondary suppliers still have stock at reasonable prices. Unfortunately I’ve learned not to trust this, and some of the secondary suppliers seem to be intentionally deceptive. They list parts they don’t have, at prices they’re not actually willing to accept.

Twice in the past week, I’ve placed an order with a secondary supplier for thousands of dollars in hard-to-find chips. The order is made and confirmed and paid. My credit card is charged $4000. Feeling good that I’ve sourced the hard-to-find part, I go ahead and buy all the other parts needed to begin manufacturing. And then three days later I get an email from the supplier saying “those parts are already sold, sorry”. Or worse, “the price on those parts you already bought for $5.17 is retroactively increasing to $20.97.” I try not to take it personally, but I can tell you I really blew up over that second one.

 
Yellowstone’s First Manufacturing Run May Be Its Last

I’ve been working on the design of the Yellowstone universal disk controller for Apple II for more than four years. I’m finally almost to the point of starting manufacturing. At the time I started the design, the Lattice MachXO2-1200 FPGA was a widely-available part for about $5. Now it’s basically impossible to find. A few of the questionable secondary suppliers may have some, for a much higher price, or their listings may be fiction. I could substitute the next higher member of the same FPGA family, the MachXO2-2000, but that’s also unavailable.

Will MachXO2-1200 availability improve any time soon? I’m not counting on it. I have about 250 of them that I bought last summer, so in theory I can manufacture at least that many Yellowstone cards, if I can also find the other necessary components. Even very common parts like a 74LVC244 bus driver are becoming difficult to find, so that’s not guaranteed. Unfortunately this means Yellowstone’s first manufacturing run may also be its last manufacturing run, at least for a while.

 
The Future of the Floppy Emu

The inability to build Yellowstone cards would be very unfortunate, but the inability to build Floppy Emu boards would be disastrous from a business standpoint. But I fear this may be where things are headed. The two primary chips used in the Floppy Emu are the ATMEGA1284 microcontroller and the XC9572XL CPLD, and both of them now have limited or no supply. I’m currently in the midst of starting a new production run of Floppy Emu boards, and it’s been very tough to source all the parts. I was eventually successful (assuming suppliers don’t retroactively cancel or modify my paid orders for a third time), but in six months when I go to do this again, there’s a good chance the parts simply won’t be available anywhere at any price.

 
Forcing a Redesign

Faced with zero availability of a key part, the only real solution is to redesign the product to use a different part. That costs lots of time and money, could introduce new bugs, and is definitely not something I’m eager to do. But if the alternative is retiring the product, do I really have a choice? There could be some upside too, as different parts or newer parts might eventually help support new product features.

If I’m going to redesign an existing product around new parts, it’s absolutely critical that the new parts are free from supply risk themselves. I can’t spend months redesigning a product, only to have the new parts become unavailable just like the old ones did. So if I’m choosing a new part now, I want to see that it’s available from several different authorized suppliers, and that the suppliers have thousands of them currently in stock. That will tell me it’s a popular high-volume part, not some niche part that may have its own supply problems in six months.

Looking for alternatives to the XC9572XL, I searched for other programmable logic parts with a similar amount of logic space (72 macrocells here). I eliminated all the parts in BGA packages, because I’m just not prepared to tackle BGA prototyping or assembly – I need chips with actual exposed pins. Then I eliminated all the parts that don’t have widespread availability and thousands of stock. That didn’t leave many options remaining. The leading choices are probably the XC9536XL, which is the same part I use now except with half the logic space, or else some low-end members of the same MachXO2 FPGA family that I’m using for Yellowstone.

 
Microcontrollers – The New Unobtanium

Next I turned my attention to finding an alternate microcontroller, and this is where my heart really sank. I searched for microcontroller options with:

  • at least 40 I/O pins
  • at least 16K RAM
  • exposed pins
  • a mainstream architecture like ARM or AVR
  • stock level in the thousands

The result of my search was basically nothing. I was floored.

What happened to all the Atmel microcontrollers? The STM32 stuff? Texas Instruments? Microchip? Anybody? Hello?

I tried going directly to the ST Microelectronics store, and viewed their STM32 Mainstream MCU section. There are 1152 different microcontrollers in this section, of which only 16 are in stock! How is that possible? Has their entire factory shut down?

OK, you’re right, it’s a lie to say there are no microcontrollers that satisfy my search. DigiKey shows 932 results when I ignore the stock level. But if I’m going to bet the farm on a new MCU, I want to see a stock level in the thousands, and that limits the choices to just a few dozen, virtually all of which aren’t what I’d consider “mainstream”. It’s 2021, am I really going to redesign my whole product around a dsPIC or some weird thing from Renesas? Does anybody actually use the AVR32 stuff for real products? The ARM Cortex parts dominate the microcontroller space these days, and they’re available from several different manufacturers. If I’m starting a new design, that’s probably the most sensible choice to ensure long-term availability. So let’s make that a requirement.

Despite their huge popularity, ARM Cortex MCUs matching my requirements just aren’t available in large quantities from anyone right now. What are other businesses using for their new microcontroller-based product designs? I don’t know.

If I eliminate microcontrollers from obscure vendors I’ve never heard of, the only parts I can find that might possibly work and are widely available are:

  • Atmel SAM3 and SAM4 families
  • Texas Instruments Tiva C family

That’s the whole list. I don’t really know anything about the Tiva C or the Texas Instruments dev tools, so that leaves the Atmel SAM stuff as the only choice. The available options aren’t the most desirable ones, though. If I were choosing an ATSAM part, I’d probably choose a Cortex-M0, or something like the popular ATSAMD21 or ATSAMD51. But the available options are in the less popular members of the SAM family like the SAM4L and SAM3S. So… yeah. I’m not sure what to do. I’m afraid we’re headed for a difficult time ahead. Here’s hoping the global chip shortage starts to ease soon.

Read 22 comments and join the conversation 

Cyber Monday 10% Off at the New BMOW Store

Help celebrate the opening of the new BMOW store with a Cyber Monday 10% off sale! You get a discount, I get more people trying the new store platform, and everybody wins. From now until the end of Monday, all BMOW products in the new store will be 10% off the regular price. This sale ends at 11:59 PM Pacific time (UTC-8) on Monday night.

You can visit the new BMOW store at shop.bigmessowires.com, and enter the discount code cyberbmow21 at checkout to get 10% off. This isn’t yet the default store, and most internal and external links still point to the original store at www.bigmessowires.com/shop. Purchases from the original store aren’t eligible for this sale – I need as many people as possible to exercise the new store to confirm it’s running smoothly, before I can deactivate the original store. Thank you for trying out the new store, and happy shopping!

Read 2 comments and join the conversation 

BMOW Store Gets a Major Remodeling

Just in time for the holiday shopping season, the BMOW store has been completely rebuilt using an entirely new technology platform! This brings an armload of major improvements, including new payment and shipping options, the restoration of Australia and New Zealand shipping, and overall much improved features and stability. This is something that’s been on my to-do list for years, but I kept putting it off, as there was always something more urgent. My procrastination continued until the old store mysteriously imploded on the night before Thanksgiving, leaving BMOW unable to process any new orders. Ouch.

 
The Store Died, Sorry

Some time on November 23rd or 24th, the old store’s payment processing quietly stopped working. There were no errors or other problem indications, except a mysterious lack of new orders and a large number of abandoned shopping carts. I didn’t notice anything was wrong until late night just before Thanksgiving as I was making pie. I found that customers could begin the checkout process, enter their payment details, and see the final order review page. But when they clicked the button to submit their order, it redirected them to a blank page. No visible error, nothing in the logs, but no payment was made, no order was submitted.

Unfortunately there was no one I could ask for help; I needed to solve the problem myself. I’ve done it before when something unexpectedly broke, sifting through logs, updating plugins, patching PHP files, changing security settings, or increasing server memory limits and timeouts. The old BMOW store is an awkward collection of cheap and free components, including a generic web hosting plan (Dreamhost), content management software (WordPress), e-commerce software (Woocommerce), a variety of different WordPress and Woocommerce plugins, and a payment gateway plugin (PayPal Express Checkout, from the PayPal for Woocommerce plugin). This system frequently breaks at unexpected times, when one of these components gets automatically updated, or a database table grows too large, or an external dependency changes. But I’m responsible for keeping it all working, and that’s not really a job I want to have.

I sunk many hours into troubleshooting this problem unsuccessfully. The logs didn’t reveal any errors or obvious failures. Dreamhost couldn’t help, beyond suggesting that I update everything to the latest versions. This is a double-edged sword in the land of WordPress, because updates are often needed to fix security issues or other problems, but they’re also the most common source of problems. When you’re relying on a dozen different components, and you update one of them, the others might not be happy.

After a frantic late night, I finally managed to fix the problem temporarily on Thanksgiving morning, by eliminating the order review step entirely. But it was now glaringly clear how much risk I was facing by depending on this fragile technology stack with nobody to maintain it but me.

 
Problems With The Old Store

Even if the old store had continued working OK, it just wasn’t great for customers or for me and Lea. On the customer side, it was very slow and sluggish. Submitting an order would sometimes lead to a 500 internal server error, or a payment made but no order recorded, or two orders for a single payment. Something about shipping options didn’t work correctly, so I was forced to limit customers to a single shipping method instead of offering a choice. The order confirmation and shipping confirmation emails sometimes didn’t work.

And then there was PayPal. The old store only offered a single payment processor, PayPal, which some customers simply hate and refuse to use. PayPal used to offer an option for credit card payments without creating a PayPal account if you didn’t want to, but that seems to have disappeared and now a PayPal account is required for making any kind of payment. Another drawback was the lack of payment integration – customers needed to leave the BMOW site in order to enter payment info at the PayPal site, before returning to the BMOW site for final order review.

PayPal also made it strangely difficult to edit the shipping address. If the customer already had a PayPal account, it would auto-populate the shipping address with their saved address, with no option to change it except by cancelling the order in progress and visiting PayPal preferences. Moved recently, or shipping to an alternate address? Too bad. PayPal’s address validation was also weirdly broken. I’ve had countless orders where people omitted required parts of their shipping address like the building number, city name, or state/territory name.

On the back end, searching for and updating customers’ order records was slow and awkward. Some apparently basic features were not included, like embedded the tracking number into the shipment confirmation email. I had to implement that myself with some custom coding. And while this may not be a fair criticism, the old store simply took orders but offered nothing towards managing shipping. I used a separate third-party shipping solution (Shippo), with a custom-made C# tool to gather the Woocommerce data, process it as needed, generate packing slips and invoices, and purchase and print the postage labels from Shippo.

 
Hello Shopify!

Shopify is a very popular solution for running e-commerce stores, and I’d already made up my mind long ago to try Shopify when I was ready to update the store. Shopify replaces the entire technology stack of the old store: Dreamhost, WordPress, Woocommerce, plugins, PayPal, and Shippo. It’s a cloud-based solution, so Shopify actually runs your store on their servers, using DNS magic to make it appear under your domain name as part of your regular web site.

Unfortunately Shopify isn’t exactly cheap, at $79/month for the plan I would probably need. But after all the pains experienced with the old store, I decided the money was well worth it. And now that I’ve dug into Shopify’s details further, I think the net cost may be much less than $79/month, thanks to lower fees for credit card processing and shipping than I was paying before. In fact, I may actually see a net savings from switching. Wow!

I was able to create and configure the new Shopify store in one very long day. So far I’m extremely pleased and impressed with what I’ve seen. Everywhere I look, from the theme designs to the order processing and back-end reporting, I find great new features that are a huge step up from what the old store provided. I should have made this switch long ago.

 
What’s New

The new store looks completely different from the old one, but the customer-facing changes are more than just a facelift. Here’s a tour of what’s new:

New Payment Options – The store uses Shopify Payments, which seems to be a rebranded Stripe. It allows for a dozen different payment methods, including credit cards and others like Apple Pay. It’s all directly integrated into the store, with no separate account needed. Basically it just works. GOODBYE PAYPAL! I will not miss you at all. But PayPal is still an option for anyone who wants it.

New USA Shipping Methods – For customers in the USA, instead of one-size-fits-all shipping, you now have a choice of Standard (3-4 day) or Economy (5-8) day shipping. For the moment these correspond to USPS Priority and USPS First Class Mail, but that may change in the future. If you want to save a few bucks on shipping and are willing to wait slightly longer, now you have that choice.

Free USA Shipping – Orders over $150 to destinations in the USA will now ship free. Hooray!

New International Shipping Methods – Worldwide DHL and UPS shipping is here! This is a big change, and I’m excited and slightly nervous. These new shipping methods aren’t cheap, at around $50 for most destination countries, but they should be faster and much more reliable. Higher tiers of USPS international service are now available too, including Priority Mail International and Priority Mail Express International. USPS First Class Package International Service remains an option too, and is the lowest cost international shipping method. FCPIS is sometimes great, and you can typically get a package delivered in a week to destinations like Canada, France, or UK. But sometimes FCPIS experiences wild delays with months passing in shipping purgatory without any updates to the tracking info. This is very frustrating for both customers and for me.

I hesitate to call any of these new international shipping methods “express shipping”, because there’s still a few days of order processing time before BMOW can fulfill your order, no matter how fast the actual shipping may be. Our fabulous office specialist Lea normally works three days per week, so the processing time is 1-2 business days in most situations.

The shipping fee for international orders is also now calculated based on the actual carrier cost, plus a small handling fee for the extra work that these orders involve. This is an improvement over the old pseudo-flat-rate pricing scheme for international shipping, which lumped everything into a few broad categories based on weight tiers and geographic region.

Australia and New Zealand Shipping – US postal service FCPIS shipments to Australia and New Zealand have been suspended for the past two months, due to some unspecified “carrier disruptions” related to COVID-19 and global supply chain problems. The introduction of new shipping methods means that BMOW can once again serve customers in those countries. Woo-hoo! Australia Post’s ShopMate USA service also remains an option for customers looking for the cheapest Australian shipping, although the service will reportedly be phased out next year. We hope that FCPIS service to these countries will become available again soon, but until then, these are good alternatives.

Better Tracking – Improved package tracking and notifications are one of the many other improvements made possible by the new store. For international shipments, DHL and UPS will provide much more timely and detailed tracking information than FCPIS. And for customers everywhere, you’ll get email notifications not just when your package is shipped, but also when it’s out for delivery. There’s even an option to get notifications on your phone by text message.

 
Tour The New Store Today!

The new BMOW store is live and taking orders now at shop.bigmessowires.com. Give it a try! For the moment, both the old store and new store are running and accepting orders, so I can monitor what’s happening at the new store and ensure everything goes smoothly. All the store links are still pointing to the old store, so you’ll need to enter the URL directly (or click the link in this paragraph) to visit the new store. Eventually I’ll retire the old store and set up redirection links for each of the product pages to the new store. If you try the new store, please let me know how the experience goes. What are your thoughts on the visual design and page layout? Any issues with order processing or payments? Thanks for your feedback!

Read 6 comments and join the conversation 

Tech Support Dilemmas

When I first began selling hardware to other vintage computer collectors, I never gave tech support much thought. I imagined I could design something, build it, sell it to somebody, and they’d happily go use it. End of story. Reality has proven different. Today I spend more time answering tech support questions from customers and potential customers than on any other aspect of the business. I get large amounts of mail, often with questions that are long and complex and need considerable time to answer properly. It’s a challenge for which I’ve yet to find any good solution.

A portion of the tech support is really sales support – questions about ordering and shipping. If you’ve been through the BMOW store in the past six months, you may have seen there’s a new person helping with these kinds of inquiries, which has been a huge help. But even after these are filtered out, the tech support load remains high.

 
RTFM?

Can I solve the tech support problem with more documentation? I’ve put a lot of time into the BMOW product documentation, particularly for the Floppy Emu. Every customer receives a printed one-page quickstart guide covering the basics, with a link to the full instruction manual on the web. Whenever the same question gets asked two or three times, I add the answer to the instructions or the quickstart. At least in theory, everything anyone would ever want to know should be covered in the documentation.

Yet I can’t escape the basic fact that there’s a lot of inherent complexity with these old computer systems and the products designed for them. In the case of the Floppy Emu, there’s a ton of ground to cover in the instructions between three different computer families, a dozen different drive emulation modes, different types of disk images, different operating systems, different ways of connecting to the computer, interactions with other drives, and much more.

The wall of documentation can be daunting. Even with the one-page quickstart that’s bundled in the box, it can be too much for some people. They throw up their hands and reach out to tech support (me) for assistance. In many cases I’m able to gently point people back to the relevant section of the manual: “It sounds like you may need to select a different disk emulation mode. Please see section 3.2 of the instruction manual for details.” But sometimes people get upset or offended if I refer them to the manual. I had one angry customer send me a video message in which he outright refused to read the instructions, insisting the he didn’t want to learn Computer Nerd 101 or get a history lesson in Apple computer models.

 
Whose Problem Is It?

A second challenge is simply identifying who or what is responsible when something doesn’t work as expected. The line between questions about BMOW hardware and questions about general usage of the Apple II or Macintosh is often blurry. For many customers, they’ve hauled an old computer out of their dusty attic, purchased a bit of BMOW hardware to run with it, and are using the machine for the first time in thirty years. If a questions arises, it will often come to me first, regardless of the source or the issue. I’ll get general questions about the Apple IIc, or about Macintosh System 7, or StuffIt. Some requests ask how to use particular software programs, or how to eject a disk, or reboot the computer. Much of this is now covered in the Floppy Emu instruction manual, since they’re common questions even if they’re not features of BMOW hardware.

What’s more puzzling are the questions about unrelated third-party products. If someone contacted me once with a BMOW tech support question and found my answer helpful, sometimes they’ll contact me again later even when their question has nothing to do with BMOW. I get a surprising number of tech support requests about the SCSI2SD. I’ve also had support requests asking about RAM upgrades, joysticks, and other peripherals. Sometimes I know the answer, sometimes I don’t. Sometimes I’ll decline to answer even if I know it, which makes me feel like a jerk, but I need to politely discourage this kind of tech support usage. Usually I’ll point people to another suggested information source instead.

This challenge isn’t limited to the Floppy Emu. The original Mac ROM-inator kit for the Mac 128K, Mac 512K, and Mac Plus was also a big source of general questions about classic Macintosh usage and problems. As a low-cost kit generating a high level of tech support, it just didn’t make sense, and this was one of the primary reasons the original ROM-inator kit was eventually discontinued.

 
Solutions?

I’m unsure how other small businesses solve this problem. Some of them may sell products that are so simple, they don’t need much tech support. Or their tech support is just weak, and they willingly suffer the reputational hit, choosing it as the least bad option compared to hours and hours of support time.

Documentation can always be improved, so that’s another area I can work on. I’m discovering that technical writing is a critical skill, and it’s not enough to simply have the necessary information in there somewhere – it needs to be presented clearly, and in a logical order. And even if the documentation is perfect, some people simply don’t want to read it.

Many companies now have support forums where customers answer questions from other customers, and the business itself is mostly or entirely absent from the conversation. This is certainly one way of reducing a company’s tech support requirements, but from my personal experience the customer experience is almost universally poor. Discussions often devolve into angry denouncements of the business, like “why isn’t (company) saying anything about this???” At worst, it can create the impression that customers have been cast out into the wilderness to fend for themselves. That’s not good.

I’ll keep searching for better solutions. Until then, I’m off to visit the tech support queue…

Read 4 comments and join the conversation 

New Zealand Shipping Suspension

The world of international shipping continues to grow more difficult. As of October 1, the US Postal Service has halted acceptance of international package shipments to New Zeleand, due to COVID-19-related service disruptions at the destination. The same suspension was applied to Australia on September 3, so BMOW is now unable to make direct shipments to customers in either country. I had hoped this would be only a brief disruption while the countries’ post offices worked on mitigating COVID-19 impacts. But with Australia’s suspension already stretching to more than a month, my hopes for a quick resolution are dimming.

One bit of good news is that Australia Post’s ShopMate service is still working. It’s a package-forwarding service for Australian residents who are buying products from vendors in the USA. ShopMate provides you with a USA address, so BMOW and other vendors in the USA can make a regular domestic shipment and you pay them the domestic shipping rate. Then ShopMate repackages the shipment and dispatches it to your home in Australia, in return for an additional fee. A few BMOW customers have already tried buying through ShopMate during the past month, and reported that it worked. I’m not aware of a similar forwarding service for New Zealand, but if I find one, I’ll report it here.

Read 4 comments and join the conversation 

Temporary Suspension of Australia Shipping

With apologies to BMOW readers Down Under, international deliveries to Australia from the BMOW Store have been temporarily suspended. A few weeks ago, Australia Post paused its processing of inbound international mail shipments due to COVID-19-related service impacts, and on September 3 the US Postal Service halted acceptance of Australia-bound international mail. I had hoped this would be only a brief interruption, but after sixteen days there have been no further updates, and there’s no current timeline for when normal operations will resume.

At the moment there are 21 international destinations for which USPS has halted mail delivery “due to impacts related to the COVID-19 pandemic and other unrelated service disruptions.” Australia is the only large country on that list, and the only one where BMOW has customers. The other countries are Afghanistan, Bhutan, Brunei, Cuba, French Guiana, Guadalupe, Laos, Libya, Martinique, Mayotte, Mongolia, Reunion, Saint Pierre and Miquelon, Samoa, South Sudan, Syria, Tajikistan, Timor-Leste, Turkmenistan, and Yemen.

International shipping is hot mess across much of the world right now, and this is just the latest incident. Thank you for your patience through all of this, and I hope to see Australia deliveries resuming soon.

Read 2 comments and join the conversation 

« Newer PostsOlder Posts »