BMOW title
Floppy Emu banner

Archive for the 'Business' Category

Falling into an Email Blacklist with DreamHost

naught-list

A blacklist can be a powerful tool for identifying spam email senders, but if you find yourself unfairly blacklisted, it’s maddening. Since sometime last September, roughly 30% of all my outbound customer-related emails have been rejected by the destination email server. Most of these are order confirmations or shipment notifications, and when they go missing, I get lots of frustrated inquiries from customers wondering why they never heard anything after placing an order. The rejections from the destination email server typically look like this:

<xxxxxx@provisoire.fr>: host mail.provisoire.fr[50.87.141.14] said:
550-“JunkMail rejected – pdx1-shared-relay1.dreamhost.com
[66.33.200.130]:40663 550-is in an RBL on rbl.unified-contact.com, see
Blocked – see 550 http://psbl.surriel.com/listing?ip=66.33.200.130″ (in
reply to RCPT TO command)
Reporting-MTA: dns; pdx1-shared-relay1.dreamhost.com
X-Postfix-Queue-ID: D4C0A30000327
X-Postfix-Sender: rfc822; steve@bigmessowires.com
Arrival-Date: Mon, 9 Jan 2017 14:07:03 -0800 (PST)

The exact message varies, but it usually mentions being on a realtime blacklist, or simply says my email was suspended, blocked, or refused. Other mail hosts such as Yahoo and Outlook.com take a passive-aggressive approach, and just drop the connection when I try to send email to one of their customers:

<xxxxxxxxx@yahoo.com>: delivery temporarily suspended: lost connection with
mta6.am0.yahoodns.net[66.196.118.34] while sending RCPT TO
Reporting-MTA: dns; pdx1-shared-relay2.dreamhost.com
X-Postfix-Queue-ID: B795D38088EC2
X-Postfix-Sender: rfc822; steve@bigmessowires.com
Arrival-Date: Wed, 4 Jan 2017 15:51:07 -0800 (PST)

I haven’t tested it thoroughly enough to be certain, but I believe the problem only occurs for auto-generated emails from the BMOW store, and not for customer support emails that I compose manually – even though both are sent through mail.bigmessowires.com to the same destination email server.

 
Identifying a Spammer

So how did I get on these blacklists? It turns out it has nothing to do with the content of my own emails, but is entirely due to my web and email hosting provider, DreamHost. They offer cheap and convenient hosting, which doubtless attracts a few people using their servers for evil purposes, sending spam. This causes the DreamHost email relay server to be placed on multiple blacklists, affecting all the other DreamHost customers who share that relay. While I only started to notice the problem last fall, this forum discussion reveals it’s been happening since at least 2013.

I’ve contacted DreamHost customer support several times about this issue. At first, they said the problem was resolved, and they had confirmed with all major blacklist providers that the block on the affected relay had been removed. And the situation did seem to improve temporarily, though it was never completely resolved. When the blocks grew more frequent again, I contacted DreamHost a second time on December 8 and received this reply:

The IP that’s showing up as blocked is actually a load balancer used for
sending mail, and it is used by hundreds of individual users. …
Over the last week, we have experienced a surge of compromised customer
SMTP users that were being used to send out malicious emails. Although we
monitor outgoing mail traffic closely and were able to stop these
compromised domains quickly, enough email managed to get through to cause
several blocklist providers to block a percentage of our email servers.
Many providers have already delisted the IP, but some holdouts do remain,
with whom we are actively working to fully resolve the block. If these
rejection notices continue for more than about 48 hours, please don’t
hesitate to let us know.

Sorry, we’re working on it, everything will be back to normal soon. But unfortunately it didn’t go back to normal, and a few weeks later I contacted them a third time. I received a detailed technical reply that focused primarily on a specific provider named 1&1. Apparently 1&1 doesn’t like the way DreamHost mail servers identify themselves when communicating – an issue related to reverse lookups involving a load balancer – so the DreamHost servers get blacklisted regardless of the content of the email. It wasn’t clear if a solution to this identification problem was imminent, or even possible. Customer support also mentioned that it can take up to a month to be removed from a blacklist:

some blacklist providers (Mostly European providers such as UCEProtect,
Backscatter, and LashBack), provide a paid “express” delisting, while
imposing an unreasonable long wait for manual or automated delisting (In
the case of LashBack, they autodelist after a month). As this amounts to
extortion, it is Dreamhost policy not to utilize paid delisting services
(they provide no added benefit to customers, encourage “bad behavior”,
and are generally a sign of an overzealous mail system administrator).

It seems unlikely that 1&1 is the only remaining problem, since my emails to domains like Yahoo and Outlook.com are also being rejected. As far as I’m aware, these are unaffiliated with 1&1.

 
Getting Past the Block

Monkey-Fix-it-300x285

DreamHost’s responses have all been apologetic, giving the impression that service should be back to normal soon. Maybe I should just be patient and wait, but it’s been three more weeks since that last customer support response, and the situation hasn’t improved. The 2013 forum discussion complaining of this same problem proves it’s not a one-time occurrence. And I received no reply to my most recent CS inquiry asking for a status update or work-around suggestions.

Maybe I should move bigmessowires.com to a Virtual Private Server with a unique IP, instead of relying on shared hosting. I’d consider that if I were confident it would fix the problem, but that’s exactly what the 2013 forum poster tried and complained didn’t work. It’s unclear to me whether that was his fault or DreamHost’s. Even if I knew it would solve the email problem, I’m a little reluctant to jump to a VPS due to the extra server admin hassles it would involve. I really like the convenience of shared hosting, where I focus entirely on the content and leave the server administration to someone else.

Perhaps it’s time to migrate the whole site to another hosting provider, but I don’t think so. I expect most other shared hosting providers will have similar issues, and possibly worse service. During the 13 years I’ve been with DreamHost, their customer support has been excellent. This email blacklist problem is the first time I’ve felt let down by their service.

The best option I’ve come up with is to move BMOW’s email functions to a more “trusted” provider, while leaving the web site and store with DreamHost. That would mean monkeying with DNS entries to relocate mail.bigmessowires.com and a few others, or else simply using a different domain like bmowmail.com for all email. Zoho looks like it might fit my needs, and it would be free for my level of usage. I need to dig into the technical details to confirm it would do what I think it does, and would actually solve the blacklist problem.

If you’ve ever dealt with an email blacklist dilemma, or have any other suggestions on how I might resolve this one, please leave your feedback in the comments. Thanks!

Read 10 comments and join the conversation 

Lower International Shipping Costs!

international-shipping

Good news for BMOW fans from outside the United States: international shipping costs for most orders should now be substantially lower than before. I wrote about the pain of international shipping costs a few weeks ago, and ever since then it’s been on my mind. Since about half of all BMOW customers are outside the US, I want to do everything I can to make their shopping easy and inexpensive, and I’m glad I’ve finally been able to address the shipping issue.

So how did I do it? The weight-based shipping rates haven’t changed, and those are set by the US Postal Service. Since I can’t lower the postage cost for a given weight, I instead focused on reducing the weight for a given item by using ultra-light packaging material whenever possible. Instead of shipping international orders in bulky and heavy cardboard boxes, many orders will now ship in padded mailing envelopes with a triple-layer of bubble wrap inside to protect the contents. I’ve tested this to several different destination countries, and it’s proven to protect the contents just as well as a box. Lower weight means lower shipping costs, so everybody wins.

International shipping costs for typical orders will be 40% cheaper thanks to this change. Most orders will now fall under the critical 0.5 pound threshold where higher postage rates take effect. Overseas customers will see typical shipping costs reduced from $24.25 to $15.25, and Canadian customers will see a reduction from $17.00 to $11.00. Hooray!

Read 2 comments and join the conversation 

Crikey! USPS International Shipping Costs

international-shipping

Running a mail-order business gives me many opportunities to think about the cost of shipping. I’ve just made a change to the BMOW store that enables small, lightweight packages sent within the USA to be shipped by USPS First Class Mail rather than Priority Mail, cutting the shipping cost in half. Woohoo! But that’s about all the help I’m going to get from the US Postal Service, and I’m slowly realizing that USPS has some of the least competitive postage rates in the world.

A typical BMOW package weighs about 10 ounces, or 283 grams. To mail that package to the UK, Australia, Germany, or other countries where customers may live costs me $22.75 in postage via “First-Class Package International Service”, which is the cheapest option available. That’s a significant amount of money. If my store sells a 10 ounce item that costs about $20, it’s unlikely anyone outside the USA will be buying it, unless they relish paying more for postage than for the item itself.

I decided to compare the cost of international shipping for a comparable 10 ounce / 283 gram package, sent from a few other major countries:

County Cost for shipping 283g
International Package
Cost in US Dollars
USA USD $22.75 $22.75
Canada CAD $19.39 $14.90
UK £14.90 $21.20
Australia AUD $14.10 $10.83
Germany     €3.70 $4.22

The US has the most expensive international postage rates of any of these countries. (But at least this gave me a chance to learn how to type £ and € on a US keyboard, where those symbols don’t appear.)

The real anomaly in international shipping is China. Sending a package from the USA to China costs me the same as to any other country. But sending a package from China to the USA appears to be nearly free. Take one look at any of the many “free shipping” deals available from Chinese sellers on Aliexpress, eBay, or similar sites, and you’ll wonder if reality has been suspended. And it’s not that they’re absorbing the cost of shipping into the item price. Take this LED voltage meter for example: it costs $0.78, is available in single quantities, and ships free to the USA. Granted the delivery time stinks (up to 40 days), but at that price few people are complaining. How are postage prices like that possible? I can’t even ship within the USA for that price, let alone internationally.

I’m always looking for the most economical way to get BMOW packages to customers. I only wish USPS could be more help!

Read 16 comments and join the conversation 

New Project Ideas

thinking

It’s high time for a new project here at BMOW. These days I spend so much time on Floppy Emu, it’s hard to squeeze in anything else! A new project would help stir things up, and get the creative juices flowing. Here are some ideas I’ve been kicking around:

Electric Scribbling Machine – I described this one in yesterday’s post. It’s a tripod with colored pens for legs, and a motor that makes it move, drawing tracks on the paper beneath it as it goes. Yesterday’s post described a “vibrabot” design using an offset weight on the motor shaft, which randomly jiggles the machine and creates interesting erratic drawings. I’ve since been working on an alternate design in which the motor is used directly as one of the legs, which spins the whole machine in rapid circles, and creates elegant looping drawings somewhat reminiscent of a Spirograph.

USB Keyboard/Mouse Adapter for Vintage Macintosh – Classic Macintosh computers like the Mac SE and Mac II series used input devices based on ADB – the Apple Desktop Bus. The Mac Plus and Mac 128K/512K were more primitive, and used custom protocols for keyboard and mouse rather than any bus-based system. Both systems are fairly well documented, and I have some experience with them already. Using a modern microcontroller, it shouldn’t be too difficult to build an adapter that functions as a USB host for a modern keyboard and mouse, and translates the input data to ADB or the Mac Plus protocol.

Weather Logging Station – A few years ago I designed the Backwoods Logger, an ultra-tiny portable weather station. I envisioned it mainly as a graphing altimeter for people going on mountain hikes. After many discussions with interested people, it became clear that most people didn’t care about portability, or about having graphing functions or even a screen. What they wanted was a stationary module that could take regular temperature, pressure, and maybe humidity readings, and save a history of weeks or months of data. There are already a few designs like this, but maybe they’re too intimidating or their feature set isn’t quite right, because I still get occasional emails from people asking for something like this.

Nibbler KitNibbler is a 4-bit CPU that I designed entirely using basic 7400-series logic elements. It was a one-off project for my own entertainment. William Buchholz later designed a nice Nibbler PCB for his hacker group, which got me thinking that something more polished would be nice. My web stats say the Nibbler pages are some of the most popular content on this site, so maybe there would be enough interest to justify a Nibbler kit? My only fear in offering a kit for something this complex is the potential support headaches. It might make more sense to offer finished Nibblers instead of kits, though that would take some of the fun out of it.

Electric Bow Tie 2.0 – As with the Backwoods Logger, I think I misjudged popular interest when I designed the Electric Bow Tie Kit. It’s fun to have neckwear that blinks and beeps, but mostly people don’t seem interested in assembling a kit – they just want something kitschy they can wear to a special event and get some laughs. I can say from personal experience that the blinking effect is fun, but the beeping effect starts to grate on your sanity after about 60 seconds. Electric Bow Tie 2.0 would probably drop the sound effects, but add many more LEDs controlled by a microcontroller, enabling all kinds of entertaining and annoying patterns like chase lights and starbursts. I’d also try to replace the 9V battery with one or two smaller CR2032 batteries to reduce weight.

Read 17 comments and join the conversation 

A New BMOW Storefront!

bmow-store

At last, BMOW is moving into the modern world with a new online storefront! As I mentioned in the Profit and Loss discussion a couple of weeks ago, the old solution of PayPal payment buttons and order tracking left much to be desired. The new storefront isn’t anything fancy, but it’s still a tremendous leap forward in terms of ease of use for me, and is also far better looking for customers.

You can browse the new store using the “BMOW Store” link at the upper right, and it’s fully functional and ready for purchases. For the time being, the “add to cart” buttons on project pages like Floppy Emu’s will continue to point to the old PayPal payment buttons. This provides an easy back-up solution in case anything goes wrong with the new system. Visitors using the BMOW Store link will make purchases through the new system, while those using purchase links on the BMOW project pages will continue to use the old system for now.

 
WooCommerce and Friends

After looking at many different e-commerce and shopping cart solutions, and installing and uninstalling Zen Cart, I finally settled on using WooCommerce. WooCommerce is a popular e-commerce platform that’s implemented as a WordPress plugin. The basic package is free, and the developers earn their income by selling a variety of add-ons and themes. As I discovered, it’s quite easy to set up a functional and attractive storefront in just a few hours. In order to minimize the risk of conflicts with my existing, heavy-customized WordPress install, I created an entirely new WordPress installation in a subdirectory of the BMOW site, used exclusively for the WooCommerce store.

I’m still using PayPal for payment processing, but the method is now PayPal Express instead of the clumsy PayPal payment buttons. The shopping cart functionality is implemented entirely on the BMOW site, within WooCommerce. When a shopper is ready to checkout, he’ll be redirected to a PayPal page, where he can enter a credit card number or his PayPal account credentials to authorize the purchase amount. He’ll then be redirected back to the BMOW site for a final review, at which point he may cancel or submit the order. Using this method, I avoid needing to ever directly handle sensitive payment info like credit card numbers. But customers still complete their transaction experiences on my site, where I can show them a thank you page, and maintain purchase statistics.

A few additional WooCommerce-related plugins proved handy:

  • PayPal for WooCommerce – Enables use of PayPal Express functionality, and tighter integration with PayPal
  • WooCommerce Weight Based Shipping – Charges different shipping amounts depending on the calculated total weight of items, destination country, and shipping method
  • Storefront Site Logo – The default theme inexplicably lacked a way to add a logo to the store’s header

All of these plugins were free.

 
Cleanup

The only major concern I have isn’t a technical problem, but a design issue. With the introduction of a storefront, all of my creations now have product pages, in addition to the project pages that already existed (and that formerly doubled as product pages). This will be a source of confusion. For example, which page should be considered “the” ROM-inator page: the pre-existing ROM-inator project page or the new ROM-inator product page? Which one should a reviewer link to, or a potential buyer look to for details? Which do I want to appear first in the Google search results for “Mac ROM-inator”?

This may seem like a minor issue, but I’ve spent significant amounts of time thinking about it. I definitely want to avoid a situation where the mind-share for each of my inventions is semi-randomly split between two different web pages, with duplicated information. I looked for examples of how other sites addressed this problem, but couldn’t find any similar examples. My current approach is to keep the project page as the “official” page, with the product page having a much shorter description focused on actual purchase details, with several links back to the project page for further reading. This hopefully makes it clear which page is the main one, but at the expense of removing information from the very place in the store that would-be buyers need it in order to make a buying decision. So it may actually cost me sales. I’ll evaluate how this works for a while, and consider making the product page be the main one if necessary.

Happy shopping!

Read 3 comments and join the conversation 

WordPress, https, and Canonical URLs

padlock

About a week ago, I added an SSL certificate to the BMOW web site, in preparation for some improved shopping cart features. With an SSL certificate signed by a certificate authority (free from Let’s Encrypt), the site can serve pages using the encrypted https protocol as well as the standard non-encrypted http. Pages encrypted with https will show a padlock icon or something similar in the address bar of most web browsers, and are normally used for handling sensitive content like payment info for a web store. My plan was to continue serving the existing blog pages using http, and use https for the new shopping cart pages. But it’s technically possible to serve any page from the site using https – try it! Just manually edit the URL of this post in your address bar, and change http to https.

The main blog pages aren’t designed to be served with https, however, and they contain embedded non-secure content like images and comment forms that use the http protocol. If you view this post as https, it will work, but your browser will probably display a warning about insecure content. If you try to post a comment, you’ll see a warning about a non-secure form, and if you persist in posting the comment you’ll see an error 403: forbidden message.

Since nobody ever visits the BMOW site using https, I thought those security warning didn’t matter, until I discovered that Google has started replacing all of the BMOW links in its search results with https versions of those same links. Someone who searches Google for “KiCad vs Eagle” might see a result with an https link to my post on that topic. Following the link, they’ll get a bunch of security warnings from their browser. And after commenting on the post, they’ll get the dreaded error 403. Oops.

I learned that Google prefers to index pages as https rather than http, if it discovers that a web server supports both. After doing more research I considered a few paths out of this mess:

  • Go full blown https everywhere on the site. Fix images, comment forms, and other content that use http.
  • Redirect https requests to http versions of the same URL.
  • Use canonical URLs to instruct search engines to index the http versions of pages, not https.

Switching everything to https would be lots of work, and wasn’t the end result I wanted anyway. Redirecting all https requests to http would probably be OK, but seems a little bit drastic, and I’d need to carve out exceptions for the shopping cart and admin pages.

Canonical URLs

Canonical URLs are a nice feature,  and I decided to use them to solve this problem. In the header section of any HTML document, you can include a link like this one:

<link rel="canonical" href="http://www.example.com/mypage/" />

and search engines will index the page as http://www.example.com/mypage/, regardless of whether they reached the page as

http://www.example.com/mypage/
https://www.example.com/mypage/
https://www.example.com/mypage/?q=vegemite

WordPress automatically adds canonical URLs to some pages, but not all, so I installed the Yoast SEO plugin to gain more control over canonical URL generation. Yoast added the canonical URLs as expected, but not in the way I needed. If I visited a page on the site using http, then Yoast would generate a canonical URL link beginning with http://. But if I visited a page on the site using https, Yoast would generate a canonical URL link beginning with https://, which was exactly what I didn’t want. I was finally able to force canonical URLs to always start with http:// by inserting this code snippet into my WordPress install’s functions.php:

function design_canonical() {
  global $post;
  if(isset($_SERVER['HTTPS']) && $_SERVER['HTTPS'] == "on") {
    $find = 'https://www.exampledomain.com';
    $replace = '';
    $theurl = str_replace($find,$replace,get_permalink($post->ID));
    return site_url( $theurl , 'http' );
  } else {
    // Leave blank and Yoast SEO will use default canonical for posts/pages
  }
}

add_filter( 'wpseo_canonical', 'design_canonical' );

Fixed?

This should be all that’s needed to make Google, Bing, and other search engines use http for indexing all my content. It may take a few days for the Google index to be updated with the new links, but eventually everything will be http. And that should be enough to prevent visitors from accidentally viewing the site content as https, right? Well, maybe not. I had forgotten about the existence of browser plugins like HTTPS Everywhere that attempt to force use https wherever they can. Even if Google’s no longer sending traffic to https versions of my pages, then, other sources of https traffic still exist. And those visitors will have all the security warning and error problems I described.

I’m scratching my head, wondering how to proceed. Redirect all https traffic to http, as I’d originally considered? Or leave everything as is, and let HTTPS Everywhere visitors deal with the problems that extension creates? Maybe there’s another simpler solution. It all makes me appreciate how complex the job of a web site admin can really be.

Read 10 comments and join the conversation 

Older Posts »