The products were expensive laboratory grade instrument modules (costing tens of thousands each) that were designed to be stacked and work together to gather and consolidate data.  But there were problems with the communications between the modules and the host PC.  Customers would spend hours preparing for a test, and a few more hours to run it.  But partway through the test, the system would suddenly hang up, data would be lost, and the test would need to be run over again from scratch.  And there were all kinds of other mysterious and problematic symptoms that were hard to find and even harder to explain.

The individual instruments were extremely reliable, and had been for years.  The problem was with the added inter-module communications, both the hardware and the software.  The instruments were designed to talk to a single host, not a network.  So the company designed add-on boards to allow for a mini-network of instruments using CAN bus communications hardware and a proprietary communications protocol.  The new boards also provided a USB connection for a host PC to monitor and download the test data.

There were many problems, but here are the main ones:

Poor protocol design

When communications problems exist, I look for two things — the documentation describing the protocol and some tools to capture and analyze the communications bus traffic to see what is really going on.  They had neither.

A protocol is a specification of the formatting of the data and messages that will travel between devices.  The design of the protocol will usually provide a way to minimize the chance of errors and detect them when they occur, typically through some form of checksum or error correcting code.  And it will anticipate the different errors that might occur and describe how to handle them so normal communications can resume.  A protocol document allows all the scenarios to be thought out ahead of time, and for peer review and corrections to polish the plans to provide a bulletproof scheme.

They didn’t have a protocol specification document.  And they had never gone through the design process.  Instead, the ad hoc protocol evolved over time with more and more patched on to it as new issues arose, with little to no documentation.  If anyone had ever seen the big picture of what was being created, they would have realized that it was a mess and a minefield of problems waiting to happen.

Since all of the original programmers were gone, we spent months trying to understand what it was supposed to do, while also trying to figure out why it kept failing and how to fix it.

In an ideal situation, you would have a well thought out and documented protocol specification and tools to sniff the communications and capture snapshots of any faults.  Within days you would find the problem and solve it.  Instead, the days or weeks they saved by skipping these steps meant months of time spent on the back end trying to clean up the mess.  Even worse, there was some serious damage to the company’s reputation and sales.

Lack of safeguards

The system included thousands of individual settings for calibration and configuration.  But these settings were unprotected and could be wiped out or changed accidentally in an instant by an errant command, all without any warning or indication that it had happened.

The protocol had no provision to identify what type of data packet was being sent, such as a packet wrapper describing the contents.  The receiver was supposed to know when the sender was changing states from, for example, a command mode to a file transfer.  But if the mode switch was lost or delayed, the receiver could see the file and try to interpret its bytes as commands, potentially wreaking havoc as it went nuts doing all kinds of random things it wasn’t supposed to do, such as overwriting critical settings.

Bottlenecks

There was a big problem with getting the data out of the system.  After a four hour test, it could take an hour or two just to transfer the data file to a PC for analysis.  There were two big bottlenecks:

First, the CAN bus connection between the modules was being run at 125,000 bits per second.  That might sound like a lot, but standard USB speeds are nearly 4000 times faster.  So a file that takes only a second to transfer over USB would take over an hour on the CAN bus!  (Actually, it is even worse than that since the CAN bus has a very high overhead, using 5 to 8 bytes of wrapper around each tiny 8-byte packet.  So the effective data rate is close to half of the bit rate.)

The diagram above shows the communications flow between the different components.  Each module consists of a laboratory instrument and an interface board which includes an MCU and a CAN and USB interface.  The different modules in a stack talk to each other through the CAN bus backbone.  And a host PC could monitor or download the data through a connection to one of the modules’ USB ports.

Since you could never know which module the PC might be plugged into, the programmers decided to make each module keep a copy of every other module’s data.  This would avoid the bottleneck of using the CAN bus to do large file transfers since every MCU could send it all directly to the USB port. But it didn’t work, and everyone was scratching their head wondering why the PC transfers were so slow. One other bottleneck had been overlooked.

The product specs could say that they had a USB 2.0 standard port, capable of 480 million bits per second rates.  But do you see the black line that transfers data from the MCU to the USB adapter?  That is a standard serial port line limited to only 230 thousand bits per second, about 2000 times slower than the USB port.  It’s like trying to supply a firehose with a narrow IV tube.  The flow will never be more than a dribble.

File transfers took so long that customers would instead attach a dedicated PC throughout the test to download the data as it happened.  Or they would take the modules apart to get to the CompactFlash memory cards that were hidden inside.

Eliminating the communications problems

One thing that became clear over time was that the problems were even worse than we thought.  It seemed like a bottomless pit of new issues to discover and fix.  New product plans were in motion, so it was time to step back and consider changing the architecture rather than perpetuating such a troublesome design.  My proposal was accepted, so I stayed on for another year to implement the new system.

fast and powerful at a low cost

When I first started designing electronics years ago I noticed something interesting.  Some chips that had way more features and capabilities could be far less expensive than some less capable alternatives.  The reason was volume.  If a chip was used in a high volume product like phones or personal computers, then the volumes would drive the price way down and I could take advantage of that even in a low volume product.

The smartphone market has made some incredibly powerful, tiny and highlight integrated chips available at extremely low prices.  System-on-a-chip devices with a powerful CPU, DSP, Video, Audio, Ethernet, USB and more are available in a single chip costing just a few dollars.  Add some memory, connectors and an operating system and you have a powerful computer.  That has led to the creation of powerful single board computers covering just a few square inches, perfect for dropping in an embedded system.

For the next generation of this troubled product, we decided to replace their proprietary hardware and communications with an off-the-shelf single board computer that cost a fraction of their old design, yet had way more capabilities.  The included standard Ethernet was capable of 100 million bits per second, a 400 to 800 fold increase in speed over the CAN bus and throttled USB ports on the old board.  And the PC would connect to the Ethernet bus, not a specific module, so all modules could be reached equally as fast.  The revised communications bus structure is shown below:

Linux Operating System

The boards also ran Linux, the open source (and free) operating system that is already widely used in a lot of complex embedded systems like DVR’s and cable boxes.  That gave us several big advantages.

We were able to completely replace the troubled proprietary protocol with standard Internet protocols (TCP/IP), all included with Linux.  Since these protocols drive the Internet and local area networks, they are obviously well documented, robust and proven, and there are excellent tools (such as Wireshark) to debug any problems during development.

A more modern User Interface

Another huge advantage of Linux is that includes a web server.  (This website runs on Linux.)  My client’s old system was also burdened by a native Windows PC host application that nobody liked and had a number of its own problems.  We created a new user interface with a web application hosted on one of the Linux boards on a private network (not on the Internet).  Since the application now runs in a browser, it is cross-platform and can be used on Windows, Mac, Linux and any other device that has a browser, and it can now be viewed by more than one person at a time.  With the addition of an integrated WiFi hub, the user interface is also available on tablets and smartphones for quick control and monitoring during a test.  In fact, an iPad was included with the new system to highlight this useful feature.

The new product is a big success.  The communication problems are gone, the speeds are fast, and the user interface is a big improvement.  And the new architecture can serve as a platform for adding new modules and creating new products from different combinations for years to come.

Lessons

This story illustrates the value of a few themes that I promote:

• Take the time to design and properly vet a system on paper before you dive into the final implementation.  It will save you time and headaches in the long run.
• Use a highly qualified and experienced person to design the system, or at least to review the plans.
• Write performance specifications and test to confirm they are met.  It was embarrassing that the high-speed USB port was almost uselessly slow and made it to market like that.
• If the system is complicated, take the time to gather or create the tools to analyze it and prove that it is working according to spec during development.  This client should have created a tool like Wireshark for their custom protocol to analyze the data on the bus.  They would have quickly identified many of the problems and fixed them before the product was shipped.
• Consider using existing industry standards.  They are proven, robust and well supported.  There are times when a proprietary scheme is appropriate.  But designing your own encryption algorithm or file transfer protocol is probably a bad idea.

The post Networking Products with Embedded Linux appeared first on Cunningham Consulting.

Motivation

Sunbeam was in trouble due to the disastrous management by “Chainsaw Al” Dunlap that left them with two billion in debt.  Jerry Levin had taken over and was looking for ways to save the company.  He explained that while running Revlon, he had the R&D department develop a new lipstick that would last all day, far longer than the competition.  It worked, and their sales margins and profits increased dramatically.  And Revlon rose from third to first in cosmetics market share.  My job was to do something similar for Sunbeam’s appliance lines to give them a boost.

When you go to an appliance store to buy a coffee maker or curling iron, you will find very similar products from several different brands.  Since their features are often nearly the same, you buy based on price, looks or reputation.  With stiff price competition, all the appliance manufacturers had slim margins.  But if one of the appliances on the shelf had some cool new feature, it would stand out, sell more and could command a higher price.

So Sunbeam’s idea of making the appliances talk to each other and the Internet had a lot of promise.  Their initial plan of how to pull it off did not.

A Heavy-Handed Approach

Sunbeam’s strategy was to sell a dedicated PC in a box that would serve as the hub for all the appliances, and a gateway to the Internet.  It would connect to the appliances through WiFi.  This was the most obvious and straightforward approach.  But I told them that I thought it would be a commercial failure for many reasons:

• Such a PC would have cost a couple thousand dollars back then.  Who would spend that much to hook up some $30-50 appliances?
• WiFi was uncommon in homes at the time.  So the consumers would have to buy that and figure out how to set it up too.
• WiFi hardware was way too expensive.  If Sunbeam wanted their smart appliances to stay within $10 or $20 of the base product, the manufacturing cost of the extra hardware had to be under a few dollars.
• WiFi used too much power.  In products like their First Alert smoke alarms, the battery would be dead within minutes.
• Their customers were used to products that just plugged into the wall and worked, not that required large manuals and an IT department to set up.

An Improved Architecture

My proposal was to eliminate the PC altogether and use a very inexpensive hub instead.  And we’d replace the WiFi communications with a much cheaper and easy to set up networking scheme.

I argued that an inexpensive alarm clock could take the place of the PC as the hub of the system.  Smartphones hadn’t been invented yet, so nearly everyone still had an alarm clock next to their bed.  It was a logical place to check on the status of everything and turn things on and off when you went to bed or woke in the morning. As a clock, it was perfect for scheduling when your electric blanket turned off and your coffee maker started in the morning, for example.  And with a cheap dial-up modem, we could download weather and traffic information early in the morning, ready for when you woke up.

High-speed communications are important for streaming videos and sending large files, but are overkill for a few dozen bytes to broadcast when the dryer is done or your coffee maker is out of water.  So we would use inexpensive low data rate powerline and wireless communications rather than WiFi.  If the product plugged into the wall, it would talk through the house wiring.  And if it ran on batteries, then it would use short bursts from an ultra-low power radio to talk.

This scheme meant that a customer could get a starter kit of networked appliances for $80 or less, such as an alarm clock plus a coffee maker, or alarm clock and a smoke alarm.  They would try it out, see the benefits, and then start buying more smart appliances to add to the system.

Thalia Products

Sunbeam liked the idea, so I was contracted to create a suite of eight prototype products including the alarm clock and a touchscreen kitchen organizer within five months.  We assembled a team and pulled it off, and were the hit of the annual Housewares Show in Chicago.  I then joined Sunbeam as Chief Technology Officer of their new tech division, Thalia, and later became VP of Technology for the corporation.

My role was to develop the technologies (such as the low-cost communications circuits that we called HLT, or Home Linking Technology) and coordinate the incorporation into the products of the different divisions within Sunbeam and to promote and sell it to other companies.  Consumers would benefit if we were also able to communicate with products from other manufacturers.  For example, we wanted our alarm clock to be able to control Lutron light switches, check that your garage door was closed or tell you when your Maytag dryer was done.  That meant that we needed some standards for both the communications hardware and the messages that were passed between the devices.

AHAM

The Association of Home Appliance Manufacturers (AHAM) organized discussions about smart appliances and how to connect them.  I chaired the committee that was writing the standard for the communications protocols, and we met every two to four weeks.  Agreeing on the hardware was difficult since a company like Maytag or GE would be content with adding $20-30 to their cost for a high-end large appliance like a washing machine, while we wanted to keep it to a couple of dollars for our small appliances.  But we were making good progress on the messaging protocols.

Someday I may write a blog post about my meeting with Bill Gates.  Microsoft was trying to convince the AHAM members that we all needed to use some form of Microsoft operating system and an adaptation of their Universal Plug and Play (UPnP) architecture.  I was arguing that we didn’t need all that complexity and to be paying licensing fees to Microsoft just to turn a light on or tell when your dishwasher was finished, and we had proven that it could be done.  So Bill called me out to his office in Redmond to talk…

Sunbeam’s Demise

Unfortunately, that all came to an end for us when Sunbeam’s financial woes from Al Dunlap’s management caught up to it.  In 2001, less than a year before we would have started selling the new products, Sunbeam filed for bankruptcy. It was sad that we didn’t get to finish, but I was very proud of what we were working on and what we accomplished.

Today, this type of work continues with the Internet of Things (IoT), the Cloud and other technologies to connect more and more of the devices and points of information in our lives.  There are still many opportunities and a lot to be done.

The post Smart Appliances appeared first on Cunningham Consulting.