March News

The wheels are turning but not moving

Dressed up like a car crash
Your wheels are turnin' but you're upside down.

   Stay (Faraway, So Close!), track 5 — Zooropa album, 1993.
By U2, Lyrics at

Software Development

The website has become difficult to change, plus we will be posting projects with documentation. This led to a bit of PHP with its learning curve. The NetBeans IDE and the PHP plugins were installed on an Apple server. The MAMP was downloaded from MAMP Pro website. NetBeans then asked for a choice of frameworks. After a brief peek at available documentation for PHP frameworks, CakePHP was chosen. Over 1000 pages later, plus PHP manuals, we are encouraged by what is possible. A couple of days was all we could spare, but this will be given a slot each month.

Other embedded matters demanded our undivided attention. The smaller I/O cards with CAN started off with some NXP Xpresso evaluation boards we purchased almost a year ago (LPC11C24 based).

NXP LPC11C24 Xpresso board

The usual path starting with the blinking LED, then slowly ported our tick-based RTOS. Next, we trimmed down on the sample code, as this device only has 32kBytes of Flash and 8kBytes of RAM.

The development used the Code_Red toolchain, which is free for target code of less than 256kBytes. The tools and device are impressive, but we soon ended up in the weeds.

Four boards in quick succession refused to be Flashed after making some large changes — after two weeks we would learn about the “Code Read Protect” word that the Code_Red default project handled behind the scenes. We somehow managed to change this, but it is a very useful feature. The on-line help suggested that the USB power needed to be increased, so the Code_Red toolchain was downloaded for Windows, but we proceeded to disable our last board.

At €14 a board, not a train smash, but a few more boards will be bought before going for the tray of parts on a MOQ. On closer inspection of the internal functions, one feature we were interested in was to see the internal clock taken out onto a pin. Often this feature is unavailable on pin-limited packages. We looked at the 48 MHz clock, but it did not look like a square wave with the setup we used, so the final output was divided down by 48 to give a nice 1 MHz square wave clock for viewing. This pin is also used for the boot process, which is when we dug a bit deeper into the documentation to discover what we did. The boards still blink the LED and the clock wave looks good, but the debugger was unable to erase the Flash. At least we know that the devices will be difficult to clone if necessary.

Tales from the Woods


Tree absorbs conduit

A tree in Griffith along their pretty main street slowly absorbing a plastic conduit and electrical cable.

Out and About

At the end of February we travelled from Adelaide to Brisbane, returning via Newcastle and the Hunter Valley two weeks later. A few weeks before Queensland and New South Wales had several bush fires during a particularly dry season. There were livestock being driven from Queensland south to greener pastures.

On the return trip the rain fell so hard in parts of Queensland making it difficult to drive. With so many low lying creeks and flood markers, this was going to be interesting. Luckily, there was no flooding.

We visited family in Brisbane with a gift of our old MIG welding equipment as our intention is to get TIG equipment later. A really nice telephoto lense was passed on by my brother — thanks Bruce! We bought stainless steel polishing equipment from the Eisenblatter agents in Brisbane, bought electronic components, then visited friends in Newcastle. The Hunter Valley was skipped on a previous trip from Sydney to Brisbane going via the Pacific Highway, but there was so much construction that it was not a nice drive.

Besides the Hunter Valley wine region, there were two interesting micro breweries — the New England Brewing Co. at Uralla along the A15 (New England Highway), the other in the Hunter Valley along the B82 outside Cessnock (Potters Brewery or previously the Hunter Beer Co).

A few years back I did a similar trip but via Sydney on the way up. The north east part of South Australia and north west Victoria are very flat at only a few metres above sea-level. The road is mostly straight, so you cannot out run any trucks, but they have to do this on a weekly routine and stay awake. Certainly a challenge on highways that do not reflect the importance of road freight without a shoulder to pull over for most of the way. The distances are vast and it was not easy to do over 1000 kms in a single day, particularly with the number of kangaroos dead at the side of the road. Travel at night is not recommended for cars.

Along B12 a couple of hours out of Adelaide

16:30 in the afternoon, about five hours east of Adelaide along the B12 (Mallee Highway).

The road was flat for hours with the occasional salt pan. This was most likely the Murray-Sunset National Park.

Salt pan from air near Swan Hill

Flying a couple hundred metres above this area in May 2010 with Lloyd Bentley (pilot and owner-builder) was a lot more spectacular. I think this was after we refuelled in Swan Hill, flying towards Adelaide into miserable weather.

Cockpit of experimental plane

Cockpit of Lloyd's experimental plane.

Horse stud farm along Golden Highway

A horse stud along the Golden Highway (near Singleton), NSW.

The Hunter Valley was green, but several vines were in poor health compared to McLaren Vale. The owners said they had to choose which vines to irrigate during the drought, plus mechanical harvesting could be improved. The early harvest compared to McLaren Vale wine region was also cited for the poor state of the vines, but on returning to McLaren Vale, the harvest was over and the vines still look healthy. We bought a couple of bottles of reds — some that we could not taste for whatever reason, so the results were mixed. None of them were cheap either!

Tree splits rock over time

Rock, tree, scissors? An opportunistic tree grows through a split rock that had rolled down the hills many years ago. No doubt it squashed the tree's great-great-great grandfather.

This was not taken with a phone camera, so there is no GPS data, but I think it was on the way to Dubbo.

Early morning, 9 Mar, freight burning

Early morning, 9th March. The smouldering remains of a fruit juice and fizzy drinks delivery. All along the trip the monopoly of the fridges in the service stations is obvious, with even the fruit juice and water being bottled by Coca Cola. They also cost a lot more than petrol, but it seems that the freight is not immune to fire. There were some cans spraying out their last gasps, but not enough to put out the fire. Not sure what happened, but the tyres and some wooden pallets have passed into the next world as seen in a closer up shot below.

The wheels are turning but not moving

Bit of a reminder about no matter how good the tyres are, they need to stay on the road. The accident might of happened the night before we passed through on the way up. The tyres could not even remain on the vehicle, let alone the road. On the way up (Mallee Highway — B12 — before joining the Sturt Highway — A20) the wheels were still on the car without police tape, but a couple of days later they found a new home.

Closer up, truck freight burning

Embedded Software Lawsuits

Vehicle Software

How reliable is vehicle software? Toyota has reached a $1 billion settlement with the US Justice Department over handling of customer complaints as reported in Toyota in U.S. settlement over unintended acceleration: CNN. The story was from Reuters, 19th March. In the same article, Toyota recalled millions of vehicles related to the acceleration issue, prompting more than $1 billion to resolve economic-loss claims. (That is in addition to the recent $1 billion).

In EE Times, Jack Ganssle went into a bit more detail about the software. See two articles, Adding Up Toyota's Expensive Software (31st March, where the price due to the fines and an estimate of a million lines of code to be about $1,200 per line of code — more than the original $1000 per line of code for the space shuttle). In Toyota's case, that excludes the engineering effort. Mike Barr was cited in an article by Junko Yoshida on EE Times, Toyota Case: Vehicle Testing Confirms Fatal Flaws, where two instrumented vehicles were tested (31st Oct, 2013).

Posts on various sites claimed US engineers would not have written such poor code, but then again, GM only recently recalled 2,6 million cars with faulty ignition switches that inadvertently switched off, stalling the engine, disabling power steering and brakes, and disabled airbags, resulting in several deaths and possible criminal charges. My guess is that there is software behind airbag deployment, or at least a lot of electronics in MEMS sensors for detecting sudden deceleration and setting off the explosive charge to deploy the airbag.

In their defence

We recall the “bit flipped” case that was blamed on causing a death, with NASA being tasked on examining the Toyota source code. Apparently, NASA ran out of time and did not conclusively find errors that caused the flipped bit. Floor mats, sticking accelerator cables and driver error were blamed by lawyers for Toyota, however, some vehicles that crashed did not have floor mats. People writing software are very junior in any corporate structure, and even if they knew of errors, they would not be able to state that publicly. They would also be under incredible pressure and not be able to run the software in a decent instrumented setup with replay etc, as few development environments support replay under real-time constraints. The cubicles in corporate software remind one of battery chickens — as laptops became smaller, desk real-estate decreased to a setup where it is almost elbows-to-elbows. And that is not because teams have huge numbers, but because any land grab that engineering cedes is dished out by HR to more deserving staff members in marketing and legal departments. Their salaries also reflect the pecking order, so why are some of the lowest paid workers expected to write fault-free software under unrealistic deadlines?

Aerospace embedded development

Embedded computers in ECUs in vehicles are a lot smaller than the those on board aircraft or spacecraft. The rigour of the specification, the development, correctness proofs, testing, 100% coverage statements, etc. did not stop the Ariane 5 loss due to software, and no single software developer was martyred, but it would be difficult to get people to step up to the plate on similar risky projects if the hundreds of millions of dollars were charged against the ESA software developers. See A Bug and a Crash, by James Gleick for an analysis of the bug that tried to stuff a 64-bit number into a 16-bit space, causing an overflow, which destroyed a giant rocket on a maiden flight that took 10 years and $7 billion to develop. Not to mention the uninsured satellites that were to be launched.

NASA has many reports on identified problems with various launches. This is to be expected in any pioneering field where the difference between a rocket and a bomb seems to be that the one has a hole in the bottom and that everyone hopes it will fly vertically rather than blow up. Software, differences between imperial and metric units, and a host of others show that this is not limited to vehicles. It is just very difficult to work on such complexity. Working on a window the size of a computer screen into source code is not fast when trying to understand any system with millions of lines of software. Even Mike Barr took eighteen months to scan Toyota's source code. Imagine being tasked with having to understand it to be able to modify or upgrade it?

Much was said about better simulation that might have caught the Ariane 5 software defect, but having written software for aerospace, that is often a luxury that you do not have, as there are too many things that can go wrong writing software next to a missile that has powerful radar in the seeker. Apparently, the whites of your eyeballs can be cooked if accidentally triggered outside a suitable chamber. Forget trying to realistically emulate a rocket's acceleration every time you do an edit-compile-download-debug cycle.

Is this common?

Is the embedded landscape really that bad? I think most people who have spent a few years on the software coal face will have a story or two about lack of enough resources or time. Many will have no problem sending staff into areas that are questionable and might have huge consequences for the company. For the company funding the work, they need to set time limits otherwise it would drag on forever without being released beyond 90% complete. Below are two seemingly simple cases that could have become headline news if they went wrong.

I was asked to look into replacing the control system of a steel mill. The managers asking for the upgrade, the sales people/ agents for the proposed boards, and the software staff at the mill were all on different paths. There was little chance of instrumenting the mill or setting up a parallel simulation, and the money I was paid for having a look at the scope of the project was less than three months salary. Only the software developers at the mill agreed with the report, as they had to make changes and maintain the existing system. The managers and sales folk were upset that there was not a quick fix at very little cost, but a couple of tons of 1300 °C stainless steel at 10 m/s running through mills is not for the faint hearted, and certainly not for a visiting casual observer to the system.

On another occasion, I was asked to upgrade a gas plant SCADA without a sign-off list, or anyone to verify the software. Nobody in the company was a gas plant expert to jointly take responsibility. On a recompile of the old software on the newer version, there were well over a hundred duplicate alarm errors — yet nobody would put into writing which ones to remove. There was not even a bund to deflect a blast at the control room, or a main emergency shutdown system. In fact, almost half the system was disconnected at the plant — which was not reflected in the software. There was not even a system where the software could be downloaded to do some preliminary testing on a similar PLC. In fact, we did not even have access to any PLCs made by the company we worked for! There was only a phone and laptop on the shared desks. Needless to say, the software for the PLC was pirated as the gas plant did not have a copy!

I recall the unpleasantness and dismissive attitude of management in the unwillingness to take responsibility for this plant, but none of them would. So when there are tales of woe on major software projects, some of them cancelled after billions had been spent, then all we can do is sip another glass of wine and recall many embedded projects in corporate locations as a contractor.

Why is this important?

After such high profile cases on embedded software where the errors are unidentified, but the developers and lawyers are keen to do battle, the days of single processor designs might be over for all but the most simple systems. Watchdogs need to be enabled which pretty much requires permanent instrumentation as common JTAG probes only have a view when target execution is suspended. For our future systems, this is at least dual core and SPI based instrumentation.

For readers unfamiliar with SoC devices, there is little visibility of the core from the pins, so other than built-in software instrumentation, it is impossible to instrument a target without sacrificing limited peripheral functions. For many targets, memory for code (non-volatile) and instrumentation (volatile) is severely limited. Time will tell if releasing the source code is a good idea. In the Toyota case, the secrecy was blamed on poor code quality, but then do the vehicle companies want to allow anyone to copy their code? Who is going to volunteer to look at the code, or if paid, at what cost compared to the original development costs where the designers have more invested than casual observers?

It seems that the arguments are converging on safety certification, formal methods, aerospace or military style software development, and even more expensive tools. However, the hourly rates remain depressed, salaries are under pressure as this work can be outsourced, and the skills required are not in line with the rewards. This is not going to end well, as where do you get certification done at an affordable rate? Can insurance help? Would limiting damages help? Would certification even guarantee error-free code?

Although we have heard of legal cases against embedded software developers, what about the cases against insurance companies who might decide not to cover the claims against them by the embedded developers?

Last word

For another analysis of the Toyota code, see The Quality of Embedded Software, or the Mess Has Happened, by Andrey Zubinskiy, dated 29th October, 2013. There were almost 80 thousand MISRA violations, plus many other errors listed.

As a frequent visitor to filling in on software projects under pressure, almost all have their own standards for tabs versus spaces, or how to indent, where to put the braces, and comments. Doxygen was rarely used. Even in major open source projects, there is little consensus on even the most basic editing. Real-time kernel? Just another can of worms. For the vehicle industry — we cannot even agree on which side of the road to drive on, or which side of the steering wheel to put the indicator lever.