Work

Retired My Work Girling Test Set Hardware Software (Real-Time) The last few years

Well the un-thought of has happened I have retired and I am looking for a new challenge so what have you got to offer?

I never even thought about it and I would probably have gone on for a goodly number of years but my wife said "Why not?" and I didn't have an answer! I'm still in love with what I do and I am looking for a new challenge to keep that fire stoked. I have a project I have supported for a goodly few years now and they are totally dependant upon me. If nothing else comes along I will work my way through that and make it open source but not until it is better structured and documented.

However, I also want to do some work on art installations utilising light and a video wall. I have already built one video wall for my work using Raspberry Pi and I intend to do another utilising the upgraded Raspberry Pi 3, the lessons I learned and the better hardware.

So any good ideas for tasks for a Real Time Systems Engineer with a thirst for a new challenge. I guess the restriction is only a few days a week!

Frank

A little background about me and the work I have been doing for many years now. I’m a design engineer who has a great deal of experience in all aspects of computer, software, instrumentation and real-time system design, hardware, software and firmware. I have designed and build a wide and diverse set of systems everything from anti-lock controllers for commercial vehicles and trains to custom measurement systems that measure the railway infrastructure from a train moving at speed. My formal training was as an electronics engineer but as software has become a major part of most systems I have continually evolved and re-educated myself.

My speciality is an understanding of what makes such systems work successfully in a vehicular environment, everything from the environment to the way the people work. I profess not to be a manager but I have often arrived to take a technically lead role on a failing project which has finished as a success. However, I am not foolish enough to forget I have had an occasional failure and learn the lessons from these no matter how painful.

As the systems have become controlled and driven by software I have worked in everything from assembler and C through to G, the graphical language used by LabView. I suppose if I need to define something within the field but outside my comfort zone its large currents and voltages but even those are not unfamiliar as I designed a communications system to talk to electronics on the live side of a pantograph on a train. For anybody who does not know the pan on a train connects to the 25kV overhead wire. Like all good engineers working with real time systems I have an understanding of issues associated with EMC Interference and Susceptibility although I am not a specialist.

On several occasions I have built models of systems to help in the design process, prove functionality or simply to convince senior management that a project is possible. However, models also provide a great way to reduce risk as they can allow the testing of sub-systems and components long before the physical construction starts.

Because of the nature of some of the systems I have designed I have learnt a great deal about Linux, Lamp and Web Services. When a system produces large volumes of data it also has to become its own data centre or the customer will not be able to manage.

So in summary everything from driving a solenoid to driving the Internet.

Frank

Although this project was not quite my first design project it is a very early example. At the time I was working for Lucas Girling at the Fen End Test Track. It was one of the most idyllic places to work, we were out in the country a few miles south of Solihull on a World War Two airfield, surrounded by tree, grass and green fields: wonderful!

The project was very simple in concept design a test set that would allow field engineers to find the faults on Anti-Lock Systems fitted to commercial vehicles. At the time we were reasonably successful at selling the systems but the maintenance network was failing to keep the vehicles in a serviceable condition and we were getting a lot of second and third level support calls.

I had very little software experience but I was very capable of designing the hardware therefore I set out a simple fault analysis tree, subcontracted the software to the software team who worked in the other half of the lab and set about designing and manufacturing the hardware. The result was a disaster; my software inexperience was used as an excuse to bin my fault tree and follow "best software design practice". As a result the software was of no practical use and I was blamed for the system not working.

However, I was well enough considered by my manager at the time to allow me the freedom to perform a rework. The discoveries and the learning process that followed still influence my thinking. "Keep it simple", "Junk in Junk out" and "If you don't understand, ask!" are phrases I suspect everybody who knows my work will have heard me use ever since. Six months later the Test Set does the job perfectly but only a few are in use.

Remember the three levels of support, well I was the level three support. Every week I would go out and deal with a few major problems that nobody could solve. I started taking one of the Test Sets and using it, soon our second level service engineers were asking for Test Sets and I was suggesting to the directors they be given them. Eventually, after a heated argument where the managing director “We might as well give them away, nobody wants them!”, the second level engineers were given one each and they started to sell.

Within six months we were sold out and had to manufacture a second batch of three hundred. Eighteen months later the same director apologised to me and told me he was glad I had the belief to argue with him.

One of the most rewarding hardware design tasks I have ever completed was to redesign a very compact signal conditioning and multi-purpose I/O board for a small company. I completed everything from the circuit concept through design to the PCB Layout and onto procurement and manufacture. One of the big advantages of a small company is the ability to influence all stages of the process. However, it is also the biggest disadvantage as you also have to accept sub-tasks you don't like. I loved it; the picture below shows a populated prototype board which has had a few components changed by hand, a task that requires a steady hand.

The project was rewarding on many levels from the personal proof, that I could still do it, to the realisation that more functionality had been crammed onto the board and that overall the new boards work to much higher standard.

The picture below is for part of the circuit diagram for an Anti-Lock Controller which illustrates very nicely that a good understanding of analogue electronics is essential even today when connecting sensors with small output signals to digital systems. This circuit was able to detect open and short circuit conditions along with shorting to ether supply I designed the circuit from basic principles and worked with a young mathematician to optimise the component selection.

To the best of my knowledge the circuit was never put into production. However, it was well proven during the development and test program.

For me the decision to implement a project in a particular language has never been an issue. Historically the project has chosen the language to be used ether because there was nothing else available that could do the job or the preferred platform restricted the choice. However, today the problem is significantly reduced as the various languages are available over a greater spread of platforms.

One of my favourite software projects was the Track Recording System for which I lead the design team for a number of years. The Track Recording Analysis Computer System (TRACS) from which DeltaRail's TrackLine was derived was a multi-processor system which used dedicated hardware to perform some of the very high speed preprocessing of both video signals and sensors. It cycled six hundred and sixty six times a second but by the use of carefully designed hardware managed to capture some signals at twenty two kilo-hertz. Practically calling it a software project belies the complexity of the hardware.

The software was based upon a real-time Unix platform with a Windows user interface. However, some of the system ran without an operating system because the time restraints were extremely critical and the processor performance available was insufficient. The software was written in Assembler, C, C++, Perl, Bash Script and SQL the language being decided by getting the best out of the platform and simplifying the task at hand.

The software captured the measurements in hard real-time and recorded the raw data to disk. Thus if everything else failed it was possible to re-analyse the data at any time and to do so for a completely different set of results. All the recorded data carried the calibration information and therefore all the calculations and results were in real world understandable units. Further, every stage of the system was configurable and adaptable to any requirements change.

One of the demonstrations that always impressed engineers and customers alike was to start a journey at a modest speed and see the results being charted and printed within a few seconds of passing over a track section. Then as the speed increased to two hundred kilometers per hour the results would lag further and further behind until if the journey was long they might be twenty minutes behind. However, by the time the train reached it's destination the results were again being produced within a few seconds.

The only effect of increasing the complexity of the analysis was that the lag was longer and recovery slower. A chart recorder would run out of paper and an error message would be produced on the user interface. A new set of paper would be loaded, the system told to continue and the chart would restart from a point just before the last sheet was printed and proceed to catchup.

One of the major design features of this system was the flexibility that resulted from the system being modular. This meant that as the system developed we could move processes between processors or let the processing lag behind the data capture without having to modify any of the components. It also meant debugging was much simpler because bugs could generally be localised to a specific small process. Further, as more powerful computers became available or more complex processing was required it was a simpler task to integrate the system.

After a period of contracting which was terminated because of poor accountancy by my accountants, who caused a run-in with Inland Revenue, I returned to permanent employment. This was a simple expedient to allow the trouble to flush out of the system and it took over three years of discussions and negotiations before it all started to settle. Whatever you do don't fall out with Inland Revenue; if you do you will have lots of trouble!

I started work at a company in Derby which makes products for super cars, nice location, nice co-workers, great work and bad management. The point however is that I worked on the Tesla model S my little bit was the door mirror controller, tilt, pan, forward, reverse, custom driver settings and park. Interestingly, from the point of view of someone who had worked in the automotive industry for a long time that was almost the specification! At the time Tesla were only interested in the functionality if it worked it almost didn't matter how or why. The automotive industry is almost the other way up the functionality does not matter as long as you can prove it meets the specification and complies with all the legal requirements.

As part of the integration I had a trip to San Francisco and took a quick look at all the Silicon Valley companies which surround Tesla. We also had a look at the Golden Gate Bridge, Monterey and that part of the coast; I enjoyed seeing a small bit of America.

While at Tesla I spotted a large splash of metal hanging on the wall and asked what it was. The answer is enlightening to anybody who is interested in science or technology. It was the metal from the inside of a Tesla battery pack. One of the tests they did was to drive a spike into a fully charged battery pack. This of course short circuits the batteries and melts the spike and the batteries in a very short period of time. The heat causes a lot of expansion and vaporisation of plastic insulation and the whole lot is ejected as a molten jet!

Why hang it on the wall? Simple it reminds everybody why the battery pack has to be protected and illustrates the problem to anybody who does not have the necessary understanding. I also have to say as an engineering artefacts go it has to be one of the prettier modern examples I have seen!

This period ended when I was told I was not going to be offered the job I had been promised and told to go on gardening leave. My line manager, who was looking very embarrassed, in the same meeting suggested two companies where, he thought, I would easily find employment. I have had a number of odd meetings over the years but this has to be one of the strangest. To paraphrase; I'm sorry you are leaving but here is where you will get a job no problem, I didn't want either.

My Final Hora

By the time the gardening leave ended I had another job in Nottingham at a company called TEW Engineering. Later I discovered I got the job simply on the basis that I was clear I loved what I did and showed a lot of enthusiasm. However, I landed on my feet and as the only electronics/software engineer in the company I soon became involved in many aspects of the company activities.

The first task interestingly enough was not truly electronic and used no software; it was a very high reliability switch with two sets changeover contacts, one activated by pulling the knob and one activated by pushing it. A separate circuit, I developed, provided a visual feedback mechanism that indicated the status of the circuit controlled by the switch. This circuit used a very interesting trick to allow the LED indicators to be supplied from AC and DC supplies over a very wide supply range but the really tricky bit was the mechanics and packaging.

The switches required mechanical accuracy and robustness of operation or they would simply be destroyed. One million activations in each direction was just the start of the specification. The package space was very compact and specific, requiring the use of flexible PCB and surface mount components. The rear surface of the switch is literally the size of the connector plus a couple of millimeters, so plus the metal wall thickness.

It was extremely gratifying when it passed all the tests and was approved by the very demanding customer.

CAD PCBs

One of the things I never thought I would see again is PCB Layout done by hand but I can confirm mechanical CAD packages are being used to draw two layer PCBs and they are being used in anger. This makes no sense today because a PCB package does so much for you once you have drawn the schematic and many of them are available for free.

Core Software Subcontracted!

Like a lot of companies who have electronic systems as part of their products the electronic and software design had been subcontracted. My perception of this is that for a lot of companies this becomes a real problem. The product has at its heart a circuit with or without software that makes the product viable. However, because a third party has designed and manufactured that component you don't have control. Everything is fine until the companies fall out or the subcontractor goes bust.

In this case both had happened a court case had resulted in the source code being provided for one system but it was obfuscated so of no real practical use! So my task should I choose to do it was to recover one reasonable chunk of real-time software from the obfuscated source and rebuild two more chunks by using the running system as a reference. If you like a challenge the obfuscated code is doable and quite interesting but decompiling the system code and reverse engineering anything of any complexity which was written in C is much more difficult. The decompiling of the C code was made even more interesting as the code was for a PIC and I had no idea which compiler was used, had never worked on PIC software and could only use free tools! Further, the Microchip decompiler is only intended to be a diagnostic tool so although it did decompile the Hex it did not help any further.

Just to summarise that assembler source for an unfamiliar processor with no meaningful labels and no variable names. Oh! I forgot one other thing the circuit diagram was a poor quality photocopy.

Guess what - I did it!


All material on this site is copyright and cannot be reproduced without consent.
© T&F Smith 2004-16