What does the Delta-T Probe do again?

The Delta-T Probe was designed to do two things: (1) Detect the presence of date and time usage within an embedded system, by detection of RTC chip or time/date presence in the system's memory. (2) If time and date is being processed, capture the relevant code sections to establish how the system uses the date, and if it is doing so in a compliant manner. The Delta-T Probe has proved itself to be a useful tool covering a wide range of applications outside of building services, for which it was initially developed. It was developed as a tool for grabbing code, but what has become clear is that in many industrial sectors it is being bought as a tool to verify that time and date does not exist within an embedded system. Many current blue-chip companies Delta-T Probe have all bought it for that reason alone.

	Is the test done with the target system on-line?
	Yes, the test is done on-line. If the system is mission-critical we prefer to connect to a standby system, and once successfully hooked up transfer control from the main processor to the standby. If there is no dual CPU control then you have to connect straight to the live system.

Does the system need to be powered up?

Yes, the system must be powered on to conduct the tests. It is possible to attach clips while the system is powered off, and then power up, but the majority of the testing carried out to date has been while the systems are on and running. Each pin on each IC clip represents 0.05 of a TTL load and 6pF of extra capacitance... in other words an insignificant loading on ordinary logic connections between ICs.

We have many PLCs and we wish to check them for embedded clocks. There seems to be no easy way of doing this. Is the Delta-T Probe of any assistance?

With regard PLCs (process controllers), the Delta-T Probe is an ideal test tool. The difficulty with most PLC applications in demonstrating to an adequate level of confidence that they do not use date and time. The Probe can be used to demonstrate that a PLC does not contain date/time components, or if it does, that the date portion of the information is not being used. There are limitations to its use - mainly in the area of access to ICs and the fact the Probe is designed for looking at older types of chip technology (i.e. JDEC style packages). The primary application is on equipment 5 years+ in age, with an 8 or early 16 bit CPU running at less than 20MHz.

	What happens if there isn't a record of a chip in the database, but it is highly likely it is a ROM/RAM/CPU chip? How will the Database be updated?
	If we receive an enquiry regarding a chip via one of our distributors we investigate it, and if it is a ROM/RAM/RTC chip which can be connected to we will add it to the chip database typically within two working days of the request submission. The database updates are posted on the embedded science FTP site.

Is a Delta-T Test invasive? Does it force new data into the system? How come this does not damage the equipment under test?

A Delta-T test requires the connection of test clips to certain chips in the target system, to allow the Probe to 'spy' on what the system is doing. The Probe does not require the system to be switched off, or any chips to be removed. It does not introduce any signals to the system under test and is designed to operate without the target system being aware of its presence. The clips are designed to be mechanically and electrically safe to minimise the risk of interfering with the system operation during clip attachment. The technical spec is that each channel on each clip represents 0.05 of a TTL load at 6pF capacitance, which for non-technical people means that it presents a negligible additional loading to the target system electronics.

	If you are extracting code, does that not breach copyright laws?
	If we were to extract and disassemble the entire firmware, then of course the answer is yes. What the Delta-T Probe does is extract only very small portions of code for analysis, akin to photocopying a page of a book, which copyright laws permit provided you do not financially gain from re-selling the photocopy.

	If I get a third-party to analyse the code captured by the probe, can I expect them to guarantee or warrant the compliance status?
	Absolutely not. Nobody is in a position to guarantee or warrant the Y2K compliance status of someone else's product. A code validation service will give you a good indication of whether a system is likely to fail, which provides you with a greater confidence factor in the compliance status of the system.

	What if a system only schedules a task every six months. Does that mean I have to run a test for six months to capture the relevant code?
	Let me turn that question around… how does the system know that six months is up? It has to check, on a continual basis, to see whether the desired time period has elapsed. Most systems do this hundreds of times per second, so picking up code dealing with large interval time/date comparisons can be done in a very short space of time.

Is Delta-T testing risk free?

No testing is risk free. Attaching clips to chips carries a certain amount of risk, primarily from accidental static discharge into the chips being tested. It is important that the Delta-T Probe and the operator is connected to the target system's earth (usually via the casing) using the clips and wrist-strap provided with the Delta-T Probe. Some very old systems may have corroded contacts where chip pins enter sockets, which may be disturbed as a result of physically attaching clips. Results so far indicate no such failures have occurred in testing, but there is an ever-present danger that they may. In other words, there is no excuse for not planning a test properly, which includes the provision of appropriate contingency measures should a system fail.

I would be grateful if you could supply me with a list of the potential areas of use of the probe within the utilities industries, based on your current operational experience and customer base.

The Delta-T Probe does have a serious application in many of the industrial systems used by the different utilities sector industries. An international gas supplier was the first to identify the industrial application of the Probe, as they have a number of safety critical systems where they cannot prove compliance just by looking at the ladder logic on the PLCs used to implement the system. They need to look at a lower level to categorically rule out the presence of date processing. This is especially true for most environments heavily dependent upon process control hardware, as very few process controllers contain real-time-clock chips, and those that do tend to not use them. In addition to PLCs there is an application of looking at how distributed SCADA data logging devices record their information – the Delta-T Probe cannot check out the SCADA head-end (as this is usually PC based), but does allow a 'bottom up' systems investigation strategy where a 'top down' date-change test has proved to be inconclusive. Absence of date-related functionality can be demonstrated by the following results: (a) No RTC found and no clock image detected in memory chips (b) Clock image detected, but no application is reading the clock image from memory (c) Clock image or RTC chip detected, but application only reads seconds, minutes, hours and day of week The above conditions usually account for more than 80% of industrial systems. The Delta-T Probe is intended for testing older hardware, typically of more than five years in age. The reasons for this are multiple; older systems account for the bulk of the unknowns, the processors required external RAM and ROM chips (those that did not only had very limited internal capability), and the processor speeds were sub-20MHz. Use of the Delta-T probe is limited by physical access to the chips for connecting clips to them. Many industrial systems are rack-mounted, which makes it almost impossible to attach clips with the boards in situ. In such cases a 'back-plane extender' is required into which the CPU card can be plugged, which in turn allows free access to the chips. Back-planes come in standard and non-standard forms, so this is not applicable to all industrial rack-mounted systems. In some cases CPU cards use a plug-in 'daughter board' for I/O connections or other uses. This again makes the relevant chips hard to get at. Header extender cables can be used to connect the daughter board to the main CPU card, allowing access to the key chips. Another limitation is to do with conformally coated boards - the clips supplied with the probe are capable of cutting through a thin conformance coating, but specialised clips are required for heavily coated (or paint coated) boards. Conformal coatings are often found on circuit boards used in off-shore and environmentally hostile environments.

I have seen some commentary in the Y2K newsgroups that suggests the Delta-T Probe is really just another In-Circuit-Emulation (ICE) tool, so is not unique. Is this true?

The Delta-T Probe and ICE tools are completely different in what they do and how they work – there is simply no comparison between the two. If you know the design of a system, including the memory and I/O mapping, and have the source code, then you can run in-circuit emulation and play about with different code variations to see how they will behave post-2000. I have seen ICE tools being sold as Y2K test tools, only to have the claim withdrawn a short while later (such as Validor2000, March 1998). Basically they are only good if you (a) have the code, and (b) know the device the code is written for from a designer's viewpoint. The Delta-T Probe is completely different. It does not run code... it simply 'snoops' or spies on the code being fetched from ROM/EPROM's, and combines this with a year-date access trigger (based on a memory location access, RTC register access, or both) to freeze the relevant code. The unique, and heavily patented part, is the way it is able to build a parallel memory map to the system under test, which mirrors the data stored in the system's RAM, and the way which analysis for date or time-related activity is performed on this memory. In other words, the Delta-T Probe is not an emulator. You cannot 'fix' the code and tell the Probe to run it. It is simply a diagnostic tool to help you rapidly locate and validate the date-sensitive bits of code.

I am trying to formulate a claim to describe what the Delta-T Probe does, so that I can recommend its use to my supply chain organisations. I am particularly concerned that they understand its limitations and the context in which it is to be used.

The Delta-T Probe was conceived as a device to detect date and time usage within a target system, and grab year-date related code samples for analysis. It works by 'snooping' or spying on the memory and code activity of a live embedded system. The purpose of this was for testing systems of unknown (or doubtful) compliance to gain a greater level of confidence in their ability to operate post-2000. Many UK companies have budgeted huge figures for the replacement of unknown or 'doubtful' systems, and the Delta-T Probe has been shown to create huge savings by dramatically reducing the number of systems that have to be replaced. Since putting it on the market, the buyers of the Delta-T Probe have found a number of new or related applications that have their own selling points... * Proving that a system does NOT have time and date in it. This is the reverse of what it was designed for, but it is very good at doing it. Companies with production control equipment (such as huge numbers of motor drives and process controllers) have found this to be invaluable. * Independently verifying vendor claims. In the above case this is especially true, as many vendors of this type of equipment claim their systems are compliant without any real justification for the claim. The key in the US market seems to be risk mitigation, to which the Delta-T Probe is very well suited. * Testing the validity of upgrades or 'fixes'. Two cases of bogus firmware upgrades being sold by vendors have been picked up by Delta-T Probe users. One of those would have cost several hundred thousand pounds to implement in an upgrade roll-out programme. If you are formulating a claim for what the Delta-T Probe does, please make it clear that nobody is able to provide a guarantee or warranty for other people's systems with regard Y2K. Compliance must be measured on a confidence scale, ranging from [no confidence at all] to [100% confident]. The more investigation and testing you do on a system the closer you will get to 100% - the Delta-T Probe is a diagnostic tool which dramatically speeds up this process. Even total system code disassembly and analysis (used extensively in the petrochems industry), which takes many weeks, will not give you 100% confidence. The more critical a system is, the more confident you need to be in its ability to operate. The Delta-T Probe is an invaluable tool for investigating systems and determining their compliance to a high degree (I hate being asked to put a figure to it, but it ranges between 95% and 99.9% depending upon what is found) in a very short space of time (hours). It would be great if we could test everything with it, but again the Delta-T Probe has its limitations. It was designed for testing older systems, where compliance is more often less certain. The technology it tests is principally the 'through hole' chip types - these are the chips where their pins go through holes in the circuit board and are soldered from the other side. More modern systems use a mixture of through-hole and / or exclusively 'surface mounted' chips (SMT) which are glued and baked onto a circuit board, and have very high pin densities. We can provide additional clips for testing 28 and 32 pin SOIC package SMT memory chips which caters for some of the more recently designed systems. Again, if I have to put a figure on it, anything over 5 years old is usually no problem. Many manufacturers are still using through hole designs, or sometimes mixing the technology to include SMT RAMs, which is also fine, so 5 years+ is not a hard and fast rule. In a typical building we find we can test at least half of all the systems present, and the other half are either too simple to test (signed off on inspection) or very modern and easy to date test. Some PCBs are 'conformally coated', which means they have a thick lacquer coat sprayed over the chips to protect them from moisture or chemical corrosion. This is quite common in military and marine hardware. The standard test clips do not pierce the lacquer when applied to the chips, but 'paint cutter' versions are available as an add-on if this is a requirement. I know one of the US manufacturers is looking at the market requirement for addressing that issue. Access to the relevant chips is sometimes a problem. Most building related systems are simply put in a box, so accessing the chips is easy. In industrial environments most systems are rack-mounted, so backplane extender boards are required to allow access to chips on a CPU module. Daughterboards obscuring chips can also be dealt with by using header extender cables. To test a system, the Delta-T Probe requires access to (static) RAM and ROM external to the CPU chip. This is fine in almost all cases, but there are some 'mask programmable' CPUs (SCM) that contain internal RAM and ROM space, and usually some input/output capability. The memory in a mask-programmable CPU part is usually very limited, so they tend to be limited to use in 'smart sensor' applications (look inside a Microsoft mouse!) - they require external time reference for doing anything date-related, so in most cases they can quickly be ruled out by inspection as being totally risk-free. Some CPUs (such as the more modern Motorola 68000) cache, or pre-fetch, instructions that they don't even end up using. This makes code validation a little tricky, but our next release of the disassembler applications is designed to filter out cached instructions by maintaining its own program counter to follow the flow of execution. This is a minor limitation. Finally, the Delta-T Probe has an upper target clock speed limit of 20MHz. This requires much of the functionality within the Probe to operate at over 200Mhz to track it... hence the limitation. I have given you chapter and verse on the Delta-T Probe limitations. The Delta-T Probe was designed to tackle an impossibly big problem, and was engineered by a very skilled design team to deal with as large a slice of that problem as possible. The end result is a unique diagnostic tool which many organisations have found has made a significant contribution to the requirements of their embedded systems Y2K programmes.

Are there any post-year 2000 uses for the Delta-T Probe?

The Delta-T Probe was designed with a very specific purpose, to diagnose date dependency and capture code, so its use in its current form is limited outside the scope of Y2K issues in embedded systems. The Year 2000 problem is not over on 31 December, or even on 29 February 2000 as many believe – businesses have to cope with ongoing version control and configuration management of spare embedded systems parts and inventory items, which if left un-checked could re-introduce Y2K problems. The following observation was made by Harlan Smith, a well-known Y2K commentator: "It occurs to me that the Delta-T probe will have a life long past 2000-01-01, as there will always be embedded systems with misplaced source code that need evaluation. If nothing else, you might be able to non-intrusively detect what software version appears in the firmware, a question that otherwise might be difficult to answer. Configuration management efforts are never perfect and you could close the loop around that and make positive identification of the actual firmware loaded into PROM." The Delta-T Probe also has a use as an 'expert witness' tool, for determining the cause of failure of an embedded system. By examining the code, even retrospectively, it is possible to determine whether a system failed as a result of a Y2K problem, or whether the failure was due to other causes.

I follow your explanation of how the product works in broad terms, but to reassure some of the more sceptical members of my technical staff I require a more detailed description of how the Delta-T Probe really works.

I hope the following information meets your requirements. It is a non-technical explanation of what it does, annotated with some technical notes [in brackets] for people with a deeper understanding of embedded systems components… The DELTA-T Probe is a combination of diagnostic hardware, and specialist software running on a laptop computer, specifically designed to answer the two questions that determine Embedded Systems Y2K compliance. It will: (1) verify the presence of time and date information within an embedded system, and (2) capture the relevant code to allow the method of time and date comparison being used by the system to be examined. [The Delta-T Probe is in effect a memory-probe, capable of doing real time memory contents analysis, combined with a 200 channel 200MHz logic analyser circuit used to capture code information from a memory or RTC based trigger. The software takes the technical skill out of configuring the probe, and has a number of self-checks built in to make sure the right information is being captured] The DELTA-T Probe overcomes the problem of getting at code and time/date information by directly attaching to chips that hold the relevant information. Specialised test clips are used to attach to one or more of the system's memory (RAM), code (ROM) and Real Time Clock (RTC) chips. This is performed while the system is running. [The software has a huge database of chip datasheets, so that all an engineer needs to do is type in a chip identifier numbers, such as HY6264ALP-10, and the software then tells the operator which chips to attach to, and what type of clip to attach to them. It even gives the operator a picture of the type of chip the clip needs to be attached to. At the same time it uses the chip information to map the signal lines on the clip pins onto appropriate internal bus lines.] The RAM is scanned for clock information, and the presence of time and date is visually confirmed to the probe operator. This provides the answer to question (1). [The Probe creates a parallel memory map to the Embedded System's memory using a very fast dual-port RAM, which it then scans using its own processor for data that changes at a regular interval, say every second or every 60 seconds. It applies a number of different search criteria in order to identify possible clock information.] The DELTA-T Probe captures the program code data from the ROM chip as it is being read by the microprocessor to answer question (2). The probe uses access signals to the Real Time Clock chip or the clock information address in RAM as a 'trigger' to freeze the captured code information and isolate the correct section of code. [This invokes the logic analyser capture function, which 'clocks in' instructions as they are read from the ROM chips. It uses the RTC chip access or the memory clock image access to freeze the code information pre-and post access. The trigger is always set to the year date portion of the information, so we can see how a system uses the year date, and if it makes any erroneous comparisons. Disassemblers for most 8 and 16 bit CPU types are built-in. The code validation work (where it is required) is usually out-sourced.]

	Do you have a database of chip types that you know the probe to be able to connect to (ie. the database within your PC). Is this available to me? Also, is there a list of chipsets that you know you cannot connect to?
	Yes, it is part of the software. We do not give it away, as it has taken 1.5 man-years to put together, and covers more than 85% of commercially available ICs.

	What is the latest situation with respect to a database which lists known tested/problem firmware versions. You mentioned that this may become available in the near future. I would be grateful if you could keep me posted.
	Each code-validation house is building its own database. There are currently no plans for combining and releasing them into the public domain, as they represent intellectual property owned by the different code validation companies. There are also legal liability problems preventing one validation house from relying on validation carried out by competing companies.

I am keen to know if you have had any experience of a use for the Delta-T Probe in larger embedded systems/process control such as PLC'c or DCS's. What other types of systems have been tested with the probe other than HVAC and fire alarms? Where do I start? We have looked at Industrial DCS (pipeline monitoring), Building Management Systems, Security Systems, Industrial Process Controllers (PLC's) in both manufacturing (frozen food, ice cream, pharmaceutical packaging) and heavy industry (cargo moving), Lift controllers, specialised automation controllers (pick&place / flow solder machinery), military environmental testing systems (thermal shock), refrigeration systems (food retail), weighing and labelling equipment (food retail), and lots more... the list is growing all the time. In what circumstances would you connect the probe to the processor (in your demo you connected to RAM, ROM and RTC only)? If the circuitry is reported as being 'glitchy' - the Delta-T Probe cannot get accurate strobe timings due to poor system design. Extra control signals from the CPU are required to create 'clean' decoding signals. We haven't encountered situations where the vendor has claimed no RTC and then been disproved, although if this has occurred we'd be very interested in pursuing details. I'm still not convinced that the Probe can be used to 'prove or disprove' RTC usage in any case - from what I've seen and read, it would be very difficult to capture a once of access to the RTC (ie initialise current time / date on boot). If the ROM code is not exercised, the probe cannot possibly identify a date usage, and so the only way to 'prove' or otherwise would be to exhaustively 'exercise' the code which even if feasible (in most cases I believe it would not be) would not be practical. Is that an accurate assessment?

We also have not encountered a case where the vendor has claimed no RTC and subsequently disproved it. It is not a likely scenario. What we have found, however, are a great number of process control systems where the application code does not make use of date or time, yet the process controller does have a real-time-clock chip on-board. There is always a level of uncertainty in such cases as to whether the PLC might be using the date and time for timing functions 'at a lower level' (e.g. BIOS equivalent). The Delta-T Probe can be used to determine whether such activity is taking place. Another case is where a system has date and time evident, and it is not clear whether the year-date portion is being used for scheduling decisions. The Delta-T Probe can provide an immediate yes / no answer on this, which effectively determines whether the system is at risk of failure or not. You highlighted one example which we see quite frequently - a system only reads the RTC on start-up (a bit like a PC) and then maintains its own time and date while it is running. This is the most interesting case, as RTCs do not get Y2K wrong... the problem is always in the vendor code, and in such a case there is more likely to be a problem. If a system is maintaining a 'soft' clock (i.e. doing its own counting) the clock image can be found in memory, which the Delta-T Probe is designed to do. From there, both the code that maintains the clock, and the code that uses the clock, are captured for validation. This is also applicable to systems which receive clock information from elsewhere. Basically, if a system is doing any sort of date-related scheduling, the Delta-T Probe will detect it and capture the year-date related code. If the code is not being exercised on a regular (less than 65 seconds) basis then the system is not performing date-scheduling activity and is not directly at risk. A program may check several hundred times per second to see if something which only happens once a month needs to be scheduled... is not missed!

What information does this form of testing provide that thorough roll over testing does not?

None. If you can roll-over test then that is usually the easiest way of doing a test. DELTA-T testing fits in where (a) it is not known whether a system really uses the date and time at all, (b) a system uses date and time, but there is no way of setting it, or (c) changing the date may be too risky to do. It is an alternative test, which we use only when date roll-over is not possible by our own testing procedures.

How does the test predict the behaviour of the embedded software at the moment when it should update the century data, as this software routine will only be used, and therefore detected by the probe, when the century is updated, in which case you would have to perform roll over testing?

The Delta-T Probe captures how the code performs year-date comparisons in its main mode of operation regardless of what the date is set to, from which a programmer can see how the comparisons are made. So it is true that we cannot see the post-millennium code being executed, but we can see how the system checks for post-millennium dates in date comparisons it makes. Date comparisons being executed wrongly are the only real cause of millennium-related failures, so we can be pretty confident in the outcome from the code that we can see.

How can you test the embedded software for compliance without copying it all from the systems ROM?

Firstly, the probe can be used to detect that date and time is not present in a system, which is often difficult to prove in any other way. This does not require analysis of any ROM code. If ROM code is captured, it is only a small date-related portion. Examining an entire ROM takes many weeks to do, but examination of code captured by the DELTA-T Probe reveals whether a system vendor has taken post-millennium year-dates into account in date comparisons, or whether the comparisons will go wrong. This tells us whether the manufacturer has addressed the compliance issue, from which we can gain a very high degree of confidence that the system will or will not fail. In capturing regularly executed date-related code functions the Delta-T Probe is not guaranteeing that all the date-related functions have been covered by the test, but from what is captured it is possible to achieve a high degree of confidence. The Delta-T Probe usually captures all code samples that are responsible for scheduling activities within a control system, which are the ones most likely to go wrong.