Isco D-Series Pump Controller firmware bugs

[banksey255]

Teledyne Isco are not able to provide support for their D-Series pump controllers due to no longer having any employees with the required knowledge. Therefore, I am hoping that someone on this board has encountered and devised a solution to a firmware bug that I recently discovered. To help others who may encounter this equipment, I will also document a second firmware bug that I am aware of and the solution I have devised for it at the end of this post. To be clear, I remotely control these devices using their serial interface.

The firmware bug that I have discovered that I want a solution for to prevent it occurring is that Isco D-Series pump controllers will (seemingly) randomly not move the pump connected to connector A, despite being commanded a non-zero refill rate or independent constant flow rate and the G& command reporting it is not stopped (I have observed this problem with multiple pumps and controllers running revision M firmware). This is contradicted by the STATUS command that reports it is stopped. If the controller is again commanded to begin motion, it reports an error that it is already running. The only way I have found to recover is to send the STOP command. Currently I have formulated a detection approach based on the aforementioned contradiction between the G& and STATUS command responses, however what I want is a solution that prevents the problem from occurring in the first place. My efforts to that end have been unsuccessful as I have been unable to identify any patterns for what causes it to occur (I have experimented with different command sequences and timings, but so far identified nothing). My code that sends commands to the pump does so to control a process variable using a PID algorithm and frequently switches the controller between independent constant flow mode and refill mode based on the sign of the PID algorithm output and sets the corresponding flow or refill rate to the magnitude of the PID algorithm output, with the problem occurring very roughly at a rate of once per 24 hours. There is the possibility that the problem occurs in other modes of operation, which I haven't investigated yet.

The second firmware bug that I will document here to help others is that if a pump is stopped then the refill rate is set less than 0.00011 mL/min and then the pump is commanded to begin refilling, it will begin refilling at maximum speed then shortly thereafter report a position error and stop. In contrast, if the pump is refilling, then the refill rate can be set less than 0.00011 mL/min without problem. Therefore, the solution to prevent this problem occurring is to simply never start refilling with the refill rate set less than 0.00011 mL/min, and then adjust the refill rate to whatever value is desired once refilling has commenced.

[banksey255]

I suppose I may be out of luck.

We continued to investigate the problem, which had been occurring on their firmware revision M, and we subsequently found also occurred on their "demo 79" revision firmware, which apparently was the last firmware released for that generation controller that they discontinued in 2013. We then tried an Isco D-series 4th generation controller running version 1.10 firmware and found it also had the same problem, and whilst they discontinued the 4th generation model in 2022, our hope is that it is recent enough that they can identify the cause and a solution.

If it's helpful to anyone, we also discovered a new firmware problem on their 4th generation controller where, intermittently, when pump B is running and switched from INDEPENDENT CONST FLOW to REFILL mode, or vice versa, it will continue to maintain the flow from the original mode, despite all commands reporting that the pump is operating in the new mode. The only way we've found to recover is to send the stop command.

[banksey255]

Here is a final summary of the Isco D-series pump controller problems we discovered when using the serial interface. If anyone ever discovers additional problems, please post to this thread as I want to know. Also, if anyone ever finds a solution to either problems (2) and/or (3) that prevents them occurring in the first place, in contrast to my solutions that detect and recover from them once they occur, I would very much like to know also. The problems are:

1. A pump will begin refilling at the maximum rate and shortly thereafter stop reporting a position error if started with a refill rate less than 0.00011 mL/min (110 nL/min). This problem has been observed in 4^th generation and older controllers. My solution is to simply never start refilling with a rate less than this, and then once refilling, set to whatever rate is desired.
2. Intermittently, pump A will not flow in REFILL or INDEPENDENT CONST FLOW mode. Other symptoms are a contradiction between STATUS reporting stopped despite G& reporting not stopped and the pump rejecting commands to run reporting that it is already running. The only way to recover is to stop the pump. This problem has been observed in 4^th generation and older controllers. My solution is if the flow is zero and the volume doesn't change for two successive G& commands, then simply stop then start the pump (although I suspect this may result in some false positives, so a more reliable test may be to instead see if there is a STATUS command contradiction, which I don't do simply as the time to execute the STATUS command is comparable to the combined time for commanding the pump to stop then start).
3. Intermittently, pumps B, C, and D do not switch from REFILL to INDEPENDENT CONST FLOW mode, or vice versa, and instead maintain flow in the original mode, despite the G&, MODE, and STATUS commands all reporting operation in the new mode. The only way to recover is to stop the pump. This problem has been observed in 4^th generation controllers. My solution is to compare the sign of the reported flow rate to that commanded, and if there is a disagreement, then simply stop then start the pump.

I suspect both problems (2) and (3) are associated with switching between REFILL and INDEPENDENT CONST FLOW modes. The very limited support I could obtain from Isco was a suspicion that it was associated with my code switching the modes as fast the Isco controller would allow and that a solution may be to increase the interval between commands, however my limited attempts to prove this failed (I inserted a delay between stopping one mode and starting the next that automatically increased by 25ms each time a problem was detected, but problems still occurred even once this delay had reach 150ms, which seemed already excessive given that it takes 25ms for it to respond to commands at its fastest baud rate, so I ended the investigation). If anyone wants to try to replicate either of these two problems, a method I used was to simply sinusoidally oscillate the flow rate with a one minute period and small 0.025 mL/min amplitude, and then simply log the data and look for a half period where the flow rate doesn't respond (in my experience this occurred within three hours of starting).

For reference, the older generation (three pump) controllers I ran tests with were running either revision M or Demo 79 firmware (which Isco informed me was the latest available for that generation), and the 4th generation (four pump) controllers were running version 1.10 firmware.