FP CPU loses communication with modules

[wd8ivl]

My FP system consists of a FP-2020, a DI-301, a DO-403, 2 x RLY-420's, a AI-110, and 2 x TB-10's with a mix of DO, AI & AO.

The program that controls the I/O lives in the attached PC and communicates over a dedicated the ethernet port to the FP-2020 (not the company network).

Occasionally, some of the modules will lose communications with the FP-2020.  I will get bad data or lose ouput control, and will then find the "Ready" light of the module will be blinking.

I had thought the problems were caused by intermittent connections between the modules, but wiggleing them doesn't solve it.

What does solve it is cycling power on the processor. (I don't have to do anything to the FP program running in the PC)

So the questions:

1. Is it possible for me to test for this intermittent condition? (it's not always apparent the data is erroneous)

2. Is it possible for me to send a reset to the FP-2020 that will reboot it when this happens?

Thanks,

Mike   

[Khalid]

Hi Mike,

If you find the commnication loss to be very frequent, you may want to bring this to NI's attention; and maybe get the Controller/modules checked out for any defects.  I personally would not try to workaround such hardware issues in software. 

Sorry for not answering your direct questions.

-Khalid

[wd8ivl]

Khalid,

It doesn't happen often (maybe once a week per installation), but it causes over an hour of lost production every time it happens, (trouble call to Maintenace, Shutdown & Restart of the system, etc) which translates to over $20,000 in product that wasn't made available for sale during that hour..  

We have been having issues with Fieldpoint modules since we started using them almost three years ago.  Intermittent hardware connections, AO modules received DOA, lost communications, bad data, etc...All of my emails and phone calls to Austin have fallen of deaf ears.   Everything I've solved, I've done so myself.

I asked this same question of Austin on Friday, and they completely misunderstood it...they suggested a work-around for checking communications between the PC & CPU...which is not the problem.

That's why I've posted the question here - there seams to be a much better knowledge pool in the LV community than in Austin.

So, is there a "quality" bit I can read from each of the modules to tell me if the data is good? (MAX is able to tell if the data is good...can I read what MAX reads? If so, how?)

Thanks,

Mike

[Otis]

Hi Mike,

I'm sorry to hear that you've run into so many problems in the past.  Hopefully we can try to figure out what is going on here.  You mentioned in the first post that there appears to be a communication problem with the cFP-2020 that you have.  In the first post you mentioned that power cycling the module appears to fix it, so I doubt its actually a problem with the connections or how they are made.  That means we should probably focus on the software and how it is interacting with the hardware.

You mentioned that you would like some way of confirming that the data is coming from a working module (a "quality" bit I can read).  One way that you could do this is read the Ready LED on the Controller and if that toggles state then it means that something erroneous is occurring.

You might also want to look in the cFP-20xx User Manual , particularly in Appendix A where it mentions what the flashing LED light indicates.  As you can see, different numbers of flashes correspond to different issues.

You mentioned that you contacted Tech Support on Friday and that they seem to have misdiagnosed the issue.  I would highly recommend continuing working with them and explaining to them why it is not a network issue (which shouldn't take long seeing as you are able to communicate with the module).  National Instruments' Applications Engineers pride themselves on being able to help our customers, and if they gave you information that did not help the first time, then they can use the knowledge they already have of your system to help you find the best solution possible.

Ultimately, since we are dealing with computers there is a logical reason as to why you are getting this error.  While it may be difficult to diagnose because of the long mean-time between failures, there is a solution or a workaround out there that can get your system back up and running.

Regards,

[wd8ivl]

Otis,

Thanks for the responce. 

When I went to the installation on the shop floor, the Status LED was off (not blinking).  All of the "Ready" lights on all of the modules were on steady, except for the "Ready" light on module #7, the AI-110.  That "Ready" light was blinking eratically (without a blink pattern)  (The Modules "Power" LED was also on steady).  Further, all of the other modules were working fine - being controlled and read by the Fieldpoint program that was running in the PC.  So all I did was cycle power to the Fieldpoint processor by unplugging an plugging in the 120V source to the FP-PS-4, and when the system came back up, the "Ready" light on the AI-110 came back on steady and I I started getting good data from it in the Fieldpoint program on the PC.  I never stopped or started the PC program during this process, never unplugged the TCP/IP cable, never unplugged/plugged the module, never wiggeled the rack, etc.

So I don't think reading the Status LED on the FP-2010 will do me any good.

So I ask again - is there some type of "data quality" bit that can be read from each module?  In MAX->My System->Data Neighborhood->FieldPoint Items->Fieldpoint IO->FP-AI-110 @7" there is a column called "Status".  So MAX can see if the data is good.  How can I read those Status bits in my Fieldpoint program running in my PC?

Mike  

[Aaron_G]

Mike,

The status in MAX is intrinsically part of the data when you are using the FieldPoint VIs in LabVIEW, it is output in the error cluster. It is not a separate channel that you can read.

A blinking Ready LED indicates that the FP-AI-110 is powering up, completing at least part of its internal diagnostics, and signaling to the network module that it is ready for configuration. It will then attempt to complete its onboard diagnostics when it receives configuration information from the controller. If it fails, it will reset itself and start over, thus leading to the blinking LED. Some possible reasons are that the FP-AI-110 may not be fully seated in its terminal base, or that the FP-AI-110 may have been damaged (either physical from excess vibrations) or electrical (how is the common to the module hooked up?).

Regards.

Aaron

[wd8ivl]

Otis,

In this case, the AI-110 had been working - giving good data for several weeks without a problem.  It just started giving bogus data and it's "Ready" light was blinking.

It's input signals come from seven Crompron transducers.  (one channel of the AI-110 is unsused)  Three are for current measurement:0-5A AC IN/0-10VDC Out.  Three are for voltage measurement:0-120VAC IN/0-10VDC Out. One is for 3 Phase Power measurement, which has two 0-5A Current sources and two 0-120V AV voltage sources as inputs and a 0-10V DV output.  That makes all of the transducers 0-10DVC outputs isolated from eachother (until they get to the AI-110).  All of the 0-10V signals are in shieled cable, and the shields are earthed at the AI-110 end (but not to the AI-110).

Like I said, I tried wiggling the Fieldpoint rack (because we've had problems with intermittent connections between the bases in the past) and it didn't solve the problem.  SO that's when I cycled power to the rack and the module re-booted and started giving good data.

The system is used to test large (10-50KW) varaible voltage, variable current power sources that are capable of operating from a range of input voltages.  The Current and Power transducers are used to actually take measurements of the unit-under-test.  The voltage transducers are used as feedback verification to make sure we preset the correct input voltage before we apply it to the unit under test.  A catastrophic failure would occure if we applied 575V to the input of one of these power sources that was set to operate from 230V.

It would not be as big of an issue if the bogus data was "0"..but it was in the area of 2800 (on a 0-4095 scale) which represents about 400V in my system.

You said "The status in MAX is intrinsically part of the data when you are using the FieldPoint VIs in LabVIEW, it is output in the error cluster. It is not a separate channel that you can read."   So is this the same "Status" that is in the error cluster I connect to the output of my "FP Read.vi", and will it indicate a false if the data is bogus?  (keeping in mind that my FP Read.vi was still executing, just getting bogus data)

Thanks,

Mike

[Aaron_G]

Mike,

The status boolean may or may not be set depending upon the specific error. In most cases, the boolean will be false since the errors are not considered fatal (and are thus considered warnings). You will want to look at the code value in the error cluster. FieldPoint warnings will be in the range of 32000-34000 if I recall correctly.

One reason that Ready LED will flash is if the microcontroller (on the isolated backplane side of the board) loses communication through the optocouplers to the ADC chip. If it does not get a response from the ADC, it resets itself, powering down and up the non-isolated ADC side of the board. You mention that the Crompron cables are shielded but the shield is not wired to the Com of the FP-AI-110. How are the FP-AI-110 Com terminals wired? Is the backplane common grounded in any way (e.g. FP-2020 C terminal)? It may be that the Crompron signal commons are pulling the FP-AI-110 non-isolated common more than 2300 Vrms transient, 250 Vrms (sustained) away from the backplane common (as determined via the FP-2020 C terminals).

Regards,
Aaron

[wd8ivl]

Aaron,

The transducers are wired to the AI-110 using two conductor twisted - shielded cable.  From each transducer, the transducer's (+) output connects to the AI-110's "Vin" connection for the respective channel, and the transducer's (-) output connects to the AI-110's "C" for the respective channel.  The shield is "earthed" to the metal panel near the Fieldpoint base.  All of the "C" termianls for all of the bases are connected together, then connected to the "C" terminal of the FP-2010 and the "C" terminal of the FP-PS-4.  There is no external connection between the FP-PS-4's "C" terminal and it's "earth" terminal, and they do not seamt to be connected internally.  All of the bases are powered by an Idec 24VDC, 5.2A power supply.  The FP-2010 is powered by a FP-PS-4.  The FP-PS-4's ground terminal is "earthed".  The +24V connections of the Idec and the PS-4 ARE NOT tied together.  All of the transducers have their own internal power supplies and are powered from the 120VAC system, and all of thier (-) outputs are isolated from earth (and obviously from each other) 

I did not see anything in the manuals for the bases, the PS-4, or the FP-2010 about earthing the "C" terminals. Further, because all of the transducer wiring is physically isolated from the control & power wiring, I would expect that anything induced on it would be induced on it's shield and drained off to earth befor reaching the module. 

When the program started giving the bogus data, the system was not being used at the time - it had been sitting idle for a time, so there would have been nothing to cause excessive voltage transients at the time.

So, if you coud dig out some more information on those warnings in the 32000 to 34000 range (like, exectly where I's look for them), that may help.

Thanks,

Mike 

[Aaron_G]

Mike,

By connecting all of the C or Com terminals from one terminal base to the next, you bypass the isolation between those two IO modules. If you connect the C from the network module to the C/Com of a terminal base, you bypass the isolation entirely. The only IO module that you listed that requires external power (e.g. Power wired to the Vsup or V terminals is the FP-DO-403). The rest of the modules draw their power entirely from the backplane and do not need to have their V/C terminals wired to a power supply. You should  disconnect the C terminals of the bases from each other. However, there may be some exceptions in your system, depending upon how some of the sensors are wired. For example, if you had any loop power current sensors, you might need to wire the V/C terminals of an AI module to make powering the sensor easier. From your description, it sounds like you probably can afford to isolate most or all of the modules. Of course, without a wiring diagram, I can not make a definitive call on that.

Isolation is probably one of the least well understood topics with FieldPoint. I recommend that you search for some of my previous posts and KnowledgeBase articles.

The error and warning codes are documented in the online help (Start>>All Programs>>National Instruments>>FieldPoint 4.X>>FieldPoint Help).

Regards,

Aaron