PCMCIA-CAN Series 2 hanging

[Balaboum]

Hello,

We are using a PCMCIA-CAN series 2 board on two of our machines to get/send PDO data to CANOpen network of 6 nodes.

Built in 2007, Labview 2010 application (using CANOpen library 1.1.3 in combination with NI-CAN version 2.7.4) seemed to work OK mainly because we used it on short period of time (under 12h). To run more longer tests, we run now the machine over 24h continuously and this revealed a small bug. We investigated the case first with NI-SPY. It reported a communication lost problem with the board firmware and propose different solutions or checks (windows interrupts, application code, etc...). 

In our application, we have two loops to send and get PDOs on the network. We use function "Wait for PDO" to wait for a TPDO during a time slot and if not received we switch to wait for another one and so on. The other loop takes care of the RPDOs the same way. We have no more than 14 RPDOs or TPDOs for our network. We checked the code and even slow down the switching but bug is still appearing.  

We then run the application with other faster computer and made all software updates. Now, NI-TRACE reports (attached with this post) other errors but symptoms are still the same. Application is not able to communicate with board to get/send the PDOs values (or send/recaive SDOs) even the nodes on the CAN network are running OK. It seems that firmware implemented on board stopped functionning or hanged.

We contacted NI sales representative in order to get version 1.1.4 of the CANOpen library (support for LV 2010) but did not get any proposal at this time. We suppose that there is no firmware update as the product is now obsolete. Of course, recommandation was also to switch to new NI PCI board with new library for industrial communications but which is unforunately a little too large to be installed in our panel PC. Another manufacturer hardware will fit for sure...

Reading back my posts in 2007-2008, I remembered also that the PCMCIA boards had more limitations than others PCI boards for example. But, before changing hardware and rewriting the code, I want to give these boards (and all the work done in the past) a last chance.

Does one of you have suggestions or solutions for me ?

Thanks in advance.

[Jesse_D]

Hi Balaboum, 

I can't find anywhere we have run into that 12 hr problem before- it certainly isn't a known issue. It make me think there is something specific to your setup that is causing problems. Unfotunately we also haven't seen the errors you are hitting. If you can post your code I might be able to make more progress.

I do recommend an upgrade based on the above, sorry to hear that the NI PCI doesn't fit. Is the PCI slot a standard size? 

[Balaboum]

Hello,

Please find below two snapshots of the code and in particular the TPDO and RPDO loops I have in my main vi.

The TPDO loop receives an array of clusters containing each with  TPDO elements declared. Array contains 7 elements actually. TPDO time refresh time has been set to 200ms so that loop switches on a waiting time for each TPDO around 28ms. CAN network speed is 1Mbits/s and the 6 CAN nodes has been setup to send their TPDOs every 20 or 30ms. Only the one used are sent.

The vi "Get TPDO data" is implementing the TPDO wait function. If a new TPDO is available, its value is compared with the stored one (in array element). If new, the TPDO cluster is updated and put in a queue. Then, in this Producer/Consumer queue principle, the second loop dispatches the values in the variables.

The RPDO loop is based on same principle except that the 2 RPDOs are set every 50 ms. The application then uses SDO to send other commands to the nodes.

When our problem appears, we notice that values are not refreshed anymore on the front panel, even commands with SDO are no more possible. Spying the CAN network shows normal activity of the nodes.

First error message we got from NI-SPY was :

“Overflow in the lower level read queue of the CAN card (frames lost).  NI-CAN reads this queue at Windows interrupt-time. Solutions: Avoid tasks that generate excessive interrupts on your PC (mouse, ethernet, ...); Avoid running other
applications during your test (screen savers, MAX, ...); use Series 2 Filter Mode to filter incoming traffic; For CAN Objects (Frame API), increase read queue length or call Read more frequently”

So we checked our configuration and computer (scren saver, network connection, etc...). We also ran the application with another one, made updates... Now NI-TRACE reports the information contained in the files attached to my last post. Hard to find the bug if it occurs only after several hours of running time...

With these information, any suggestions or comments ?

Thanks.

[Jesse_D]

Hi Balaboum, 

I am not sure that is a bug at all - we have no known or reported issues. We do have some documents on the error. Try decreasing your RPDO refresh time. It is possible that you are reading slightly slower than the frames are coming it. You could try increasing the queue size for your CAN card. 

[Balaboum]

Hi Jesse,

I udpated CANOpen library to latest, i.e. 1.1.4.

Following your suggestions, I slow down the CAN network by sending a SYNC message every 15ms instead of 10ms. As well, I set the RPDO refresh time to 50ms and TPDO refresh time to 100ms. No effect, our application stopped communicating the same way as before however it failed with 0xBFF62028 error (see attached nitrace file)

I did not changed the queue sizes and will try that now. Queues for TPDO objects will be set to 10 and to 0 for RPDO objects. Is my reading of your document correct ?

Thanks.  

[Balaboum]

So, I increased the TPDO buffers size to 10. Well, things get worse... The application stops communicating after network initialisation (see nitrace file). Back to small buffer sizes (0) and everything is fine.

[ThomasV]

Hi,

Were you ever able to solve this or get more information about this?

I am seeing similar behaviour. We have a Windows 7 x64 PC with a NI PCI-8512 CAN-card, and using LabVIEW 2011 and the CanOpen Interface Toolkit 1.1.4, we are communicating with 11 motion controllers.

The setup works quite smoothly, everything is done using PDO and busload is about 25%.

After several hours, we see that the drives do not receive the PDO commands anymore. However there are no errors on the PDO Send VI's.

From some drives, we still receive PDO information from some drive not.

At this point, when we connect to the drive using a service console (RS-485), there is nothing wrong with the drives. It seems that the CAN-card just stopped

sending the PDO information to the drives.

Can anyone please advise?

[Jesse_D]

Hi Thomas, 

 I don't recognize the CanOpen Interface Toolkit 1.1.4, is it the same canopen library referenced above? If so there might be a bug and we would need to narrow down and reproduce it. If you have a service contract I recommend opening an SR at ni.com/support.

[Balaboum]

Hello,

We have a look to this with NI engineers but were not able to find a solution for it. Nothing seems to be wrong in the board firmware and I agree it may be difficult to reproduce the error. No one really had time to spend on long period of tests. End of story come with this simple comment from NI "... we have now this NI-XNET interfaces and our recommandation is to use them instead...". My boss and final users also asked me to move forward quickly.

Well, I had a look to the NI-XNET hardware and software. No doubt it may be a good product but I finally chose another manufacturer of industrial hardware/software solution as for me developpement efforts and results were the same. The final user had a doubt on the NI hardware and good experience the other manufacturer. The price difference finished to convinced my boss...

No one is perfect but final customers have always the last word...

Knowing that this bug exists on long period of time, I took the 3 pcmcia boards out of the field but still using them for diagnostics or other small CAN monitoring tasks in combination of the last NI-CanOpen library.

Regards,