Thanks a lot for looking into this issue, thanks again for offering such comprehensive help!
Regarding the specific remarks and questions in your post:
Since rtcp starts with rebooting the controller and obviously sets it into a special mode then wherein the customer's software would not run (status LED flashing three times in repetition),
I doubt that we could face some sort of starved communication on the controller's side.
The RealTime Execution Trace toolkit is something that I do know by name and I think I know vaguely about its capabilities, but that's it. We didn't decide to use it yet because
we did not yet have RT-Troubles that we thought the Trace toolkit could help us out with. Besides, for continuous use, there is a price tag attached. So we would need assistance in
how to use the RT Execution Trace toolkit to help us solve this problem.
The RealTime system manager however we used a bit to look at CPU and memory usage from time to time, but not that much lately because we don't have any issues with memory
whatsoever and the CPU usage display is not that usable for a cFP device (being at 100% all the time for cFP). Another point is that the system manager does not respond well,
especially with CPU monitoring in effect, if 'connected" to a cFP2100. It is very slow then (this is very much different with cFP2200, which we only begin to use now). And again:
You would have to instruct us in how the system manager could help us here.
We would gladly accept your offer to provide us with a service request number so that we then could send you the image we're transferring. However: I tried to mail zipped data
to NI mail adresses not too long ago, and I had to learn that the NI mail server rejected these mails, stating that it was an illegal type of attachment. Disguising the .zip file as
a .txt file and then as a .jpg file did not help. I ended up having to send that data to your colleague in germany via a CD in an envelope... Maybe it wasn't the type of attachment,
maybe it was its size (3.5 MB)? Nobody was able to tell me what the reason for these rejections was. Nevertheless, I hope sending you the image will work better.
I will add here as an attachment the messages we got from rtcp when such an interruption recently took place.
In the last days, we tried to better reproduce the problem. This lead us to the following observation: Some kind of 'catalyst' seems to be present so that the problem occurs. For instance,
it seems to happen more often when the image is not read from the PC's hard drive, but from a network drive. But this is not a provision, it did happen when the image came from the
local hard disk too.
We have a PC that has a high interruption rate, then we have another one with a very low one (despite the fact that the latter one also gets the image from a network drive -- so this
is far from being a guarantee that an interruption will take place).
In at least one case observed, the PC with the high interruption rate came up with a message from a firewall that was locally installed right before the interruption took place.
Unfortunately, for a variety of reasons, we were unable yet to get the following setup: A high-interruption-rate PC WITH LabVIEW installed... The last days, the PCs with the problem
showing were of course NOT ones with LabVIEW installed. Murphys law!
(One might think that this is more than a pure coincidence -- but on the other hand, we did experience at least one confirmed interruption when a PC was used having LabVIEW
Another thing, however, we could make out clearly: This phenomenon takes place with older FieldPoint Software on the cFP ('older' meaning the LV 8.5 stuff -- 6.0 if I'm not mistaking)
and with brand new FieldPoint Software also (6.0.3). We could only reproduce the problem with cFP2100 controllers yet, but not with the newer cFP2200 -- mainly because we did not
have the opportunity to try out the cFP2200 seriously (we only have one cFP2200 which we got just recently, and this unit has been needed for other testing purposes). We do not have
other cFPs at our site.
Lets assume that we will manage to set up a PC that 'reproduces well' AND has LabVIEW installed: we would need some hints what exactly we should look for then in rtcp's
By the way -- having successfully recompiled the rtcp stuff now using LabVIEW 8.6: Would it be a good idea to generally work with an .exe from this compilation (at our site and
at our customer's sites as well) rather then with the much older rtcp.exe that this package comes with? Would that be more reliable to start with, regarding the fact that it has to run
in an all-8.6-environment?
Doing so would at least no longer make it necessary to provide our customers with the LabVIEW 7.1.1 runtime (which we must add as an "additional installer" in order that they
also can run rtcp).
Thank you very much!
With kind regards
Another thing the RT gang should look at is the age of the VIs. I noticed that right after installing the replication toolkit and placing "Find All Targets" on a block diagram, there was load warnings like replacing constants with hidden controls. I'm also fairly sure the replication toolkit uses the FTP toolkit. The FTP toolkit hasn't recieved any attention since well before LV 7.1 because when using those it inserts code around file I/O to preserve compatability.
Both these toolkits should probably be refactored under lv8.6 to bring them up to date. The FTP toolkit could use some improvements as well...such as exposing timeouts and adding in support for secure FTP methods like SSL (I get beat up for not being able to support secure transfers).
Hello MKr and Jason,
I have continued to look into this issue further and I found out that there is a new version of the System Replication Toolkit that just became available on our website. There have been many bug fixes implemented into this new version and could very well give you better results. Could you give this new version a try and let us know if you are still running into the same issues? I hope this helps! Have a great day!
Thanks a lot for this information. Of course we will give the new version a try -- and surely more than one.
Will be back with results as soon as we have some.
Having taken a first glance at the new package, I noticed that the file rtcp.exe is exactly identical to the old one.
I guess that this is a mistake, or isn't it?
Unfortunately, rtcp.exe in the version8 replication package on the NI website is still the very same as in the old one (please see my last post).
In the meantime, I tried to generate an rtcp.exe based on the VIs in the new package myself using LabVIEW 8.5. This failed, because it did not find a lot of necessary TCP... VIs which are apparently part of the internet toolkit which we do not own.
However, we need a version8 rtcp.exe compatible with the LV 8.5 runtime, because we would have to deploy it to customers of ours which have an 8.5 runtime (and not an 8.6 one).
Nevertheless, I tried to generate our own version8 rtcp.exe using LV 8.6 also. Much to my surprise, this worked. Maybe it's because we once had the internet toolkit installed for the 30 days test period in LV 8.6 (but which expired quite some time ago)??
So, we are able to finally start testing the version8 rtcp.exe now at our site. But if that turns out to be successful, we would finally need an rtcp.exe compatible with a LV 8.5 runtime for some of our customers.
Maybe you could see to it that the version8 replication package on your website now gets a version8 rtcp.exe instead of the version7 one it currently still has?
Thank you and best regards
We now tried out the new version8 rtcp enough so that I can post the results.
Please bear in mind that we had to compile our own rtcp.exe using LabVIEW 8.6 because the file rtcp.exe in the new version8 replication package is the very same as in the version7 package (updating the .exe seems having been forgotten for the version8 package). So, it is possible that we did something wrong compiling the .exe.
What we tried:
We tried to transfer an image to a cFP2100 which we read from a cFP2100 some time ago using the 'old' (version7) rtcp.exe. We did it this way mainly because this is exactly our field scenario. Any rtcp we distribute to our customers must of course be able to do exactly that flawlessly. In particular, we won't be able to re-read older images using version8 rtcp (e.g. simply because we might not have that controller anymore at our site). It must be possible to use images created by the 'old' rtcp. But in my understanding of the basic procedure, this in itself won't be a problem (or will it?).
The good results:
We did NEVER see an interrupted transfer like we did so often with the 'old' rtcp!
The not-so-good results:
We were NEVER able even once to transfer an image successfully to the controller (although rtcp often reported a successful transfer at the end).
Sometimes, there is no error reported by rtcp ("Image applied successfully").
Sometimes, there is a "Reboot timeout"
Sometimes, when starting the transfer, this message appears:
TCP Open Connection in NI_Real-Time Libraries.
lvlib:FPC open connection.vi->Get Target
Info (IP).vi->Set Target Image.vi->RTcopy.vi->replication.vi
...and that's all what happens in that case (no transfer at all).
Most of the time, the controller won't run our program at all after rebooting (which must sometimes be initiated manually).
Sometimes -- and this is the best result we got -- our program runs, but there is something wrong accessing the network variables we use for comunication with a connected host: These variables are hosted on that very same controller, and accessing them in our non-time-critical loop won't work (i.e. yields an error in every cycle). Behavior is just like variable deployment to that target didn't take place (but the image used was taken from a controller which had the variables successfully deployed to, of course).
Then, when using the 'old' rtcp.exe for a change to transfer the very same image to the very same controller, everything is running flawlessly again -- provided that there is no transfer interruption (which is the -- only! -- problem with the 'old' rtcp, lest we forget).
After one of our many trials, we transferred the image back from the controller to disk using the 'old' rtcp (!). Then, we compared the original image with the downloaded uploaded one using "total commander". The result is attached.
Especially the fact that some file sizes differ slightly (binary/text transfer handling difficulties??) and the fact that variable access never works (some files/filetypes not handled properly??) leads to suspect that the new rtcp now has file(type) handling problems that the old one did not have.
(Forgive me for maybe thinking too simple, but: wouldn't it be a good idea to mix the new rtcp's file transferral capabilities with the old rtcp's general file and file type handling?)
Please let us know if you should need more specific information to resolve this issue.
I apologize that I have not gotten back to you recently. For some reason there is no service request for this discussion and I did not receive email updates that you had posted. I am going to look into this issue for you to determine the best method to continue. Thank you very much for your patience on this issue. I will make sure to get back to you with an update within 1 business day. If you have any new information since your last post please just let me know.
You are correct that the rtcp.exe is the old version still even with the new System Replication Toolkit. This is really meant only as an example, not a solution. However, I would like to try to reproduce what you are seeing on my end. Can send me the image you are using? Once I am able to reproduce it then I can work with R&D to determine if there is a workaround for you to get your system up and running. If you are unable to post the image to this forum please let me know and we can work to set up a service request for you so we can use either email or FTP to transfer the file. I look forward to hearing from you!
Thanks for getting back. Regarding the 'old' rtcp in the 'new' replication toolkit: People without the internet toolkit won't be able to use the toolkit otherwise (please see my post from November 17 and please correct me if I'm wrong). At least for them, an rtcp.exe would be more thant just an example.
And in fact, in the past, we used just the file rtcp.exe, and that's it. And I think we would like to continue that way in the future. rtcp.exe (if working) is a bit more than just another example. In my opinion, it's pretty much exactly what you need for RT replication. There's not one function too much (which is great for our customers to use it). Additionaly, we like the idea to have that functionality NOT integrated in our software, for a lot of reasons (e.g. being able to update rtcp -- as we are about to do here! -- without having to touch our software at all). Besides: NI personnel also recommended to us to use rtcp.exe. And finally: Why reinvent the wheel?
And clearly, technically, not touching rtcp.exe in the new package must have been in error, that I'm pretty sure of.
Regarding the image we're using: I still would appreciate to receive a service request number so that I could provide you with our image.
Currently, we're just running a few more tests the results of which could be of interest. This time, we're transferring an image to a cFP 2200. One thing seems already to be pretty obvious: Rebooting the controller after completion of the transfer hardly ever works (this was already true when transferring to the cFP 2100, please see my post from November 20). The funny thing is, that even a manual reboot via MAX or by pressing the reset button won't work. It is necessary then to power down the controller.
But again: These tests are not completed yet. It might take us appx. one more day, maybe two. I'll post the results rightaway.
Thank you and best regards