tdms performance

[xl600]

I am trying to compare TDMS write performance between similar systems to determine if I stand a chance of being able to capture data from my cameras at anywhere near the speed needed (240fps at 640x480 is my basic target).

The two systems are:

Windows 7 laptop with LV2014, 16GB of memory, Core I7 2.5GHZ, 500GB ST500 hard drive (SCSI)

PXIe-8135 RT (PharLap 13.1), 4GB memory, Core I7 2.8GHz, 500GB hard drive (SCSI)

I have a VI which simulates storing video frame data to a TDMS file (Raw U8) (Attached).  When this is run on the laptop, with all of the heavy applications like antivirus, on-access scanning, etc.. I get about twice the storage performance that I get when the same VI is run on the PXIe-8135.  Given their similar hardware specifications, that has taken me by surprise and I fear that the entire system may be non-viable.  I would have expected better performance from the PXIe-8135 in fact (Even with the lower total memory given that this test isn't even scratching the surface of the available memory).

So, I suspect I am missing something fundamental about how to get higher performance disk I/O out of LabView RT and would like to hear any suggestions that may be available.  Or does labView RT simply have this poor of performance compared to a Windows 7 installation?

Thanks,

Robert

[mikeporter]

Part of the issue is that RT systems are built to make the time required to perform key tasks very deterministic. For a variety of reasons, File IO is not very deterministic.

My advice would be to consider an approach that is sort of a rift on the concept of the producer/consumer pattern, but in this case the producer is on the RT system and the consumer (the file IO) is on another computer -- perhaps even a PC. Comm between them is via something fast like perhaps UDP (if you can afford the occasional lost packet).

Mike...

[deppSu]

On windows platform, TDMS has leverage some windows system owned IO technology to improve the performance. Thus, It's true that the performance on RT is not as good as it on windows for TDMS.

[christian_w]

Use the NI_MinimumBufferSize property to reduce file access and fragmented tdms file when writing different channels.

[mikeporter]

Just haad another unconnected thought. If the data is video, why are you saving it in a TDMS file. Wouldn't it be more efficient to save video data as video?

Mike...

[xl600]

It sounds like I will have to throw out the use of LabView RT as a result.  There simply is no way to get enough network performance to handle the data load of dual Camera Link cameras running at high speed.

Can an SSD be fitted to a PXIe-8135?

Can LabView RT be dual booted with LabView on an 8135RT?

Perhaps those are questions for a different forum...

Thanks,

Robert

[xl600]

The video data is raw data coming from two CameraLink cameras on two PXIe-7966 FPGA boards with NI1483 front ends.  Conversion of the raw frame data to video formats would add a lot of overhead which I can't afford.  So, I need to push all the raw data coming off the FPGAs into TDMS files and process into video (And perform a myriad of other processing) in non-realtime.

Conversion of the raw data to video in the FPGA has turned out not to be possible because of the low level of IMAQ support and vision support for the NI1483.  So, I've moved to using the FPGAs simply to buffer frames (Using the entire DRAM), time tag frames (Embedded time stamping to 100ns resolution), and to push (DMA) into the 8135's main memory (Which is pretty small at 4GB).

[xl600]

Setting NI_MinimumBufferSize in this application appears to have no significant effect (I've tried it).  Items being written are large frames of raw video data from two cameras.  If there's an optimal way to organize frame data into a TDMS file for the highest write performance, I'd love to hear some suggestions.  I think the basic issue is probably just plain old I/O at the OS driver level though (As mentioned in an earlier response).

Thanks,

Robert

[xl600]

Question for Mike Porter:

I took your suggestion and reworked my experiment to store to avi2 using Y800 uncompressed video.  There is a marked improvement in disk storage speed (About 30%) with a corresponding increase in CPU% (About 15% more on one processor).  I attribute the CPU% increase to the need to perform the IMAQ image conversion from the raw data coming from my FPGA. 

This is a very interesting outcome.  I will experiment more with AVI2 files and perhaps I can coax more out of it.  I thought TDMS was supposed to be the highest performance mechanism so i'm wondering what the underlying difference is for sequential data?

Thanks,

Robert

[crossrulz]

---

xl600 wrote: I thought TDMS was supposed to be the highest performance mechanism

---

It all depends on what you are doing.  Writing a raw binary file will be faster than TDMS.  TDMS does have some extra overhead to maintain certain things and store metadata.