Re : Increase the speed reader

[ThomasAu]

Hello,

Actually, I use a cRIO for a control from captors and a file.

The FPGA is use for control (PID, conversion and emission).

My problem is the time to read the file of value. My VI use 600ms for read the good line. If I reduce to 72ms, that's perfect. But if i reduce to 100ms that's very good too.

Before the control, I make a TDMS file with all values usefull for the control (like command). An other format of file is better ?

What is the better approache for read this file fastly ?

Unfortunately, i cannot post all my code. In the parts not posted, I develop the cluster and save informations in shared variables.

[SLee2004 · Accepted solution]

If the file is not too big, you should read the file into memory first then access the data points instead of trying to access the file line by line. If the file is too big, you can use a queue structure to read portions of the file into memory (producer & consumer structure). One loop will control how much data is loaded from file into the queue and the other loop can dequeue the data and sent to your control device.

[ThomasAu]

Thank you for your answer. It's a good idea.

The file is very big (2 hours with a step of one hundred miliseconds from the begin to 10 minutes. After, it's a step of a one second)

I think I will make my VI like that :

- The productor loads 100 lines from the file to the queue. When the number of elements decreases in the queue under a level, the productor loads a package of lines (for example 50).

- When the main loop (the cusmer) wants read the queue, it searchs the line with the same time like the time of application. When it find, it use the selected line for save parameters.

But I'm not sure that the cusumer loop will have the same frequency all time.

It's a good trail or you have a good idea ?

[Yamaeda]

How big is big? Several gigabytes? Your idea should work, but read a million lines or something. 🙂

/Y

[SLee2004]

I would read in more than 100 lines at a time (possibly 10,000 lines), but just pay attention to how much memory your application is using (http://zone.ni.com/reference/en-XX/help/371361H-01/lvconcepts/vi_memory_usage/). Your consumer loop time depends on how fast you want to send the data out to your physical DAQ and how fast your DAQ responds.

[ThomasAu]

The idea for use queue is very fast. But a new problem begun.

In my file (7Mo of data is not very big after consideration) regroup two files. The two files have a variable timing step.

The cRIO need use two queues and two loops for control the number of eache queue. In first, I have use normal queues, but they blocked execution. With RT queues I haven't this problem. An other problem is the time for execute. It's faste alone, but when all control is use, the CPU is overbooked (before the control use 70-80% if the CPU).

I will continue to seek a solution

[ThomasAu]

I use not a DAQ. I use a cRIO 9075.

I had this problem of memory wich a previous solution. It's why I want to read a file by package of line and not all file in one time.

[Paolo_P]

The fastest way is using Binary files as said in this link:

http://www.ni.com/white-paper/9630/en

Hope this can help you.

[ThomasAu]

Hello

I have progress significantly in my problem.

I have use the structure of productor/consumer. The problem of this structure is the time CPU which is important.

For reduce this time I have build a new file. I use TDMS format (not binary because I have not the time for make a test). The TDMS have not two groups (my control need two file), but one. Now I need just one queue with a cluster where else value are named (this very useful). I reduce the occaption of the CPU by decrease the number of reading in TDMS and writing in my queue.

My second diffculty was the variable step time. I have build my TDMS with a constant step. Now, I can regroup my two file and I simplified the reading of the queue by the consumer (I read the first value of my queue if the time of my line is lower I unstack the value. If the next time value is upper, I take this value and I unstack.

Now it's more efficient and more simple. The productor loop is use every 2,5second and the consumer is use every 100miliseconds. But I have an important jitter.

Thank you for your help.

Thomas