Memory Problem - Split File

[dan07]

Altenbach

I tried what you said, to work from the beginning until the end as string, not convert the data to DBL, and it worked!!! The code is much more fast (3-fold more or less).

If I would like to upgrade this VI and start to plot in a graph the data loaded, I will need to convert the string into array and in the sequence I will be able to plot in a Graph, correct? Is there a way to plot the string values direct on a graph with no conversion of string into array (Spreadsheet string to array)?  Is it possible to build a waveform with the string data, without conversion?

Thanks for help

Dan07

[altenbach]

---

dan07 wrote:

If I would like to upgrade this VI and start to plot in a graph the data loaded, I will need to convert the string into array and in the sequence I will be able to plot in a Graph, correct? Is there a way to plot the string values direct on a graph with no conversion of string into array (Spreadsheet string to array)?  Is it possible to build a waveform with the string data, without conversion?

---

My guess is that you should be more that 3x faster. Can you show us your code? Remember, to lower the memory footprint you can also do things in chunks, e.g. 10% at a time.

No, you cannot plot "strings", you need to parse them into a numeric datatype.

However, remember that most likely you don't have a monitor with 9 Million pixels across, so you only need to plot a very small percentage of all points.

You might also want to study Managing Large Data Sets in LabVIEW.

[TCPlomp]

@TonP wrote:

 Is the amount of bytes per line always the same?

---

dan07 wrote:

TonP

The files loaded don't have always the same size.

Dan07

---

You gave an answer to the wrong question.

However when I look at your code, it shows 8 bytes of visible ASCII data per data-point, together with a two byte CRLF combo for the linefeed (check that), I get 10 bytes per line.

You could start and stop reading the data easily at such a point and direcly write it to the target file.

Ton

[dan07]

Altenbach

I examined carefully the segmented file generated and it was not correct.  I don't know what it's happening since that the number of values in the Data.txt (the file that I am using to test the VI) is 65536, and when I load the file in my VI, the number of values its much much bigger (the String Lenght VI its taking into consideration the whole number of characters). The problem is to convert the string into a sequence of values, but without change it to DBL, and take only a subset of the values, since that the string subset is not taking a sequence of values.

I said that was everything ok, but when I checked the number of values I realize that the VI was with problem.

Attached its the VI and the example file to load

Thanks for attention

Dan07 

[TCPlomp]

Here's what I was talking about.

Remember you are not reading lines anymore but bytes!

 

Ton

Message Edited by TonP on 12-02-2008 08:08 AM

[dan07]

TonP

You are right about your comments. Now the code its perfect! The only "problem" is that I created a constant (10) to show that after 10 bytes its the end of the line and the next 10 bytes will be of the next line. But sometimes, the values range from 90 (or even less) until 130 (or even more) , so sometimes I could get a value like 94.546 (6 bytes) and sometimes could be 120.321 (7 bytes). Is there a way to progamatically verify the lenght and change as needed?

Attached is a example data.

Thanks for attention

Dan07

Message Edited by dan07 on 12-02-2008 01:57 PM

[DFGray]

Since your lines are of unequal length, you will need to base your calculation on lines instead of bytes.  There is a right-click option to read lines on the Read From Text File primitive.  The count input is then lines.  This should make your job much easier.  Remember to keep your chunks close to 65,000 bytes!

[altenbach]

---

dan07 wrote:

Attached its the VI and the example file to load

---

A few comments to your code:

• You should disable "convert EOL" in the text file IO VIs. While it probably won't make a difference in windows, it will shorten the file if you run it on e.g. linux because of different EOL definition. You want to keep the file structure exactly "as is".
• When you built the path in the "RUN" event, you should wire the string (which is the filename!) and NOT convert the string to a path first.
• Since the read and write operations must occur sequentially, you can place them into the same error thread. This also has the advantage that the writing get skipped if the reading fails, something that would probably be reasonable. 😉 No need for "merge errors".
• The LED blinking property belongs outside the FOR loop. It needs to be written only once per event. Also the LED terminal can be outside your code since you are never interested in its T/F value.
• The "first point, last point" values in your cluster should be integer (e.g. I32), because that's what they are. You can never read 5.3 data points. (Is the sample rate integer too or can it be fractional?)
• Personally, I would keep the path and array in shift registers instead of local variables.
• In the "Path" event, you don't need to wire N of the FOR loop.
• I would add a vertical scrollbar to the "array" control. What if you have more than 9 files??
• The FOR loop should contain a case that skips all FIle IO if "last point" < "first point".
• Don't read the entire file, but start at "first point". (as in Ton's example).
• Since processing your files could be slow, I would add a progress indicator (e.g. 100N/i using Q/R to keep things blue)
• ASCII formatted datafiles should never be used for large files such as this. Consider flat binary files and everything will be much faster and easier (no expensive text parsing, exact bytes/point, etc.). Where do the files come from? Are you generating these yourself in another one of your programs?
• While you can use DFGray's advice to read variable-lenght lines, it is still much more expensive than working with binary files because the code needs to inspect every single byte, constanty searching for EOL characters for each value to be read.

let me know if you have any questions. Good luck!

[dan07]

Altenbach, thanks for your attention and comments. I followed your suggestions and modified my code.

 Is the sample rate integer too or can it be fractional?

It is always integer. I changed the sample rate control to I32.

The FOR loop should contain a case that skips all FIle IO if "last point" < "first point".

Good suggestion. I inserted a new case to the event structure to handle this problem. The case does not skip File IO, instead of this it avoid that impossible numbers had been entered in the "From" and "To" controls.

Since processing your files could be slow, I would add a progress indicator (e.g. 100N/i using Q/R to keep things blue)

I was not able to build my own progress bar. Instead of this, I got a example of progress bar here in the forum and I am considering to inserted it in my code. How should I proceed to build the progress bar as you suggested?

 Where do the files come from? Are you generating these yourself in another one of your programs?

These files are exported as ASCII files by other programs, since that Labview is not able to handle the original files of the programs.

In my next codes I am considering to first of all load the ASCII files and convert them to binary (using the Write to Binary File.vi). Next, I can perform all the operations by loading this binary file, what do you think?

Attached are my code and the VI to create progress bar

Thanks

Dan07

Message Edited by dan07 on 12-04-2008 01:34 PM

[altenbach · Accepted solution]

The FOR loop should contain a case that skips all FIle IO if "last point" < "first point".

Good suggestion. I inserted a new case to the event structure to handle this problem. The case does not skip File IO, instead of this it avoid that impossible numbers had been entered in the "From" and "To" controls.

The problem is that you compare old and new, but originally all sets are 0,0 and this should not be valid. I would use a boolean to indicate to the operator which sets are not OK. It is very frustrating to the operator if entered values are not accepted without feedback. Bad UI!

Since processing your files could be slow, I would add a progress indicator (e.g. 100N/i using Q/R to keep things blue)

I was not able to build my own progress bar. Instead of this, I got a example of progress bar here in the forum and I am considering to inserted it in my code. How should I proceed to build the progress bar as you suggested?

You can replace your LED with a progress indicator and show/hide it as approrpiate. Not bulky code needed. 🙂

 Where do the files come from? Are you generating these yourself in another one of your programs?

These files are exported as ASCII files by other programs, since that Labview is not able to handle the original files of the programs.

In my next codes I am considering to first of all load the ASCII files and convert them to binary (using the Write to Binary File.vi). Next, I can perform all the operations by loading this binary file, what do you think?

You could probably write a VI that reads the propritary binary files directly. Do you have the specs?

Anyway, here's a quick rewrite of your last code incorporating a few of the suggestions. I cannot run it, so there might still be a few bugs, bu you should still get the ideas. 🙂

Message Edited by altenbach on 12-04-2008 01:04 PM