Add column to large array

[Humada33]

All-

I have my data (20 columns by ~700,000 rows) stored in a binary file and I would like to add a timestamp to each row of data.  I plan to use the sample rate with the number of samples to add the time at which the sample was recorded to the data.  I read the data in (~700,000 samples) and use for a loop to great a 1D array with the same number of rows of data and then insert the 1 column into the larger array that has my data.  However this seems to take a significant amount of time and I am searching for a faster/simpler way. Ny significant I mean the vi runs for ~20min before I give up and stop it.  I posted a png of the block diagram.  Any help will be much appreciated.

Thank you,

Chris

[aeastet]

Can you send a copy of this vi and a copy of your data?

[jack47]

If you have LV2009 you can 'Configure Iteration Parallelism' on the for loop and that will speed up the loop significantly.  However, it would probably be better to add the time column when the data is written, so you don't have to load this huge array into memory.  Or if you can keep the number of samples from previous in the experiment, or write it to the data file then you could just append the time to the end of each row of the file.

Here is the nugget on parallel loops

[Humada33]

Here is the vi but the data file is 45mb.

[Humada33]

Jack,

Thank you for the reply.  Unfortunately I do not have LV2009, using 8.5.1 and I have several trials with data so it is too late to add the timestamp when the data was originally written.  I will try to have this done for future trials.

I am missing what you mean by append the time to the end of each row. Do you mean that I could write the timestamp to the original file instead of bringing the data into memory and then writing the data to a file?  Could you elaborate?

Thank you,

Chris

[GerdW · Accepted solution]

Hi humada,

try this...

[jack47]

In the File IO palette, there is the Advanced File Functions palette.   One of the functions is 'Set File Position'  where you set the number of bytes from the start, end or current file position.  If you set the position and then do the write to binary file, the text is written at that point.  I does overwrite what is there, however.

[smercurio_fc]

---

jack47 wrote:

If you have LV2009 you can 'Configure Iteration Parallelism' on the for loop and that will speed up the loop significantly. 

---

That would only help if there's multiple loops. There are no multiple loops in the VI.

As for the code: there are several errors/oddities with the code:

• You said you're using LabVIEW 8.5, but you're using the old Read From SGL File. This seems odd.
• What is the purpose of the offset control? I know it's so you can start reading at a specific row, but why? Also, why a "# of rows" control? Why convert only part of the file?
• The method being used for replacing array substitutes is incorrect. You have to use a shift register for the array. The way you have it now, only the first element will get replaced with the last value.
• Avoid coercions, especially when dealing with arrays! The output of the loop is an array of DBLs, which you are trying to add to an array of SGL. This requires downcasting.

[smercurio_fc]

---

jack47 wrote:
In the File IO palette, there is the Advanced File Functions palette.   One of the functions is 'Set File Position'  where you set the number of bytes from the start, end or current file position.  If you set the position and then do the write to binary file, the text is written at that point.  I does overwrite what is there, however.

---

Then what would be the point of this?

[GerdW]

Hi smercurio,

the vi seems to be loaded from an older "project" (pre LV8) as is shown by the automatically inserted error cluster unbundle function after the file selection...

But LV8.5 also explains why the diagram cleanup wasn't used before posting

Message Edited by GerdW on 08-20-2009 04:43 PM