How to save captured data to a log file to prevent lag?

[milkbottlec]

Here is my VI.

I read in an string of this format (the channels number are now 32 only but will like to increase up to more than 10000 later)

Time\sStamp\tChannel:00\tChannel:01\tChannel:02\tChannel:03\tChannel:04\tChannel:05\tChannel:06\tChannel:07\tChannel:10\tChannel:11\tChannel:12\tChannel:13\tChannel:14\tChannel:15\tChannel:16\tChannel:17\tChannel:20\tChannel:21\tChannel:22\tChannel:23\tChannel:24\tChannel:25\tChannel:26\tChannel:27\tChannel:30\tChannel:31\tChannel:32\tChannel:33\tChannel:34\tChannel:35\tChannel:36\tChannel:37\tIP\sAddress\t\n

The problem I am now having is that, the data is through UDP sending in, the program start off normal capturing all data, but when data are captured and saved to the .log file after an hour or more, some of the data will start to be missed.
I guess this is beause the .log file is becoming larger, and in the rate of 10Hz (now), the file open and close can't chase up? so some data are missed?

Anyone could advice or amend the VI so that I could have a better way to save the captured data? making a buffer? or what so ever? Hope someone could help =]

Thanks.

[tbd]

Hi milkbottlec,

      I doubt it's a file-size problem, though, could be.  More likely you're building a bad path based on incomming data.  Your logic is testing for a bad path when opening the data-file, but the error case assumes the problem is a missing file or folder - it could be an invalid filename.

The attached code is modified to append headings when the filepath changes, and record paths that fail open/create.

Hope it helps!  

[milkbottlec]

Thanks for changing it, but will it be better to just give it a box to select a file to save to?
How Could you change it?

I will try the amended one to see if there is any bad file path pops up.

[milkbottlec]

Just found some error when the data are saved.
At some point, there data just copy this whole thing and save in the middle of the excel

""Time\sStamp\tChannel:00\tChannel:01\tChannel:02\tChannel:03\tChannel:04\tChannel:05\tChannel:06\tChannel:07\tChannel:10\tChannel:11\tChannel:12\tChannel:13\tChannel:14\tChannel:15\tChannel:16\tChannel:17\tChannel:20\tChannel:21\tChannel:22\tChannel:23\tChannel:24\tChannel:25\tChannel:26\tChannel:27\tChannel:30\tChannel:31\tChannel:32\tChannel:33\tChannel:34\tChannel:35\tChannel:36\tChannel:37\tIP\sAddress\t\n""

I don't know which part handling the error is wrong.

If possible, maybe could you help me delete the whole auto naming function (move it aside) and just add in a promt user for saving directory thing ok?
Thanks!

Here is the file and i added the excel in.

[tbd]

--------------------------------------------------------------------------------
milkbottlec wrote:

Just found some error when the data are saved.
At some point, there data just copy this whole thing and save in the middle of the excel

""Time\sStamp\tChannel:00\tChannel:01\tChannel:02\tChannel:03\tChannel:04\tChannel:05\tChannel:06\tChannel:07\tChannel:10\tChannel:11\tChannel:12\tChannel:13\tChannel:14\tChannel:15\tChannel:16\tChannel:17\tChannel:20\tChannel:21\tChannel:22\tChannel:23\tChannel:24\tChannel:25\tChannel:26\tChannel:27\tChannel:30\tChannel:31\tChannel:32\tChannel:33\tChannel:34\tChannel:35\tChannel:36\tChannel:37\tIP\sAddress\t\n""

I don't know which part handling the error is wrong.

If possible, maybe could you help me delete the whole auto naming function (move it aside) and just add in a promt user for saving directory thing ok?
Thanks!

Here is the file and i added the excel in.

--------------------------------------------------------------------------------

Hi milkbottlec,

      Regarding the headings in the middle of your file, I imagine the filename changed temporarily.  When the name changed BACK to an exisiting name, the headings were added (anticipating a new file.)  I'm more curious about the "lost" data problem.  Do you still see holes in the data?  If so, were any bad filenames recorded?

Sure! Why not present a file-dialog instead of automatically generating filenames,  Looks like you've mastered the "True/False" case, why not build on what you know?  Experiment with the File Dialog express VI on the FILE\Advanced File Functions palette.  Personally, I'm a fan of auto-generated filenames, though, not necessarily built from the data. 

Cheers!