Cannot pass finite data in one iteration

[deskpilot]

I started down this path on a post a couple of weeks ago, but it didn't go the right direction. Turns out the problem I'm having is that my finite tasks creates a file that is too large to pass in one iteration. My DaqMx read is in a while loop so that it can operate in parrallel with other operations and was set up to read all samples (samples to read set to -1) and then stop the loop when it output the data to a queue (which writes to a .tdms file).

I am attempting to acquire 48 channels at an average of about 1 MHz for a second (there are three tasks at different speeds and lengths), and this created a not enough memory error unless I reduced that to less than half that number of samples. The solution was to only read a million samples at a time and allow the read loop to iterate until all three DaqMx Reads error out, causing the loop to stop. This allows me to acquire much more data, but when I look at it, my channel names are missing. It appears that the data is all there, but everything is untitled.

So, how can I get all 300 MB of data out of my finite task without losing the Channel Names?

[BCho]

Hey deskpilot-

It is unclear why exactly you seem to be loosing the channel names from your data. As a test, do you see the channel names in your data if you simply take a million data points and only iterate your while loop once? How are you viewing your data once you have successfully saved it? Would it be possible to post a screen shot of your code and a screen shot of you untitled data?

[deskpilot]

Thank you for the reply. It turns out I half cured my own problem. I was still using a setup with less than one million samples per channel, but telling the read to get 1 million samples. When I decreased the read to 100,000 samples per channel, the channels retain their names. I do, however, still get untitled empty channels at the end of the data when I look at it with .tdms viewer.  I have a program that converts the data to .txt files post test, and it was showing untitled also on the channel attributes. I have not had a chance to run it on the data after reducing the read to 100,000 lines and getting the right channels with the untitled channels at the end.  

If I iterate the while loop once, I still get the untitled channels at the end, but I am building an array out of three different reads (from three different finite tasks at three different speeds), so in order to iterate once, each read has to output a different number of channles.

Here is a quick example I built that represents what I am doing. It would be nice if I get get rid of the untitled empty channels?

Message Edited by deskpilot on 06-10-2009 12:29 PM

[BCho]

Hey deskpilot-

It is interesting that you seem to loose the channel name information if you go above 100,000 samples. In your code it looks as though you have three different rates and times and two channels for each. Did you intend on having two channels sampled at each rate? What happens if you take out two of the tasks and simply sample from one channel, do you retain the channel name? It looks like in your while loop you are reading in all of your readings, which sort of makes the while loop unnecessary. This is because the DAQmx read is waiting until all of those samples are obtained before proceeding, and since you are already read in all of the finite samples you intended to read there are no more samples to be read in subsequent iterations of the while loop.

So, to sort of narrow down when this loss of information is occurring, I would start with one channel and one task and see if the channel name is retained. Then I would add the second channel to the first task. I would continue to do this until I was sure of where the channel name is lost. Have you tried to use the TDMS File Viewer.vi at the end of your code to view your file? This is done in many of the write to TDMS file shipping examples if you are unsure of what I am referring to. Are you able to see the channel names there?

[deskpilot]

The problem does not exist above 100,000 samples, it exists if the read is looking for more samples than are actually there.  

The loop only iterates once in that example because you asked me what happens if I only iterate once. The real program has dozens of channels at each rate, and needs to iterate more than once because I cannot pass 400 MB in one loop iteration. If I try, I get "not enough memory" from both windows and the .tdms write. And yes, as I stated before, I look at them in the .tdms viewer.

So here's the problem: any time you poll the read on a finite task and you are asking it for more samples than what are left in the buffer, of course it errors. But it also fails to pass the channel name along with the data that it does take out before it errors. So now the data for the same channel is broken up in to two places, one titled, the other untitled. I can solve this by making sure that I never ask for more samples than are in the buffer, but then I still have to poll it one more time to get the error (thus stopping the loop). When I poll it this last time, it creates new untitled channels, so I still end up with twice as many channels in the file as I actually have. I cannot figure out a way to iteratively read a finite task, but then stop polling it when it has read all samples (other than the method shown in the example)?

[BCho · Accepted solution]

Hey Deskpilot,

Sorry for the confusion. What happens if you set up your while loop to stop when you have have sampled all your data instead of waiting until you generate an error. For example, if you are going to sample 100 samples total and 25 samples each time through you while loop, have your while loop stop when your while loop iteration terminal equals 3. This way, you are still sampling all 100 samples but are not relying on the DAQmx Read to error out to stop your while loop.

This may be a bit of a work around. But with my initial messing with it, it seemed to do the trick of eliminating the extra 'untitled' data field.

[deskpilot]

I guess thats a pretty simple fix. I hadn't thought about that because the real program I've been troubleshooting I was trying to read the same number of samples per iteration from each task. It also iterates numerous time before I arm the high speed, but I guess I can just check what iteration I'm on when I arm it and then stop n number of iterations later. Seems simple now, thanks for the help.