How to read from txt file that has words in between?

[zientist]

Hi all,

I am using Labview 8.2.

I would like to read from a text file.  I have data (after each time it is has averaged over 100 waveforms) repeatedly stored on to the file.  The idea is to further improve SNR in post processing by again averaging the data (that has been averaged over the 100 waveforms).  

I can get LabView  to save the data repeatedly into the file, so it keeps getting appended.

The problem is to read the data in labview so I can now again average it.  The problem is the labview seperates the sets of data with the following:

" Channels    1    
Samples    9925    
Date    2008/10/28    
Time    17:16:11.638363    
X_Dimension    Time    
X0    -3.0125000000000013E-3    
Delta_X    2.500000E-6    
***End_of_Header***        "

So When I read it, it only sees the first set of data.

Can someone please tell me how to read all the sets of data in labview?

I have attached the file I want to read "acquiredwaveform.txt"  and the basic VI (really basic btw) to read the file.

Thanks

[Andrew McL]

Hi zientist,

I've had a look at both your VI and your data file. Are you deriving this text file from a lvm file? Could you possibly upload the VI that generates the text file? I think you will need to restructure the way you generate the text file as it just appears to be one long data set. 

Thanks,

[johnsold]

zientist,

It should not be too hard.  You could search for the "***End of Header ***" string.  That will locate the header, but not the end of the header.  You would then need to look for the "Filter order" which appears to near the end of the header.  The data starts on the next line after the line containing "Filter Order".  The data set appears to end with one or two blank lines - also fairly easy to find.

As Andrew McL said, it might be easier if the files were written differently, but as long as the headers are consistent, you can extract the data.  We did this kind of thing back in LV 1 and LV2 when only very elementary file functions and string matches were available.

Lynn 

[DFGray · Accepted solution]

This appears to be a standard LVM formatted file.  You can read the different data segments by using the LVM Read Express VI in a loop (make sure the file maintains its .lvm extension).  I modified your example to show this.

Note that my example is fairly inefficient. The build array primitive causes a massive memory copy every time it is used and you would probably be better off averaging as you read in.  I did it this way to make it more obvious how to read in segments from an LVM file.  In the process, I noticed you have a fair amount of phase jitter (zoom in on the graph of all three waveforms).  Simple averaging will cause you to smear out your waveforms.  You can either fix the problem in hardware (recommended) or phase shift your data so it lines up before averaging.  If you need help with the latter, let us know.

[zientist]

Thanks DFGray for the code it works fine. In terms of the phase shift problem I have a written a code that phase shifts the data before averaging. Say if I want to average 100 times, I do this by using my first signal as a reference and then taking the auto-correlation (with itself) and the cross-correlation (with the other 99 signals).  I find the difference between the maximum peak of these correlations and then shift the other waveforms accordingly, after this I average.  It works fine.  If there are better ways of doing this, please let me know.

Answering a few questions raised: The main reason why I saved it as a measurement file (although I would prefer to save it as a txt file) was because I wanted to incorporate the inital conditions into the saved file.  The measurement file had the comments string that enabled me to do this.  I didn't know any other way to do this.  Are there any better ways of saving initial conditions onto the same data file? Also it would also be nice to be able to start new lines, i.e. in my case I have:

"X_Value    Untitled    Comment
-0.003120    1.027881E-5    Fs= 400kHz, Fp= 23Hz, Amplitude= 2Vpeak"

Since the initial condition comes in at the same line as -0.003120.... it becomes a bit inconvinient to read in matlab. How can I save it such that I get for examle:

 "X_Value    Untitled    Comment

Fs= 400kHz, Fp= 23Hz, Amplitude= 2Vpeak
-0.003120    1.027881E-5"

DFGray, with regards to your comments, I would like to know why you insist on saving it in .lvm format.  As I can save in .txt as also read it easily.

Thanks

[DFGray]

Your method of using correlations should work pretty well.  However, the details may cause you to degrade your data a little.  When you find your peak, do you find the highest point or do some sort of interpolation (e.g. fit to parabola) to the peak and the surroundings to find the peak position?  If you know the shape of this peak, you may be able to generate a fairly fast, closed form solution.  If you are simply finding the highest point, you will get phase jitter of plus or minus about half a sample period.  To actually use a phase shift which is more accurate than your sample period, you need to be able to shift your data less than a sample period.  You can use one of the resample VIs/Express VIs that come with LabVIEW for this.  Alternately, since you have a lot of unneeded data at the ends of your waveforms, you can take a Fourier transform, change the output to magnitude, phase, modify the phases to shift the waveform in time (this is a linear ramp change to phase in frequency space), and do an inverse transform.  Do the sample level shift first, then the subsample shift for best results.

Unfortunately, there is no easy way, other than the one you are using, to save extra information in the file.  One option is to post process the file to move the data, although it is probably just as easy to handle this with your read code as you are currently doing.

<shameless NI marketing>Have you tried MathScript in LabVIEW?  It would allow you to use the LVM read to get your data and eliminate having to write parsing code yourself.  Your m-scripts should copy and paste in.</shameless NI marketing>

I recommended keeping the LVM suffix as a way of keeping track of what file format is what.  LVM is a specific text format, while TXT is generic.  LVM has a specific internal header, so the suffix is not needed, but I like to know what format my files are in.

Let us know if you have any more questions.

[zientist]

Thanks again DFGray for the comments. 

After the correlations to find the peak positions, i just take the max value.  And you are right the accuracy is limited by the number of  samples per cycle.  Perhaps it would be clearer if you see the code.

1) Basically I get a signal on the up and downslope of the sine wave.  On the down slope however the signal is negative, i.e. its is flipped.  So before I shift and average...I 'cut' the waveform into 4 (when cycles per buffer = 2, then I get 4 signals, 2 on the up slope and 2 on down slope) bits.  Counting from one, I flip the even number, cut it, and but an array of waveforms which is then sent to be convolved and shifted.

2) Array of waveforms are stored to be phased shifted (Convolve and shift vi) and averaged (entire averaging vi which uses the convolve and shift vi as a sub vi). 

* Phase shifting doesn't work when I cut and put it together (So something is wrong in cut waveform vi) 

* Also if it isn't too time consuming could you give me an example of interpolating and shifting thing.

* Also if you have any comments regarding the following VIs please let me know.  

Thanks 

Attached is:

1) Cut waveform vi

2) Convolve and shift

3) Entire averaging 

[zientist]

Hi,

I have tried fitting and resampling.  Doesn't work very well.  For now I have fitted the polynomial to the entire waveform.  I think I will try finding the max, and then fitting the waveform around that max, then finding the max again for the fitted waveform.

For now, before I convolve it, I fit it, then resample, then convolve.

I think I am on the right track? (Attached is convolveshift2)

[DFGray]

You have some frequency variation from shot to shot in your waveforms which can cause you issues (and prevents Fourier Transform methods of finding phase from working well).  Attached is a modification of my original VI which does a correlation of the first waveform with the others, then does a parabola fit to the peak of the result, calculates the vertex, and uses this to time shift the waveforms.  It seems to work OK, given the frequency variation issue.  You should look into why your frequency is varying from shot to shot.  It may be important.