Read many files, recognise the source and make operations across.

[JimAt93]

Hi everyone,

I am from the biomechanics field where we have multiple trials across participants and many participants. All these exist in the same file. I am trying to figure out

1) a mechanism that will recognise where the data came from when I read many files (i.e. trials)

2) When I have a few specific values (peaks, times etc), I know that I need to create an array or something(haven't figured out which is the best option yet), but what do I need to do to find the means across the trials(or different files) 

I do have a specific VI to upload but I am looking more to understand the concepts first (newbie in LabView).

Thanks for your time in advance guys.

[Bob_Schor]

We know nothing about the structure of the file, nor how data are organized in the file to identify participants associated with particular data sets.  Is it a Text file?  A Binary file?  An Excel WorkBook?  A file with a particular Format (perhaps created by another program and saved according to some File Protocol)?

LabVIEW has a number of File I/O routines.  At one end, there are Text Files, which are "human-readable" and generally allow humans to distinguish one person's data from another, and at the other end, there are binary files where the data representation is effectively "unknown", and requires "extra knowledge" to interpret the bits and bytes.  You absolutely need to know (or be prepared to deduce) the file structure in order to read it.  Once you know the structure, you use the appropriate routines in the File I/O palette to get the byes into the PC and structure them to useable LabVIEW data.

So the first thing to determine is the File Structure.  When/if you submit LabVIEW code, be sure to include an example of the Data File you are trying to read and parse.

Bob Schor

[JimAt93]

Thanks Bob for your reply. My data currently are in .txt files and I want to extract them in a spreadsheet. That's not always the case though that's why I was so general. I was/am more interested in to learn how to do these things and what do they mean. 

Further below I have included my VI and two txt files for two different participants. The ultimate goal here is:

1. To find a mechanism for knowing where the data came from (e.g., subject and trial)
2. Organizes the output so that it is easy to calculate mean values across trials for each dependent variable

[Niatross]

I am not entirely sure I understand your question at the moment. What do you mean when you say you want to know where the data came from? As far as I can tell there is nothing in the files attached which says what the subject or trial of the data was. This need including in the header of the file when it is created.

Averaging the data for each channel across lots of files isn't trivial, especially if they are large files as you might start running into memory problems. I have attached one way of doing it. This requires all of the test data files to be put in a folder with nothing else. The program then:

- opens each file in turn

- removes the header

- sums up each channel

- adds this to an overall running total for each channel

- adds the number of frames in the current files to the total number of frames processed

- Calculates the mean of each channel by dividing the running total of each channel by the total number of frames processed.

Let us know if this isn't waht you were after.

[Bob_Schor]

Hi, Jim@93.  Thank you for attaching two data files.  They appear to be tab-separated files (which LabVIEW calls a Delimited Spreadsheet File), but appear to lack much in the way of identifying information that would allow you to process across different files.  But I have some suggestions about how to process within a file.

First, look at the format of your two data files.  There are 4 lines of information (not sure how these would be used), a blank line, then what looks like a multi-column delimited spreadsheet with one row of Column headings and many rows of data.

Here is how I would read it, breaking it into a String Array called "Header" (consisting of the 4 rows of header information and the blank row), another String Array called Column Labels, and a 2D Dbl Array called Numeric Data.

Some notes on rarely-used features:  Read Delimited Spreadsheet, by default, reads the entire file.  In the first function, I have it read the (first) 5 lines.  I use the output, "mark after read (chars)" to tell the next Read function where to start reading ("start of read offset (chars)", and have it read 1 row of Column Labels.  I finally finish reading the File, this time as Dbls (and change the format specification to 6 decimals to match your data).

You now have a way to find the data you want to analyze.  Suppose you want to compare "R Sag Knee Angle" with "L Sag Knee Angle".  Search Column Labels for "R Sag Knee Angle", which should return 4.  You can now pull out Column 4 from Numeric Data, and pull out Column 25 (for Left Sag Knee Angle), and do whatever you need to do with them.  

Bob Schor

[JimAt93]

Thanks for your reply Niatross.

You can differentiate where the data came from the title s1t1= subject 1 trial 1, and so on. I tried to perform your VI but it didn't work. Probably that is because I am not familiar still with all the LabVIEW commands etc. If you see at my VI at the end I have an array of 6 variables. These 6 (for the moment at least) are the ones that will be extracted from each participant's trial and will be used to calculate their means across 3 trials(or 3 files). I am trying to figure out how to create an excel/text file as an outcome that will contain something like along the following:

Participant 1
trial 1	value 1	value 2	3	4	5	6
2
3
4
Participant 2
trial 1
2
3
4

I hope this makes a little bit clearer what is my target. Thanks for your help again.

[JimAt93]

Hi Bob, thanks for your reply.

As I mentioned to Niatross, you can differentiate where the data came from the title s1t1= subject 1 trial 1, and so on. I don't know if it is possible to do something with this information in LabVIEW. The first four lines that you mentioned are not to be used within the program, just informational for further data process. 

Your answer is very detailed and I think I understand most of it but I still couldn't figure out how to create and connect to the "all rows" features the following:

1) String Array 

2)2D Dbl Array

I following any possible instruction that I could find online but most likely I wasn't looking at the right place.

Additionally, and as I mentioned to Niatross, I already have done some calculations/computations within my VI (probably the long way), and I have added these 6 variables to an array (they can go towards an xlsx/txt file doesn't matter really). These are the ones that need to be computed across the x amount of trials of each subject and ideally, I would like them to look as an outcome something like the following table:

Participant 1
trial 1
2
3
4
Participant 2
trial 1
2
3
4

[Bob_Schor]

---

@JimAt93 wrote:

Your answer is very detailed and I think I understand most of it but I still couldn't figure out how to create and connect to the "all rows" features the following:

1) String Array 

2)2D Dbl Array

---

Here's how you figure it out:

1. Put a Read Delimited Spreadsheet function on your Block Diagram (or find one already there).
2. Right-click the function and from the Drop-down, choose "Help".
3. Look at the Connection diagram.  Not all of the connectors are labeled, but if you read the Detailed Description, you'll find every connector described.

You should see that there are 5 (not the usual four) right-hand Indicators for the Read Delimited Spreadsheet function.  The second one is "All Rows", and the third is "First Row".  If you look closely at the Snippet in my earlier reply, you can see that the thicker (because it is a 2D array) wire from the first function ("All Rows") is higher (second position) than the thinner (1D Array) wire from the second function ("First Row").

Bob Schor

[JimAt93]

Thanks Bob for all the replies.

First, the where to wire was not so much of the issue, you did a nice job explaining, rather how to create this abc think (String Array) to receive things. When I am taking an array and dropping the string constant in it the small black arrow is at the end of the abc. The ones below I copied them from another VI in the forum. Plus as you can see almost nothing works. Any ideas/sources on how to improve the code/my knowledge of Lab View?

[GerdW]

Hi Jim,

---

@JimAt93 wrote:

When I am taking an array and dropping the string constant in it the small black arrow is at the end of the abc. The ones below I copied them from another VI in the forum. Plus as you can see almost nothing works. Any ideas/sources on how to improve the code/my knowledge of Lab View?

---

When the "small black arrow is at the end of the abc" then you created a control. When the arrow is on the left side it denotes an indicator: the arrow indicates to DATAFLOW: controls generate data, indicators consume data…

When you want to learn LabVIEW (start with proper spelling) you should take note of the Training resources offered right in the header of the LabVIEW board. And hurry up: all self-paced courses are available for FREE until end of April!