Read Data from file and bifurcate it

[Aziroazide]

Hello people,

[Brief Intro:---  I'm working on interfacing Arduino UNO with LabVIEW, no success until now even after reading couple of forum solutions. So what i did was dumped the serial monitor data into a text file and hooked up the text file into VIEW ]

The problem I'm facing here is:

LabVIEW: (Hex 0x1) An input parameter is invalid. For example if the input is a path, the path might contain a character not allowed by the OS such as ? or @.

Error Pic

Searched the whole community forums, everybody's suggesting to either check the file path or relocate it somewhere with shorter file name. (Tried both, didn't work)

[tried this too: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000P6rgSAC&l=en-US]

So, after researching a lot and with no result I'm asking my Question here.

Kindly do help me.

Basically the file contains this format of data:

DD-MM-YYYY HH-MM-SS.SSSSSS Data_Type Data_Value Data_Unit endl

for eg:

2019-05-24 16:19:59.934504 b'Flow rate: 0.00 L/s\r\n'

This is the data dumped by the arduino serial monitor.

(As a matter of fact it's more than 1 sensor data with values of each sensor in new line)

[ufaruque123]

It would be easy if you can attach the VI.

[Bob_Schor]

In addition to attaching the VI, attach a sample of the data file you are trying to read and parse.  You should expect a rapid response once we understand the problem better ...

Bob Schor

[Aziroazide]

Here's the VI and the database is also attached with it.

TIA

[Bob_Schor]

A bunch of comments:

• Your "SimpleDB_Store.txt" is clearly a Text file, not a binary file.  You should not be using Binary I/O functions to read it.
• The format for the data is not quite as you describe, and is also somewhat difficult to deduce.  The following seems to be "mainly true":
  • Every line starts with a TimeStamp, the letter "b", and a string between single-quotes.
  • The string can be of several types:
    • A three-line header starting with \n, then a string of asterisks ending in \r\n, then a "packet" number in the form <ordinal number>\r\n.
    • A named quantity, without units, e.g. Temperature = 41.00\r\n
    • A named quantity, with units, en..g. Voltage = 2.40 V\r\n.
    • A "multi-line" cluster (Angle: and Temp: are examples).
    • A "single-line" cluster (Accelerometer and Gyroscope are examples).
  • Not every record will have all of the data elements.  It is unclear what to do about "missing" elements (e.g. Record 2 has Flow Rate, but Record 1 does not).
  • It is unclear why "Cluster" values have different formulations (single line and multi-line).

One could treat each "record" as a Cluster (elements such as Temperature, Humidity, Flow Rate, etc.) and build up an array of 256 of them (from this sample file), but the parsing rules would be idiosyncratic and hard to generalize.  Also it is not clear what to do with the multitude of TimeStamps which are associated with lines of the file -- it seems more logical to have a single TimeStamp for each of the 256 "records" (who should care when Temperature was transmitted, as opposed to Angle?).

Bob Schor

[Aziroazide]

---

@Bob_Schor wrote:

• The format for the data is not quite as you describe, and is also somewhat difficult to deduce. 

Bob Schor

---

Ok, lemme clarify in layman's term:

The data is read by arduino through sensors (Hence sometimes there's a possibility of value=NA), then arduino dumps all these data onto serial monitor. After that I wrote a python script that dumps the data into a temporary text file (acting as a temp. Database). After this process, a VIEW module reads the data from this txt file and (I tried to bifurcate the data of each sensor and display it in form of meter or graphs).

Now, I this might confuse you hence I'll straightaway get to your doubt: The b' signifies that data is printed in binary form. (If you think it's wrong, I'll fix that later, but that's how python dumps the data onto a text file.)

Secondly, 

---

@Bob_Schor wrote:

• • Not every record will have all of the data elements.  It is unclear what to do about "missing" elements (e.g. Record 2 has Flow Rate, but Record 1 does not).

Bob Schor

---

The record-1 doesn't have flow rate due to my bad, the wire was lose maybe. 

Also this DB is dynamic, this text file gets updated every half a second. So the data keeps on getting appended at the EoF until unless the clock strikes 00:00 (After that the file resets to NIL).

---

@Bob_Schor wrote:

• • • A three-line header starting with \n, then a string of asterisks ending in \r\n, then a "packet" number in the form <ordinal number>\r\n.
  • It is unclear why "Cluster" values have different formulations (single line and multi-line).

Bob Schor

---

The /r/n is due to the Arduino's serial monitor's config: both NL & CR

NL is new line: /n and CR is Carriage Return: /r

(If that's a problem then I can fix that too, no issue: I'll remove it.)

Serial monitor config.

If timestamp at the start is an issue too, I'll remove that too. I just need to extract the data from the text file.

The ** after that is just for user understandability, just to differentiate data counter, else it'd be hard to comprehend.

Clusters here represent a sensor, hence a sensor can give a single value (eg: Flow Sensor: flowRate in L/s), or multiple values (eg: Gyrosensor: temp+acceleration+gyration).

Baseline: 

I'm a school student, trying to make a project: I'm monitoring the health of a vehicle, these sensors are just for representational purpose, the data is to be extracted from them and displayed onto "vi"

For example, I'm attaching my waterInjection.vi module here (Just for reference for you to understand what I'm doing here)

So, the data from the text file is to be displayed here, I just need to extract the data from txt file and later on I'll load this (and other) vi so as to visually display the values on meter or graph.

[Aziroazide]

So I came up with a workaround but now am facing different issues. What I did was, I removed all the texts and converted the format to simple list and now I'm storing that onto a txt file which is then loaded by a new vi that I've made. But the problem is: now am not able to convert this dynamic data to numeric data but somehow it's being displayed onto the graph.

Attached below is my DB, python script to dump the data and the new vi

"The issue now is that I'm getting the data on graph but when I connect any numeric indicator to it, it doesn't show anything. Even the probe shows either 0, or some dynamic value such as [1]"

I want to connect these resulting values to either a meter, gauge or something like that.

The value coming out is NIL, i dunno why.

Kindly do help me here.

Data format now:

{

iteration, humidity,flowRate, PPM-value, deviceVolt, pitch, roll, temp(F), temp(C), AcX,AcY,AcZ,GyX,GyY,GyZ,Knock

}

iteration is iteration value

humidity ranges from 0 to 100

flowRate ranges from 0 to 1

etc etc...

"Also i guess append array at the end is useless but I placed it to check for incoming values"

[Bob_Schor]

From your description, it sounds like you've reduced the problem of parsing the Arduino string to a (slightly simpler) parse a (better-defined) string whose entries look like entries into a LabVIEW Cluster (I hope you know what this is -- if not, take a few more LabVIEW Tutorials or ask your Instructor) and assembling each entry into an Array (note that the first item, "iteration value", become (off-by-one) the Index of the entry into the Array) (and I also hope that also makes sense, else learn more about LabVIEW Arrays and how to access their elements).

Here is a little VI that assumes you have an Array, "Data" that holds the pH, Flow Rate, and Temp (°F) data that you have parsed, and want to plot on your indicators.  It uses a For loop to iterate through the entire Array, plotting each set of data at a rate of one update per second.  In my example, the Array is empty, so it won't show anything, but if you fill it by parsing your data, it will display it all.

You'll want, of course, to build a larger Cluster, as you have more than the three elements I used in this demo.  I'd also suggest that you spend more time learning how to program LabVIEW than you have in making fancy Front Panels -- having "pretty" LabVIEW code is nice, but having working LabVIEW code is better.

WaterInjection (BS)

Bob Schor

[Aziroazide]

Thanks a ton boss, I'll surely consider your opinion on learning more on labVIEW (currently on a deadline here!!).

I'll give your idea a shot and reach you as soon as possible to let you know if it worked or not or if I encounter other issue.

Thanks and regards.