RS232 DASYlab parsing help needed

[miller863]

Hi, I am running DASYlab 13, and have set up an RS232 input with one channel (0) for a USB sensor that can be normally read through the COM port (COM port 13 in this case, at 115200 8-N-1.  I can read data fine with the RS232 monitor, I am having trouble with parsing the data using the format in the "Measurement Data Format" field of the RS232 Input configuration box so as to get meaningful data to display using the Digital Meter, chart recorder, etc.

To get data from the sensor (outside of DASYlab) I can close DASYlab (which closes the COM port), and issue a command to the sensor from the command prompt in a terminal program:  B followed by the enter key.  I.e. "B\r\n" (without quotes).

The sensor responds with "AA 3B xx xx xx xx" hex, where "AA 3B" is a header, and the x's are the data.  The data is in INT32 format (IEEE 754).  However, apparantly to get it to convert correctly with an int32 converter, the hex must be reversed.  I.e. if the sensor reports the data as 12 34 56 78 (this is a made up datapoint), I have to change it to 78 56 34 12 to get what I expect as an output.  Therfore, I cannot do a Measurement Data Format of "2x f" or "2x fy" because neither one takes in account the reversed byte format.

How can I parse the data so I ignore the two byte header (would this be 2x or 4x??), and reverse the data byte order, so that the display on the digital output is correct?  I have tried all kinds of things, looked in the help files, forums, etc., and an coming up short.  I am a beginner DASYlab user, so I don't know how to do scripting, and I don't understand why there are multiple channels within each RS232 input.  I'm thinking that they allow you to separate data from an RS232 "packet" into individualized data points, but from the examples I have seen, the channels don't appear to allow you to reorder data....i.e. using my earlier example, 12 can go into channel 0, 34 can go into channel 1, etc., but I can't put 78 into channel 0, 56 into channel 1, etc.  Also, I don't understand how the data in the channels translate into the output what is displayed on the output (i.e. digital meter, etc.).

DASYlab1.png image shows a screenshot of my setup.  You can see the COM port monitor, and the data output by the sensor.  In this case it is AA 3B 74 94 62 41 hex.  Removing the header, and reversing the bytes gives me 41 62 94 74 hex, which is 14.16 when I convert it with http://www.h-schmidt.net/FloatConverter/IEEE754.html.  This reading (after manipulation) is correct.  However, with a Measurement Data Format setting of "2x  f" I get 00000000000000000000000000001110 on the digital readout, which does not correspond to anything meaningful.

Your help is much appreciated.

Thanks,

[hw70]

Hmm, I think you should change the display format of the digital meter to decimal, not binary.

Because the format string "f" does what you want (see image).

"2x" to skip two "header bytes"

"f" for the ieee data

[miller863]

Ah thanks that worked well for using the digital meter.  Although I don't know what the "\Xx74\X94\X62\X41" means in your attachment, and what this window is.  I tried to find the equivalent window in "Options-->Global Variables and Strings.....", but could not find.

I have an additional challenge: When the data of the sensor matches either of the header values, i.e. AA or 3B, the sensor will insert an additional byte equal to the AA or 3B.  The data portion of the "packet" is normally 4 bytes, except in the case where the data contains AA or 3B, in which an additional AA or 3B is inserted for each instance.  (this is called byte stuff).  In the extreme case, where all the data is equal to one of the header bytes, the entire packet could be 10 bytes (e.g. AA 3B AA AA 3B 3B AA AA 3B 3B).

In the present scenario, I am sending the "B" command ("B\r\n") once a second.  When DASYlab encounters an extra byte, it appears to paste that byte onto the beginning of the next packet, and go "out of sync".  What i mean by "out of sync" is that now DASYlab treats the extra byte as the first byte in the next message, which is like the "packet" has been shifted by a byte (the "3B" packet header is now part of the data field).  This shift continues until another repeated AA or 3B is in the data area, and now the "packet" is shifted by another byte.  Maybe an example is a better explanation of this phenomenon:

Packet                               Correct Decimal          Packet seen by DL             DL display (decimal)

AA 3B EE C5 A9 41              21.22                         AA 3B EE C5 A9 41                  21.22 

AA 3B EE C5 AA AA 41         21.33                        AA 3B EE C5 AA AA                 -3E-13

AA 3B 8F C1 81 41              16.22                         41 AA 3B 8F C1 81                  -7E-38

...

AA 3B 8D 97 A9 41              21.20                          41 AA 3B 8D 97 A9                  -6E-14 

AA 3B 87 1C AA AA 41         21.26                         41 AA 3B 87 1C AA                   -1.39E-13

AA 3B C4 A4 94 41              18.58                         AA 41 AA 3B C4 A4                  -8.5-17

(where DL=DASYlab)

This can also be seen in the DASYlab3.png image attached to this.  DASYlab5.png shows the data in the digital display when my data format is set to 2x l.  (where "l" = lowercase "L"), i.e. Long.

Therefore I think I need to insert some intelligent parsing, which filters out a repeated AA and/or 3B.  

Does DASYlab inherently have this capability?  

From the forums, it looks like I will have to do some Python scripting (unfamiliar), and create a custom module to house it.

ftp://ftp.mccdaq.com/downloads/example_programs/DASYLab_App_and_App_Notes/DASYLab%2013%20Script%20Ex...

Python has a bitstream module that, from first glance, seems suited for this application:

http://pythonhosted.org/bitstring/

My image is that the input to the module would be the packet, and the module would check for AA 3B header, then examine each byte to see if it is AA or 3B, then reconstruct the packet without the repeated bytes, and output just the data portion.  However, DASYlab has behaved in an unexpected manner (to me) by shifting the received packets (as described above), so I am not enrirely confident that spending the time learning the custom module and learning python programming will achive any different results than what I have achieved so far.  

Before I get neck deep in this, could you (or anyone else in this forum) tell me if I am headed the right direction?

Thanks!!

Scott

[CJ_Butler__DASYLab_Pro]

Sadly, I think that you're headed in the right directly.

Sadly, because this is a case / scenario that we have not seen before, and do not have a way to handle it in the RS232 module.

The trick with the Python module will be how to handle the comm port. I've been working on a GPS example... I'll try to finish it and post it. It works, except that I messed up the timestamp. It's a good example because it reads the comm port, opening it at Measurement start, closing it at Measurement stop. I import the pynmea libraries to parse... 

Maybe Holger has a generic example that he wants to share? 

There are DASYLab Python examples posted at www.dasylab.com. Look in the Downloads section.

[miller863]

Is the "shifting" of data that I presented above an intended and/or expected function of DASYlab?  A bad datapoint in DASYlab here and there (when 1-4 extra AA and/or 3B is in the data area) from my sensor might be tolerable in some instances.  But since any extra data introduces an error that propogates through and corrupts future readings (the "shifting" effect, as I described above), it obviously is not tolerable.  

It almost appears that anything received from RS232 that doesn't match what is handled by what is in the "measurement data format" area (i.e. "leftover data"), DASYlab stores it into a buffer, and adds the new data to the leftover data to form the next data point that is displayed, and so on and so forth.  If I try a measurement data format of "2x f 4x", my expectation would be that such a format would ignore the first two bytes (header), process the next 4 bytes as a float, and ignore the up-to-4 extra bytes that might be received as a result of AA or 3B in the data field.  In practice, this data format gives me really weird results, i.e. bezerk data, which confirms my theory about the way DASYlab handles the RS232 data.  From what I see, DASYlab just fills up a buffer with whatever is received on the RS232 COM port, regardless of the timing of the Data Request interval (I have it set to 1second) or Measurement data request.  Is this the case?  Is there any way around this?  Like flushing the buffer between each measurement data request send interval?

The expected operation of the RS232 would be for the RS232 to send the Measurement Data Request, process data that is received on the RS232 port according to the measurement data format, and discard any other data received.  Then when the Measurement Data Request is sent again (due to the data request sample interval being set to some value), this process repeats over and over.  In my case, if there were ever a repeating AA or 3B in the data portion of the packet received from the sensor, it would show up as a strange datapoint, which I could ignore or delete during post processing.

Thanks!

[CJ_Butler__DASYLab_Pro · Accepted solution]

If you configure the parsing string to assume a fixed length, then yes, this is an expected error.

Often, we can compensate for variable length strings by using an end of line or end of data point delimiter... the a\r is the flexible length parse for an ASCII string. With the delimiter, we always end at the end of the line, starting on the next one.

One way to alway anchor at the start of the string is to use a search string... if the data always starts with AA 3B, you might try a search string using it...

"\xAA \x3B" l

You need the \x to notr that this is hex. Also note that we are not skipping them with 2x anymore, but searching for them. This should keep it anchored so that you get a bad data point periodically, but legal data points are fine.

[miller863]

That data format works great, except for the known issue with the occasional glitchy data point when the byte stuff occurs.  As stated before, a true solution would be for me to write a Python script to truly parse the data, as unusual situations like this can not be handled by DASYlab's standard interface.

Thanks so much to both of you for helping me out!

[CJ_Butler__DASYLab_Pro]

I've got my GPS input module working with the Python script... if nothing else, it's an example for reading the comm port, doing some parsing (in this case I use pynmea to parse it), and sets up the output with timing. 

Would you like to look at it? Are you familiar with Python? Send me a PM, and I'll share (that applies to everyone who's interested).

Once I'm convinced that it works, I'll be documenting it and putting it up at www.dasylab.com and kb.mccdaq.com.