how to parse serial data?

[guyh2]

Hi, really new to LV and I just can't get my head around how to solve this problem.

I have a pretty much continuous stream of serial data @57600 baud

It is composed of 9 signed 16-bit ints sent big-endian and then terminated with 2 bytes of 0xAA,

The data is just streamed so when LV opens the port it can start at any point in the stream.

I can open the port, read from it, convert the bytes to integers BUT what I can't work out is how to 'synchronise' my loop to an 0xAAAA boundary

My simple idea was to somehow discard the stream until 2 consequutive 0xAA bytes then enter the main loop

My complicated idea was a FIFO with some sort of string or byte-array parser to pattern match back from the head of the queue the 0xAA,0xAA if there are not 18 bytes between the pattern & the head then discard else remove the 18 & process

The latter seems way too complicated especially as the situation should only happen once (I have had my loop running for hours without dropping a byte, so I am happy with it).

Problem is, I can't implement either 😞 I think I just need a shove in teh right direction please ...

[johnsold]

I would probably use a two loop system like the Producer/Consumer Design Pattern, with the serial Read in the Producer.  Pass data (as string) from Producer to Consumer via a queue.  In Consumer concatenate dequeued data onto a string which is passed from iteration to iteration via a shift register.  If the string length is less than 18, wait for more data.  Once it is longer than 18 characters, search for AAAA.Take the 16 characters before AAAA and convert to the integers.  Retain any characters after the AAAA as part of the next data set.

If the string keeps growing, your processing is too slow and eventually you will have memory problems.  That is not too likely at the speed you are using unless you are doing something really slow with the data after you decode it.

Lynn

[SteveChandler]

Yes just like what johnsold said. Here is an example done in a slightly different way. I tried using a regular expression but that will not work if your data ever contains a 0. LabVIEW regular expressions puke with a NULL character.

There example has three loops.

The first loop you can delete. It just generates random packets for testing and simulates your device sending data.

The middle loop recieves the bytes from the first loop via a queue. Replace the dequeue with a VISA one byte read from the serial port. This loop is a simple state machine. There is an initialization state (where you can open your comm port), a get byte state, a check for packet state, and a packet received state. The packet received state is entered when the buffer starts with and ends with 0xaaaa. It sends the packet to the third loop.

The third loop will iterate for each packet.

Here is the vi and an enumerated typedef for the state machine.

Now I am just waiting for someone to show a much simpler way

Edit: I had attached a broken vi! This one should work

[guyh2]

Thanks Lynn & Steve, it would seem the 2nd idea is the way to go in LV!!

I had been struggling to implement my 1st idea with a case condition starting by taking a byte at a time and checking via a shift register if 2 adjacent bytes were 0xAA but then I couldn't change the case state from inside - as the 'True' was tunnelled out, it became a 'false' lol

My version of LV is an old steam driven v8.5 so I can't open your attachment - but thank you - I will hunt down the producer-consumer examples

[Coq_Rouge]

---

@guyh2 wrote:

Thanks Lynn & Steve, it would seem the 2nd idea is the way to go in LV!!

I had been struggling to implement my 1st idea with a case condition starting by taking a byte at a time and checking via a shift register if 2 adjacent bytes were 0xAA but then I couldn't change the case state from inside - as the 'True' was tunnelled out, it became a 'false' lol

My version of LV is an old steam driven v8.5 so I can't open your attachment - but thank you - I will hunt down the producer-consumer examples

---

This does not have to be to complex. Take advantage of the serial buffer. The first step would be to get in synch with the instrument. Read byte by byte until you have received the 0xAA. This can be done with a simple state machine. After reading the two 0xAA. You know that the next 2x9+2 bytes in the buffer will be your data plus synch bytes. As an error check. Check if the the two last bytes read is 0xAA. If not resynch. And so on.

In this case you do not really need a Producer/Consumer structure. As you already have it in the serial buffer. This is 1024 bytes by default (I think). Post your code(NOT as pictures) so we can take a look at it   

[SteveChandler · Accepted solution]

You don't need a producer consumer architecture. That is correct. But it is not that complicated and if you are new to LabVIEW you should familiarize yourself with it sooner rather than later.

The stream parsing state machine should be in it's own loop doing it's own thing. As packets are detected and checked they should be sent to the loop that is going to do something with it. This way no matter what you are doing with the data right now, whether it is just displaying it, saving it to a file, or whatever, you can change all that later without touching the serial processing code. The code in the serial loop is just sitting there doing it's thing and doesn't know or care what happens to the packets.

Attached is the code saved for 8.5

[guyh2]

Steve, thank you 🙂

I have just spent the last hour and a bit going through your VI and I have learnt so much from it.

A very elegant solution which demonstrates many LV features.

The mindset you have to get into with LV is quite a leap from embedded assembler !!

I will have to modify it very slightly as teh packets only have AAAA tails 🙂

Thanks again

[guyh2]

@Coq Rouge

Thank you for your comments

I did try that, and it works as long as the stream is not already running.

If I start the VI and then start the stream, it ignores the text header and properly processes the packets

BUT if the stream is already running it is not so reliable it seems to overrun sometimes when I switch from reading 1 byte at a time to 20 bytes.

With execution highlighting on, it works 100% at full speed it works 50%

Probably some weird PC biffering issue

[GregFreeman]

---

@COq Rouge wrote:

---

 This does not have to be to complex. Take advantage of the serial buffer. The first step would be to get in synch with the instrument. Read byte by byte until you have received the 0xAA. This can be done with a simple state machine. After reading the two 0xAA. You know that the next 2x9+2 bytes in the buffer will be your data plus synch bytes. As an error check. Check if the the two last bytes read is 0xAA. If not resynch. And so on.

In this case you do not really need a Producer/Consumer structure. As you already have it in the serial buffer. This is 1024 bytes by default (I think). Post your code(NOT as pictures) so we can take a look at it   

---

Ironic, just did this on a recent project. Once you are in sync, you just read the correct number of bytes and as long as the buffer doesn't overflow because you are reading too slow, you should always be in sync. I had a queued state machine, and if it was out of sync, I just sent it back to the resynch case. Simple enough.

[SteveChandler]

---

@guyh2 wrote:

Steve, thank you 🙂

The mindset you have to get into with LV is quite a leap from embedded assembler !!

I will have to modify it very slightly as teh packets only have AAAA tails 🙂

Thanks again

---

You're welcome. I also started out with assembler and it took a while to get my head around how to do LabVIEW right. It is very easy to start writing programs but not so easy to make sure they are scalable and maintainable. Well the same can be said for assembler

Some suggestions. Use the LabVIEW project, use libraries with apropriate access scope to create clean public APIs, use subVIs and create and maintain a reuse library. You can search the forum for all of those things.

The example I posted has three loops. Each loop can be in it's own subVI. Now you run into the question of how to stop all of the loops when your main application stops. There are many ways to do it.

Attached is a project containing the same code that I posted before. The top two loops are now in a subVI. You can see how much cleaner it starts to become. If you ever feel the need to suggest that NI implement a zoom feature then you are doing it wrong. The project also includes a simple module from my reuse library that I sometimes use to stop multiple loops. I use named queues for simplicity and convenience but I typically never use them. I will have an initialization function that stores the queue reference in a shift register but I wanted to keep this somewhat easy to follow.

This does go way beyond your original question but I think some of these concepts will be useful. I sure wish I knew about them when I first started using LabVIEW!