producer consumers questions

[tysonl]

I am developing an application that reads and processes serial data from a prototype device.  The data is transferred from my device at 160 Hz and is comprised of about 200 bytes of data. Each data packet has header, data, and checksum bytes.  A illustration of my data packets would be the following: [Header bytes][Data bytes][Checksum bytes].

The main design pattern I am using in my application is the producer consumer design pattern.  The producer reads in the data byte-by-byte (via VISA Read) and enqueues it for the consumer loop.  The consumer loop dequeues the data byte-by-byte and then passes each byte through a state machine that finds the header, forms the data bytes into an array, and then finally parses, processes, and saves the data. The states of my state machine are:  find header, build packet array, and process and save.

I've used this approach in the past and it has always worked for me, but I've always wondered if it is the best solution for this type of application.  Are there better solutions out there?  Reading and processing data in byte-by-byte doesn't seem like a good idea, but it is simple and straightforward. 

I have been considering pushing all of the parsing, checking, processing, and saving into a third loop.  In this approach, my current consumer would act as both a consumer and a producer.  It would consume data by finding packet headers and building data packet arrays, and it would produce data by enqueuing data packet arrays for a new, second consumer loop.  The reasoning behind this approach is that it would separate tasks and allow me to only have to change the second consumer for different projects that use my base hardware.  Is this kind of approach reasonable or is it too complex and/or ill advised?

[Mark_Yedinak]

First, reading data one byte at a time is inefficient. You really should be reading in chunks and passing that to your parser/processing code. Use a shift register on your parsing code so that you can concatenate new data to any data you have left over from the last buffer that was posted for processing. This is necessary if the chunks you are reading may not read all of the data.

I have used both two and three loop implementations. You have to look at your data rates and how much effort is required to parse and process the data. Since you are using a serial port you should have no problem doing the parsing and processing in the same state machine.

[tysonl]

If I read in and enqueue chunks of data (byte arrays), then how do I find the header bytes inside a chunk?  The only way I know of to do this is to search the cunks byte-by-byte. 

For example, if I read in and enqueue eight bytes at a time, then one read could produce the following:  [122, 13, 1, 2, 3, 4, 5, 6].  If my header is: [1, 2, 3, 4, 5, 6, 7, 8], then how do I process the read and enqueued byte arrays so that I can build my data packets?  In the example read, the first two bytes in the array appear to be the last two bytes of the previous packets checksum.  The last six bytes appear to by the first six bytes of the new packets header.  I don't know how to deal with this situation other than examine the data byte-by-byte.  I can't enqueue byte arrays, but dequeue single bytes can I? 

[Mark_Yedinak]

If your header is well defined and constant you can use regular expressions to find your header. When using this approach you leave the buffered data as a string. You may also use the Scan String for Tokens as well. Generally once you find your first data packets all subsequent data packets follow immediately in the buffer. This type of processing can be done inside a state machine.

[tysonl]

Will something like this work?

It looks like it should, but Match Regular Expression is throwing null character errors.