Best method for reading and storing TCP data sent by server?

[TestEngineer11]

Hello,

I am writing a TCP client DLL which is designed to read/write data with my UUT (which acts as the server). I am using the CVI TCP library. Per the examples of the library, I will use a synchronous callback function which will process messages that I receive as a client. This callback function will execute ClientTCPRead() when data is available as seen in below snippet. As you can see, when data is ready, the read fucntion will store the data in the global g_recieveBuf array which is of type uint32_t. 

int CVICALLBACK ClientTCPCB(unsigned handle, int event, int errCode, void *callbackData)
{
    int status = 0;
    switch (event)
	{
        case TCP_DATAREADY:
			g_dataReady = 1;
			if ((status = ClientTCPRead (handle, g_receiveBuf, dataSize, TIMEOUT)) < 0);
			{
				reportTCPErr(status, g_errBuf); 
			}
      		break;
			
        case TCP_DISCONNECT:
            MessagePopup ("TCP Client", "Server has refused or closed connection!");
            break;
    }
    return status;
}

Question 1:

when it comes to large messages that I receive and due to other TCP timing issues, I will often need multiple calls to read all the data. I believe the callback will handle this case. The case TCP_DATAREADY will continue executing until all the data is read. This, I believe is the utility provided by this library and what makes it such an easy process. My question is how do I properly concatenate data into my global array across multiple calls? I don't want to overwrite my array each time ClientTCPRead() is executed. For example if one large message is being sent by the server and multiple executions of the callback function take place to read all that data, I want it all captured properly in my array as one contiguous message. One of my colleagues suggested using a List construct (programmer's toolbox library). I don't quite understand what those are or what the advantage is of using them but any advice to solve above problem would be appreciated. 

Question 2:

The data coming from the server is in binary format. It's a proprietary message structure with a known header size. The header also contains a field that tells me the total byte count (header and body) of the sever message. Is there any advantage to knowing this in terms of properly reading the data? I'm thinking that it doesn't matter because I am using the in-built callback function which will always read data if it's available. So why do I care how many bytes are coming if the callback function ensures it will all be read and stored in my global array (or other construct such as list if I use that method). 

Thank you in advance. 

[RobertoBozzolo · Accepted solution]

Answer 1: You could try using the reading method suggested in the function help,  i.e. reading in a loop until the whole message has been read. Receiving the size of the message at the beginning will help you to read the entire message from the server. If you prefer to rely on the callback mechanism, you will need to store somewhere the index in the array and fill it in chunks upon repeated calls of the callback. An alternative could be to to use a queue (look at Thread Safe Queue functions in Utility Library >> Multithreading) but again the size of the message will be useful for the queue reader to properly rebuild the function upon repeated calls

Answer 2:I can think of two scenarios where the data size is relevant. First of all you may be receiving messages with different sizes. This may not be your case but in general it is a good thing to be prepared for a large number of possibilities. Second, the server could enqueue more than one message one after the other, so you may find yourself to read the second message immediately after the preceding one: in this case knowing the size of each message will help in splitting them. Again, it may not be your case if you are in a query-answer scenario but this may not be the only possible one.

[TestEngineer11]

Thanks Roberto. I am going to use the size of message data to drive how I read the message as suggested.

[Bertrand_ENAC]

Thank you Roberto.

Concerning answer 1.

I had seen in the function help how to get the complete message.

But how do I get at the beginning the size of the message?

Thank you

[RobertoBozzolo]

As you see in MessageReader example, first of all a message is sent with the size of data to exchange, then actual data are sent so that the reader is able to rebuild the complete message.

[Bertrand_ENAC]

Hi Roberto

Thank you for your quick answer. 

What would we do without you?

Unfortunately the MessageReader example is not working in my case because the number of data is not given at the beginning of the stream.

I have to find another way to manage this issue.

Bertrand

[RobertoBozzolo]

What I have done in the past is something along this line (some sort of pseudo-code, missing error checking: use it only as a guide).

I had to transmit different messages each with a different length. I formatted messages in a readable string even for binary data. Each message includes data length at the beginning.

Prepare and send a message (WRITER application):

	char	buf[512], msg[512];

	memset (msg, 0, 512);
	memset (buf, 0, 512);

	sprintf (buf, "%d;%d;%d;", SomeData1, SomeData2, SomeData3);
	// Binary data formatted in 2-bytes, zero-padded hex numbers without separators
	Fmt (buf + strlen (buf), "%*d[zb1w2p0r16]", sizeof (SYSTEMTIME), (char *)&DateTimeVariable);
	sprintf (msg, "%d;%d;%s", TypeOfMessage, strlen (buf), buf);
	err = ServerTCPWrite (handle, msg, strlen (msg) + 1, 1000);

Receive the message (READER application):

// Part of the Tcp callback:
case TCP_DATAREADY:
    if(ClientTCPRead(tcpH, buf, TCPBUFSIZE, 250) < 0) {
    	// Error receiving messages
    	return 0;
    }
    Scan (buf, "%d[x]%d[x]", &TypeOfMessage, &NumberOfChars);
    // Read from the tpcserver until received 'NumberOfChars' characters
    // .....

    // Strip out message header
    x = NumFmtdBytes ();
    memmove (buf, buf + x, TCPBUFSIZE - x);

    // Pass the message to interpreting routine
    InterpretTheMessage (buf);
    break;

Interpret the message (READER application):

void InterpretTheMessage (char *buf)
{
	char	*el;

	el = strtok (buf, ";");  sscanf (el, "%d", &SomeData1);
	el = strtok (NULL, ";"); sscanf (el, "%d", &SomeData2);
	el = strtok (NULL, ";"); sscanf (el, "%d", &SomeData3);
	el = strtok (NULL, ";");
	Scan (el, "%*x[b1w2]", sizeof (SYSTEMTIME), (char *)&DateTimeVariable);

	return;
}

[RobertoBozzolo]

---

@Bertrand_ENAC  ha scritto:

Unfortunately the MessageReader example is not working in my case because the number of data is not given at the beginning of the stream.

The corresponding writer application sends an array of n+1 elements whose first element holds the number of actual data sent (n). The reader reads the first integer and then iterates reading up to 'n' elements.

[Bertrand_ENAC]

I cannot change the received data format as the data are comming from an external device.

What I did is to check if the received data set is smaller than the reciever buffer.

If the size of the received data set is equal to the size of the data buffer I guess there are additionnal data available and I continue the reading process until the size of the received data set is smaller  than the receiver buffer.

I don't know if it is a good method, but for the time being it seems to work.

[rolfk]

It's a hacky way of implementing communication.

Normally a sender has three valid methods of sending data:

1) Use of some soft of end of message indicator. In HTTP this is for instance always an \r\n followed immediately by another \r\n. Some simpler protocols use a single \r\n or EOT (03) at the end and with SOT(02) at the beginning.

2) The message is fixed size

3) The message uses a header with one of the above methods and one of the parameters in that header indicates how much of data is to follow.

Anything else is an utterly hacky protocol whose developer should be flogged, feathered and tared. 😁