how can I determine number of bytes are received At TCP/IP server?

[Y@sh001]

Hi,

I am using TCP/IP protocols to make Communication in between Labview and another application.

From Client Side Code I am creating Frame of ByteArray.

Here my Frame code.
 

#define FRAIMING_BYTES  0xFFDD

 typedef struct {
	uint16_t frame;
	uint8_t command;
	uint8_t channelNum;
	uint8_t deviceNameLength;
	uint16_t dataLength;
	char data[];
}TxPacket;

        TxPacket *pack = (TxPacket*)toSend.data();
	pack->frame = FRAIMING_BYTES;
	pack->command = comm;
	pack->channelNum = channel;
	pack->deviceNameLength = deviceName.size();
	pack->dataLength = data.size();
	memcpy(pack->data, deviceName.toLatin1().data() , deviceName.size());
	memcpy(&(pack->data[deviceName.size()]), data.data(), data.size());

I attach Server Code, which is created in Labview. I have question for TCP Read function. How can i determine the number of receive Byte after TCP Listen.

I am sure I am receiving correct order or Frame, But it is split frames in 10 bytes. 
I would like to determine the exact number of receiving bytes in Single Frame. So I can set the "bytes to read " of TCP read. 

[rolfk · Accepted solution]

TCP/IP communication should never rely implicitely on the number of bytes transfered. TCP/IP gives you a number of guarantees, such that that a whole buffer sent will either eventually arrive at the receiver end, or never, but it does not guarantee you when it will arrive and if it will arrive in one go or in smaller chunks. TCP/IP read will return any data as soon as there is a continous stream of bytes in the buffer without any missing fragments in between but that can be the first few bytes of a much longer buffer, half of the buffer or the entire buffer. You have also no guarantee that all the data has arrived in a certain amount of time, such that you can simply wait this amount of time before trying to read any data and then be sure that you got all the data. TCP/IP operates fully within its promisses if you receive the first few bytes of your message after a few ms after it was send and the remainder half an hour later.

So if you want to communicate over TCP/IP you have really three methods that will work:

1) Fixed size messages, each message has a fixed size, either a fully fixed size for all messages of a particular protocol or a fixed size for each type of message in the protocol, with the type of the message being encoded in the header of the message. This way you can read the fixed header, determine the type of the message and then read the remainder.

2) Variable sized messages with a length indication in the fixed size header part. Here the message consists of a header that contains among other information also of the size of the variable sized data that follows. This can be applied to embedded data frames too, as it is done with the different layers of an IP frame that contain the length of the payload which could be an embedded TCP frame and the TCP frame contains the length of the actual payload which is your data you send. But this length is the length of the actual TCP payload, not the length of your original message which could be longer than a TCP frame (message fragmentation), or also shorter (when multiple data frames are combined into a single TCP frame for instance due to the Nagle algorthme).

3) Variable sized messages with a specific end of line and/or end of message termination character. HTTP is such a protocol where each line of the header is terminated with a carriage return/line feed character sequence and the entire header is terminated with two such character sequences without any other character in between. The body of the message is then either send with a length indication in the header or as so called chunked encoding or as last possibility by terminating the connection.

For your situation I think number 2 is the easiest method. Also please note that your code most likely contains an error. You copy the content of the Latin1 encoded device name into the buffer but use the length for the unencoded device name. That is almost certainly not always the same

Also how do you guarantee that pack->data is always large enough to contain the information you memcpy() into it? Or do you rely on exceptions to not crash your app when such a situation occurs?

Any other method that relies on the implicit length of the data in the receiving buffer is doomed to fail sooner or later and will cause lots of complicated workarounds to make it work anyhow, but these workarounds will never be able to guarantee failure free transmissions, they only may reduce the probability that failures will happen.

[Y@sh001]

Thank you rolfk for Explanation about TCP/IP Protocols. 

"

 You copy the content of the Latin1 encoded device name into the buffer but use the length for the unencoded device name. That is almost certainly not always the same

"

I checked in debugger, and at that time I got correct data and size. However, i change toLatin1() to toLocal8Bit(), I can get surety about single bytes. In addition, Here deviceName type is QString, and Size() method of QString will give me number of letter in string without last Null Corrector. I am sure this is correct. 

 "

Also how do you guarantee that pack->data is always large enough to contain the information you memcpy() into it? Or do you rely on exceptions to not crash your app when such a situation occurs?

"

Thank you for notice this error. I did not handle any crash condition but now I set framing code in try and Catch block. I think it is enough to handle crash.

void SocketCommunicator::SendCommand(TxCommand comm, QString deviceName, quint8 channel, const QByteArray &data) {
	try {
		QByteArray toSend(sizeof(TxPacket) + deviceName.size() + data.size(), 0x00);
		TxPacket *pack = (TxPacket*)toSend.data();
		pack->frame = FRAIMING_BYTES;
		pack->command = comm;
		pack->channelNum = channel;
		pack->deviceNameLength = deviceName.size();
		pack->dataLength = data.size();
		memcpy(pack->data, deviceName.toLocal8Bit().data(), deviceName.size());
		memcpy(&(pack->data[deviceName.size()]), data.data(), data.size());
		emit SendData(toSend);
	}
	catch (...) {
		qDebug() << "error is generated while froming TCP Data frame";
	}
}

[rolfk]

---

Y@sh001 wrote:

I checked in debugger, and at that time I got correct data and size. However, i change toLatin1() to toLocal8Bit(), I can get surety about single bytes. In addition, Here deviceName type is QString, and Size() method of QString will give me number of letter in string without last Null Corrector. I am sure this is correct.  

---

I'm not sure about your library but toLatin1 or toLocal8Bit almost certainly involves some encoding of some sort which might or might not match character for character with your original string which is most likely in Unicode format of some sort. So while the size of the QString will indicate the number of characters (or maybe codepoints) this is not necessarily the number of bytes for the 8bit converted string.

From the documentation:

Returns the local 8-bit representation of the string as a QByteArray. The returned byte array is undefined if the string contains characters not supported by the local 8-bit encoding.

So if you can guarantee that your device name will always only contain data that is representable by 7-bit ASCII characters you are indeed safe, but you know that ASSume contains a three letter word that almost always comes around to bite you in exactly that body part sooner or later If any other characters are possible you may have made the code even worse by choosing toLocal8Bit() and for transmission over a network a local encoding is anyhow almost always wrong because the remote side knows absolutely nothing about your local on your machine and uses its own, which may or may not be Latin1 (codepage 1252).

And you still don't add the length of your string and the whole data block into the stream that you send to the other side, so you still rely on the receiver to guess how much data is arriving.

[Y@sh001]

"

So if you can guarantee that your device name will always only contain data that is representable by 7-bit ASCII characters you are indeed safe, but you know that ASSume contains a three letter word that almost always comes around to bite you in exactly that body part sooner or later  

"

Hahaha You are right. ASSume is dangerous word. I confirm with my supervisor. our device Name is always 7-bit ASCII characters. However, let me figure out something, which is copy string data to Frame. 

"

And you still don't add the length of your string and the whole data block into the stream that you send to the other side, so you still rely on the receiver to guess how much data is arriving.

"

	pack->deviceNameLength = deviceName.size();
	pack->dataLength = data.size();

 This two variable are contain length of device name and length of data block. I will use this two variable on other side to identify device name and  data block length.  

[rolfk]

---

Y@sh001 wrote:

	pack->deviceNameLength = deviceName.size();
	pack->dataLength = data.size();

 This two variable are contain length of device name and length of data block. I will use this two variable on other side to identify device name and  data block length.  

---

Well if these variables get indeed flattened into the data stream your problem about how to determine the length of the data on the other end was already solved from the beginning. From the code you show this was not clear. That implementation detail is hidden somewhere in the TxPacket implementation. But we can't really see that.

So you basically already implemented my solution 2), with the TxPacket being your fixed size header that contains all the information you need to retrieve the reminder of the message. Your only challenge now is to determine the byte size of each data element in TxPacket and to be aware of that your system most likely runs on a Little Endian machine while LabVIEW's Unflatten from String uses the network byte order standard by default which is Big Endian. So all numbers containing more than one byte (Short, Integer, Quad, Float, Double) will be swapped that way.

Solution is to either standardize your data on the sender end to network byte order too, so that your network protocol is independent from the system your software is running, or to set the Flatten from String primitive to expect Little Endian through the according input.

The first option is more complicated unless your QByteArray implementation already supports some endianize configuration, the second is simpler as you will need the Unflatten from String anyhow on the LabVIEW side, but it means that your protocols endianess will depend on the system you run your software on. Since most CPUs that run a common OS nowadays do operate in Little Endian mode, this risk is not so big, but it is something to be concerned of if you design a real protocol and not just some experiment setup for a school project. LabVIEW still knows one platform, the VxWorks based PPC cRIO controllers that run with BigEndian format. But LabVIEW originated on the Macintosh which used Big Endian for both the Motorola 68k and Power PC based machines. Only when Apple joined the dark side and started to use Intel CPUs did they switch to Little Endian format too.

Most ARM platforms could support both Endian modes but most OS implementations for the ARM choose Little Endian probably because Intel does so too.

[Y@sh001]

Thank you rolfk for explaining about the Endianness on different system, and Labview. here I am little confuse about creating packet on sender side. Here My Frame data type is QByteArray on sender side. I am doing copy of Structure (TxPacket) into QByteArray. QByteArray is nothing but "array of Char". So that, Char is single bytes.

I am developing for window 64-bit (Little Endian system). I do not require to take care of Endianness because of Single Byte array. Right? ( while send data from My application -> Labview ) 

whenever I will write data from Labview to My Application through TCp/IP at that time I may need to use flatten to string with byte order 2 (Little Endian). Right?

"

to set the Flatten from String primitive to expect Little Endian through the according input.      

"

Just I am curious to know "set flatten from String" means convert all the other data type to string data type in frame? 

sorry for asking silly question. 

[rolfk · Accepted solution]

Sorry for causing confusion here.

First while the QByteArray is of course simply an array of bytes, the endianess issues happens at the step where you copy the TxPacket header into that array. Unless all the member of that data structure object

		pack->frame
		pack->command
		pack->channelNum
		pack->deviceNameLength
		pack->dataLength

are themselves byte integers, which they are probably not, you do have an endianess issue here. Since you run on a Windows 64 bit system you will use Little Endian format unless you take extra steps to force a specific endian format.

You will need to use Unflatten from String on the LabVIEW side to turn the TCP/IP byte stream (which LabVIEW unfortunately presents as a string) into the numeric values again. The endianess input selector remains the same for both functions.

The nice thing about the LabVIEW Flatten and Unflatten functions is that they can directly operate on clusters too, so you can build a cluster that resembles your TxPacket object by adding the according integers with the correct byte size and in the correct order into it and directly unflatten the TCP/IP stream into this cluster. Determine the byte size of this cluster and wire that value as Bytes to Read to the TCP Read primitive, unflatten into the cluster and proceed from there.

Now you have the deviceNameLength and dataLength values and can do another TCP Read with these values to get the actual data for these components.

[Y@sh001]

Now I perfectly understood. Thank you so much rolf for explaining in very detail.