Video transmission from one computer to another computer over TCP

[billko]

---

@constructionworker wrote:

---

@Mark_Yedinak wrote:

I would still recommend a rewrite of the code. The server logic is bizarre to say the least. A true server would have a task which simply listens and spawns next tasks as clients connect. This is not necessary if the server supports one, and only one client at a time. You should not use the timeout event as a trigger to actually read the data. this is a very strange construct. As mentioned earlier, a basic protocol would help quite a bit. For instance, every message you send would include a header which contains a message type and a length field. The length field would specify the amount of data which is contained in the message. So, the server could simply read with a timeout of -1 waiting for x number of bytes. X is the size of your header. If the message ID is a U16 and the length is a U32, your header is 6 bytes. After reading this you check the length field to see if there is more data. If so, read it. If not process the message and then go back to waiting for the read of a message header. There is no strange "look for a timeout which will then trigger me to actually read data" construct. If necessary, you can put a short timeout on the read of the header which simply allows the server to do some other processing between receiving messages. However, hear when you receive a timeout you clear the error, do some basic task/processing if necessary and then go back to waiting for a message header.

This is a much cleaner server implementation than what you have posted. It also has a basic protocol which allows you to have a variety of message types. Other messages could be an exit/stop message, clear buffers or any other type of processing that may be needed.

---

To be honest, I changed the server and client logic a little bit for me.
Let me explain the type of application I want to have as follows;
Application on first computer:
-It is the application to run first. The application will start running and wait for the second computer to send the videos.
-If the application on the second computer starts running, it will display the number of connections to understand that the application on the second computer is connected.
If the videos start coming, it will start sending some numerical information.

Application on Second Computer:
-It is the second application that will work.
The application will run, send the connection information to the first computer and start reading and sending videos from the cameras.
-It will display the incoming numerical data on the screen as an extra.

I'm trying to understand your suggestion to rewrite the code, it sounds like a cleaner implementation.

But I suspect there is some difference between the method you suggested and the application I want to create.
Even if the application on the second computer has a problem and closes, I do not want the application on the first computer to close. I tried to explain the application I want to create above in detail. I added the generated code.
If you can make changes to the code, I can try so I can better understand what you mean.

-I removed the timeout case structure.
Thank you for all your suggestions.

---

Really, it doesn't really matter what your applications do.  The communications protocol is part of the very underpinnings of the whole application suite, and if it is not robust, your applications won't be, either.

Setting up a listener is pretty common, and there is no reason to reinvent the wheel.  Start with a robust protocol, and implement a standard communication layer.

[Bob_Schor]

I've done something similar to what you are trying to do.  I don't have the exact code in front of me (since I'm not doing exactly what you want), but I think I can describe the process that I'm 95% confident will do what you want.

First, here's the "one-computer" method (be patient, it will become clear).  This is a simple Producer/Consumer Design Pattern.  The Producer loop opens the Camera and starts taking Frames.  As each Frame is acquired, it is put into a Queue (or Channel-Wire Stream) and sent to the Consumer Loop.  The Producer Loop also reads the "Stop" button, and if it is true, sends a "Sentinel" (a signal that means there are not more data coming to the Consumer, so the Consumer can exit when it receives the Sentinel) and exits (the Producer).  This guarantees both Producer and Consumer stop at the same time.  For a Network Stream, the Sentinel is built into the Stream Writer (and Reader).  If I'm using Queues, I use a "blank" input to the Queue (not quite certain how to do this for an Image, but we can figure this out later).

Now let's place the Producer on Computer 1, and the Consumer on Computer 2.  Run the same code, except Computer 1 has only the Producer Loop, and Computer 2 has the Consumer (plus other stuff, of course).  The Images are transmitted by a Network Stream, configured to send Images.  You'd need to use something for a Sentinel, but the logic is the same, except the Network Stream replaces the Queue.  You may want two more Network Streams to send "commands" from 1->2 and from 2->1 (you can "fake" a sentinel this way).

Bob Schor

P.S. -- I notice @billko suggests settling on the Communications Protocol.  I agree, and would suggest Network Streams, as they are robust, flexible, and (relatively) easy to set up (once you understand the protocol -- help is available here on the Forums, if not in the Help files).

[constructionworker]

---

@Bob_Schor wrote:

I've done something similar to what you are trying to do.  I don't have the exact code in front of me (since I'm not doing exactly what you want), but I think I can describe the process that I'm 95% confident will do what you want.

First, here's the "one-computer" method (be patient, it will become clear).  This is a simple Producer/Consumer Design Pattern.  The Producer loop opens the Camera and starts taking Frames.  As each Frame is acquired, it is put into a Queue (or Channel-Wire Stream) and sent to the Consumer Loop.  The Producer Loop also reads the "Stop" button, and if it is true, sends a "Sentinel" (a signal that means there are not more data coming to the Consumer, so the Consumer can exit when it receives the Sentinel) and exits (the Producer).  This guarantees both Producer and Consumer stop at the same time.  For a Network Stream, the Sentinel is built into the Stream Writer (and Reader).  If I'm using Queues, I use a "blank" input to the Queue (not quite certain how to do this for an Image, but we can figure this out later).

Now let's place the Producer on Computer 1, and the Consumer on Computer 2.  Run the same code, except Computer 1 has only the Producer Loop, and Computer 2 has the Consumer (plus other stuff, of course).  The Images are transmitted by a Network Stream, configured to send Images.  You'd need to use something for a Sentinel, but the logic is the same, except the Network Stream replaces the Queue.  You may want two more Network Streams to send "commands" from 1->2 and from 2->1 (you can "fake" a sentinel this way).

Bob Schor

---

Thanks for your suggestion.
I understand what you want to say, especially it makes sense to use the producer consumer model, but I want the application of the first computer not to close even if the application on the second computer (cannot read the video or similar) closes.
Because if the second computer shuts down, the application on the first computer will be running during the re-run and will wait for data to come from the second computer.

---

@Bob_Schor wrote:

P.S. -- I notice @billko suggests settling on the Communications Protocol.  I agree, and would suggest Network Streams, as they are robust, flexible, and (relatively) easy to set up (once you understand the protocol -- help is available here on the Forums, if not in the Help files).

---

I'm having trouble understanding some things about how to send videos as there are no examples of sending videos using network stream.

Thanks for your suggestion.

[rolfk]

I’m not quite understanding your reasoning. You seem to try to defend a poor choice for a flawed protocol by some requirements that you made up. In reality those requirements are not contradictory to a properly designed protocol, rather tbe opposite

As to missing video streaming examples, that has a very specific reason. Doing video streaming properly is an inherently difficult task that can only be done efficiently with data compression schemes. Those data compressors are however often patent encumbered AND at the same time need active interaction with the higher level protocol layers to work efficiently. This makes them either very complicated to implement, which goes far beyond what an example is supposed to be or pretty useless. And you need an army of lawyers to determine who you need to pay royalties to in order to be allowed  to distribute such an example. The most sensible choice simply is to not create such examples.

If you do you will be damned one way or the other!

[Bob_Schor]

Let me try to answer your questions.

Here is how I have structured my "two computers/one cooperative task" routines using Network Streams.  I'm going to name the machines "Host" and "Remote".  These names are taken from the terminology that NI uses for Real-Time Projects, where the Host is a PC running LabVIEW and the Remote is, say, a cRIO running NI LabVIEW Real-Time Linux.  But in this situation, the Remote is just another instance of LabVIEW running on a PC.

• I've structured both the Host and the Remote routines as a State Machine implemented as a Message Handler.  I'm a Channel Wire enthusiast, so I code the State Machine as a "Channel Message Handler", but it can also be implemented using a Queue as a "Queued Message Handler"
  
  • The Message consists of a State (which I represent as an Enum, with the first State being "Init" and the last one "Exit", with one called "Error" near the end), and a variant called "Data" that is often empty.
  • Inside the State Machine's While Loop, you Channel Read (for a CMH) or Dequeue (for a QMH)  the incoming Message, passing the State to the Case Statement (to select the State code-to-run) and also passing in the (optional) variant Data.
  • Within each State, you "do what is necessary".  Typically, at the end of the State, you Channel Write (or Enqueue) the next State.
• Sometimes State transitions come from interactions with the Front Panel controls (e.g. pressing the Stop button).  This gets implemented by an Event loop (which I put just above the Message Handler Loop) that "enqueues" the appropriate State to the State Machine.  [You can find examples of the QMH and CMH, I believe, in the Examples or on the Web].
• In your situation, you want to have two independent State Machines, running on two independent PCs, handling a one-way transmission of data (frame images from a camera) from Remote to Host.  To do this, I would create three Network Streams, since a Network Stream is a one-way transmission of a specific type of data from Host to Remote (H->R) or Remote to Host (R->H).
  • The first Network Stream is a H->R Stream of "State" Messages for the Remote.  The "State" part of the Message is the State you want to run on the Remote, while the "Data" is any incidental data the Remote might need to carry out the duties of the State.
  • Reciprocally, the second Network Stream is a R->H Stream of "State" Messages the Remote sends to the Host when it wants the Host to undertake a particular action.
  • Finally, the third Network Stream is another R->H Stream of "Frame" Messages where the Data part of the Message consists of successive Image Frames from the Camera, direct from the IMAQdx Read.  When the Host receives the Message (which would direct it to a State maybe called "Get Frame from Remote"), it would look at the accompanying Data, use Variant to Data to transform it back to the appropriate IMAQdx Image Frame, then display it on the Host, write it to disk, or whatever else you need to do.
• Note that when you establish a Network Stream, you define the type of data that the Stream will contain.  The first Stream (H->R State) consists of Remote State clusters, the R->H State Stream has Host State clusters, and the R->H Data Stream has IMAQdx Frames.  The first action you need to do is to get Host and Remote to "talk to each other" and create the three Network Streams.  Here's my procedure:
  • The Host starts and starts trying to connect with a specific Remote (whose IP you know and provide).  I typically provide a 15-second time-out to get all three Streams connected, with the ability to "give up" and exit gracefully if I don't connect in a certain number of tries.
  • The Remote starts and "listens" for anyone connecting to it using Network Streams.  I also typically provide 15'second time-outs, but just keep looping "almost forever" (or until I push the "Quit" button on the Remote).
  • These routines run either in the "Init" States of the Host and Remote, or run before entering the State Machines (by issuing the respective "Init" Message, which is what I usually do).
• Now you have your two machines running independently.  How do you want your Host to "control" the Remote?  If you have a "Start Recording" button on the Host to tell the Remote to "Start Recording", then you'd do the following:
  • In the Host, have an Event that monitors the Start Recording Boolean control (a "Rectangular Button" with Latch Until Released mechanical action) and sends the Host to the "Start Recording" Host State.
  • In the Host "Start Recording" State, you send the Remote the "Start Recording" Message, along with any parameters the Remote might need.
  • The Remote needs to have a parallel Loop running that "listens" for Network Stream H->R Messages, and when it finds one, it reads it and puts it on the Remote's Message Channel (or Message Queue), where it goes to the "Start Recording" State.
• You know, I've been writing this for about 30 minutes, and just realized I might not need the R->H Data Stream!  In the former Bullet Point, the Remote goes to the "Start Recording" State.  The next State for the Remote can be "Get and Send Frame".
  • "Get and Send Frame" can get a Frame from the Camera and bundle it into a R->H Message to the Host with the State being the Host "Save Frame" State and the Data being the "Frame".
  • Similar to the Remote, the Host needs a parallel Loop listening to Host Messages from the Remote. When it gets a "Save Frame" Message, it reads it and puts it on the Host's Message Channel (or Queue), and in the "Save Frame" State, does whatever you want to do with the Frame.
• You mentioned not wanting one computer stopping to (necessarily) affect the other.  That's basically the case -- unless you tell the Host to send an "Exit" Message to the Remote, the Remote can keep running until it shuts itself down (or someone "pulls the plug").  Similarly, stopping the Remote doesn't directly affect the Host.  One thing you do need to consider is the effect that an unexpected shutdown of Host or Remote can cause -- the other side might just "hang", waiting for a response or acknowledgement.

Hope this has been helpful.

Bob Schor

[constructionworker]

Thank you for your answers. By the way, I had to read it many times 🙂 I tried to develop a small code as far as I understood from you.

I could not create exactly what you said because I think it is necessary to proceed step by step.

The method you describe is exactly what I want, but I'm having some trouble creating it.
- VI's working as local hosts on the same computer do not transmit data streams on different computers. There is no problem with the network or anything.I encounter this problem when I try to send a normal data.
I don't know where the problem originates from.

I have included example below.

[constructionworker]

---

@rolfk wrote:

I’m not quite understanding your reasoning. You seem to try to defend a poor choice for a flawed protocol by some requirements that you made up. In reality those requirements are not contradictory to a properly designed protocol, rather tbe opposite

As to missing video streaming examples, that has a very specific reason. Doing video streaming properly is an inherently difficult task that can only be done efficiently with data compression schemes. Those data compressors are however often patent encumbered AND at the same time need active interaction with the higher level protocol layers to work efficiently. This makes them either very complicated to implement, which goes far beyond what an example is supposed to be or pretty useless. And you need an army of lawyers to determine who you need to pay royalties to in order to be allowed  to distribute such an example. The most sensible choice simply is to not create such examples.

If you do you will be damned one way or the other!

---

I took this example from the shared examples in labview, apart from the video sending and receiving part. I just changed the multiserver TCP example a bit.

Do you mean that my delay in sending and receiving videos is not due to the generated code method?
Are you talking about the need to use the compression method, which is a very difficult task.

[Mark_Yedinak]

I was feeling very generous so I rewrote your server and modified the client. This still is not the way that I would really do this. Since your system is not a command response type of protocol I would use separate connections for the data coming from the server to the client and the image data being sent to the server. Using the single connection like you are both data streams are very dependent on each other. That is, the data rate for the server data is synchronized with the image data being received. I don't have the IMAQ stuff installed for 2019 so I didn't do anything with that portion of the client. I would not have both image captures in the same loop. I would use a separate loop for each one.

[constructionworker]

Thanks for your generosity, but there's a bit of a problem.

I am using 2018 labview by the way.

[Mark_Yedinak]

Here you go.