H.264 raw byte streams to VLC

[NapDynamite]

Hi everyone

I have a laboratory setup for a communication system prototyping with two FlexRIOs and a PC. I'll be more verbose here.  

Tx:

IP camera --> video encoder (SDI-to-H.264) --> FlexRIO(tx) controller ethernet port --> UDP read in LabVIEW RT --> FPGA modulation backen --> on Air

Rx:

From Air --> FPGA demod backend --> to RT UDP write --> FlexRIO(tx) ethernet port --> to PC Ethernet 

PC:

UDP Read --> getting raw h.264 streams. 

What now I need to do is stream those packets to VLC and playback the video. Since VLC won't just understand the raw byte streams if I send it via localhost, I have looked upon something known as NAL unit handling. My question is, do I need to perform any byte reordering for VLC to understand? 

Any help would be appreciated. 

Regards.

[Bob_Schor · Accepted solution]

Sounds like LabVIEW is being used as a way to get data from your Camera into your PC to send directly to VLC.  If we assume your LabVIEW hardware is correctly passing the data down the chain and delivering it as UDP packets to your PC, then it becomes more a question of how does VLC handle "raw h.264 streams", which is not something that LabVIEW knows anything about.  Incidentally, support for H.264 itself in LabVIEW is hard (for me) to find ...

Bob Schor 

[MarcDub]

I don't think that this is a solution to the problem mentioned here but I have developed using the Libav library (www.libav.org) a dll that could be called from LabVIEW to get a H.264 stream as raw bytes. The dll can actually read a lot of different codecs from files as well. It was 5 years ago and I have not updated the Libav library. I have converted the VS solution and project to VS 2019 and I have spent some time to make sure it still worked. Below is the block diagram of the VI that uses a 2D picture control (no need for Vision toolkit) to display the images.

I have uploaded the C code and the VI with all the required dll's. I hope it might help someone. You should be able to just unzip the C code, open the solution and start from where I left off. The VI should work as is but you have to leave all the dll's in the same directory.

Provided without any guarantee...

[NapDynamite]

Thanks MarcDub,

I sure will try this. 

In my scenario, I am receiving raw bytes from UDP (a stream of strings) and then again I am writing that stream to another UDP socket where VLC is listening. I think there is some problem with the raw H.264 bytes (not actually a problem, I think this is how they are made) is that it doesn't have any header byte for VLC to recognize. When I read a simple .mp4 file (read binary and then converting the string to byte array) and perform the above chain, VLC plays it like a charm. 
I am now on the verge of expanding my knowledge base of H.264 and drop in some header bytes in the h.264 stream and see if it works. 

Regards. 

[MarcDub]

Some of you have complained that the initial H.264 stream connection is unreliable. Here is a small improvement to the VI and dll that made the connection nearly 100% reliable for me.

First, replace the Video_DLL_x86.dll  file with the one in attachment. I have added a reset of the parameters in case of error at initialization. The modified code is also attached.

Second, in the VI, add a For Loop around the Init call as shown below.

I would suggest in case of streaming to put the timeout value at 0. This timeout is mostly to adjust the video rate for files. It might take several seconds to connect to the stream.

Make sure that the firewall is off and that no other network connection can interfere with connection to the stream. 

This code is provided with no warranty whatsoever. 

Hopefully, you will be able to grab that code and fly on your own. You can even upgrade the code with the most recent version of the Libav library and share it in this forum.

Good luck

[NapDynamite]

Thanks MarcDub for this. 

I will try this, make some tweaks and sure will post a blog on NI, targeting some application, probably with an NI hardware in the scene. All that, of-course, duly and thankfully referenced. 

Will update this thread if any trouble arrives. 

regards. 

[armaghan]

hi

Mardub im facing issue while running this vi on stream at given ip and its showing nothing on 2d pic.but its working fine in file option.

[MarcDub]

Did you try with a For loop for the Init function and the DLL I posted in this thread?

In my case, it works with a drone that streams a video. I had the firewall off and it was connected to its own network (I had to turn off the other network connection but I am not sure it was critical). Initially, I had a connection rate below 50% (I had tested only a few times before the initial post). Now, with the For Loop and the modified DLL, it is nearly 100%. Firewall surely prevents connection. I only have one source of h.264 stream for testing. Any chance someone knows an online source that can be used to test h.264 streaming?

[armaghan]

hi Marcdub i tried everything which u said but still it is not working for streaming at udp.

[MarcDub]

I have just downloaded OBS Studio and I started UDP streaming. I was able to read the UDP streaming with the provided VI. All you have to do is to replace "http" by "udp" in the string constant wired to the "Format String" VI.

Notice that you should always start UDP streaming before attempting to read UDP with the VI. If the dll cannot find a UDP stream, it will hang indefinitely and you'll have to restart LabVIEW using Task Manager. It is apparently a documented bug for that version of libav. It was apparently fixed with later versions. It might be a problem if you have firewall issues as well. It is probably better to test with another application like VLC to see if your streaming gets through first before testing with the VI.