MJPEG stream over RTSP

[yog.muskrat]

This is based on RTSP communication example by GriffinRU. Only instead of saving H.264 stream to file, it converts MJPEG stream to JPEG files according to according to RFC 2435 standard.

What is not implemented:

• Generating quantization tables from Q-factor. Since video-server I'm working with always sends tables itself. There is an example in RFC 2435 appendix A, which is really simple.
  
• Using restart markers. Since I'm not sure why would I need them.
  
• Converting JPEG-string to LabVIEW picture is pretty lousy, since there is no native solution to it, and I didn't want to include extra DLL's.
  

VIs description

RTP and RTSP VIs are described in aforementioned document.

RTP_RTSP - Parse JPEG payload

VI parses the payload, extracts headers and appends image data to previous data until RTP marker becomes TRUE.

Parse main header

Extracts JPEG's main header

Parse restart markers

Reads Restart Marker Header if image type is 64-127.

Parse quant tables

If Q-factor is 128-255 VI reads quantization table header and data from the payload. If quantisation header is present, but tables length == 0, it uses tables from previous payload.

Generating tables from Q-factor can be added here, or in the next VI.

JPEG is marked as ready according to RTP marker.

Create JPEG image

Recreates JPEG image from recieved data. Restart marker header is not used. Huffman tables are created according to example in RFC 2435 Appendix B.

It might be wise to check for EOI marker (0xFFD9) here, but I was in a hurry.

JPEG string to picture

That is just sad. I'm going to try this solution someday, and update this document.

That's it!

[GriffinRU]

Great job,

As far as FFD9 check good idea, if someone would like to use it outside this project, might be very handy.

Converting jpeg string to LV_Image can be done over bitmap API (.NET) or over 3rd party jpeg encoder dll. Do decoding in LabVIEW directly would be less efficient. Check examples here https://decibel.ni.com/content/docs/DOC-24019

If you create Bitmap once and use it as circular buffer you can achive great performance.

-Artur

[yog.muskrat]

Hi, Artur!

It's been almost a year, and I've finally hit the necessity to convert JPEG-string to LV image on the fly. I've tried that solution you've mentioned, and its working great.

However, I can't find a way to use a System.Drawing.Bitmap in a way you've suggested. There's no way to read data from MemoryStream other than to use an appropriate constructor, so I can't reuse the same Bitmap for each frame I receive. Am I missing something?

Thanks again!

[GriffinRU]

Hi,

I am sorry if I confuse you, but jpeg is compressed data of a given bitmap image. To extract 2D image data from jpeg string would require CPU and memory, now if you do not need 2D in LabVIEW than .NET way of getting from JPEG string to BITMAP is very efficient and if you just need to display image with PictureBox than you can replace pictures with "FromStream" invoke mode and "Dispose" the privious one.

If you would like to obtain 2D image array in LabVIEW as LV Image than you must go through new bitmap constructor with every JPEG string. I am sure there is a way to go from JPEG string to the same BITMAP constructor but such mechanism is not easily exposed and would take a lot more programming in LabVIEW and as a result would defeat all the benefits.

-Artur

[yog.muskrat]

Thank you, Artur.

You're a great help!

[GriffinRU]

By the way, if you can switch to PNG than you will have LabVIEW-native way of going from PNG-string to LV Image...

-Artur

P.S.

It would make sense to check performance of extracting 2D data from BITMAP vs getting array from stream after converting from JPEG to PNG within .NET...