Example Code

RTSP over TCP with LabVIEW

Code and Documents

Attachment

Overview

LabVIEW-based RTSP over TCP, RTP methods: DESCRIBE, PLAY, OPTIONS, TEARDOWN, SETUP. Other methods can be developed in similar fashion and other stream types can be saved. Complex interfaces with over the net control can be developed, RTP messages can be send to control remote devices (PTZ, focus, firmware updates, device health-monitor...)

Tested with following IP cameras:

Constants for the following cameras provided as:

Sanyo HD cameras: VideoInput/1/h264/1 (up to 4 streams available)

Axis cameras stream: axis-media/media.amp?videocodec=h264

Arecont camera stream: h264.sdp?

Note: names above are defaults names, user can modify them. Correct name can be extracted from DESCRIBE method reply.

Warning: no anonymous access

Description

Real Time Streaming Protocol from Wikipedia.

DESCRIBE

describe.png

SETUP

setup.png

PLAY

play.png

TEARDOWN

teardown.png

EXAMPLE

example.png

  • Describe -> Setup -> Play
  • Parsing RTP header
  • Parsing RTP Payload
  • Saving data to file
  • Teardown

NOTE:

H264 file can be re-encoded to any format by VLC, Gstreamer or Live555.

Can be saved to ts.file with proper formatting (might add later)

Frames can be viewed with ffmpeg (might add later)

By writing to the file with every RTP payload further improvement can be achieved...

Steps to Implement or Execute Code

  1. Set Camera's IP Address
  2. Set File path
  3. Run Example

Requirements to Run

Software

  1. LabVIEW 11.0 or later
  2. Base64 utility

NOTE:

Can be down converted as low as LabVIEW 8.5 (For-loop conditional terminal). With re-programming around "conditional terminal" can be pushed lower...

-Artur

Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.

Comments
rajez
Member
Member
on

Hi GriffinRU,

i also want the same thing but for another IP Camera (dahua IPC-HDW2100). I had downloaded your program and put my RTSP address but it is not working. But the RTSP is working with vlc.

plz help me to find required commands to tcp in labview.

GriffinRU
Member
Member
on

Hi,

I do not have such a camera on-hand, but with manual or web search you should be able to tweak my example to work with your camera.

I can help you, but I need access to your camera. You can provide link and credentials over PM or here.

-Artur

rajez
Member
Member
on

Thanks for replay.

I had read about RTSP in this link "http://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol". And they are saying about protocol directive which if you send to server it will respond with some information.

My doubt is how we can send and recieve these directives( i require a basic solution because i am just a biginner in networking and protocols). I ran simple labview program to communicate to labview it is showing time out error.

RTSP_error.PNG 

GriffinRU
Member
Member
on

Hi,

Looks like you are missing CR and LF as line terminators and double termination at the end of request...

-Artur

GriffinRU
Member
Member
on

Any success?

-Artur

yog.muskrat
Member
Member
on

Hi, GriffinRU!

Could you, please, explain, how could I display H.264 video, acquired with this solution, in LabVIEW?

Thank you!

yog.muskrat
Member
Member
on

And I mean in realtime!

GriffinRU
Member
Member
on

Hi,

You need to re-construct H264 datagrams...

I do no see any reason to do that in LabVIEW directly but you can try.

There are 3rd party solutions like DShow, VLC, GStreamer, QT, FFMPEG and probably more where you can pass packets and receive image.

Idea here was to capture raw data and post process later.

-Artur

yog.muskrat
Member
Member
on

You see, what I need to do, is to capture a stream from camera, add some informational layers (like timestamp, zoom level, etc) and display the result in a LabVIEW app.

I used to work with a camera that provided MJPEG stream via simple HTTP request, and that was  simple. But now we have another camera, that only has a H.264 RTSP stream which I can capture and save, thanks to your code, but have no idea how to display. So I'd really appreciate if you could shed some more light on how to display that video stream.

Thanks!

GriffinRU
Member
Member
on

Hi,

Can you share camera brand and type? I haven't seen a camera yet without jpeg stream...

You will get timestamp from RTSP stream but not sure about other camera properties/parameters. Some camera brands place this info into jpeg footer some not. With RTSP stream you will be limited to parameters coming from DESCRIBE request only and datagram timestamps.To get more you need to check camera documentation and get parameters over HTTP requests, like you done before. And, once you get this parameters, you can overlay them anyway you like in your LabVIEW app, even over 3rd party indicators.

-Artur

P.S. You can open multiple sessions to the same camera...

yog.muskrat
Member
Member
on

By "informational layers" I meant "informational graphic layers" i.e. text and geometrical shapes. So, basically, I nedd to to draw some stuff on a video frame )

That camera I've mentioned is actually a network video server. (this one). It's supposed to have a MJPEG stream, but it's firmware is pretty broken, so it won't work. Manufacturer's techsupport was unable to help.

Soooo... we're down to two options now - to use H.264 stream, which is working properly, or to buy another video server. The latter, obviously, is the correct way to do things, but, unfortunately, it'll require a lot of additional paper and designer work.

And therefore!

I need to somehow decode H.264 compressed frames to a LabVIEW-friendly format.

GriffinRU
Member
Member
on

Ok, I will check what I can do.

Meanwhile looks like your server is ONVIF complaint, so check ONVIF Streaming Specs on how to retrieve information and jpeg.

-Artur

yog.muskrat
Member
Member
on

I will do that. Thank you!

GriffinRU
Member
Member
on

Итак, перед тем как я пойду рулить с avcodec_decode_video хотелось бы все-таки уточнить, что необходимо получить в конце?

У нас имеется rtsp стрим.

Что с него взять:

  • таймстэмп
  • сжатый поток (картинка, звук, текст...)

Что нужно:

  • таимстэмп
  • параметры камеры (видео ресурса)
  • другие атрибуты видео потока (отсутствуют в rtsp стрим)

Цель:

  • Сохранить поток
  • Отобразить текущее состояние видео/айдио регистратора
  • Real_time latency?

Сохранить поток в формате (h264) можно с представленным примером.

Собрать дополнительные данные об источнике необходимо из докуметации ONVIF.

Я думаю, что получив картинку из H264 стрима вы потратите значительное время в LabVIEW для наложения 2-3 строк текста информации и в лучшем случае удастся получить 5 к/с.

Почему бы вам ни сохранить стрим в h264, а затем проиграть его в VLC с overlay того что вам необходимо, с задержкой скажем в 100-200мс,

-Артур

yog.muskrat
Member
Member
on

Дело в том, что в LabVIEW приложении реализовано управление набором различных устройств, и в т.ч. камерой, которая должна наблюдать за экспериментом. Поэтому видеокадры мне необходимо в первую очередь максимально оперативно отображать на экране, а потом уже сохранять на диск. Поэтому, к сожалению, вариант с VLC не подходит.

Всю дополнительную информацию для отображения я беру из внешних источников (PTZ от самой камеры, заранее созданные картинки с масштабными сетками и т.п.), поэтому по сути, от видеосервера нужен только видеопоток.

Вобщем, если кратко, то мне представляется такая последовательность:

1. Стартую свое приложение LW

2. С помощью вашего решения шлю RTSP запрос к видеосерверу.

3. Получаю от него видеопоток.

4. Каждый полученный кадр (если хватит производительности) декодирую в знакомы LW формат.

5. Средствами LW накладываю на этот кадр нужную мне информацию.

6. Отображаю этот измененный кадр средствами LW.

7. GOTO 4.

Спасибо, что тратите свое время, Артур.

Если что, я смогу сам подцепить к LW DLL'ки от ffmpeg, но т.к. прежде мне с ним работать не доводилось, то я уму не приложу какой там должен быть порядок действий и какие функции меня должны интересовать.

Снова спасибо )

yog.muskrat
Member
Member
on

ВНЕЗАПНО мне удалось вытащить MJPEG поток с камеры.

Во время последней отчаянной попытки выбрать из сервера хоть что-нибудь еще, я просканировал его порты и обнаружил, что помимо 554, по которому я получаю H.264 поток, у него открыт еще и 1554. С этим портом прежний RTSP запрос успешно стартует нужный поток.

Поэтому теперь моя задача, судя по всему, сменилась, и теперь мне нужно по аналогии с вашим разбором H.264 NAL unit написать парсинг для payload типа 26.

Я правильно понимаю, что мне для этого нужен вот этот документ?

GriffinRU
Member
Member
on

Я уж собрался с вами FFMPEG замочить в LabVIEW , а тут так подфортило вам...

Насчет прицепленного документа, надо посмотреть документацию и на ответ сервера, там явно будет указано по какому стандарту упакован MJPEG, другими словами говоря - вы на правильном пути, но 100% гарантии дать ни могу без предварительного просмотра пакета.

-Артур

P.S. По поводу FFMPEG , если будет желание, то начинать надо с закачки development архива и работать с примером encoding/decoding и гуглить avcodec_decode_video2...

Но скорее всего лучше весь процесс от соединения с камерой до прокрутки кадров в FFMPEG, а потом попробовать качать данные из LabVIEW.

yog.muskrat
Member
Member
on

Я вас понял, Артур. Еще раз спасибо за потраченное время и данное решение, мне все это здорово помогло.

Кстати, насчет того, чтобы весь процесс поединения и получения кадров вынести в ffmpeg. Я, пока на это вот ваше решение не наткнулся, планировал для этих целей openCV использовать. Но очень уж не хотелось ради одного кривого видеосервера во всем этом разбираться.

Вобщем, буду пробовать писать разбор JPEG over RTP. Если интересно, потом выложу результат.

GriffinRU
Member
Member
on

Good,

Check this one:

https://decibel.ni.com/content/docs/DOC-36763

It can be optimized alot, but simple from LabVIEW stand-point, I might re-write it over the week-end

And, yes, any success on JPEG parsing would be great asset for this post.

-Artur

yog.muskrat
Member
Member
on

Ok, so I've come up with a working version of MJPEG over RTP via LabVIEW.

Should I create a new post, or upload it somewhere and post description here?

GriffinRU
Member
Member
on

Good,

It is up to you, but I think it would make sense for you to create your own post and crosslink the together.

-Artur

yog.muskrat
Member
Member
on

Here it is.

GriffinRU
Member
Member
on

Thank you,

I spent some time with FFMPEG and if you would like i can share some vi's for ffmpeg.exe and ffplay.exe over process + pipes.

Programming over .dll requires alot, but possible as well, I can share startup vi's.

-Artur

mgtrtc
Member
Member
on

Hello Griffin.

I noticed that your .vi does not include the audio track.  How difficult would it be to include the audio track to the H.264 binary file so that the VLC media player could run the AV file. 

GriffinRU
Member
Member
on

Hello,
It would require to parse audio packets and interleave them with video payloads.
Adding example on how to do that, might take some time.
Artur

mgtrtc
Member
Member
on

If it's possible, I would like to see an example on how to interleave the audio packet with the video.  I'm using a Vivotek FD8151V and I need to stream both video and audio to file.  I appreciate your help!

Mark

mgtrtc
Member
Member
on

Hello,

If possible I would like to see an example of how to interleave audio and video. I'm using a Vivotek FD8151V and I would like to understand how to stream this cameras H.264 video and audio to file using LabVIEW! I appreciate your help!

Mark

Sent from my iPhone

GriffinRU
Member
Member
on

Mark,

As I said, it might take some time, I plan to get back to my computer in January...

If you need sooner, I would suggest save to file audio stream only first, as wav-file.

  • create wav-file header
  • parse incoming packets
  • write to file

-Artur

mgtrtc
Member
Member
on

Artur,

Ok, I understand.

I will be looking forward to more of your posts.

Mark

HENIGMAN
Member
Member
on

Hello,

I used your RTP_RTSP vi with success. It is working smoothly with IP camera ENEO with FULL HD 30 frames/s. I have no much experience in programming because I am specialized on low noise analog front-end sensor amplifiers.

What I need. I would like to view Video from camera. I would like to add Text to this video in real time ( data measured and processed with LabView ), then the Video with overlayed Text should be saved. Have you idea how to solve this problem.

Have a nice day

Franci

GriffinRU
Member
Member
on

Hi,

I don't think it is possible in LabVIEW to decode H264 packets into Bitmaps while text/image stamping at full HD rate.

I would suggest to do that afterwards with 3rd party tools like VLC, FFMPEG, Gstreamer or such.

H264 decoding and encoding is complex enough and CPU hungry, while LabVIEW is not efficient in large data memory handling, using 3rd party to do heavy lifting is Ok, but transferring every frame back and force would kill your performance. What you can do in LabVIEW is get H264 stream pass it to 3rd party program and once in a while request single frame (at 1-5 fps) add overlay elements and display as intermediate on the screen.

-Artur

HENIGMAN
Member
Member
on

Hello Artur,

Thank you very much. I will develop my project in way you proposed.

Franci

Erlking
Member
Member
on

hi griffin.

your code is very impressive for me, and i tried to work for streaming ip camera immedietly with ffmpeg several days, but it's not work.

if you have some code or hint, plz show me(if you can).

also i already read all about of ffmpeg and rtsp post in ni.com community.

Intergration rtsp and ffmpeg is very hard to me. thanx

ps. sorry for rudeness... im bad at english so .. plz understand my weird english

GriffinRU
Member
Member
on

Hi,

Well, show me what you have done or tried so far with ffmpeg, so I can provide help accordingly. I haven't read any ffmpeg threads at ni.com (provide links if you have) but there are multiple ways to integrate ffmpeg into LabVIEW.

-Artur

Erlking
Member
Member
on

hi. thank you for reply.

I found a post about ffmpeg and link is here .  you can download ffmpeg reader here

ffmpeg reader need 64bit system and provide a recorded file playback.

and here is the my work(and im still working)

캡처.PNG

first, i focus on string data, so i tried to string data to image for labview.

of course, that's not work because of codec. so im trying to integrae your code and ffmpeg reader.

but, im very newbie in codec and mpeg, so i need your help

thank you for reading. Arthur!

you are my only hope i think. haha : )

Erlking
Member
Member
on

this is the best rtsp code i have ever seen in ni community.

very great perfomance and stable cpu using.

GriffinRU
Member
Member
on

Hi,

I think, I will create another thread and step-by-step show how to work with ffmpeg.

Thank you for your feedback.

-Artur

Erlking
Member
Member
on

That's great.

Now I'm trying to work with Opencv's ffmpeg.dll.

Maybe this library will be helpful.

thank you so much.

GriffinRU
Member
Member
on

Sorry for delay,

I will publish examples later this week in new thread:

https://decibel.ni.com/content/docs/DOC-43009

-Artur

KarthikMysoreSrinivasa
Member
Member
on

Great work. Have you ever run into Digest Realm authentication issues while using DESCRIBE method?

Thanks for the answer in advance

-Karthik

GriffinRU
Member
Member
on

Thank you,

I haven't, just showed how it can be done. Now to get over this road block you shoud probably explore "https" examples and use LabVIEW functions to encrypt credentials.

-Artur

KarthikMysoreSrinivasa
Member
Member
on

Artur,

Thank you very much. I have looked through the HTTP examples. All those examples use "GET" method to get the nounce. But, RFC 2326 that ONVIF specification refers to, only has the following public methods: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, SET_PARAMETER. Computing the nounce value using either GET or DESCRIBE didn't help much. However, after going through the following link

http://stackoverflow.com/questions/2384230/what-is-digest-authentication

I'm not sure if I have to keep the connection open and programatically compute the nounce for step 3

-Karthik

GriffinRU
Member
Member
on

I will take a look late this weekend, not ready at this point.

But take closer look at "https" not "http" examples and just "google" around.

-Artur

KarthikMysoreSrinivasa
Member
Member
on

Artur,

Thank you very much

-Karthik

GriffinRU
Member
Member
on

Hi Karthik,

It end up pretty busy weekend, sorry no LabVIEW examples.

Looks like you are on the correct track with your link to stackoverflow, try it and if something won't be clear let me know.

I was sending you to "https" route which wasn't exactly what you were looking for, but from security stand points, it is the prefered method (from wiki) over Digest Realm.

-Artur

KarthikMysoreSrinivasa
Member
Member
on

Hi Artur,

Thanks for taking time to look through. No worries about the examples; I'll keep trying and post if I find anything interesting

Best,

Karthik

DrManuva
Member
Member
on

First, thank you for the examples that have work perfectly for getting a stream of Sony block camera with Atomas mini. I am working on a project that requires me to have a RT text overlay on the video stream. From my understanding, it is best to use the LabVIEW IMAQ format, but am at dead end with changing for N264 Naval string to an IMAQ. I have limited understanding with video steams a push in the right direction would be much apricated.

 

Thanks,

Jarred

basilalias
Member
Member
on

hi Griffin is it possible to stream using sjcam sj4000, my ip is 192.168.1.254.
it is well working with vlc player. can u pls help me out..

CHRISTIANroB
Member
Member
on

Hi my name is Christian, i have the data stream in h264 format from my ip camera, but i don't know decode this frames to image jpg, i need help about this, thanks 

rolfk
Knight of NI Knight of NI
Knight of NI
on

You do not want to do H.264 encoding in LabVIEW, EVER! It is very complex to program and therefore a lot of work to do, and LabVIEW simply doesn't lend itself for lots of low level bit twiddling with extremely high throughput.

 

It's not impossible but simply not economical to try that. Such a solution would require many man months if not years of development work by highly skilled programmers for something that has been already solved numerous times in various C(++) libraries, and with much more experienced programmers who know exactly how to squeeze out more performance from the various CPU architectures out there. But to do that you need low level control of the CPU (including possibly assembly coding) and that is simply not an option in a high level language like LabVIEW.

 

As an extra complication for H.264 there is a licensing requirement. As developer of such code you may be held responsible to make sure users of your code follow these license requirements. That is a possible liability that not everyone is willing to take.

 

Rolf Kalbermatter
My Blog
Contributors