Example Code

FFMPEG scripting in LabVIEW (.NET)

Code and Documents

Attachment

Overview

"CONVERTING VIDEOS AND AUDIOS HAS NEVER BEEN SO EASY." - And Finally in LabVIEW

How to use "Zeranoe FFmpeg" static 32/64-bit FFmpeg executables with LabVIEW under Windows OS.

FFmpeg examples provided below can be used with any supported sources, i.e DShow devices, network streams, files.

FFmpeg can be used for video analysis and data broadcasting, for more details on how to use FFmpeg refer to https://www.ffmpeg.org/

Description

Attached examples were created in response to question posted at "RTSP over TCP with LabVIEW" thread.

.NET implementation explained:

  • Create process (FFmpeg.exe)
  • Set process parameters
  • Subscribe to process-related Events
  • Use pipes (stdin/pipe0/stdout/pipe1/stderr/pipe2) to move data from FFmpeg to LabVIEW and back

Steps to Implement or Execute Code


  1. Download and install "Zeranoe FFmpeg" latest static build and place into FFMPEG directory.
  2. Folder snapshot:FFMPEG folder.png

  3. Enjoy FFMPEG scripting with LabVIEW

Requirements to Run


Software

  1. LabVIEW 11.0 or later
  2. "Zeranoe FFmpeg" 32/64-bit static build  -  http://ffmpeg.zeranoe.com/builds/
  3. "RTSP over TCP with LabVIEW" library for RTSP examples - https://decibel.ni.com/content/docs/DOC-35217

Hardware

  1. USB (DShow device) or IP camera or video/audio files

Snippets

***SORRY for broken VI's - LabVIEW's snippet tool backfiring... Files inside zip archive are not broken. I will fix snippets later...***


Open DShow device.png

"FFPLAY - Example - DShow device.vi" - opens user specified DShow device.

DShow save to file.png

"FFMPEG - Example - DShow - stream to file.vi" - Saves DShow device data to mp4 container, user can specify duration or cancel recording at any time.

RTSP save to file.png

"FFMPEG - Example - RTSP - stream to file.vi" - Saves RTSP stream to mp4 container, same as above.

LabVIEW-RTSP-FFMPEG.png

"FFMPEG - Example - RTSP-LabVIEW-FFMPEG - stream to file" - Sometimes when FFMPEG fails to establish connection with network device you can relay data with LabVIEW over the pipes (stdin/pipe0 and stdout/pipe1 and stderr/pipe2).

Extract frame to JPEG.png

"FFMPEG - Example - RTSP - Extract frame as JPEG.vi" - Extracts single frame from network stream, passes it over pipe to LabVIEW and saves as JPEG file.

Extract frame to PNG.png

"FFMPEG - Example - RTSP - Extract frame as PNG.vi" - Extracts single frame from network stream, passes it over pipe to LabVIEW and draws it.

Extract frame to PPM.png

"FFMPEG - Example - RTSP - Extract frame as PPM.vi" - Extracts single frame, stamps image with timestamp, frame count and duration. Pipes resulting frame to LabVIEW, converts PPM data to 2D Image array.

More examples would come later...

Update 06/29/2015:

PPM_Movie.png

"FFMPEG - Example - RTSP - Extract frames (PPM).vi" - FFmpeg processes video stream and passes individual frames to LabVIEW.

Update 07/01/2015:

"FFMPEG Examples v1.1.zip" - updated with "yuv420p" and NI Vision Graph examples.

yuv420p.png

"FFMPEG - Example - RTSP - Extract frames (yuv420p) Intensity Graph.vi" - yuv420p to 2D RGB

Note: To improve performance, Intensity graph update rate was reduced to 1/5 of frame rate.

Vision Graph.png

"FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph.vi" - yuv420p to NI Vision Graph

Update 12/30/2015:

With script -f sdl "window name" - you can create FFmpeg rendering window (CPU usage is not great but better than what you can do in LabVIEW)

You can specify size, title, background and window title.

FFmpeg sdl documentation

-Artur

Example code from the Example Code Exchange in the NI Community is licensed with the MIT license.

Comments
Erlking
Member
Member
on

Hi. Artur.

i tested your code few days, and recording very well with many codec.

your code include extracting one image, so i trying to make a streaming mode. but(you know) it didn't work.

i used ffmpeg command vframe over 1(ex. -vframe 240) and my computer memory overflow and shut down. i think extract ppm.vi always extract one image.

how to extract few frames like movie. in fact i need streaming. i'm still trying to work and help me if you can. : )

i'm machine vision and vision processing developer... so i am weak in this field.

thank you! your code is always very helpful !

GriffinRU
Member
Member
on

Hi,
Well the down side of using pipes is if you do not unspool data fast enogh you are going to lock/crash LabVIEW or any other application. Examples above were for single frame extraction, I can show how to extract frames as they become available, but you need to make sure that your code is keeping up.

What is your final goal, real-time or post processing? And you should be able to script ffmpeg to dump as many frames as you want in any format to a given directory...

-Thanks for comments,
-Artur

Erlking
Member
Member
on

hi

my final goal is real-time. i can play ip camera in real-time using vlc player in labview. but i have to use labview machine vision tools in my field. so i need image available in labview.

i'm still trying to work with your examples.

i don't know what is the missing link. haha

thank you! and you r the best in this labview community!

GriffinRU
Member
Member
on

Hi,

Example attached,

In some cases you might also consider "yuv420p" format, I have "yuv420p to RGB.vi" inside "SubVIs" folder.

-Artur

Erlking
Member
Member
on

omg. what a great code!

it works very well!

why images are showed black & white? because of allocated memory ?

i'll change b&w image to color image. i am seeking color control in your code.

thanx a lot!

ps. if you know why images are gray, teach me plz

GriffinRU
Member
Member
on

FFmpeg does the job, LabVIEW sits and relax

Images are grayscale, because provided example uses only one component... 2D intensity graph is not the best tool for 32-bit color images. To display color images in 2D intensity graph you need to either index values with custom color table, or create color table and convert array to RGB values.

I think, what you need is attached in latest examples (yuv420p and Vision Graph), let me know if that's true.

Enjoy,

-Artur

Note: PPM vs yuv420p (if supported) half the payload from FFmpeg.exe to LabVIEW over pipes.

Treat examples as prototypes and add error checking and cleanup accordingly.

Erlking
Member
Member
on

omg you did it again! professional!

i'll use this code in my image processing works and share with you later.

very helpful and amazing code.

Erlking
Member
Member
on

hi Artur.

i'm testing your code at few computers and i encouter some error(In fact, not my pc)

< error > ----------------------------------------------------------------------------------------------------------------------------

error 1172: FFMPEG - Get Stream Info.vi->FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph.vi invoke node System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.

          Inner Exception: System.ComponentModel.Win32Exception: cannot find file

------------------------------------------------------------------------------------------------------------------------------------------

i trying to find some solution and all solution had bad result. what's wrong with my pc?

thank you.

GriffinRU
Member
Member
on

Hi,

Can you identify what part of the code gives you an error?

Sounds like "FIREWALL" settings...

-Artur

Erlking
Member
Member
on

< error > ------------------------------------------------------------------------------- ---------------------------------------------

error 1172: FFMPEG - Get Stream Info.vi->FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph.v  event callback

labview : .net external assembly exception occured.

---------------------------------------------------------------------

error 1172 occured. ffmpeg example - rtsp - extract frames(yuv420p) vision graph.vi  event call back

=---------------------------------------------------------------------

2.jpg

error occured here. call back fuction

thank you for your reply and i'll check my pc's fire wall

thanx!

GriffinRU
Member
Member
on

Hi,

LabVIEV's "brain-freeze" - to solve this do the following:

1. Delete register for events code

2. Save vi and exit LabVIEW completely

3. Open LabVIEW, open blank vi and check if register for events works (you can load any LabVIEW provided examples)

4. If works - rebuild my examples (they are simple) - do not copy and paste

5. If doesn't - follow NI support on how to handle 1172 error

-Artur

Keyvan86
Member
Member
on

hi

please help me.

FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph.vi working very good for one IP camera.How read Two IP camera RTSP Address and display in Two labview vision graph?

Thank you

GriffinRU
Member
Member
on

Hi,

It should work with as many cameras as you want, up to system limits, just change IMAQ image name... and of you go.

If you compile the code you can just call clones as many as you want.

Maybe, attach your code, so I can point what to fix?

-Artur

Keyvan86
Member
Member
on

hi

Thank you for reply

Now working correct and no problem.change IMAQ and FIFO name for another camera.

Thank you

gharris
Member
Member
on

Hi Artur,

Great code - thanks so much for posting it!  I was looking for a way to extract a single frame into a LabView picture, so I made a variation of your examples.  I put it in the list of attachments - I hope you don't mind.  It seems to work with any video format - what a great utility.

Thanks again,

George

GriffinRU
Member
Member
on

Hey George,

You are welcome, and i am glad my example helped you out and i do not mind adding examples to my thread.

-Artur

Erlking
Member
Member
on

i think some pc has a problem

some pc are encounterd error in BeginErrorReadLine(invoke node).

what's wrong in this node.

GriffinRU
Member
Member
on

Hi,
Can you attach your code?
-Artur

Erlking
Member
Member
on

hi artur.

i still have used your code because its great.("FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph.vi")

but i have some problems.

when i connect my ip camera(1920x1080) cpu use rate is too high! about 50%!

and other problem is consumed frames don't follow produced frame.

so, if play this code long time, produced frame got all my pc's memory and stuck.

why this happend?

can you give a solution to make lower cpu using rate, and memory issue ?

GriffinRU
Member
Member
on

Hi,

I think you need to check if you parsing stream correctly, I would wire constants for sizing array first and stop execution after successful frame received.

Get some probes and post results.

-Artur

Corny
Member
Member
on

Nice code Artur

When having an output pipe from FFMPEG to Labview, have you tried configuring FFMPEG to convert the frames to RGB32 by using the output command "-pix_fmt rgb32", instead of needing to convert YUV to RGB in labview?

GriffinRU
Member
Member
on

Thank you,

I have not used "-pix_fmt rgb32", I am glad you found it, should be more efficient

-Artur

Corny
Member
Member
on

I just tried it, and the first test gave good results.

Downside is that you'll end up with more data through the pipe (to be precisely its 2.67 times more data), but the upside is that LV is not converting it.

Note: when using the command "-pix_fmt rgb32" this gives you rgba values at the pipe. Imaq wants to have argb values. Instead of converting these from rgba to argb values in labview, rather use the command "-pix_fmt argb" (instead of "-pix_fmt rgb32")

GriffinRU
Member
Member
on

There is always a catch but still great tool to have...

-Artur

GriffinRU
Member
Member
on

You can request "pix_fmt bgr24" or "pix_fmt rgb24" and while doubling memory transfer -> you can optimize performance by skipping data manipulation in LabVIEW...

-Artur

zahidkjatoi
Member
Member
on

Hello,

 

I was trying to use your Vi's for getting an image from Usb 2.0 camera. But i got multiple errors (Actually same error 1172 and it loops back). I am not sure why its happening. I think it is due to one thing but not sure. Snapshots of errors are attached for review.

error1.JPG

error2.JPGerror3.JPGerror4.JPGerror5.JPG

Kindly guide,

 

Khan.

MindLoops
Member
Member
on

Hi Artur,

            I had tried your program for two cameras (vision graph) and it works. But i having trouble when my one of camera is in offline mode. When either one camera is offline, i cannot get image from another working fine camera. I found that the program is stuck in the ffprobe.exe. May i know how to add timeout for ffprobe.exe to stop it if it failed to connect the camera? I am duplicate same program in a vi and rename the IMAQ&FIFO for second camera.

NapDynamite
Active Participant
Active Participant
on

Hi GriffinRU,

Great work.
I have tried your code and works like a charm for my IP camera. Is there any possibility to play the raw h.264 byte stream in to an IMAQ compatible format using FFMPEG scripting? 


mrk_valeo
Member
Member
on

Hi Mr. Artur,

 

It is a great code to push data from LabVIEW in to FFPMEG, do conversion and pull it back to LabVIEW for display or further processing purpose. I am using your code for converting H264 video data from camera in to RGB format to be used further in LabVIEW. I am converting live stream of video data from camera. Steps followed,

1. I am collecting camera video data through hardware interface in to LabVIEW

2. Pushing this video data from LabVIEW in to FFPMEG via a TCP port 35001 as below,

TCP to FFMPEG.JPG

2. Do conversion in FFPMEG and pull it back to LabVIEW via pipe buffers as below,

FFMPEG to LabVIEW.jpg

I am able to get converted RGB data back to LabVIEW. This works good for live streaming. This works fine for one single camera live streaming purpose.

 

Now I am trying to scale this up to two camera live streaming. I am doing pre allocated reentrant for the code shown in both the above images. The conversion for live streaming for two cameras still works fine. Now I am doing close camera / disconnect camera 1 and expecting first instance of the FFMPEG process closed down independent of second camera live conversion process. But first process pauses at read FIFO that I highlighted in second image. Please provide solution to avoid this pause at pipe buffer read (by doing different way of configuring process and pipe?). And also please note that if I do close / disconnect camera 2, then first instance of the process reads out certain amount of data the gets closed without any issue and both the instances are closed. So my observation is, if I do close / disconnect camera, there is some residual amount of data left out at the buffer (less than 65536 the buffer size that is configured), and the buffer is waiting indefinitely to get data of array size 65536 get accumulated / filled up. And once if the second instance is also closed, then both the process instances closed after reading residual amount of data (ignoring 65536 data to get accumulate)  and closed without any error / issue.

 

So the solution needed is to flush out buffer residual data and close the process independent of other parallel process once if the particular camera closed / disconnected.

 

Thanks for your help,

Ramesh

mrk_valeo
Member
Member
on

Hi Mr. Artur,

It is a great code to push data from LabVIEW in to FFPMEG, do conversion and pull it back to LabVIEW for display or further processing purpose. I am using your code for converting H264 video data from camera in to RGB format to be used further in LabVIEW. I am converting live stream of video data from camera. Steps followed,

1. I am collecting camera video data through hardware interface in to LabVIEW

2. Pushing this video data from LabVIEW in to FFPMEG via a TCP port 35001 as below,

TCP to FFMPEG.JPG

2. Do conversion in FFPMEG and pull it back to LabVIEW via pipe buffers as below,

FFMPEG to LabVIEW.jpg

I am able to get converted RGB data back to LabVIEW. This works good for live streaming. This works fine for one single camera live streaming purpose.

Now I am trying to scale this up to two camera live streaming. I am doing pre allocated reentrant for the code shown in both the above images. The conversion for live streaming for two cameras still works fine. Now I am doing close camera / disconnect camera 1 and expecting first instance of the FFMPEG process closed down independent of second camera live conversion process. But first process pauses at read FIFO that I highlighted in second image. Please provide solution to avoid this pause at pipe buffer read (by doing different way of configuring process and pipe?). And also please note that if I do close / disconnect camera 2, then first instance of the process reads out certain amount of data the gets closed without any issue and both the instances are closed. So my observation is, if I do close / disconnect camera, there is some residual amount of data left out at the buffer (less than 65536 the buffer size that is configured), and the buffer is waiting indefinitely to get data of array size 65536 get accumulated / filled up. And once if the second instance is also closed, then both the process instances closed after reading residual amount of data (ignoring 65536 data to get accumulate)  and closed without any error / issue.

So the solution needed is to flush out buffer residual data and close the process independent of other parallel process once if the particular camera closed / disconnected.

 

Thanks for your help,

Ramesh

MoviJOHN
Active Participant
Active Participant
on

Hi Artur,

 

I hope you are doing well. I modified the FFMPEG - Example - RTSP - Extract frames (yuv420p) Vision Graph and gave the following source for mjpeg instead of mpeg4:

 

-i rtsp://admin:admin@192.168.1.93:554/mjpg?source=0 -an -f image2pipe -vcodec rawvideo -

 

The first few frames appear OK, but the image immediately shifts, and the lower portion becomes partially corrupted. Thinking it was the -vcodec rawvideo causing this, I tried -vcodec mjpeg, but this results in an RGB noise image.

 

I have seen as many as the first 50 frames come through OK, and the stream remains stable and responsive, even with the corruption, so it must be something to do with parsing. Any thoughts?

 

Best regards,

 

John

Machine Vision, Robotics, Embedded Systems, Surveillance

www.movimed.com - Custom Imaging Solutions
Contributors