NI Home > Community > NI Discussion Forums
Reply
Member
glennjammin
Posts: 46
0 Kudos

PQA Help

I am new to NI Picture Quality Analysis (PQA).

In the PQA software you have multilple processors (Blockiness, Checkers, Color, Dropped Frames, Gradient, MTF Contrast, Pattern Match, PSNR,SSIM).

Then you have a trigger to start the video at a specific time.

Then have the metrics or results of those processors, which has a max and min limit for each.

 

Currently I am struggleing with getting a good trigger.  It doesn't catch the trigger consistantly in the right place.

I have used OCR, and Intensity.

 

The other issue is how do you determine what processors to use and what limits or metrics to use.

For example you process 100 frames and you are only given a max and min, But there are multiple points that could changed the quality of the picture.

 

For my application the requirement isn't very stringent on the quality of the video, I am just trying to make it less subjective and automate instead of asking the user how it looks.

 

If anyone can point me in the right direction that would be great.

 

Thanks

Test Engineer
L-3 Communications
Member
Aaron O
Posts: 111
0 Kudos

Re: PQA Help

Hello,

 

Thank you for posting.

 

I think the first thing we should concentrate on is the triggering issue. I have a few questions. When you are attempting to use OCR, what part of the image are you triggering from? If you don't mind explaining what area of you're reference file you are focusing on and if you are using a Char set file, that would be great. If you are using a char set file, how was it generated? Many times triggering is done from a frame number located on the reference stream, and that number's reference image comes from the char file. 

 

The PQA is only as good as the reference stream. What are using for this? 

 

Any additional information would be helpful!

 

Regards,

Aaron

National Instruments
Member
glennjammin
Posts: 46
0 Kudos

Re: PQA Help

I am using a Char Set file that I generated using the Vision Trainer.

The trainer used few snapshots of various frames from the referance video.

The area of reference is a Time Stamp on the video.

The reference stream is captured from a known good Video Stream.

 

I like the idea of using a frame counter.  That would be more accurate.

 

If a frame counter is recommended, what is a good method of generating it?

 

Also it may be helpful to describe my setup.

 

  1. My device under test (DUT) has a composite video input that is connected to a video source (Reference video).
  2. The DUT encodes the video and sends the video over Ethernet via UDP
  3. The PC Decodes the video using a DirectShow Graph
  4. PQA Analyzes the video.

Thanks for your help!!!

Test Engineer
L-3 Communications
Member
Aaron O
Posts: 111
0 Kudos

Re: PQA Help

Using a frame counter is a good method, but if you're reference video doesn't already have a frame counter, it might be difficult to add.

 

There are several key parts that are important for good a video trigger:

 

(1) Make sure that the video trigger is unique. The algorithms don't work right if the trigger is not unique throughout the entire reference video.

(2) You also must make sure that the trigger is on the very first frame that you would like to align you're video to. If it isn't, the PQA will be incorrect.

(3) It is also important that the OCR file that is trained using the Vision Trainer is done well. If it isn't, then this could cause incorrect triggering. If you are sure that the video trigger on the time stamp is unique, then you could try recreating the OCR file.

 

Some of these thoughts might help you create a good trigger. If you would like a more in-depth knowledge on what might be wrong with the trigger, then we would probably need you're reference video and the OCR file here to test with.

 

Let me know if you have thoughts or questions!

 

Regards,

Aaron

National Instruments
Member
enigma123
Posts: 4
0 Kudos

Re: PQA Help

Hi Aaron,

Is the tirgger required for blockiness and framedrop tests ? Also, if I don't do a media reference stream or data reference stream, then should my trigger be unique ?

 

My setup - I have a 30fps clips and the output of my player is 60fps. Hence, it duplicates frames to make the 30fps clip into 60fps. Hence, I don't have a unique frame trigger as the frames are repeated. So, what is the solution in this case. 

Active Participant
Greg_S-Video
Posts: 224
0 Kudos

Re: PQA Help

Hello enigma123

 

Thanks for getting in touch with us.  A video trigger isn't required but is prertty much essential if you're looking to start at a specific frame or synchronize two video streams. 

 

You are able to use the Blockiness or Dropped Frame processors without a video trigger but then you wouldn't be performing the the measurement off that same frame every time in your clip.

 

The media reference stream a binary capture of your audio and video stream which is used either for DiskBuffer playback or for use with reference tests so it does not have any direct have any direct influence in software over how you select your video trigger.  Of course your media reference will affect how the user selects the video trigger as you'll want your source and reference to snychronize.

 

The data reference stream is basically a binary reference of the metrics you acquire in a given session or group of sessions.  This also does not affect your video trigger behind the scenes in the software, only how the user selects the trigger.

 

I always recommend a unique trigger if possible.  My expectation of our software is that even though you have a 2 frames with the same video trigger that the acquisition would begin when PQA observes the first instance of that trigger.

 

Let's look at this example.

 

Your source video has Frames: 1, 2, 3, 4, 5.

Your playback devices doubles the frames and so your PQA input will be: 1, 1, 2, 2, 3, 3, 4, 4, 5, 5.

Now, you set your vide trigger to trigger off frame 1 and you want to acquire 5 frames.  You will acquire:  1, 1, 2, 2, 3.

 

If this is not the behaviour you are experiencing?  If not, then let's continue our discussion.  If you like, can you send me a download link for your 30 fps source file?  The solution if it is causing a problem, would be to conver to a 60 fps source.  However, let's discuss this in more detail if you like.

 

Also, I will be answering the following question:

 

How frame drop test is executed using PQA in TestStand ?
http://forums.ni.com/t5/NI-TestStand/How-frame-drop-test-is-executed-using-PQA-in-TestStand/m-p/2305...

 

However, let's continue all discussion of this topic in this thread.

 

Thank you.

Sincerely,

Greg S.
Applications Engineering Specialist | Automated Test
Multimedia Test, Vision, Control Design, Motion
National Instruments
Active Participant
Greg_S-Video
Posts: 224
0 Kudos

Re: PQA Help

[ Edited ]

Hello Enigma123

 

For future reference, I am posting my response to your previous question in the NI TestStand forum:

 

How frame drop test is executed using PQA in TestStand ?
http://forums.ni.com/t5/NI-TestStand/How-frame-drop-test-is-executed-using-PQA-in-TestStand/m-p/2305...

 

How frame drop test is executed using PQA in TestStand ?
02-06-2013 03:23 AM

Can someone shed some light on the frame drop test which is executed using PQA in TestStand ? Also, what is the use of reference stream and trigger in the frame drop test ?

According to the manual, the Dropped Frame processor computes the difference in pixels between consecutive frames. So, in this scenario, how does the processor compare the framedrop of the test stream with reference stream (if any).

Thanks

 

The Dropped Frames processor returns a metric that is the average of the absolute difference between each pixel in a frame compared to its previous frame.  Depending on scene transitions or a known video playback performance, you can use this processor to determine if you are missing frames.  For example, imagine two 3x3 images

 

2 2 2           1 1 1
2 2 2     &    1 1 1
2 2 2           1 1 1

The average difference between these frames is 1.  A media reference stream is not used with this processor when it runs.  However, and I always recommend doing this for unreferenced tests, a data reference can be created from a reference stream to compare with future tests results. The data reference is a calcuated set of metrics that corresponds with a given PQA acquisition/processes or a set thereof.

 

Here is the process I would take for unreferenced tests.

 

  • Take a reference video that you know plays back properly.
  • Set that you wish to create a data reference, acquire that video/audio stream and perform and activate the desired processors.
  • Now, select that you would like to use that data reference for future acquisitions.
  • Perform new acquisitions using the same processors and observe the new results stream
  • You will see your results stream now centred at zero according to the following calculation  (Test Results - Reference Results) / Reference Results  [Relative Difference]
  • If you see any major deviations, this is a probably cause for a concern, in our case, for dropped frames

Let's continue this discussion in the following location:

 

PQA Help

http://forums.ni.com/t5/NI-VideoMASTER-and-NI/PQA-Help/m-p/2305242/

 

Thank you.

Sincerely,

Greg S.
Applications Engineering Specialist | Automated Test
Multimedia Test, Vision, Control Design, Motion
National Instruments
Active Participant
Greg_S-Video
Posts: 224
0 Kudos

Re: PQA Help

Hello Enigma123

 

To make sure I actually answered your question, to use PQA in NI TestStand -- please see the attached image for opening PQA in TestStand and navigating to the Dropped Frames processor.

 

First, ensure NI TestStand and PQA is installed on your system and open a new sequence.

 

1., 2. Drag an NI PQA step onto your sequence.

3.  In the bottom, in the Step Settings for NI PQA divisinon, go to the NI PQA tab and click the "Configure NI PQA..." button.

4,  Select the Processors tab

5.  Selected "Dropped Frames" from the list of processors.

 

Of course, you will need to set up your provider at the very least and a video trigger and data reference if you wish.

 

Please let us know how we can be of assistance.

 

Thank you.

Sincerely,

Greg S.
Applications Engineering Specialist | Automated Test
Multimedia Test, Vision, Control Design, Motion
National Instruments
Member
enigma123
Posts: 4
0 Kudos

Re: PQA Help

Thanks Greg.

 

I tried out the steps. I am doing following steps. Correct me if I am wrong. 

 

1. Play a video stream and capture 350 samples without any reference. I have selected Dropped Frame in processor tab.

2. Select a unique trigger from which I want to start recording. I chose intensity based trigger. I selected a red patch which appeared on 10th frame. [Even though the red patch occurs on later frames - say 25 or 78 - the tigger will start from the 10th frame itself]. 

3. Go to refs and disk tab and select REC for the data reference. 

4. Play my video again and the recording for data stream will start from the 10th frame when tigger is hit and I will get a data reference. 

 

Upto this point, I got a data reference stream for my video.

 

5. In the refs and disk tab, tick the data ref stream box. [Will use this as reference] 

6. In the metrics tab, I have added 'Difference' and defined the limits as follows -

Upper limit - 0
lower limit - 0
count over - 0
count under - 0
frame errors - 1000
start frame - 0
End frame 5000

 

7. Play the same video on the same device with the same trigger again.

 

Now, in the output, I get 158 frame errors even though I am playing the same video. So is this the correct behaviour? What should be the ideal output in this scenario. Also, for calculating frame error count, does the software do a frame matching between current input video and data referene stream ? If there are no frame drops, then what will be the output ? Also, for some other video, I got mean diff as 8k  when I did the same procedure as above. 

 

I am attaching a snapshot for reference.

 

Thanks,

Darsh

 

Active Participant
Greg_S-Video
Posts: 224
0 Kudos

Re: PQA Help

Hello Enigma123

 

I am experiencing a pretty high support volume right now.  I'm going to throw some quick comments but I might be able to address this personally later in the week.  I've got an answer to this concern but I can't put it together right this second.

 

The reason you are getting all those errors is because your upper and lower acceptable limits have no deviation.  At zero for both, if the metrics go either above or below zero, then you are going to get an error.  Notice the frame errors correspond with anything that is not a zero on the metrics tab.  Would you want to upload your source video for me?  I can go into more details about the Dropped Frame processor metric later this week so please make sure to respond if you would like to continue addressing this issue. :-)

 

Rest assured that if you respond, this issue will be addressed by one of our applications engineers in the mean time.

 

I would also like to point you to our latest version of this software for your interest.

 

NI Picture Quality Analysis 1.6 - Windows 7/8/Vista/XP 32-bit
http://joule.ni.com/nidu/cds/view/p/id/3753/lang/en

 

Thanks!

 

Sincerely,

 

Greg Schumacher

 

Sincerely,

Greg S.
Applications Engineering Specialist | Automated Test
Multimedia Test, Vision, Control Design, Motion
National Instruments