How to write AVI fast - Gigabit Ethernet camera FullHD

[k_pala]

Hello

     I am going to make some acquisition of images from FullHD Gigabit Ethernet camera (1920x1080) and I am not sure how to do it.

When I am trying to save next frames of avi in while loop (after grabbing image from camera) it takes quite long, but without compression

with some codecs it is acceptable. The problem is  that I am not sure how long the film would be so storaging everything on hard drive might be simply impossible.

I was wondering about putting everything into Images made by IMAQ Create but it rather quite good idea, especially that I don't know how many

frames I need. I am looking for some suggestions, tips..  I was wondering about writing small parts of film (i.e. 20 minutes), compressing it in background

and then writing on DVD, but I still need to much of my CPU, so I am quite confused.

Thank you very much for all help

Karol 

[SMeserve]

Hi Karol,

We do not have compression that would allow you do what you are describing.  The solution would be to obtain more storage or find an option to push the data off the hard drive (to DVD, perhaps) in batches during acquisition. You would need to find a 3rd party piece of software to burn those DVDs however.

[k_pala]

 What about using two core processor and do acquisition and saving with two cores separately ? I'm still looking for solution

Karol Kapała 

[SMeserve]

Hi Karol,

Processors won't help you create more storage space on your hard drive.  No matter how fast you go, you cannot record more data than you have space for. Thanks.

[unclebump]

Are you using SATA hard drives in the computer?? You can get them with 3.0Gbps transfer speeds. Might have to use and add-in card to get that type of buss.

[smcclana]

What is important to note is the physical write speed of the hard-drive.  In many cases, the published maximum write speed is theoretical if not down right misleading. When writing HD GigE camera data, your bottleneck is going to be the write speed of your storage device.  You can use various codecs to shove this bottleneck into the processing and compression in the CPU, but then you'll most likely have a lossy AVI.

Another approach is to shove the data into RAM, later writing to AVI.  You could do this with a queue (circular buffer) system.  However, given the size of data you're accumulating, you'll hit your RAM limit within a short amount of time (depending on your camera frame rate).

You are faced with the classic dilemma, either you have to minimize your frame rate and save all data to AVI uncompressed or use a compression routine and keep your frame rate constant.  Something has to give.

If you are able to, you could also investigate a third party off-the-shelf high-speed storage system compliant with GigE vision cameras.  These systems are designed just for this kind of scenario and are your best bet of having full HD AVI data.  However, they come with a hefty price-tag.

Good luck,

Stephen McClanahan

[Joseph_Loo]

If you want a fast storage system, you should investigate in some real RAID hardware. If necessary go to ISCSI drives instead of the slower e-sata drives. What you are talking is throwing hardware to solve your cost. I do know that some of high speed equipment do exists but they cost a lot of money, e.g., CERN system. I believe they stream at the gigabyte/sec range.

[Matt_W1]

A large scale raid system is hardly necessary for a single GigE Camera. Even raw, GigE is 120 mebibyte/s, current commodity SSDs can do 130-200 mebibyte/s (you need to pick the right ones though) and are about $300. If you choose your codec carefully you can losslessly shrink the stream rate roughly in half (unless you're video has very high entropy of course). With that a single dedicated VelociRaptor should be able to handle it. You'll want some amount of frame buffer to handle hiccups in the system though.

I'm streaming one of these (300 megabit/s of Fire Wire bandwidth usage, the video stream alone is ~200megabit/s of that) onto a cheap 3 year old hard drive using 12 mebibyte/s of space with a lossless codec. Technically there's a color space change but that's hardly a loss. I think I could probably stream 3 GigE camera's onto a single SSD. I might need to interleave frames while saving since the avi compression system LabVIEW uses might suffer from contention trying to simultaneously compress multiple streams, in that case a post process would be needed to take it back apart (and the camera's should be the same res).

[C.K]

Hi Matt

I'm struggling with the same problem as k_pala. I've got a Gige camera that requires a max. 88.5 MB/s write speed, but I know my recordings are never going to exceed 4 minutes in size, i.e 21.2 GB. Too much to store into RAM though.

As smcclana said the write speed in the spec may be far from the real write speed. For example for the velociraptor 600GB, some tests say the max speed is 150MB/s, but minimum is below 100MB/s. Do you know if this is the case for the SSD drives too? Also, I'm worried the speed is affected by the computer, mine is an older pentium 4 (R), with 3.8GHZ processor, and 3.24 GB ddr ram.

I'm not sure if the Velociraptor would provide a reliable solution, although it would be cheaper than an SSD for a given size. Since you have a working system which SSD would you recommend? Kingston seems to make SSD kits for desktops...

Also, I could do a lossless compression on my video, but my application requires no loss in the colour information, and I need to be able to get hue out.  Any idea about which codec could I use? 

[Matt_W1]

A normal harddrive is slower at the end of the drive vs the beginning so a dedicated drive (and you should really be using a dedicated drive for capture) that you keep mostly empty would stay near the higher end of it's rated speed (bad sector remapping can put hiccups in this).

A good ssd should hold the same speed over the entire drive. There can some heavy fluctuations due to fragmentation with the wear-leveling algorithms (this is different than file system fragmentation). Wear-leveling shouldn't be a problem on Win7 with a ssd that supports TRIM (Note TRIM doesn't currently work on raided ssds), when you keep a bit of free space. Internal data compression on the sandforce controllers can also vary writespeed.

My system doesn't have a SSD (the 3 gige camera was an estimate of I thought I could do with a good SSD). I get most of my ssd info from

http://www.anandtech.com/tag/storage

so you can read stuff there, currently it looks like the Crucial C300s, OCZ Vertex 2 and Intel SSDs are the good ones. If you don't use Win7 then OCZ's Vertex 2 is probably the best bet (isn't affected much by fragmentation) and is quick with a consistent write speed of about 180MB/s while fragmented. A OCZ revodrive should be 200 MB/s write while fragmented.

For compression I use the Huffyuv with adaptive huffman tables codec in the ffdshow_tryouts codec pack. I don't think much color information is lost from the transformation. Color is encoded per 2x2 pixel block instead of 1x1 at least with the YV12 colorspace I'm using. Sadly the ffdshow huffyuv doesn't include the rgb colorspace. The stand alone huffyuv does, but didn't perform as well for my use. If you're camera spits out YV12 or YUY2 then you can encode without a color transformation. Huffyuv is light on the cpu, at least for a video codec. On an P4 you could run into bus bandwidth limits (depends on motherboard). For instance if your connection to the harddive is through a 33MHz 32bit PCI bus then at best you can get ~130MB/s assuming that nothing else is using the bus (like a network card). This isn't a problem with the newer PCIe based systems since the bus isn't shared. Depending on what else is going on the P4 could get bogged done though. I run a multicore system, so I made modified the avi write vi, to be reentrant when calling the dll (so the UI Thread can't block it). This hasn't caused me a problem but I'm very careful about linearizing access to the avi commands. Since marking things reentrant when they aren't can easily cause very nasty problems.