How to store 10,000 frames?

[Basanta]

Hi

I need to store a sequence of 10,000 frames (768x576) at the speed of 30 frames/sec. Our color CCD is interfaced with the PC through NI 1411 card. Currently I am using Vision Stage software to store the frames, but unfortunately I am able to store only 1000 frames (max) in a sequence. Can anybody please let me know whether I need to change anything? Is it possible to use LabVIEW to store that many image? I want to confirm before using LabVIEW.

With thanks and regards,

Basanta

[Pnt]

As far as i know Labview has no limitations to the data you can store.

Possible reasons for your problem:

-Your computer runs out of memory

-there is something wrong with your code. (You can post it if you want).

[iRish]

Hi Basanta,

Pnt is correct and there is no upper limit for image storage in LabVIEW.  Your available RAM will be a limit.  If you are doing 'snaps', then you need to be sure to allocate enough buffer memory for this.  Otherwise if you are doing a 'grab', LabVIEW only processes one frame at a time before saving to AVI.

Regards,

[Kevin_Price]

Just estimating in my head, but if you're talking about each pixel as a 4-byte (32-bit) integer, then at your resolution of 768x576, you may be approaching 2 GB as you get to a bit over 1000 frames. 

The significance of 2GB relates to maximum file size for certain disk formats and/or OS'es.  Ok, here's a link.  If that site's accurate, 2GB maximum file size applies to disks formatted as FAT-16.  This seems a bit too archaic to be likely, but you should verify to be sure.

I know nothing about "Vision Stage" software, but perhaps *it* has trouble as you approach 2GB of memory usage?  Another fun fact that may prove relevant is that 32-bit OS'es can only address 2GB of memory space.  (Presumably the memory pointers are stored as signed 32-bit integers rather than unsigned?)

If you code in LabVIEW, there should be ways to be more efficient about re-using RAM memory so you don't bump up against the 2GB limit.  If needed, you can also distribute the 10000 frames across multiple files to avoid disk-format file size limitations.  FAT32 is fairly common and is limited to 4GB files as seen in the earlier link.

-Kevin P.

[Ben]

Kevin pondered "(Presumably the memory pointers are stored as signed 32-bit integers rather than unsigned?)"

I BELIEVE the MSB distingished between User and System memory. The OS uses the top half of memory space and the apps use the bottom. When I first recognized this issue and brought it to the instructors attention (the class was VMS internals and data structures) responded "For that to be an issue, someone would have to write an application that required more than 2 Gig of memory!" I was laughed out of the class for asking such a silly question.

Ben

[waldemar.hersacher]

Ben you don't need to believe it its true for Windows.

Although there is a switch for the boot.ini entry of the OS to boot named /3GB. Setting this switch increases the user adress space to 3GB. To access the additional space the application must be built for it and marked for the OS that it can do.

Source: Inside Microsoft Windows 2000, page 155 (in German edition).

I don't think that LV is built for that.

[DFGray]

Most of the problems you will run into will be relatively independent of LabVIEW.  As mentioned before, you won't be able to buffer all the frames in memory.  For a a variety of reasons, the maximum RAM most apps in Windows can access is about 1.5GBytes, minus any memory the app itself takes (see the tutorial Managing Large Data Sets in LabVIEW for details).  This means you will need to buffer to disk.  With 32bits/pixel, this is about 53MBytes/sec, so you will probably need a RAID array for robust operation (a really fast SCSI or Raptor disk may do it with a single disk, but the RAID array is probably cheaper to implement).  You will want to stream to a single file or create your files ahead of time to avoid stalling your stream operation.  If you design your LabVIEW program correctly, it should be disk speed limited.  You will want to use a binary file format.  Pick one depending on where the data will be analyzed.  TDMS is native to newer versions of LabVIEW, but not widely supported outside of NI software.  HDF5 is difficult to learn, not natively supported, but widely supported by many analysis packages.  Flat binary is always the fastest option, but may give you archiving issues down the road.  Using NTFS as your file system format instead of FAT16 or FAT32 will make your life much easier, since it supports files over 4GBytes in size.  Note that it is a bit slower than FAT32, so you can use this to your advantage if your system is just barely not good enough.

Good luck.


Message Edited by DFGray on 01-07-2008 09:31 AM

[Matt_W1]

I know how to stream (so only limited by hard drive space) 1024*768@30Hz color video, to a hard disk at ~18 MB/s (well within most current hard drive speeds, the speed requirement's can be higher depending on what you're recording) with lossless quality (well almost, there's a color conversion) in an AVI file. It's kind of a pain to setup properly though (not something I would want a user to have to do, although I think all the settings can be set via the registry). I can explain it if someone's interested, assuming I still remember all the important gotcha's. Note: this uses the vision module

Matt W


Message Edited by Matt W on 01-07-2008 05:42 PM