Stream Serial Data to Disk

[JMorgan]

I am looking for a sample .vi that I can use to read in data from my RS-422 port, and stream it straight to disk so I don't have to worry about buffering the incoming data. My incoming data files can be anywhere from 1MB up to about 150MB or so, and it will vary, so I need something, or ideas anyway, how to do this. Any help would be greatly appreciated.

Jimmy

[JMorgan]

I have one correction to my main post: I just found out that my file size will be 500-750MB.....

[JenK]

Hi Jimmy,

There is a "Basic Read and Write.vi" in the NI Example Finder that you can modify to write data to a file. If you have never used the NI Example Finder before you can find it in LabVIEW by going to Help>>Find Examples. This will launch the Example Finder. Click on the Search tab, type "serial", and hit enter. You should see the "Basic Read and Write.vi".

Once you have that opened you can modify the block diagram. You might put the VISA Read portion in a while loop and write the data to a file in that loop. For an example on how to write the data to a file look at the "Write to Text File.vi" which can also be found in the NI Example Finder by typing in keyword "write".

If you have further questions please repost.

JenK

[DFGray]

Dealing with that amount of data will require you to pay attention to data copies for best throughput. Check out Managing Large Data Sets in LabVIEW for some tips. You will definitely want to optimize your read and write chunk sizes (and they will probably be different). For Windows (FAT, FAT32, and NTFS file systems), the optimum write-to-disk chunk size is about 65,000 bytes, just under the 64kByte boundary. Your mileage may vary on RT systems.

If you need different chunk sizes for optimum performance, my approach would be to set up a ring buffer using a LabVIEW 2 style global. Feed it with the data input at one chunk size. Stream to disk from it at a different chunk size.

Good luck.

[JMorgan]

The data I'll be recieving is going to be a binary file. Will these methods still work?

[DFGray]

Most data transfer methods work better with binary, and the ones we have listed below are no exception. The LabVIEW file read and write primitives take and arbitrary data type, so that is not an issue (endian-ness is - if you see weirdness, check this out with a hex editor). Serial ports usually take strings, but they are easy to convert to binary using the typecast operator. Check this forum or the LabVIEW examples for how to do this if you are unfamiliar with it.

[JMorgan]

I have been looking over the large file sets example, and had a question that I'm really hoping you could answer. I can't seem to figure out how to change the example to read data from a serial port. I'm pretty sure that I need to modify the while for-loop, but I just can't seem to figure out how to read in and write my data to the file instead of writing the index.

any help would be appreciatd.

Jimmy

[DFGray]

Take a look at the attached VI. It has none of the buffering I mentioned earlier, but you may not need it. Disk transfer will probably be inefficient, but should be able to keep up with the serial port. You may want to put a delay in the loop to keep from sucking up processor resources and to allow some data to come in before the next iteration. This will depend upon your data rates and what else you are doing at the time. I would suggest about 10msec.

[JMorgan]

I am checking out the .vi you posted, and as far as i can tell, it will work great. the only problem i am having is when I'm working on a checkout. I have a .vi that is writing a binary file out to the serial port, and i am using a loopback connector to allow me to use the same serial port for both. I'm using your .vi to read the file, but am getting error -1073807339. Also, when I run NISpy, I get the following message over and over. Is this caused by me using the loopback connector, or should I look somewhere else. the file i'm trying to transfer is about 10MB.

> 24069. viWaitOnEvent (0x001BA008, 0x3FFF2009, 0, 0, 0x00000000)
> Process ID: 0x00000EBC Thread ID: 0x00000EFC
> Start Time: 13:07:15.345 Call Duration: 00:00:00.000
> Status: 0xBFFF0015 (VI_ERROR_TMO)

[DFGray]

At this point, you are getting out of my field of expertise. I have not used serial ports in years. However, you may want to check the following:

1. Your error is a timeout error. Is your timeout set too low? If so, bump it up a bit and see if the error goes away.


2. Are your hardware/software set up correctly? Since you are getting a timeout error, the software does not think any data has arrived. This could be due to either hardware or software handshaking problems.


3. Does your hardware support simultaneous read and write from the same serial port? There may be only one buffer for both input and output.


4. Does VISA support read and write from the same serial port simultaneously? I have never tried.

Keep us posted. I am sure there is someone out there who knows serial ports far better than I and can probably steer you in the right direction.