LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

How to achieve the maximum file transfer rate from PXI to local host?

I will have to copy a huge amount of data (as big as 50 GB) from a PXI-8106 unit on site to a laptop. Currently it would take several hours. It is critical to max out the data transfer rate.

 

The standard method we've been using is just drag and drop using Windows Explorer via FTP. I tried to use FTP VIs with Filezilla Server as well but the transfer rate is only like 1.5 MB/s. Is this normal? Once I saw it was 3.0 MB/s but for some reason, that isn't happening now.

 

For another option, I tried to transfer data to a USB flash drive but it's even slower. As far as I know, USB 2.0 transfer rate is supposed to be around 60 MB/s but why would it be slower than 1.5 MB/s? What could be the speed limiting factor in my file transfer setup?

 

Any tips on improving file transfer rate would be appreciated!

 

0 Kudos
Message 1 of 11
(4,516 Views)

I noticed that in general you can transfer files to NTFS-formatted USB flash drives much faster than FAT32 ones. However, LabVIEW RT OS doesn't seem to recognize USB drives in NTFS format. Is there any way to make it work?  

0 Kudos
Message 2 of 11
(4,486 Views)

Hey There!

 

First, the easy question - NTFS on USB.  We do not support any of the NTFS file systems in LabVIEW Real-Time.  Yes, there are actually several versions, and we don't support any of them.  FAT32 is the only supported filesystem on USB.

 

USB data transfer rates on LabVIEW Real-Time are optimized for worse-case support across all our supported platforms.  This unfortunately means it's optimized for the cRIO-9004, which uses a block transfer size that is non-optimal on PXI-based controllers.  Versions of LabVIEW RT 2011 and better have better transfer rates, but it maxes out more at 13MB/s instead of the theoretical optimum limit for USB 2.0.  We are looking into optimizing USB throughput within our OS.

 

Transferring files across FTP got a huge boost in LabVIEW RT 2012.  We realized we were using a block transfer that was abhorrently small, and increasing that block transfer was relatively easy (not like messing with USB and Disk caching like in the case of USB).  In LabVIEW RT 2012 we increased the block transfer rate for FTP, and we increased our throughput significantly.  

 

If you want to stream data off a PXI-8106 optimally, you would need to write your own TCP/IP application and stream the data via TCP/IP to a system.  SMP network traffic on a PXI-8106 prior to LabVIEW RT 2012 is also half as slow as performing the same operations in single-core mode, because the network stack in pre-2012 versions of LabVIEW RT aren't optimized for SMP.  In LabVIEW RT 2012, we made significant upgrades to the network stack and optimized the network stack for SMP - now the data transfer rates of ethernet are the same for SMP and UP modes.

 

-Danny

Message 3 of 11
(4,471 Views)

Thank you for the answer, Diaz. I'm currently using LabVIEW RT 2012. I tried running simple TCP file sender/receiver VIs on the PXI and a laptop but it didn't work fast for me. There might be some important trick I missed. By the way, I noticed something very interesting in the meantime.

 

Let's say the program in the PXI finished writing to a tdms file. I copy it to my laptop via Ethernet FTP connection. The transfer rate is usually capped at 1.5 MB/s. Then I copy it back to the PXI. Now if I copy it once again from the PXI to my laptop, the transfer rate is like more than 10 times faster than before although it's essentially the same file with the exactly same content. It seems the file structure is re-organized and optimized for file transfer while being copied back and forth. Is there any way to have that optimal file structure without copying it back and forth?

0 Kudos
Message 4 of 11
(4,439 Views)

That's odd, it sounds like there is a cache somewhere getting involved in this process - I promise no file structures are being "optimized" in any case, it just sounds like some or all of the file is being cached somewhere.  LabVIEW Real-Time has a file cache, but it's incredibly shallow - I would be VERY surprised if any caching mechanism on the side of the RT system would have anything to do with our use-case here.  

 

Using RT 2012 I wrote a quick RT app on a PXI-8106 controller that created a massive text file on the disk - about 500MB worth (this is guaranteed to defeat caching on either side).  I am running FileZilla on a host PC connected to the target on a gigabit network, and I connected to the target via FTP using FileZilla and "pulled" the file from the RT target to the Host PC, and I got a sustained data rate of 40MB/s.  This is also about the speed (within 10%) that I can pull files from my internal gigabit network servers.  

 

1.5MB/s sounds eerily familiar - If I recall correctly, that's about the max throughput on a 10Mb/s network.  However, if you were connected at 10Mb/s I wouldn't expect you to get 10x performance after uploading the file back to the target and re-attempting the transfer - unless whatever you're using is lying to you, and not actually uploading the file (because maybe the file on both sides is already the same).  Windows Explorer FTP has a nasty cache that infuriates me and always leads to me putting money in the "curse jar" so I always use FileZilla.  

 

Here's a few questions for you:

  1. How exactly are you transferring the files from the RT target to the Host?
  2. If you're running an FTP server on an external system, what server are you using?  If you upload a file to that FTP server from a separate machine, what throughput rate do you get?
  3. What connection speed does the Network Interface report on the PXI-8106?  There are 2 LEDs on the network interface, the top is an Activity light that will probably be blinking, and the bottom LED is the connection speed with 3 "colors" -- off=10Mb, Green=100Mb, Orange=Gigabit.

 

-Danny

Message 5 of 11
(4,411 Views)

Sustained 40 MB/s? That is my dream speed! The maximum speed I've ever seen here was about 25 MB/s, which is still very good, and that happened only after copying back and forth, which I explained above. Let me go through what I did with screencaps first.

 

These are the test tdms files to transfer in the PXI: an original file witten in the PXI and its copy file. I copied the original file from PXI to laptop and renamed it (added '_copy') and then copied it back to PXI.

 

1.png

 

 

FTP module is transferring the original file to the Filezilla server on laptop. Note the low speed.

 

2.png

 

 

FTP module is transferring the copy file to the Filezilla server on laptop. Note the higher speed.

 

3.png

 

 

This is the FTP sub VI I'm using. It transfers files in sub-folders (one level lower), too. The data connection is set to 'passive' on the FTP Put Multiple Files VI but it doesn't seem to make any difference. I attach the sub VI.

 

FTPvi.png

 

You tried the file transfer with a 500 MB text file. Would there be any difference if the file type was tdms? Would the RAM size matter? It is 2GB here.

And the LED is orange, which means Gigabit.

 

Another question: Is there any way to programmatically stop file transfer in progress? Currently, I just have to restart the PXI.

 

Thank you!

0 Kudos
Message 6 of 11
(4,396 Views)

I tried pulling files from Filezilla Client instead of sending them to Filezilla Server but I didn't see much improvement. I tried setting the number of maximum simultaneous transfers at from 1 to 10 but there was no difference. The transfer rate for a single file usually settles at 1.6~1.7 MB/s. If I download multiple files at the same time, the single-file transfer rate is roughly divided by the number of transfers.

 

I thought about installing Windows OS to utilize the NTFS file system, which may enable faster USB file transfer. If all else fails, would this be a feasible option to try?

http://digital.ni.com/public.nsf/allkb/227CF36870DB7BDB86256F790057F20F

 

0 Kudos
Message 7 of 11
(4,364 Views)

If pulling files onto your laptop using FileZilla Client is equally slow, then I would question whether or not you truly have a 100Mb or Gigabit connection from your PXI to your laptop.  I don't know if you've tried to use a cross-over cable from your PXI to your laptop - both your PXI and Laptop need to either have static IPs or both set to DHCP (so that both will fail-over to Link Local) so that they can talk to one another.  Try to FTP using a cross-over cable, and if things magically perform better then look at your network and see if there's a 10Mb hub in the path from your PXI to your laptop.  Also ensure your laptop's NIC is not set at a fixed 10Mb/s.

 

Another issue to consider here with laptops is hard disk speed.  If you've got an incredibly slow hard disk, your performance will be capped at the rate at which your hard disk can write the data to the disk.  This is generally why gigabit file transfers are only half (or less) the actual gigabit rate, because most hard disks cap out at about 40MB/s - transferring RAM-to-RAM (via a RAM drive or similar) can net 80MB/s transfers on gigabit!

 

If you have another computer handy, slap FileZilla Client / Server on another computer, and see if you can transfer files at a faster rate between computers.  Put your laptop in place of your PXI controller and another computer in place of yours, and see if the transfer speed is similar.

 

Something tells me here that your problem doesn't seem related to the PXI itself, but either in the network topology (limiting bandwidth) or on the laptop itself limiting the speed of the connection or the rate at which you can write to the Hard Disk.

 

-Danny

Message 8 of 11
(4,340 Views)

I am currently using a CAT 6 Ethernet cable but I was using a crossover cable before. It was just as slow unless I'm mistaken. Theoretically, there should be no practical difference, right? My laptop and the PXI have static IPs (10.3.8.73 and 10.3.8.74) with the same subnet mask (255.255.248.0) under TCP/IPv4 settings.

 

4.png

 

5.png

 

 

I tried transferring the same files as above (HMHI_0_130202_0.tdms and its copy file, 50 MB each) in these ways to compare:

 

1. Pulling files from Filezilla Client (one file at a time)

original file: 36 sec

copy file: 11 sec

total elapsed time: 47 sec

 

2. Sending files to a USB flash drive

original file: 52 sec

copy file: 29 sec

total elapsed time: 1 min 21 sec

 

3. Using Filezilla Client for the original file and using USB flash drive for the copy file simultaneously

original file: 1 min 6 sec (FTP)

copy file: 1 min 3 sec (USB)

total elapsed time: 1 min 6 sec

 

The copy file is always transferred much faster under the same transfer method. Isn't the USB transfer rate abnormally slow here? Would this greatly depend on the specific USB flash drive model I use? Would it be any faster if I use a USB external hard drive? The PXI doesn't seem to recognize certain types of USB external hard drives. Is there a list of USB HDD products that will work with PXI-8106 and 8108? I'm just wondering I should invest more in this USB option.

 

I doubt my laptop's hard disk speed might be the limiting factor for FTP transfer since it's not that old. Here is the specification.

 

Processor: Intel Core i5-2520M CPU @ 2.50 GHz 2.50 GHz

RAM: 4 GB

OS: 64-bit Windows 7 Professional

 

What exactly did you do to get 40 MB/s? Could you share any relevant VI you used? I just wanna replicate it here.

I will try to establish FTP connection between two computers later and see if there is any difference.

 

Thank you for all your help!

0 Kudos
Message 9 of 11
(4,331 Views)

Okay, you're right - my deepest apologies.

 

I THOUGHT I was using RT 2012, but in actuality I was using the soon-to-be-released RT 2012SP1.  

 

Using RT 2012 on my PXI-8106, I can use FileZilla Client to upload a 1GB file from my PC to a PXI-8106 at an average rate of 40MiB/s, but when using FileZilla Client to transfer the same 1GB file from my PXI-8106 to my PC I capped out at about 1.7MiB/s.

 

Using RT 2012SP1 on my PXI-8106, I can use FileZilla Client to upload a 1GB file from my PC to a PXI-8106 at an average rate of 40MiB/s, and when using FileZilla Client to transfer the same 1GB file from my PXI-8106 to my PC the rate was the same, at about 41MiB/s.

 

I verified that with RT 2013 on the same PXI-8106 (LabVIEW 2013 is released in public Beta, but the Real-Time component is not in public Beta) I received approximately the same performance as with RT 2012SP1 (RT 2013 was a little better, but I can't tell if that's within normal operating limits of 2012SP1 from my limited testing).  

 

I do not know when the official release date for 2012SP1 is, but we traditionally release Service Packs by March 1 (I think 2010SP1 was released in late January).  I am blissfully unaware of the metrics required to get approval for release, or where it is in the release process, but I personally would expect it to be available on SSP within a few weeks.

 

-Danny

Message 10 of 11
(4,297 Views)