Improve performance of Copy/Move VIs in Labview RT on PXI-8xxx controllers

[twopac]

At the moment Labview RT does not support high speed USB 2.0 (EHCI) functionality on PXI-8xxx controllers even if the controller has supporting hardware.  We implemented a copy operation to transfer data files from a harddrive on the PXI chassis to an external USB connected flash drive using the copy VI, and only obtained speeds at USB 1.1 (OHCI) levels.  We also tried a move version using the Move VI with similar results.  The controller used in our tests was a PXI-8186, which has USB 2.0 ports.

My discussions with NI Support and the R&D team indicate this a limitation imposed by the ETS RTOS from Phar Lap that Labivew RT runs on.  This is inline with the fact that VxWorks supported controllers have access to EHCI speeds as per this faq:

http://digital.ni.com/public.nsf/allkb/BE80D012BB933B54862572D6004FE5F9

Could we get high speed USB support implemented in Labview RT/ETS?

Here are some benchmarks that we obtained:

	Size (kB)	Time	MB/sec
Laptop to USB	55,535	14	3.97
Labview Copy VI Laptop to USB	55,535	14	3.97
Laptop FTP to RT	55,535	17	3.27
Labview Copy VI RT to USB	55,535	73	0.76

[LabVIEWRT]

Actually at the moment LabVIEW RT on PXI *does* support speeds faster than USB 1.1 if the controller has EHCI hardware - we're currently able to do about 12MB/s under optimal conditions.  However, that's only reads/writes from/to memory - not to/from another storage medium.  The reason for this is because the cache size and chunk/block size required to get optimal performance is different for disk storage versus USB transfers on Pharlap ETS, and so when you combine them you currently cannot get optimal performance on both.  If you were to directly write the data from memory to USB it is faster, since you're not reading one chunk/block size from one storage medium and then writing that same chunk/block size (which may not even be close to ideal) on another.  You'll find you will even get a different performance factor if you write to a USB drive via FTP!

If the support engineer that you spoke with did not clearly explain this to you, I apologize for the confusion.  You are correct, though, that our speeds are not able to hit the practical maximum of USB 2.0 (about 40MB/s) even on our fastest RT controllers under optimal conditions.  We are actively investigating USB technologies to improve our performance and capabilities for LabVIEW Real-Time (we'd be foolish not to with USB 3.0 coming over the horizon, and the growing need to support more than just Mass Storage devices), and so this will be one of the major areas where we will look to improve our current performance and functionality offerings for USB on LabVIEW Real-Time.  

-Danny

[twopac]

Thanks for the response Danny.  The numbers for RT/USB transfer line up with v1.1 expected speeds (i.e. 0.76*8 ~ 6Mb/s).  The support engineers I talked to seemed to agree that EHCI support did not exist under ETS, and the FAQ mentioned in my original post agrees with this.  Is the FAQ incorrect?

The last support engineer I talked to said the R&D team was not surprised at the low transfer rates I was using, keeping in mind that I am using built in VIs for this.

I heard from one support engineer that 4092 bytes is the optimal size for the USB transfers, and I am going to assume ETS is using 512 bytes reads from the disk.  We might be able to address this issue by allowing the VIs to be configured to use 4092 bytes chunks instead 512 bytes.

Thoughts?

[twopac]

Incidentally the controller in question is a PXI-8186, which is spec'ed as having USB 2.0 ports.

[LabVIEWRT]

The FAQ does indeed need to be updated.  

Updates for the FAQ:

- UHCI/OHCI/EHCI are *controllers* not *devices*

- Low-speed / High-Speed would have been better expressed as USB 1.1 and USB 2.0 because 1.1 also support full-speed (1.1 thumb drives would probably do full-speed)

- NI PharLap ETS most certainly does support EHCI controllers (whereas VxWorks-based cRIO and cFP controllers do not), but it will also not tell you when it will not perform at USB 2.0 "High Speed" because of device incompatibility (or perceived device incompatibility); the USB controller and device will settle for the slowest of the two.

- USB drive letters start with Drive U: - that way there is little risk of collision between drive letters on IDE/SATA drives versus USB.  That may not always be the case, but currently it is the case.

- While it's true that ETS could use a Reliance-formatted USB drive, there currently does not exist -anywhere- a means of formatting a USB drive for Reliance that will work for ETS.  The only way to create a USB drive with Reliance is via a tool that Reliance provides, but ETS uses a slightly different partition ID for its Reliance partitions (since Reliance did not reserve a partition ID) and so a Reliance-formatted USB drive would never work in ETS.  Good idea, though.  🙂

You are correct, ETS uses a much smaller blocksize read from the disk storage than is optimal for USB.  Windows has a massively different caching mechanism than we do in ETS, and our USB performance is crazy sensitive to the chunk size being used (and we're not currently doing anything to compensate for differences).  I agree with the last support engineer you spoke with, I am not surprised at the low transfer rates you are getting simply because of the differences in block/chunk sizes and cache sizes for the different storage mediums/buses; this hasn't been an area we've attempted to optimize in quite some time.  I sincerely believe that if you were to stream data directly from memory to one of the storage mediums (USB or On-Disk Storage) you'll find your performance numbers to be very different.  

Using the Copy/Move VIs just calls into the OS to perform the required action, so it would be a modification in the OS (instead of the VIs) to align the block/chunk sizes used to find a sweet spot in performance in this specific case (or generically when dealing with different storage types).  This is something we can look to address in future releases of LabVIEW Real-Time.

-Danny

[twopac]

Very useful information Danny. 

Since I already have a version that iterates through a list of files/subdirectories to copy them individually, it shoulds like I could really improve the performance if I wrote my own copy function that read 4K chunks from source files and wrote them to destination files on the USB drive.  Is this correct?

Our production version of Labview RT is v8.5, does it have EHCI support?

[LabVIEWRT]

LabVIEW RT 8.5 should definitely have access to EHCI - there have been many changes/upgrades to the USB support since LabVIEW RT 8.5, but from what I recall it has been primarily support for newer targets and newer onboard chipsets (such as ICH8/9/10) - I don't believe your ICH4 support on the PXI-8186 should have changed any, and should support EHCI.

We ran some tests ourselves, and everything concurs with your assumption that by reading larger chunks from source files and writing them to destination files you can improve USB performance significantly.  We tested varying chunk sizes from 1k (which is what we use for the hard disk) all the way through to 64k, and the performance curve is fairly linear up through 32k chunks (and then levels off and/or dips past 32k).  There is a decent performance benefit by using 32k chunks for USB writes, but the significant performance benefit is using 32k chunks in USB reads.  4k is the inflection point that you would start with if you were to test yourself the performance of HDD-to-USB performance by varying the chunk size being written to the USB side.

-Danny

[twopac]

Thanks Danny.  I will modify my code and try it out.

[twopac]

Just to follow up, the custom write managment does make a difference:

Block Size (Bytes)   MB/S
4096                       2.727272727
16384                     4.166666667
32768                     4.918032787
65535                     1.171875
131070                   1.923076923
1000000                  2.307692308
10000000                2.459016393

This is much better than the sub 1MB/s we were seen with the generic copy/move VIs.

Going from 32K to 64K seems to be causing a buffer overflow, so the writes block for a while until the underlying OS function recovers and starts taking in writes.  At 32K, we seem to be balanced in terms of the fill and drain rates for the OS buffer (seems to be around 1M) for a USB flash drive.  With an external HD, we get faster numbers so it seems the flash drive is the bottle neck at least for single thread writes.

If I wanted to add more drives, I am assuming the next letter for a USB drive is V:, is this correct?

Thanks for your help Danny.

4096	110	2.727273
16384	72	4.166667
32768	61	4.918033
65535	256	1.171875
131070	156	1.923077
1000000	130	2.307692
10000000	122	2.459016

[Texas_Diaz]

Yes, that is correct.

-Danny