Send huge data through the network

[Logic]

Hi all,

I need to transfer large image (around 200MB) through the network to the neighborhood PCs for processing. The time i have is very short (Say 4 seconds).

Since, i need to send the image to multiple PCs, i thought UDP will be helpful. But the LabVIEW UDP vis having limitation of 548 bytes reading for each run. So i need to run in a loop and get the image. Building the string also will be of great memory related task. So the time taken is very high (in the order of minutes). often, i am missing some of the bytes during the operation.

Then i have tried TCP/IP, planning to send to only one PC. There also when the size is more than 78MB, i am not able to send in single time. Is it so? But i have not seen any limitations in the documents. For 78 MB it take around 18 seconds. But the data is safe.

The great disadvantage of these two method is, i need to flatten as a string and and rebuild the image in the other end.

I have tried one more option also, using save as bmp, and read from the network. This work quite better compared to the other two. Time take to transfer 200MB image is 24 seconds ( 5 seconds to save in the same PC + 19 seconds to read through the network). Still it is far from my requirement. One advantage here is, i can read as image datatype it self.

I have used 100Mps LAN, yes, i can go far 1GPS. Then any other idea how to transfer these huge files in quick manner.

Thanks,

Logic

[chilly_charly]

Just a short comment : do not confuse megabytes (the size of the image) and megabits (the maximum transfer rate over the network). As a rule of thumb, a 100 Mb network can transfer as much as 100 / 10 = 10 megabytes/s. So your 200 megabytes image will require 20 seconds. Even with a 1 Gb network ( 2 secs) you may be in trouble if your time constraint is very narrow.

[Logic]

Hi cc,

Yes, i was quite clear on that. The point i want to explain is - because of the limitation of the VIs (like 548 bytes for UDP & 78MB for TCP/IP) we are not able to achieve this.

1. I need to flatten to string the 200MB image.
2. I need to send this in a loop (loop delay may needed to synchronize)
3. I need to concatenate at the other end
4. Unflatten to image

Look at the timing i have send in my last mail. Its huge for UDP & TCP/IP

But the File I/O looks simple and comes closer to your calculation. In my trial i have sent 180 Mega byte image - it took 19 seconds to read from the network PC Harddisk in 100mbps line - which looks good. In a way File I/O also, should do the same (Flatten - read thru network -May be in an ideal way). Anyway, I have left with 2 questions now.

1. In File I/O, i need to first write image in to Harddisk of the same PC - that time going to remain the same, any ideas on reducing this? Any chance for without saving into HD - transfer?
2. Will it reduce directly 10 times if i go for 1Gbps? (200MB in 2 Secs?) Anything else decides this timing?

Thanks,

logic

[unclebump]

Could you compress the flattened string data before sending and inflate on the other end?? Here is a link to the Zlib deflate and inflate.

http://digital.ni.com/public.nsf/allkb/529A66352E02A25686256C2F00783073

[Ben]

1) Map the drive on the "other" machine and just save your file there.

2) Search this site for "Nagal" (?) algorithm. There is an "opimal size" for TCP/IP data transfers.

3) Forget UDP, it is just good for junk mail.

4) Concider VI server and do a "call by reference" on a VI running on the target.

5) TCP/IP looses efficiency after you start to use more than 40% of the bandwidth. Do not expect to get dedicated use of the network unless you can pull all of the other nodes off the network. I have added extra NIC's to allow this option.

Gotta run....

Ben

[Ben]

Correcting myself.

its is ethernet that looses efficiency. It is based on CDMA. CDMA is Collision Detect Mulltiple Access. Simply stated, it works like a phone meeting. If someone has something to say and no one else is talking they just start talking. If hear they are talking over someone else, they stop talking and execute a delay. They again wait for a quiet moment and try it again. When you get into high throuhput situations, there is a high likly hood of a collision.

The link below should help with ethernet throughput issues. In this thread 70 MHz results were reported.

http://forums.ni.com/ni/board/message?board.id=170&message.id=71098&requireLogin=False

Ben

[tst]

---

@Ben wrote:

If hear they are talking...

---

Yoda?

Have you gone AWOL?

Are you also like our englishly-impaired friends, using august for the "vacance", or are you just starting NI-week a couple of weeks early?

And BTW, the algorithm is called Nagle.

[Ben]

Hi tst,

Thank you for hearing what I meant and not what I said. That is what my internal dialog sounds like before I do the transform into something that resembles english.

I have two designs to get out this week and have no time to proof read. That said, I could not sit back and let LV look bad when playing the comm. game.

Ben