Buffered Downloading of a File using a URL

[Prasaanth]

Hi All, I have been trying to download a file (online - using URL) using the Data Socket Read. But on downloading the file (Minimum Size: 250 MB) the labview memory gets built up i.e I will have to wait till all the packets pertaining to that file are received by the client. I tried using buffered read in Data Socket Open, so that i would to able to obtain instantaneous packets received. But still the memory builds up during the download process. Can anyone please suggest me a solution so that i would be able to obtain the packets as and when they are received by the client instead of waiting for the entire file to be received by the client ? The data downloaded will be in unformatted text and i have attached the VI for reference. Thanks in advance. Regards, S.P.Prasaanth

[Prasaanth]

Please find the Attachment VI

[Sam_Sharp]

If you are keeping the file in memory as you are downloading it then it will of course use lots of memory (essentially an array of bytes). I think the key thing here is to periodically flush the chunks data received to disk.

I don't know if this is possible using the DataSocket functions (it maybe buffers the network data, not the file part?) but it should be possible using the raw TCP/IP functions or perhaps even a .NET library (HttpWebRequest?).

Edit: On looking at your VI, I can't see any obvious cause for the whole file to be in memory unless it's something relating to how DataSockets are implemented internally. In which case - my point still stands that you could try the .NET library or doing the http by raw TCP/IP.

[krist81]

To download  a big file via http, you need to use the Range header keyword in the GET request, to partially download a file.
Example of header HTTP:

"Range: bytes=0-99"

to download the first 100 bytes

Take a look a complete example in the zip file.

[GerdW]

Hi krist,

so you uploaded the very same VI(s) 3 times in different threads!?

- Your VI results in error 59 when I try to download a file from this LabVIEW forum…

- Your VIs could be cleaned up a lot!

- Your main VI should be placed in the main folder, while the subVIs should be located in a sub-folder…

- Your upload is missing a project file…

[krist81]

Dear GerdW,

Sorry for multiple posts, but I don't understand well how to reply to all the discussions that ask to solve the problem of the large file download.

So, this new zip file try to manage all the possible status code replies:

 - "200 OK": 

althought you ask a partial download, the server reply with the complete raw data and status code 200 (the link inside this forum are in this case)

- "206 Partial content"

it respond correctly like this link 

try this link Logo NI

- "301 Moved Permanently" or "302 Found" or "307 Temporary Redirect" or 308 Permanent Redirect

the VI read the new link from HTTP Header parameter "Location:" and redo the request until they get the 206 code 

The update include:

- Now you can see the project file.

- Some calculation inside the Downloader.vi are needed to balance the block size of the request.

try this links to test:

ISO Debian  redirect

Logo NI   small link

NIVISA1600full.exe  redirect