Can Load a URL with Browser, but not with URL Get HTTP Document.vi - Any Takers?

[Wiredup]

OK, desparate people do desparate things:

I used the MS Network Monitor 3 to capture the GET command that IE7 uses to fetch the page. Here it is:

GET /index.htm HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
If-Modified-Since: Fri, 23 Feb 2007 19:51:39 GMT; length=9515
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Host: www.invicis.com
Connection: Keep-Alive

I removed the gzip and If-Modified lines, and put this formatting into the "URL Get HTTP Document.vi" GET text formatting and all is magically just fine. I'll let those who actually understand this stuff figure out why this works (HTTP 1.1 vs 1.0?), but I'm off to tidy up the new vi and chase other bugs.

I'll post the replacement vi to this thread when complete.  Thanks to all for your help!

[Wiredup]

Forgiving the mispellings in the prior post....

As promised, here's the HTTP 1.1 version of  the vi supplied in the Internet Toolkit "URL Get HTTP Document.vi". The attached vi is in LabVIEW version 7.0 format. It does not leverage any of the advantages of HTTP 1.1 as described in:

www.w3.org/Protocols/HTTP/1.1/rfc2616.pdf

but seems to work where the normal .vi would not. 

[tbd]

---

@Wiredup wrote:

"tbd", would you be so kind to save that vi IWebBrowser.vi as V7.0 or earlier (I'm not up to speed on LV Releases yet). I cannot find it by searching the forums.

---

Here you go!

While it's possible to extract the html text via this ActiveX control, it's much simpler to use Datasocket Read to grab a file/document.  See attached.

Tried to open "URL Get Document.vi" - looks like it requires a few sub-VIs also from the Internet toolkit.  You can use "File\Save with options" to save entire hierarchy into an LLB.

Cheers!

Message Edited by tbd on 02-28-2007 09:54 PM

[Wiredup]

Thanks for re-posting the vi's. These are both great. I'm a bit worred from what I've read in the related threads that the [text] trick with the DataSocket may not survive in later releases of LV. You are right, though. It is hard to get much cleaner than the DataSocket call!

Here's the earlier posted vi with referenced sub-vi's. It's too big to attach, so I posted externally here. Now that I better understand HTTP 1.1, there's much that can be done with this vi to make it better, such as:

1) Hold open the connection for subsequent calls (avoids re-connecting overhead on multiple calls to same site)

2) Support for gzip allowing compressed transfers

3) ....

Perhaps another day.

Thanks again for your help.

Message Edited by Wiredup on 03-01-2007 08:24 AM

Message Edited by Wiredup on 03-01-2007 08:26 AM

[Wiredup]

...and just a quick followup post: If you use the vi attached to this thread:

URL Get HTTP 1-1 Document.vi

be sure and mark it as reentrant. It was unmarked for editing and debugging, and I forgot to put it back to reentrant before posting. The corrected vi is attached (LV 7.0 or later and Internet Connection Toolkit required).

[dan_u]

This is indeed interesting. But what I find strange is that your router or software firewall or whatever blocks the HTTP 1.0 protocol.
Anyway, it looks like the internet toolkit would require some work from the NI side...
HTTP 1.1 is definitely much better in terms of connections, you just open it once and keep it open to send several commands and receive several pages. But I'm not familiar enough with the HTTP 1.1 protocol to correctly modify the URL Get HTTP Document VI. Your VI works, but is it really compliant to the 1.1 standard?
Anyway, I'm glad you have a solution that works!

[Wiredup]

The 1.1 version I think is pretty compliant - although it uses little (only the bare mins) of the 1.1 protocol which forces the server to fall back to almost 1.0 protocol responses.  It's biggest area of sketchiness is the reporting of the "user-agent". I just mimicked the IE7 user-agent text. There's probably better text to use here.

Maybe someone who's got some time (perhaps at NI) can work on this vi to make it a full-featured HTTP 1.1 URL document fetcher.

[Steven_C]

Hi Wiredup,

Thank you for your post, we certainly appreciate the feedback! In order to provide this suggestion to R&D for it to be potentially included in a future version of our product, the best place to go would be our product suggestion page. The following link is where official product suggestions to R&D are made:

Product Suggestion Center

http://digital.ni.com/public.nsf/allkb/EDA7C01C684ACB6286256FF0000238D5

Thanks again for the feedback Wiredup and have a great day!

Best regards,

Steven

[Wiredup]

Steven,

I posted the suggestion using your link. Thank you.

Best Regrards,

Brad

[Wiredup]

In the spirit of saving a future Forum viewer some problems, I discovered the previously posted HTTP 1.1 URL Get Document vi had a glitch when trying to open documents with 'chunked' transfer-coding. The ability to handle chunked documents is a requirement of a minimalist 1.1 compliant reader. The attached .llb contains the previously posted vi along with required sub-vi's to properly read 1.1 documents (like yahoo's website). These are in Version 7.0