Search google and download results (Sentiment Analysis)

[Zhounmg]

I'm building a sentiment analysis (sum of negative vs positive words) that I'd be happy to share with you once done. I'm half way through. Currently I'm able to download RSS feeds and analyse them. I would like to know if anybody here has figured a way or knows a way to search google (without having to open a browser) and download the results. It would of been more than perfect if the link of those results could also be downloaded. I know this would involve some javascript. Can anybody give me any advice or a way to solve this problem. I would like to use the TCP connection. Any feedback would be grealy appreciated. Thanks

[James.M]

Google has implemented lots of safeguards to ensure that it's search engine can't be scraped.

So the only way to do it is with a webbrowser. I think this is your best option.

[altenbach]

Is this related to LabVIEW? Just curious since you are posting here but don't mention it.

[MilanR]

Twitter search feeds could also be a good source of text to do the analysis. If you are using LabVIEW, I worked on a Twitter toolkit that you can use to interface with and make queries to Twitter from LabVIEW: https://decibel.ni.com/content/docs/DOC-25055

[tst · Accepted solution]

To expand on James' reply, the example he linked to opens the page in a browser window. Here's an example which uses the .NET browser, rather than the ActiveX one, and shows interacting with the page - http://forums.ni.com/t5/LabVIEW/Mouse-event-in-embedded-internet-explorer/m-p/2853440#M832224

This doesn't demo reading data from the page, but it does show how to interact with it. Note that once you have the URLs from the search, you might be able to use the HTTP VIs or Datasocket Read to get the HTML of the pages, but you would still need to parse the text out of the HTML, which is page dependent and a lot of work. You might be better off using something like Twitter, which has an API, as suggested above, or looking for more organized sources of text. I seem to remember seeing a presentation (probably a TED talk) by people who did exactly that - scraped Twitter or Facebook to do analysis on that content.

[Zhounmg]

I have already implement twitter feeds in the program, but thanks. 

[Zhounmg]

I could pretend to be a webbrowser and google wouldn't know, to work with google that would involve java. So right now we should be talking about tcp, websockets & java.....anybody has a solution? Does labview have a plugin that I could use ? 

[Zhounmg]

I will try this out and let you know. Thanks TST. Smart

[Zhounmg]

Hello tst, could you hit me up with that template you built please. Thanks in advance...

[tst]

I'm not sure which template you're referring to, but the code in the other thread is pretty much all I've done with this. The code is a snippet, which means you can save the image to your hard drive and then drag it into LV and LV will convert it into actual code. Like I said, that is all I've done with this, so if you want to extract data from the document you'll have to figure out how to parse the document on your own. If you're not familiar with DOM you'll need to read up on that and read the documentation for those .NET classes which I used on the MSDN. Like I said, because the way content is placed in web pages isn't something which is standardized, it might be impractical to get the text you want automatically.