Re: Google Trends Data and Mathematica
- To: mathgroup at smc.vnet.net
- Subject: [mg103896] Re: Google Trends Data and Mathematica
- From: Jonah <jgollub at gmail.com>
- Date: Sun, 11 Oct 2009 08:06:07 -0400 (EDT)
- References: <hahsrv$os$1@smc.vnet.net> <han6j2$pme$1@smc.vnet.net>
On Oct 9, 4:25 am, Chris Osborn <chris... at gmail.com> wrote:
> On Oct 7, 6:08 am, jroge <jr... at mac.com> wrote:
>
> > I am looking for some advice for directly importing data from Google
> > Trends? I am able to download the CSV files from Google and then load
> > into Mathematica easily enough, but I would rather skip the
> > intermediate steps and directly pull the data from Google into my
> > Mathematica notebook. I think where I am getting stuck is that Google
> > requires authentication and programming that into Mathematica is a bit
> > beyond my skill sets. Any thoughts?
>
> > Jon
>
> Have you tried "Import?" It allows you to specify a URL; perhaps it
> will suffice.
>
> Chris
Hi Jon,
I've been working on a way of importing data using mathematica's java
functionality (using apache's Httpclient). It worked until a couple
weeks ago. It still gives a successful http code response when
requesting data but that data is not read correctly. It seems not to
be recognizing the data format. I'm not sure what changed... maybe
google updated something on its webpage. I've been distracted from
working on it lately but I'd like to figure out what's going on. If
anybody has any suggestions on how to fix it I'd definitely appreciate
the help. Don't forget to change the PASSWORD and EMAIL to correlate
to your account.
<< Jlink`
passwordattribute = "PASSWORD"; (*google account password*)
emailattribute =
"EMAIL at gmail.com";(*google email*)
(*Initiate Http client*)
client = JavaNew["org.apache.commons.httpclient.HttpClient"];
client@getHostConfiguration[]@setHost["www.google.com", 443, "https"]
(*prepare Request Parameters*)
gat3t = JavaNew["org.apache.commons.httpclient.NameValuePair", "GA3T",
"ouVrvynsses"];
galx = JavaNew["org.apache.commons.httpclient.NameValuePair", "GALX",
"ouVrvynQwdf"];
continue =
JavaNew["org.apache.commons.httpclient.NameValuePair", "continue",
"http://www.google.com/insights/search/#";];
nui = JavaNew["org.apache.commons.httpclient.NameValuePair", "nui",
"1"];
hl = JavaNew["org.apache.commons.httpclient.NameValuePair", "hl",
"en-US"];
rmshown =
JavaNew["org.apache.commons.httpclient.NameValuePair", "rmShown",
"1"];
persistent =
JavaNew["org.apache.commons.httpclient.NameValuePair",
"PersistentCookie", "yes"];
accountId =
JavaNew["org.apache.commons.httpclient.NameValuePair", "Email",
emailattribute];
pass = JavaNew["org.apache.commons.httpclient.NameValuePair",
"Passwd", passwordattribute];
service =
JavaNew["org.apache.commons.httpclient.NameValuePair", "service",
"trends"];
nameValuePairs = {gat3t, galx, continue, nui, hl, rmshown,
persistent, accountId, pass, service };
postMethod =
JavaNew["org.apache.commons.httpclient.methods.PostMethod",
"https://www.google.com/accounts/ServiceLoginBoxAuth";];
postMethod@setRequestBody[nameValuePairs]
client@executeMethod[postMethod]
postMethod@releaseConnection[]
(*make request for data*)
insights[query_String] := (
redirect =
JavaNew["org.apache.commons.httpclient.methods.GetMethod",
"http://www.google.com/insights/search/overviewReport?q="; <>
query <> "&cmpt=q&content=1&export=1"];
client@executeMethod[redirect];
(*get data and drop first 5 lines "header"*)
(* data = Drop[ImportString[redirect@getResponseBodyAsStream[]],
5];*)
data=Drop[ImportString[redirect@getResponseBodyAsString[]],5];
insightDateAndData = {DateList[
StringSplit[#[[1]], " - "][[2]]], #[[2]]} & /@ (Take[
data, (Position[data, {""}][[1]])[[1]] - 1]);
redirect@releaseConnection[];
Return[insightDateAndData]
)