Comments on: Fetching Android Market Stats with Python, MozRepl, and BeautifulSoup

By: craiget

craiget — Sat, 16 Jan 2010 02:51:15 +0000

Oops.. I may have mangled the script when copying it between blogs. The 2nd profile thing is not strictly necessary. I have two Google accounts, so I am using two profiles here so they don’t stomp on each other. You can probably drop the “-P profile” flag altogether without any problem. There’s more info about setting up a second profile here: http://support.mozilla.com/en-US/kb/Managing+profiles.

There was another typo on that line too – both should be fixed now.

By: Henrik

Henrik — Sat, 16 Jan 2010 01:50:03 +0000

Hi there,

You’ll have to excuse me, but I get this:

os.system(“firefox -no-remote -P %s &” % (profile, url))
NameError: name ‘profile’ is not defined

I don’t think I understand how to handle that profile?

Henrik

By: Will

Will — Wed, 01 Jul 2009 05:53:42 +0000

The free version of the iMacros Firefox addon doesn’t allow you to scrape webpages so isn’t a choice for some.

MozRepl works or at least it did for an AJAX-heavy website that seemed otherwise impossible to scrape with mechanize.

There doesn’t seem to be a convenient way to check the ‘readyState’ of a AJAX request so the automation can be flaky – I had several time.sleep’s interspersed in the code.

By: Mike

Mike — Tue, 26 May 2009 20:40:01 +0000

nice, but wouldn’t that be easier with the iMacros Firefox addon?

By: James Grafton

James Grafton — Fri, 15 May 2009 14:19:43 +0000

I have recently been attempting this, using a Python module called mechanize. Thus far I have manged to authenticate with the marketplace server and grab the marketplace webpage contents…..however the actual stats are generated via a JSON post request which I am having difficulty emulating…Feel free to try and get it working:

def gatherMarketplace():
print “gathering marketplace stats for “+product.product_name
##Initialise cookies
cookiejar = cookielib.LWPCookieJar()
cookiejar = urllib2.HTTPCookieProcessor(cookiejar)
opener = urllib2.build_opener(cookiejar)
urllib2.install_opener(opener)

#Create browser object
br = Browser()
br.set_handle_robots(False)
response = br.open(“https://www.google.com/accounts/ServiceLogin?service=androiddeveloper&passive=true&nui=1&continue=http%3A%2F%2Fmarket.android.com%3A80%2Fpublish&followup=http%3A%2F%2Fmarket.android.com%3A80%2Fpublish”)
responseBody = response.read()
form = br.select_form(nr=0)
br["Email"] = “gbjgandroid”
br["Passwd"] = “********”
response2 = br.submit()
#At this point response2.read() will be you android market home page

This is the post request that I am trying to emulate:

5|0|4|http://market.android.com/publish/gwt/|20D26AAE72C63DE79F3EE065E818E545|com.google.wireless.android
.vending.developer.shared.AppEditorService|getFullAssetInfosForUser|1|2|3|4|0|