2

Getting your stats from the Android Marketplace with PHP/cURL

by craiget | May 15th, 2009

A few weeks ago I mentioned one way to get your developer stats off the Android Developer Console automatically in the post “Fetching Android Market Stats with Python, MozRepl, and BeautifulSoup“. Unfortunately, despite being very awesome, Firefox + MozRepl is not super-great for this task. When a plugin is updated, Firefox hangs on startup. That’s fine, but it kinda sucks for scripting. I’m sure there’s a way around it, but that difficulty makes a good excuse for coming back to solve this problem the right way.

Following is a PHP script that uses cURL to login to the Developer Console and grab the market stats. Unfortunately, Google’s app is written in GWT and the its Javascript is completely obfuscated. The market stats are fetched as JSON data and then somehow parsed, but I haven’t been able to figure out how exactly. If you run this script (or just look using Firebug), you’ll see that the JSON is a gigantic array. While the data of interest are clearly present in this array (total downloads, current installed base, rating, etc..), I haven’t been able to figure out how to parse it reliably. If you’ve tried this and figured it out, I’d love to know!

This script was assembled from a bunch of random PHP/cURL tutorials and may contain redundancy, unnecessary cURL settings, etc. Python fans, see the comments of my other post on this topic where a kind soul has demonstrated the same thing in Python using mechanize.

<?php
//setup a temp file to store cookies
$ckfile = tempnam ("/tmp", "CURLCOOKIE");
 
//do google authorization
$data = array('accountType' => 'GOOGLE',
          'Email' => 'YOUR_ACCOUNT_EMAIL_HERE',
          'Passwd' => 'YOUR_ACCOUNT_PW_HERE',
          'source' => '',
          'service' => 'androiddeveloper');  
 
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.google.com/accounts/ClientLogin");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
 
//grab the AUTH token for later
$auth = '';
if($info['http_code'] == 200) {
    preg_match('/Auth=(.*)/', $output, $matches);
    if(isset($matches[1])) {
        $auth = $matches[1];
    }
}
 
//login to Android Market
//this results in a 302
//I think this is necessary for a cookie to be set
$ch = curl_init ("http://market.android.com/publish?auth=$auth");
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//go to the Developer Console
$ch = curl_init ("http://market.android.com/publish/Home");
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//grab the JSON data
//perm and postdata seem to have changed in the last 6 months
//if the script isn't working, try using firebug to inspect the Request when
//http://market.android.com/publish/editapp gets fetched
$perm = "81E29277804F7729E9B743A43B2EFD07";
$headers = array(
    "Content-Type: text/x-gwt-rpc; charset=utf-8",
    "X-GWT-Permutation: $perm",
    "Referer: http://market.android.com/publish/gwt/$perm.cache.html");
//not sure what x-gwt-permutation means, I think it may have to do with which version of GWT they serve based on your browser
$postdata = "5|0|4|http://market.android.com/publish/gwt/|09C42EAE15B55219550B2D800FAC1644|com.google.wireless.android.vending.developer.shared.AppEditorService|getFullAssetInfosForUser|1|2|3|4|0|";
$ch = curl_init ("http://market.android.com/publish/editapp");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//now what?!?
echo('<pre>');
$output = json_decode(substr($output, 4));
print_r($output);

If you run this script and are willing to send me your stats, that would be super-helpful. Maybe I’ll be able to get enough data to figure out why some apps have more fields than others. With only 3 apps currently on the market, I don’t have much to go on. Feel free to obscure your data, but please make the changes obvious and note whether the app is free/paid and what part of the market it appears on (games/apps and sub-category). Here is a link to my best guesses so far in an Excel worksheet: market-json

0

Geocoding in Sharepoint Lists

by craiget | April 14th, 2009

Today I was thinking it would be nice to do some Geocoding in Sharepoint. Specifically, I wanted to make it so list items could have longitude and latitude fields that could be populated with a button click from the EditItem page. Geocoding is pretty easy to do with both Google and Yahoo. For my situation, Yahoo seemed more appropriate.

My first intuition was that this should be easy to do with Javascript. Just call the Yahoo Maps API with the right data and parse the response. Except.. this usage violates the same site origin policy for Javascript. Drat. Well, there are several things that can be done. If you’re handy with C# and M$ technologies, you can just create a proxy on the same server. Unfortunately, I don’t know the first thing about the Microsoft stack and I’m too lazy to learn. As an alternative, I opted to create a PHP proxy on another server and force it to return JSON, which skirts around the same site origin problem.

First, the code for the PHP proxy which lives somewhere besides the Sharepoint server. This proxy forwards requests to Yahoo, parses the response, and emits JSON back to the caller.

<?php
//get params from request
$appid = 'YOUR_YAHOO_APPID';
$street = $_GET['street'];
$city = $_GET['city'];
$state = $_GET['state'];
 
//build new request
$req = 'http://local.yahooapis.com/MapsService/V1/geocode?';
$req .= 'appid=' . $appid;
$req .= '&amp;street=' . urlencode($street);
$req .= '&amp;city=' . urlencode($city);
$req .= '&amp;state=' . urlencode($state);
 
//fetch XML using cURL
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $req);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = trim(curl_exec($ch));
curl_close($ch);
 
//parse XML
$xml = simplexml_load_string($result);
$lat = $xml-&gt;Result[0]-&gt;Latitude;
$lng = $xml-&gt;Result[0]-&gt;Longitude;
 
//return JSON
echo $_GET['jsoncallback'] . "({lat: \"$lat\", lng: \"$lng\"})";
?>

Now, the Javascript part of this uses jQuery to make life much much easier. It basically injects a new Button onto the page. When clicked, it builds a request from the location data on the page, sends the request to the proxy, parses the response, and puts the latitude and longitude into the form fields. Then the user can click ‘Save’ as usual. This code needs to be added to the EditItem.aspx page for that particular list. You also need to have jQuery on the page. In my case, I just included them both as external scripts to keep my changes to .aspx pages to a minimum.

$(document).ready(function() {
	//create a new Button, match Sharepoint styles
	var geoButton = $('&lt;input type="button"&gt;').attr({'class':'ms-ButtonHeightWidth', 'value':'GeoCode'});
 
	//add our Button after the default 'Cancel' Button
	$('.ms-formtoolbar .ms-toolbar:last').after($('&lt;td&gt;&amp;nbsp;&lt;/td&gt;').attr({'class':'ms-separator'}));
	$('.ms-formtoolbar .ms-separator:last').after(geoButton);
 
	//wrap Button in a Table to match Sharepoint's style
	geoButton.wrap($('&lt;td&gt;&lt;/td&gt;').attr({'class': 'ms-toolbar', 'no-wrap':'true'}))
		 .wrap($('&lt;table&gt;&lt;/table&gt;').attr({'cellspacing':'0','cellpadding':'0','width':'100%'}))
		 .wrap($('&lt;tbody&gt;&lt;/tbody&gt;'))
	         .wrap($('&lt;tr&gt;&lt;/tr&gt;'))
                 .wrap($('&lt;td&gt;&lt;/td&gt;').attr({'nowrap':'','align':'right','width':'100%'}));
 
	//onClick, perform geoCode and put Long/Lat into form fields
	geoButton.click(function() {
 
	//get data from form fields
	var street = $('input[title=Street]').val();
	var city =  $('input[title=City]').val();
	var state =  $('input[title=State]').val();
 
	//fail early if some data is absent, since we wouldn't get a good geocode
	if(street == '' || city == '' || state == '')
  	    return;
 
	//wrap data into a URL so we can do an HTTP GET
	var address = '&amp;street='+street+'&amp;city='+city+'&amp;state='+state;
 
	//use jQuery.getJSON to avoid that pesky cross domain security restriction
	//?jsoncallback=? is a peculiarity required by jQuery, the server must echo this back
	$.getJSON("http://path_to/yahoo-geocoder.php?jsoncallback=?"+address,
		function(json) {
    		    //async callback, unpack the data
 		    var lat = json.lat;
		    var lng = json.lng;
       		    //simple error detection
		    if(lat == '' || lng == '') {
			//show '!!!' after form fields to indicate that geocoding failed
			$('input[title=Latitude]')
			.after($('&lt;span&gt;!!!&lt;/span&gt;').attr({'id':'error-lat'}).css('color','red'));
			$('input[title=Longitude]')
			.after($('&lt;span&gt;!!!&lt;/span&gt;').attr({'id':'error-lng'}).css('color','red'));
		    } else {
			//success, place results into form fields
			$('input[title=Latitude]').val(lat);
			$('input[title=Longitude]').val(lng);
			//remove any previous error indicators
			$('#error-lat').remove();
			$('#error-lng').remove();
		    }
		});
	});
});
0

SPARQLing The Highest Point in Every US State

by craiget | February 4th, 2009

In a previous post, I mentioned that it should be pretty easy to use SPARQL to make a map of the highest point in each of the 50 US States. Having written that, I thought I should maybe actually, you know, try it.

The following chunk of code uses ARC2, an rdf/semantic web library for PHP to query the dbpedia endpoint and then put the results on a Google Map.

To try this out, you need to:

  1. Have a functional PHP installation
  2. Download ARC2 into your web path (no setup required)
  3. Set the path to ARC in the code below
  4. Get a Google Maps API Key (free)
  5. Set your API key in the code below
  6. Run

Note that this is a demo and is written to be easy to run – a real application might separate the data logic from the webpage and make more sophisticated use of Javascript/Google Maps API.

<?php
//include ARC2 libraries
include_once("path/to/ARC2.php");
//instantiate a RemoteStore
$config = array('remote_store_endpoint' => 'http://dbpedia.org/sparql');
$store = ARC2::getRemoteStore($config);
//build the SPARQL query
$q = '
PREFIX dbpedia2: <http://dbpedia.org/property/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>
SELECT ?state ?mtn ?lat ?long
WHERE {
?state skos:subject <http://dbpedia.org/resource/Category:States_of_the_United_States> .
?state dbpedia2:highestpoint ?mtn .
?mtn geo:lat ?lat .
?mtn geo:long ?long
}
';
//process the results
$results = array();
if ($rows = $store->query($q, 'rows')) {
foreach ($rows as $row) {
$state = substr($row['state'], strlen("http://dbpedia.org/resource/"));
$mtn = substr($row['mtn'], strlen("http://dbpedia.org/resource/"));
$lat = $row['lat'];
$lng = $row['long'];
$results[] = array($state, $mtn, $lat, $lng);
}
}
?>
<!DOCTYPE html "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<title>Google Maps JavaScript API Example</title>
<script type="text/javascript" src="http://maps.google.com/maps?file=api&amp;amp;v=2&amp;amp;key=YOUR_KEY"></script>
<script type="text/javascript">
function initialize() {
if (GBrowserIsCompatible()) {
var map = new GMap2(document.getElementById("map_canvas"));
map.setCenter(new GLatLng(37.4419, -122.1419), 3);
map.addControl(new GMapTypeControl());
map.addControl(new GLargeMapControl());
<?php
//populate map with results
foreach($results as $result) {
list($state, $mtn, $lat, $lng) = $result;
echo("map.addOverlay(new GMarker(new GLatLng($lat,$lng), {title: '$mtn'}));n");
}
?>
}
}
</script>
</head>
<body onload="initialize()" onunload="GUnload()">
<div id="map_canvas" style="width: 100%; height: 100%"></div>
</body>
</html>

* Using substr() to chop off “http://dbpedia.org/resource/” from the names is probably cheating. I think you’re supposed to use rdfs:label@en instead.

2

Hello Sharepoint, Meet PHP

by craiget | January 22nd, 2009

I am not a huge fan of Sharepoint, but we are starting to use it at work and it does do certain things well.

Trying to make Sharepoint do something non-obvious can be a lot of fun, especially on a slow day.

So I was trying to create a simple feedback form that would be stored by Sharepoint as a List. While this is well within Sharepoint’s capabilities, it ends up being kind of a pain. You end up either creating a custom WebPart or designing something in InfoPath or writing ASP code. None of these options really appeals to me, mostly because I don’t know much about Sharepoint and I don’t have priviledges to deploy code on the server. Luckily most of the Sharepoint API is exposed through Web Services. Just make a PHP-backed webpage, produce the XML and fire it off to Sharepoint – no problem, right?

Unfortunately, this is easier said than done. SOAP is typically an all-or-nothing proposition. You don’t “kind of” get a response, you either get success or an opaque error message.

I spent the better part of a morning piecing this together from a zillion blog posts. There are two big tricks – getting the XML right and using NTLM authentication.

This code sample requires the NuSOAP library for PHP – you could also do it with PHP’s native SOAP I suppose. You will need to have a fairly recent version of CURL to get the NTLM authentication.

<?php
/* Requires the NuSOAP library */
require_once('lib/nusoap.php');
 
/*  Your username and password, separated by a colon
    Domain may be optional, depending on your setup */
$auth = "domainusername:password";
 
/*  Location of the Lists.asmx file
    If the list is in a subsite, the subsite must be in the path */
$wsdl = "http://domain.com/some-site/some-subsite/_vti_bin/Lists.asmx?WSDL";
 
/*  GUID of the list */
$guid = "{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}";
 
/* Setup NuSOAP
   Sharepoint requires NTLM Authorization
   You need a fairly recent version of CURL installed for this	*/
$client = new soapclient($wsdl, true);
$client->setCredentials("","","ntlm");
$client->setCurlOption(CURLOPT_USERPWD, $auth);
 
/*  XML for the request, add extra Fields as necessary */
$xml =
'
<UpdateListItems xmlns="http://schemas.microsoft.com/sharepoint/soap/">
<listName>'.$guid.'</listName>
<updates>
<Batch>
<Method ID="1" Cmd="New">
<Field Name="Title">My Title</Field>
<Field Name="ABC">My Value</Field>
</Method>
</Batch>
</updates>
</UpdateListItems>
';
 
/* Invoke the Web Service */
$result = $client->call('UpdateListItems', $xml);
 
/* Check for Errors */
if(isset($fault)) {
  echo("<h2>Error</h2>". $fault);
}
 
/* Debugging Info */
echo("<h2>Request</h2>" . htmlspecialchars($client->request, ENT_QUOTES));
echo("<h2>Response</h2>" . htmlspecialchars($client->response, ENT_QUOTES));
echo("<h2>Debug</h2>" . htmlspecialchars($client->debug_str, ENT_QUOTES));
 
unset($client);
?>

That’s pushing data into Sharepoint. You can also pull data using the GetListItems Web Service. The code is basically the same with different XML.