Archive for the ‘Code’ Category

Batch Extracting MP3s from YouTube Videos

Saturday, January 9th, 2010

Last night I wanted to extract audio tracks from a number of YouTube videos that I’d downloaded using youtube-dl. Being only a so-so shell scripter, I’ve always resorted to ugly for-loops when manipulating multiple files. This invariably ends badly when my loop improperly handles whitespace and mangles the filenames.

No more! Skimming a tutorial last night I stumbled on something that heavy shell users already know: the -exec parameter for the find command. This allows you to specify a command to run on everything that find finds. In the case of extracing audio from MP3s, it works like this:

find . -name '*.flv' -exec ffmpeg -i '{}' '{}.mp3' ';'

This command looks in the current directory for flv files and uses ffmpeg to extract the audio to another file with the same name, plus the .mp3 extension. The funny brackets {} are substituted for the file name.

A downside to this approach – your files end up with names like .flv.mp3 instead of .mp3. If that bothers you, you can fix it with the rename command which uses regexes to rename files:

rename 's/\.flv\.mp3/\.mp3/' *.flv.mp3

Ubuntu users like myself will need to install ffmpeg and ubuntu-restricted-extras to get the necessary encoder.

There are certainly lots of other ways to encode a directory worth of files, but I think this one is pretty cool.

Another interesting python snippet

Thursday, July 2nd, 2009

Well, I think it’s interesting anyway..

So today I was trying to express the idea “do something to all these files, unless the filename matches the list of things we don’t care about”. I have to do a BUNCH of find-and-replace kinda stuff in the next week on a couple thousand webpages, so my plan is to write a little script to make sure I don’t miss anything. Sometimes the script will turn up a false positive that I know I want to ignore..

EDIT – The following snippet doesn’t do quite what I thought it did.. more later..

There are at least three distinct and reasonable ways to do this in Python:

# skip if name matches any of these
ignore = ["thing1", "thing2"]
 
# Iterate over the strings, see if a substring matches
# This is reasonably clear, but it seems long and requires a flag
for filepath in dirwalk(path):
    found = False
    for item in ignore:
        if filepath.find(item) > 0:
            found = True
            break
    if found:
        continue
    # do stuff..
 
# The next two ways build a list of matches
# If there are any matches then skip this file
 
# This is a more functional style
for filepath in dirwalk(path):
    if filter(lambda item: filepath.find(item) != -1, ignore):
        continue
    # do stuff..
 
# Another way to do the same thing
for filepath in dirwalk(path):
    if [item for item in ignore if filepath.find(item) != -1]:
        continue
    # do stuff..

Interestingly, the last two use the same number of characters. I’m not sure which I prefer. While I suspect the final one would be considered the most Pythonic, I do have a soft spot for lambda. Eh, maybe someday when I’m not lazy I will see which is the fastest.. though that doesn’t really matter for my purposes.

Extending Android’s Chronometer to get Elapsed Time

Sunday, June 7th, 2009

In revamping one of my Games on the Android Market, I wanted to add an onscreen timer. The built-in Chronometer class does 95% of what I needed, but getting that extra 5% was annoying enough that I decided to post my solution.

The OOTB Chronometer keeps tracks of time, but it can’t tell you the number of seconds elapsed since it started. This is easily remedied, as I discovered from the fine people on StackOverflow. However, simply getting the elapsed time is not enough because the Chronometer will reset itself everytime the app is Paused or Killed. The two cases are quite different. When the app is Paused, it is still alive and may resume (ex. an incoming call). When the app is Killed, it needs to store the elapsed time somewhere so it can be recovered when the app is Restored (ex. orientation change, or b/c the OS needs memory).

So here is a stripped down solution that seems to do the job:

import android.app.Activity;
import android.content.Context;
import android.os.Bundle;
import android.os.SystemClock;
import android.util.Log;
import android.view.Menu;
import android.widget.Chronometer;
 
public class CustomChronometerActivity extends Activity {
	private static final String TAG = "CustomChronometerActivity";
	private static final String MS_ELAPSED = "com.etc.etc.MsElapsed";
 
	private MyChronometer chrono;
 
	@Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
 
        //start the chronometer
        chrono = new MyChronometer(this);
        chrono.start();
        setContentView(chrono);
    }
 
	@Override
	protected void onPause() {
		Log.i(TAG, "onPause()");
		super.onPause();
		chrono.stop();
	}
 
	@Override
	protected void onResume() {
		Log.i(TAG, "onResume()");
		super.onResume();
		chrono.start();
	}
 
	@Override
	protected void onSaveInstanceState(Bundle outState) {
		super.onSaveInstanceState(outState);
		Log.i(TAG, "onSaveInstanceState()");
		chrono.stop();
		outState.putInt(MS_ELAPSED, chrono.getMsElapsed());
	}
 
	@Override
	protected void onRestoreInstanceState(Bundle savedInstanceState) {
		super.onRestoreInstanceState(savedInstanceState);
		Log.i(TAG, "onRestoreInstanceState()");
		int ms = savedInstanceState.getInt(MS_ELAPSED);
		chrono.setMsElapsed(ms);
		chrono.start();
	}
 
	class MyChronometer extends Chronometer {
 
		public int msElapsed;
		public boolean isRunning = false;
 
		public MyChronometer(Context context) {
			super(context);
		}
 
		public int getMsElapsed() {
			return msElapsed;
		}
 
		public void setMsElapsed(int ms) {
			setBase(getBase() - ms);
			msElapsed  = ms;
		}
 
		@Override
		public void start() {
			super.start();
			setBase(SystemClock.elapsedRealtime() - msElapsed);
			isRunning = true;
		}
 
		@Override
		public void stop() {
			super.stop();
			if(isRunning) {
				msElapsed = (int)(SystemClock.elapsedRealtime() - this.getBase());
			}
			isRunning = false;
		}
	}
}

Certainly there are other ways to do this – either by implementing your own timer using Threads or Handlers, or perhaps by implementing an OnChronometerTickListener and subscribing to events. I rather like this solution, but if you’re the clever sort and see some situation where this doesn’t work or some reason why it might be a bad idea, please let me know.

More Reasons I Love Python..

Tuesday, May 19th, 2009

I was cleaning up some folders the other day at work where the files had been named using one of several naming schemes (or a few with no particular scheme at all). After brief consideration, I decided to do the legwork of renaming all the files with a naming scheme that actually makes sense:

Category_YYYY-MM-DD

That way, the files will stay grouped together if they get copied around to other folders, and they sort alphabetically by date. Then there’s the task for regenerating all the HTML for these baddies. Happily, Python was up to the task:

import os
from datetime import date
files = os.listdir("Path\\To\\File")
files.sort()
files.reverse()
for file in files:
    # chop the prefix, chop the suffix, split into (year, month, date), convert to int
    x = [int(x) for x in file.split("_")[-1][:-4].split('-')]
    print "<li><a href=\"/path/to/%s\">%s</a></li>" % (file, date(x[0], x[1], x[2]).strftime('%B %d, %Y'))

Well, it’s nothing like the real pros can do. But you gotta love a few links of code that save your fingers from a repetitive and typo-prone task like manually editing hundreds of links.

Getting your stats from the Android Marketplace with PHP/cURL

Friday, May 15th, 2009

A few weeks ago I mentioned one way to get your developer stats off the Android Developer Console automatically in the post “Fetching Android Market Stats with Python, MozRepl, and BeautifulSoup“. Unfortunately, despite being very awesome, Firefox + MozRepl is not super-great for this task. When a plugin is updated, Firefox hangs on startup. That’s fine, but it kinda sucks for scripting. I’m sure there’s a way around it, but that difficulty makes a good excuse for coming back to solve this problem the right way.

Following is a PHP script that uses cURL to login to the Developer Console and grab the market stats. Unfortunately, Google’s app is written in GWT and the its Javascript is completely obfuscated. The market stats are fetched as JSON data and then somehow parsed, but I haven’t been able to figure out how exactly. If you run this script (or just look using Firebug), you’ll see that the JSON is a gigantic array. While the data of interest are clearly present in this array (total downloads, current installed base, rating, etc..), I haven’t been able to figure out how to parse it reliably. If you’ve tried this and figured it out, I’d love to know!

This script was assembled from a bunch of random PHP/cURL tutorials and may contain redundancy, unnecessary cURL settings, etc. Python fans, see the comments of my other post on this topic where a kind soul has demonstrated the same thing in Python using mechanize.

<?php
//setup a temp file to store cookies
$ckfile = tempnam ("/tmp", "CURLCOOKIE");
 
//do google authorization
$data = array('accountType' => 'GOOGLE',
          'Email' => 'YOUR_ACCOUNT_EMAIL_HERE',
          'Passwd' => 'YOUR_ACCOUNT_PW_HERE',
          'source' => '',
          'service' => 'androiddeveloper');  
 
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.google.com/accounts/ClientLogin");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
 
//grab the AUTH token for later
$auth = '';
if($info['http_code'] == 200) {
    preg_match('/Auth=(.*)/', $output, $matches);
    if(isset($matches[1])) {
        $auth = $matches[1];
    }
}
 
//login to Android Market
//this results in a 302
//I think this is necessary for a cookie to be set
$ch = curl_init ("http://market.android.com/publish?auth=$auth");
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//go to the Developer Console
$ch = curl_init ("http://market.android.com/publish/Home");
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//grab the JSON data
//perm and postdata seem to have changed in the last 6 months
//if the script isn't working, try using firebug to inspect the Request when
//http://market.android.com/publish/editapp gets fetched
$perm = "81E29277804F7729E9B743A43B2EFD07";
$headers = array(
    "Content-Type: text/x-gwt-rpc; charset=utf-8",
    "X-GWT-Permutation: $perm",
    "Referer: http://market.android.com/publish/gwt/$perm.cache.html");
//not sure what x-gwt-permutation means, I think it may have to do with which version of GWT they serve based on your browser
$postdata = "5|0|4|http://market.android.com/publish/gwt/|09C42EAE15B55219550B2D800FAC1644|com.google.wireless.android.vending.developer.shared.AppEditorService|getFullAssetInfosForUser|1|2|3|4|0|";
$ch = curl_init ("http://market.android.com/publish/editapp");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
 
//now what?!?
echo('<pre>');
$output = json_decode(substr($output, 4));
print_r($output);

If you run this script and are willing to send me your stats, that would be super-helpful. Maybe I’ll be able to get enough data to figure out why some apps have more fields than others. With only 3 apps currently on the market, I don’t have much to go on. Feel free to obscure your data, but please make the changes obvious and note whether the app is free/paid and what part of the market it appears on (games/apps and sub-category). Here is a link to my best guesses so far in an Excel worksheet: market-json

Geocoding in Sharepoint Lists

Tuesday, April 14th, 2009

Today I was thinking it would be nice to do some Geocoding in Sharepoint. Specifically, I wanted to make it so list items could have longitude and latitude fields that could be populated with a button click from the EditItem page. Geocoding is pretty easy to do with both Google and Yahoo. For my situation, Yahoo seemed more appropriate.

My first intuition was that this should be easy to do with Javascript. Just call the Yahoo Maps API with the right data and parse the response. Except.. this usage violates the same site origin policy for Javascript. Drat. Well, there are several things that can be done. If you’re handy with C# and M$ technologies, you can just create a proxy on the same server. Unfortunately, I don’t know the first thing about the Microsoft stack and I’m too lazy to learn. As an alternative, I opted to create a PHP proxy on another server and force it to return JSON, which skirts around the same site origin problem.

First, the code for the PHP proxy which lives somewhere besides the Sharepoint server. This proxy forwards requests to Yahoo, parses the response, and emits JSON back to the caller.

<?php
//get params from request
$appid = 'YOUR_YAHOO_APPID';
$street = $_GET['street'];
$city = $_GET['city'];
$state = $_GET['state'];
 
//build new request
$req = 'http://local.yahooapis.com/MapsService/V1/geocode?';
$req .= 'appid=' . $appid;
$req .= '&amp;street=' . urlencode($street);
$req .= '&amp;city=' . urlencode($city);
$req .= '&amp;state=' . urlencode($state);
 
//fetch XML using cURL
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $req);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = trim(curl_exec($ch));
curl_close($ch);
 
//parse XML
$xml = simplexml_load_string($result);
$lat = $xml-&gt;Result[0]-&gt;Latitude;
$lng = $xml-&gt;Result[0]-&gt;Longitude;
 
//return JSON
echo $_GET['jsoncallback'] . "({lat: \"$lat\", lng: \"$lng\"})";
?>

Now, the Javascript part of this uses jQuery to make life much much easier. It basically injects a new Button onto the page. When clicked, it builds a request from the location data on the page, sends the request to the proxy, parses the response, and puts the latitude and longitude into the form fields. Then the user can click ‘Save’ as usual. This code needs to be added to the EditItem.aspx page for that particular list. You also need to have jQuery on the page. In my case, I just included them both as external scripts to keep my changes to .aspx pages to a minimum.

$(document).ready(function() {
	//create a new Button, match Sharepoint styles
	var geoButton = $('&lt;input type="button"&gt;').attr({'class':'ms-ButtonHeightWidth', 'value':'GeoCode'});
 
	//add our Button after the default 'Cancel' Button
	$('.ms-formtoolbar .ms-toolbar:last').after($('&lt;td&gt;&amp;nbsp;&lt;/td&gt;').attr({'class':'ms-separator'}));
	$('.ms-formtoolbar .ms-separator:last').after(geoButton);
 
	//wrap Button in a Table to match Sharepoint's style
	geoButton.wrap($('&lt;td&gt;&lt;/td&gt;').attr({'class': 'ms-toolbar', 'no-wrap':'true'}))
		 .wrap($('&lt;table&gt;&lt;/table&gt;').attr({'cellspacing':'0','cellpadding':'0','width':'100%'}))
		 .wrap($('&lt;tbody&gt;&lt;/tbody&gt;'))
	         .wrap($('&lt;tr&gt;&lt;/tr&gt;'))
                 .wrap($('&lt;td&gt;&lt;/td&gt;').attr({'nowrap':'','align':'right','width':'100%'}));
 
	//onClick, perform geoCode and put Long/Lat into form fields
	geoButton.click(function() {
 
	//get data from form fields
	var street = $('input[title=Street]').val();
	var city =  $('input[title=City]').val();
	var state =  $('input[title=State]').val();
 
	//fail early if some data is absent, since we wouldn't get a good geocode
	if(street == '' || city == '' || state == '')
  	    return;
 
	//wrap data into a URL so we can do an HTTP GET
	var address = '&amp;street='+street+'&amp;city='+city+'&amp;state='+state;
 
	//use jQuery.getJSON to avoid that pesky cross domain security restriction
	//?jsoncallback=? is a peculiarity required by jQuery, the server must echo this back
	$.getJSON("http://path_to/yahoo-geocoder.php?jsoncallback=?"+address,
		function(json) {
    		    //async callback, unpack the data
 		    var lat = json.lat;
		    var lng = json.lng;
       		    //simple error detection
		    if(lat == '' || lng == '') {
			//show '!!!' after form fields to indicate that geocoding failed
			$('input[title=Latitude]')
			.after($('&lt;span&gt;!!!&lt;/span&gt;').attr({'id':'error-lat'}).css('color','red'));
			$('input[title=Longitude]')
			.after($('&lt;span&gt;!!!&lt;/span&gt;').attr({'id':'error-lng'}).css('color','red'));
		    } else {
			//success, place results into form fields
			$('input[title=Latitude]').val(lat);
			$('input[title=Longitude]').val(lng);
			//remove any previous error indicators
			$('#error-lat').remove();
			$('#error-lng').remove();
		    }
		});
	});
});

Fetching Android Market Stats with Python, MozRepl, and BeautifulSoup

Thursday, April 2nd, 2009

A few weeks ago I was quite keen on the idea of gathering stats and creating charts to track the popularity of my Android apps. Alas, despite digging around in various packages and experimenting with cURL, I could never seem to get logged in programmatically to the Android Marketplace Developer Console. So I gave up to continue working on my next app. Now I’ve come up with another reason to do some screen-scraping, so I thought I should give this another try.

Half the magic here belongs to a very cool Firefox plugin called MozRepl which lets you open a telnet connection to Firefox and interact with it via Javascript. Awesome, no?

All you have to do is ask MozRepl to go to the Developer Console, download the HTML, and run it through BeautifulSoup (the rest of the magic) to extract the data.

It turns out to be just slightly trickier because MozRepl needs to talk to Python via Telnet. I suppose this script could be setup in cron to grabs stats a couple of times each day. I think I’m just gonna run it manually every once in awhile.

import BeautifulSoup, re, time
import os, telnetlib
# Install MozRepl Plugin
# http://wiki.github.com/bard/mozrepl
# Setup MozRepl to start automatically with FF, check that port number is 4242
# Login to Developer Console once manually so login credentials get saved
 
# Create a new profile and set this accordingly
# http://support.mozilla.com/en-US/kb/Managing+profiles
profile = 'my_firefox_profile'
 
# go to Developer Console using new profile
url = 'http://market.android.com/publish/Home'
os.system("firefox -no-remote -P %s %s &" % (profile, url))
time.sleep(5) #wait a sec for FF to start
 
#connect to MozRepl and fetch HTML
t = telnetlib.Telnet("localhost", 4242)
t.read_until("repl>")
t.write("content.document.body.innerHTML")
body = t.read_until("repl>")
t.close()
 
#is there a better way to do this?
os.system("killall -9 firefox")
 
#yank stats out of HTML
now = time.strftime("%Y-%m-%d %H:%M:%S")
soup = BeautifulSoup.BeautifulSoup(body)
table = soup.find("div", { "class" : "listingTable" })
for row in table.findAll('div', {'class':'listingRow'}):
  app = row.find("div", { "class" : "listingApp" })
  rating = row.find("div", { "class" : "listingRating" })
  stats = row.find("div", { "class" : "listingStats" })
  if app and rating and stats:
    name = app.next.next.string
    total = stats.next.string.split()[0]
    active = stats.next.nextSibling.string.split()[0]
    nratings = rating.next.string[1:-1]
    stars = len(rating.findAll(attrs={'style':re.compile("scroll -78px")}))
    print now, name, total, "total", active, "active", nratings, "ratings", stars, "stars"
#that's it, now maybe save these to a CSV or a log file..

I debated whether to show my actual numbers. Here you go, enjoy:

2009-04-03 17:45:15 Measure Stuff 4 total 1 active 2 ratings 1 stars
2009-04-03 17:45:15 Measure Stuff Lite 3006 total 995 active 28 ratings 2 stars
2009-04-03 17:45:15 RGB Probe 4 total 2 active 2 ratings 1 stars
2009-04-03 17:45:15 Thumb Maze 112 total 39 active 8 ratings 3 stars
2009-04-03 17:45:15 Thumb Maze Lite 16313 total 8813 active 172 ratings 3 stars

Uh oh, those numbers are not very good at all! So far my plan to live off Android looks doomed, but maybe things will pick up in the future. Two of the apps appear twice because there is a paid version and a free one. Can you tell which is which? =). Also, I think there is something wrong with RGB Probe. I’ve gotten a couple of e-mails saying the download failed.

So I hope folks will find this script useful. Obviously, use of this code is completely at your own risk. Screen scrapers are an arguably questionable enterprise, so don’t blame me if you hose your Firefox profile or Google gets mad at you.

Also, if anyone knows the cURL incantation that will do the same thing sans Firefox, I’d love to hear it. I kept getting a 302 response and never quite figured it out. I’ve taken several suggestions based on other Google services that ’should work’, but for some reason don’t.

There are certainly pros and cons to screen scraping through the browser; I’ll only point out two advantages: First, you get ‘real’ Javascript executed right in Firefox. With many of the big data sites being Ajax-heavy, simply fetching the HTML without executing the JS only gets you halfway there. Second, it is possible to detect and block screen scrapers by looking for unusual or suspicous request patterns. I don’t know if any sites actually do this, but it could be done. For example, a simple fetch via wget looks different to a server than a fetch with Firefox and it goes beyond User-Agents. The css, images, javascript, and such will also be fetched in a particular way and a server can look for anything unusual in the order or timing with which resources are requested. Sound crazy? You’re right! It probably is and I’m not sure anybody actually does this. In fact, it very possibly wouldn’t work well at all in practice. For one, it could screw up text-only browsers. But I think it is still within the realm of possibility..

Now for balance, two downsides: First, the browser needs a window to run in. This means it is kinda slow, hijacks your computer for a few seconds, and doesn’t really lend itself to parallelization. Second, tools like cURL and wget and many language-specific libraries are practically standard.

Restyling Android ImageButtons

Thursday, March 26th, 2009

Discovered an interesting thing about Android today: the fairly ugly default ImageButton can be replaced with a custom drawable while retaining the very user-friendly highlighting behavior.

I was trying to create three buttons for an application I am working on: zoom-in, zoom-out, and next. For development purposes, I had just been using regular (text-only) Buttons, but for the final product I wanted something a little snazzier. While it is certainly possible to create your own button functionality, I’m guessing it is wise to stick to the defaults when possible – enter the ImageButton class. It works the same as a Button, but shows an image instead of text. The problem is, the default look was gonna be kinda ugly in my application.

My first thought was – well, maybe I can just set the background to black. That works, but you lose the highlighting that makes the ImageButton nice in the first place. So, a few searches later, here is what you do – this example is for a two-state (“normal” and “pressed”) button called “Next”:

1. Create two image files (one for the “normal” state and one for the “highlighted” state

2. Create an XML Drawable (that is, a .xml file in your /resources/drawable/ directory

3. Paste the following XML:

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
<item android:state_window_focused="false" android:state_enabled="true"
android:drawable="@drawable/btn_next_normal" />
<item android:state_window_focused="false" android:state_enabled="false"
android:drawable="@drawable/btn_next_normal" />
<item android:state_pressed="true"
android:drawable="@drawable/btn_next_pressed" />
<item android:state_focused="true" android:state_enabled="true"
android:drawable="@drawable/btn_next_pressed" />
<item android:state_enabled="true"
android:drawable="@drawable/btn_next_normal" />
<item android:state_focused="true"
android:drawable="@drawable/btn_next_normal" />
<item android:drawable="@drawable/btn_next_normal" />
</selector>

4. In your Java code (should also work in XML), use it just like any other Drawable:

ImageButton next = new ImageButton(this);
  next.setImageResource(R.drawable.btn_next);
  next.setScaleType(ScaleType.CENTER_INSIDE);
  next.setOnClickListener(new View.OnClickListener() {
  @Override
  public void onClick(View v) {
    //do stuff here
  }
});

Note that this button only shows two images “normal” and “pressed” because that seems sufficient for my situation, but the XML allows 7 views depending on the state of the button.

For more information, see:

http://stackoverflow.com/questions/606694/android-different-image-for-rollover-on-imagebutton

http://groups.google.com/group/android-developers/msg/ef6014f46c35a824

Bridging Android’s ListPreference and Database

Sunday, March 1st, 2009

Long time, no post.. I’ve been waay too busy programming Android for the last few weeks to get anything useful done. As mentioned a few days ago, I’m getting deluged by weird comment spam, so comments are turned off for the moment – sorry.

Today I was trying to mix Android’s two storage mechanisms. The application needs to store a small number of settings which a user can create/update/delete -  a database. Additionally, one of these settings is considered to be “selected” – probably SharedPreferences, though one could argue for the database here too. My first intuition was to specialize a subclass of ListPreference to override the getEntries() and getEntryValues() methods – nope. The next idea was to call setEntries(CharSequence[]) and setEntryValues(CharSequence[]) in the constuctor – that seems to work fine. However, since I was reading from a database, I wanted the Cursor to be managed by an activity. Since I was subclassing ListPreference, that wasn’t gonna work. Ultimately, I settled on subclassing PreferenceActivity and building the menus in code instead of XML. Adapting the sample database code from Notepadv3, here is a custom Preferences menu:

public class EditPreferences extends PreferenceActivity {
 
public static final String SELECTED_TARGET_KEY = "SelectedTargetKey";
public static final String NO_SELECTION  = "0";
 
private TargetDbAdapter mDbHelper;
 
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setPreferenceScreen(createPreferenceHierarchy());
}
 
private PreferenceScreen createPreferenceHierarchy() {
PreferenceScreen root = getPreferenceManager().createPreferenceScreen(this);
PreferenceCategory dialogBasedPrefCat = new PreferenceCategory(this);
dialogBasedPrefCat.setTitle(R.string.pref_cat_title);
root.addPreference(dialogBasedPrefCat);
 
//builds list from DB
mDbHelper = new TargetDbAdapter(this);
mDbHelper.open();
Cursor c = mDbHelper.fetchAllNotes();
startManagingCursor(c);
 
int count = c.getCount();
CharSequence[] entries = new CharSequence[count];
CharSequence[] entryValues = new CharSequence[count];
 
c.moveToFirst();
for(int i=0; i&lt;count; i++) {
entries[i] = c.getString(c.getColumnIndexOrThrow(TargetDbAdapter.KEY_TITLE));
entryValues[i] = c.getString(c.getColumnIndexOrThrow(TargetDbAdapter.KEY_ROWID));
c.moveToNext();
}
 
ListPreference targets = new ListPreference(this);
targets.setEntries(entries);
targets.setEntryValues(entryValues);
targets.setDefaultValue(NO_SELECTION);
targets.setDialogTitle(R.string.pref_dialog_title);
targets.setKey(SELECTED_TARGET_KEY);
targets.setTitle(R.string.pref_title);
targets.setSummary(R.string.pref_summary);
dialogBasedPrefCat.addPreference(targets);
 
//add other preference screens
 
return root;
}
 
}

SPARQLing The Highest Point in Every US State

Wednesday, February 4th, 2009

In a previous post, I mentioned that it should be pretty easy to use SPARQL to make a map of the highest point in each of the 50 US States. Having written that, I thought I should maybe actually, you know, try it.

The following chunk of code uses ARC2, an rdf/semantic web library for PHP to query the dbpedia endpoint and then put the results on a Google Map.

To try this out, you need to:

  1. Have a functional PHP installation
  2. Download ARC2 into your web path (no setup required)
  3. Set the path to ARC in the code below
  4. Get a Google Maps API Key (free)
  5. Set your API key in the code below
  6. Run

Note that this is a demo and is written to be easy to run – a real application might separate the data logic from the webpage and make more sophisticated use of Javascript/Google Maps API.

<?php
//include ARC2 libraries
include_once("path/to/ARC2.php");
//instantiate a RemoteStore
$config = array('remote_store_endpoint' => 'http://dbpedia.org/sparql');
$store = ARC2::getRemoteStore($config);
//build the SPARQL query
$q = '
PREFIX dbpedia2: <http://dbpedia.org/property/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>
SELECT ?state ?mtn ?lat ?long
WHERE {
?state skos:subject <http://dbpedia.org/resource/Category:States_of_the_United_States> .
?state dbpedia2:highestpoint ?mtn .
?mtn geo:lat ?lat .
?mtn geo:long ?long
}
';
//process the results
$results = array();
if ($rows = $store->query($q, 'rows')) {
foreach ($rows as $row) {
$state = substr($row['state'], strlen("http://dbpedia.org/resource/"));
$mtn = substr($row['mtn'], strlen("http://dbpedia.org/resource/"));
$lat = $row['lat'];
$lng = $row['long'];
$results[] = array($state, $mtn, $lat, $lng);
}
}
?>
<!DOCTYPE html "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<title>Google Maps JavaScript API Example</title>
<script type="text/javascript" src="http://maps.google.com/maps?file=api&amp;amp;v=2&amp;amp;key=YOUR_KEY"></script>
<script type="text/javascript">
function initialize() {
if (GBrowserIsCompatible()) {
var map = new GMap2(document.getElementById("map_canvas"));
map.setCenter(new GLatLng(37.4419, -122.1419), 3);
map.addControl(new GMapTypeControl());
map.addControl(new GLargeMapControl());
<?php
//populate map with results
foreach($results as $result) {
list($state, $mtn, $lat, $lng) = $result;
echo("map.addOverlay(new GMarker(new GLatLng($lat,$lng), {title: '$mtn'}));n");
}
?>
}
}
</script>
</head>
<body onload="initialize()" onunload="GUnload()">
<div id="map_canvas" style="width: 100%; height: 100%"></div>
</body>
</html>

* Using substr() to chop off “http://dbpedia.org/resource/” from the names is probably cheating. I think you’re supposed to use rdfs:label@en instead.