Browsing iTunesU without intalling iTunes

Boris Fritscher — Wed, 13 May 2009 20:51:59 +0000

Updated version of the script

On some pages the location-bar information is available (third-level pages), but 1st and 2nd level pages do not have a location-bar. Even in iTunes it has been removed in favor of a location-bar managed by iTunes. I suppose this is done in order to make use of a cached version of the page and speed up loading. Having no location-bar at 2nd level pages is an awkward design choice from Apple. Therefore location-bar has to be managed by the script and 3th level pages need also to be re-adapted to be displayed correctly.

Fixed version can be tested here: http://1.latest.unil-podcast.appspot.com/index.groovy?handle=[itunesU url]

the new source files are available here:
complete source code at github http://github.com/bfritscher/www-itunesu

index_fixed.groovy – new version with navigation fixed
proxy.groovy – proxy class because iTunes checks user-agent to link to some video files behind a redirect.

The problem

More and more universities publish courses as podcasts on iTunesU and neglect to give a direct access to their content. This hinders sharing of knowledge, because of the obligatory Apple’s iTunes application needed to access iTunesU.

The solution

Emulating the iTunes client: the script can be tested at

http://www2.unil.ch/itunesu/index.groovy?handle=[itunesU url]

It only needs the url which would open in iTunes and converts it to a displayable webpage.

For example for mit.edu: http://deimos.apple.com/WebObjects/Core.woa/Browsev2/mit.edu

http://www2.unil.ch/itunesu/index.groovy?handle=http://deimos.apple.com/WebObjects/Core.woa/Browsev2/mit.edu

Source download

index.groovy version 1 iTunesU xml to html groovy script

Why groovy?

Having experience with Java and Grails development using Groovy especially due to its powerful closures, xml navigation and html building was a obvious choice. Also this project started as a Google AppEngine prototype, but hit the no-user-agent change limitation.

Limitations

Of the three types of layout used in iTunes U only the two column standard university page view and detailed course view (track list) are supported. The iTunes main site listing and main page three column layout is not supported.

Solution explained

iTunes uses a custom xml page to receive and render its content. This file can be access when switching the user-agent to “iTunes”.
Switching user agent is easy under Firefox with the user-agent switcher add-on
Part of the xml file

The xml file is quite verbose and it takes some time to identify the interesting content.
The hard part of this project was to identify the right way to identify and select the wanted data from the xml tree.

Following are code snippets of the source file with their most important or interesting aspects explained.

First task of the script is to retrieve the xml page by settings the right user-agent [3] and then use the powerful groovy xml parser api [13] to have an easily navigable xml object.

def xml
def conn = new URL(handle).openConnection()
conn.setRequestProperty ( "User-Agent", "iTunes/8.1" )
def putBackTogether = new StringBuffer()
def r = new InputStreamReader ( conn.getInputStream(), "UTF-8" )
char [  ]  cb = new char [ 2048 ]
int amtRead = r.read ( cb )
while  ( amtRead  >  0 )   {
	putBackTogether.append ( cb, 0, amtRead )
	amtRead = r.read ( cb )
}
xml = putBackTogether.toString()
xml = new XmlSlurper().parseText(xml)

To work around default namespace bugs when recreating a xml string from an xml object the Markupbuilder has to be set to the apple namespace[6].

import groovy.xml.StreamingMarkupBuilder
def getXml(item){
	def outputBuilder = new StreamingMarkupBuilder()
	outputBuilder.encoding = "UTF-8"
	String result = outputBuilder.bind{
		mkp.declareNamespace("":"http://www.apple.com/itms/")
		mkp.yield item
	}
}

Example of the powerful way to generate html code with the groovy MarkupBuilder.

import groovy.xml.StreamingMarkupBuilder
html.html(xmlns:"http://www.w3.org/1999/xhtml",lang:"en",'xml:lang':"en") {
	head {
		title xml.Path.PathElement.collect{it.'@displayName'}.join(" > ")
		meta('http-equiv':"X-UA-Compatible", content:"IE=8")
		meta('http-equiv':"content-type",content:"application/xhtml+xml; charset=UTF-8")
		//...

Identifying the type of layout the xml file is meant to build is done by looking at a reflection attribute from the first image found in the file.

def topimg = xml.'**'.find{ it.name() == "PictureView"}
def reflect = topimg?.'@reflection'==1 //true = course page

Getting page colors from FontStyle tags

def titleFontStyle = xml.'**'.find{it.name() == "FontStyle" && it.'@name' == "normalTitleFontStyle"}?.'@color'
if(titleFontStyle == null){
	titleFontStyle = xml.'**'.find{it.name() == "FontStyle" && it.'@name' == "outlineTitleFontStyle"}?.'@color'
}

Some more examples of the groovy MarkupBuilder: especially interesting is the mkp.yieldUnescaped [10] method to allow usage of html otherwise automatically converted symbols like &.

body(style:"background: #${xml.'*'.find{ it.'@backColor' != ''}.'@backColor'}"){
	div(class:"container"){
		div(class:"span-24 last nav"){
			def list = xml.Path.PathElement
			def last = list.size()-1
			list.eachWithIndex{ pe,i ->
				if(i == 0){
					a(class:"first",href:"index.groovy?handle=${pe.text().trim()}"){
						span(){
							mkp.yieldUnescaped " "
						}
					}
				}else{
					a(class:"${i == last ? 'last' : ''}",href:"index.groovy?handle=${pe.text().trim()}", pe.'@displayName')
				}
			}
		}
	}
}

Being able to identify the right element is sometimes a bit tricky, especially that not all pages seem to follow the same structure rules.

div(class:"description"){
	xml.ScrollView.View.MatrixView.View?.'**'.find{it?.name()== "VBoxView" && it?.parent()?.name()== "VBoxView" && it?.parent()?.parent()?.name()== "VBoxView" && it?.parent()?.parent()?.parent()?.name()== "VBoxView"}?.'*'.findAll{ it?.name() == "TextView"}.each{ text -> p text }
	mkp.yieldUnescaped " "
}

Usage of the xmlwise library[15] to parse the standard plist part of the xml containing the track list .

import xmlwise.*
div(class:"span-24 last tracklist"){
	table(cellspacing:"1", class:"tablesorter"){
		thead{
			tr{
				th ""
				th "Name"
				th "Time"
				th "Artist"
				th "Release Date"
				th ""
			}
		}
		tbody{
			def plist = Plist.fromXml(getXml(xml.TrackList.plist))
			plist.items.eachWithIndex{ track, i ->
				int seconds = (track["duration"] /1000)
				int minutes = seconds % 3600
				int hours = (seconds - minutes) / 3600
				seconds = minutes % 60
				minutes = (minutes - seconds) / 60

				tr(class:i % 2 == 0 ? "even" : "odd"){
					td track["rank"]
					td track["songName"]
					td ((hours > 0 ? String.format("%02d",hours)  + ":" : "")+ String.format("%02d",minutes) + ":" + String.format("%02d",seconds))
					td track["artistName"]
					td track["releaseDate"][0..9]
					td{
						a(href:track["previewURL"],rel:"lightbox[set 480 380]",title:track["songName"], "view")
						mkp.yieldUnescaped(" ")
						a(href:track["previewURL"],title:track["songName"], "download")
					}
				}
			}
		}
	}
}

Resources used for html design

http://www.blueprintcss.org/
http://jquery.com/ (thankfully, compatible with mootools)
http://www.digitalia.be/software/reflectionjs-for-jquery
http://mootools.net/ (needed for mediabox advanced)
http://iaian7.com/webcode/mediaboxAdvanced
http://tablesorter.com/docs/
http://comparenetworks.com/developers/jqueryplugins/jbreadcrumb.html (only used images)

No related posts.

Slowness with localhost on Vista and seaside/squeak in firefox

Boris Fritscher — Sun, 10 Feb 2008 20:48:59 +0000

While testing seaside when loading the pages locally in firefox I had a long delay. After some digging,

it turns out that the slowness is caused by an IPv6 issue with DNS and can easily be resolved by turning IPv6 support off in Firefox while doing localhost testing. To make the change, type about:config in the address bar, locate the network.dns.disableIPv6 setting and double-click on it to set it to true. This does the trick for the Firefox localhost issue on Vista and everything is running fast again.

as it turns out other people had the same issues but not in relation with seaside, therefore I found it in google only after having discovered the solution by myself.

I am hopping that blogging it under seaside/squeak will help someone save some time.

No related posts.

Think you know JAVA?

Boris Fritscher — Tue, 28 Aug 2007 22:06:43 +0000

Think again!

Advanced Topics in Programming Languages: Java Puzzlers, Episode VI – Google Tech Talks July 23, 2007

Many other interesting talks at: Google Video Tech Talks

No related posts.

amfPHP & PHP 5.2.2 solved!

Boris Fritscher — Sun, 13 May 2007 19:13:35 +0000

After upgrading php to version 5.2.2 amfPHP or cakeamfphp stopped working!
Thankfully i found a quick fix on sourceforge

sourceforge.net

You only need to add this code at the top of gateway.php
if (!isset($HTTP_RAW_POST_DATA)){ $HTTP_RAW_POST_DATA = file_get_contents('php://input'); }

Dashboard (Tableau de bord) avec Flex et PHP

TechnoBabel » Computer Programming

Browsing iTunesU without intalling iTunes

Updated version of the script

The problem

The solution

Source download

Why groovy?

Limitations

Solution explained

Resources used for html design

Slowness with localhost on Vista and seaside/squeak in firefox

Think you know JAVA?

Think again!

amfPHP & PHP 5.2.2 solved!