Browsing iTunesU without intalling iTunes

Updated version of the script

On some pages the location-bar information is available (third-level pages), but 1st and 2nd level pages do not have a location-bar. Even in iTunes it has been removed in favor of a location-bar managed by iTunes. I suppose this is done in order to make use of a cached version of the page and speed up loading. Having no location-bar at 2nd level pages is an awkward design choice from Apple. Therefore location-bar has to be managed by the script and 3th level pages need also to be re-adapted to be displayed correctly.

Fixed version can be tested here: http://1.latest.unil-podcast.appspot.com/index.groovy?handle=[itunesU url]

the new source files are available here:
complete source code at github http://github.com/bfritscher/www-itunesu

index_fixed.groovy – new version with navigation fixed
proxy.groovy – proxy class because iTunes checks user-agent to link to some video files behind a redirect.

The problem

More and more universities publish courses as podcasts on iTunesU and neglect to give a direct access to their content. This hinders sharing of knowledge, because of the obligatory Apple’s iTunes application needed to access iTunesU.

The solution

Emulating the iTunes client: the script can be tested at

http://www2.unil.ch/itunesu/index.groovy?handle=[itunesU url]

It only needs the url which would open in iTunes and converts it to a displayable webpage.

For example for mit.edu: http://deimos.apple.com/WebObjects/Core.woa/Browsev2/mit.edu

http://www2.unil.ch/itunesu/index.groovy?handle=http://deimos.apple.com/WebObjects/Core.woa/Browsev2/mit.edu

Source download

index.groovy version 1 iTunesU xml to html groovy script

Why groovy?

Having experience with Java and Grails development using Groovy especially due to its powerful closures, xml navigation and html building was a obvious choice. Also this project started as a Google AppEngine prototype, but hit the no-user-agent change limitation.

Limitations

Of the three types of layout used in iTunes U only the two column standard university page view and detailed course view (track list) are supported. The iTunes main site listing and main page three column layout is not supported.

Solution explained

iTunes uses a custom xml page to receive and render its content. This file can be access when switching the user-agent to “iTunes”.
Switching user agent is easy under Firefox with the user-agent switcher add-on
Part of the xml file

<FontStyleSet setName="basic22" normalStyle="helvetica22"
	linkStyle="link" linkPressedStyle="linkPressed" linkRolloverStyle="linkRollover" />
<FontStyleSet setName="normal40" normalStyle="lucida40"
	linkStyle="link" linkPressedStyle="linkPressed" linkRolloverStyle="linkRollover" />
<FontStyleSet setName="normal40Bold" normalStyle="lucida40Bold"
	linkStyle="link" linkPressedStyle="linkPressed" linkRolloverStyle="linkRollover" />
<MatrixView rightInset="0" bottomInset="0" leftInset="0"
	topInset="0" rowFormat="100%,*" viewName="Form">
	<View rightInset="0" bottomInset="0" leftInset="0" topInset="0"
		minWidth="948" minHeight="400">
		<VBoxView>
			<!-- Layer 1 = Banner Image -->
			<!-- BEGIN ReplaceableImageXMLView -->
			<View alt="" height="400" width="600">
				<PictureView proportional="1" verticalAlignment="top"
					shouldBeVisible="true" shadowHeight="0"
					alt="MIT offers a selection of video and audio from several of our groundbreaking projects: MIT OpenCourseWare, MIT World, MIT TechTV, and Visualizing Cultures."
					shadowWidth="0" height="400" width="600"
					url="http://deimos3.apple.com/indigo//2c/c5/f3/93/2cc5f393018ecb9b1a3991063e10376b093cd1918f43b3847af32a8a9ba35103-1502158938.jpg"
					addShadowSizes="false" />
			</View>
			<!-- END ReplaceableImageXMLView -->
		</VBoxView>
		<VBoxView>
			<!-- Layer 2 = link box area -->
			<MatrixView leftInset="0" rightInset="0" topInset="25"
				columnFormat="65%,25,35%">
				<!--
					This ensures we leave enough room for the banner image in the 3
					column view
				-->
				<View minHeight="400" />
				<View />
				<VBoxView rightInset="25" bottomInset="0" leftInset="0"
					topInset="0">
					<VBoxView>
						<VBoxView leftInset="0" rightInset="0" topInset="0"
							bottomInset="0">
							<!-- BEGIN description box-->
							<!-- END description box-->
							<!-- START ListBoxStack -->
							<View>
								<FontStyle name="outlineTitleFontStyle" color="ffffff" />
								<FontStyle name="outlineTextFontStyle" color="ffffff" />
								<!-- BEGIN RoundedBox -->
								<View rightInset="0" bottomInset="0" leftInset="0"
									topInset="0">
									<Test value="7.0.0" comparison="greater or equal"
										property="iTunes version">
										<!-- BEGIN MaskedView -->
										<View rightInset="0" bottomInset="0" leftInset="0"
											topInset="0">
											<PictureButtonView rightInset="0" topInset="0"
												bottomInset="0" leftInset="0" alt="" color="rgba(0,0,0,0.75)"
												mask="http://deimos3.apple.com/rsrc/Images/masks/rounded_box.png"
												cap="4" />

										</View>
									</Test>
								</View>
							</View>
						</VBoxView>
					</VBoxView>
				</VBoxView>
			</MatrixView>
		</VBoxView>
	</View>
</MatrixView>

The xml file is quite verbose and it takes some time to identify the interesting content.
The hard part of this project was to identify the right way to identify and select the wanted data from the xml tree.

Following are code snippets of the source file with their most important or interesting aspects explained.

First task of the script is to retrieve the xml page by settings the right user-agent [3] and then use the powerful groovy xml parser api [13] to have an easily navigable xml object.

def xml
def conn = new URL(handle).openConnection()
conn.setRequestProperty ( "User-Agent", "iTunes/8.1" )
def putBackTogether = new StringBuffer()
def r = new InputStreamReader ( conn.getInputStream(), "UTF-8" )
char [  ]  cb = new char [ 2048 ]
int amtRead = r.read ( cb )
while  ( amtRead  >  0 )   {
	putBackTogether.append ( cb, 0, amtRead )
	amtRead = r.read ( cb )
}
xml = putBackTogether.toString()
xml = new XmlSlurper().parseText(xml)

To work around default namespace bugs when recreating a xml string from an xml object the Markupbuilder has to be set to the apple namespace[6].

import groovy.xml.StreamingMarkupBuilder
def getXml(item){
	def outputBuilder = new StreamingMarkupBuilder()
	outputBuilder.encoding = "UTF-8"
	String result = outputBuilder.bind{
		mkp.declareNamespace("":"http://www.apple.com/itms/")
		mkp.yield item
	}
}

Example of the powerful way to generate html code with the groovy MarkupBuilder.

import groovy.xml.StreamingMarkupBuilder
html.html(xmlns:"http://www.w3.org/1999/xhtml",lang:"en",'xml:lang':"en") {
	head {
		title xml.Path.PathElement.collect{it.'@displayName'}.join(" > ")
		meta('http-equiv':"X-UA-Compatible", content:"IE=8")
		meta('http-equiv':"content-type",content:"application/xhtml+xml; charset=UTF-8")
		//...

Identifying the type of layout the xml file is meant to build is done by looking at a reflection attribute from the first image found in the file.

def topimg = xml.'**'.find{ it.name() == "PictureView"}
def reflect = topimg?.'@reflection'==1 //true = course page

Getting page colors from FontStyle tags

def titleFontStyle = xml.'**'.find{it.name() == "FontStyle" && it.'@name' == "normalTitleFontStyle"}?.'@color'
if(titleFontStyle == null){
	titleFontStyle = xml.'**'.find{it.name() == "FontStyle" && it.'@name' == "outlineTitleFontStyle"}?.'@color'
}

Some more examples of the groovy MarkupBuilder: especially interesting is the mkp.yieldUnescaped [10] method to allow usage of html otherwise automatically converted symbols like &.

body(style:"background: #${xml.'*'.find{ it.'@backColor' != ''}.'@backColor'}"){
	div(class:"container"){
		div(class:"span-24 last nav"){
			def list = xml.Path.PathElement
			def last = list.size()-1
			list.eachWithIndex{ pe,i ->
				if(i == 0){
					a(class:"first",href:"index.groovy?handle=${pe.text().trim()}"){
						span(){
							mkp.yieldUnescaped "&nbsp;"
						}
					}
				}else{
					a(class:"${i == last ? 'last' : ''}",href:"index.groovy?handle=${pe.text().trim()}", pe.'@displayName')
				}
			}
		}
	}
}

Being able to identify the right element is sometimes a bit tricky, especially that not all pages seem to follow the same structure rules.

div(class:"description"){
	xml.ScrollView.View.MatrixView.View?.'**'.find{it?.name()== "VBoxView" && it?.parent()?.name()== "VBoxView" && it?.parent()?.parent()?.name()== "VBoxView" && it?.parent()?.parent()?.parent()?.name()== "VBoxView"}?.'*'.findAll{ it?.name() == "TextView"}.each{ text -> p text }
	mkp.yieldUnescaped "&nbsp;"
}

Usage of the xmlwise library[15] to parse the standard plist part of the xml containing the track list .

import xmlwise.*
div(class:"span-24 last tracklist"){
	table(cellspacing:"1", class:"tablesorter"){
		thead{
			tr{
				th ""
				th "Name"
				th "Time"
				th "Artist"
				th "Release Date"
				th ""
			}
		}
		tbody{
			def plist = Plist.fromXml(getXml(xml.TrackList.plist))
			plist.items.eachWithIndex{ track, i ->
				int seconds = (track["duration"] /1000)
				int minutes = seconds % 3600
				int hours = (seconds - minutes) / 3600
				seconds = minutes % 60
				minutes = (minutes - seconds) / 60
				
				tr(class:i % 2 == 0 ? "even" : "odd"){
					td track["rank"]
					td track["songName"]
					td ((hours > 0 ? String.format("%02d",hours)  + ":" : "")+ String.format("%02d",minutes) + ":" + String.format("%02d",seconds))
					td track["artistName"]
					td track["releaseDate"][0..9]
					td{
						a(href:track["previewURL"],rel:"lightbox[set 480 380]",title:track["songName"], "view")
						mkp.yieldUnescaped("&nbsp;")
						a(href:track["previewURL"],title:track["songName"], "download")
					}
				}
			}
		}
	}
}

Resources used for html design

23 thoughts on “Browsing iTunesU without intalling iTunes

  1. Thanks a lot, this is what I am exactly looking for. I wanted to watch some videos on my Linux from iTMS and there is no iTunes here to handle it. No apps worked either. This script was the rescue :-)

    Any help on running this script locally would be great, I am getting an exception like this:
    Caught: groovy.lang.MissingPropertyException: No such property: html
    on line
    html.html(xmlns:”http://www.w3.org/1999/xhtml”,lang:”en”,’xml:lang’:”en”) {

  2. Would it be possible to expose a “download all” link? I realize that could cumbersome for the browser, but with an extension like DownThemAll, that would make for a really nice touch.

    This app works great on Android 2.0 by the way! Thanks for your hard work!

  3. Pingback: iTunes in a Browser «

  4. I just wanted to thank you for your work! This works just as advertised and spares a lot of headache to those of us who really do not want to install Apple’s bloated media player. Thank you!

  5. This is a very useful web-app, but it seems the iTunesU system has changed, and now the location-bar on top doesn’t show anything but the home location. From that example link above you can go to MIT > Architecture > Global History of Architecture, but it won’t have links to MIT, Architecture, only a link to home. Same problem happens with TunesViewer.

  6. Thanks for the information. I took a quick look at the new xml file, but it seems that the location-bar information is no more inside the main file. I also noticed that some pages are not rendering correctly, but this will take more time to analyze.

  7. I’ve looked into this problem some more, and it seems the path is never sent unless the user-agent is iTunes/9.0 or later, and even then, it is not in the usual section of the xml. Instead, the location-bar info is in tags right after the tags.

  8. Sorry, the tags got erased, that should have said: it is not in the usual path pathelement section of the xml. Instead, the location-bar info is in gotourl tags right after the FontStyleSet tags.

  9. I had to install iTunes :( to check the layouts. As you mentioned on some pages the location-bar information is available (third-level pages), but 1st and 2nd level pages do not have a location-bar. Even in iTunes it has been removed in favor of a location-bar managed by iTunes. I suppose this is done in order to make use of a cached version of the page and speed up loading. Having no location-bar at 2nd level pages is an awkward design choice from Apple. Therefore location-bar has to be managed by the script and 3th level pages need also to be readapted to be displayed correctly…

  10. The test version seems to work well on third lavel pages – navigation bar magically appears – and works – at least on my university site!

    Could you publish the fixed script? Thanks a lot for your clever work

  11. Hi Boris, thanks a lot. Your work is very useful – I’m trying to make our small iTunesU site available also to Linux users and your script is a godsend. But…

    I am puzzled – I installed groovy (1.7.3) on my machine (a mac) which has a Tomcat server running. I also installed you “index_fixed” and “proxy” as groovlets following the instructions given by http://groovy.codehaus.org/Groovlets.
    Everything seems to work, apart of he graphical rendering. If I use your site
    http://1.latest.unil-podcast.appspot.com/index.groovy?handle=https://deimos.apple.com/WebObjects/Core.woa/Browse/unitn.it
    everything works well – while if I use your script on my site
    http://latemar.science.unitn.it:8080/groovy/index.groovy?handle=https://deimos.apple.com/WebObjects/Core.woa/Browse/unitn.it
    the graphical rendering is wrong – as if some css was missing…
    Any hint?

  12. Pingback: iTunesU without iTunes « My Mac Notes

  13. THANK YOU, THANK YOU!

    I loathe how Apple has tricked so many universities who are trying to give to the community into giving to Apple instead. And I loathe how iTunes insists on running in the background at all times, draining CPU and memory, regardless of whether I’m using it or not. There’s no good reason for it to do that.

  14. Um… I spoke too soon. How am I supposed to use this? How am I supposed to open it? I tried saving it as a .js file, and as a .htm file, but firefox wouldn’t open it either way. How am I supposed to give it the URL that it needs?

  15. Hi,
    the script is provided as-is. full source is available at http://github.com/bfritscher/www-itunesu

    for just using the script without installing it yourself you can test
    http://1.latest.unil-podcast.appspot.com/index.groovy?handle= and append the itunesU url page you wish to see. The link for your university, is certainly published somewhere on your universitie’s page. Otherwise you will have to guess it.

    the format is http://deimos.apple.com/WebObjects/Core.woa/Browse/%5Buniversitywebpage%5D

    for example:
    http://1.latest.unil-podcast.appspot.com/index.groovy?handle=http://deimos.apple.com/WebObjects/Core.woa/Browse/unil.ch

    http://1.latest.unil-podcast.appspot.com/index.groovy?handle=http://deimos.apple.com/WebObjects/Core.woa/Browse/mit.edu

    I hope this helps

  16. Hi it seems there is a new kind of page in iTunesU, I finally check it out on a machine, which had iTunes installed. The pages are different in the sens that they look more like a real webpage and have the playlist directly embedded in the page. I will have to check out the xml behind, to see if i can adapat it to my script. Unfortunately, I do not have time for this right now.

Leave a Reply

Your email address will not be published. Required fields are marked *