Building a Website on the DigitalNZ API

Introduction

After an eighteen-month spell at Weta Digital I am back at the National Library, or more precisely the hallowed ground of Digital New Zealand.

When I left, Digital New Zealand was a newborn widget. I have returned to find a fully-fledged search service and API suite, with over 100 content partners.

That's one hundred organisations who have signed on and made their digital material searchable through the DigitalNZ APIs. Just wrangling all that disparate data into a searchable common metadata standard is a mind-boggling achievement, let alone the background work required to make such things happen. Having so many organisations on board makes the search API more valuable than ever.

So let's build something with that DigitalNZ Search Records API, and see how it all works.

We could do something client side with javascript, perhaps some kind of play on the DigitalNZ Search Widget.

Or we could do the obligatory google maps mash up, although the search API isn't yet geographically potentised. Hopefully soon.

Or we could just make a website for discovering images and videos, since the vanilla standard http://search.digitalnz.org isn't particularly exhilarating when used for this purpose. Yes, that sounds like fun.

Fast forward a few days and I had what I thought was a pretty reasonable website built on the DigitalNZ search API. There was only one problem.

When I told the DigitalNZ team about it:

So here's this website I built on the DigitalNZ API: http://elliottyoung.com/labs/nzpictureshow/

They said:

Looks good!
Good? Didn't you see how fast it was? It searches half a million images and videos in a few hundred milliseconds! And shows hundreds of thumbnails at a time!
oh, yep
Users can even narrow their search by decade, content partner, placename and rights, using cool retro tag clouds!
and geotag any result by clicking on the small map pin!
uh-huh
So did you notice that it's all done with only 150 lines of code? That's a pretty incredible website for 150 lines of code. This API changes everything!
yeah
and do you know I could actually host this awesome website for like $1/week because the API is doing all the heavy lifting and the actual images and videos are delivered to the user's browser directly from the content providers thanks to the miracle of hypertext?
sure

This went on.

At first I couldn't tell what was happening, but eventually I realised that what I thought was so great about my website was really just what was so great about the DigitalNZ APIs, and the team have been living and breathing them for quite some time. They've become accustomed to these miraculous APIs.

But I think that something this awesome has to be shouted about. I knew just what to do.

I'm going to blog about it

It will start with a story about how I came to DigitalNZ and built this picture show website with their API. Then there will be this weird conversation between the DigitalNZ team and me that possibly never happened. Last of all, we'll unpick the website, show all of the source code, and hopefully help everyone on their individual ways to doing really cool things with the DigitalNZ API.

And so it begins.

This may reveal that developing with the DigitalNZ APIs is easier than you thought. At the very least, you can surely pinch some code to use in your own projects.

I need to apologise to users of older versions of Internet Explorer, for how the code snippets appear on your screen. For the rest of us they should just overflow the righthand margin in an ugly but perfectly functional manner.

Building a Website on the DigitalNZ API

Use any Language

I have used PHP and MySQL because they're widely known, friendly, and come for free on this $1 per week hosting plan I found. Best of all, PHP won't snipe at you for a hundred minor things that fancier languages will, so you can write code that just gets it done. Oh, and when it hits a bug, it won't print a whole page of barely intelligible stack trace to the screen without giving a clue to where the error even occurred. No, it'll just say something like: "on line 15 in index.php you missed a semicolon" or something like that. It wins.

If you're making a widget — a small "panel" that sits within another website — you will have to use Javascript or Flash, since widgets run inside the user's browser and the options are limited. But if you're making a normal website with the DigitalNZ API, almost any language and database you can think of will be fine.

Of course, if developing on your own computer, PHP comes pre-installed on Mac OS X and Linux, and you can download a perfectly good setup for Windows called xampp. Perhaps best of all, you can google site:php.net <what I need> and you'll almost always find documentation about the function you need, with lots of examples.

Start with a "Hello World"

Before we look at the NZ Picture Show in detail, we'll start with a really simple script that uses the DigitalNZ Search API to display some results and thumbnails for the search_string given in the URL. Here's how it looks in action - and here's the script:

Notes Code
you can get your API key here •

retrieve the parameter from the URL •
search for sunrises if nothing else •

set up the API call •
make the API call •
dump the results to the screen (disabled) •

loop through the results •
linking each one to its original •
displaying a thumbnail •
and a title •

<?php

$api_key = '<your api key>';

$search_text=stripslashes($_GET['search_text']);
if(!$search_text) { header('Location:'.$_SERVER['PHP_SELF'].'?search_text=sunrise'); exit; }

$api_call = 'http://api.digitalnz.org/records/v2.json?search_text='.urlencode($search_text).'&api_key='.$api_key;
$api_response = json_decode(file_get_contents($api_call),true);
//var_dump($api_response['results']);

foreach($api_response['results'] as $result) {
	echo '<a href="'.$result['source_url'].'">
	      <img src="'.$result['thumbnail_url'].'"/>
	      <br />'.htmlspecialchars($result['title'],ENT_COMPAT,'UTF-8').'</a><br /><br />';
}
?>

One of the great things about PHP is that you can probably copy the code above, paste it into a new text editor window, save it as helloworld.php to the www or htdocs folder on your own computer or on your own webhost, and point your browser at http://localhost/helloworld.php. The only thing you'll need is your own API key, which is just a long string of characters by which the API will know you. Luckily, getting one is free, quick and easy - just follow the instructions at http://digitalnz.org/developers/getting-started.

If you're wondering how this API call actually works:

$api_response = json_decode(file_get_contents($api_call),true);

file_get_contents is a PHP function that retrieves a URL (just like a web browser, only it puts the output in a string instead of on the screen). Because we want the data in an array (which allows us to use the super handy foreach loop), we then pass it to json_decode. Job done!

Use what you like from the Source Code of the NZ Picture Show

The website comprises 4 PHP files, included in full below:

index.php - the main script
functions.php - a bunch of functions used by index.php
constants.php - just some constants, such as API key, number of results to return, etc
db.php - a small script that adds a suggestion to the database for the 'Popular' section on the front page. Very optional; I just wanted to show MySQL in use.

There's also a simple css file, some javascript (jquery and jquery.infinitescroll, to automatically load more images as you approach the bottom of the page) and a logo. But we're not concerned with them.

Let's look at the php files one at a time

The Source Code, Part I: index.php

The default landing page is called index.php for historic reasons; it doesn't have anything to do with indexing.

Note that any raw html in a php file is just output directly to the browser as html; to invoke php code you need to enclose the code in <?php and ?>.

Notes Code

get the
parameters
from the
url


call the API •




output the html to set up the 
page, its search forms and so on.







































if there's some results...

display the 'tag clouds' at the top •

display the thumbnails •


make a link to the next page •

or there's no results...

this was a user search •
that found nothing
or the API returned an error •

or this is the landing page so
display the 'Popular' cloud •

and let them add a Popular term •










load the javascript for twitter,
and the infinite scroll feature

and configure it
<?php
include 'constants.php';
include 'functions.php';

$search_text=stripslashes($_GET['search_text']);
$applied_filters=stripslashes($_GET['applied_filters']);
$page=max(1,floor($_GET['page']));
$suggestionadded=floor($_GET['suggestionadded']);

if($search_text or $applied_filters) {
	$api_response=call_api($search_text,$applied_filters,$page);
	if($api_response) $result_count=$api_response['result_count']; else $api_failure=1;
}
?>

<!DOCTYPE html>
<html>
<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
	<link rel="stylesheet" type="text/css" href="css/basic.css" media="screen" /> 
	<script type="text/javascript" src="js/showhide.js"></script>
	<title>NZ Picture Show | <?php if ($search_text) echo htmlentities($search_text); else echo 'Home';?></title>
</head>
<body OnLoad="document.searchform.search_text.focus();">
	<div class="block banner">
		<div class="headline"><a href="?">NZ Picture Show</a><br />
		<a href="http://twitter.com/share" class="twitter-share-button" data-count="none" data-via="digitalnz">Tweet</a></div>
		<div class="search">
			<form name="searchform" action="">
					<input name="search_text" type="text" class="textbox" value="<?php echo htmlspecialchars($search_text,ENT_COMPAT,'UTF-8'); ?>"><br />
					<input class="Search button" value="Search" type="submit"> &nbsp;
					<?php if ($result_count>0) echo $result_count.' Results found in '.$execution_time.' seconds. &nbsp; 
					<a target="_blank" href="http://search.digitalnz.org/search?search_text='.
					urlencode(DEFAULT_SEARCH_TEXT.' '.$search_text.' '.$applied_filters).'">View on DigitalNZ</a>'; 
					else echo 'Type a word or two and click Search'; ?> 
					 &nbsp; <span onclick="showhide('abouttext'); return true;">About</span> &nbsp; <a href="?">Popular</a>
			</form>
			<div id="abouttext" style="display:none;">
					<div class="h">About this website</div>
					This website demonstrates use of the <a href="http://digitalnz.org" target="_blank">
					Digital New Zealand</a> API to query and discover Digital New Zealand content.  A tutorial, including the 
					full source code, is available at <a href="http://digitalnz.org" target="_blank"
					>http://digitalnz.org</a>.
					<div class="h">Terms of Use</div>
					Use of this website is governed by the terms of use of DigitalNZ at 
					<a href="http://digitalnz.org/terms/" target="_blank">http://digitalnz.org/terms/</a>.
			</div>		
		</div>
		<div style="float:right;">
			<a target="_blank" href="http://www.digitalnz.org"><img style="border:0;" src="images/DNZ_Logo.jpg" alt="Powered by Digital New Zealand"></a>			
		</div>
		<div style="clear:both;"></div>		
	</div>

		<?php 
	
	if($result_count>0) {

		display_clouds($api_response,$search_text,$applied_filters); 
		echo '<div id="content" style="clear:both; ">';
			display_thumbs($api_response); 
		echo '</div>';
		echo '<div style="clear:both;"></div>';
		echo '<div class="navigation"><a href="?search_text='.urlencode($search_text).'&applied_filters='.urlencode($applied_filters).'&page='.($page+1).'">Next</a></div>';

	} else {
	
		if($api_response) echo '<div class="block"><span class="heading">NO RESULTS</span>
			<br />&nbsp;<br />Sorry, your search returned no results.  Please try again, or click a suggestion below to get started.</div>';
		elseif($api_failure) echo '<div class="block"><span class="heading" style="color:red;">ERROR</span>
			<br />&nbsp;<br />Sorry, your search returned an error from the search service.  Please try another search.</div>';
		echo '<div class="block"><span class="heading">POPULAR</span>';
			display_popular();
		echo '</div>';
		echo '<div class="block"><span class="heading">SUGGEST A SEARCH TERM</span><br />&nbsp;<br />
			'.($suggestionadded?'Thanks!  Your suggestion has been queued for moderation. ':'').'Add your favourite search to the Popular section:
			<form name="suggestionform" action="db.php">
					<input name="suggestion" type="text" value=""><br />
					<input class="Search button" value="Add" type="submit">
			</form>
			</div>';
		
	}
	?>

	<script type="text/javascript" src="http://platform.twitter.com/widgets.js"></script>
	<script type="text/javascript" src="js/jquery-1.3.2.min.js"></script>
	<script type="text/javascript" src="js/jquery.infinitescroll.min.js"></script>
	<script type="text/javascript">
		$('#content').infinitescroll({
		navSelector  : "div.navigation",            
		nextSelector : "div.navigation a:first",    
		itemSelector : "#content div.post",         
		bufferPx     : 100,
		loadingImg   : "images/blank.gif",          
		loadingText  : "",    
		donetext     : "" 
		});
	</script>
</body>
</html>

The Source Code, Part II: functions.php

This contains the functions used to render the search results and facet tag clouds.

Notes Code
call_api
Calls the Search API and
returns the results


All these API parameters 
are documented here.




The actual API call


If there's an error, return false


tag_size
returns a font-size %


sanitise
takes care of macrons etc


display_clouds
displays a tag cloud for each
facet in the results

the facets in the array can best be
seen with a var_dump from the api, 
but they are returned as an array 
of facets, where each facet is made 
up of a facet_field (eg decade) and 
an array of values, where each value 
has num_results (the number of results 
that would be returned if that facet were
applied) and the string (eg "1960-1969")



array_multisort puts the results in
alphabetical order within each facet

loop through the tag_fields, tag_values 
and tag_counts arrays











Add a string to the search to filter
the results based on the tag clicked, eg
decade:1960-1969






display_thumbs
Displays a page of thumbnails with
captions beneath


display the thumbnail •




the pin •






the title •
and the content provider •







display_popular
Display a tag cloud of popular options


Retrieve all the 'published' suggestions •




Print them out at the appropriate size •
<?php

function call_api($search_text,$applied_filters,$page) {
global $execution_time;
	$api_call='http://api.digitalnz.org/records/v2.json?num_results='.NUM_RESULTS.
		'&search_text='.urlencode(implode(array(DEFAULT_SEARCH_TEXT,$search_text,$applied_filters),' ')).
		'&facets='.SHOW_FACETS.
		'&facet_num_results='.FACET_NUM_RESULTS.
		'&start='.max(0,($page-1)*NUM_RESULTS).
		'&sort='.SORT_KEY.
		'&direction='.SORT_DIRECTION.
		'&api_key='.API_KEY;
	$time_start = microtime(true);
	$result=json_decode(file_get_contents($api_call),true);
	$time_end = microtime(true);
	$execution_time = round($time_end - $time_start,2);		
	if(is_array($result)) return $result; else return false;
}

function tag_size($tag_count,$max_count) {
	return min(TAG_MAX_SCALE,floor(TAG_SIZE*(1.0+(TAG_RATIO*$tag_count-$max_count/2)/$max_count)));
}

function sanitise($text) {
	return htmlspecialchars($text,ENT_COMPAT,'UTF-8');
}

function display_clouds($api_response,$search_text,$applied_filters) {
		
	$max_count=0; 

	foreach($api_response['facets'] as $facets) {
		$facet_field= $facets['facet_field'];

		foreach($facets['values'] as $value) {
			$facet_value=$value['name'];
			$facet_count=$value['num_results'];
			$tag_values[]=$facet_value;
			$tag_fields[]=$facet_field;
			$tag_counts[]=$facet_count;
			$max_count=max($max_count,$value['num_results']);
		}
	}
	array_multisort($tag_fields,$tag_values,$tag_counts);
	$c=count($tag_values);

	echo '<div class="block">';
	for($i=0;$i<$c;$i++) {
		# Add Tag Category heading if it's the first tag of its category
		if ($tag_fields[$i]<>$last_tag_field) { if($last_tag_field) echo '</div><div class="block">'; 
			echo '<span class="heading">'.strtoupper($tag_fields[$i]).'</span>'; $last_tag_field=$tag_fields[$i]; }
		if(!(strpos(urlencode($applied_filters),$tag_fields[$i].'%3A%22'.urlencode($tag_values[$i]).'%22')===FALSE)) { 
		# Tag has already applied
			echo '<a class="tag'.($i%2).'" title="remove '.htmlspecialchars($tag_fields[$i].':"'.$tag_values[$i].'"',ENT_COMPAT,'UTF-8').' 
				from filters" style="font-weight:bold; color: #c8470f; font-size: '.tag_size($tag_counts[$i],$max_count).'%;" 
				href="?search_text='.urlencode($search_text).'&amp;applied_filters='.str_replace($tag_fields[$i].'%3A%22'.
				urlencode($tag_values[$i]).'%22','',urlencode($applied_filters));
		} else {
		# Tag has not been applied
			echo '<a class="tag'.($i%2).'" title="'.$tag_counts[$i].' items" style="font-size: '.tag_size($tag_counts[$i],$max_count).'%;" 
				href="?search_text='.urlencode($search_text).'&amp;applied_filters='.urlencode($applied_filters).
				($applied_filters?'+':'').$tag_fields[$i].'%3A%22'.urlencode($tag_values[$i]).'%22';
		}
		echo '">'.str_replace(' ','&nbsp;',$tag_values[$i]).'</a> ';
	}
	echo '</div>';
}		

function display_thumbs($api_response) {
	foreach($api_response['results'] as $value) {
		echo '
			<div class="post">
				<a target="_blank" href="'.sanitise($value['source_url']).'">
					<img src="'.sanitise($value['thumbnail_url']).'" 
						title="'.sanitise($value['title']).' ('.sanitise($value['content_provider']).'): '.sanitise($value['description']).'"/>
				</a>
				<div class="metadata">
					<a target="_blank" href="http://search.digitalnz.org/records/show/'.$value['id'].'.html">
						<img class="pin"  src="images/pin.gif"/>
					</a>
					<div>
						<a title="'.sanitise($value['title']).'" target="_blank" href="'.sanitise($value['source_url']).'">'.
							substr(sanitise($value['title']),0,45).'</a>
					</div>
					<div>
						<a class="provider" title="'.sanitise($value['content_provider']).'" target="_blank" href="'.
							sanitise($value['source_url']).'">'.substr(sanitise($value['content_provider']),0,45).'</a>
					</div>
					
				</div>
			</div>';		
	}
}

function display_popular() {
	$connection = mysql_connect('localhost', MYSQL_USER, MYSQL_PASSWORD);
	$max = mysql_fetch_row(mysql_query('select max(hitcount) from younge_dnz.suggestions where published=1'));
	$max_count=$max[0];
	$suggestions = mysql_query('select suggestion,hitcount from younge_dnz.suggestions where published=1 order by suggestion');
	$i=0;
	while($row=mysql_fetch_row($suggestions)) {
		$suggestion=$row[0];
		$hitcount=$row[1];
		echo '<a class="tag'.($i%2).'" title="'.$hitcount.' items" style="font-size: '.tag_size($hitcount,$max_count).'%;" href="?search_text='.
			urlencode($suggestion).'">'.str_replace(' ','&nbsp;',$suggestion).'</a> ';
		$i++;		
	}
}

?>

The Source Code, Part III: constants.php

This one's simple enough. A bunch of things you can configure.

Notes Code
your API key •
how many thumbnails to grab at once •
the filter that is applied to all the searches •
how many tags are in each cloud •
how the results are sorted - a field•
how the results are sorted - asc/desc •
database username •
database password •
which tag clouds to show •
tag cloud largest font % scale •
tag cloud overall font size •
tag cloud size differential •
<?php

define('API_KEY','put your key here');
define('NUM_RESULTS',50);
define('DEFAULT_SEARCH_TEXT','(category:Images OR category:Videos)');
define('FACET_NUM_RESULTS',20);
define('SORT_KEY','date');
define('SORT_DIRECTION','asc');
define('MYSQL_USER','put your database username here');
define('MYSQL_PASSWORD','put your database password, if any, here');
define('SHOW_FACETS','decade,placename,rights,category,content_partner');#choose from category,century,decade,year,creator,rights,collection,content_partner
define('TAG_MAX_SCALE',250);
define('TAG_SIZE',200);
define('TAG_RATIO',1.6);

?>

The Source Code, Part IV: db.php

Takes a script parameter and adds it to the MySQL database of popular searches, if valid. Very optional.
Notes Code

get the parameter from the URL •
if there's a suggestion and it's not too long •
connect to the database •
see how many results this search would return •

if it would return more than one result •
add it to the database •
and flag the fact that we added it •



redirect back to the home page •
<?php
include 'constants.php';
include 'functions.php';

$suggestion = stripslashes($_GET['suggestion']);
if($suggestion<>'' and substr_count($suggestion," ")<4) {
	$connection = mysql_connect('localhost', MYSQL_USER, MYSQL_PASSWORD);
	$api_response=call_api($suggestion,'','',0,0);
	$result_count=$api_response['result_count'];
	if($result_count>0) {
		mysql_query('insert into younge_dnz.suggestions (suggestion,hitcount,published) values ("'.mysql_real_escape_string(ucfirst($suggestion)).'",'.$result_count.',0)');
		$suggestionadded=1;
	}
}

header("Location: http://".$_SERVER['HTTP_HOST'].rtrim(dirname($_SERVER['PHP_SELF']), '/\\')."/?suggestionadded=".$suggestionadded);
exit;

?>

I'm using a MySQL database to store the popular searches that appear on the homepage. Like PHP, MySQL is free and ubiquitous. The table can be created by executing the following SQL, using PHPMyAdmin or similar:

Notes SQL
the key words •
how many hits this search gets •
whether this search is visible •
CREATE TABLE `suggestions` (
  `suggestionid` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `suggestion` varchar(255) NOT NULL,
  `hitcount` int(10) unsigned NOT NULL DEFAULT '0',
  `published` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`suggestionid`),
  UNIQUE KEY `suggestion` (`suggestion`),
  KEY `published` (`published`)
) ENGINE=MyISAM AUTO_INCREMENT=132 DEFAULT CHARSET=latin1

The Wrap Up

The DigitalNZ APIs open up an unbelievably rich and deep content source for you to use in your own projects.

As you've seen here, you can use the APIs to create something new, and then share that website or widget with the world.

Using the same techniques, you can also combine our APIs with a growing number of other organisations' APIs, to create a "mash up" (or something so cool it doesn't even have a name yet) that is greater than the sum of its parts. This is a very exciting area for development.

Whatever you're doing, or thinking about doing, with the DigitalNZ APIs, I hope you've found something useful in this post, and I look forward to seeing what you've made! Please send me a link, or post a comment below.

Elliott Young
DigitalNZ