Coming Home... a search showcase

Posted on 24 November 2008 by Thomasin

The Coming Home search experience went live on 11 November 2008, to celebrate the 90th Anniversary of the World War One Armistice. This was a sneak preview of the DigitalNZ tools that allow users to build their own NZ search tool, finding hidden and buried NZ content about their topic of interest. We undertook to connect together content from a whole range of sources, and in the process learn about how to open up more of NZ's rich digital treasures.

How did we do it and what did we learn?

Getting metadata content

The Coming Home search was mainly fuelled with content contributed by members of the National Digital Forum (NDF). A key feature of the DigitalNZ search approach is that we focus on aggregating metadata content, and on collecting descriptions of content objects, so we can drive visitors to websites around NZ. We had a three-pronged approach to gathering that content:

  • Direct harvesting via OAI-PMH, XML sitemaps, or RSS feeds

  • Collaboration with aggregating sources that contributors were already part of, including Matapihi , KRIS ,  NZ Museums , and NZ On Screen

  • Providing a repository website for content that couldn't immediately get online otherwise. We call this service Kete DigitalNZ , and if you don't have a website we can host your organisation's digital treasures for you

What did we ask from content contributors?

We approached organisations and individuals to participate and asked them to let DigitalNZ do one, some, or all of the following:

  • Regularly harvest and cache metadata and thumbnails so the content is discoverable in search experiences powered by DigitalNZ (a bit like Google).

  • Establish and maintain RSS feed services so the content can be discovered through other websites and online services

  • Allow public API  access to the harvested metadata so it can be queried, discovered, and connected with other data sources (including in third party applications).

So what does all this mean? We asked that people share their metadata with us, so that we could share it with everyone. This is not a straight-foward thing to agree to, and even now we are still working through the details. We thank everyone who gave us the benefit of the doubt when all we had were concepts and ideas.

Building a search tool

Very early on in the project we decided building another website or online exhibition around 'Coming Home' would be a wasted effort. Instead we set out to allow anyone to build a search tool that explored the NZ content that they were interested in. We created the search builder and then used it to build our very own 'Coming Home' search. The search builder uses a straight forward boolean search query to gather up all the content we we're interested in, for example here's the query for 'Coming Home':

((War OR WW? OR "great war" OR peace OR flu OR influenza OR day? OR victory OR treat* OR parade OR celebration OR armistic* OR soldier* OR army OR anzac OR wounded OR gallipoli OR commemorat* OR memorial OR monument? OR carillon OR medal* OR military OR (maori AND battalion)) AND (191? OR 192?)) OR ComingHome2008 OR (returned AND servic*) OR "R.S.A" OR RSA OR veterans

At the time of writing this post, the query pulls together 14,156 related items for the 'Coming Home' search. The result is unique New Zealand content on a chosen topic being brought together and accessed through discovery points all over the web.

What has been learnt?

There is so much more to say, but we are only at the beginning of the journey. Here are a few things we've noted:

  • Build solutions to fit the environment rather than expect the environment to fit the solution. More methods of harvesting give us more flexibility.

  • Developing a good search experience is not just about being able to harvest metadata. Good quality metadata aids discoverability of content.

  • Good quality metadata is not just about structure. Shared approaches to data values in dates, subject terms, and so on also aid discoverability of content.

  • Letting users enhance metadata might allow us to benefit both content providers and users – a test for another day.

  • A lot of metadata exists telling people what they can’t do with content and far less telling people what they can do. We need to make both obvious.

  • Explaining new generation web concepts such as ‘API’ in a vacuum is difficult without examples. 

What's next?

You can of course use the search builder to create your own search experience, it's a demonstration of what's possible. What can you add? Can you become a contributor? Can you develop using the APIs? Do you know of a burning issue in this digital space that needs attention? Have something to share or comment on? We'd love to hear from you.

We've turned off comments here, but we'd still love to know your thoughts. Visit us on our Facebook Page @digitalnz or on Twitter @DigitalNZ to share any ideas or musings with the DigitalNZ team.