Google search within a site made easier for navigational searches

On Wednesday, we noticed Google was starting to surface a search within box for some results for navigational searches:

It turns out it’s a wider roll-out by Google as the Official Google blog reports.

“However, one of the trends we noticed while studying teleporting was that there were lots of searchers who would type the name of a specific website as if they wanted to teleport, but would then immediately issue another more a refined search within this site.

Through experimentation, we found that presenting users with a search box as part of the result increases their likelihood of finding the exact page they are looking for. So over the past few days we have been testing, and today we have fully rolled out, a search box that appears within some of the search results themselves. This feature will now occur when we detect a high probability that a user wants more refined search results within a specific site. Like the rest of our snippets, the sites that display the site search box are chosen algorithmically based on metrics that measure how useful the search box is to users.”

The last sentence is interesting. We’ve certainly noticed a wide implementation across UK government websites.

RSS feeds for search results

I did a web search for ‘RSS feeds for search results’ and found a Live Search blog entry dating from 1995 that talks about the alpha feature of providing RSS feeds from search results: just add ‘&format=rss’ to the URL of the results and copy the full URL into your RSS Reader.

This still works on the main Live search site and is made easier by just clicking on the orange RSS icon in the URL. Here’s the ‘RSSed’ URL for Flooding information from the UK government: http://search.msn.com/results.aspx?q=flooding+site%3agov.uk&format=rss

Very nifty, but interestingly, Live don’t seem to draw users’ attention to it or even provide information in Help.

Yahoo! offers the similar features to Live, with different syntax, but the RSS icon.

In contrast, Google only offers RSS feeds (RSS and Atom) for its news results but does publicise it clearly. Similarly, the BBC offers RSS of its news and sports results.

Other web search engines such as Ask, Exalead and Clusty do not appear to offer RSS at all.

Search usability

Shari Thurow has recently blogged on Understanding search usability 1 and 2. I was expecting something on designing search interfaces for usability. Rather, Thorow focuses on the message that optimising the search experience is much more than traditional SEO and should take account of users’ search behaviour.

Thurow argues that search usability is misunderstood by a range of new media disciplines and aims to dispel misconceptions around the use of usability studies, focus groups and web analytics. She argues that key to understanding SEO and web site usability are the human factors. “Why do people do what they do before and after they arrive on your web site? By objectively observing target audience members and carefully analyzing their search behavior, web site owners can improve their web sites.”

Thurow outlines Marcia Bates’ concept of berrypicking - searching is not a linear behavior, rather it comprises a wide variety of behaviors including, but not limited, to: querying, refining, expanding, browsing/surfing, pogo-sticking, foraging, scanning and reading. So users’ berrypicking behaviour needs to be recognised in designing more effective search interfaces.

“The term “search usability” addresses all search behaviors on a single web site, not only querying behavior, and not only browsing behavior. A user-friendly, search-friendly web site accommodates berrypicking behavior and delivers searchers to the information they desire as quickly and easily as possible.”

Search usability needs to incorporate not only the usability of the query interface (search box and results pages etc.) but also ensuring sites are search-friendly (rather than purely search-engine friendly).

“The term “search usability” addresses all search behaviors on a single web site, not only querying behavior, and not only browsing behavior. A user-friendly, search-friendly web site accommodates berrypicking behavior and delivers searchers to the information they desire as quickly and easily as possible.”

So it is important to:

• Identify types of search behaviour
• Understand how these search behaviours are all related
• Design a web site that addresses all or most of these search behaviors, by understanding the searcher’s experience as an objective observer.

Thurow advises remembering three things:

1. You are NOT the user
2. Even if a user fits a profile, persona, or role, a user is not objective or accurate about evaluating his own behavior
3. Users are not always right

So Thurow concludes that “Search usability is a complex subject. There are many types of search behaviors, and plenty of elements on a web page that need to be formatted in such a way to accommodate these behaviors.” I particularly liked her point that “You are here” cues are very important. When users click on a link from web search, they are likely to land in the middle of the site. So good label to help searchers orientate and form a mental model of the site are important in instilling confidence.

The Social, Political, Economic, and Cultural Dimensions of Search Engines

The April 2007 edition of the Journal of Computer-Mediated Communication ran a special theme on the Social, Political, Economic, and Cultural Dimensions of Search Engines.

Despite search engines’ central role in how people access information, little social science study has focused on the non-technical dimensions, the search companies or users’ the practices.

As the search industry has consolidated, decisions made by just a few players can have considerable repercussions for what material is realistically within the reach of users. So it is important to look at what factors determine inclusion and exclusion criteria in search results and how users approach them; so as to gain a better understanding of how users’ access to content is being mediated by a handful of commercial services.The first two articles examine people’s search engine uses at the level of responding to results pages.

• Heuristic and Systematic Use of Search Engines
• In Google We Trust: Users’ Decisions on Rank, Position, and Relevance

The next cluster of articles considers the larger social context of searching.

• Searching for Culture—High and Low
• Learning to Search and Searching to Learn: Income, Education, and Experience Online

The last three articles focus on material covered by search engines, including a look at what decisions influence the content and presentation on these services, a comparison of search engines in different countries, and the possible manipulation of one such tool.

• Is Relevance Relevant? Market, Science, and War: Discourses of Search Engine Quality
• Equal Representation by Search Engines? A Comparison of Websites across Countries and Domains
• Google Bombing from a Time Perspective

Strategies for improving enterprise search

John Ferrara argues the importance of going beyond the out-of-box experience at Boxes and Arrows Sep 07.

I’d agree with his argument that quality results only come about through applied effort.

The conceptual task

1. Search engine. The algorithmic gears that parse the query and assign pages relevance.
2. Content. The documents searched.
3. Index. A catalogue of the locations of every word in every document.
4. User input. The keywords and other parameters the user submits.
5. Results display. The way the data returned by the search engine is presented.

Strategies

1. Make content machine readable.
   • Structural markup - using semantic elements and class attributes.
   • Use the keywords and descriptions tags and build a controlled vocabulary to populate. Well, I think there’s rooms for a and the tactical application of wild keywords.
   • More metadata - audience, sector etc.
   • Ontology - Get the search engine to exploit the relationships between concepts.
2. Index All of the Right Data
   • Ignore unnecessary content - navigation, footers, adverts etc.
   • Get all resources - what about content in PDFs, Word documents etc?. I’d agree that search needs to be aware of them, but disagree about indexing them all. I often find a more usable experience is to index a web page that provides links to and a summary of such documents. The ranking of digital assets can be unpredictable and getting it right is, as Ferrara says, a big job
3. Make the most of user input - it is important to make the best of the user’s intent on their first search attempt- after all the user has probably a complex intent behind those few keywords and users are much less likely to make a second. (People Search Once, Maybe Twice - Jared Spool 2001).
   • Query expansion - stemming, thesaurus.
   • Syntax conventions - get the parser to understand common human syntax - and use AND as the default operator - not sure about that; it can be too restrictive.
   • Assisting query formulation - Did you mean?, others searched for…
4. Build results around the user’s needs
   • Demonstrate relevance
     • Generate a snippet which contains the user’s terms. I disagree - in an enterprise solution, they often not very good and you have the opportunity to teach editors to write good, relevant descriptions which include the most important keywords.
     • Bold the terms that match the user’s query terms.
   • Best bets/ recommended links. I’m a huge fan of these as they can really help orientate users and push important content for the business.
   • Conditional content. Take best bets further and display contextually appropriate content when a query indicates a user has a particular interest.

Information behaviours

“People don’t understand their own information behaviors, and they don’t really understand much about search or the web, so they will have to learn. It could take generations.”

Christina Wodtke from Boxes and Arrows interviews Amanda Spink in Oct 2006.

Most web queries short:
2- 3 terms

Sessions:
2-3 queries in length with little query modification

Long tail:
The long tail makes ‘relevant’ retrieval very hard - especially if users use only 2 or 3 words - search is not very interactive. Personalisation is an attempt to get more info from user to help determine relevance.

Complex search behaviours:
Often long and complex - beyond one topic/one search paradigm most search engine assume

• Successive searching (e.g. looking for information on cars)
• Multitasking search (batch searches - time constraint or new topics emerge)

Search is challenging and interactive:

• no silver bullet or ’single’ feature that’s effective
• clustering + relevance feedback +??
• lack of people trained in info and web retrieval, web design and usability
• how to elucidate real intent from a small number of keywords?

Wikia search launches

Wikia Search launched today. Nice simple interface; but, as the site says, “the results are pretty bad”. But the concept is that trusted user feedback from a community of users acting together in an open, transparent, public way will improve results. This seems to be based on providing user feedback and creating mini articles that provide short definitions, disambiguations, photos and ’see also’s.

Users get to see the Nutch relevancy score by clicking on the numerical score by each result.

The seven deadly sins of site search according to Vivisimo

A Vivisimo white paper, available at searchdoneright.com lists the seven deadly sins of site search. It’s a basic, but useful introduction:

1. Omission - Make sure you offer search - some users prefer it and it’s an escape route when navigation fails.
2. Apathy - Make sure you check how search is functioning - relevancy, scent of information from titles an descriptions.
3. Complexity - Keep search interface simple - don’t offer >1 search box on a page.
4. Omnipotence - Beware the conceit that your engine will understand the user’s intent and respond with an ‘answer’ to a ‘question’.
5. Egoism - Don’t ask too much of a visitor - make it user-friendly.
6. Brand confusion - Ensure the search results has your brand and your URL.
7. Multiple personality - Ensure search pages have a consistent branding and look and feel to the rest of the site.

But search will eventually change all websites

Matt Chapman in Information World Review 01 Jun 2007, summarises John Batelle’s conversation at HP ’s Print 2.0 conference in New York.

Google is now the default interface for the web but will not always be the dominant point of access, “The web has an interface and I would argue that it is Google right now.”

“Search is that interface but it is not always going to be, just like it was not always DOS and it is not always going to be Windows.”

Batelle compared Google’s sparse homepage to the command line from DOS that was used to get information from a computer.

“Where are we now in search? Well, the command line, but with a huge difference: we are not talking in the computer’s language, we are talking in our language,” he said. So search is facilitating a conversation.

Batelle argued that the way search treats its users would eventually have wide-reaching effects for all websites.

“Think about what search does. You come to a place (Google, MSN, Yahoo, whatever), you say something and the whole place reorganises around what you just said,” he explained.

“And this is an interface that we are getting so used to that we are going to start getting mad at businesses that do not do that for us.”

See the videos at HP Corporate TV.

Internet World

I attended Internet World at Earl’s Court 2 on 1 May. The attraction was the free seminars across wide variety of subjects. The reality was a trifle disappointing - clashing events on similar subjects, rather basic presentations and a bit crowded. Did I learn much, well not too much, but I came away with some ideas. I attended these seminars:

Using Google Website optimiser to improve site search
A Google salesman talking about Google Analytics and Google Website Optimizer .

Website Optimizer, Google’s free multivariate testing application, helps online marketers increase visitor conversion rates and overall visitor satisfaction by continually testing different combinations of site content (text and images).

London Borough of Southwark: enterprise search in the real world with Google
Alison Leahy and Stephen D’Arcy wanted one place to search across repositories, including public site and intranet.

Working with Jadu, they implementing Jadu Rupa for Google. Allows user to set and save preferences for scopes. They called this clustering…

Implemented IPSV as a thesaurus for preferred and non-preferred terms. Planning iterative addition of new collections.

Using a content management system to achieve effective search engine marketing
Nigel Jackson from Immediacy did a SEM 101:

CMS should deliver:

• W3C XHTML compliance
• clean uncluttered code
• friendly URLs
• autogeneration of site maps
• accessible navigation (text based)
• mandated actions for editors - e.g. mandatory metadata
• link checker

CMS lets user do:

• optimisation of titles (allows short titles for nav pages)
• summaries
• manage links
• encourage back links
• easily create landing pages, targetting keywords

Useful links:
www.highrankings.com
www.longtail.com

Maximising customer catch points in natural search
Your Amigo which has a technology that spiders sites, identifies important keywords and gaps and creates loads of extra pages optimised around these terms.

Managing huge paid search campaigns
The most interesting presentation of the day. Efficient Frontier use algorithms developed for financial markets to plan SEM campaigns in the increasingly opaque markets where paid position is now determined by click through rates and quality as well as cost per click. The algorithms handle lots of parameters that help optimise your ROI.