First, I wanted to give readers a heads up that I’ll be in Seattle this Friday and Saturday. I’ll spend Friday afternoon at the University of Washington, meeting with some of their outstanding computer science doctoral students. My schedule filled up with unexpected haste! But if you’re on campus and urgently want to meet, let me know and I’ll see what I can do.

Saturday I’ll be attending the social track of WSDM 2012, the premier international ACM conference covering research in the areas of search and data mining on the Web. I’m excited about the program, as well as the opportunity to catch up with friends and make new ones. Back in 2010, I had the pleasure of co-organizing the Workshop on Search and Social Media (SSM 2010) and being the official ACM blogger for WSDM 2010. You can read my posts here.

Then, on Saturday evening, I’ll be heading to Microsoft Research to attend the Social Search Social (SSS 2012). Hats off to organizers Meredith Ringel Morris, Gene Golovchinksy, Jeremy Pickens, Madhu Reddy, Chirag Shah, and Michael Twidale for creating what looks to be a fun (and very social!) event. I’m especially looking forward to the 45-second “madness” presentations (in which I’m participating) and the “speed dating” to help cross-pollinate  the WSDM and CSCW communities.

Hope to see some of you there, and of course will share what I learn here at The Noisy Channel. I also encourage you to follow the tweet streams for #wsdm2012 and #sss2012.

As regular readers know, I have a deep affection for Carnegie Mellon University, where I did my graduate work. I’m happy to announce that two of my colleagues (both fellow CMU PhDs) will be giving talks at CMU in a couple of weeks, and I hope that some of you will have the opportunities to attend.

On Tuesday, February 7th, Abhimanyu Lad will be hosting an information session at 6pm in Scaife Hall, Room 214. Abhi is rock star on our data science team, and he’s been working on the next generation of LinkedIn search. You can get a taste of his work from his recent HCIR 2011 presentation, “Is it Time to Abandon Abandonment?“. Abhi will talk about a variety of technical challenges that data scientists and engineers are working on at LinkedIn.

On Thursday, February 9th, Paul Ogilvie will talk about “Where Big Data Meets Real-Time: Efficiently Indexing and Ranking News using Activity” at 3:30pm in GHC 6115. Paul is responsible for article relevance infrastructure and algorithms on LinkedIn Today, a great example of social navigation – not to mention a great success for users. Paul will talk about the technical details that make LinkedIn Today possible, including a novel use of inverted lists to efficiently index and support real-time updates to document representations.

And, even if you can’t make it to the talks, I encourage you to visit the LinkedIn booth at the EOC fair on Wednesday, February 8th. We’re looking for great software engineers and data scientists, and we’re especially interested in interns.

This post is last in a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

The final talk of the CIKM 2011 Industry Event was a talk from Yandex co-founder and CTO Ilya Segalovich on “Improving Search Quality at Yandex: Current Challenges and Solutions“.

Yandex is the world’s #5 search engine. It dominates the Russian search market, where it has over 64% market share. Ilya focused on three challenges facing Yandex: result diversification, recency-specific ranking, and cross-lingual search.

For result diversification, Ilya focused on queries containing entities without any addition indicators of intent. He asserted that entities offer a strong but incomplete signal of query intent, and in particular that entities often call for suggested query reformulations. The first step in processing such a query is entity categorization. Ilya said that Yandex achieved almost 90% precision using machine learning, and over 95% precision by incorporating manually tuned heuristics. The second step is enumerating possible search intents for the identified category in order to optimize for intent-aware expected reciprocal rank. By diversifying entity queries, Yandex reduced abandonment on popular queries, increased click-through rates, and was able to highlight possible intents in result snippets.

Ilya then talked about the problem of balancing recency and relevance in handling queries about current events. He sees recency ranking as a diversification problem, since a desire for recent content is a kind of query intent. A challenge is managing recency-specific ranking is to predict the recency sensitivity of the user for a given query. Yandex considers factors such as the fraction of results found that are at most 3 days old, the number of news results, spikes in the query stream, lexical cues (e.g., searches for “explosion” or “fire”), and Twitter trending topics. He also referred to a WWW 2006 paper he co-authored on extracting news-related queries from web query logs. The results of these efforts led to measurable improvements in click-based metrics of user happiness.

Ilya talked about a variety of efforts to support cross-lingual search. Russian users enter a significant fraction (about 15%) of non-Russian queries, but many still prefer Russian-language results. For example, a search for a company name return that company’s Russian-language home page if one is available. Yandex implements language personalization by learning a user’s language knowledge and using it as a factor in relevance computation. Yandex also uses machine translation to serve results for Russian-language queries when there are no relevant Russian-language results.

Ilya concluded by pitching the efforts that Yandex is making to participate in and support the broader information retrieval community, including running (and releasing data for) a relevance prediction challenge. It’s great to see a reminder that there is more to web search than Google vs. Bing, and refreshing to see how much Yandex shares its methodology and results with the IR community.

This post is part of a series summarizing the presentations at the CIKM 2011 Industry Event, which I chaired with former Endeca colleague Tony Russell-Rose.

Given the extraordinary ascent of all things social in today’s online world, we could hardly neglect this theme at the CIKM 2011 Industry Event. We were lucky to have Ed Chi, who recently left the PARC Augmented Social Cognition Group to work on Google+, presenting “Model-Driven Research in Social Computing“.

Ed warned us at the beginning of the talk that his focus would be on work he’d done prior to joining Google. Nonetheless, he offered an interesting collection of public statistics about social activity associated with Google properties: 360M words per day being published on Blogger, 150 years of YouTube video being watched everyday on Facebook, and 40M+ people using Google+. Regardless of how Google has fared in the competition for social networking mindshare, Google is clearly no stranger to online social behavior.

Ed then dove into recent research that he and colleagues have done on Twitter activity. Since all of the papers he discussed are available online, I will only touch on highlights. I encourage you to read the full papers:

• Want to be Retweeted? Large Scale Analytics on Factors Impacting Retweet in Twitter Network
• Tweets from Justin Bieber’s Heart: the Dynamics of the “Location” Field in User Profiles
• Is Twitter a Good Place for Asking Questions? A Characterization Study
• Language Matters in Twitter: A Large Scale Study
• Eddi: Interactive Topic-based Browsing of Social Status Streams
• Short and Tweet: Experiments on Recommending Content from Information Streams
• Speak Little and Well: Recommending Conversations in Online Social Streams

Ed talked at some length about language-dependent behavior on Twitter. For example, tweets in French are more likely to contain URLs than those in English, while tweets in Japanese are less likely (perhaps because the language is more compact relative to Twitter’s 140-character limit?). Tweets in Korean are far more likely to be conversational (i.e., explicitly mentioning or replying to other users) than those in English. These differences remind us to be cautious in generalizing our understanding of online social behavior from the behavior of English-speaking users. Ed also talked about cross-language “brokers” who tweet in multiple languages: he sees these as indicating connection strength between languages, as well as giving us insight to improve cross-­language communication.

Ed then talked about ways to reduce information overload in social streams. These included Eddi, a tool for summarizing social streams, and zerozero88, a closed experiment to produce a personal newspaper from a tweet stream. In analyzing the results of the zerozero88 experiment, Ed and his colleagues found that the most successful recommendation strategy combined users’ self-voting with social voting by their friends of friends. They also found that users wanted both relevance and serendipity — a challenge since the two criteria often compete with one another.

Ed concluded by offering the following design rule: since interaction costs determine number of the people who participate in social activity, get more people into the system by reducing interaction cost. He asserted that this is a key design principle for Google+.

My skepticism about Google’s social efforts is a matter of public record (cf. Social Utility, +/- 25%; Google±?). But hiring Ed Chi was a real coup for Google, and I’m optimistic about what he’ll bring to the Google+ effort.

ps. My thanks to Jeff Dalton for live-blogging his notes.

Wednesday evening before the workshop, we were treated to a welcome reception, the first of a few meals provided by Google’s excellent chefs. It was a great opportunity to reconnect with old friends and meet many first-time HCIR attendees.

Thursday started with a scrumptious breakfast that included chilaquiles, coconut fritters, and bacon. Last year’s keynote and this year’s local host Dan Russell pulled all the stops — apparently BigTable is the only Google cafe that serves bacon for breakfast! We then proceeded to a poster boaster session in which each poster presenter had a minute to pitch his or her poster. This session set the tone for the rest of the workshop: concentrated ideas and intense audience engagement.

Then came this year’s keynote, Gary Marchionini. It was a particular treat to have Gary as a keynote, since his lecture on “Toward Human-Computer Information Retrieval” inspired me to conceive the HCIR workshop back in 2007. And Gary delivered the goods. He started with a review of the history of HCIR, including some lesser known figures like Don Hawkins (who was in the audience) , Pauline Cochrane, Richard Marcus, and Charles Meadow.  He brought a few chuckles by citing Nick Belkin (who was present) and Sue Dumais (who was not) as the father and mother of HCIR. Naturally he described some of his own work at the University of North Carolina, including the Open Video, Relation Browser, and ResultsSpace projects.But the highlight of his talk was a graph he presented showing two paths to the same user end-state, one of the paths being a smooth progression and the other being a roller-coaster of ups and down. The question of which one was better drew a wide variety of responses, my favorite being Gene Golovchinsky observing that learning is the friction of the information-seeking process.

We broke for coffee and then came back to the first session of paper presentations. Sofia Athenikos presented a semantic search engine that outperformed IMDB in a user study. Chang Liu explored the effect of task difficulty and domain knowledge on dwell times, finding counterintuitive results (at least for me) regarding the correlation of expertise to dwell time. Jingjing Liu presented research on knowledge examination in multi-session tasks. Then came the lightning talks: Mark Smucker on how users examine and process ranked document lists; Jin Kim on simulating associative browsing; Bill Kules on visualizing the stages of exploratory search; and Michael Cole on user domain knowledge and eye movement patterns during search. Way too much goodness to summarize here — I suggest you read the full papers on the workshop site.

Then came lunch — again in BigTable, but this time with outdoor seating — and the poster session. As always, this it the most interactive part of the day: two hours of non-stop discussion that start over food and end with prying people away from discussions about posters. I was especially proud of LinkedIn’s contributions to the poster session, which covered faceted search log analysis, social navigation, and whether it is time to abandon abandonment.

Then back to the second session of  paper presentations. Luanne Freund talked about document usefulness and genre, finding that genre, besides being hard for users to reliably identify, only matters for tasks that involve doing, deciding, learning; but not for those that involve fact finding or problem solving. Gene Golovchinsky presented work on designing for collaboration in information seeking, previewing the system he used for his challenge entry.  Alyona Medelyan used the Pingar search engine to evaluate how search interface features affect performance on biosciences tasks. Then more lightning talks: Rob Capra analyzing faceted search on mobile devices; Keith Bagley on conceptual mile markers for exploratory search; Xiaojun Yuan on how cognitive styles affect user performance; and Mike Zarro on using social tags and controlled vocabularies as search filters.

Last but not least came the HCIR Challenge:

The HCIR 2011 Challenge focuses on the case where recall is everything – namely, the problem of information availability. The information availability problem arises when the seeker faces uncertainty as to whether the information of interest is available at all. Instances of this problem include some of the highest-value information tasks, such as those facing national security and legal/patent professionals, who might spend hours or days searching to determine whether the desired information exists.

The corpus we will use for the HCIR 2011 Challenge is the CiteSeer digital library of scientific literature. The CiteSeer corpus contains over 750,000 documents and provides rich meta-data about documents, authors, and citations.

There were four entries:

• FreeSearch – Literature Search in a Natural Way
  Claudiu S. Firan, Wolfgang Nejdl, Mihai Georgescu (University of Hanover), and Xinyun Sun (DEKE Lab MOE, Renmin)
  
• Session-based search with Querium
  Gene Golovchinsky (FX Palo Alto Lab) and Abdigani Diriye (University College London)
  
• GisterPro
  David L.Ostby and Edmond Brian (Visual Purple)
  
• Query Analytics Workbench
  Antony Scerri, Matthew Corkum, Keith Gutfreund, Ron Daniel Jr., Michael Taylor (Elsevier Labs)

The competition was fierce. Claudiu showed off the Faceted DBLP interface, which is well suited to the information availability task on CiteSeer data. Ed showed how GisterPro uses visualization to support the information seeking process. But it came down to a close call between the Query Analytics Workbench and Querium. Despite the Elsevier team’s impressive functionality and animated presentation, Gene’s simpler interface and application of ranked fusion won the day. Congratulations to Gene and Abdigani, this year’s HCIR Challenge winners!

We wrapped up the evening at the Tied House, a local microbrewery. And of course the discussion turned to where, when, and how we will hold next year’s workshop. Watch this space. In the meantime, my heartfelt thanks to everyone who made this year’s workshop such a success — and especially to our sponsors. Thank you Endeca, Kent State, Microsoft, and Google!

A major feature of the recently released Google+ is Circles, which allows you to “share relevant content with the right people, and follow content posted by people you find interesting.”

Most people seem to look at Circles as a privacy feature — and indeed Google’s official description gives the impression that Circles exist to manage privacy based on real-life social contexts. Of course, re-sharing can result in unintended consequences, and Google even offers a warning that:

Unless you disable reshares, anything you share (either publicly or with your circles) can be reshared beyond the original people you shared the content with. This could happen either through reshares or through mentions in comments.

Privacy is a big deal, especially for Google — and particularly in the context of rolling out a new social network. Still, I’m not persuaded that privacy is the only or even the primary concern motivating the concept of social circles.

Sharing content with someone is not just about giving that person permission to see it. Sharing content with someone asserts a claim on that person’s attention. While it may be a privilege for me to have access to your content, it may be even more of a privilege for you that I allocate my scarce attention to consume it.

What if we focus on routing content to the people who would find it most interesting? Such an approach works best if all of the shared content is public with respect to permissions — that is, people post it without any expectation of privacy. Twitter demonstrates that many people are comfortable with such a sharing model. Imagine if they could learn to trust a system that optimizes (or at least attempts to optimize) the allocation of everyone’s attention. This is not an easy problem by any means, nor is it one that is likely to be solved by algorithms alone. It will take a strong dose of HCIR to get it right. But, at least in my view, optimizing the allocation of human attention is the grand challenge that everyone working with information retrieval or social networks should be striving to address.

Privacy is important, and social networks should offer simple, robust privacy controls that users understand. We all have experienced the problem of filter failure. But sharing isn’t just about privacy. Our attention is our most precious cognitive asset, both as individuals and as a society, Moreover, our attention faces ever-increasing demands as our social lives evolve in an online world relatively free of physical constraints. Social network developers would do well to pay attention…to attention.

When I left Google last December, it was an open secret that Google was developing a social networking product. Now that Google has released Google+, I am at liberty to share my personal impressions.

Let’s start with the clear wins.

• Impressive launch. Google has certainly learned its lesson from the past launches of Wave and Buzz. Google+ is unambiguously opt-in — no one is going to complain about being ambushed. People have been begging for invites. But Google is wisely releasing invites quickly enough to build critical mass. I’d say that Google has at least picked up the Quora crowd of early adopters in Silicon Valley.

• Clean design. Design lead Andy Hertzfeld (of Macintosh fame) has nailed it, leading bloggers to comment that this looks too well designed to be a Google product. Comparing Google+ to Facebook now, I’m reminded at least a little of comparisons between Facebook and Myspace. Great move for Google here.

Now let’s talk about Google’s three big features here: Circles, Sparks, and Hangouts.

• Circles. Straight out of Paul Adams’s presentation of social networking (which he created before he left Google for Facebook), the idea is simple: a person doesn’t have a single group of friends, but rather several groups that tend are mostly disjoint. Through Circles, Google+ makes this soft partitioning of the social space a core design principle. You add people to one or more circles, follow the stream of activity from a circle, and share with circles. It’s great in theory. But in practice it creates friction, especially for people trained on Facebook. There’s a trade-off between simplicity and expressive power, and Google is placing a strong bet on how users will make this trade-off.  I’m inclined to agree with Yishan Wong that “the sorting of friends into buckets (friend lists) is something that only nerds do”. Given Google’s deep expertise in machine learning, I’m expecting Google to reduce this friction by give users intelligent suggestions. Full disclosure: my colleagues at LinkedIn built InMaps, which infers communities from your social network.

• Sparks. The tagline for Sparks is “For nerding out. Together.” It feels like a positioning designed by Googlers for Googlers– you can see promotional videos here and here. I haven’t seen much talk about Sparks, and what little commentary I’ve seen is less than gushing. I’ve experimented with it a bit from a consumption side, and I confess I’m underwhelmed. Perhaps it’s a chicken-and-egg problem — Sparks will only be useful if users populate their profiles with interests, but right now users have no incentive to do so. If Sparks is Google’s attempt to make Reader more social, there’s still a ways to go. Full disclosure: LinkedIn has its own approach to social news, LinkedIn Today, which seems to be doing something right.  

• Hangouts. In plain English, Hangouts are group video chat embedded in a social network. Which sounds a lot like what Facebook is rumored to be releasing this week through a partnership with Skype. Which in turn was just acquired by Microsoft. Will Apple join the party too by implementing group chat in FaceTime? Competitive dynamics aside, this is a very cool feature that hopefully won’t devolve into Chatroulette. Nothing to, um, disclose here.

But the $64B question is whether all this will matter. Can Google+ sustainably co-exist with Facebook? Will people use both services — and, if so, how will they allocate their attention between them? Or is the success of Google+ predicated on displacing Facebook? Or Twitter? Either of those would certainly qualify as a Big Hairy Audacious Goal.

Like Fred Wilson, I’m rooting for Google+ to succeed — but even Fred notes that he would not be able to get his family on Google+, as they are already happy with Facebook. It’s not clear to me what I can get *today* from Google+ that I can’t get from Facebook.

Granted, I’m not a heavy Facebook user, so I’m not the best person to ask this question. So readers, I ask you: why will or won’t you use Google+?

Last weekend I had the extraordinary privilege to attend Foo Camp, an annual gathering of about 250 Friends Of O’Reilly (aka Foo). Tim O’Reilly, Sara Winge, and their colleagues have amazing friends, as you can see if you scan this unofficial list of attendees working on big data, open government, computer security, and more generally on the cutting edge of technology and culture (especially where the two overlap).

Foo Camp is an unconference, which merits some elaboration. No fees, no conference hotel (many attendees literally set up camp in the space O’Reilly provided), and no advance program aside from some preselected 5-minute Ignite presentations. Attendees proposed and organized sessions, merging and re-arranging them to optimize for participation. It was a bit chaotic (especially the mad rush after dinner to secure session slots), but very effective.

The minimalist format brought out the best in participants.

For example, I am passionate about (i.e., against) software patents, so I organized a session about them. I did a double-take when I realized that one of the participants was Pamela Samuelson, perhaps the world’s top expers on intellectual property law. I braced myself to be schooled — as I was. But she did it gently and constructively. Specifically, she pointed me to work that her colleagues Jason Schultz and Jennifer Urban were doing on a defensive patent strategy for open-source software (including a proposed license), as well as reminding me of the Berkeley Patent Survey supporting the argument that software entrepreneurs only file for patents because of real or perceived pressure from their investors. I also heard war stories from lawyers who have done pro bono work against patent trolls, reinforcing my own resolve and also reassuring me that the examples I’ve seen at close range are not isolated.

Another session asked whether we are too data driven in our work. What was notable is that this session included participants from some of the largest internet companies debating some of the must fundamental ways in which we work, e.g., do we actually learn from data or do we engage in assault by data to defend preconceived positions (cf. argumentative theory). Like all of the conference, the discussion was under “frieNDA”. so I’m being intentionally vague on the specifics. But it was refreshing to see candid admission that all of us know and have experienced the dangers of manipulating an audience with data, and that there are no algorithms to enforce common sense and good faith.

I won’t even try to enumerate the sessions and side conversations that excited me — topics included privacy, the future of publishing, a critical analysis of geek culture, and irrational user behavior. I missed the session on data-driven parenting, though others have pointed out to me that you can only learn so much if you don’t have twins and perform A/B tests. The best summary is intellectual diversity and overstimulation. If you’d like to get a general sense of the discussion, check out the #foocamp tweet stream. I also recommend Scott Berkun’s post on “What I learned at FOO Camp“.

As someone who organizes the occasional event, I’m intrigued by the unconference approach — especially now that I’ve experienced it first-hand. Moreover, I feel strongly that the academic conference model needs an upgrade. But I also know that open-ended, free-form discussion sessions are not a viable alternative — indeed, a big part of Foo Camp’s success was how it inspired participants to organize sessions — and to vote with their feet to attend the worthwhile ones. And of course part of that success came from inviting active, engaged participants rather than passive spectators.

Many of you also organize events, and I’m sure that all of you attend them. I’m curious to hear your thoughts about how to make them better, and happy to share more of what I learned at Foo Camp. After all, Foo is for (inspiring) thought.

The war for talent. It’s the latest metaphor for the challenge that tech companies face as excitement is building in Silicon Valley again. Well, not really — McKinsey coined the phrase in 1997 and used it as the title of a book published four years later.

But anyone who has been trying to hire great software engineers in recent months knows how hard it is to do so. Particularly for folks like me who are trying to hire data scientists — apparently there’s a national shortage. This is nothing new — as Joel Spolsky noted in a 2006 post, “the great software developers, indeed, the best people in every field, are quite simply never on the market.”

I’m not an expert (or ninja) on the subject of recruiting or employer branding in general, but I’ve seen enough of how companies go about hiring software engineers to know that we can do better. I’d like to share some of my thoughts and experiences, and I hope that you will reciprocate and share your thoughts in the comments. I’m especially interesting in hearing from folks who are at universities (aka hunting grounds) or who are involved in organizing academic conferences.

First, let’s talk about how we measure success. As Lord Kelvin famously said, “If you can’t measure it, you can’t improve it.” I’m not going to talk about how to handle active candidates — that’s a filtering problem which, in my opinion, is much more tractable. For example, see what Joel has to say about interviewing developers. Rather, I’m concerned with the challenge of discovering qualified passive candidates and converting them into active ones. Hence, I propose we make our metric the number of qualified applicants.

The baseline strategy is sourcing, i.e. have sourcers or hiring managers scour the world for qualified candidates (there’s an app for that), entice them with your best recruiting pitch, and then go hog wild on the folks who respond. The success of this strategy depends mainly on the rate at which you, your sourcers, or your hiring managers find qualified candidates — which in turn may split into the two subtasks of finding candidates and filtering them — and the conversion rate for the qualified candidates you find. Since the best candidates are often happy in their current positions, sourcing passive candidates requires a lot of work and a thick skin for rejection.

What are other ways to attract qualified passive candidates? Here are a few, with examples from my experience at LinkedIn:

• Hosting events. Last week at LinkedIn, we hosted CMU professor Christos Faloutsos, who delivered a fantastic talk on “Mining Billion Node Graphs” — a topic we thought interesting enough to justify opening up the talk to the general public. We had a few hundred guests, many of whom are precisely the kinds of folks we are trying to hire. Even more people watched the live stream online or will watch the video when we post it to YouTube (coming soon — stay tuned!). While this was not a recruiting event (we did not even announce that we are hiring), it was a great opportunity to associate LinkedIn with the hard computer science problems we solve on a daily basis.
• Sponsoring events. Sponsorship is tricky — if you’re not careful, you spend a lot of money for a glorified display ad. Sometimes sponsorship offers speaking slots as part of the package, but audiences are rightfully skeptical of speakers who have paid for their slots — especially at conferences that charge hefty fees for attendance. But sometimes sponsorship works. For example, LinkedIn’s was a sponsor of the O’Reilly Strata Conference, and the perks of sponsorship complemented our earned speaker slots, helping us bring enormous visibility to our data scientist team and its recent innovations like InMaps (we has a booth there to print attendees’ InMaps) and Skills (which launched during the conference). While Strata generated few direct leads, it left a lasting impression in the big data community, and I regularly hear candidates refer to it.
• Participating in events. As the Beatles tell us, money can’t buy you love. If you want to make an (positive) impression at a conference, you have to contribute people and ideas. This is especially true at academic conferences, where attendees quickly throw out the the extra weight in their tote bags and focus on the conference’s content and professional networking opportunities. It’s great if you are Microsoft with a team of close to a thousand researchers and can dominate a conference like SIGIR. But smaller companies can still make a strong impression on researchers — and especially on students who may be looking for internships or full-time positions — by taking an active role at conferences. The traditional approach is to submit papers to the main conference track — but other avenues include tutorials, workshops, and industry events. Such participation is often invited, but such invitations are in turn earned by cultivating relationships with researchers — especially the ones who find themselves on organizing committees.
• Contribute to open source projects. The Search, Network, and Analytics (SNA) team at LinkedIn contributes frequently to open-source projects and publicizes some of its work at http://sna-projects.com/. Open source projects are a great way to earn the respect of engineers who value source over PowerPoint. Especially when your employees include committers to key technologies like Hadoop. Moreover, open-source projects are social communities, so contributing to them offers opportunities for employees to interact with potential hires.
• Social media. By now, I’d like to think that marketers understand social media to simply be another set of marketing channels. But I think the territory is still pretty new for employers. Here is a simple suggestion: encourage (but do not try to force) employees to express themselves professionally online. Enforce the standard non-disclosure rules, of course, but don’t try to manage their voices. Authenticity speaks for itself — for example, look at what Adam Nash says about LinkedIn on his personal blog. Or my own posts here. Engineers don’t read press releases or  corporate blogs, but they do pay attention to their peers. And there’s nothing unique about blogs — the same principle applies to platforms like Twitter, Facebook, Quora, and of course LinkedIn. Not all employees enjoy being online extroverts, but those that do not only act as brand ambassadors, but also are likely to eventually strike up conversations with passive candidates about employment opportunities.

Finally, don’t forget measure the results of these efforts! Some activities generate leads directly, in which case you can make an apples-to-apples comparison of their results and costs with the baseline strategy of sourcing. It’s harder to measure the longer-term effect of efforts to raise visibility, but you can at least ask candidates if they are aware of those efforts — after all, efforts to raise visibility should be visible to candidates! You can also ask candidates if those efforts were a factor in their decision to apply. These measures aren’t perfect, but they are a lot better than nothing, especially when you’re trying to decide how best to invest limited resources.

Of course, even an optimal strategy can’t substitute for offering a combination of interesting work, competitive compensation, and a work hard / play hard culture. As with all marketing efforts, you need to start with a great product. But great products don’t sell themselves: you need to invest in a combination of outbound and inbound marketing to have a fighting chance in the war for talent. Good luck! And, in case you didn’t notice, we’re hiring!

I thought I’d prepare for the discussion by sharing my thoughts here. Perhaps some of you will even be kind enough to add your own ideas, which I promise to share with the reading group.

I encourage you to read the paper, but here’s a summary of its results:

• A user’s influence on Twitter is the extent to which that user can cause diffusion a posted URL, as measured by reposts propagated through follower edges in Twitter’s directed social graph.
• The best predictors of future total influence are follower count and past local influence, where local influence refers to the average number of reposts by that user’s immediate followers, and total influence refers to average total cascade size.
• The content features of individual posts do not have identifiable predictive value.
• Barring a high per-influencer acquisition cost, the most cost-effective strategy for buying influence is to target users of average influence.

Let’s dive in a bit deeper.

The definitions of influence and influencers are, by the authors’ own admission, narrow and arbitrary. There are many ways one could define influence, even within the context of Twitter use. But I agree with the authors that these definitions have enough verisimilitude to be useful, and their simplicity facilitates quantitative analysis.

It’s hardly surprising that past influence is a strong predictor of future influence. But it might seem counterintuitive that, for predicting future total influence, past local influence is more informative than past total influence. The authors suggest the explanation that most non-trivial cascades are of depth 1 — i.e., total influence is mostly local influence. But at most that would make the two features equally informative, and total influence should still be a mildly better predictor.

I suspect that another factor is in play — namely, that the difference between local influence and total influence reflects the unpredictable and rare virality of the content (e.g., a random Facebook Question generated 4M votes). If this hypothesis is correct, then past local influence factors out this unpredictable factor and is thus a better predictor of both future local influence and future total influence.

I’m a bit surprised that follower count supplies additional informative value beyond the past local influence; after all, local influence should already reflect the extent to which the followers are being influenced. It’s possible that past influence lags the follower count, since it does not sufficiently weigh the potential contributions of more recent followers. But another possibility is one analogous to the predictive value of past local vs. global influence: past local influence may include an unpredictable content factor which follower count factors out.

Of course, I can’t help suggesting that TunkRank might be a more useful indicator than follower count. Unfortunately the authors don’t seem to be aware of the TunkRank work — or perhaps they preferred to restrict their attention to basic features.

I’m not surprised by the inability to exploit content features to predict influence. If it were easy to generate viral content, everyone would do it. Granted, a deeper analysis might squeeze out a few features (like those suggested in the Buddy Media report), but I don’t think there are any silver bullets here.

Finally, the authors consider the question of designing a cost-effective strategy to buy influence. The authors assume that the cost of buying influence can be modeled in terms of two parameters: a per-influencer acquisition cost (which is the same for each influencer) and a per-follower cost for each influencer. They conclude that, until the acquisition cost is extremely high (i.e., over 10,000 times the per-follower cost), the most cost-efficient influencers are those of average influence. In other words, there’s no reason to target the small number of highly influential users.

The authors may be arriving at the right conclusion (Watts’s earlier work with Peter Dodds, which the paper cites, questions the “influentials” hypothesis), but I’m not convinced by their economic model of an influence market. It may be the case that professional influencers are trying to peddle their followers’ attention on a per-follower basis — there are sites that offer this model.

But why should anyone believe that an influencer’s value is proportional to his or her number of followers? The authors’ own work suggests that past local influence is a more valuable predictor than follower count, and again they might want to look at TunkRank.

Regardless, I’m not surprised that a fixed per-follower cost makes users with high follower counts less cost-effective, as I subscribe to its corollary: as a user’s follower count goes up, the per-follower value diminishes. I haven’t done the analysis, but I believe that the ratio of a user’s TunkRank to the user’s follower count tends to go down as a user’s follower count goes up. A more interesting research (and practical) question would be to establish a correctly calibrated model of influencer value and then explore portfolio strategies.

In any case, it’s an interesting paper, and I look forward to discussing it with my colleagues next week. Of course, I’m happy to discuss it here in the meantime. If you’re in my reading group, feel free to chime in. And you’re not in you’re not in my reading group, consider joining. We do have openings.

I like Google…

I’ve been a regular Google user since the day I first discovered its existence in 1999. Indeed, I’ve consistently found Google to be the most useful service on the web. That’s not love, but it’s a very strong +1.

Moreover, I’d say that my preference for Google is an informed one. I’ve given all of the major search engines a fair chance, and even tried a fair number of obscure ones. They all have their strengths, but none have delivered enough utility to me to justify the cognitive load of using more than one search engine for the open web.

…but I don’t need Google.

Nonetheless, I know that, if Google disappeared tomorrow or became inconvenient to access, I’d be content with one of its competitors. I have no particular investment in Google beyond brand loyalty.

Actually, that’s not entirely true. I could easily walk away from Google search, but I’d be apoplectic if I suddenly lost access to my Gmail account — much as if I lost access to my LinkedIn or Twitter accounts. Indeed, Gmail is the only way in which Google has me locked in, but I don’t see my Gmail account as entangled with my access to Google’s other services.

Perhaps that not a bug but a feature: after all, Google trumpets the virtues of “open” and the portability of user data (including Gmail) through the Data Liberation Front. Nonetheless, it’s no secret that Google has a major case of Facebook envy. And if rumors hold, Google is now making the success of its social strategy a major component in all employee compensation.

Social is Give to Get.

Google critics often assert that Google doesn’t get social. But I think the problem isn’t so much with what Google gets as what it gives. When it comes to social, you have to give to get. That is, to get data and engagement, you have to provide social utility.

To start off, Google would love to know who you are. That’s why it developed Google Profiles in 2007. People are more than willing to provide data about who they are, as proven by the hundreds of millions of people who create profiles on Facebook and LinkedIn. Perhaps Google was a little bit late to the game. More likely, people didn’t see enough utility in creating Google profiles. Facebook, on the other hand, helps people be found by their friends and family in a context designed for social interaction. LinkedIn offers people the opportunity to be found by people who can help you professionally: colleagues, classmates, potential employers, etc. Google didn’t give people much reason to invest effort — in fact it seems to treat Profiles as a dumping ground populated by Google’s other products, rather than valuable piece of online real estate embedded in a living social context. Not surprisingly, users invest their efforts elsewhere.

Google would also love to know where you are and where you’ve been — that’s why Google created Latitude in 2009. Moreover, Google developed this pioneering location-based service as a complement to Google Maps, perhaps the best product Google has produced outside of search. Given it’s dominance in mapping services, directions, and local search, Google should be the leader of all things local. And yet, while Latitude has flopped, Foursquare — which launched in the same year as a tiny startup after Google acquired and shut down its previous incarnation– succeeded in defining location-based services as a category. Before Foursquare, the idea of a service tracking your location was one that most of us associated with Lo-Jack and Big Brother — if not with modern totalitarian regimes. Yet, by making a game out of “checking in” to venues, Foursquare inspired its users to willingly — and eagerly! — share and publish their whereabouts. It’s unclear whether this model will create sustained interest (cf. Mark Watkins’s analysis at ReadWriteWeb), but Foursquare’s success thus far is predicate on its offers social utility in exchange for data and attention.

Of course, Google also wants to know what you like. That’s why Google developed SearchWiki (RIP), Hotpot (now merged into Places), and most recently +1. As Amazon, Facebook, Netflix, and Yelp have demonstrated, people aren’t shy about sharing their opinions publicly, given the right social context and utility. Unfortunately, Google seems to struggle with that last part. Google embedded SearchWiki in the non-social context of search — and has launched +1 the same way. It’s not at all clear what users would gain by going out of their flow to annotate search results. Hotpot may simply be a case of too little, too late — people are already trained to go to Yelp and Facebook Fan pages for subjective information about service businesses. Overall, Google has not given users a reason to believe there is significant return on their investment in sharing opinions.

Collecting Data Doesn’t Count.

Of course Google is able to collect a significant amount of data about users’ identities through their search history, cookies, browser toolbars, and purchase history (if they use Google Checkout). Indeed, it is Google inference of user intent in search queries that has allowed Google to become the poster child of online advertising.

But collecting data is not the same as having the user volunteer it. Most users have a transactional relationship with Google, tolerating data collection and advertising in exchange for a free service. Google wants more — it wants users to invest in identities associated with their Google accounts. But Google doesn’t seem to undertand that users don’t make these investments unless their receive some social or professional utility in return.

If it’s true that Larry Page is making “social” Google’s top OKR, then I hope for the sake of my former colleagues that Google has learned from its past experiments.

Note: This post was written by Scott Nicholson, a Senior Data Scientist at LinkedIn. Scott is data and modeling geek with a passion for startups, product and user experience. His work at LinkedIn focuses on analyzing and improving user engagement and monetization.

I’m happy to report back on my experience at the Data 2.0 conference, an event organized by midVentures and targeted at entrepreneurs building products to leverage the dramatic increase in publicly and privately collected data. The conference has four main themes: what data is available, how to obtain data, how to store and access data, and how to create value from data products. For data nerds or hackers, the conference offered a delightful stream of  “you know what would be cool…” ideas.

The morning started off on a strong foot with a talk by Vivek Wadhwa on how data is going to define the next generation of successful startups in a new information age. He observed the increasing online access to data that has previously been restricted to offline access (or no access at all). He also emphasized the importance of  new sources of data, such as medical records and genome data. We need to think of social use of data beyond Twitter, Facebook and LinkedIn: for example, genome data will allow us to connect to each other in ways that helps us better understand our similarities and differences. Meanwhile, some existing data sources will become increasingly open and available to all. Wadhwa stressed the importance of leveraging the open sources of federal, state and local government data to come up with solutions to the existing closed and clunky legacy systems that governments used to generate data reports (a pity that data.gov and related programs may be defunded — DT).

The morning keynote segued nicely into the panel on open data sources. Jay Nath, Director of CRM for the city of San Francisco, noted that, while many applications are using government data and APIs, they mostly address consumer convenience (e.g., public transit apps) rather than government efficiency.  Panelists agreed that government employees have few incentives to take risks by using new technology: legacy systems might be expensive, inflexible and inefficient, but they do perform their limited function. Alluding to Eric Ries’s idea of a “lean startup“, Nath suggested the concept of a “lean government” that lowered costs, sped up its operations, and avoided procurement processes by using open source technology — all in the context of providing services to its citizens.

The inspiring mid-day keynote by former Amazon Chief Scientist Andreas Weigend took a different perspective from the morning sessions: he focused on the how data sharing can provide tangible value to end-users, even resulting in significant behavior change. He cited products like tweeting weight scales, FitBit, and Nike + that allow people to share data about their fitness efforts, thus leading to social reinforcement for positive behaviors. I personally see this area as a great example of where data scientists and engineers can create enormous economic value and increase people’s welfare

The day also featured a various product launches and presentations. Here are a few that caught my attention:

• Micello: Google maps for indoors. They won the startup competition that was held in conjunction with the conference.
• Tropo: API for voice calls and SMS
• DataStax Brisk: Technology unifying Hadoop, Hive & Cassandra. A new Hadoop distribution powered by Cassandra.
• Neer: always-on location awareness app from Qualcomm. Privately share location with groups and families.
• Heritage Health Prize: $3MM prize for predictive modeling around who will require hospitalization (a follow-up on their announcing the prize at Strata)

Overall, it was great to see hundreds of people exploring innovations and opportunities to use data to improve business, technology and society.

Last October, a certain blogger said:

LinkedIn needs to implement some kind of concept extraction to provide a useful topic facet (something I’d also love to see for their regular people search). This is a challenging information extraction problem, especially for the open web, but I also know from experience that it is tractable within a domain. Given LinkedIn’s professional focus, I believe this is a problem they can and should tackle.

Shortly after writing that post, I interviewed at LinkedIn and met Pete Skomoroch, who showed me an early preview of the work his team was doing to make skills a facet for exploring the space of LinkedIn member profiles. That demo made a strong impression on me, giving me a taste of the great products LinkedIn’s data scientists were working on in the lab.

And now I’m delighted that everyone can try out the beta launch of LinkedIn Skills which was announced today at O’Reilly’s Strata 2011 conference on Big Data.

As Pete says in his blog post:

If you search for a particular skill, we’ll surface key people within that community, show you the top locations, related companies, relevant jobs, and groups where you can interact with like minded professionals.  You’ll also be able to explore similar skills and compare their growth relative to each other.

I encourage you to check it out — whether you’re looking for experts on Hadoop, cheese, or anything else! It’s a beta, so I’m sure you’ll find rough edges; but I hope it gives you a sense of how LinkedIn’s data can enable a incredibly powerful and useful exploratory search experience.

No forward-looking statements, except to say that it only gets better from here!

But I have personal reasons to be interested in Quora. One of their lead engineers, Albert Sheu, was a star intern of mine at Endeca. And Quora raises lots of interesting questions about search, user experience, knowledge management, and online reputation. How could I resist?

I see three potential reasons to use Quora:

1. Objective question answering.
2. Subjective question answering.
3. Community participation.

Let’s consider how Quora fares today on each of these, and where it might go.

1. Objective question answering.

When I blogged about Quora early last year, I said that “I don’t see Quora as a knowledge base of first resort–except possibly to learn more about software startups.” Despite Quora’s recently growth surge, I am not ready to change my answer significantly — I find that Quora’s topics are pretty sparse when I stray from its Silicon Valley focus.

Within that focus, Quora is nailing it. For example, I was curious to learn whether someone who signed a non-compete agreement outside of California was still subject to it if he or she moved to California, where such contracts are legally unenforceable. Not surprisingly, non-compete agreements are a topic on Quora, and I quickly found a useful answer from a lawyer.

But for most objective questions, I’m still turning to Google and Wikipedia — or to Twitter if both of those fail and I am willing to ask a favor of my followers (who kick ass!). Sometimes Google will take me to Quora, but I can’t imagine Quora will succeed through this flow in the long term.

2. Subjective question answering.

I see subjective question answering as Quora’s strongest suit. A good subjective question on Quora — often a “why” question — generates a diverse collection of interesting and informed perspectives. A couple of good example are “Why did Google Wave fail to get significant user adoption?” and “What is lacking in social networking now?“.

Again, these questions are well within the Silicon Valley focus, but I could see Quora extending this value proposition to other verticals if it can grow the communities successfully. And I certainly don’t see myself going to Google or even Twitter to get useful answers to subjective questions. The closest is Topsy, and Quora has the advantage of being explicitly organized around questions and topics.

3. Community participation.

Is Quora a question answering site or a social network? Quora users and employees have tried to answer that question (on Quora, natch), but I’m not sure Quora’s converged enough for anyone to know. What is clear is that Quora emphasizes conversation, making it more like a blog or wiki than an answers site.

Conversation certainly engages its participants. But it also raises the cost of participation. One of the things I love about Google is that it gives me information without unnecessary overhead. When I want conversation, I go to social venues like Twitter.

Perhaps Quora can be both a question answering site and a social network. But I suspect it will need to choose. Most people don’t have the time or patience to participate in additional communities, so question answering is the easier sell to a mass audience. But the participation is what makes Quora especially distinctive today. Perhaps it’s a question of quality vs. quantity.

So, quo vadis, Quora? I suppose I’ll have to check Quora (or Cwora) to find the answers.

I hope everyone I’ve invited is enjoying Quora. But I also hope you take it upon yourselves to circulate more invitations to those who want them. Any Quora user can send out invites — that’s how these viral sites work. If you’re still looking for an invite, I urge you to use Twitter or some other broadcast mechanism to request it. As of today, I will stop responding to Quora invite requests through my blog or email, and I will also delete comments requesting them. I am sorry if this is a bit harsh, but I hope folks understand.

But perhaps you’d like to make a career out your jones for big data. Luckily for you, some of the hottest companies around are hiring data scientists!

Of course, those jobs aren’t for everyone. To get an idea of the necessary qualifications, I suggest you read the answers on Quora for “How do I become a data scientist?” to get an idea of the requisite math and computer science skills. I’m also a fan of Hilary Mason‘s definition which was cited in Ryan Kim’s “Wanted: Data Scientists to Turn Information Into Gold“: a data scientist is someone who can obtain, scrub, explore, model and interpret data, blending hacking, statistics and machine learning. You can see Hilary’s full explanation in a blog post she co-authored with Chris Wiggins, entitled “A Taxonomy of Data Science“.

If the qualifications haven’t scared you off, then it’s just a question of where you can best apply your data scientist skills. The good news is that there are a lot of different ways to make a career out of working with big data. Here are some suggestions for what to work on. I apologize in advance for taking a US-centric perspective — if you’re outside the US, I can only hope that the examples have local analogs.

1) Web search.

Google, Yahoo, and Bing all collect an enormous amount of data from people’s web search activity. Google is, of course, the 800-pound gorilla, but don’t dismiss the others — even a single-digit market share is enough to derive extremely valuable insights from user activity. And, since every major search engine makes the bulk of its revenue from advertising, they all present the big-data challenges associated with computational advertising. Search is, in my view, the web’s killer app, so you can’t go wrong working on it. But temper your expectations — despite heroic efforts from various parties, it seems difficult to deliver revolutionary improvements to this field.

2) Social networking.

Here the biggest players are Facebook and Twitter, but you can find a more comprehensive list on Wikipedia. Many consider LinkedIn to be a social network, but I’ll take the liberty to discuss it in its own section. Social networks attract an outsized share of users’ attention: Facebook alone accounts for a quarter of US page views on the web! All of this user activity means a lot of data to crunch, so it’s not surprising that LinkedIn, Facebook, and Twitter are recognized as having the best data science teams. How much you’ll enjoy working at these companies will in part reflect the value (and values) you perceive in their offerings, but they are all playgrounds for data scientists.

3) Electronic commerce.

While ad-supported web search may be the killer app of the web, what opens up people’s wallets is e-commerce. Led by Amazon and eBay, e-commerce sites deserve much of the credit for turning the web from an esoteric research project into a mainstream staple. And, like their offline counterparts, e-commerce sites generate vast amounts of data from how users view and purchase products. This data drives user recommendations, merchandising campaigns, pricing strategy, and much more. If you’d like to pursue data-driven capitalism, then e-commerce may be for you. A word of caution: if you are one of a crowd of merchants selling the same products as everyone else (as opposed to a site like Etsy selling unique products), make sure you have a sustainable competitive advantage. Data science is necessary for success in e-commerce, but it may not be sufficient.

4) Digital content.

Whether its books, music, video, or apps, the long-prophesied digital convergence has arrived: almost every newly created piece of digital content is now distributed in electronic form. Here the biggest players are Amazon, Apple, and Google (particularly its YouTube subsidiary), but there is still a lot of flux as new hardware, software, and business models compete for dominance. Digital content poses two daunting challenges: the volume of published content far exceeds people’s available attention, and digital media products are experience goods than people can only evaluate after consuming them. For both of these reasons, the digital content industry depends on data scientists to help people find and discover what they like. The catch: from its advent, the digital content industry has struggled with unauthorized distribution (aka piracy), and the results of this struggle will determine which business models are viable.

5) Finance.

Money, money, money. Working in finance has always been a data-intensive business, but advances in technology have only increased the industry’s reliance on data scientists. Algorithmic trading — and high-frequency trading in particular — mean that those who can most effectively and efficiently mine financial data can derive enormous financial benefits. Finance isn’t for everyone — the hours are long, the stress is high, and the compensation is highly variable. That said, the financial upside can be quite compelling, and some even enjoy the lifestyle.

6) Public sector.

Given the libertarian leanings of the software industry, the public sector might not seem like an obvious career choice. But some of the largest repositories of data reside there–from public repositories like census data to highly classified repositories restricted to the TLAs. Better understanding of this data can improve public policy, national security, and much more. Not everyone has the temperament to deal with government bureaucracy, but those who do have the opportunity to turn big data into big public good.

7) LinkedIn.

OK, I’m being self-serving, but after all this is my blog! LinkedIn is widely recognized as being one of the top data science teams on the planet. But LinkedIn has more than just talent — it has what Pete Warden of ReadWriteWeb described in “Secrets of the LinkedIn Data Scientists” as “detailed information on millions of people who are motivated to keep their profiles up-to-date, collect a rich network of connections and have a strong desire from their users for more tools to help them in their professional lives.” Indeed, I don’t know of anyone who has a dataset that competes with the combined quantity, quality, and utility of LinkedIn’s data. Moreover, working as a data scientist at LinkedIn means helping make people more professionally successful by connecting the to opportunities, information, and of course other people. It’s a wonderful way to create value, and it doesn’t hurt to do so in the context of a profitable, rapidly growing company.

And LinkedIn recognizes the extraordinary value of data science. Don’t take my word for it — listen to LinkedIn CEO Jeff Weiner’s interview at the 2010 Web 2.0 Summit:

To wrap up, data science is more than just an opportunity to have fun and make the world a better place — it might even be how you make an honest living!

I began this year as a Noogler and leave it as a Xoogler. I hope I left Google better than I found it — I’m certainly proud of the improvements my team made to the quality of local authority pages. I also tried to infuse Google with some of the scrappy start-up culture I’d picked up at Endeca, particularly focusing on the hiring process. In information retrieval terms, I’d say that Google’s hiring process does extremely well when it comes to precision, but could use improvement in the areas of recall and efficiency. Still, I’m impressed at how well Google has maintained its quality standards as the company has grown. Finally, I couldn’t help being an extrovert: I developed warm relationships with the lead bloggers covering local search, including Andrew Shotland, David Mihm, Gib Olander, Greg Sterling, and Mike Blumenthal. Indeed, when I announced my departure, Mike wrote a really nice post about the friendship we cultivated over the past year. I hope that he continues to have such relationships with my former co-workers.

Looking back at what was on my mind when this year began, I had lots of questions around exploratory, mobile, real-time, social/collaborative search. I also wondered whether it was possible to offer more transparency in relevance ranking without losing ground in the battle against spam and black-hat SEO.

I’m as bullish as ever on the value of exploratory search:  part of why I joined LinkedIn is that a significant fraction of the site’s value comes from supporting users’ exploratory search needs. I also published a position paper at the SIGIR 2010 Workshop on Simulation of Interaction proposing the use of query performance prediction to model the fidelity of communication between user and system, thus helping HCIR researchers to simulate query refinement with standard test collections. And of course exploratory search was a major theme at the HCIR 2010 workshop, not only providing the basis for the first HCIR Challenge, but even extending to new territory with Max Wilson and David Elsweiler’s work on casual leisure searching.

As for mobile search, I’d say that 2010 has been the year of “mobile first“. Thanks to a generous gift from my former employer, I’ve become a regular user of the mobile web–and of search in particular. To my surprise, the communication bottleneck has not been screen real estate, but rather the difficulty of entering text. And innovative approaches like voice search and Swype go a long way to mitigate that difficulty.

On to real-time search. Not surprisingly, my favorite innovation in this space is LinkedIn Signal, which offers exploratory search for Twitter. I still struggle to find use cases that emphasize the “real-time” aspect of Twitter and other microblogging services, but I am convinced that the path to utility lies in tools that support organization, analysis, and exploration.

On the social/collaborative front, I’m happy to work for a company whose charter includes “supporting mediated search by linking people to people, rather than directly to information”. While the biggest event in this space in 2010 was Facebook’s introduction of the Like button, I’m not convinced that “likes” have supplanted links. I’m still looking to niche players like Topsy and Blekko to push innovation in this space.

Speaking of Blekko, they’ve made an impressive attempt to increase the transparency of relevance ranking. But, as I blogged earlier this year, I think that, at least for the time being, Google is making the right decision to keep some of its details secret. Now that web search is essentially a duopoly (at least in the US), I believe the real test of the value of transparency to users will be whether one of the two parties employs it as competitive differentiator.

What’s in store for 2011? LinkedIn CEO Jeff Weiner has a vision of using data science to provide a “Pandora for people“, and that’s a vision I’m eager to help realize. Not surprisingly, when I blogged in 2008 about where Google wasn’t good enough, two of the four areas I cited were finding jobs and find employees. Even then I recognized that LinkedIn was the best at both. But LinkedIn can be so much more, and I am looking forward to working with an incredible team and incredible data on a delightful set of information science challenges.

Happy New Year! I hope that 2011 brings you great answers — and great questions!

Consider information retrieval. Perhaps I’m dating myself, but an undergraduate computer science major, I hardly imagined that information retrieval would have much significance outside of academia. Sure, there were commercial IR systems being built in the 1980s, but it wasn’t until the late 1990s that web search brought IR to the mainstream. Today, it’s hard to imagine studying computer science without learning about IR. Sure, my career makes me a tad biased, but it is undeniable that information retrieval is one of the defining problems of our generation.

And then there are social networks. When I studied graph drawing in the 1990s, the canonical example of a social network was “Six Degrees of Kevin Bacon“. Sure, many of my peers would talk about their Erdős numbers (they were more discreet about their placement in the Tarjan graph), but the study of social networks was surely an academic pursuit. Who would imagine that, barely a decade later, a movie entitled The Social Network would be a blockbuster movie grossing $175M? Leaving aside Hollywood, social networks have become a significant part of our daily lives. Not only do Facebook, Twitter, and LinkedIn account for a large fraction of our time online, but they also affect our offline personal and professional lives.

From childhood, I’ve been interested in mathematics, computer science, and psychology. Living in an age of information retrieval and social networks means that I can apply these interests in my daily work. Today I give thanks for being born at the right place and right time, blessed with a lifetime of interesting and practical problems to solve. Happy Thanksgiving to all, and enjoy the leftovers!

Like many interface design principles, the principle of least surprise reflects the premise that software applications exist to be useful. In utility-oriented applications, surprise means distraction and delay — negatives that good designers work to avoid.

But we increasingly see applications whose main value to the user is not utility, but entertainment. Indeed, a recent Nielsen report claims that the top two online activities for Americans are social networks / blogs and games. I take the report with a grain of salt, but it seems safe to argue that people have come to expect the internet to be at least as fun as it is useful.

Even search, which would seem to be the poster child for the utility of online services, is being pressed into the service of entertainment. Max Wilson and David Elsweiler argued as much in their HCIR 2010 presentation about “casual leisure searching“. They mined Twitter to analyze a variety of scenarios where search isn’t about the use finding something, but rather about enjoying the experience. Indeed, their controversial definition of search is broad enough to include the possibility that the user does not have an information need.

Like the businessman in Antoine de St. Exupery’s Le Petit Prince, I’ve long felt that, as “un homme sérieux”, my job is delivering utility to users. Users already have lots of ways to waste time; I focus on making their productivity-oriented time more effective and efficient. I’m glad there are folks who devote their lives to making the rest of us have more fun (especially all the computer scientists who left academia for Pixar), but entertainment simply isn’t a vocation for me.

However, I’ve been coming around to the realization that fun and utility are not mutually exclusive. For example, news serves the utilitarian ideal of informing the citizenry, but many (most?) of us read news as a pleasant way to pass the time. Social networks are another example serving a similar function–perhaps with a balance that is more toward the entertainment of the spectrum but still providing genuine social utility.

A common feature of both of these examples is that users regularly return to the same site expecting the unexpected. The transient nature of news and social news feeds promises an endless supply of fresh content, produced more quickly than users can consume it. This situation is in stark contrast to those of typical web search queries, for which the results are expected to be largely static. Indeed, we may set up alerts to inform us of novel search results, but we are unlikely to regularly visit a bookmarked search results page the way we regularly visit a news or social network site.

Is novelty the only source of surprise? Novelty certainly helps, but it is not a necessity. An alternative source is randomness. I’m known people to use Wikipedia’s “random article” feature. But a more plausible place to introduce randomness is in recommendations — whether for products or content. Since recommendations are good guesses at best, a bit of randomness can help ensure that the guesses are interesting. Indeed, a SIGIR 2010 paper by Neal Lathia, Stephen Hailes, Licia Capra, and Xavier Amatriain on “Temporal Diversity in Recommender Systems” explored the use or randomness to induce diversity in recommendations and arrived at the conclusion that people don’t like being recommended the same things over and over again.

Can we generalize from these examples? I think so. For utility-oriented information needs, it is important to provide users with accurate, predictable, and efficient tools. But we can’t dismiss everything else as frivolous. Sometimes we just need to offer our users a little bit of surprise to keep it interesting.

Or, as Mary Poppins tells us: “In every job that must be done, there is an element of fun. You find the fun, and – SNAP – the job’s a game!”

I like Twitter. Yes, I know that a lot of its content is noise. But I’ve found Twitter to be a useful professional tool for both publishing and consuming information. Publishing to Twitter is the easy part: I publish links to my blog posts and occasionally engage in public conversations.

Consuming information from Twitter is more of a challenge. I follow 100 people, which is about the limit of my attention budget. I use saved searches to track long-term interests (much as I use web and news alerts), and I perform ad hoc searches when I am interested in finding out what people are saying about a particular topic.

But Twitter search is not a great fit for analysis or exploration–unless you count trending topics as analysis. Originally, the search results were simply the tweets that contained the matching tweets in order of recency. The current system sometimes promotes a few “top tweets” to the top of the results. Still, if you’d like to get a summary view, slice and dice the results, or perform any other sort of HCIR task, you’re out of luck.

Until now.

The LinkedIn Search, Network, and Analytics team–the same folks that built LinkedIn’s faceted search system and developed open-source search tools Zoie and Bobo–just introduced a service called Signal that is squarely aimed at folks like me who use Twitter as a professional tool. It is still in its infancy (in private beta, in fact), but I think it has the potential to dramatically change how people like me use Twitter. You can learn more about its architecture and implementation details here.

Signal joins the often cacophonous Twitter stream to the high-quality structured data that LinkedIn knows about its own users. For example, when I post a tweet, LinkedIn knows that I am in the software industry, work at Google, and live in New York. LinkedIn can only make this connection for people who include Twitter ids in their LinkedIn profiles, but that’s a substantial and growing population.

Signal then lets you use this structured information to satisfy analytic and exploratory information needs. For example, I can see which companies’ employees are tweeting about software patents (top two are Google and Red Hat).

Or compare what Microsoft employees are saying about Android…

…to what Google employees are saying about Android.

As you can see on the right-hand side, Signal also mines shared links to identify popular ones relative to given search–and allows you to see who has shared a particular link. This functionality is similar to Topsy, but with the advantage of allowing structured searches. Like Topsy, it wrangles the mass of retweeted links into a useful and user-friendly summary.

Signal is still very much in beta. An amusing bug that I encountered earlier today was that, due to some legacy issues in how Linkedin standardized institution names, the system decided that I was an alumnus of the Longy School of Music rather than of MIT. Fortunately, that’s fixed now (thanks, John!)–I love karaoke, but I’m not ready to quit my day job!

Also, Signal only exposes a handful of LinkedIn’s facets, which limits the breadth of analysis and exploration. I’d love to see it add a past company facet, making it possible to drill down into what a company’s ex-employees are saying about a particular topic (e.g., their ex-employer).

Finally, while Signal offers Twitter hashtags as a facet, these are hardly a substitute for a topic facet. In order to provide such a facet, LinkedIn needs to implement some kind of concept extraction to provide a useful topic facet (something I’d also love to see for their regular people search). This is a challenging information extraction problem, especially for the open web, but I also know from experience that it is tractable within a domain. Given LinkedIn’s professional focus, I believe this is a problem they can and should tackle.

Of course, Linkedin also needs to convince more of its users to join their LinkedIn accounts to their Twitter accounts–since that is their input source. But I suspect it’s mostly a matter of time and education–and hopefully the buzz around Signal will help raise awareness.

All in all, I see LinkedIn Signal as a great innovation and a big step forward for exploratory search and for Twitter. Congratulations to John Wang, Igor Perisic, and the rest of the LinkedIn search team on the launch!

If you’re a search engine junkie like me, you’ve probably heard about Blekko, a search engine that has been percolating for over two years and recently launched a private beta. If not, I encourage you to watch the TechCrunch video I’ve embedded above. You can join the beta by following them on Twitter. I did that earlier this week, and my invitation arrived via a direct message the next day.

Blekko’s main differentiating feature is that it supports “slashtags”. These aren’t the same as the Twitter microsyntax proposed by Chris Messina and named by Chris Blow. Rather, they are a way for users to “spin” their search results using a variety of filters. For example, [climate /liberal] and [climate /conservative] return very different results, because they are restricted to different sets of sites.

In addition to providing a set of curated slashtags, Blekko allows users to define their own slashtags by specifying the sets of sites to be included. There’s a social aspect here too: you can use (and follow) other users’ slashtags. Blekko also has some special slashtags that don’t act as site filters, e.g., /date shows recent results and /seo offers indexing information about web sites.

Blekko emphasizes two characteristics that I find very appealing: transparency and user control. While they do not disclose their relevance ranking algorithm, they do expose some of the information they use to compute it. More significantly, their emphasis on slashtags de-emphasizes default ranking, but rather encourages users to take more responsibility in the information seeking process. Very HCIR!

I like the concept. But I’m not sure how I feel about the execution. I have three main concerns.

First, the set of slashtags is somewhat haphazard–to be expected in a beta, but I’m not sure how it will evolve. I’d love to see a vocabulary collectively (and transparently) curated like Wikipedia, but I fear it will look more like social tagging site Delicious, which is a case study in the “vocabulary problem“. As any information scientist can tell you, managing vocabularies is hard!

Second, I’m not sure if site filters are the right model. What happens to sites with heterogeneous content? Or to sites that have one-hit wonders and therefore are unlikely to show up in any slashtags? I’d prefer to see the sites used as seeds to train classifiers that could then be applied to the entire index. Something a bit more like what Miles Efron implemented in this research–only on a much larger scale and applied at a page rather than site level.

Third, I think there’s a third ingredient that is essential to complement transparency and user control: guidance. As a user, I need to know what slashtags would lead me to interesting results, and ideally I’d want some kind of preview to make exploration as low-cost as possible.

I know I’m asking for a lot–especially from an ambitious startup that has just launched its private beta. But I think the stakes are high in this space, and going easy on a newcomer is no favor. I offer the tough love of a critic who would really like to see this kind of vision succeed.

If you are are interested in attending and haven’t already registered, please let me know as soon as possible via email or Twitter (speaking of which, follow the #hcir2010 hash tag). We’re making the remaining slots available to the community on a first-come, first-serve basis.

Google user experience researcher Dan Russell will be delivering this year’s keynote on “Why is search sometimes easy and sometimes hard? Understanding serendipity and expertise in the mind of the searcher“.

Here is the list of accepted papers:

Oral Presentations

• VISTO: for Web Information Gathering and Organization
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• Time-based Exploration of News Archives
  Omar Alonso (Microsoft Corporation), Klaus Berberich (Max-Planck Institute for Informatics), Srikanta Bedathur (Max-Planck Institute for Informatics), and Gerhard Weikum (Max-Planck Institute for Informatics)
• Combining Computational Analyses and Interactive Visualization to Enhance Information Retrieval
  Carsten Goerg, Jaeyeon Kihm, Jaegul Choo, Zhicheng Liu, Sivasailam Muthiah, Haesun Park, and John Stasko (Georgia Institute of Technology)
• Impact of Retrieval Precision on Perceived Difficulty and Other User Measures
  Mark Smucker and Chandra Prakash Jethani (University of Waterloo)
• Exploratory Searching As Conceptual Exploration
  Pertti Vakkari (University of Tampere)
• Casual-leisure Searching: The Exploratory Search Scenarios that Break our Current Models
  Max L. Wilson (Swansea University) and David Elsweiler (University of Erlangen)

HCIR Challenge Reports

• Search for Journalists: New York Times Challenge Report
  Corrado Boscarino, Arjen P. de Vries, and Wouter Alink (Centrum Wiskunde and Informatica)
• Exploring the New York Times Corpus with NewsClub
  Christian Kohlschütter (Leibniz Universität Hannover)
• Searching Through Time in the New York Times
  Michael Matthews, Pancho Tolchinsky, Roi Blanco, Jordi Atserias, Peter Mika, and Hugo Zaragoza (Yahoo! Labs)
• News Sync: Three Reasons to Visualize News Better
  V.G. Vinod Vydiswaran (University of Illinois), Jeroen van den Eijkhof (University of Washington), Raman Chandrasekar (Microsoft Research), Ann Paradiso (Microsoft Research), and Jim St. George (Microsoft Research)
• Custom Dimensions for Text Corpus Navigation
  Vladimir Zelevinsky (Endeca Technologies)
• A Retrieval System Based on Sentiment Analysis
  Wei Zheng and Hui Fang (University of Delaware)

Research Posters

• Improving Web Search for Information Gathering: Visualization in Effect
  Anwar Alhenshiri, Carolyn Watters, and Michael Shepherd (Dalhousie University)
• User-oriented and Eye-Tracking-based Evaluation of an Interactive Search System
  Thomas Beckers and Norbert Fuhr (University of Duisberg-Essen)
• Exploring Combinations of Sources for Interaction Features for Document Re-ranking
  Emanuele Di Buccio (University of Padua), Massimo Melucci (University of Padua), and Dawei Song (The Robert Gordon University)
• Extracting Expertise to Facilitate Exploratory Search and Information Discovery: Combining Information Retrieval Techniques with a Computational Cognitive Model
  Wai-Tat Fu and Wei Dong (University of Illinois at Urbana-Champaign)
• An Architecture for Real-time Textual Query Term Extraction from Images
  Cathal Hoare and Humphrey Sorensen (University College Cork)
• Transaction Log Analysis of User Actions in a Faceted Library Catalog Interface
  Bill Kules (The Catholic University of America), Robert Capra (University of North Carolina at Chapel Hill), and Joseph Ryan (North Carolina State University Libraries)
• Context in Health Information Retrieval: What and Where
  Carla Lopes and Cristina Ribeiro (University of Porto)
• Tactics for Information Search in a Public and an Academic Library Catalog with Faceted Interfaces
  Xi Niu and Bradley M. Hemminger (University of North Carolina at Chapel Hill)

Position Papers

• Understanding Information Seeking in the Patent Domain and its Impact on the Interface Design of IR Systems
  Daniela Becks, Matthias Görtz, and Christa Womser-Hacker (University of Hildesheim)
• Better Search Applications Through Domain Specific Context Descriptions
  Corrado Boscarino, Arjen P. de Vries, and Jacco van Ossenbruggen (Centrum Wiskunde and Informatica)
• Layered, Adaptive Results: Interaction Concepts for Large, Heterogeneous Data Sets
  Duane Degler (Design for Context)
• Revisiting Exploratory Search from the HCI Perspective
  Abdigani Diriye (University College London), Max L. Wilson (Swansea University), Ann Blandford (University College London), and Anastasios Tombros (Queen Mary University London)
• Supporting Task with Information Appliances: Taxonomy of Needs
  Sarah Gilbert, Lori McCay-Peet, and Elaine Toms (Dalhousie University)
• A Proposal for Measuring and Implementing Group’s Affective Relevance in Collaborative Information Seeking
  Roberto González-Ibáñez and Chirag Shah (Rutgers University)
• Evaluation of Music Information Retrieval: Towards a User-Centered Approach
  Xiao Hu (University of Illinois at Urbana Champaign) and Jingjing Liu (Rutgers University)
• Information Derivatives: A New Way to Examine Information Propagation
  Chirag Shah (Rutgers University)
• Implicit Factors in Networked Information Feeds
  Fred Stutzman (University of North Carolina at Chapel Hill)
• Improving the Online News Experience
  V. G. Vinod Vydiswaran (University of Illinois) and Raman Chandrasekar (Microsoft Research)
• Breaking Down the Assumptions of Faceted Search
  Vladimir Zelevinsky (Endeca Technologies)
• A Survey of User Interfaces in Content-based Image Search Engines on the Web
  Danyang Zhang (The City University of New York)

You can also download the full proceedings here.

So there’s no question that community question answering is hot. The question is why? In particular, is community question answering a step forward or backward relative to today’s search engines, or is it something different?

Regarding Facebook Questions, Jason Kincaid writes in TechCrunch:

Given its size, it won’t take long for Facebook to build up a massive amount of data — if that data is consistently reliable, Questions could turn into a viable alternative to Google for many queries.

That’s a big if.  But I think the bigger caveat is the vague quantifier “many”. The success of community question answering services will depend on how these services position themselves relative to users’ information needs. Anyone arguing that these services can or should replace today’s web search engines might want to consider the following examples of information needs that are typical of current search engine use:

• How do I get an iPhone case?
• Who sings the “choco latte” song?
• What movies are playing in my neighborhood?
• How do I get to Boston from New York?
• What is the best selling netbook?
• Who offers the best cell phone reception in New York?
• What was the score in the North Korea – Portugal game?

I hope I don’t have to keep going to convince you that web search engines have earned their popularity by serving a broad class of information needs (i.e., answer lots of questions)–and that’s without even using the wide variety of personalized and social features that web search engines are rapidly developing.

The common thread in the above questions is that they focus on objective information. In general, such questions are effectively and efficiently answered by search engines based on indexed, published content (including “deep web” content made available to search engines via APIs). There’s a lot of work we can do to improve search engines, particularly in the area of supporting query formulation. But it seems silly and wasteful to route such questions to other people–human beings should not be reduced to performing tasks at which machines excel.

That said, I agree with Kincaid that there are many information needs that are well addressed by  community question answering. In particular:

• Questions for which point of view is a feature, not a bug. Review sites succeed when they provide sincere, informed personal reactions to products and services. Similarly, routing questions to people makes sense either when we care about the answerer’s a point of view. For some questions, I want the opinion of someone who shares my taste (which is what Hunch is pursuing with its “taste graph“). For others, I want a diversity of expert opinions–for which I might turn to Aardvark (which tries to route questions to topic experts), Quora (where people follow particular topics), or LinkedIn Answers. Over time, the answers to many such questions can be published and indexed–and indeed some answers sites receive a large share of their traffic from search engines.
• Niche topics. As much as web search as improved information accessibility for the “long tail” of published information, the effectiveness of web search can be highly variable for the most obscure information needs. Moreover, this effectiveness depends significantly on the user: some people are better at searching than others, especially in their areas of domain expertise. Social search can help level the playing field. Much as Wikipedia has surfaced much of the expertise at the head of the information distribution, community question answering can help out in the tail.
• Community for its own sake. Even in cases where search engines are more effective and efficient than community question answering services, some people prefer to participate in a social exchange rather than to conduct a transaction with an impersonal algorithm. Indeed, researchers at Aardvark found that many of the questions posed through their service (pre-acquisition) could be answered successfully using Google. I’ll go out on a limb and assume that Aardvark’s users were early technology adopters who are quite conversant with search engines–but in some case chose to use a social alternative simply because they wanted to be social.

Conclusions? Community question answering may be overhyped right now, but it isn’t a fad. There are broad classes of subjective information needs that require a point of view, if not a diversity of views. And even if much of the use of community question answering sites is mediated by search engines indexing their archives, there will always be a need for fresh content. I also believe that social search will continue to be valuable for niche topics, since neither search engines nor searchers will ever be perfect.

But I think the biggest open question is whether people will favor community question answering simply to be social. I conjecture that, by very publicly integrating community question answering into is social networking platform, Facebook is testing the hypothesis that it can turn information seeking from a utilitarian individual task into an entertaining social destination. Given Facebook’s highly engaged user population, we won’t have to wait long to find out.

But of course she did a lot more than defend Cranfield. She offered a comprehensive and fascinating history of the Cranfield paradigm, starting with the Cranfield 1 experiments in the late 1950s which evaluated manual indexing systems.

Most importantly, she defined the Cranfield paradigm as defining a metric that reflects real user model and building the collection before the experiments to prevent human bias and enable reusability. As she noted, this model does not say anything about only returning a ranked list of ten blue links–which is what most people (myself included) associate with the Cranfield model. Indeed, she urged us to think outside this mindset.

I loved the presentation and found the history enlightening (though Stephen Robertson corrected a few minor details). Still, I wondered if she was defining the Cranfield paradigm so broadly as to co-opt all of its critics. But I think the clear dividing line between Cranfield and non-Cranfield is whether user effects are something to avoid or embrace. I perceive the success of Cranfield as coming in large part from its reduction of user effects. But I think that much of the HCIR community sees user effects as precisely what we need to be evaluating for information seeking support systems.

In any case, it was a great keynote, and Donna promises me she will make the slides available. Of course I’ll post them here. In the mean time, check out Jeff Dalton’s notes on his great blog and the tweets at #sigir2010.

• “Query Quality: User Ratings and System Predictions” by Claudia Hauff, Franciska de Jong, Diane Kelly, and Leif Azzopardi offered the startling (to me at least) result that human prediction of query difficulty did not correlate (or at best correlated weakly) to post-retrieval query performance prediction (QPP) measures like query clarity. I talked with Diane about it, and I wonder how strongly the human prediction, which was pre-retrieval, would correlate to human assessments of the results. I also don’t know how well the QPP measures she used apply to web search contexts.
• Which leads me to the next poster I saw, “Predicting Query Performance on the Web” by Niranjan Balasubramanian, Giridhar Kumaran, and Vitor Carvalho. They offered what I saw as a much more encouraging result–namely that QPP is highly reliable when it returns low scores. In other words, a search engine may wrongly believe that it did well on a query, but it is almost certainly right when it thinks it failed. This certainty on the negative side is exactly the opening that HCIR advocates need to offer richer interaction for queries a conventional ranking approach recognizes its own failure. While some of the specifics of the authors’ approach are proprietary (they perform regression on features used by Bing), the approach seems broadly applicable.
• Next I saw “Hashtag Retrieval in a Microblogging Environment” by Miles Efron. He provided evidence that hashtags could be an effective foundation for query expansion of Twitter search queries, using a language model approach. The approach may generalize beyond hashtags, but hashtags do have the advantage of being highly topical and relatively unambiguous by convention.
• “The Power of Naive Query Segmentation” by Matthias Hagen, Martin Potthast, Benno Stein, and Christof Brautigam suggested a simple approach for segmenting long queries into quoted phrases: consider all segmentations and, for a given segmentation, compute a weighted sum of the Google ngram counts for each quoted phases, the weight of a phrase of length s being s^s. I don’t find the weighting particularly intuitive, but the accuracy numbers they present look quite nice relative to more sophisticated approaches.
• “Investigating the Suboptimality and Instability of Pseudo-Relevance Feedback” by Raghavendra Udupa and Abhijit Bhole showed that an oracle with knowledge of a few high-scoring non-relevant documents could vastly improve the performance of pseudo-relevance feedback. While this information does not lead directly to any applications, it does suggest that obtaining a very small amount of feedback from the user might go a long way. I’m curious how much is possible from even a single negative-feedback input.
• “Short Text Classification in Twitter to Improve Information Filtering” by Bharath Sriram, David Fuhry, Engin Demir, Hakan Ferhatosmanoglu, and Murat Demirbas challenged the conventional wisdom that tweets are too short for traditional classification methods. They achieved nice results, but on the relatively simple problem of classifying tweets as news, events, opinions, deals, and private messages. I was offered promises of future work, but I think the more general classification problem is much harder.
• “Metrics for Assessing Sets of Subtopics” by Filip Radlinski, Martin Szummer, and Nick Craswell proposed an evaluation framework for result diversity based on coherence, distinctness, plausibility, and completeness. I suggested that this framework would apply nicely to faceted search interfaces, and that I’d love to see it demonstrated on production systems–especially since I think that might be easier to achieve than convincing the SIGIR community to embrace it.
• Which leads me nicely to the last poster I saw, “Machine Learned Ranking of Entity Facets” by Roelof van Zwol, Lluis Garcia Pueyo, Mridul Muralidharan, and Borkur Sigurbjornsson. They found that they could accurately predict click-through rates on named entity facets (people, places) by learning from click logs. It’s worth noting that their entity facets are extremely clean, since they are derived from sources like Wikipedia, IMDB, GeoPlanet, and Freebase. It’s not clear to me how well their approach would work for noisier facets extracted from open-domain data.

As I said, there were over a hundred posters, and I’d meant to see far more of them. Hopefully other people will blog about some of them! Or perhaps tweet about them at #sigir2010.

Jinyoung Kim of UMass presented joint work with Bruce Croft on “Ranking using Multiple Document Types in Desktop Search” in which they showed that type prediction can significantly improve known-item search performance in simulated desktop settings. I like the approach and result, but I’d be very interested to see how well it applied to more recall-oriented tasks.

Then came work by Googlers Enrique Alfonseca, Marius Pasca, and Enrique Robledo-Arnuncio on “Acquisition of Instance Attributes via Labeled and Related Instances” that overcomes the data sparseness of open-domain attribute extraction by computing relationships among instances and injecting this relatedness data into the instance-attribute graph so that attributes can be propagated to more instances. This is a nice enhancement to earlier work by Pasca and others on obtaining these instance-attribute graphs.

The session ended with an intriguing paper on “Relevance and Ranking in Online Dating Systems” by Yahoo researchers Fernando Diaz, Donald Metzler, and Sihem Amer-Yahia that formulated a two-way relevance model for matchmaking systems but unfortunately found that it did no better than query-independent ranking in the context of a production personals system. I would be very interested to see how the model applied to other matchmaking scenarios, such as matching job seekers to employers.

After a wonderful lunch hosted by Morgan & Claypool for authors, I attended a session on Filtering and Recommendation.

It started with a paper on “Social Media Recommendation Based on People and Tags” by IBM researchers Ido Guy, Naama Zwerdling, Inbal Ronen, David Carmel, and Erel Uziel. They analyzed item recommendation in an enterprise setting and found that a hybrid approach combining algorithmic tag-based recommendations with people-based recommendations achieves better performance at delivering interesting recommendations than either approach alone. I’m curious how well these results generalize outside of enterprise settings–or even how well they apply across the large variation in enterprises.

Then came work by Nikolaos Nanas, Manolis Vavalis, and Anne De Roeck on “A Network-Based Model for High-Dimensional Information Filtering”. The authors propose to overcome the “curse of dimensionality” of vector space representations of profiles by instead modeling keyword dependencies in a directed graph and applying a non-iterative activation model to it. The presentation was excellent, but I’m not entirely convinced by the baseline they used for their comparisons.

After that was a paper by Neal Lathia, Stephen Halles, Licia Capra, and Xavier Amatriain on “Temporal Diversity in Recommender Systems”. They focused on the problem that users get bored and frustrated by recommender systems that keep recommending the same items over time. They provided evidence that users prefer temporal diversity of recommendations and suggested some methods to promote it. I like the research, but I still think that recommendation engines cry out for transparency, and that transparency can also help address the diversity problem–e.g., pick a random movie the user watched and propose recommendations explicitly based on that movie.

Unfortunately I missed the last paper of the session, in which Noriaki Kawamae talked about “Serendipitous Recommendations via Innovators”.

Reminder: also check out the tweet stream with hash tag #sigir2010.

I arrived here on Monday, too jet-lagged to even imagine attending the tutorials, but fortunately I recovered enough to go to the welcome reception in the Parc de Bastions that evening. Then a night of sleep and on to the main event.

Tuesday morning kicked off with a keynote by Microsoft Live Labs director Gary Flake entitled “Zoomable UIs, Information Retrieval, and the Uncanny Valley”. Flake’s premise is that information retrieval is stuck in the “uncanny valley“, a metaphor he borrows from the robotics community. According to Wikipedia:

The theory holds that when robots and other facsimiles of humans look and act almost like actual humans, it causes a response of revulsion among human observers. The “valley” in question is a dip in a proposed graph of the positivity of human reaction as a function of a robot’s lifelikeness.

Flake offered Grokker (R.I.P.) as an example of a search interface that emphasized visual clustering and got stuck in the uncanny valley. He called it “the sexiest search experience that no one was going to use”. Flake then went on to propose that moving beyond the uncanny valley would require replacing our current discrete interactions with search engines into a mode of continuous, fluid interaction where whole of data greater than sum or parts. He offered some demos, emphasizing the recently released Pivot client, that he felt provided a vision to overcome the uncanny valley.

As became clear in the question and answer period, many people (myself included) felt that this rich visual approach might work well for browsing images but not as clear a fit for text-oriented information needs–despite Flake offering a demo based on the collection of Wikipedia documents. In fairness, it may be too early to assess a proof of concept.

Nonetheless, I feel that links supply context, and I’m a fan of keeping context nearby. Indeed, I find that clicking on a link incurs a much lower cognitive cost than flipping to the back of the book, searching for an endnote. I’ve had readers specifically thank me for including links to Wikipedia entries for technical terms. I assume those readers are fully capable of finding those Wikipedia entries themselves, but that they appreciate the convenience of the links.

Some of the commenters on Carr’s post suggest that we use technology to address this tension between preserving the reader’s focus and supplying nearby context. Specifically, we can use CSS and have a JavaScript button that toggles the link style between visible and invisible. I like the idea of handing readers control of the presentation style, though I still think it’s important to pick a sensible default. At the very least, a document should be self-contained so that a reader can choose if and when to look at the material it cites. The document should also give credit where it’s due, linking to the material it cites in a way that is visible to people and search engines. Beyond that, I think it’s really a matter of author style.

Still, I’m curious what folks here–especially long-time readers–think. Do I link so heavily that it’s distracting? Would it be easier to read my posts if the links were in a block at the end? I write for you, so please let me know how I can make this blog better. I don’t have the resources to conduct cognitive load experiments, but I’m very receptive to comments.

The focus of his talk, naturally, was patterns. Specifically, he advocated that we take the behavior patterns of information seekers that library and information scientists have been studying for years, and use them to inform design patterns for search user interfaces.

One point he raised that deserves a deeper dive: number of media (mobile, kiosk, TV) environments push people to browse, partly because of limitations of the medium but also taking advantage of the novelty and relative lack of user habits. Unfortunately, browsing doesn’t always scale in those environments, so search is usually available as a contingency.

Interestingly, while Peter promotes rich interfaces in many of his patterns, he noted that great results ranking plus speedy response (he uses Google “classic” as his example) does allow users to rapidly reformulate their queries while staying in the flow of the information seeking experience. He returned to Google later in his talk, noting that the new interface goes beyond ranking to support a richer user interaction.

And, like me and Marti Hearst (yesterday’s keynote), Peter advocates faceted navigation (I won’t quibble on whether to call it navigation or search) as his favorite search design pattern. He uses the NCSU library as an example not only of a great implementation but also of an organization that continues to experiment with incremental design changes. He also showed faceted search examples from other domains, including Amazon and Buzzilions.

Other patterns he discusses included question answering (his example being Wolfram Alpha) and decision making (his example being Hunch). He didn’t go deep on these, but rather invited the audience to consider a broad palette of strategies for supporting information seeking. Indeed, when I asked him about question answering, he conceded that he was a skeptic and preferred a conversational (i.e., HCIR) approach akin to a librarian’s reference interview.

His closing note was about bridging the gap between physical and digital information, where he offered a potpourri of examples (from Redbox to a tweeting plant). I work in local search, so in my case he’s preaching to the converted. But I think he’s right that everything is only recently coming together–specifically, the ubiquity of digital data on the internet and of mobile devices in the physical world that can both consume and produce that data. Many of us take these developments for granted, but it’s important that we adapt our approach to search to address what is a very recent phenomenon.

Fun stuff! I didn’t get to attend the rest of the summit, but I encourage you to check out the tweet stream at #ESS10.

Her major take-aways:

• The user’s emotional response is a key aspect of the information seeking experience.
• There is a double vocabulary problem: different ways to express same concept (cf. Furnas et al.), and users stubbornly anchor on initial query terms (cf. Kahneman, Tversky, et al.)
• Recognition is easier than recall, so interfaces need to support the recognition process.
• Don’t personalize search, socialize it!

She peppered her talk with concrete examples and scholarly references. Given that her book is available online for free, I won’t try to replicate them all here! Still, I’ll single out two Noisy Community members: FXPAL researchers Jeremy Pickens and Gene Golovchinsky (for their SIGIR 2008 work on collaborative exploratory search) and user experience designer Greg Nudelman for his proposal of faceted breadcrumbs as a search user interface.

If you missed her live, you check find a video of a tech talk she gave at Google a few months ago. You can also check out the conference tweet-stream at #ESS10.

Speaking of Jason, check out the latest improvements he’s made to the TunkRank interface. Pretty slick! To learn more about the TunkRank measure of Twitter influence / authority, check out this post.

I know there’s lots of interesting stuff coming out at the Chirp Twitter developer conference this week, and I’m still catching up on it all. But I am happy to point folks to a Google Labs application that was announced this morning: Follow Finder.

It’s not the first application to suggest Twitter followers based on analysis of the social graph, but I’ve actually found its suggestions to be quite plausible. For example, it suggests @fredwilson, @cshirky, @mattcutts, @peteskomoroch, and @msftresearch as “tweeps” I should follow, and suggests that the following users have similar followers to mine: @endeca, @lemire, @yahooresearch, @googleresearch, and @mattcutts.

There’s a bit of an “everything sounds like Coldplay” effect (e.g., @fredwilson shows up in a lot of the searches I tried), but overall I’m impressed with the quality, especially compared to the other suggestion tools I’ve tried.

I haven’t talked much about TunkRank in the past months, largely because Jason Adams, who stepped up to the TunkRank Implementation Challenge last year, has been leading the charge. Indeed, all I did, beyond lending my first syllable to its name, was to propose the measure and get it implemented “Tom Sawyer” style.

Since then:

• TunkRank was cited in a WSDM 2010 paper entitled “TwitterRank: finding topic-sensitive influential twitterers“
• Nicolas Cerrato implemented an influence measure for gamer site Gamocracy based on TunkRank.

And, most recently:

• University of Oviedo professor Daniel Gayo-Avello published a research paper entitled “Nepotistic Relationships in Twitter and their Impact on Rank Prestige Algorithms“, based on a follower graph of 1.8M Twitter users, in which he reports:
  
  Lastly, there are one method clearly outperforming PageRank with respect to penalization of abusive users while still inducing plausible rankings: TunkRank. It is certainly similar to PageRank but it makes a much better job when confronted with “cheating”: aggressive marketers are almost indistinguishable from common users –which is, of course, desirable; and spammers just manage to grab a much smaller amount of the global available prestige and reach lower positions –although they still manage to be better positioned than average users. In addition to that, the ranking induced by TunkRank certainly agrees with that of PageRank, specially at the very top of the list, meaning that many users achieving good positions with PageRank should also get good positions with TunkRank. Thus, TunkRank is a highly recommendable ranking method to apply to social networks: it is simple, it induces plausible rankings, and severely penalizes spammers when compared to PageRank.
  
  You can read a summary version in his blog post, descriptively titled “Research on a 1.8M Twitter user graph. Conclusion: TunkRank is your best option.“

I’ve excited that an idea I came up with on a whim (or perhaps out of excessive idealism) has taken such a life of its own. And hey, I do work for a company that is into real-time search and that knows a thing or two about adversarial information retrieval. Hopefully I’ll find way to apply TunkRank–or at least its intuition–in my own work. In the mean time, I offer those who have already done so my congratulations and gratitude.

I’ve been following Hunch for a while, and my impression has evolved from the initial skepticism with which I greeted it a year ago (to the day!). Given the track records of co-founders Caterina Fake and Chris Dixon, perhaps I should have expected their success at obtaining traffic and funding.  But what interests me more is that they are doing interesting things with data mining and putting a new twist on social media analytics.

For those unfamiliar with Hunch, it is a decision engine (cf. [real decision engine] vs. [decision engine]). For example, it can help you decide whether to buy an iPad or how to name your baby. While it’s not clear to me how much people are using Hunch for utility vs. entertainment, Hunch is certainly accumulating users–as well as the data that those users volunteer.

Hunch recently released two applications that mash up that data with the Twitter follower graph. The first is a “Twitter Predictor Game” that attempts to calculate your taste profile from your Twitter id and then predict how you’ll answer Hunch’s taste questions. Just to keep the game honest, you can look at the Hunch’s guess either before or after you provide your answer. The second is called “Twitter Follower Stats“: given a Twitter user, it reports the salient information it has inferred about that user’s followers (e.g., @maddow vs. @karlrove).

I think this stuff is neat, and a great testament to the “unreasonable effectiveness of data“. The question-answer data still feels a bit sparse for my taste, and I suspect there’s still room for more dimensionality reduction. I’m sure Hunch CTO Matt Gattis and colleagues are working on it! Also, it would be neat to direct the follower analytics rather than simply see the ones that Hunch deems most salient.

In summary, Hunch is keeping it interesting. Definitely a startup to watch and learn from.

Mathew Ingram posted an interview with Craig Newmark (the Craig of craigslist fame) in which the latter argued that what the web needs is a “distributed trust network” to manage our online reputations. As it happens, this is an idea that has occupied me for several years. So I figured it was about time that I shared my thoughts on the subject.

When we think of how trust works online, two of the most prominent examples are Google’s PageRank measure and eBay’s feedback scores. But neither of these measures addresses what I think Craig has in mind. PageRank is a great way of using citation analysis to determine the most authoritative citations, but the trust in a page should consider its out-links (i.e., can we trust the page not to point us to untrustworthy ones?) and not just its in-links. eBay’s feedback scores have a different problem: they count positive and negative ratings without considering the social network of buyers and sellers–and approach that is vulnerable to fraud through shill ratings. Incidentally,LinkedIn recommendations have a similar weakness if viewed in strictly quantitative terms, but the potential for abuse is mitigated by the endorsements being signed–and by their being more than just binary or numerical ratings. Incidentally, here’s a site you can use if you’re too lazy to actually write the recommendations yourself.

But I digress. Propagation of trust does seem like the perfect application to build on top of social networks. Consider any problem that involves getting advice to inform a decision. If we regularly solicit advice from our first-degree connections, then we should be able to learn over time whose advice we can trust. We can then vouch for these connections, which offers the connections who trust us a basis for trusting their second-degree connections through us. And so forth through our social network. Of course, trust is not irrevocable: loss of trust should propagate similarly.

I’ve talked about this problem with two of the leading experts on social networks, Jon Kleinberg and Prabhakar Raghavan, and as far as I know no one has built a system along these principles. In economic terms, I envision a system where a person’s reputation truly is his or her coin. One person might think of bribing one another to exploit the latter’s established reputation, but a rational person with a strong reputation would demand an exorbitant bribe to put that reputation at risk.

Of course, a lot of information would have to propagate throughout the social network–and be stored–for this system to work. Regardless of how the information is abstracted, such a reputation index would raise thorny privacy issues. Nonetheless, I don’t know if we can build a reputation system that is entirely privacy-preserving–since reputation is an inherently public mechanism. In addition, any such system would have to consider the implications of defamation laws. These are some major hurdles!

Nonetheless, I agree wholeheartedly with Craig that a distributed trust network could be “the killingest of killer apps”. I just hope we can find a way to build and use it!

Note: Chris Rines suggested I look at Advogato’s Trust Metric, and a quick investigation led me to the Wikipedia entry for trust metric. Looks like I have some homework to do!

In a post on ReadWriteWeb, Sarah Perez summarizes “Measuring User Influence in Twitter: The Million Follower Fallacy“, a recent research paper by Meeyoung Cha, Hamed Haddadi, Fabricio Benevenuto, and Krishna Gummadi. The punch line should hardly be surprising to regular readers here given my variety of rants on the subject: follower count isn’t great measure of influence.

The authors focus on measuring three quantities: followers (which they call indegree), retweets, and mentions. Their main results is that, while the number of followers is strongly correlated to the numbers of retweets and mentions for the general user population, the correlation is much weaker for the users with high follower counts, e.g., in the top 10%. Indeed, the authors believe that the correlation for the general population is “an artifact of the tied ranks among the least influential users, e.g., many of the least connected users also received zero retweet and mention.”

The authors further note that:

Across all three measures, the top influentials were generally recognizable public figures and websites. Interestingly, we saw marginal overlap in these three top lists. These top-20 lists only had 2 users in common: Ashton Kutcher and Puff Daddy. The top-100 lists also showed marginal overlap, as shown in Figure 1, indicating that the three measures capture different types of influence.

The authors ultimately conclude that:

• Follower count represents a user’s popularity, but is not related to notions of influence such as engaging audience, i.e., retweets and mentions.
• Retweets are driven by the content value of a tweet, favoring mainstream news organizations.
• Mentions are driven by the name value of the user, favoring celebrities.

I can’t argue with any of the above, but I do wonder if any of them are ideal measures of influence. All three measures are easy to game–and none of them model the scarcity of user attention, which is the motivating principle of TunkRank. Nor do they ground “influence” in any outcome external to Twitter.

Still, it’s an interesting negative result. If nothing else, it helps reinforce the argument that follower count isn’t a useful measure–at least once you get beyond the very low end of the range.

in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it (Simon, 1971)

I hope everyone agrees that attention is a scarce good. But I’m curious how people measure it. After all, if we’re going to talk about an economic good being scarce, we ought to quantify it!

One approach is to measure attention at a specific moment in time, measuring how much of our instantaneous cognitive capacity we devote to a task. This approach is useful for evaluating a user interface–in particular, for determining how users allocate their attention among the various interface elements. Another approach is to measure attention in units of time, e.g., how many of our waking hours do we devote to a particular activity. This latter strikes me as more of what Herb Simon had in mind.

We can interpret the two definitions as equivalent–after all, cumulative attention devoted to a task is simply the sum (or integral) of instantaneous attention over time. But thinking this way so misses a key consideration: we pay a significant price for context switching.

A familiar example is email. The total time we spend reading email is a productivity concern, but the larger concern for many of us is the frequency with which email causes us to interrupt our workflow. Knowing this, I made a brief attempt in 2008 to check email only once a day. Unfortunately, this approach would have violated too many of my peers’ expectations. I returned to status quo, reading my email (or at least scanning headers) as it arrives. Other messaging tools, such as instant messaging and Twitter, only add to the challenge of managing our personal communication flow.

Of course, what I really want is for my messaging tools to distinguish urgent messages from non-urgent ones, and to only interrupt my workflow for the former. I know that no system, whether based on manual filtering or algorithmic analysis, can make this subjective classification with 100% accuracy, but I’d certainly accept a handful of false positives in exchange for far fewer interruptions. I suspect I’m not alone.

Moreover, this approach extends beyond personal communications to more public ones, such as social media platforms and even web search. On one hand, the passing of time offers an opportunity to accumulate reliable content analysis; on the other hand, we don’t want to miss time-sensitive content just because the system waited too long to determine the content’s relevance to our information needs. Still, the low signal-to-noise ratio on social media platforms suggests to me that many information consumers would be amenable to a different tradeoff than the one we experience today.

What I’d really like to see is systems take advantage of the differences in users’ personal senses of urgency. Some examples:

• A widely broadcast email isn’t delivered all at once, but first goes to users with higher urgency settings. Because those users mark it as spam, the email is already marked as spam for users with lower urgency settings. Conversely, if enough high-urgency users mark it as important, then it may be sent to lower-urgency users sooner.
• High-urgency users frequently check news sites and blogs. If an article attract a threshold level of engagement from high-urgency users, then low-urgency users are notified. This approach could apply to general news or to news in a specific topic that the user follows.
• Same as above, but applied to activity feeds and based on engagement within your social network. But again, high-urgency users lead the way, seeing updates sooner but at the price of experiencing a noisier stream.

To some extent, our existing systems already approximate this approach. Mechanisms like favoriting and re-tweeting propagate signal from information scouts to their followers, as do algorithms that rank real-time information based on engagement. Still, as an information consumer, I’d appreciate an interface that explicitly and transparently adapts to my priorities, and that manages interruption of my workflow accordingly.

What do folks here think? Is information delayed tantamount to information denied? Or is time on our side, potentially offering us a better tradeoff than the one we experience today?

Today is the last day of WSDM 2010, and I unfortunately spent it at home drinking chicken soup. But I’ve been following the conference via the proceedings and tweets.

The day started with a short session on temporal interaction. Topics included clustering social media documents (e.g., Flickr photos) based on their association with events, statistical tests for early identification of popular social media content, and analysis of answers sites (like Yahoo! Answers) as evolving two-sided economic markets.

The next session focused on advertising. Two papers focused on click prediction: one proposing an Bayesian inference model to better predict click-throughs in the tail of the ad distribution; the other presenting a framework for personalized click models. Another paper addressed the closely related problem of predicting ad relevance. The remaining papers discussed other aspects of search advertising: one on estimating the value per click for channels like Google AdSense, where ad inventory is supplied by a third party; the other proposing an algorithmic approach to automate online ad campaigns based onlanding page content.

The following session was on systems and efficiency, a popular topic given the immense data and traffic associated with web search. Two papers proposed approaches to help short-circuit ranking computations: one by optimizing the organizations of inverted index entries to consider both the static ranks of documents and the upper bounds of term scores for all terms contained in each document; the other using early-exit strategies to optimize ensemble-based machine learning algorithms. Another used machine learning to mine rules for de-duplicating web pages based on URL string patterns. Another focused on compression, showing that web content is at least an order of magnitude more compressible that what can be achieved by gzip. The last paper proposed a method to perform efficient distance queries on graph (i.e., web graphs or social graphs) by pre-computing a collection of node-centered subgraphs.

The last session of the conference discussed various topics in web mining. One presented a system for identifying distributed search bot attacks. Another proposed an image search method using a combination of entity information and visual similarity. The final paper showed that shallow text features can be used for low-cost detection of boilerplate text in web documents.

All in all, WSDM 2010 was an excellent conference, and I’m sad to not to have been able to attend more of it in person. I’m delighted to see an even mix of academic and industry representatives sharing ideas and working to make the web a better place for information access.

Unfortunately, I woke up this morning rather under the weather, so I’m having to resort to remotely reporting on the second day of WSDM 2010 conference, based on the published proceedings and the tweet stream.

The day started with a keynote from Harvard economist Susan Athey. Her research focuses on the design of auction-based markets, a topic core to the business of search which largely relies on auction-based advertising models (cf. Google AdWords). Then came a session focused on learning and optimization. One paper proposed a method to learn ranking functions and query categorization simultaneously, reflecting that different categories of queries leads users to have different expectations about ranking. Another combined traditional list-based ranking with pair-wise comparisons between results to separate the results into tiers reflecting grades of relevance. An intriguing approach to query recommendation treated it as an optimization problem, perturbing users’ query-reformulation path to maximize the expected value of a utility function over the search session. Another paper looked not at ranking per se, but rather at improving the quality of training data for using machine learning for ranking. The final paper of the session, which earned a best-paper nomination, modeled document relevance based not on click-through behavior, but rather on post-click user behavior.

The next session was about users and measurement. It opened with another best-paper nominee: a analysis of over a hundred million users to understand how they re-find web content. Another offered a rigorous analysis of the often sloppily presented “long-tail” hypothesis: it found that light users disproportionately prefer content at the head of distribution while heavy users disproportionately prefer the tail. Another log-analysis paper analyzed search logs using a partially observable Markov model, a variant of thehidden Markov model in which not all of the hidden state transitions emit observable events–and compared the latent variables with eye-tracking studies. An intriguing study demonstrated that user behavior models are more predictive of goal success than models based on document relevance. The final paper of the session proposed methods for quantifying the reusability of the test collections that lie at the heart of information retrieval evaluation.

The last session of the day focused on social aspects of search. Two of the papers were concerned with modeling authority and influence in social networks, a problem in which I take a deep personal interest. Another inferred attributes of social network users based on those of other users in their communities (cg. MIT’s Project Gaydar). Another analyzed Flickr and Last.fm user logs to show that users’ semantic similarity based on their tagging behavior is predictive of social links. The final paper tackled the sparsity of social media tags by inferring latent topics from shared tags and spatial information.

Not surprisingly, a disproportionate number of contributors to the conference work at major web search companies, who have both the motivation to improve results and the access to data that is needed for such research. One of the ongoing research challenges for the field is to find ways to make this data available to others while respecting the business concerns of search engine companies and the privacy concerns of their users.

It is my pleasure to report on the 3rd Annual Workshop on Search in Social Media (SSM 2010), a gathering of information retrieval and social media researchers and practitioners in an area that has captured the interest of computer scientists, social scientists, and even the broader public. The one-day workshop took place at the Polytechnic Institute of NYU in Brooklyn, NY, co-located with the ACM Conference on Web Search and Data Mining (WSDM 2010). The quality of the presenters, the overbooked registration, and the hundreds of live tweets with the #ssm2010 hashtag all attest to the success of this event.

The workshop opened with a warm welcome from Ian Soboroff (NIST), immediately followed by a keynote from Jan Pedersen, Chief Scientist of Bing Search. Jan established a clear business case for search in social media: the opportunity to deliver content that is fresh, local, and under-served by general web search. He drilled into particular types of content where social media search is most useful: expert opinions, breaking news, and tail content. The benefits of social media search include trust and personal interaction (as compared to web content that is often soulless and of uncertain provenance), low latency (though perhaps at the cost of accuracy), and access to niche or ephemeral information that web search rarely surfaces. But delivering social media results to searchers creates its own variety of challenges, such as weighing freshness against accuracy and relevance, coping with loss of social content’s conversational context, managing low update latency when search engines have not been optimized for it, and fighting new kinds of spam. Despite these challenges, it is clear that the major web search engines have embraced the brave new world of real-time social content.

Eugene Agitchein (Emory University) then moderated a panel representing the world’s leading search engines: Jeremy Hylton (Google), Matthew Hurst (Microsoft), Sihem Amer-Yahia (Yahoo!), and William Chang (Baidu). Jeremy justified the universal interface approach, pointing out that users don’t want to have to figure out what kind of search site to use for their queries, and that they expect a familiar interface. He also noted that Google has made great strides on update latency: it can index the Twitter firehose in the same amount of time as serving a query. Matthew offered various analyses of the social search problem, based on whether the information signal resides in content (e.g., web) or attention (e.g., Twitter), or whether the information need is expressed in an explicit search query or inferred from the user’s context. Sihem offered a counter-point to Jeremy, arguing that social media search queries often represent broad or vague information needs, and thus call for a more browsing-oriented interface than web search, which is optimized for highly specific needs. William noted that the biggest competitive threat he sees for web search engines comes from social media players–and he credits much of Baidu’s success to its surfacing of social media content.

Then came a flurry of questions, perhaps the most interesting of which was how to address identity management. William argued that people prefer interacting with real-named (or pseudonymous) people to whom they are directly connected. Sihem offered the counter-example of obtaining recommendations through community aggregation. Matthew noted the incongruity of there being no economic relationship between social network companies that maintain proprietary social graphs and people whose identities and relationships those graph represent. Jeremy pointed out that users benefit if the data is as open as possible.

Given the almost even split between academic and industry participation in the workshop, the panelists were also asked to present research challenges to academia. Jeremy posed the problem of determining when social media results are actually true. Matthew wants to see more interdisciplinary work between computer scientists and social scientists. Sihem offered two challenge problems:  scalable community discovery and evaluation of collaborative recommendation systems. William wants to see a rigorous axiomatization of social media search behavior.

After lunch, Jeremy Pickens (FXPAL) moderated a panel representing social media / networking companies: Hilary Mason (bit.ly), Igor Perisic (LinkedIn), and David Hendi (MySpace). Hilary noted that, while bit.ly does not have access to an explicit social graph, it captures implicit connections from user behavior that may not be represented in the graph. Jeremy asked the panelists how much a person’s extended network matters; David and Igor pointed out research indicating correlations of mood and even medical conditions between people and their third-degree connections. Again, the audience was full of questions, especially for Igor. As a fan of faceted search, I was glad to see him touting LinkedIn’s success in making faceted search the primary means of performing people search on the site. For an in-depth view, I recommend “LinkedIn Search: A Look Beneath the Hood“.

The afternoon continued with a poster / demo session emphasizing work in progress: tools, interfaces, research studies, and position papers. I particularly enjoyed listening to the stream of interaction between academic researchers and industry practitioners.

The final panel session assembled academic researchers to discuss their views of the challenges in social media. Gene Golovchinsky (FXPAL) moderated a panel comprised of Meena Nagarajan (Wright State University), Liangjie Hong (Lehigh University),Richard McCreadie (University of Glasgow), Jonathan Elsas (CMU), and Mor Naaman (Rutgers University). Meena highlighted the need to build up meta-data to describe the context around social utteracnces. Liangjie took a position similar to William Cheng’s, calling for a framework to model the tasks and behavior of users who interact with social media. Richard focused on the intersection of social media and news search, and noted that some of the most useful information is private and proprietary (e.g., search and chat logs). Jonathan offered a variety of challenges: determining the right retrieval granularity, managing multiple axes of organization, aggregating author behavior, and multidimensional indexing of social media content. Finally, Mor noted that we’re moving from a world of email to a “social awareness stream”, in which the content we directed content at a group and have lower expectations of readership than email. As with all of the panels, there were countless questions from the moderator and audience, particularly about determining the truthfulness of social media content and delivering social content in an effective user interface.

The final conference session was a conference was a full-group discussion that dived into the various topics addressed throughout the day. But Gene Golovchinsky provided the “one more thing” at the end, showing us a glimpse of a faceted search interface to explore a Twitter stream. It was an elegant finish to a day filled with informative and engaging discussion, and I look forward to seeing many of the participants in the WSDM conference over the next few days.

Of course, I’ll cross-post here. I also encourage folks to follow the live tweet streams at #ssm2010 and #wsdm2010, as well as Gene and Jeremy’s posts at the FXPAL blog.

To those attending: see you all tomorrow through Saturday! To everyone else: I will try my best to communicate the substance and spirit of the conference.

As my wife, daughter, and I were walking home from a playground, we noticed a large number of fire trucks congregating a block away from our house. A quick search on Twitter explained what was going on, particularly by pointing us to this post on Gothamist–which as of this writing seems to be the only reporting about this incident.

I think this example tells us a lot about the utility of real-time search. Most of us don’t need real-time search to tell us about the news in Haiti, since a critical mass of major news providers is covering the story around the clock. Where real-time search matters most is at the personal level–specifically, when our personal urgency to obtain information is higher than that of the general population. In such situations, we’re willing to accept less polished–and even risk less accurate–information, particularly if the alternative is to wait until if and when news providers cover the story. At least to some extent, urgency trumps authority.

Yes, there are other use cases for conversational media like Facebook and Twitter, such as sharing the experience of watching a live event, or simply chatting with friends and strangers about arbitrary topics. But I wouldn’t consider such use of these media to be search. Real-time search, in my view, is about helping users obtain the latest information available–in accordance with their personal needs. Twitter and Google served me well today, and I’m grateful that real-time search gave me real-time peace of mind.

Earlier this week, Peter Morville and Mark Burrell presented a UIE virtual seminar on “Leveraging Search & Discovery Patterns For Great Online Experiences“. It sold out! And I thought Pete Bell and I had done well with our seminar on faceted search!

But I’m hardly surprised. Although I wasn’t able to attend it myself, I gather from Twitter and the blogosphere that it was a great presentation. I enjoyed serving as a reviewer for Peter’s new book on Search Patterns, and I contributed a bit to Endeca’s UI Design Pattern Library while I was there and Mark’s team was developing it.

In reading reactions to the seminar, I was particularly intrigued by a post entitled “Search and Browse” by Livia Labate on her fantastically named blog, “I think, therefore IA“. She raised a question that I think needs to be asked more often: when is (or isn’t) faceted search appropriate?

Her conversation with readers in a comment thread offered some possible answers:

• Faceted search helps users who think in terms of attribute specifications as filtering criteria.
• Faceted search supports search by exclusion, as opposed to by discovery.
• Faceted search requires a set of useful facets that is neither too small nor too large.

I’d like to propose my own answers. Here are the conditions for which I see faceted search being most useful:

• Faceted search supports exploratory use cases, in contrast to known-item search. For known-item search, users are better served by a search box to specify an item by name, or a non-faceted hierarchy to locate it. In contrast, faceted search optimizes for cases where users are either unsure of what they want or of how to specify it.
• Faceted search helps users who need or want to learn about the search space as they execute the search process. Facets educate users about different ways to characterize items in a collection. If users do not need or want this education, they may be frustrated by an interface that makes them do more work.
• The search space is classified using accurate, understandable facets that relate to the users’ information needs. As I’ve discussed before, data quality is often the bottleneck in designing search interfaces. Offering users facets that are either unreliable or unrelated to their needs is worse than providing no facets at all.

Given the above criteria, it’s not surprising that faceted search has been a huge success in online retail: shopping is often an exploratory learning experience, and retailers tend to have good data.

But the success of faceted search in retail overshadows other domains where faceted search may be even more valuable. My favorite example is faceted people search, most recently demonstrated by LinkedIn. I would love to see other entities (locations, businesses, etc.) receive similar treatment, at least in contexts where exploration is a common use case.

I think Livia is right to be skeptical about any interface that introduces complexity–and facets do introduce complexity. I hope that my guidelines help answer her question as to when that complexity is worthwhile and perhaps even necessary to help users satisfy their information needs.

committed to providing a framework for any company big or small, rich or poor to do a deal with us to get access to the Firehose in the same way we did deals with Google and Microsoft. We want everyone to have the opportunity — terms will vary based on a number of variables but we want a two-person startup in a  garage to have the same opportunity to build great things with the full feed that someone with a billion dollar market cap does. There are still a lot of details to be fleshed out and communicated, but this a top priority for us and we look forward to what types of companies and products get built on top of this unique and rich stream.

That and some other details, like raising the API rate limit from 150 requests per hour to 1500,  may well bring on what Marshall Kirkpatrick of ReadWriteWeb calls “Twitter 2.0“. But it was something else in Kirkpatrick’s write up that caught my attention–this quote from Wow.ly co-founder Kevin Marshall:

The more I do with and around social data, the less interested I seem to become in ‘realtime’ and the more interested I become in ‘over time.’ When I first started hacking on Twitter (and Facebook) apps, I was in love with the idea of parsing and analyzing data in real-time and I was very link/content focused. But the more I build and use these tools, the more I see the value in the history and the trails of the data set.

I couldn’t have said it better! Not that I haven’t tried: you look back at my post about Topsy, you’ll see where real-time and over time meet. Recency matters, but the signal is far too sparse without some way to aggregate and analyze over time.

I’m thrilled that Twitter plans to open up its platform in a way that could enable analysis over semantic, social, and temporal dimensions. Now I’m curious to see what that access will look like, and what everyone has been clamoring for that access will do with it.

• BIA / Kelsey Blog
  • By The Kelsey Group, a division of BIA Advisory Services that provides data and analysis on directories and local media.
• Exploring Local
  • By Mike Dobson, President of TeleMapics, a company that provides consulting services focused on local search.
• Local SEO Guide
  • Andrew Shotland‘s blog on local search optimization, small business marketing & search engine optimization strategy.
• Localsearchdatabase
  • By Gib Olander, Director of Business Development for Localeze, an online content management company serving businesses, local search engines and consumers.
• Mihmorandum
  • David Mihm‘s blog on local search engine optimization and marketing.
• Screenwerk
  • Greg Sterling‘s thoughts on online and offline media. Sterling used to run The Kelsey Group’s Interactive Local Media program.
• SEO Igloo Blog
  • By Solas Web Design, which specializes in web design and search engine optimization for small businesses.
• Understanding Google Maps & Local Search
  • By Mike Blumenthal, whose company offers consulting services and market research advice relating to maps and local search.

Not surprisingly, these blogs offers me a critical perspective on how Google and other search engines serve the local space.  Granted, everyone has their own motives–and it’s hard to avoid some tension in a space with the competitive dynamics of local search. But now that I’m no longer an outsider myself, I appreciate having others to help keep me honest as I work to make local search better for users and businesses.

Consider MusicLab, an “experimental study of self-fulfilling prophecies in an artificial cultural market“. In this study, sociologists Matt Salganik, Peter Dodds, and Duncan Watts manipulated the social information available to consumers (specifically teens) regarding their peers’ musical tastes. The experimenters’ goal was to empirically validate a quantitative model of social contagion.

But we can look at this study another way: by isolating the social factors that influence musical taste, the experimenters were also isolating the non-social signal–in theory, how popular a song would be in the absence of social signaling. Indeed, they found that, if they measured a song’s quality by isolating out the social factor, “the best songs never do very badly, and the worst songs never do extremely well, but almost any other result is possible”.

It’s interesting–interesting to me, at least!–to ask if search engines can do the same for search. One of the frequent objections to link-based authority measures like PageRank is that they make the rich get richer. “Real-time” variants like re-tweet frequency (and even TunkRank) suffer from the same weakness. Unchecked, these measures can cause authority / influence market has to resemble a winner-take-all market.

It strikes me as interesting to learn from cases where searchers swim upstream against the social signals to find information. Of course, you may already see the contradiction–this is just another kind of social signaling! Still, it seems like it might be a way to hedge our bets and against the weaknesses of positive feedback and spammers. In a similar vein, we might look at how users find information that suffers from poor accessibility or retrievability.

I don’t have answers about how to pursue such an approach, or whether it would even be feasible to do so. But I hope you agree with me that it’s an interesting question.

In “Using Twitter Lists To Judge Influence“, Todd Zeigler of the Bivings Report writes:

I think Twitter Lists will end up helping separate the men from the boys when it comes to influence.  In addition to seeing a Twitter users follower count, we can now see the number of other Twitter users who have added them to lists (example to the right).  I would argue that getting added to a list is a bigger deal than simply getting someone to follow you.

I’m certainly intrigued by Twitter Lists, but I’m skeptical that counting how many lists someone is on will prove that much more useful than follower count. For example, I currently have 1159 followers, am on 33 lists, and have a TunkRank of 24.1. For grins, here’s a handful of people who have similar stats:

• Evan Sandhaus: 796 followers, 21 lists, TunkRank = 17.2
• Josh Young: 801 followers, 25 lists, TunkRank = 14.3
• Chris Ahearn: 1108 followers, 14 lists, TunkRank = 30.1
• Brynn Evans: 1303 followers, 33 lists, TunkRank = 18.9
• Eric Andersen: 1543 followers, 37 lists, TunkRank = 3.1

While I can’t generalize from a few arbitrarily selected data points (though Gladwell seems to have no trouble doing so in Outliers), my suspicion is that list count will be highly correlated to follower count–and may actually be a noisier signal because the numbers are so much smaller.

Of course, there’s no reason we should use raw list counts–any more than we should use raw follower counts. Just as TunkRank aspires to model attention scarcity and recognizes that not all followers are created equal, an effective measure of how lists contribute to influence must recognize that not all list memberships are created equal either.

I’ve been chatting with Chris Langreiter, who is working on enhancements to TunkRank to address some of the oversimplifications of its model, as well as with Jonathan Glick and Ken Reisman at TLists. I’d like to see online influence–on Twitter and in general–measured more effectively. It will be great if lists can help, but we can’t make the same naive mistakes as those who were quick to embrace follower count as a measure of authority.

Google may be an also-ran in the social networking market with its Brazil-centric Orkut service, but that hasn’t stopped the search giant from adding social features to its products. A post at the (unofficial) Google Operating System blog recounts the history of Google Reader’s social evolution, up to but not including its latest update last week. SearchWiki, though not a social search feature per se, allows users to share personal annotations of their search results, as does the more recently introduced Sidewiki. And, like Bing, Google has established a partnership with Twitter in order to surface “social” results.

But the feature announced today, which Google is actually calling “Social Search“, is a much bigger step, even if it is tucked away as an experiment on Google Labs. From the official blog post:

With Social Search, Google finds relevant public content from your friends and contacts and highlights it for you at the bottom of your search results. When I do a simple query for [new york], Google Social Search includes my friend’s blog on the results page under the heading “Results from people in your social circle for New York.” I can also filter my results to see only content from my social circle by clicking “Show options” on the results page and clicking “Social.”

I gave it a whirl, search for “noisy channel” and then restricting the search to content from what Google considers my social circle. The results are as promised, and could further refine to results by author name, selecting from a familiar list of Neal Richter, Jason Adams, Daniel Lemire. Ken Ellis, and Joshua Young (though for some reason Josh’s link didn’t work). Cool! Except that there are a lot of names missing (check out the bloggers in The Noisy Community) and, more importantly, I can’t further refine or even sort the search results. Indeed, the ordering of search results seems quite arbitrary–a phenomenon I’ve noticed more generally for search engine ranking of social media content.

In short, Google Social Search is a welcome initiative, but there’s a lot more work to do before I would find a productive use for it. Given the mismatch between social search and black-box relevance ranking, a little bit of HCIR would go a long way towards making this feature practically useful.

But this year was different. We were overloaded with strong submissions from all corners, and we had to turn people away for lack of capacity! While we didn’t relish saying no to prospective participants, these are great problems to have! And, thanks to Nick Belkin and Diane Kelly, we’ve arranged to greatly increase that capacity at HCIR 2010–more on that in a moment.

Max Wilson has already written up an excellent summary of the workshop, which I encourage you to read. You can also see the live tweet stream at #hcir09. Rather than duplicate these efforts, let me add my personal reflections as an organizer and participant.

Ben Shneiderman‘s keynote address was sweeping and inspiring. I expected him to talk about information visualization, the area where he is most known for his contributions. He did present some examples of his group’s work on visualization-centric interfaces to support medical research, but his overall presentation took the much more ambitious approach of discussing the past, present, and possible future of HCIR. Specifically, he urged us to link our work to societal goals, such as the United Nations Millennium Development Goals. His challenge may seem impossibly idealistic, but I agree with his assertion that it is a practical one: we will do our best research by grounding ourselves firmly in the real and pressing problems of our age. Last year’s keynote speaker went on to win the Gerard Salton Award; I can only hope that Ben receives comparable accolades for his past accomplishments and future contributions to HCIR.

A new feature for this year’s workshop was having a “poster boaster” session, in which each of the presenters in the poster session had one minute to pitch his or her work.  For those of you unfamiliar with this format, I highly recommend it. The compressed format forces presenters to distill the essence of their contributions–a useful exercise in general. And the audience doesn’t get bored: if you decide halfway into a presentation that you aren’t interested, then you only have to wait 30 seconds until the next one! Not the we had that problem: the posters were consistently interesting, as the submissions were unusually strong this year. You can download the full workshop proceedings here.

Even the full presentations weren’t that long. The five speakers were each allotted ten minutes, with a healthy amount of time reserved for a panel-style Q&A sessions. The papers in this session were, by design, some of the more controversial ones. In particular, Ellen Voorhees delivered a full-throated defense of Cranfield / TREC-style evaluation: “I Come Not to Bury Cranfield, but to Praise It” (similar to her presentation at the 2006 Workshop on Adaptive Information Retrieval that I discussed on this blog last year). Her reminder of HCIR’s challenges on the evaluation front surely ruffled some feathers, but all of us HCIR avocates need to address these challenges if we want researchers (and practitioners) outside our community to drink our kool-aid.

The above format was already quite interactive (as befits a workshop about interaction), but the second half of the day was explicitly designed to facilitate discussion. We had lunch on site, followed by a one-hour poster session.  We then had two one-hour guided discussion sessions to address the theoretical and practical concerns of HCIR. As organizers, we seeded both sessions with questions, but we also incorporated concerns that had come up during earlier discussions.

Finally, I am grateful to our sponsors. Catholic University was a gracious host and sponsor, providing the workshop with a great space and very helpful student volunteers. Between that and the financial contributions of Endeca and Microsoft Research, we were able to continue our tradition of not charging attendees for the workshop. I can’t promise that will continue indefinitely, but I am glad that our insistence on emphasizing substance over frivolous amenities has helped us deliver what I believe to be some of the best bang-for-buck in the scholarly community.

I’m already excited about HCIR 2010. Unlike the past three workshops, which have been held as independent events, next year’s workshop will be co-located with the Information Interaction in Context Symposium (IIiX’10) in New Brunswick, New Jersey. The workshop will take place on August 22nd, breaking our unintended tradition of holding the workshop on October 23rd. Nick Belkin assures us that there will be lots of space, so hopefully we’ll be able to accommodate everyone who is interested. We’ll also be soliciting sponsors for both the workshop and the broader symposium.

But there’s more to HCIR than enjoying each other’s company at workshops. We must spend the remaining 364 days of the year fleshing out our vision, and relating that vision not only to the disciplines HCIR explicitly integrates, but to pressing social concerns. It is up to us all to make our work relevant.

But apparently the Huffington Post is blazing new trails in this area. The Nieman Journalism Lab reports that:

The Huffington Post applies A/B testing to some of its headlines. Readers are randomly shown one of two headlines for the same story. After five minutes, which is enough time for such a high-traffic site, the version with the most clicks becomes the wood that everyone sees.

NJL also reports that Huffington Post social media editor–and long-time Noisy Channel reader–Josh Young uses Twitter to help crowd-source better headlines.

I’m sure this approach must rattle some old-school journalists. And there is a real danger of optimizing for the wrong outcome. For example, including the word “sex” in this message might improve its traffic (the popularity of this post attests to that), but to what end?

Still, I don’t see this use of technology as cramping anyone’s style. Most of us write to be read–especially those in the media industry who are trying to monetize their audiences. Measurable success matters, and there’s no harm in trying to maximize it.

Recurring Tweets are a violation no matter how they are done, including whether or not someone pays you to have a special privilege. We don’t want to see any duplicate tweets whatsoever- They pollute Twitter, and tools shouldn’t be given to enable people to break the rules. Spinnable text seems to just be a way to bypass the rules against duplicate updates and essentially provides the same problems.

Hence, from Thursday, October 15th, 2009, 00:00 AM CST we will prevent the entry of recurring tweets on Twitter accounts within the SocialOomph system. Existing recurring tweets on Twitter accounts will all be placed in paused state at that time, so that the content of the tweet text is still accessible to you, but no publishing to Twitter of those tweets will take place.

Not everyone is thrilled with this new feature. My friend (and Noisy Channel reader) Eric Andersen notes: “this doesn’t make a lot of sense to me – many highly regarded Twitter users (e.g. @GuyKawasaki) regularly re-post tweets…primarily because of the “dip” model: re-posting the same tweet means more people will see, especially with an int’l audience.”

On one hand, I loathe inefficient communication, and I see repeated tweets as exposing the inefficiency of the dip model. We won’t get into my differences of opinion with Guy Kawasaki. If Twitter offered better search and control to users, then I think it would make sense for them to consider duplicate tweets as a spam issue.

On the other hand, Twitter search is crude. And the dip model, much as it may raise my personal hackles, is, in fact, what many users embrace. Twitter takes pride in letting users drive innovation, and I think they should be cautious about being too autocratic. Surely many of the people who post duplicate tweets do so with unspammy intentions.

Let’s face it: Twitter is going through growing pains, even if it just inherited the mother of all trust funds. They really do have to address spam. But they might consider doing so in a less heavy-handed way. I suspect that duplicate tweets are mainly a problem because they affect the statistics for Trending Topics–a problem they could easily address without prohibiting the tweets themselves. Better search would make it users to take charge of the user experience–a small dose of HCIR would go a long way.

I think Twitter has the best of intentions, and that it is confronting a real problem. I hope they work harder to find the right solution.

But the trickle that Twitter returns is hardly enough.

I believe this limitation is by design–that Twitter knows the value of such access and isn’t about to give it away. I just hope Twitter will figure out a way to provide this access for a price, and that an ecology of information access providers develops around it. Of course, if Google or Microsoft buys Twitter first, that probably won’t happen.

Now that Twitter has raised $100M at a valuation of $1B, I doubt any acquisition will happen anytime soon. But, according to Kara Swisher’s unnamed sources:

Twitter is in advanced talks with Microsoft and Google separately about striking data-mining deals, in which the companies would license a full feed from the microblogging service that could then be integrated into the results of their competing search engines.

If so, then it’s about time! How much either Microsoft or Google would pay for this feed is an interesting question. It’s probably not a coincidence that Twitter raised its last round of funding before pursuing this path–the revenue they obtain this way could be significant, but is unlikely to justify a $1B valuation.

In any case, I’m excited as a consumer that Twitter may finally allow Google and Microsoft to better expose the value of its content. But I’m also curious what my friends on the Twitter Search team think of the potential competition from the web search titans. Until now, no one has been able compete effectively with Twitter’s native search because of  lacking access to the firehose. Having such access would give Google and Microsoft more than a fighting chance. Given the centrality of search to Twitter’s user experience, it’s an interesting corporate strategy.

Another topic in that conversation was pseudonymity. Someone pointed to a 2000 USENIX paper entitled “Can Pseudonymity Really Guarantee Privacy?” The challenges of implementing pseudonymity have, of course, received lots of attention in the past few years. The most notorious example is the AOL search data scandal, which made the front page of the New York Times. But there’s also the work co-authored by my friend Vitaly Shmatikov on de-anonymizing Netflix data. Indeed, some have expressed concern that the new Netflix competition is a privacy lawsuit waiting to happen.

Finally, danah boyd‘s master’s thesis on “faceted id/entity: managing representation in a digital world” also came up–and I recently discovered by way of Robert Scoble that she’ll be keynoting at SXSW next year. Now I feel even more proud that I convinced her to speak at the SIGIR Industry Track this year. But I digress.

What does any of this have to do with copyright? Watch Lessig’s presentation–it’s long, but I promise you it’s worthwhile and entertaining to boot. Besides, I’ve made it easy by embedding it for you! He makes an analogy–rather, he makes fair use of Jonathan Zittrain‘s analogy–between privacy rights and copyright.

The executive (and overgeneralized) summary is that both privacy-holders (“consumers”) and copyright-holders (“industry”) have complained that technology has undermined their rights, and both have sought out legal remedies. Consumers push back on industry, frustrated with legal strategies to enforce copyright at the expense of consumer freedom, preferring instead to let technology dictate policy; industry pushes back on consumers, frustrated with their legal strategies to enforce privacy rights at the expense of industry freedom, in this case preferring instead to let technology dictate policy. The analogy may not be perfect, but it is close enough to be compelling.

But I’d like to stretch the analogy further than Lessig and Zittrain to consider pseudonymity and derivative works. The pseudonymity challenge (e.g., the recent reports about Project Gaydar) remind us that privacy isn’t binary, and that we have to accept at least some loss of privacy if we are going to live in a social world. Similarly, provisions like fair use exist because copyright is an inherent trade-off between protecting creators’ rights and embracing the value of creation in a social context.

As I said, I find the Zittrain’s analogy and Lessig’s presentation compelling. While it may not answer any of society’s urgent questions about privacy and copyright, it may at least further the conversation. At the very least, I hope the topic is intellectually stimulating.

Maggie Fox, founder and CEO of Social Media Group, will moderate a panel composed of Augie Ray, who blogs at Experience: The Blog) and is Managing Director of Experiential Marketing at interactive and social media agency Fullhouse; John Jantsch, who blogs at Duct Tape Marketing and is a marketing and digital technology coach; and yours truly. It’s free to attend; just register here.

Panel Presentations

• Usefulness as the Criterion for Evaluation of Interactive Information Retrieval
  Michael Cole, Jingjing Liu, Nicholas Belkin, Ralf Bierig, Jacek Gwizdka, Chang Liu, Jun Zhang and Xiangmin Zhang (Rutgers University)
• Modeling Searcher Frustration
  Henry Feild and James Allan (University of Massachusetts Amherst)
• Query Suggestions as Idea Tactics for Information Search
  Diane Kelly (University of North Carolina at Chapel Hill)
• I Come Not to Bury Cranfield, but to Praise It
  Ellen Voorhees (National Institute of Standards and Technology)
• Search Tasks and Their Role in Studies of Search Behaviors
  Barbara Wildemuth (University of North Carolina at Chapel Hill) and Luanne Freund (University of British Columbia)

Posters and Demonstrations

• Visual Interaction for Personalized Information Retrieval
  Jae-wook Ahn and Peter Brusilovsky (University of Pittsburgh)
• PuppyIR: Designing an Open Source Framework for Interactive Information Services for Children
  Leif Azzopardi (University of Glasgow), Richard Glassey (University of Glasgow), Mounia Lalmas (University of Glasgow), Tamara Polajnar (University of Glasgow) and Ian Ruthven (University of Strathclyde)
• Designing an Interactive Automatic Document Classification System
  Kirk Baker (Collexis)
• The HCI Browser Tool for Studying Web Search Behavior
  Robert Capra (University of North Carolina at Chapel Hill)
• A Graphic User Interface for Content and Structure Queries in XML Retrieval
  Juan M. Fernández-Luna, Luis M. de Campos, Juan F. Huete and Carlos J. Martin-Dancausa (University of Granada)
• Improving Search-Driven Development with Collaborative Information Retrieval Techniques
  Juan M. Fernández-Luna (University of Granada), Juan F. Huete (University of Granada), Ramiro Pérez-Vázquez (Universidad Central de Las Villas) and Julio C. Rodríguez-Cano (Universidad de Holguín)
• A visualization interface for interactive search refinement
  Fernando Figueira Filho (State University of Campinas), João Porto de Albuquerque (University of Sao Paulo), André Resende (State University of Campinas), Paulo Lício de Geus (State University of Campinas) and Gary Olson (University of California, Irvine)
• Cognitive Dimensions Analysis of Interfaces for Information Seeking
  Gene Golovchinsky (FX Palo Alto Laboratory, Inc.)
• Cognitive Load and Web Search Tasks
  Jacek Gwizdka (Rutgers University)
• Visualising Digital Video Libraries for TV Broadcasting Industry: A User-Centred Approach
  Mieke Haesen, Jan Meskens and Karin Coninx (Hasselt University)
• Log Based Analysis of How Faceted and Text Based Searching Interact in a Library Catalog Interface
  Bradley Hemminger (University of North Carolina), Xi Niu (University of North Carolina) and Cory Lown (NC State Libraries)
• Freebase Cubed: Text-based Collection Queries for Large, Richly Interconnected Data Sets
  David Huynh (Metaweb Technologies, Inc.)
• System Controlled Assistance for Improving Search Performance
  Bernard Jansen (Pennsylvania State University)
• Designing for Enterprise Search in a Global Organization
  Maria Johansson and Lina Westerling (Findwise AB)
• Cultural Differences in Information Behavior
  Anita Komlodi (University of Maryland Baltimore County) and Karoly Hercegfi (Budapest University of Technology and Economics)
• Adapting an Information Visualization Tool for Mobile Information Retrieval
  Sherry Koshman and Jae-wook Ahn (University of Pittsburgh)
• A Theoretical Framework for Subjective Relevance
  Katrina Muller and Diane Kelly (University of North Carolina)
• Query Reuse in Exploratory Search Tasks
  Chirag Shah and Gary Marchionini (University of North Carolina at Chapel Hill)
• Augmenting Cranfield-Style Evaluation with GOMS to Obtain Timed Predictions of User Performance
  Mark Smucker (Waterloo University)
• Text-To-Query: Suggesting Structured Analytics to Illustrate Textual Content
  Raphael Thollot (SAP Business Objects) and Marie-Aude Aufaure (Ecole Centrale Paris)
• The Information Availability Problem
  Daniel Tunkelang (Endeca)
• Exploratory Search Over Temporal Event Sequences: Novel Requirements, Operations, and a Process Model
  Taowei Wang, Krist Wongsuphasawat, Catherine Plaisant and Ben Shneiderman (University of Maryland)
• Keyword Search: Quite Exploratory Actually
  Max Wilson (Swansea University)
• Using Twitter to Assess Information Needs: Early Results
  Max Wilson (Swansea University)
• Integrating User-generated Content Description to Search Interface Design
  Kyunghye Yoon (SUNY Oswego)
• Ambiguity and Context-Aware Query Reformulation
  Hui Zhang (Indiana University)

I don’t know how much the audience took heed of his call, but he certainly followed his own advice. He and Endeca alum Mark Watkins just launched Goby, a vertical search engine that exhorts you to “create your own adventure”.  It’s fun–a sort of exploratory search for explorers. And it uses a deep web crawl to populate its index with semi-structured data.

Anyway, try it out! I’ve been in the private beta, but haven’t had the chance to see what they’ve been up to in the final stretch leading to the launch. You can also read more on Search Engine Land or CNET.

The morning kicked off with a series of 18 lighting talks in 90 minutes–that was 5 minutes apiece, with a ruthless gong for anyone who went overtime. The presentations were consistently intense, and I had the misfortune to follow one of the best talks–a very passionate presentation about crowd-sourced translation by IBM’s Uyi Stewart. Other notable presenters included design ninja Alexis Lloyd from the New York Times R&D Lab, Karrie Karahalios from the University of Illinois talking about the experimental WeMeddle Twitter client,  MIT Media Lab professor and Berkman Fellow Judith Donath showing a stunning gallery of “data portraits”, and Dragon Systems co-founder Janet Baker explaining how the brain recognizes speech–with an skull as a prop! The session was incredible, and I hope other conferences adopt this model.

After the coffee break, there was a session on Text Analysis in the Large, featuring Dan Gruhl (IBM), Gary King (Harvard), and David Ferrucci (IBM). Dan Gruhl talked about web-scale text analysis–a topic up his alley, considering his role in architecting the IBM WebFountain project. Gary King gave a fascinating talk about using ensemble methods to improve on existing clustering methods–the idea is to synthesize a collection of derived clusterings and place them in an explorable metric space. You can read the full paper here. But the winner for this session was definitely David Ferrucci, who described the work IBM Research is doing to develop a machine Jeopardy player. He spent much of the talk building a case for the difficulty of the problem–and then delivered the punchline: In less then three years of research, they’ve developed a machine player whose performance is comparable to that or jeopardy winners. Hopefully they’ll be competing on live television by next year!

After lunch, there was a session on Investigation, featuring MAPLight Research Director Emily Calhoun, UC Berkeley law professor Kevin Quinn, and Guardian news editor Simon Rogers. Emily Calhoun showed how MAPLight illuminates the connections between money and politics–it was great seeing data to correlate who supports and opposes bills with the associated campaign contributions from interest groups. Kevin Quinn’s presentation was a bit more technical, but his work reminds me a lot of Miles Efron’s work on estimating political orientation in web documents–but Quinn’s work is more general and goes beyond co-citation analysis to analyze the actual language of the documents. Great application of topic modeling! But my favorite presentation in this session was the one from Simon Rogers: he told the story of how the Guardian successfully crowd-sourced a project to investigate the expenses of UK Parliament members.

The final session was a panel discussion about how visualization might elevate or advance the debate over health care policy. The panelists were Ben Fry, Marti Hearst, Gary King, and Simon Rogers; Fernanda Viégas and Martin Wattenberg moderated. Unfortunately, the overwhelming sentiment from the panel was pessimism that anything we could do might actually lead to improved outcomes. Nonetheless, it’s clear that a lot of people are going to try.

Again, I want to thank Fernanda, Martin, Irene Greif, and everyone at IBM for organizing this fantastic event–and for inviting me to attend! I am impressed that anyone could manage to assemble such an impressive set of speakers in one place, and I appreciate the effort that everyone put into making the past two days so worthwhile. I look forward to seeing the videos available online, and I hope those who weren’t able to attend take the opportunity to watch some of them. I also encourage you to check out the live Twitter stream at #tt09 while it’s still available.

I’ll start with Deputy U.S. CTO Beth Noveck‘s keynote about the Open Government Initiative. First, the very existence of such an initiative is incredible, given the culture of secrecy traditionally associated with Washington. Second, I like the top priority of releasing raw data so that other people can work on analyzing it, visualizing it, and generally making it more accessible either to the general public or to particular interest groups. This is very much what I had in mind in January when I posted “Information Sharing We Can Believe In” and I’m glad to see tangible progress. I was never a big fan of faith-based initiatives.

The next session was a group of talks about watchdogs and accountability–people looking at how to ensure government transparency from the outside. New York Times editor Aron Pilhofer and software developer Jeremy Ashkenas talked about DocumentCloud, an ambitious project to enable exploratory search for news documents on the open web. Sunlight Foundation co-founder and executive director Ellen Miller offered a particularly compelling example of the power of visualization: a graph correlating the campaign contributions and earmark associated with a congressman under investigation. But my favorite presenter in this section was ProPublica‘s Amanda Michel, whose thoughts about a “human test of transparency” are worth a talk in themselves. For now, I recommend you look at the two projects she discussed: Stimulus Spot Check and Off the Bus.

After lunch, we shifted gears from government transparency to more of a focus on text. The first of the two afternoon sessions was entitled “Analyzing the Written Record” and featured Matthew Gray from Google Books, Tom Tague from Open Calais (a free text annotation service that almost all of the previous speakers raved about), and Ethan Zuckerman from Harvard’s Berkman Center. All of the talks were solid, but Ethan’s was outstanding. I blogged about his Media Cloud project back in March, but it’s come a long was in the past six months and is doing something I’ve been waiting years to see someone do: comparing how different news organizations select and cover news.

The final session was about visualization. David Small offered a presentation about literally transparent text that was, in the words of Marian Dörk, “refreshingly non-utilitarian and visually stimulating”. Ben Fry showed the power of visualizing changes in a document over time–specifically, a project called “the preservation of favoured traces” that illustrates  the evolution of Darwin’s On the Origin of Species. But, as expected, IBM’s Many Eyes researchers Fernanda Viégas and Martin Wattenberg stole the show with an incredibly informative and entertaining presentation about the visualization of repetition in text. No summary can do it justice, so I urge you to watch the video when it is available.

After all that, we enjoyed a nice reception at the IBM Center for Social Software. I’m incredibly grateful to IBM for organizing and sponsoring this event, and to Martin Wattenberg for being so kind as to invite me. I’ll try to earn my keep in my 5 minutes at the “Ignite-style” session tomorrow morning.

The Transparent Text symposium is a free event that will focus on ways to make large collections of documents understandable to laypeople and experts alike. We are interested in approaches that shed light on unstructured text, ranging from novel statistical techniques to web-based crowdsourcing.

The speaker list is impressive, ranging from familiar (at least to me) interface experts Ben Fry and Marti Hearst to social scientist Gary King and Sunlight Foundation Executive Director Ellen Miller. IBM also contributed some of its own researchers to the program, including David Ferrucci, who has been leading the Jeopardy project. There’s even an “Ignite-style” session where all attendees will have the opportunity to give five-minute presentations.

I’m looking forward to the eclectic mix of speakers and attendees. As Chris Dixon recently reminded us, it’s important to introduce some randomization into our intellectual diets so that we don’t get stuck in a rut of local optimization. While an event with a theme of transparency and interacting with textual information is hardly a detour for me, I am excited about the opportunity to hear a diversity of new perspectives on this topic. There will be videos of the speakers posted after the event, as well as a  Twitter stream at #tt09.

Of course, I’ll blog about what I learn and recycle it in the discussion activities at the HCIR workshop next month.

But sometimes I like to get away from all that and turn back to my roots inside the ivory tower. And that leads me to one of my favorite university blogs: the Haystack Blog.

The Haystack Blog is published by faculty and grad students in the MIT Computer Science and AI Lab (CSAIL)–specifically those in the Haystack group. Principal Investigator (and occasional dance instructor) David Karger is its most prolific blogger–you might have read some of his SIGIR 2009 posts or his debate with Stefano Mazzocchi about how to properly use RDF. But other people’s posts are just as interesting–check out the most recent post by Eirik Bakke about bridging the gap between spreadsheets and relational databases.

I wish that more universities and departments would encourage their faculty and students to blog. As Daniel Lemire has pointed out, it’s a great way for academic researchers to get their ideas out and build up their reputations and networks. He should know–he leads by example. Likewise, Haystack is setting a great example for university blogs, and is a credit to MIT and CSAIL.

I’ve been invited to participate in a webinar on the ethics of blogging that will take place Thursday, September 24th at 1 PM EST. It’s free to attend; just register online here.

Maggie Fox, founder and CEO of Social Media Group, will moderate. My two co-panelists are Augie Ray, who blogs at Experience: The Blog) and is Managing Director of Experiential Marketing at interactive and social media agency Fullhouse, and John Jantsch, who blogs at Duct Tape Marketing and is a marketing and digital technology coach.

Among the topics to be discussed:

• Transparency: How and when should a blogger reveal revenue sources?
• Pay for play: Blog posts, tweets, and more as marketing tools
• Online privacy
• Astroturfing: Organizations creating artificial “grassroots” campaigns
• Compliance and Legal: What should a corporate blog policy look like? What are a blogger’s legal obligations?

I hope some of you will be able to attend! Regardless, please use the comment thread make suggestions here about topics you’d like me to cover or concerns you’d like to see me address. I know that a lot of you have thought hard about these issues, and I’d like to ethically exploit your collective wisdom.

Their approach strikes me as a generalization of measuring retweets, but perhaps I’m giving it too cursory a read. I did compare their results to TunkRank: we at least agree that mashable is more influential than CNN–though even as simple a measure as follower count would confirm that judgment.

Anyway, I am delighted to see serious researchers looking at this problem. I’m still hoping to investigate hypotheses regarding TunkRank and friend:follower ratios.

Library scientists make a distinction between known-item and exploratory search. The former is about locating information: as an information seeker, you know the information exists, and you can even characterize it unambiguously; but the challenge is to convert that description into a location that allows you to retrieve the information. The latter is about discovery: you don’t know that the information you seek exists, and you may be sure of how to characterize what you are looking for–or even know what exactly you want until you’ve learned something about what is available.

These are extreme points on the information seeking spectrum, and most real-world tasks are in the middle, or combine subtasks of both types. For example, in physical libraries (yes, I’m that old!), I remember finding a book in the stacks and then browsing the nearby books in the hopes of serendipitous discovery. These days, I’d be more likely to scan its bibliography–or to look at the books and articles citing it. A known item can be an excellent entry point for exploration. Conversely, exploration can lead you to discover the existence of information that you then simply need to retrieve.

In common use, words like searching and finding cover this entire spectrum of information seeking activity. This breadth of meaning causes a lot of confusion. I’ve blogged about this before: “What is (Not) Search?“:

At the very least, I propose that we distinguish “search” as a problem from “search” as a solution. By the former, I mean the problem of information seeking, which is traditionally the domain of library and information scientists. By the latter, I mean the approach most commonly associated with information retrieval, in which a user enters a query into the system (typically as free text) and the system returns a set of objects that match the query, perhaps with different degrees of relevancy.

Back to Campbell’s article. His main points:

• Social networks have dramatically expanded our network of contacts.
• Search engine optimization (SEO) experts have killed their own game.
• The flow of information has changed: information now comes to us, rather than us having to go out and find it.

I like the spirit of the post, but I think he overstates his case. SEO isn’t all bad–in fact, it’s probably a key factor in Google’s effectiveness. And, while social networks enable social search in theory, and information does come to us; we are experiencing filter failure (Clay Shirky’s term) in a big way.

My conclusion: I agree with him about Google’s limitations–Google is primarily a locating tool, not a discovery tool. Unfortunately, I’m not persuaded that social networks and our theoretical ability to construct an ideal in-flow of information have actually delivered on the promise of more efficient information access. But I’m optimistic that we’ll eventually get there.

And I’ve dabbled a bit on the theoretical side myself. The TunkRank measure (I’m indebted to Jason Adams for his implementing it on a live site!) attempts to take the most basic assumption about our social behavior–the constraint that we have a finite attention budget–and explore its implications for influence over social networks. I have a few unexplored hypotheses queued up for when I can find the spare time to try validate them empirically!

But why settle for theory? We live in an age where social networks compete with web search (and perhaps complement search) as the hottest online technologies. If we’re not reading about Google vs. Bing, we’re reading about Facebook vs. Twitter, with LinkedIn offering a third way that seems to co-exist with its more storied peers. In this post, I’d like to focus on LinkedIn.

LinkedIn, despite its feature creep, is still fairly old-school: its raison d’être is for users to build, maintain, and exploit their professional networks. In theory, connections on LinkedIn represent present or past working relationships that become the basis for referrals–whether the goal is employment, sales, or partnership. LinkedIn is not the only professionally oriented social network, but at this point it’s certainly the dominant one.

But I’ve found at least two additional ways to use LinkedIn that I’d like to share:

Intelligence gathering. For reasons I don’t yet claim to understand, people share far more information about themselves–and in a much cleaner, structured form–on LinkedIn than in perhaps any other online medium. Most people’s resumes are not available online, but their LinkedIn profiles are tantamount to resumes. Moreover, their structured format makes it possible for LinkedIn to assemble aggregate profiles of companies, revealing composite pictures that must drive some of those companies’ legal and HR departments batty! At a higher level, LinkedIn also works well as a discovery tool–much more so now they’ve enabled faceted search. It’s still a bit tricky to explore people and companies by topic, but far more effective using LinkedIn than using any other tool I’m aware of.

Meeting new people. Cold-calling, spamming–pick your poison. In short, LinkedIn doesn’t have to only be about connecting with people you already know. But there’s an art to sending unsolicited messages: you have to pass the moral equivalent of a CAPTCHA by proving that your communication strategy isn’t indiscriminate. Let me use a personal example (that Maisha Walker was nice enough to write up in her Inc. magazine column). I decided that I wanted to find everyone on LinkedIn who might be interested in HCIR ’09. So I searched for everyone whose profiles indicated interests in both IR and HCI and sent out a targeted message (in fact, a invite with personalized message–a feature I recently feared they’d killed). The results were overwhelmingly positive. I’m not sure how many of the people I contacted will attend, but I raised awareness without inflicting annoyance. Better yet, one of the people I contacted then discovered I was looking for volunteers to review the draft of my book–and I thus obtained hours of help of someone who, just a day before, had never heard of me!

What intrigues me about LinkedIn (and other social networks) is the extent to which I am exploiting attention market inefficiencies (as LinkedIn may be doing as well). For example, LinkedIn makes it easy to send unsolicited invitations to anyone. Granted, you can lose this privilege by even having a couple of people respond to invitations with “I don’t know this person”. There’s also the question of why people’s social norms around disclosure are so different on LinkedIn than anywhere else–people not only post the content of their resumes, but go through the effort of providing it to LinkedIn in a structured form! Meanwhile, LinkedIn keeps tightfisted control over the information it aggregates–understandably, they recognize that this content is their most valuable asset.

People are still getting used to the idea of social networks. It will be interesting to see how their use evolves, particularly in term of information and attention market efficiency.

Since we didn’t manage to get to all of the questions (over 40–possibly over 50 counting the activity on Twitter!), we’re going to do a follow-up podcast that will be available even to people who didn’t attend the seminar. And, since even that might not be enough, I’m saving all of the questions as blog fodder.

To all who attended–and to Jared, Adam, and all the folks of UIE–thanks from me and Pete for giving us this great opportunity to connect with folks interested in faceted search and user experience.

But I noticed on Twitter today that Mark Williams (aka @Mr_LinkedIn) had noticed the same change and followed up on it with LinkedIn’s customer service department. I never assume any site behavior on a freely provided service is permanent, but it is starting to look like this is a deliberate decision and not a transient bug.

If so, it’s an annoying change, though I can see the merits. I’ve made heavy use of the connection message, especially when inviting someone I don’t know all that well–or don’t know at all. A personal message can be what distinguishes a welcome cold call from spam. But I’m guessing that others have abused that capability, filling it with spam or worse. Still, I feel like LinkedIn may be throwing the baby out with the bathwater. Will follow up if / when I hear more.

UPDATE: Just saw this message on the LinkedIn site via Twitter:

Unable to Personalize Invitation Message

Why can’t I personalize the message in my Invitation?

We are aware of an issue preventing some members from customizing their Invitation messages. There is no need to contact Customer Service as our team is reviewing the issue to determine the best overall solution.

As a temporary workaround, the following message (with your name in the signature) is being sent when you click on the ‘Send Invitation’ button: ‘I’d like to add you to my professional network on LinkedIn.’

As long as you approve of this message, you may continue to take advantage of this feature. If you prefer a more customized message to be sent, you may delay sending your Invitations until the functionality has been restored.

UPDATE #2: Looks like the problem is resolved.

Let’s say I know almost nothing about wines/digital cameras/cars and a search site offers me “options” to drill down. However, I can’t use those effectively and eventually it comes down to availability and price for me. My questions are what are your thoughts on these kinds of situations and is there a scientific explanation/theory on this case?

This may be why Google does not endorse faceted search except for experimental projects.

It’s a great question. There’s been a lot of research on how people make decisions when they have to manage trade-offs among multiple attributes, and the increasing interest in behavioral economics since Daniel Kahneman won the Nobel Prize in 2002 has helped some of that research has even percolated into the mainstream thanks to bestsellers like Freakonomics and Dan Ariely’s Predictable Irrationality.

The short answer is that there’s no point in offering users options that they can’t (or won’t) use effectively. Choice overload is certainly a problem, and our reaction to it is to satisfice, typically resorting to “fast and frugal” heuristics that throw out most of the potential decision criteria and instead focus on one or two attributes, e.g., price and availability.

But that’s no reason to dumb down the data we make available to decision makers. We make hard choices all the time, and fast and frugal can be horrendously suboptimal. We don’t hire employees based solely on their price and availability–or at least good employers don’t! For that matter, I don’t think most people pick wines that way, given that even Trader Joe has to diversify beyond “Two Buck Chuck“. And, while there’s probably more of a market for cheap cameras and cars, I’m pretty sure you’re an extreme outlier if you completely ignore other criteria.

That said, there are some caveats about exposing options to users. Faceted search is hard, especially on the open web. Take it from the folks at Microsoft Research–but I’m sure Googlers would be the first to agree, especially given their experience with projects like Google Squared that, while promising, are nowhere near ready for prime time.

I appreciate that Google is conservative about embracing faceted search–and HCIR in general. I’m actually impressed by the steadily improving quality of their related terms for search queries–even if they do hide them behind two clicks (show options -> related searches). Perhaps they’re feeling some pressure from Bing. But I think they’re largely following the dictum of “if it ain’t broke, don’t fix it”. Google is an extremely successful company. And, as Clayton Christensen argues, successful companies are great at incremental innovation and bad at disruptive innovation. As far as I can tell, faceted search is very disruptive to their model.

I’ve link-baited Summize founder and Twitter Chief Scientist Abdur Chowdhury here once or twice, but I understand that he’s no longer running Twitter Search. They’ve got a new guy, Doug Cook, as Director of Search.

This is great news, because Doug is someone who’s thought a lot about search and user experience. He was one of the early web search guys at Inktomi and also spent some time at Yahoo!, but what impresses me most is a project he’s pursued as a labor of love: Able Grape.

From their about page:

We’re a wine search engine — not for comparison shopping, but for learning and research. We aim to be the world’s most comprehensive, up-to-date, and authoritative source for online wine information.

Great, another vertical search engine, just what the world needs (unfortunately WordPress 2.8.4 doesn’t support sarcastic font). But seriously, Able Grape is worth a look, even if, like me, you are not a wine nerd. So wash your glasses and let’s have a quick tasting.

First off, Able Grape is not searching a proprietary document collection. Rather, it’s based on a focused crawl of “more than 38,000 sites and some 18 million pages.” In other words, Able Grape is in no position to ask anyone to add meta-data. Even at the site level, I doubt Doug had the time to customize the handling of content for each of 38,000 sites. In other words, there’s enough scale here to make the problem interesting.

Now let’s look at some examples of the site in action. I’m a fan of Spanish wines, so I’ll start with one of their example queries, tempranillo. The first page of results looks relevant to the topic, but so far that doesn’t distinguish them from Google, Yahoo, or Bing. What surprises me is that the “Filter by Region” offers regions outside of Spain–like California and even New York! Yes, I might have learned some of that from Wikipedia–though it would not even have occurred to me to ask about non-Spanish Tempranillo. That’s exploratory search and serendipitous discovery for you!

Let’s try a different example, this time not from their list. I like Malbec wines (which I associated with my maternal link to Argentina), but the only local wine region for me is the North Fork of Long Island. So here’s a search for north fork malbec, filtered to Long Island. It certainly gives me ideas of which wineries to check out on my next trip there. Though, to be fair to the competition, G/Y/B all handle this query pretty well–though none of them offer refinement by region to disambiguate “north fork”.

Able Grape has lots of cool features, ranging from how they handle multilingual content to clever use of constrained “wildcard” terms like anyvariety to match any wine variety (aka varietal). I suspect that there is much to learn from its design that applies to a broad variety (sorry!) of search applications.

I’m a wine dilletante, so it’s hard for me to spend too much time on this site without any deep-seated information needs to fulfill. But I’m a card-carrying member of searchaholics anonymous (well, maybe not so anonymous), and I’m impressed by what Doug’s done with this vertical.

Which brings us back to Twitter Search. Director of Search for Twitter is a high-profile, high-pressure job, even without Facebook nipping at Twitter’s heels. I’m sure Able Grape will ferment for a while as Doug devotes his creative energies to improving Twitter Search. I certainly hope he brings the same focus and sensitivity to his new endeavor and makes Twitter Search a grand cru of search engines.

• Facebook rolls out new version of search
• Facebook acquires FriendFeed
• Google testing new “caffeine” web search infrastructure

OK, perhaps that last item isn’t strictly search news, but it may as well be, given that the microblogging wars are in no small part about “real time” search.

I’m not a huge Facebook fan (as those of you who have looked at my spartan profile page may have noticed), but I am curious about how they’re implementing search over their sprawling collection of content. I’m underwhelmed with my own search experience on the site, but that might be my own fault for not being an active Facebook participant. Perhaps folks here who are more active can share their own experiences.

As for the acquisition of FriendFeed, I’m surely in good company to assume this was Facebook’s second choice after the attempt to acquire Twitter fell through. If, as has been reported. Facebook only paid $50M for FriendFeed, then the acquisition was pocket change compared to the $500M they offered Twitter  (granted, some or all of that being based on a controversial valuation of the Facebook). Anyway, it should keep life interesting in the status-sphere.

And then there’s Google’s preview site, which you can try here. The only difference I see between it and the non-preview Google search is that the estimated result counts tend to be slightly higher. The top-ranked results seem almost identical, modulo tiny permutations for the queries I checked, as do related searches and any other features I tried. But apparently that’s the idea:

The new infrastructure sits “under the hood” of Google’s search engine, which means that most users won’t notice a difference in search results. But web developers and power searchers might notice a few differences, so we’re opening up a web developer preview to collect feedback.

Anyway, it’s more fun reading all of this stuff than hearing the CEO of the web’s great search brands proclaim that her company has never been a search company–or wondering where all the great search people I know there will land as Yahoo search is assimilated into Bing. Don’t get me wrong, I’m looking forward to the competition between Google and Microsoft–one that I think will finally be waged in earnest. But I’m still sad for Yahoo and its employees.

Which bring us to the last news item: Doug Cutting is leaving Yahoo for Cloudera, where he’ll continue to work on Hadoop. According to his blog post about it, “This move will not fundamentally change my day-to-day activities.” It will certainly be interesting to see what comes next from someone who has been instrumental to so many major open-source packages associated with search.

I’m strongly if not absolutely in favor of freedom of expression, so it’s hard not to find such cases depressing. Nonetheless, I don’t think  the legal landscape hasn’t changed.

Rather, what has changed (or accelerated) is that:

• It is easier for people to express themselves publicly–and hence far more people are doing it.
• The detached nature of online communication releases people’s inhibitions. Moreover, people not only don’t self-censor, but in some cases are deliberately provocative to attract attention.
• The speed and efficiency of distribution (especially through search / alerts) means that the people most likely to be or feel damaged by an act of public expression are far more likely to discover that act.

So it’s not surprising that users are being sued for what they say online–it’s an expected consequence of the democratization of publishing, especially in the litigious English-speaking countries on both sides of the pond.

I’d personally like to see it a higher bar for someone to initiate a defamation lawsuit–let alone win it–but I’m not holding my breath. Instead, I expect that we’ll see more anonymous expression by people who don’t feel the authenticity of disclosure justified the risk of retaliation. Oh well.

Tuesday morning, the second day of the conference and the day before the Industry Track, I woke up to an email from Tip House of the OCLC, whom I’d planned to have speak about his experiences developing Worldcat.org, the world’s largest bibliographic database. Unfortuantely, he had fallen ill and would not be able to make it to the conference.

I was determined not to have a hole in the program. I immediate sent an email to the Director of Search at LinkedIn, whom I had just met at the poster session the previous evening, hoping he might have a presentation tucked away about LinkedIn’s recent launch of faceted people search. I turned to Twitter–which actually earned me a plausible suggestion.

But it was during the morning coffee break that serendipity struck. As I walked by the Bing exhibitor table, I saw Jan Pedersen, Chief Scientist of Core Search at Microsoft, chatting with Peter Bailey, an applied researcher on the Bing team. I turned to them and, in my most charming voice, asked if they might be interested in having someone on their team taking about Bing the next day. They took a few minutes to think it over, and then replied in the affirmative, producing Nick Craswell, also an applied researcher. Problem solved, and I can proudly say that I Binged for it!

Nick talked about how query modeling, focusing issues like query ambiguty, session context, and temporal query dynamics (particularly seasonality). He talked a bit about a technique that involved random walks on click logs–a technique I remember striking me when I first heard him talk about it at ECIR 2008.

The talk was a bit raw–understanably so given the short notice. But it was great to see a major web search practitioner connecting information retrieval research to actual product. Yes, there were the standard caveats about not revealing secret sauce, but the talk was open and substantive. Indeed, I hope Nick will be able to share the slides!

UPDATE: Nick emailed me the slides and gave me permission to post them here.

It’s nice to see this piece joining a growing chorus questioning the way we conflate the distinct concerns of disseminating knowledge, establishing professional reputation, and building community. This problem is not unique to computer science, but we are certainly in a position to lead by example in addressing it.

In age where distribution is nearly free, I agree that we should move the filtering role from content publishers to content consumers. There’s no economic reason today why scholarship (or purported scholarship) shouldn’t be published online. Of course, the ability to publish digital content for free (or close to free) does not imply anyone will (or should) read what you write. The blogosphere offers an instructive example: the overwhelming majority of blogs attract few (if any) readers. I suspect that the same holds true for arXiv.org. Of course, peer-reviewed content may not fare that much better, particularly given the proliferation of peer-reviewed venues. Regardless, it makes no sense for publishers to act as filters in an age of nearly-free digital distribution.

That brings us to the question of how researchers should establish their professional reputation–and, in the case of academics, obtain tenure and promotion. Today, they have to publish in peer-reviewed journals and conferences. Even if we accept the weaknesses of the current peer-review regime, we should be able to separate content assessment from distribution. The peer-review process (and review processes in general) should serve to endorse content–and ideally even to improve it–rather than to filter it.

Finally, conferences should primarily serve to build community. I find the main value of conferences and workshops to be face-to-face interaction, and I’ve heard many people express similar sentiments. Part of the problem is that so few presenters at conferences invest in (or have the skills for) delivering strong presentations. But more fundamentally it’s not even clear that the presentations are the point of a conference–after all, an author’s main motive for submitting an article to a conference seems to be getting it into the proceedings.

Here are some questions I’d like to suggest we consider as a community:

What if presentation at a conference were optional, and an author’s decision to present had no effect on inclusion in the proceedings? Would there be significantly fewer presentations? Would those fewer presentation be of higher quality?

What if the process of peer-reviewing conference submissions required the submission of presentation materials rather than (or in addition to) a paper? Would the accepted presentations be of higher quality? Would researchers invest more in presentation skills? What would happen to strong researchers without such skills?

Can we update the traditional conference format to foster more productive interaction among researchers? For example, should we have more poster sessions and fewer paper presentations?

I’d love to see the computer science community take the lead in evolving what increasingly feel like dated procedures for disseminating knowledge, establishing professional reputation, and building community. I’ve tried to do my small part, co-organizing workshops on Human-Computer Interaction and Information Retrieval (HCIR) that emphasize face-to-face interaction and organizing the SIGIR 2009 Industry Track as a series of invited talks and panels from strong presenters. But I’m encouraged to see “establishment” types like Moshe Vardi and Lance Fortnow leading the charge to question the status quo.

Last December (New Year’s Eve, to be precise), I started recruiting speakers. I started with a list of topics I wanted to see covered, and one of those topics was spam / adversarial information retrieval. My top two choices were Matt Cutts and Amit Singhal, both members of the Search Quality group at Google. I’d heard Amit speak before: he delivered one of the keynotes at ECIR 2008 (and inspired one of my first blog posts!). So I decided to aim for Matt Cutts, despite having no way to contact him (the head of Google’s Webspam team is understandably a bit protective of his personal email address). And, just two weeks later, I had Matt locked in to the program.

Matt was an incredible speaker, and he had the unenviable task of opening the Industry Track at 8:30 AM, the morning after the banquet. His title, “WebSpam and Adversarial IR: The Road Ahead”, gave him a fair amount of maneuvering room, and he used his 45 minutes to give the audience a peek into his world.

He opened the talk by inducing the audience to try to think like a spammer. He then game examples of social engineering attacks, to put us in a “black hat” mindset. He also pointed out the danger of punishing sites with spammy inlinks: people and companies would use this knowledge against their competitors / enemies (the practice has been called “Google bowling“).

He then moved on to examples of spam techniques. He showed examples of pages whose spaminess is only detectable by parsing JavaScript, something I wasn’t aware that Google could do (though apparently this has been public knowledge for a while). The theoretical computer scientist in me wonders about using random self-reducibility as obfuscation on steroids, but hopefully spammers aren’t quite that sophisticated yet!

He offered a common-sense framework for fighting spam: reduce the return on investment. Unfortuately, he sees a trend in spam where spammers are aiming for faster, higher payoffs by hacking sites and installing malware. Indeed, the democratizing effect of social media means that a lot more people have pages that can serve spam, including their Twitter and Facebook pages. He invited the information retrieval community to invest effort in learning how to automatically detect  that a page or server has been hacked.

My only quibble with the talk is that Matt did not discuss the inherent subjectivity of spam. Sure, there are many cases that are black and white, but ultimately spam (like relevance) is in the eye of the user. I’d love to see more use of techniques like attention bond mechanisms that accommodate a subjective definition of spam, e.g., “any email that you would rather have not received.”

But I quibble. Matt delivered an excellent talk to a packed audience, and it was a real privilege to have him kick off the Industry Track.

ps. You can also read Jeff Dalton’s notes on Matt’s presentation.

The second day of SIGIR (Tuesday) started straight off with research talks. I went to the web retrieval session, which consisted of two talks about anchor text and one about privacy-preserving link analysis.

“Building Enriched Document Representations using Aggregated Anchor Text“, by Don Metzler and colleagues at Yahoo Labs. They address the challenge of anchor text sparsity (the distribution of in-links for web pages follows a power law) by enriching document representation through aggregation of anchor text along the web graph. Their technique is intuitive, and the authors demonstrate statistically significant improvements in retrieval effectiveness. Unfortunately, their results are not repeatable, since used a proprietary test collection to obtain them.

The second talk of the session, “Using Anchor Texts with Their Hyperlink Structure for Web Search“, was by a group of authors from Microsoft Research Asia. They address the opposite problem of the previous paper: how to handle too much, rather than too little, anchor text. Specifically, they model dependence among multiple anchor texts associated with the same target document. Like the Yahoo folks, they demonstrate statistically significant results on a proprietary test collection.

The third talk, “Link Analysis for Private Weighted Graphs” (ACM DL subscribers only) by Jun Sakuma (University of Tsukuba) and Shigenobu Kobayashi (Tokyo Institute of Technology), was a bit of an outlier, if one can call a paper in a three-paper session an outlier. The authors offer privacy-preserving expansions of PageRank and HITS, the best-known link analysis methods associated with relevance and authority in web search. I’ve noticed an increasing number of papers like these that mix cryptography with information retrieval or database concerns. One of my frustrations in reading such papers is that I always suspect that people are re-inventing wheels because so few people are able to keep up with research in multiple disciplines.

Then I had the coffee break to solve my own research problem: how to fill the 11:30 slot in the Wednesday Industry Track, since a speaker called in sick that morning. When I walked by the Bing table, I saw Jan Pedersen (Chief Scientist for Core Search at Microsoft), and I begged him to help me out. I must have been a persuasive supplicant, because he procured me Nick Craswell, an applied researcher who works on Bing. Out of gratitude for this 11th-hour favor, I wore a Bing t-shirt all day yesterday as I went wine-tasting. Bing drinking, not binge drinking!

Anyway, that urgent problem resolved, I went back to enjoying the conference. For the second morning session, I went to the vertical search session.

As it turns out, that session kicked off the with SIGIR Best Paper winner: “Sources of Evidence for Vertical Selection” by Jaime Arguello (CMU), Fernando Diaz (Yahoo), Jamie Callan (CMU), and Jean-François Crespo (Yahoo). The authors do a lot of things I like: they apply query clarity as a performance predictor, and they bootstrap on an external collection (specifically Wikipedia). The test collection they use for evaluation is proprietary, but that seems to be the price (at least today) of doing this kind of work.

The second talk of the session was by a subset of the previous paper’s authors: “Adaptation of Offline Vertical Selection Predictions in the Presence of User Feedback” by Fernando Diaz and Jaime Arguello. The authors creatively used simulation to evalaute their approach. They did a nice job, but I have to admit I’m skeptical of results about feedback that aren’t based on user studies.

Unfortunately, I missed the third talk of the session because I had to play organizer. But I must have earned some good karma, because I got to enjoy a delightful lunch with Marti Hearst and David Grossman.

Stay tuned for more posts about the interactive search session, the keynote by Albert-László Barabási, the banquet at the JFK Presidential Library and Museum, and of course the Industry Track.

I arrived on Sunday evening, and the reception was like Cheers (“where everyone knows your name“)–only that, at least in my case, I was meeting many people face-to-face for the first time in years, and in some cases for the first time, period! I reconnected with some of the SIGIR regulars whom I’d missed last year (Singapore was a bit far for me), finally met my editor, Diane Cerra from Morgan & Claypool, and even ran into someone who is evaluating my company’s technology. And that was just Day 0.

Day 1 started bright and early with the 7:00 am newcomer’s breakfast, which brings together newcomers and “old hands”. I believe my role as organizer qualified my as an “old hand”, even though this is only my third SIGIR. Which might explain why Justin Zobel, a real old hand (and one of this year’s program chairs) joined my table. Of course, he hadn’t read my post about his recent SIGIR Forum essay, so we chatted a bit about recall. Not surprisingly, we mostly agreed, and I have to give credit to the essay for provoking that and other good discussions today.

Then the conference started in earnest, with Liz Liddy bestowing  the Salton Award to Susan Dumais. In the tradition of the award, Sue delivered a keynote recounting her personal journey through the space of information retrieval. I was thrilled that her recognition called out her working to bring together information retrieval and human-computer interaction. Of course some of us were ahead of the curve by recruiting her as the keynote for HCIR ’08. Of course, I asked her a question about why transparency, which she called out as a reason that users in her Stuff I’ve Seen work preferred to explicitly sort results by date rather than accept the systems best-first relevance ranking, was so absent in web search. Her answer was interesting: she feels that transparency is most useful for re-finding, and least useful for discovery. I’m not sure I agree with that explanation, but I’ll at least think about it a bit before I commit to disagreeing with it.

Some coffee, and then off to the first session of research papers. The presentation that stood out for me in this session was “Refined Experts“, presented by Paul Bennett. The paper offers a nice technique for improve on hierarchical classification (by addressing the problems of error propagation through the hierarchy and the inherent non-linearity of hierarchies), and Paul is an outstanding presenter.

Then Diane Cerra and Gary Marchionini took the Morgan & Claypool authors (and a few authors-to-be) to lunch at Brasserie Jo. Great food, and even better company. My only regret is that I missed one of the talks in the first session after lunch, “A Statistical Comparison of Tag and Query Logs“. I did like David Carmel‘s talk on “Enhancing Cluster Labeling Using Wikipedia” in that same session, though I’ll need to do some homework to figure out what distinguishes it from other work in this area, such as an ICDE 2008 paper by Wisam Dakka and Panos Ipeirotis on “Automatic Extraction of Useful Facet Hierarchies from Text Databases“.

In the following session, I attended a couple of the efficiency talks. The talks were well presented, but in both cases I wondered if they were addressing the right problems. I’ve felt this way before at SIGIR efficiency talks, so perhaps my tastes are just idiosyncratic.

Then came the poster / demo reception. Even with three hours, there was far too much to take in–and of course that session is as much about networking as it is about the posters and demos. I enjoyed the three hours, but I’ll have to go back to the proceedings to learn more about what I saw–and what I missed.

Finally, I wrapped up by leading a crew to Tapeo for dinner–apparently a popular choice for attendees, since another table of 6 arrived shortly afterward. It was a nice cap to a fantastic but exhausting day.

I can’t promise I’ll keep this up daily, but I will blog about the rest of the conference when I have the chance. Meanwhile, here are some other folks blogging about SIGIR ’09:

• David Karger is live blogging on the Haystack blog.
• Jon Elsas is blogging and posting some pictures.
• Jeff Dalton posted detailed notes about Sue’s keynote.
• Mary McKenna is blogging at the SemanticHacker blog.
• John Battelle and Gene Golovchinsky couldn’t attend, but have both blogged about the conference.

And follow on Twitter. The preferred hashtag is #sigir09, but I follow sigir OR sigir09 to be safe.

If you’re attending, I hope you’ll find me and say hi–after over a year of blogging, there are far too many people I’ve gotten to know but never met face to face! If you’re not attending, then I encourage you to follow the coverage on Twitter. Since there seems to be some confusion about which hashtag to use, I suggest you follow sigir OR sigir09 OR sigir2009 (yes, there is sometimes value to favoring recall). I promise to blog about it when I get back, but I hope you’ll forgive me if The Noisy Channel is a bit quiet over the next few days.

In any case, I wish I’d known about the beta launch earlier today, when I was looking for Boston-area colleagues to help me publicize the SIGIR Industry Track! The current interface is much more supportive of exploration.

It’s a nice implementation. The interface lets you refine the text search results by location, relationship (1st degree, 2nd degree, group, and other), industry, current / past company, and school. For a facet with a large number of values, like company, the interface only displays the top 10 values, and then lets you use type-ahead to refine by other companies. Unfortunately, the type-ahead was a bit buggy for me–but hey, it is a beta.

The application is fairly responsive, even for my search for “software”, which returns 2.4M results, 120K of which are 2nd-degree connections. Other than at Endeca, I haven’t seen anyone else mix faceted search with social networks, and LinkedIn has done a nice job of it.

So, if anyone from LinkedIn is reading this, congratulations and welcome to the wonderful world of faceted search. Count me a delighted customer. I hope my enthusiasm today makes up for my past criticism.

• No to SQL? Anti-database movement gains steam
• A Comparison of Open Source Search Engines
• Introducing the Google Chrome OS

Quick reactions:

Regarding the anti-SQL movement, I would have thought the main complaint would be that SQL is too arcane a language for ordinary users to ever use it directly. Instead, the article discusses developers’ complaints about databases, and these are mostly about price, speed, and scale. Evidently even free, open-source databases like MySQL are losing favor relative to tools like Hadoop and Hypertable that don’t offer support for SQL. Of course, this picture comes from a meetup of 150 people that might not be entirely representative of information technology workers.

I know first-hand from my experience at Endeca that, to quote Michael Stonebraker, the “one size fits all” approach to databases is an idea whose time has come and gone. At Endeca, we have built our own special-purpose database to address information needs ill-served by the available OLTP and OLAP technologies. Still, I think it’s premature to declare the death of SQL or of relational databases. But why let that stand in the way of a good story?

On to the open-source search engine comparison. I won’t rehash the critique of the study, which you can find in the 80+ comments from folks like Jeff Dalton, Bob Carpenter, and Otis Gospodnetic. Perhaps the most salient point is that it’s not clear how much sense it makes to perform “out of the box” evaluations. In any case, my impression is that Lucene is by far the dominant player in the open-source search space; the study, if it has any effect, will only be to reinforce that dominance.

And finally, the big news from the big G: a Google Operating System. Even my mom (who couldn’t name an existing operating system) was asking me about it, so clearly this one has made it into the mainstream media. And yet I don’t see why this is such a big deal. We have netbooks, and we even have Linux-based netbooks. As far as I’ve heard, the latter are popular with geeks and cheapskates, but that’s about it–most people are willing to pony up the few extra dollars for Windows XP. Will Google launching a netbook-oriented OS significantly affect this market? I suspect the only route to success is if they meet non-technical users’ needs (browsing, email, media, light document editing) while minimizing their overhead (maintenance, security, compatibility). Will they be a better Ubuntu? Perhaps, much in the way that Chrome is trying to be a better Firefox. Why Google choose to build its own free, open-source products rather than contribute to mature open-source projects is a mystery to me, but it’s their money and time to spend.

I think that cover’s the week’s big stories–or at least those that matter most to Noisy Channel readers. Somehow I didn’t manage to come up with an IR / HCIR angle on the Michael Jackson story, or perhaps it’s just that Danny Sullivan beat me to it.

Anyway, I’m back in the saddle, and should soon be back to my normal posting volume. Thank you all for being patient.

The first is Panjiva, a startup that I’m advising. You can read more about them here. They are looking for a hands-on developer (yes, someone who can code) with background in information retrieval or data mining. The job is in Cambridge, MA, and they want someone local. Check out their jobs page.

The second is Twitter. Yes, you’ve heard of them. What you might not know if that they’re aggressively hiring in their search group. Apparently the company is growing–I’d thought they were at ~30 people, but I just did a reference call for someone and learned that they’ve doubled in the past few months. I have no stake in Twitter except as a user, but I’d love to see them improve their search capabilities. So, if you’re in or near San Francisco and looking for a search job on the bleeding edge, check it out.

Other folks who are trying to hire people with search / information retrieval background: I encourage you to post opportunities in the comments!

Evidently Luis von Ahn used his keynote to unveil MonoLingo, a human-powered system for translation, but only using people that know one language (no idea if he used the old joke).

According to Panos:

Monolingo relies on the fact that machine translation is pretty good at this point, but not perfect. So MonoLingo starts by by translating each word using a dictionary, giving multiple interpretations for each word. The human then (who is a native speaker of the target language) selects the translation for each word and forms the sentence that makes most sense.

I’m curious to hear more: as of this writing, the site is password-protected with no further information.

Just a few days ago Habitat, a furniture store, used the trending topics as hashtags–including one associated with the disputed Iran election–to pimp their “totally desirable Spring collection”. It made for a great case study in how not to use Twitter.

And today I see that the top two trending topics are What McFLY Song Are and TweetBoard Alpha, both edging out #iranelection. The first spams through a quiz; the second through a request for invitations. It’s enough to make you want to scream, Stop Twitter Spam!

Of course, the solution may be to ignore the trending topics, which we can now see are easily gamed. Even when they’re legitimate, the topics aren’t necessarily all that useful. In the Twitterquake of Michael Jackson’s death, nine of the top ten trending topics related to the Gloved One–one of them even misspelled as Micheal. the tenth related to a hoax that Jeff Goldblum had died.

As I’ve said before, I actually look forward to a spamageddon that forces us to confront the attention scarcity problem head-on. At this rate, perhaps I won’t have to wait much longer.

Apparently Michael Arrington got the email too. But Aardvark’s larger PR coup is a feature in the Business section of the Sunday New York Times, entitled “Now All Your Friends Are in the Answer Business“.

I’ve blogged about Aardvark a bit–see my previous pair of posts about the blog + Twitter vs. Aardvark challenge. I like the idea of expert-mediated information seeking, though I have at least two concerns with Aardvark.

The first is in how Aardvark routes questions to experts–I’ve had mixed results that I attribute to the inherent challenge of inferring a topic through natural language processing. I think Aardvark would to well to offer guidance to users both in volunteering their own areas of expertise and in specifying their query topics.

The second is that questions and answers are private. I’m a big fan of “when in doubt, make it public“–and this is a clear-cut case where public is at least the right default. I’m curious how often people ask questions that someone else has already answered. Yes, there’s something to be said about getting an answer from someone in your own social network. But I don’t see any reason that the correspondence has to be private–especially for a question that you’re willing to have routed to a total stranger.

I hope Aardvark addresses both of these concerns, improving its routing and publishing question-answer pairs. As I’ve mentioned in recent posts, I think social search deserves a lot more attention than real-time search, and it’s great to see startups like Aardvark and Hunch working on it.

In practice, I’m pretty skeptical, as is Gwen Robinson at the Financial Times Alphaville blog. She writes:

That’s very interesting, because several hedge fund managers we spoke to dismissed the idea variously as “all twatter” and “rubbish” – not least because Twitter has carved a reputation more for unfounded speculation and even sensational disinformation than for ground-breaking, market-moving alerts for alpha-hungry fund managers.

I’ll concede that time really is money for for hedge funds and other traders who need to make decisions before the rest of the market catches up. But I’m dubious that Twitter–let alone an automated processing of tweets–will enable traders to make better decisions. Moreover, any success would immediately be gamed, along the lines of pump and dump scams. I suppose that hasn’t put a damper on the popularity of StockTwits, but popularity does not necessarily translate to profitability for the traders. I hear that Swoopo (a great example of exploiting behavioral economics) is popular too.

If real-time search is to be useful–and I think it really should be called alerting–then the information it provides has to have some sort of quality assurance, and not just freshness. There’s almost certainly a trade-off, since it usually takes time to vet  information for quality, even if the vetting is through crowdsourcing. But that reality doesn’t seem to have sunk in yet for the real-time advocates. I say, give it time.

I love the headlines from Techmeme:

• Mashable!: Collecta: True Real-Time Social Search
• paidContent: Startup Promises Best Real-Time Search Results Yet
• Tech Beat: Collecta Launches *Really* Real-Time Search Engine
• ReadWriteWeb: Collecta: Summize Backer Launches Broader Real-Time Search
• Search Engine Land: Collecta And CrowdEye Join The “Real Time” Search Club
• The Register: Collecta – real-time search in real-time
• VentureBeat: Collecta says it’s the fastest contender in the real-time search race
• TechCrunch: Collecta Enters The Real Time Search Wars
• AltSearchEngines: Top Story: Collecta launches real-time search engine!

Yes, folks, it’s really, really, real-time! Of course Twitter and Facebook have their own real-time search offerings. And apparently Google, Yahoo, and Microsoft are looking hard at real-time too.

I concede that there’s something in this real-time mania. I’ve live-tweeted events, and I’ve followed others who were doing so. I certainly read current news and blogs–as they say, today’s newspaper wraps tomorrow’s fish (someone will have to translate the expression for folks who’ve never read an analog newspaper). But yes, recency / freshness  is a certainly a concern in information seeking.

But it’s not the only one, and I doubt it’s the dominant one. Moreover, the dismissal of web search engines as if their index contents are ancient history is preposterous. Search for iran election on Google, Yahoo, or Bing, and you see a lot of current news. I suppose Twitter offers more recently generated bits, but the main virtue there is not the immediacy–rather, it’s the social nature of the content. For example, a number of people are following @persiankiwi for a personal perspective. I’ll let you decide for yourselves if Collecta or Crowdeye offer something new or valuable–I’m still waiting for the former to show me anything at all!

I know that the technology press likes new buzzwords, and “real-time” search is surely the buzzword du jour, even giving “semantic” search a run for its money. And I understand how many in the blogosphere feel it is their moral duty to cheer on any start-up that makes a go at disrupting the current regime. But I wish these folks would evaluate the new entrants on their merits, rather than simply on the drama of the David vs. Goliath story.

I understand what it’s like on the startup side–it wasn’t that long ago that few people outside the Boston-area technology scene had heard of Endeca. For a long time, I was jealous of people whose companies had generated more buzz. But, in retrospect, I’m at least glad that my colleagues and I had a chance to build a robust product before the press noticed us. Overenthusiastic press isn’t necessarily a good thing, as I’m sure a line-up of prematurely crowned Google killers can attest.

In that spirit, I hope that CrowdEye and Collecta bring something interesting to the market. But I doubt that “real-time” search will cut it, especially if it’s not ready for prime time.

At first glance, Twitter seems immune from this sort of spamming, since you only see tweets from the users you follow. Yes, Barack Obama and Guy Kawasaki must spend a lot of time on Twitter! But, regardless of how many users you follow, you are the one in control.

At least that’s the theory. Of course, things tend to work a bit differently in practice. Like many Twitter users, I use Twitter Search to maintain a running vanity query for mentions of my user name, employer, blog, etc. As a result, a user I don’t follow can nonetheless get my attention by tweeting an “at reply” to me. Twitter has struggled to figure out whether that is a good thing or a bad thing, but I suspect that my erring on the side of vanity is a common behavior.

But I do recognize that I’m opening myself up to alert spamming–perhaps not just in theory, but in practice. Today I read on All Things Digital that:

Pontiflex, a lead generation startup that hoovers up names and other other info from users that visit its network of publishers, then sells the data to marketers. The Brooklyn-based company is rolling out a Twitter product that lets marketers compile a list of interested Twitter users.
…
Since the users aren’t actually signing up to “follow” any of the marketers, said marketers can’t send them direct messages. The marketers could try to “at reply” their leads — the equivalent of shouting out the name of someone you think might be at a loud cocktail party, but who you can’t actually see. But that’s about it.

That’s about enough, if enough users are like me. Fortunately, I’m not enough of a celebrity to be particularly concerned about being singled out–at this stage. But I think the writing is on the wall, and spammers will innovate to embrace social media. I’ve already experienced a few examples of such innovation, and I’m sure that they are child’s play compared to what’s in store.

Personally, I look forward to this spamageddon. Why? Because I think we already have a problem managing attention scarcity in social media, but haven’t found sufficient motivation to confront the problem head on. A spam epidemic will certainly cause us to revisit our priorities, and I’m optimistic that we’ll innovate beyond the existing approaches used for email spam.

But be warned, the site may be a bit sluggish. They’re certainly getting bombarded with traffic from the blogosphere and Twitterverse!

Sergey Brin is so rattled by the launch of Microsoft’s rival search engine that he has assembled a team of top engineers to work on urgent upgrades to his Web service.

I never imagined that anyone would get their technology news from the New York Post, but evidently it’s well read in the blogosphere. Techmeme reports the following articles as citing the New York Post article:

• Larry Dignan / Between the Lines: Google missed a marketing turn with the ‘decision engine’ thing
• Steven Musil / CNET News: Does Microsoft’s Bing have Google running scared?
• Bob Caswell: It’s official: I now use Bing instead of Google
• Amit Chowdhry / Pulse2: Bing Increases Microsoft’s Market Share; Upsets Google’s Sergey Brin
• Ben Parr / Mashable!: Google vs. Bing Battle Heating Up: Is Google Scared?
• Tony Hung / Deep Jive Interests: How Many Geeks Are Even Trying Bing?

I know that the press loves a good fight, and in technology it’s hard to ask for a better pairing than Google and Microsoft. Moreover, I do think that Google should be paying attention to Microsoft’s positioning of Bing, regardless of how well Microsoft has delivered on that positioning. In any case, it makes sense for Google to keep close tabs on its competitors. After all, even a fraction of a percent of web search market share translates into millions–more than enough revenue to justify a few full-time employees.

Still, to assert that Google is gripped with fear stretches credibility, even for a tabloid. I don’t mean to suggest that Google is so self-confident as to be fearless. Google may well have reacted with fear when it looked like Microsoft would acquire Yahoo–in fact, some have suggested that Google’s proposed (but ultimately abandoned) advertising deal with Yahoo was a Machiavellian maneuver to scuttle the acquisition.

But, unless I’m missing something, Bing simply isn’t a threat to Google’s market dominance. If anyone should be concerned, it’s folks like Kayak who might lose some market share to Bing’s travel search–which seems to be generally acknowledged as Bing’s strongest vertical.

Personally, after being underwhelmed by Bing, I decided to try it for 2 weeks. I made it for about a week and a half, and you can see some of my commentary on Twitter. I stand by initial impression: it’s not bad, but it’s noticeably inferior to Google, and even parity is not enough to reverse the tide. Perhaps the tiny gain–or the slowdown in loss–that they will make in market share will justify their investment. But this is no revolution, and the Gevil Empire is not running scared.

Yesterday, I was fortunate to attend a presentation from a Google Engineering Director about Google Wave, an online communication and collaboration tool that Google recently unveiled at the Google I/O developer conference. For those who, like me, were unable to attend I/O, Google has posted the entire 80-minute presentation on YouTube (embedded above). For those of you without 80 minutes to spare, Gina Trapani has assembled a highlight reel.

The pitch is that email, the most popular technology for online communication, is a 40 years old and needs an overhaul to reflect the opportunities of an always-on world. They also emphasize that everything they’ve done works inside the browser.

The video is sexy, showing off both the real-time updating capabilities of Wave (blurring the lines between email and instant messaging) and the ability to support structure more cleanly than email (e.g., responding to only part of an email). The conversation model is also nice: for example, participants can bring someone new into a conversation, and that new person can access the evolution of a conversation (a sort of retroactive cc). Indeed, Wave looks more like Basecamp than like email.

Google is pitching Wave to developers–they even stole a page from Oprah and gave every Google I/O attendee a new Android phone in order to develop applications using their early-access Wave accounts. I haven’t studied the APIs, but the object model seems reasonable, ranging from a “blip” (a low-level event associated with content, possibly as fine-grained as someone typing a single character) to “wavelets” (the sub-conversations that comprise a wave) to of course the wave itself. And, given that the team is led by the folks who developed Google Maps, I have no doubt that they understand how to play well with mash-ups.

But I’m left with two big questions.

The first is what it would feel like to access this rich structured history of conversation. The search interface feels a lot like Gmail’s–and I don’t mean that as a compliment. I use Gmail, but I curse every time I have to deal with managing search results that include large conversational threads. I think there will be a lot of challenges for managing search results, and I’m curious how Google, with its historically spartan approach to search interfaces, will address them.

The second is about interoperability. For all of the openness, I get the sense that everything can be brought into Wave and Waves can be embedded anywhere. That feels about as open as Facebook. What I’m missing is a sense of how (or even if!) Google Wave will interoperate with other communication platforms. They do show an example of building a Twitter client within Wave–perhaps that is representative of their interoperability strategy.

The Google Wave demo is impressive, and I have no doubt that developers will play with it and build cool demos of their own. But I believe the ultimate success of Google Wave will depend on how they address the above two questions. Time will tell.

• Twitter’s employees don’t twitter very much: an average of 2 to 3 tweets per person per day.
• Twitter employees don’t follow very many other people: only 2 out of 49 Twitter team members follow more than 500 people and no one was over 1k.
• Twitter staff members aren’t following top Twitter developers in the community.

I can’t really address the third point, but the first two–and especially the second–are hardly helpful to Kirkpatrick’s case. To the contrary, they argue that the people who work at Twitter get it. And, to make sure Kirkpatrick got it, Twitter CEO Ev Williams even wrote him a letter, in which he said:

Many people fall into the trap that you should follow all or most people back out of a sense of politeness or so-called engagement with the community… At a certain point, you’re not actually reading any more tweets by following more people — you’re just dipping into the stream somewhat randomly and missing a whole lot of what people say. That’s fine, but I believe people will generally get more value out of Twitter by dropping the symmetrical relationship expectation and simply curating their following list based on the information and people they want to tune in to.

Amen! I’ve been hammering this point here in most of my posts about Twitter, but here is a handful of examples for newer readers:

• Twitter’s Twist on the Attention Economy
• An Attention Ponzi Scheme?
• Putting the Social back in Social Networks
• The Real Twitter
• Guy Kawasaki, I’ll Say It

And of course the whole point of TunkRank is to discourage the vicious circle of reciprocity and fake following. That’s baked into the the measure which, like PageRank, divides the voting power by the number of out-links.

The comments on Kirkpatrick’s post suggest that a lot of regular Twitter users also get it. I find that reassuring, especially given the hype around Twitter in the last several weeks. Twitter can be a useful tool but it will help if people don’t devalue it by imposing cultural norms that devalue the social network. I’m glad the folks who have given us Twitter realize that.

Regular readers know that I am not a Google fan boy, and that much of my commentary on Google focuses on their neglect of exploratory search. Nonetheless, when I saw the initial Youtubeware describing Google Squared a few weeks ago, my ears perked up. I decided to wait until it went live to assess it. Well, it’s live now.

The idea of Google Squared is simple: it “collects facts from the web and presents them in an organized collection, similar to a spreadsheet.” The best way to understand it is to try it. For example, search for hybrid car, and you’ll see a table of hybrids, with columns corresponding to image, description, type of transmission, yeah, and height. Add a price column if you’d like, and it will populate it for you. Very slick.

Of course, it is, as Google admits, “by no means perfect”. Most queries will show its warts, and some, like information scientists, are way off (it doesn’t even try to return results for library scientists). But it does pretty well when there is structured data out there, and it makes admirable attempt to find it! I suspect the real trick here is that it does a decent job of finding determining instances of the query category (perhaps a souped up version of work they started discussing back in 2004), and then mining structured content about those instances from repositories like Freebase.

I mean, look at these results:

• b-movies
• renaissance choral composers
• heroes characters
• sf books

To be clear, I picked these examples after a fair amount of trial and error–like Wolfram Alpha, it is hit and miss, with more miss than hit. But, as Seth Grimes said at the recent Text Analytics Summit, when Wolfram Alpha is good, it’s very very good, but when it’s bad, it’s horrid. Google Squared doesn’t fail quite so spectacularly, and it gives you a lot more of a chance to interact with it.

This is, by far, the best step I’ve seen Google take towards HCIR, and I’m impressed. It’s still a toy at this stage, but I think it has a future. My warmest congratulations to Daniel Dulitz and the rest of the magpie team that developed it; I’m looking forward to seeing it evolve.