To complete my MS in Information Architecture and Knowledge Management at Kent State I did some research on folksonomies and how the can support information retrieval. I compared social bookmarking systems with search engines and directories. I’m hoping to see the results published in an academic journal. In the mean time, you can see a pre-publication copy of my results:
Tag Archives: information-retrieval
Notes: Design of interfaces for information seeking
Marchionini, G., & Komllodi, A. (1998). Design of interfaces for information seeking. Annual Review of Information Science and Technology (ARIST), 21, 89-130.
In this chapter Marchionini and Komlodi examine the state of user interfaces for information seeking. Interfaces are defined as the conjunctions and boundaries where different physical and conceptual human constructs meet, and is at the center of information science in fields such as human-computer interaction (HCI and human factors. The chapter looks at advances in technology and research, summarizes the developments of the first two generations of user interfaces, and examines current (as of 1998) developments in the field. One way to look at the chapter is shown in figure 1, with technology, information seeking, and interface design research and development shifting from mainframes to PCs to the web, from professionals to literate end users to universal access, and from ASCII characters to graphics to multimedia respectively. Some early developments remain important today, such as the components of an interactive system – task, user, terminal and content (with context added later). Another milestone was the development of the GOMS (goals, operators, methods and selection) model, the first formal model of of HCI. Two themes throughout the chapter are the interdependent nature of research in this area and the importance of human-centered concepts and design.
This is a really good summary of the history of HCI with an eye specifically toward searching and information use. It’s not surprising the many of the names we have seen on articles this semester show up here as well. The only real regret I have is that there are no pictures. User interfaces often rely on visual display for interaction, so in addition to all the description it would be really interesting to see examples of the different generations of user interfaces. One other criticism is that little attention is paid the the interfaces of video games—I have read a lot of articles about interface design that ignore this field as well.
Although it is a little out of date, there’s a lot to be taken from this chapter’s historical perspective. I found three things in particular that were talked about in relationship to third-generation user interfaces that were particularly interesting. First was the move toward universal access or ubiquitous computing. It is in some ways a measure of success that researchers now worry about the lack of computers in Sub-Saharan Africa—this wouldn’t be a problem if information seeking computer interfaces were not so available, useful, and approachable. Second was the notion that the advance of the web in some ways slowed the advance of user interface design, although the apparent disadvantage quickly disappeared. This is something I’ve run into in a different form as a web designer—clients complaining that their web site did not look exactly like their brochure. Again, in some ways this was an embarrassment of riches—the web site cost nothing to distribute, could be found by search engines, acted as a storefront, but the lack of a particular font face was a step backward? Finally, the notion that the whole field is really interdisciplinary is important to always keep in mind.
Notes: Automatic performance evaluation of web search engines
Can, F., Nuray, R., & Sevdik, A. B. (2004). Automatic performance evaluation of web search engines. Information Processing & Management, 40(3), 495-514.
Although virtually all Internet users utilize search engines to find information on the web evaluation of search engines is often difficult. A large number of searches would need to be tested and each one would need to be judged subjectively by human participants. The authors of this paper have devised a new way to test search engines and have tested their method against evaluations done by human judges, and found their automatic Web search engine evaluation method (AWSEEM) significantly predicted the subjective judgments. In the human-evaluation control, users were given a list of resources called up by the various search engines with no idea which engine each came from and were asked to rank the relevance of each. In AWSEEM, each query was run and the top 200 results for each engine were compiled into a collection of vectors which are then ranked by their similarity to the “the user information-needs” (including the question, the query, and a description of the need). The system then looks at the top 20 ranked pages for each engine and counts how many are in the top s (50 and 100 are used) commonly retrieved pages. These are assumed to be relevant.
One possible issue with this system is that it requires a little more human interaction than first assumed—the query providers must provide more than just a query. A bigger issue, though, is the choice of measure for relevancy. AWSEEM assumes that if a result appears in the results of multiple engines, it is relevant. This may be reasonable, but does raise the question—what if all the engines studied are wrong? For a simple example, searching for my own name online will retrieve a large number of results that are the same in many search engines but have nothing to do with the particular Jason Morrison who sits here typing this. Another interesting thing to note is that they did not find much of a statistically significant difference between the performance of the different search engines using either method (although more so with the human-judgment method). Very few scholarly articles (and even fewer popular press articles) bother to do this when pitting search engines against each other. Is it possible that the very notion of the “best” search engine has been statistically meaningless for some time?
The authors make a good point about the difficulty in using real users for search engine evaluation. An automated approach is one answer, but there is another—the problem is that too much time and effort is required of a small number of users. Instead, if tiny amounts of time and effort were spread across thousands or millions of users, similar results could be achieved while still using subjective measures. For example, if every time a user got results on any search engine they were presented with a simple “rate these results on a scale of 1 to 5 stars” input, they could quickly and effortlessly contribute data toward a shootout-type study. Cooperation of the search engines would not necessarily be needed, if one could use a university’s proxy to substitute or add the input for popular search engines, for example, or if a generic search page was set up to produce results from randomized (double-blind) engines. It would be interesting to try this, AWSEEM, and individual evaluation in one study to see if there was a statistical correlation.