Testing of Search Engines for Grant Project

[Scott Frey]

I've received a grant from the Wolters Kluwer Law & Business Grant Program to develop a search engine for law-related public domain electronic books (http://www.aallnet.org/news/newsdisplay.asp?nid=146). As part of the project, I'm seeking law librarians and other legal researchers to provide feedback on the search engines' interfaces and results. I envision that the initial test in April 2009 would take about 30 minutes, followed by tests of similar length in May, August, September, and potentially October. (I figure that 30 minutes would be sufficient for good testing and feedback, without impinging unduly on people's busy schedules.) I might call or email some testers for clarification or elaboration of their feedback.

If you're interested in being a tester, please contact me at sfrey at wsulaw.edu. If you doubt you'll be available during one or more of the testing times, that's okay -- even one test could be useful. This isn't a firm commitment; some testers will have to cancel, while others might sign up at the last minute or after testing has started.

[Scott Frey]

An update regarding testing for the search engine project:

I anticipate sending instructions for the initial testing in the last week of April (after April 25) and for a second test in late May (after May 23). When I have definite dates -- or if this schedule changes -- I'll provide additional updates. Once I send the instructions, I intend to allow about 10 days for participants to send their responses.

So far, I've asked law librarians to participate. However, I'm interested in testing by others who do legal research: law professors, law students, attorneys, legal historians, etc. If you know someone who might be interested (whether now or later in the year), you can give the person my email address (sfrey at wsulaw.edu). Alternatively, if you're on my list of testers, you can provide the person with a copy of the instructions that I'll be sending you.

[Scott Frey]

A second update regarding testing for the search engine project:

I am postponing the initial testing –- originally scheduled for late April –- until late May (after May 23). (I think it makes more sense to do it right than to do it now.) I might schedule a second round of testing to start in late June (after June 20). For each round of testing, I intend to allow about two weeks for participants to send their responses to the testing questions. Throughout this project, prospective testers should feel free to drop out or join in at the last minute, as well as to invite other people who do legal research to participate.

If you have questions, feel free to contact me at sfrey at wsulaw.edu.

[Scott Frey]

An update regarding the search engine project:

In my own testing of and tinkering with several pieces of software during May, I've found that all have some drawbacks in their current form. For example, one program might have worked well when last updated in 2007, but requires modification to work in 2009. Another program might be under active development, but is awkward to use. Yet another program might be easy to use, but lacks certain features (or the flexibility for a developer to add features easily).

After much angst over these issues, I have settled on combinations of software that I think are promising. Essentially, I have done the preliminary testing that I had initially planned to have others do. Now I will focus entirely on getting search engines out to the testers. (I can always return to the angst later.)

As I have noted previously –- and wish to emphasize in light of the changes in schedule –- any tester should feel free to join or drop out at any time. Thank you for bearing with me; I'm grateful for the encouragement you've given.

I'll post another message when search engines are ready for testing. (From my experiences so far in this project, I've learned that I shouldn't promise a specific week. But let's just say that I'll finish the preparation as soon as I possibly can, and more specifically in June.) Please feel free to contact me at sfrey at wsulaw.edu.

[Scott Frey]

I've put a couple of search engines on the Web. Later today, I'll be sending the URLs plus some information and instructions to the people who asked to be testers. I'll likely run another round of testing within a couple of weeks (i.e., prior to the AALL Annual Meeting). If you haven't contacted me before and are interested in being a tester, please let me know.

[Scott Frey]

In previous testing, I've asked testers to try three interfaces to a search engine. While I got some positive feedback and do like some aspects of these interfaces, I'm not satisfied yet. Thus, I'm creating my own interface. In September and October, I'll ask testers to try it, probably in comparison with another interface.

However, I'm opening the current round of "testing" to everyone who sees this message. Rather than ask for comments on a new search engine, I'm interested in what you think about existing search engines -- whether Google, Yahoo, or Bing; LexisNexis, Westlaw, or HeinOnline; or another website you've used.

* What do you like or dislike in a search engine?

* What makes a search engine useful to you?

* Do you find what you want on search engines mainly with your initial query, trial and error with a few queries, or another method?

* Which features of a search engine -- i.e., aspects of functionality, design, etc. -- add to your experience?

* Which features detract from the experience?

* Do you find any aspects of legal search engines helpful or unhelpful?

I expect that you'll have some quick responses to these questions, which I'd like to see. But I hope that you'll also take a few minutes -- possibly after trying a couple of search engines -- to think of additional comments. Though not necessary, you can illustrate your views with sample queries or links to search engines or results pages. Feel free to answer any or all questions, or to modify the questions as you think appropriate.

Please email your replies by September 9 to sfrey at wsulaw.edu.

Also, if you haven't previously asked to be a tester, and would like to test my search engine(s) in September and October, please let me know. If you know anyone else (e.g., attorney, law professor, law student, or other law library patron) who would like to do this testing, feel free to refer them to me. Anyone who isn't a law librarian should indicate their line of work, so that I know basically what perspective they are coming from.

[Scott Frey]

I invite anyone interested in search engines or legal research to try a prototype search engine at http://li88-17.members.linode.com:8081/search3/. Any feedback is welcome. (I'd be grateful if you could note the date and time when you searched and queries that you searched for.)