----------------------------------------------------------------------
Alta Vista http://www.altavista.digital.com/cgi-bin/query?pg=q
Alta Vista a http://www.altavista.digital.com/cgi-bin/query?pg=aq
Excite http://www.excite.com/?xam
Hotbot http://www.hotbot.com/index.html
Infoseek http://www.infoseek.com/Home?pg=ultra_home.html
Lycos http://www.lycos.com/index.html
Lycos a http://www.lycos.com/customsearch.html
General criteria for assigning grades 0 through 3:
3 A three is for a site that would likely be of use to anyone
who entered the search query. Such a site would be a one that
contains a lengthy and general discussion of the topic, or
one that contains many links to other sites that deal with the
topic (a webliography, if you will).
2 A two is for a site that could be of some conceivable use to
someone searching the topic, but not necessarily. Twos are given
to sites that focus on a narrow aspect of the subject, or that
deal with the subject, but not thoroughly. For instance, in
"unemployment rate" +Illinois, a site that link to many pages
containing Illinois unemployment rates might get a three, but a page
that reported the Illinois unemployment rate for only one month would
only get a two. Also, a page with a link to a three, but without
other qualifications, will get a two.
1 A site that is technically correct, but not useful. The html code for
the page did satisfy the required components of the query. However,
the components may not have been in the correct context. To continue
with #2, the unemployment rate, if a page gave the unemployment rate
of some other place, but then happened to mention the word "Illinois,"
it technically satisfied the query without being useful. In mechanical
terms, the incorrect subject is more likely to have the components
of the query farther apart in proximity. The query might be Boolean
or unstructured, but the useful information had higher proximity.
Search engines would occassionally return a dictionary page as the
result to a query. Indeed, all of the words in the query were found
in the dictionary, but it was of no conceivable use to someone who
had issued the query. Also, pages that mentioned the correct topic,
but contained only a minimum of actual information that could be
useful to someone conducting the search were given ones.
0 A site that did not satisfy the query as submitted was given a zero.
In the unemployment case, if the either the word Illinois or the
phrase "unemployment rate" was missing, a zero was given. In #3, a
zero was avoided if the word "athletes" was present and either of
the phrases "wife abuse" or "domestic violence" was present.
PART I: Queries that could be used because of how they were selected
and examined.
Query 1: Simple: Information on the Individuals with Disabilities Education
Act
Source: Obtained locally
Searched: 1/31/97
----------------------------------------------------------------------
Alta Vista Individuals with Disabilities Education Act
Excite Individuals with Disabilities Education Act
Hotbot All the words: Individuals with Disabilities Education Act
Infoseek Individuals with Disabilities Education Act
Lycos Individuals with Disabilities Education Act
Criteria:
3 A site with a lengthy explanation of the Act or a site
with links to many other resources.
2 A site with some discussion of the Act, or focusing on some
aspect of the act.
1 A site that contained all of the words but did not deal with the
topic, or one that mentioned the topic, but was insubstantial.
0 HTML Code for the page lacked some of the key words of the
search, like "Individuals" or "Disabilities" or "Education".
Query 2: Structured: Find unemployment rate statistics for Illinois
Source: Obtained locally
Searched: 2/1/97
----------------------------------------------------------------------
Alta Vista a +"unemployment rate" AND +Illinois
Excite +"unemployment rate" AND +Illinois
Hotbot All of the words: unemployment rate Illinois
Modify: must contain the exact phrase: "unemployment rate"
Infoseek +"unemployment rate" +Illinois
Lycos a Match all words, close match: unemployment rate Illinois
Criteria:
3 A site with a full analysis of the unemployment rate of Illinois, or
with links to pages that listed the rate for different time periods.
2 A site that cited the unemployment rate for Illinois, regardless of
how recent, or that mentioned the unemployment rate for Illinois.
1 A site that contained the phrase "unemployment rate" and the word
Illinois, but did not deal with the rate in Illinois.
0 HTML Code for the page lacked some of the key words of the
search, like "unemployment rate" or "Illinois".
Query 3: Structured: Information on the problem of domestic violence among
athletes
Source: Obtained locally
Searched: 2/8/97
----------------------------------------------------------------------
Alta Vista a +athletes AND ("domestic violence" OR "wife abuse")
Excite +athletes AND ("domestic violence" OR "wife abuse")
Hotbot Any of the words: athletes "domestic violence" "wife abuse"
Modify: must contain the word: athletes
Modify: should contain the exact phrase: "domestic violence"
Modify: should contain the exact phrase: "wife abuse"
Infoseek +athletes "domestic violence" "wife abuse"
Lycos a Match 3 words: athletes sports domestic violence wife abuse
Criteria:
3 A site with a lengthy treatment of the domestic violence among
athletes, or with links to many pages that discussed it.
2 A site that made some mention of the topic, or dealt with some
aspect of it.
1 A site that contained the phrase "domestic violence" or "wife abuse"
and the word athletes, but did not deal with the two together.
0 HTML Code for the page lacked some of the key words of the
search, like "domestic violence" or "wife abuse" or "athletes".
Query 4: Simple: animal husbandry
Source: Tomaiuolo and Packer study
Searched: 2/14/97
----------------------------------------------------------------------
Alta Vista animal husbandry
Excite animal husbandry
Hotbot All of the words: animal husbandry
Infoseek animal husbandry
Lycos animal husbandry
Criteria:
3 A site which had a lengthy treatment of the topic of animal husbandry,
or which provided links to many pages that discussed it.
2 A site that made some mention of the topic, or dealt with some
aspect of it.
1 A site that contained the words animal and husbandry, but did not
deal with the topic meant by two together in a substantial way.
0 HTML Code for the page lacked some of the key words of the
search, like "animal" or "husbandry".
Query 5: Simple: cable television regulation
Source: Tomaiuolo and Packer study
Searched: 2/14/97
----------------------------------------------------------------------
Alta Vista cable television regulation
Excite cable television regulation
Hotbot All of the words: cable television regulation
Infoseek cable television regulation
Lycos cable television regulation
Criteria:
3 A site which had a lengthy treatment of the topic of cable television
regulation, or which provided links to many pages that discussed it.
2 A site that made some mention of the topic, or dealt with some
aspect of it.
1 A site that contained the words cable, television and regulation,
but did not deal with the topic in a substantial way.
0 HTML Code for the page lacked some of the key words of the
search, like "cable" or "regulation".
Query 6: Simple: MTX International Corporation
Source: Obtained locally
Searched: 2/14/97
----------------------------------------------------------------------
Alta Vista MTX International
Excite MTX International
Hotbot All of the Words: MTX International
Infoseek MTX International
Lycos MTX International
Criteria:
3 A site which had substantial information on the MTX International
corporation.
2 A site that provided some substantial information about the company,
or one that provided a link to the MTX International homepage.
1 A site that happened to mention the company, say in a list of
products supplied, but no information about the company, or that
contained the words MTX and Internation, but did not deal with the
company "MTX International".
0 HTML Code for the page lacked some of the key words of the
search, like "MTX" or "International".
Query 7: Simple: cell cryopreservation
Source: Tomaiuolo and Packer study
Searched: 2/20/97
----------------------------------------------------------------------
Alta Vista cell cryopreservation
Excite cell cryopreservation
Hotbot All of the words: cell cryopreservation
Infoseek cell cryopreservation
Lycos cell cryopreservation
Criteria:
3 A site which had a lengthy treatment of the topic of cell
cryopreservation, or which provided links to many pages that
discussed it.
2 A site that made some mention of the topic, or dealt with some
aspect of it.
1 A site that contained the words cell and cryopreservation,
but did not deal with the topic in a substantial way.
0 HTML Code for the page lacked some of the key words of the
search, like "cell" or "cryopreservation".
Query 8: Simple: classical architecture
Source: Tomaiuolo and Packer study
Searched: 2/21/97
----------------------------------------------------------------------
Alta Vista classical architecture
Excite classical architecture
Hotbot All of the words: classical architecture
Infoseek classical architecture
Lycos classical architecture
Criteria:
3 A site which had a lengthy treatment of the topic of classical
architecture, or which provided links to many pages that dealt with
it.
2 A site that gave some treatment of the topic, or dealt with some
aspect of it, such as a thorough discussion of an academic program
that focused on classical architecture. "Classical" was left to
the site to define. Roman, Greek and Turkish classical pages were
discovered.
1 A site that contained the words classical and architecture,
but did not deal with the topic in a substantial way.
0 HTML Code for the page lacked some of the key words of the
search, like "classical" or "architecture".
Query 9: Simple: ecotourism
Source: Tomaiuolo and Packer study
Searched: 2/24/97
----------------------------------------------------------------------
Alta Vista ecotourism
Excite ecotourism
Hotbot ecotourism
Infoseek ecotourism
Lycos ecotourism
Criteria:
3 A site which had a lengthy treatment of the topic of ecotourism,
or which provided links to pages from more than one tour company
that organizes ecotours.
2 A site that discussed the topic of ecotourism, or provided links
to pages describing several different ecotours offered by one tour
company.
1 A site that contained the word ecotourism, but had little discussion
of the topic, or only described one tour offered by a commercial
tour provider.
0 HTML Code for the page lacked the word "ecotourism".
Query 10: Simple: In Focus Systems
Source: Obtained locally
Searched: 2/28/97
----------------------------------------------------------------------
Alta Vista In Focus Systems
Excite In Focus Systems
Hotbot All of the Words: In Focus Systems
Infoseek In Focus Systems
Lycos In Focus Systems
Criteria:
3 A site which had substantial information on the In Focus Systems
corporation, such as the In Focus Systems homepage.
2 A site that provided some substantial information about the company,
or one that provided a link to the In Focus Systems homepage.
1 A site that happened to mention the company, say in a list of
products supplied, but no information about the company.
0 HTML Code for the page lacked some of the key words of the
search, like "In" or "Systems".
Query 11: Structured: Find information on the National Gallery of Prague
Source: Obtained locally
Searched: 3/1/97
----------------------------------------------------------------------
Alta Vista a "National Gallery" AND +Prague
Excite "National Gallery" AND +Prague
Hotbot Exact phrase: National Gallery
Modify: must contain the word: Prague
Infoseek "National Gallery" +Prague
Lycos a Match 2 words, Loose Match: National Gallery Prague
Criteria:
3 The homepage for National Gallery of Prague, or a page that had
an extensive discussion of it or links to exhibits from it
(only the National Gallery of Prague homepage qualified).
2 A site that gave information about the National Gallery of Prague
or one that provided a link to the National Gallery of Prague
homepage.
1 A site that contained the phrase "National Gallery" and word Prague,
but did not deal with the two together.
0 HTML Code for the page lacked some of the key words or
phrases of the search, like "National Gallery" or "Prague".
Query 12: Structured: The effects of caffeine on aerobic exercise
Source: Obtained locally
Searched: 2/27/97
----------------------------------------------------------------------
Alta Vista a +caffeine AND +aerobic AND exercise AND effects
Excite +caffeine AND +aerobic AND exercise AND effects
Hotbot All the words: caffeine aerobic exercise effects
Modify: must contain the word: caffeine
Modify: must contain the word: aerobic
Infoseek +caffeine +aerobic exercise effects
Lycos a Match three words, loose: caffeine aerobic exercise effects
Criteria:
3 A site with a lengthy treatment of the effects of caffeine on human or
animal performance doing aerobic exercise, or with links to many
pages that discussed it.
2 A site that made some mention of the topic, or dealt with some
aspect of it.
1 A site that contained the words "caffeine" and "aerobic"
but did not deal with the effects of caffeine on aerobic exercise.
0 HTML Code for the page lacked some of the key words of the
search, like "caffeine" or "aerobic."
Query 13: Structured: Find information on the group called "Queer Nation"
Source: Obtained locally
Searched: 3/5/97
----------------------------------------------------------------------
Alta Vista a "Queer Nation"
Excite "Queer Nation"
Hotbot Exact Phrase: Queer Nation
Infoseek "Queer Nation"
Lycos a Match All Words, Loose Match: Queer Nation
Criteria:
3 A site with a lengthy treatment of the group called Queer Nation,
or provide many links to pages that discussed it.
2 A site that made some mention of the group, or dealt with some
aspect of it, or dealt with the Australian club called Queer Nation.
1 A site that contained the phrase "Queer Nation" but did not discuss
it, or one that contained the phrase but had no substantial
information.
0 HTML Code for the page lacked the phrase "Queer Nation".
Query 14: Simple: Find information about prozac, but not the rock group
Source: Obtained locally
Searched: 3/12/97
----------------------------------------------------------------------
Alta Vista prozac -rock
Excite prozac -rock
Hotbot All of the Words: prozac
Modify: must not contain the word: rock
Infoseek prozac -rock
Lycos prozac -rock
Criteria:
3 A site which had a lengthy treatment of the topic of prozac,
or which provided links to many pages that discussed it, without
containing the word "rock," as in rock group.
2 A site that made some mention of the topic, or dealt with some
aspect of it, without containing the word "rock," as in rock group.
1 A site that contained the word prozac and not the word "rock,"
but did not deal with the topic of prozac, the drug, in a substantial
way. (This situation was common, as the word prozac has become a
popular term in popular culture.)
0 HTML Code for the page lacked the key word "prozac" or contained
the word "rock".
Query 15: Find information on Maria Luisa Bombal
Source: Obtained locally
Searched: 3/12/97
----------------------------------------------------------------------
Alta Vista Maria Luisa Bombal
Excite Maria Luisa Bombal
Hotbot The Person: Maria Luisa Bombal
Infoseek Maria Luisa Bombal
Lycos Maria Luisa Bombal
Criteria:
3 A site which had a lengthy treatment of the writer, or which
provided links to many pages that discussed her (there were none).
2 A site that had any information at all on her, other than citations
to books by her or books about her. This category was stretched
a little further for this query than for others, because of the
lack of pages that discussed her.
1 A site that contained the name "Maria Luisa Bombal" but had little
substantial infomation about her. Included here were pages that
cited one of her books.
0 HTML Code for the page lacked the key name Maria Luisa Bombal.
Lyocs had the most zeros, because it found many pages about other
women who were named "Maria Luisa."
Table B1a: Score of 1, 2 or 3 in the first:
three results/ ten results/ twenty results
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
---------------------------------------------------------------------------
1 IDEA 2/9/19 3/10/20 2/5/6 3/8/18 3/8/14
2 u. rate 3/9/16 3/9/16 2/7/12 3/10/18 0/0/0 (0)
3 athletes 3/9/18 3/10/20 3/10/18 1/5/13 1/1/1
4 animal h 3/10/17 3/9/19 2/6/13 3/9/17 2/6/15
5 cable t 3/9/18 3/9/18 2/7/15 3/8/17 2/8/13
6 MTXI 3/10/11 3/10/20 2/9/14 3/9/17 3/3/3
7 cell cry 3/10/20 3/10/20 3/9/19 3/10/20 2/7/14
8 classical 3/10/20 3/10/20 2/5/12 3/8/17 3/10/20
9 ecotour 2/9/19 2/8/18 2/8/17 3/9/15 3/9/17
10 IFS 3/8/16 3/10/20 0/2/3 3/10/20 1/1/2
11 Nat Gall 2/6/15 3/10/19 1/7/13 2/7/14 0/0/0
12 caffeine 3/8/17 3/10/20 3/9/12 3/7/12 0/1/3 (16)
13 QN 2/7/14 3/5/13 2/5/11 3/6/14 1/6/8
14 prozac -r 3/10/18 3/8/17 3/8/12 2/7/17 3/9/14
15 ML Bombal 3/10/19 3/10/18 3/9/14 2/8/11 (14) 2/3/3
(18)
Table B1b: The weighted score from test 1
Query \ Serv. Alta V Excit HotBot Infosk Lycos
-------------------------------------------------------
1 IDEA .93 1.00 .36 .88 .73
2 u. rate .83 .83 .63 .93 0
3 athletes .90 1.00 .93 .60 .07
4 animal h .89 .94 .64 .87 .71
5 cable t .90 .90 .73 .84 .69
6 MTXI .68 1.00 .75 .87 .22
7 cell cry 1.00 1.00 .94 1.00 .70
8 classical 1.00 1.00 .58 .84 1.00
9 ecotour .93 .87 .83 .80 .87
10 IFS .81 1.00 .16 1.00 .11
11 Nat Gall .71 .96 .65 .70 0
12 caffeine .84 1.00 .69 .64 .15
13 QN .70 .62 .54 .68 .45
14 prozac -r .93 .84 .66 .81 .76
15 ML Bombal .96 1.00 .76 .79 .20
Table B1c: Mean, Standard Deviation and Standard Error of Mean for Services
for Test 1
Serv\Statistic N Mean Stan. Dev. St. Error
-----------------------------------------------------------------
Alta Vista 15 0.8676 0.1048 0.0271
Excite 15 0.9314 0.1057 0.0273
Hotbot 15 0.6566 0.2013 0.0520
Infoseek 15 0.8158 0.1197 0.0309
Lycos 15 0.4440 0.3492 0.0902
All together 75 0.7431 0.2621 0.0303
Table B2a: Score 2 or 3 in the first: three results/ ten results/
twenty results
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
---------------------------------------------------------------------------
1 IDEA 2/8/15 3/7/11 2/5/5 3/7/15 3/7/11
2 u. rate 1/2/4 3/8/14 1/5/5 3/9/13 0/0/0 (0)
3 athletes 0/0/3 2/7/11 1/1/3 1/5/11 1/1/1
4 animal h 3/8/12 2/5/7 1/3/5 3/8/13 0/4/7
5 cable t 2/6/9 3/7/11 2/3/6 0/2/4 1/5/10
6 MTXI 2/7/7 2/2/3 1/6/8 2/5/11 1/1/1
7 cell cry 1/6/12 1/8/12 0/1/6 1/2/8 0/0/2
8 classical 3/10/18 1/5/12 0/0/4 3/5/9 3/8/17
9 ecotour 2/8/13 2/5/7 0/2/5 1/5/8 2/5/7
10 IFS 3/8/16 3/10/18 0/1/1 1/8/16 1/1/2
11 Nat Gall 0/1/4 3/8/15 1/6/12 2/5/10 0/0/0
12 caffeine 0/1/2 2/2/3 1/4/4 1/3/6 0/0/2 (16)
13 QN 1/2/6 3/5/7 2/4/10 3/5/10 0/2/4
14 prozac -r 2/7/10 0/4/10 0/2/5 2/5/15 1/4/9
15 ML Bombal 2/3/4 0/1/2(18) 0/1/2 1/2/3 (14) 0/0/0
Table B2b: The weighted score from test 2
Query \ Serv. Alta V Excit HotBot Infosk Lycos
-------------------------------------------------------
1 IDEA .76 .60 .33 .75 .60
2 u. rate .20 .73 .32 .72 0
3 athletes .11 .59 .14 .53 .07
4 animal h .66 .40 .27 .70 .35
5 cable t .49 .60 .31 .19 .49
6 MTXI .45 .18 .45 .54 .07
7 cell cry .59 .64 .24 .35 .07
8 classical .93 .57 .14 .48 .84
9 ecotour .69 .40 .23 .42 .40
10 IFS .81 .93 .06 .78 .11
11 Nat Gall .17 .77 .59 .51 0
12 caffeine .10 .18 .25 .27 .07
13 QN .28 .41 .48 .52 .19
14 prozac -r .56 .46 .23 .68 .43
15 ML Bombal .24 .14 .10 .21 0
Table B2c: Mean, Standard Deviation and Standard Error of Mean for Services
for Test 2
Serv\Statistic N Mean Stan. Dev. St. Error
-----------------------------------------------------------------
Alta Vista 15 0.4686 0.2723 0.0703
Excite 15 0.5042 0.2328 0.0601
Hotbot 15 0.2757 0.1454 0.0375
Infoseek 15 0.5128 0.1909 0.0493
Lycos 15 0.2481 0.2584 0.0667
All together 75 0.4019 0.2477 0.0286
Table B3a: Score of 3 in the first:
three results/ ten results/ twenty results
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
--------------------------------------------------------------------------
1 IDEA 1/4/4 2/2/2 1/2/2 1/2/3 2/2/2
2 u. rate 1/1/1 1/1/1 0/0/0 3/5/5 0/0/0 (0)
3 athletes 0/0/2 2/2/2 0/0/2 0/1/3 1/1/1
4 animal h 0/2/4 1/2/2 1/1/3 3/6/9 0/1/1
5 cable t 1/2/3 0/1/2 1/1/1 0/0/1 1/3/3
6 MTXI 0/0/0 1/1/1 0/0/0 1/1/2 0/0/0
7 cell cry 1/2/4 0/0/0 0/0/1 1/1/2 0/0/0
8 classical 1/1/1 0/3/4 0/0/2 0/1/2 0/1/2
9 ecotour 1/2/3 2/3/4 0/0/0 1/2/2 0/0/0
10 IFS 1/1/1 1/1/2 0/0/0 1/1/2 0/0/0
11 Nat Gall 0/0/0 0/0/1 1/1/1 1/1/1 0/0/0
12 caffeine 0/0/0 1/1/1 1/2/2 0/1/1 0/0/0
13 QN 0/0/0 0/0/0 0/0/0 0/0/1 0/0/0
14 prozac -r 0/0/1 0/0/0 0/0/0 1/2/4 0/0/1
15 ML Bombal 0/0/0 0/0/0 0/0/0 0/0/0 0/0/0
Table B3b: The weighted score from test 3
Query \ Serv. Alta V Excit HotBot Infosk Lycos
-------------------------------------------------------
1 IDEA .25 .14 .13 .17 .14
2 u. rate .07 .07 0 .34 0
3 athletes .07 .14 .07 .13 .07
4 animal h .19 .13 .14 .51 .06
5 cable t .17 .10 .07 .04 .19
6 MTXI 0 .07 0 .11 0
7 cell cry .20 0 .04 .11 0
8 classical .07 .22 .07 .10 .10
9 ecotour .17 .24 0 .13 0
10 IFS .07 .11 0 .11 0
11 Nat Gall 0 .04 .07 .07 0
12 caffeine 0 .07 .13 .06 0
13 QN 0 0 0 .04 0
14 prozac -r .04 0 0 .20 .04
15 ML Bombal 0 0 0 0 0
Table B3c: Mean, Standard Deviation and Standard Error of Mean for Services
for Test 3
Serv\Statistic N Mean Stan. Dev. St. Error
-----------------------------------------------------------------
Alta Vista 15 0.0875 0.0878 0.0227
Excite 15 0.0889 0.0771 0.0199
Hotbot 15 0.0487 0.0547 0.0141
Infoseek 15 0.1403 0.1294 0.0334
Lycos 15 0.0401 0.0617 0.0160
All together 75 0.0811 0.0912 0.0105
Table B4a (HotBot special): Score 2 or 3 minus duplicates in the first:
three results/ ten results/ twenty results
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
--------------------------------------------------------------------------
1 IDEA 2/9/19 3/10/20 3/6/6 (9) 3/8/18 3/8/14
2 u. rate 3/9/16 3/9/16 3/7/12 (18) 3/10/18 0/0/0 (0)
3 athletes 3/9/18 3/10/20 3/10/18 1/5/13 1/1/1
4 animal h 3/10/17 3/9/19 2/7/13 (16) 3/9/17 2/6/15
5 cable t 3/9/18 3/9/18 2/7/15 (18) 3/8/17 2/8/13
6 MTXI 3/10/11 3/10/20 2/9/14 (16) 3/9/17 3/3/3
7 cell cry 3/10/20 3/10/20 3/9/19 (19) 3/10/20 2/7/14
8 classical 3/10/20 3/10/20 3/8/12 (16) 3/8/17 3/10/20
9 ecotour 2/9/19 2/8/18 2/8/17 3/9/15 3/9/17
10 IFS 3/8/16 3/10/20 0/2/3 (19) 3/10/20 1/1/2
11 Nat Gall 2/6/15 3/10/19(19) 2/9/13 (15) 2/7/14(17) 0/0/0
12 caffeine 3/8/17 3/10/20 3/10/12 (13) 3/7/12(18) 0/1/3 (16)
13 QN 2/7/14 3/5/13 2/6/11 (16) 3/6/14 1/6/8
14 prozac -r 3/10/18 3/8/17 3/9/12 (13) 2/7/17 3/9/14
15 ML Bombal 3/10/19 3/10/18 3/9/14 (19) 2/8/11 (14) 2/3/3
Table B4b: The weighted score from test 4
Query \ Serv. Alta V Excit HotBot Infosk Lycos
-------------------------------------------------------
1 IDEA .93 1.00 .65 .88 .73
2 u. rate .83 .83 .69 .93 0
3 athletes .90 1.00 .93 .60 .07
4 animal h .89 .94 .77 .87 .71
5 cable t .90 .90 .79 .84 .69
6 MTXI .68 1.00 .87 .87 .22
7 cell cry 1.00 1.00 1.00 1.00 .70
8 classical 1.00 1.00 .77 .84 1.00
9 ecotour .93 .87 .83 .80 .87
10 IFS .81 1.00 .16 1.00 .11
11 Nat Gall .71 1.00 .87 .81 0
12 caffeine .84 1.00 .95 .64 .15
13 QN .70 .62 .66 .68 .45
14 prozac -r .93 .84 .92 .81 .76
15 ML Bombal .96 1.00 .79 .79 .20
Table B4c: Mean, Standard Deviation and Standard Error of Mean for Services
for Test 4
Serv\Statistic N Mean Stan. Dev. St. Error
-----------------------------------------------------------------
Alta Vista 15 0.8676 0.1048 0.0271
Excite 15 0.9338 0.1068 0.0275
Hotbot 15 0.7692 0.1947 0.0503
Infoseek 15 0.8309 0.1055 0.0273
Lycos 15 0.4440 0.3492 0.0902
All together 75 0.7691 0.2574 0.0297
Table B5a (HotBot special): Score 2 or 3 minus duplicates in the first:
three results/ ten results/ twenty results
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
--------------------------------------------------------------------------
1 IDEA 2/8/15 3/7/11 2/5/5 (9) 3/7/15 3/7/11
2 u. rate 1/2/4 3/8/14 2/5/5 (18) 3/9/13 0/0/0 (0)
3 athletes 0/0/3 2/7/11 1/1/3 1/5/11 1/1/1
4 animal h 3/8/12 2/5/7 1/3/5 (16) 3/8/13 0/4/7
5 cable t 2/6/9 3/7/11 2/3/6 (18) 0/2/4 1/5/10
6 MTXI 2/7/7 2/2/3 1/6/8 (16) 2/5/11 1/1/1
7 cell cry 1/6/12 1/8/12 0/1/6 (19) 1/2/8 0/0/2
8 classical 3/10/18 1/5/12 0/2/4 (16) 3/5/9 3/8/17
9 ecotour 2/8/13 2/5/7 0/2/5 1/5/8 2/5/7
10 IFS 3/8/16 3/10/18 0/1/1 (19) 1/8/16 1/1/2
11 Nat Gall 0/1/4 3/8/15 2/8/12 (15) 2/5/10 0/0/0
12 caffeine 0/1/2 2/2/3 1/4/4 (17) 1/3/6 0/0/2 (16)
13 QN 1/2/6 3/5/7 2/5/10 (16) 3/5/10 0/2/4
14 prozac -r 2/7/10 0/4/10 0/3/5 (13) 2/5/15 1/4/9
15 ML Bombal 2/3/4 0/1/2 (18) 0/1/2 (19) 1/2/3 (14) 0/0/0
Table B5b: The weighted score from test 5
Query \ Serv. Alta V Excit HotBot Infosk Lycos
-------------------------------------------------------
1 IDEA .76 .60 .53 .75 .60
2 u. rate .20 .73 .35 .72 0
3 athletes .11 .59 .14 .53 .07
4 animal h .66 .40 .31 .70 .35
5 cable t .49 .60 .33 .19 .49
6 MTXI .45 .18 .52 .54 .07
7 cell cry .59 .64 .25 .35 .07
8 classical .93 .57 .23 .48 .84
9 ecotour .69 .40 .23 .42 .40
10 IFS .81 .93 .06 .78 .11
11 Nat Gall .17 .80 .79 .59 0
12 caffeine .10 .18 .29 .32 .07
13 QN .28 .41 .55 .52 .19
14 prozac -r .56 .46 .34 .68 .43
15 ML Bombal .24 .14 .10 .21 0
Table B5c: Mean, Standard Deviation and Standard Error of Mean for Services
for Test 5
Serv\Statistic N Mean Stan. Dev. St. Error
-----------------------------------------------------------------
Alta Vista 15 0.4686 0.2723 0.0703
Excite 15 0.5061 0.2352 0.0607
Hotbot 15 0.3401 0.1992 0.0514
Infoseek 15 0.5202 0.1902 0.0491
Lycos 15 0.2481 0.2584 0.0667
All together 75 0.4166 0.2505 0.0289
Table B6a: First Twenty--dead links, duplicates, zeros
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
-----------------------------------------------------------------------------------
1 IDEA 1/0/0 0/0/0 1/11/2 2/0/0 2/0/4
2 u. rate 3/0/1 3/0/1 5/2/1 2/0/0 0/0/0 (0)
3 athletes 0/0/2 0/0/0 0/0/2 6/0/1 2/0/17
4 animal h 3/0/0 1/0/0 1/4/2 3/0/0 5/0/0
5 cable t 1/0/1 2/0/0 2/2/1 3/0/0 5/0/2
6 MTXI 0/0/9 0/0/0 1/4/1 2/0/1 6/0/11
7 cell cry 0/0/0 0/0/0 0/1/0 0/0/0 5/0/1
8 classical 0/0/0 0/0/0 2/4/2 3/0/0 0/0/0
9 ecotour 1/0/0 2/0/0 3/0/0 4/0/0 3/0/0
10 IFS 1/0/3 0/0/0 4/1/12 0/0/0 7/0/11
11 Nat Gall 3/0/2 0/1/0 1/5/1 3/3/0 7/0/13
12 caffeine 2/0/1 0/0/0 1/7/0 6/2/0 4/0/9 (16)
13 QN 6/0/0 6/0/1 1/4/4 5/0/1 7/0/5
14 prozac -r 2/0/0 2/0/1 1/7/0 2/0/1 6/0/0
15 ML Bombal 0/0/1 0/0/0(18) 4/1/1 2/0/1 (14) 5/0/12
Table B6b: Median, Mean and Mode for Dead, Duplicate and Zero Links
Query \ Serv. Alta V Excite HotBot Infoseek Lycos
-----------------------------------------------------------------------------------
Dead Median 1 0 1 3[2..5]* 4.5[3..6]*
Dead Mean 1.53 1.07 1.80 2.87 4.27
Dead Mode 0 0 1 2 5
Dup Median 0 0 4 0 0
Dup Mean 0 0.06 3.53 0.33 0
Dup Mode 0 0 4 0 0
Zero Median 1 0 1 0 5.5 [1..8.5]*
Zero Mean 1.33 0.20 1.93 0.33 5.67
Zero Mode 0 0 1 0 0
*Used an estimated population median and a confidence interval for
the median based on the Wilcoxon test.
QUERY ENGINE NUM1 NUM2 NUM3 DEN123 NUM4 NUM5 DEN45 1 1 259 212 71 279 259 212 279 2 1 232 57 20 279 232 57 279 3 1 252 30 20 279 252 30 279 4 1 249 185 54 279 249 185 279 5 1 252 138 47 279 252 138 279 6 1 189 125 0 279 189 125 279 7 1 279 165 57 279 279 165 279 8 1 279 259 20 279 279 259 279 9 1 259 192 47 279 259 192 279 10 1 225 225 20 279 225 225 279 11 1 198 47 0 279 198 47 279 12 1 235 27 0 279 235 27 279 13 1 195 77 0 279 195 77 279 14 1 259 155 10 279 259 155 279 15 1 269 67 0 279 269 67 279 1 2 279 168 40 279 279 168 279 2 2 232 205 20 279 232 205 279 3 2 279 165 40 279 279 165 279 4 2 262 111 37 279 262 111 279 5 2 252 168 27 279 252 168 279 6 2 279 50 20 279 279 50 279 7 2 279 179 0 279 279 179 279 8 2 279 158 61 279 279 158 279 9 2 242 111 67 279 242 111 279 10 2 279 259 30 279 279 259 279 11 2 269 215 10 279 269 215 269 12 2 279 50 20 279 279 50 279 13 2 174 114 0 279 174 114 279 14 2 235 128 0 279 235 128 279 15 2 259 27 0 259 259 27 259 1 3 101 91 37 279 111 94 172 2 3 175 88 0 279 178 91 259 3 3 259 40 20 279 259 40 279 4 3 178 74 40 279 185 74 239 5 3 205 87 20 279 205 87 249 6 3 209 125 0 279 209 125 239 7 3 262 67 10 279 269 67 269 8 3 161 40 20 279 185 54 239 9 3 232 64 0 279 232 64 279 10 3 44 17 0 279 44 17 269 11 3 182 165 20 279 199 182 229 12 3 192 71 37 279 199 71 249 13 3 151 134 0 279 158 141 239 14 3 185 64 0 279 192 71 209 15 3 212 27 0 279 212 27 269 1 4 245 208 47 279 245 208 279 2 4 259 202 94 279 259 202 279 3 4 168 148 37 279 168 148 279 4 4 242 195 141 279 242 195 279 5 4 235 54 10 279 235 54 279 6 4 242 151 30 279 242 151 279 7 4 279 97 30 279 279 97 279 8 4 235 134 27 279 235 134 279 9 4 222 118 37 279 222 118 279 10 4 279 219 30 279 279 219 279 11 4 195 141 20 279 202 148 249 12 4 178 84 17 279 195 84 259 13 4 191 144 10 279 191 144 279 14 4 225 191 57 279 225 191 279 15 4 172 47 0 219 172 47 219 1 5 205 168 40 279 205 168 279 2 5 0 0 0 79 0 0 79 3 5 20 20 20 279 20 20 279 4 5 198 98 17 279 198 98 279 5 5 192 138 54 279 192 138 279 6 5 60 20 0 279 60 20 279 7 5 195 20 0 279 195 20 279 8 5 279 235 27 279 279 235 279 9 5 242 111 0 279 242 111 279 10 5 30 30 0 279 30 30 279 11 5 0 0 0 279 0 0 279 12 5 37 20 0 239 37 20 239 13 5 125 54 0 279 125 54 279 14 5 212 121 10 279 212 121 279 15 5 57 0 0 279 57 0 279