CSE 473 Autumn 1998, Copyright, S. Tanimoto, Univ. of Washington 
Introduction to Artificial Intelligence (Sept. 30, 1998)

"Evaluating Artificially Intelligent Systems"



 

Extrinsic Factors: The Turing Test


What's a fair competition between a person and a computer?

Goal: to make the "user" believe it's human.

Intermediary hides irrelevant appearances.

Criterion for success is a statistical one:
  The computer is intelligent if, on the average, it wins 50%
     of the time.
 
 
 


 

Intrinsic Factors: How Does It Work?


 How does it reason?
   -- logical inference?  plausible reasoning?

 In what way does it learn?
   -- acquisition of facts? fine-tuning of skills?

 What kind of perception is it doing?
   -- vision, speech understanding, reading, feeling?

 How does its perception work?
   -- exact matching of patterns? structural analysis?

 Does it use and understand human language?
   -- can it hear, read, take commands?

 Can it solve problems?
   -- if so, must the problem be already clearly posed?

 Does it possess knowledge?
   -- in what form is the knowledge represented?
 
 


 

Evaluating Learning

  Some kinds of learning:

    acquisition of facts
    inducing rules from examples
    building skills
    forming a library of plans for reaching goals
    building semantic networks to represent situations

  Learning needs to be measured relative to a criterion function,
    or function of merit,
      f : states --> R
    Let s1 and s2 be the system's states at times t1 and t2, respectively.

    The system learns during [t1, t2] with respect to f provided
    f(s1) < f(s2)  and the change of state occurs because of the system's
    information processing activity between t1 and t2.
 
 


 

Intelligence in the Interface

  Natural language understanding --

  Poor problem solving ability does not necessarily mean
      that a system lacks intelligence....

    The intelligence might reside in its NLU ability.

    Or it may be good at recognizing your face in
      a video image.
 
 
 


 
 
 

Intelligence is Not the Only Goal

  In business it's the bottom line that counts.
  Delivery of highest quality services requires
     a mix of capabilities.

  Hybrid systems often are best, combining

  -- intelligent processing,
  -- large, high-quality knowledge and data sources,
  -- lots of computing cycles
 
 
 


 
 

Onward...

  It will become easier to evaluate the intrinsic aspects of AI systems
after we have begun building AI programs ourselves....
 
 


 
 

Last modified: September 30 1998

Steve Tanimoto

tanimoto@cs.washington.edu