The New Psychometrics

Science, Psychology and Measurement

Paul Kline

• Publisher: Routledge
• Year: 2014
• ISBN:
  9781317724636
• Edition: 1
• Topic: Psychology
• Subtopic: History & Theory in Psychology
• Index: Psychology

Many psychological factors are little more than statistical descriptions of particular sets of data and have no real significance. Paul Kline uses his long and extensive knowledge of psychological measurement to argue that truly scientific forms of measurement could be developed to create a new psychometrics. This would transform the basis of psychology and change it from a social science to a pure science.

Part 1
Scientific Method, Scientific Measurement and Psychometrics

1
Scientific Method, Realism, Truth and Psychology

In this first chapter I shall argue that the scientific method, which is put forward as the only way ahead for psychology, will not necessarily lead to success. There are a number of reasons for this,. In the first place, the fact that the scientific method has worked so well in the natural sciences does not mean that it works well in psychology, whose subject matter is different from that of the natural sciences. An examination of the scientific method reveals that, even in the sciences where it has been successful, it is not without problems. Thus the Popperian notion of refutability (Popper, 1959) has the difficulty that the meaning of refutable is not clear. Furthermore, the method itself has a number of philosophical problems. Thus it relies considerably on inferential logic, but there is no necessity that inferential logic,, in contrast to deductive reasoning, as used in mathematics, will arrive at the truth. This introduces further difficulties. The notion that science, by means of the scientific method, aims to discover the truth about the external world implies a realist view that there is a world out there that can be discovered. In addition, to be at all coherent the scientific endeavour assumes a correspondence theory of truth. Both these philosophical viewpoints - realism and the correspondence theory - have been seriously challenged in philosophy. Careful examination of these philosophical points reveals that both realism and the correspondence theory are tenable so that the scientific method, per se, cannot be dismissed as naive. However, the justification for its use in science has no other basis than its success in that field, which has been greater than for any other known procedure, rather than that it must inevitably arrive at the truth. In psychology, where, so far, it has not been as successful, it is clear that the nature of the subject may be a cause of the difficulty, since psychological variables axe far more difficult to measure than those in the natural sciences, and some subjective aspects of psychology may well resist measurement of any kind.

The success of the natural sciences in obtaining knowledge and understanding of the world has led psychologists to argue that the scientific method is the road to developing a successful psychology. This is most clearly stated by Eysenck in the concluding chapter of a book about his work. He writes:

My belief is that psychology is a science in the same sense, that physics or chemistry or astronomy are sciences. I believe that psychology should follow the same path as other sciences, namely the construction of theories, the deduction of consequences from these theories and the testing of these deductions, . . . In other words I believe that we must follow the hypothetico-deductive method of all the other sciences if we are to prosper.

(Eysenck, 1986: 397)

As a matter of fact there is something odd in this quotation. The triple reiteration of 'I believe' is in ill accord with what is generally understood by the scientific method, which is noted for the rigour of its thinking and contrasted as a basis of knowledge with the belief systems of religions and quasi-scientific cults. However, it is necessary to examine the scientific method since it is by no means self-evident that, just because a method works well in one sphere of knowledge, it is necessarily applicable to another. This is particularly pertinent to psychology since it is obvious that its subject matter differs greatly from that of the natural sciences.

The scientific method

One of the problems in attempting to define the scientific method is that it has become the province of philosophers. There is, indeed, a recognised branch of philosophy known as the philosophy of science. Needless to say, this has complicated the field without adding clarity. Furthermore, in elucidating the matter one is well aware of the claim made by Swift, that there is scarcely a matter of any importance on which philosophers have not been entirely wrong.

Despite these difficulties it is obvious that the scientific method should be a province of philosophy since it presupposes a view of the world which may not be tenable, and a superficial support of the scientific method, without understanding, can lead to a narrowing of vision which allows only somewhat trivial experiments. Often there is an emphasis on method at the expense of substance. As Bannister (1970) aptly put it, some psychologists are desperate to join the science club. There is an even greater problem which is particularly relevant to this book. This concerns the fact that some psychologists are content to assume that quantification, per se, implies that the scientific method has been followed, regardless of the nature of such quantification. Since psychometrics deals with psychological measurement it is a field where this particular error is highly important, and, as we shall see, many well-regarded areas of psychometrics have foundered on this difficulty.

The work of Popper

I shall begin with a discussion of the work of Karl Popper who has been highly influential in this field, particularly among psychologists. His is effectively an operational definition of the scientific method (Popper, 1959). To Popper the essence of the method lies in falsification. Hypotheses cannot be proven, only refuted. Thus the whole edifice of science is provisional. One counter-example is all that is needed to refute a hypothesis. This is what makes the scientist different from the believer in religion. Scientists,: the theory goes, are open minded, ready to modify their theories in the light of new evidence. The latter cling grimly to their beliefs regardless of the facts. This is another form of the old Platonic distinction between belief and knowledge.

For the working research psychologist this is a highly useful system. For example, I attempted to apply the method to Freudian theory, which is regarded by most scientific psychologists as hopelessly unscientific, a view derived from Eysenck (1953). I combed Freud for testable hypotheses and then searched the psychological literature for relevant experimental evidence. The results reported in Kline (1981) indicate that a number of hypotheses were not refuted, although, as Eysenck has argued, many were. In fact this Popperian approach to science is the one taught in many university departments of psychology in the UK and the USA. As Scruton (1994) has pointed Out, one of the attractive features of Popper's theory is that it appears to discriminate between science and pseudo-science. In the former the theories are clearly stated, with the aim of refutation in mind. In the latter they are deliberately vague with the aim of making such refutation impossible. It is this aspect of Freudian theory which both Popper and Eysenck have found particularly objectionable.

Problems with the Popperian position

Nevertheless, there are some difficulties with the Popperian position which need to be clarified. The first concerns the meaning of testable. This is crucial, since a theory or hypothesis is said to be unscientific if it is stated in such a way that it cannot be tested. After considerable and detailed analysis Gruenbaum (1984) was forced to conclude that the only meaning of untestable was that Popper was unable to devise a method of testing. There is a further point, not dissimilar to an infinite regress, namely that this aspect of Popper's theory, that good science demands refutability, is itself difficult to test. However, there are other, perhaps more substantial, points. The first of these involves the differences between inductive and deductive inference. In deductive reasoning, the reasoning of mathematics and logic, we argue from the general to the particular. If the premises are true the conclusions cannot be false. In induction, however, we reason from the particular to the general. Induction leads us to conclusions which are not entailed by the premises. Thus conclusions could be false even when the premises are true. With inductive reasoning, therefore, other evidence of the truth of conclusions is required than the truth of the premises.

Inductive reasoning

Scruton (1994) makes two further important points about such inductive reasoning, which is, of course, employed in many scientific experiments. Inductive inference often proceeds from the particular to the general by the pdstulation of laws. From the law a deductive inference is then made. Scruton has a good example in the rising of the sun. From observation one can induce the law that the sun rises in the morning and deduce that it will do so tomorrow. However, the law embraces an infinity of instances and goes well beyond the evidence. Its validity must, therefore, be in doubt. There is a further argument, which Scruton refers to as Hume's law. This is the claim that if one thing follows or precedes another then the connection between them can only be contingent since the world could come to an end at time t before the second event occurred. This is a severe blow to inductive reasoning since no matter how many times A may have preceded B, no general law can be made. Anything before time t will always be compatible with events after t.

Popper's emphasis on falsification is an attempt to overcome this difficulty with induction. Rather than induction, scientific thinking, the story goes, is a matter of hypothesis and refutation. As has been pointed out, scientific hypotheses are held provisionally until they are refuted. The fact, that, despite our best efforts, they have not been refuted is the evidence for their truth. However, as Scruton (1994) argues, this will not do, on two counts. In the first place, the fact that the method has worked in the past and yielded what are regarded as true hypotheses (although 'true', it is to be noted, means only 'as yet not refuted') does not mean, on the arguments of Hume, that it will work in the future. Indeed, the fact that it is thought to do so is itself an example of induction. Furthermore, despite obeisance to the notion of their provisionality, scientific laws are accepted as true, although because they are based on inductive reasoning they cannot attain that epistemic status.

The problems of induction and the circularity which appears to be involved in any rigorous defence of it as a method of reaching truth are beyond the scope of this first chapter, and a more complete discussion is given by O'Heai" (1989). As is pointed out there, a pragmatic approach can be adopted in which the argument supporting induction is that it works and that it is useful for those who employ it. This is implicitly the belief of the working scientist. Of course this does not evade the original problem because it requires induction to recognise that the method yields truth. On the other hand, the pragmatic position can, in principle, support methods that yield false beliefs if such beliefs are valuable. An obvious example might be the belief from the observation of a small number of judges that judges are wise and just. Such beliefs, if widely spread, regardless of their truth or falsity, greatly improve the administration of a justice system. The pragmatic justification of induction will not stand scrutiny, although the argument that inductive inference appears to work and is the best we can do, is not unreasonable.

There is a further difficulty with induction which Scruton (1994) discusses, one which deserves a brief consideration because it has relevance to some of the methods used in psychology and psychometrics. This is Hempel's paradox (Hempel, 1945). The claim that all oranges are orange is a general law which by the principles of induction is confirmed by specific instances. The obvious instance is an orange orange. However, there are other kinds of confirming instance including the claim that all non-orange things are non-oranges. However, if this so there is an infinity of confirmations that oranges are orange: black ties, white swans and so on. This example illustrates the difficulty of formulating the principles of induction, ofrecognising an instance, which is essential since laws are confirmed by their instances. The paradox lies in the fact that, despite the logical equivalence, the observation of a white swan and an orange orange are not equivalent as confirmations of the law that oranges are orange. Thus there are severe logical problems in the notion of inductive reasoning.

I have raised these problems of inductive reasoning and falsification simply because it is generally assumed by practising psychological researchers that the scientific method, as described by Popper, necessarily leads to truth and that by following it they are bound, as Eysenck (1986) desires, to construct a natural science, HempePs paradox of confirmation is actually relevant to the notion of construct validity in psychometrics. This will be discussed in detail in Chapter 2 but it is pertinent to note here that the validity of a test, say a personality test, is often said to be confirmed because it correlates zero with an intelligence test;, that is, we have shown that a non-personality test is not a personality test. These problems are, particularly important in psychology because its subject matter is different from and not so clearly defined as that in the natural sciences. However, this topic will be discussed on pp. 23—4.

Of course, as has been noted, the scientific method makes use of deductive logic. Indeed, it is referred to as the hypothetico-deductive method. Thus not only is induction employed on the basis of observations of the particular instances, but also the general laws are put to the test of refutation by deducing instances which can then be observed to see whether they are confirmed or not. The scientific method proceeds most fruitfully through an interaction of these two processe% induction and deduction. If the general laws are falsified, then they have to be rejected or modified in the light of the evidence. In fact, in the natural sciences, it is the impressive accuracy of the deductions, as evinced in their engineering applications and in medicine, which is such: convincing evidence that the basic laws are indeed true. Some philosophers have attempted to argue that the notion of truth is so problematic that the idea of science tracking it down is false and that scientific hypotheses about the natural world are no more true than any other set of hypotheses, for example hypotheses drawn from mysticism or religion. This relativistic position, which is the one favoured in social psychology, I find absurd, but it needs to be refuted and it is to this problem, the problem of truth and an external reality, to which I shall now turn. If there is no external reality and all accounts are relative then the programme of science is doomed.

Truth and reality

The programme of science is doomed because the aim, of the natural sciences at least, is to seek out the truth about the natural world, and science does this by means of the scientific method. Yet if there is no external reality, independent of the human mind, the scientific objective is impossible. The scientific account is different from other accounts only in that it was produced through the procedures of the scientific method.

The work of Searle: brute and institutionalised facts

Searle (1995) has a cogent discussion of this issue and I shall summarise some of his arguments which are particularly relevant to the utility of the scientific method in psychology and psychometrics. First, a useful distinction can be made between what Searle (1995) refers to as brute and institutionalised facts. Brute facts are those which are totally independent of human opinion. Obvious examples are the facts that elephants have four legs and pigs cannot, by their own exertions, fly. Institutionalised facts are dependent on human institutions for their existence. Examples of these are money or chess. That my heptagonal silver piece of metal is a fifty-pence coin depends on the agreement of government and people that this is so. Similar!}, that a chequered board and black or white pieces of wood is a chess set is purely a matter of agreement. This distinction is important because,, as Searle points out, some respectable philosophers, and many others, argue that all reality is a human creation and that, there is nothing beyond the human mind and that, as a consequence of this, statements are not true merely because they correspond to the facts - a view which, as has been said, is fatal to the sciences. Implicit in these statements are two philosophical viewpoints which must be scrutinised, namely realism - claims that the real woiid does exist - and the correspondence theory of truth. Each of these topics is highly complex but I shall try to deal with the most pertinent points as briefly as possible.

Realism

Realism can be defined as the view that the world exists independently of any human representation. If Homo sapiens had never developed, Loch Ness would be still as deep as it is, When I am dead it will remain the same. This is the commonsense view of the man on the Clapham omnibus, who by some Oxonian snobbery is deemed to be ignorant of philosophy. As Searle points out, realism is not a viewpoint which attempts to state how the world is, merely that there is an external world. He derives some interesting and important implicati...

Table of contents

• Cover
• Title
• Copyright
• Dedication
• Contents
• List of illustrations
• Preface
• Acknowledgement
• List of abbreviations
• PART 1 Scientific method, scientific measurement and psychometrics
• PART 2 The new psychometrics: the scientific measurement of personality and ability
• References
• Name index
• Subject index