Weekend 1 reading: Smith and the problem of criteria

Smith, J. K. (1993). The problem of criteria. In: J. K. Smith (ed.). After the demise of empiricism. The problem of judging social and educational enquiry. Norwood, NJ: Ablex.

This is another of the weekend 1 readings. Smith is an interpretivist. Again, as I don't really yet understand the topic, a lot of this is verbatim and will need to be paraphrased or quoted if I use it in work.

PEs, CTs and interpretivists can all agree:

• There's no definitive way to distinguish knowledge from opinion or from false claims;
• There's no definitive way to sift good research from bad.

All three paradigms agree that the empirical theory of knowledge is undermined by the fact that theory-free observation is impossible, and so on.

The end of the common ground

However, as the three theories have developed, so have major differences in their positions. The differences lie in the depth to which the PE ideas of theory-free knowledge, abandonment of direct realism and collapse of fact-value and subject-object distinctions are taken.

PEs: To develop a "modified" version of empiricism. They are unwilling to abandon completely the concepts of truth and objectivity as ideals. They suggest that, without these ideals, it is impossible to determine knowledge from opinion and good research from bad. They view the task as to modify empiricism rather than abandon it.

PEs have sought to develop the philosophical doctrine of realism, from 'naïve/direct' to 'sophisticated/indirect' realism. They also argue for the importance of certain values key to the scientific process, e.g. being guided by the 'search for truth'. They realise that certitude is impossible but one isn't railroaded into relativism; just because research cannot definitively be judged on its quality does not mean that some research is not more valid than others. "They are looking for a non-foundational theory of knowledge" between certitude and 'anything goes relativism' and between definitive criteria for judgement and no criteria at all.

CTs have attempted to develop an alternative foundation for knowledge, with alternative conceptions for truth, objectivity etc. Within CT, the ideal is to make transparent the historical socio-political forces that have led to false consciousness, distorted communication, etc. The knowledge must then be put to use so that the exploited can transform themselves and society. "CTs make no pretence that their theory is neutral or divorced from practical action".

The key task for CTs has been to develop a normative foundation for their position. Doing this, they've adopted a critical realism, interpreted objectivity and truth within a historical viewpoint, elaborated a theory of communicative action and rationality, etc. They do this knowing that certitude is impossible but that relativism  is unacceptable. They realise that judging research definitively is impossible but state that some research is more objective and valid than others. Their task has been to establish alternative criteria, alternative to empiricism, post-empiricism and interpretivism.

Interpretivists are guided by human solidarity. hey heavily discuss the 'no theory-free knowledge' concept, the collapse of subject/object dualism, the end of the distinction between facts and values, etc. They push further than PEs and CTs - they redefine traditional concepts such as subjective, objective, truth, relativism etc in non-epistemological terms.

The key position of interpretivists is that researchers do not have a unique hold on knowledge. We cannot judge, in a foundational or epistemological sense, knowledge against opinion. The best we can do is to describe forms of justification for knowledge that are common to a particular group or society at a particular time. Any judgements must be framed by practices and moral concerns, not epistemological ones.

Interpretivists believe that researchers are not privy to privileged knowledge about social life. A research study is thought of as another narrative account of our social lives, alongside other (research and lay) narrative accounts.

However, that is not to say that they believe all research is equally important or justified - they're not 'anything goes' relativists. However, this judgement is not made as to whether some research is more objective (giving an accurate depiction) but because some accounts make sense to us given our interests at this time and place. Judgements we make about knowledge against opinion and good against bad research are practical and moral tasks, not epistemological ones. Interpretivists must elaborate what lies beyond epistemology and beyond the thought that there are special abstract criteria for judging research quality.

1. The need to fill the vacuum left by empiricism with another theory of knowledge: PEs and CTs agree with this; we must be able to say something special about genuine knowledge against false claims and about good against bad research. Good research is defined in terms of objectivity. Interps do not desire to develop a theory of knowledge and believe that there is no vacuum to fill. The absence of a theory of knowledge does not mean that all claims to knowledge and all research are equal, but not better because they are more objective but what we can agree on at any given time/place.
2. Theories of knowledge differ between PEs and CTs. PEs want modified empiricism but CTs want an alternative basis for knowledge. Both have adopted realism and make claims to objectivity but have different concepts of these definitions. PEs see objectivity as true to reality as it exists. CTs seem objectivity in terms of historical distortions that have led to the present false consciousness. PEs believe that objectivity is a matter of detachment. CTs believe objectivity can only be understood in terms of commitment to the emancipation and empowerment of those who are unaware of the reality of their situations.

These differences are clearest when it comes to the interpretation of meaning, i.e. in hermeneutics.

PE make a distinction between meaning and significance. Meaning has independent existence and can be known (in principle if not in practice) at any given moment, as it actually is, separate from the interests and purposes of the interpreter. An objective account of what an author meant is one that has accurately captured that meaning. Meaning is given this status as an external referent point against which to assess interpretation; therefore we can assess the extent to which an interpreter has got it right or wrong.

Making this interpretation is not arbitrary. Using the regulative ideal of objectivity the author makes a hypothesis with a systematic approach. The extent to which the interpreter applies data collection and analysis can be judged. Furthermore, they explain how their subjectivities are made explicit and how discomfiting evidence was sought.

CTs make good on their versions of realism and objectivity with a critical or depth hermeneutics. Interpretation doesn't mean an accurate representation of the author's meaning, because the author may be mistaken. An objective interpretation points out to the victim of false consciousness the reasons for their illusions/distorted communication. In other words, an interpreter must depict what an author meant and also assess that meaning in light of objective historical conditions. This will lead to true meaning and the possibility that empowerment and emancipation will follow. Therefore, good critical research is that in which the researcher has been reflexive, clarified historical conditions, collaborated with those studied and educated meaning as to the true nature of their conditions.

Interps don't see hermeneutics in terms of theories, as they are not interested in theories of knowledge. They believe meaning only comes into being as a result of dialogue between the interpreter and that to be interpreted. There is therefore no objective or 'right' interpretation. However, this doesn't mean that there is no good or bad research - one interpretation is often agreed to be better than another. All one can do to convince another to accept their interpretation through providing materials that best support it. Interpretivists see criteria not as abstract standards but as open-ended evolving lists of traits that characterise what we think research should do and be like.

Criteria for judging social and educational research

PEs and CTs, based on their interest in a theory of knowledge, have both attempted to establish some form of abstract, general and universal criteria for distinguishing the quality of research.

PEs: Their realism leads them to judge that research which more accurately depicts reality (ie is more objective) from less accurate depictions (less objective). Whilst mistakes can be made, over time, with more research and analysis, more accurate judgements can be made. PEs differentiate between valid research and important/relevant research - this has been carried over from empiricism. Research can be of no practical or theoretical importance but can be valid through employing appropriate data collection and analysis and through thorough research. A view of importance is, necessarily, more subjective.

To CTs, good research must be both theoretically well developed and lead to emancipation and empowerment (i.e practically based). These are addressed through validity. Judgement is the extent to which the researcher accurately captures how people interpret their own expressions and those of others. This also needs to be understood in terms of the historical conditions that have led to these interpretations. Finally, good critical research must have catalytic validity - it must lead to or inspire practical action. Because of this, CTs are less likely to make a distinction between good research and important research. A researcher could objectively depict a person's interpretation and determine that the interpretation is mistaken or an illusion but not inspire people to act on that knowledge; therefore meeting the criterion of theoretical success but not that of practical agency, so it is invalid and unimportant.

Interpretivists: As the criteria problem is not an epistemological problem, criteria are not conceptualised as abstract standards. Instead, they are characterising traits. To argue that some research is good is to advance an argument for the particular traits that you think should characterise that study. Therefore decisions about the quality of an interpretive study draw on exemplars of that research tradition, involve judgemental interpretation and are a practical-moral affair.

Practical consequences

Within empiricism it was thought (in principle) that an objectivity of judgement about quality of research was possible. However, in reality, this was only ever loose and imprecise. PEs believe that although mistakes in judgement of research do occur, in the long haul, it will if it *is* good research, come to the forefront. This has implications for peer review of research journal articles.

P. 158: "It appears that there is little prospect for universal criteria to distinguish good research from bad" therefore there's little sense in the notion of a community of all researchers.

Potential concern related to the problem of criteria is the status of social and educational research/ers - no longer such as 'special or privileged knowledge', although it does still occur in PEs and CTs because of their self-reflection, openness to criticism, systematicness in thinking than non-researchers. Interpretivists shy away from this idea of a 'hierarchy of knowledge'. In the absence of a theory of knowledge there is no point worrying about a hierarchy of knowledge. There's no special knowledge about knowing and therefore no special, abstract way of distinguishing knowledge from opinion and good from bad research. Research is therefore knowledge that is different from, and expressed differently from the knowledge and language of the lay person. However, it is not superior knowledge - research is another voice in the conversation.

Summary

If PEs and CTs are correct that there is a theory of knowledge to replace the empiricist theory of knowledge, then there are abstract criteria/standards for distinguishing knowledge from belief and good from bad research. These criteria are loose and imprecise but are sufficient for the task of distinguishing good/bad research. Therefore these criteria can also serve as the basis for the claim that research knowledge is superior to other forms of knowledge - because it can be judged.

If there is no theory of knowledge (I.e. interpretivism) then there are no (epistemological) criteria for distinguishing good research from bad and knowledge from belief. Instead, there are characterising traits which are expressions of our values. The attempt to distinguish knowledge from opinion and good research from bad is a practical and moral task, not an epistemological one.

What  next? Questions to answer.

PEs: Standing between the correspondence theory of truth and the coherence theory of truth, they have yet to come up with a convincing theory which draws from both. They also acknowledge their uncertainty about the role of empirical evidence in theory choice in light of there being no theory-free observation.

CTs: Can they justify that their critique is more than just another possibility (amongst many others) for understanding the social and educational world? Is their critique of ideology only another ideology? Have *they* escaped the historical distortions and false consciousness that everyone else is enveloped within?

Interps: Have they actually moved beyond a theory of knowledge, or just side-stepped it? "A judgement that a research study is good is never more than an expression of the fact that people, guided by the ideal of human solidarity, have come to agreement about the quality of that study".

What this means???

There's no agreement on the criteria that are used to consider what is good/bad research. Different groups use different means or even believe that there is no clear way except through agreement on characteristic traits at that moment in time.

CTs believe that change should occur. PEs do not seem that different from empiricists.