Wednesday 30 September 2015

More discussions on criteria for assessing research... Hammersley

... and more week 1 reading. I think even I am beginning to see the theme in this early reading.

Hammersley, M. (1992). what's wrong with ethnography? London, United Kingdom: Routledge. (Chapter 4: By what criteria should ethnographic research be judged?)

Hammersley introduces the main issue with ethnography in social sciences which is, which criteria should be used to assess ethnographic studies. He suggests that there are three basic position:
  1. we should use the same criteria as are used by quantitative researchers. In this way, ethnography is one of many methods and is not associated with a distinctive methodological philosophy.
  2. Most believe that ethnography is within an alternative paradigm to positivism - a methodology which is more appropriate to the study of the social world than the quantitative methodology of the physical sciences. Therefore the criteria for judging ethnography will differ to those used by quantitative researchers. There is little agreement as to what "appropriate criteria" are.
  3. Some researchers believe that there can be no criteria that can be used to judge qualitative research - assessing research through criteria is incompatible with the nature of the social world and how we understand it.
Hammersley starts by looking at viewpoint three:

The rejection of criteria

Smith (1984) believes that establishing 'non-arbitrary' criteria for ethnographic research is marked by inconsistency and confusion. With the qualitative tradition being 'idealist', the definition of 'validity' is one of agreement on an interpretation. Because we're unable to have certain knowledge of an independent truth the correspondence theory of reality is not appropriate. This undermines the idea that there exist criteria by which we should judge research. "To accept that social reality is mind-constructed and that there are multiple realities is to deny that there are any 'givens' upon which to found knowledge. If one accepts these assumptions, different claims about reality result not from incorrect procedures but may simply be a case of one investigator's interpretation of reality against another's."

Hammersley unpicks Smith' argument:
  1. The claim that assessment of ethnographic research requires judgement, so there is always the potential for disagreement about the application of proposed criteria;
  2. The argument that there are no criteria whose validity is certain, so that assessing claims leads to the possibility of disagreement and potential change;
  3. The suggestion that there are no criteria in the sense of judgements that assume the reality of the phenomena studied to be independent of the researcher. Instead, the considerations in terms of which assessment is and should be made just refer to very particular, historically-located forms of social practice. The only legitimacy is that they belong to such a practice and that they are agreed on by those who engage in that practice.
Hammersley agrees with Smith's first two points - there are no criteria whose validity is a given. Applying criteria requires JUDGEMENT and the criteria are open to challenge and reformulation. Hammersley doesn't accept Smith's third argument. He disagrees with Smith's anti-realist position of rejecting that our knowledge can correspond to the phenomena it is intended to represent and that achieving such correspondence is one of the aims of inquiry: Hammersley develops his refutation of Smith's final point by affirming that we have no way of determining the certainty of our understanding of the world, but that this does not necessarily mean the rejection of the truth as correspondence.

This lack of certainty also fails to undermine the idea that some methods are more effective than others in producing knowledge of reality. It similarly doesn't counter the thought that there are criteria that we can use to judge empirical claims. All of these suppositions only follow (as per Smith's argument) if, when we discuss knowledge, it can only be known with ABSOLUTE CERTAINTY, that to be of any value, the methods used must ENSURE true findings and that the criteria must produce assessments that are BEYOND ALL POSSIBLE DOUBT.

But these assumptions are effectively irrelevant - Smith suggests you can only have realism (naïve realism) or idealism. However, there is a wider range of positions than this polar view. Many of these positions are philosophically defensible even though none can be established beyond all doubt.

Smith defines truth (within the interpretive tradition) as 'what is agreed upon' ; Hammersley states that this inhibits any rational discussion. He also believes that it is logically impossible and subject to a circular definition. He also suggests that Smith falsely suggests that choice of paradigm is down to personal preference - quantitative paradigms have criteria, but if you prefer the interpretive paradigm, you cannot have foundational criteria. Hammersley suggests that criteria for assessment should be applied but should be HEURISTIC and subject to debate.

Ethnographic criteria

Some researchers (e.g. Lofland & Lofland, 1984) suggest criteria for assessment, for the 'generic' style of ethnography. He suggests looking for several of the following features:

  • a GENERIC conceptual framework, applicable to a wide range of social phenomena. Can the research be applied to other areas/relationships or is it specific to one location/social phenomenon?
  • Is the framework NOVEL?
  • Is the framework ELABORATED? Is it more than a skeleton frame? rather, does it draw out implications and show major variations?
  • Is the framework 'EVENTFUL'? Is it illustrated by a richness of events and concrete episodes?
  • The framework should be interpenetrated with the empirical materials - i.e. both theoretical framework and the empirical account enrich each other.
This generic style differs from very specific ethnographic studies which do not produce generalisations, or contain generalised description rather than detailed empirical data.

Athens (1984) produced a similar set of criteria for assessment:
  • Does it generate formal theory? refining existing theory is less valuable. Formal theory relates to Lofland's 'generic' theory.
  • Is it empirically grounded? Are the theoretical concepts consistent with empirical observations from which they are derived? Observations need to be included in the report so the reader can make this assessment.
  • Is it scientifically credible? Is an account of the research process included?
Only this final criterion seems distinct from Lofland's.

Hammersley discusses another attempt to specify criteria for judging ethnographic research, that of Guba and Lincoln (1981 and 1985). They believe that the NATURALISTIC approach is needed for studying human social life. They believe that the four major traditional criteria (truth value, applicability, consistency and neutrality) can be used but must be formulated in a way that differs from the rationalist paradigm, as follows:

  1. Truth value: concerned with credibility; do the people studied find the account produced to be true?
  2. Applicability is related to transferability; although naturalists reject 'generalisability', they believe that there can be some transferability, if enough 'thick descriptive' can be produced.
  3. Consistency equates to dependability; because of the emergent nature of research design in naturalistic research, replication is impossible.
  4. Neutrality takes the form of 'confirmability'; is the analysis 'grounded in the data'? Are any inferences based on the data logical and of high utility?
This viewpoint adds to Lofland and to Athens. Hammersley synthesises these to bring together a new list:
  1. the degree to which generic/formal theory is generated;
  2. the degree of development of the theory;
  3. the novelty of the claims made;
  4. the consistency of the claims with empirical observation and the inclusion of representative examples of the latter in the report;
  5. the credibility of the account to readers and/or to those studied;
  6. the extent to which the findings are transferable to other settings;
  7. the reflexivity of the account: the degree to which the effects of the findings of the researcher and of the research strategies employed are assessed and/or the amount of information about the research process that is provided to readers.
Hammersley suggests that there are questions that need to be asked about these criteria:
  1. are they applied to all ethnographic research? For example, should all ethnographic research be concerned with developing formal theory? *Er, No!* We need to modify the application of criteria according to the intended product of the research.
  2. Credibility shouldn't be define as whether readers or people studied judge the account to be true. The researched may not want to  acknowledge the truth. The studied do not have privileged access to the truth. The responses to the ethnographic accounts are useful data sources but agreement with these accounts should not be a criterion for assessing research. If it were, bland, agreeable results would be produced.

Applying quantitative criteria

Evaluation for quantitative methods centre on:
  • internal and external validity
  • reliability and validity.
The first subsumes the second.

Hammersley suggests that application of these criteria to ethnography is unsound, even to quantitative research. Campbell's development of the concepts of internal and external validity takes the 'quasi-experiment' as its research model, but it's been used for other research. The fcus is on designing research to rule out threats to validity.

INTERNAL VALIDITY: whether, in the specific experiment, manipulation of the treatmentproduced variation in the outcome.

EXTERNAL VALIDITY: whether a relationship discovered in an experiment can be generalised to other situations.

You can have internal validity without external validity.

Hammersley believes that this talk of different types of validity is misleading. The findings of a study are either valid or not - they can't be valid in one sense but not in another.

The other framework from quantitative methodology that is often applied to ethnography is the distinction between reliability and validity. There are ambiguities in the definitions.

VALIDITY: Accuracy with which a description of particular events represents the theoretical category that it is intended to represent and captures the relevant features of those events.

RELIABILITY: The degree of consistency with which instances are assigned to the same category by different observers or the same observer on different occasions. It provides evidence about validity and tells us about the usefulness of the particular strategy used.

Validity and reliability are appropriate for judging methods and products of research, but they are not sufficient. We need to consider alternatives for internal/external validity and the question of relevance.

A reformulation of the criteria

Some of what Guba and Lincoln identify as criteria are means by which the validity of qualitative research may be assessed, rather than criteria themselves - definitely 'credibility', 'consistency', 'neutrality' and possibly 'transferability'.

Hammersley concentrates on criteria of assessment and standards by which research results should be assessed. we need to first consider: what is the PURPOSE of such research? What goal is it intended to serve? Hammersley suggests that research is "to provide information and relevant to some legitimate public concern". Based on this definition, we need to judge on two criteria: TRUTH (validity) and RELEVANCE. These apply to both qualitative and quantitative research.

The importance of validity is obvious, if not unproblematic. However, relevance is important, including it being of importance to those outside the research community. Sound research requires the criteria of validity and relevance.

Validity

Hammersley suggests that he is a 'subtle' realist, using validity as a synonym for 'truth'. "An account is valid/true if it accurately represents those features of the phenomena that it is intended to describe, explain or theorise". This assumes a correspondence theory of truth. Since we cannot know the extent to which an account is true, we have to judge validity on the basis of the adequacy of evidence used to support them. We must recognise that judgements on the truth of knowledge claims rely on assumptions, many of which we are not consciously aware of, and many of which have not been subjected to rigorous testing.

We must be aware that potentially we need to keep challenging each new piece of evidence and so on ad infinitum, so where does this stop? we make this sort of judgement on a daily basis in real life, taking into account such considerations as: judgements about what is beyond reasonable doubt; likely costs of error; the scope for acquiring further evidence before a decision has to be made; the likely value of that evidence; the costs of getting this evidence; and so on...

Limits on time and resources as well as an understanding of 'beyond reasonable doubt' will set the bar for sufficiency of evidence. Hammersley suggests there are three important considerations with regard to sufficiency of evidence:

  1. Issues of plausibility/credibility: are the claims made sufficiently plausible, given our existing knowledge? If so, accept them. If not, is the claim reasonably likely to be accurate (i.e. is it credible), given what we know about the circumstances of the research? researchers must ensure their findings are sufficiently plausible/credible including through anticipating the likely judgements of fellow researchers. They must provide sufficient evidence to counter any concerns over the research.
  2. The evidence should relate to the level of centrality of the claim to the researcher's argument. A claim that is central to the researcher's argument will require more convincing evidence than a more marginal claim.
  3. The type of claim made: we need to distinguish between: definitions; descriptions (claims about what happened at a particular time/location; explanations; theory (relationships between different types of phenomena, wherever instances of those types occur). The validity of claims for supporting theories is more complex than that of descriptions, therefore needs more evidence, via descriptions of a range of examples. We'd also need evidence to show any confounding factors could not cause the link. This requires the use of a variety of sources of information.
Hammersley suggests that the assessment of validity involves identifying the main claims made by a study, noting the types of claims these represent, then comparing the evidence provided with each claim with what is judged to be necessary, given the claim's plausibility and credibility.

Relevance

Work should not only "just" provide 'truth' but also be judged in terms of its relevance. But, relevant to what/whom? - A limted audience, generally. We need to consider the 'audience': bear in mind the researcher's own specialisations of topic but also different methodologies and theories. Hammersley believes that the primary emphasis should be on substantive relevance.

Relevance to researchers can be considered through:
  1. Importance of the topic: the centrality of the topic studied to a substantive field; the ideas about importance should reflect wider societal values and circumstances.
  2. Contribution ot the literature: mere confirmation of previous research is of little value. It must make a significant contribution to established knowledge. This links to Lofland's and Athens' concerns with novely and theoretical development.
we may also need ot consider relevance of the work to practitioners, as well as researchers, e.g. application of findings to practice; this i not so relevant in ethnography. Hammersley suggests that the role of research influencing practice is limited - we need to consider the same two aspects of relevance, though with diffferent implications.
  1. Importance of the topic: will the research help the practitioner with a current problem? A short term and specific judgement.
  2. Contribution of findings: confirmation of what is already known is of little use to the practitioner too. However, the researcher can question what the practitioner takes for granted.
Which is the most important audience for ethnographic research - other researchers, or practitioners? Assessments of practical relevance of research are continually changing. Judgements of the relevance of studies rely on prior value judgements. How can we justify those judgements? Hammersley believes there are imprtant questions about the ability of ethnography to produce relevant results. Situations studied by ethnographers rarely have intrinsic relevance; they have to be made relevant by generalisation  or theoretical inference.

Summary: Two over-arching criteria that must govern assessment of the value of social research: validity and relevance.

Who is Martin Hammersley?

Professor of Educational and Social Research at the Centre for Childhood, Development and Learning at The Open University. He has carried out research in the sociology of education and the sociology of the media. However, much of his work has been concerned with the methodological issues surrounding social enquiry. http://www.bloomsbury.com/author/martyn-hammersley#sthash.L4aMeWUO.dpuf

Any critiques of his work? I can't find any with a quick look.


OK, so what do I think? It makes sense. Sort of. I'll think some more after the first weekend session.


Sunday 27 September 2015

More reading... Sparkes on the Paradigms Debate

Alternative title: Help me, my brain is melting...

This was a big article. I got lucky with Eisner as my first read - it was an easier read and made intuitive sense. This overview of three key research paradigms, however, was not so easy.

This wasn't helped by coming at it piecemeal - 20 minutes here, half an hour in the soft play centre there, and so on.

Much of what I'm going to write (and there's lots because I found it so difficult) is pretty much verbatim from the text, as I don't yet have the knowledge to be able to condense. As I always tell students asking about paraphrasing, if you don't understand the text, you can't effectively paraphrase it. So, this is mostly direct from text.

Sparkes, A. C. (1992). The paradigm debate: An extended review and a celebration of difference. In: A. C. Sparks (ed.). Research in physical education and sport. (pp. 9-60). London, United Kingdom: Falmer.

There has been a revolution in philosophical paradigms, including in Sparkes' specialism (PE), where empirical predominance is being rethought. [Bear in mind that this was written in 1992, so there has probably been a lot more re-thinking since then!]. The purpose of the paper is as a framework to aid understanding of the debate that is taking place. It covers three paradigms, interpretive, critical and positivist.

There is a discussion on the nature of paradigms but effectively that they are a shared conception of problems and the methods shared within them. They can be described as a shared world view, a way of breaking down the complexity of the world. Paradigms are deeply embedded within the researcher's own socialisations, experiences and framework. Herein lies their strength and their weakness. The strength is that they make action possible without thinking about *every* angle of research. Their weakness is that there are hidden, unquestioned assumptions within each paradigm - an awareness of them is needed to ensure an understanding of the unseen frameworks within which the researcher operates. Paradigms act as a theoretical lens through which we experience the world.

To add to what Sparkes says, and to link to Eisner: these frameworks and assumptions within each paradigm don't just affect how we think and act but at a more fundamental level it will act as a barrier as to whether we *see* specific aspects of what we observe. I'm not sure that's clear, but I think I know what I mean there. We are blinkered, so we don't know that we can't see something, so we have no ability to research/understand it, because to us, it's not there!

I suppose it makes me uncomfortable that Sparkes states "the individual must not only learn the content of the field but also a particular way of seeing the world that eventually becomes not only unquestioned but unquestionable." Is this a healthy way of working? Perhaps so, If the alternative is a philosophical dam against which we ultimately can't move because of the inertia of philosophical vacillation.

During the 'socialisation' process into a paradigm the researcher absorbs assumptions on ontology and epistemology. Social scientists need to question whether there is an external 'reality' or one coloured by individual consciousness and experience - ONTOLOGY.

External realists of positivism vs the internal idealists of interpretivism.

The epistemology - how we know what is true or false - differs too. The objectivist view is that knowledge is capable of being transmitted in a tangible form and is 'hard' and 'real'. The subjectivist epistemology believes that the truth/knowledge is subjective and as such subject to interpretation through our individual lenses or the lenses employed when working within a specific paradigm.

There is a third set of assumptions, based on human nature:
Deterministic: people react to their environment and are products of it;
Voluntaristic: people have control over at least some of their environment and exhibit autonomy, "actively creating their environment".

The standpoint of the researcher on these three sets of assumption affects how they gather date:

1. External realist vs internal idealist
2. Objectivist vs subjectivist
3. Deterministic vs voluntaristic.

Sparkes discusses these assumptions to show that all researchers make assumptions relating to ontology, epistemology, human nature and methodology. These assumptions cluster and are given coherence within the frameworks of particular paradigms. No researcher approaches research as a blank slate - ontological assumptions give rise to epistemological assumptions which have methodological implications for the choices made regarding particular techniques of data collection, interpretation of findings and the ways they are understood and discussed in texts or orally presented.

Essentially, those working within different paradigms view the world in different ways, investigate situations in different ways and report the results of their investigations in different ways. The researchers' basic assumptions concerning ontology (reality), epistemology (truth), the physical world and the social world affect all aspects of research. Confusion is increased through the inaccurate use of words such as methods, research methods, and methodology. Sparkes uses the following definitions:
Methodology: the philosophical underpinning of the investigation
Research strategy: the design and carrying out of the investigation
Research techniques: specific methods to yield data, e.g. questionnaire construction, statistical analysis.
Methods cannot be independent of the philosophical underpinning and similarly the interpretation of results and their recording. "Techniques of data collection do not constitute the uniqueness of a paradigm" - the same data collection method can be use din both a positivist and behavioural way but the meaning put on the findings will differ, as well as the content gathered.

In investigating various paradigms, Sparkes explains that it is important to avoid caricatures. It must be remembered that within each paradigm there is heterogeneity - different traditions within each form of paradigm.

The Positivist Paradigm


This has a  long history, and uses the notions of science as a framework. It is historically important - Sparkes discusses its role within PE and that it offers "almost unquestionable respectability". For example, objective measurement of teaching and learning, the use of standardised data collection instruments. Inter-related assumptions listed by Popkewitz (1984) shape the positivist paradigm:
  1. theories are universal and not affected by the values of the researchers;
  2. the science is "disinterested" and not affected by the values of the researchers;
  3. the social world can be reduced to variables that can be studied independently;
  4. concepts can be formalised and defined so as to provide dependent and independent variables to manipulate;
  5. use of quantitative analysis to reduce ambiguity.
Ontologically: the social world is ontologically objective and concepts such as intelligence, self esteem exist separate from the individual. The researcher observes nature in such a way as to not affect nature's answers. Sources of bias must be recognised and controlled. Internal and external validity, as well as reliability must be provided by the research techniques.

VALIDITY can be established when the extent to which conclusions represent empirical reality, and also when assessing whether the constructs devised by researchers measure the categories of human experience that occur. "A judgement is 'true' when it corresponds to this external reality and 'false' when it does not - the correspondence theory of truth. Observation (empirical verification) is needed so that we can judge whether a statement is true. To achieve objectivity, positivists follow the 'scientific method'.

The interpretive paradigm


In contrast to the positivist paradigm is the interpretive paradigm, an umbrella term within which sit a whole range of methodologies. We cannot lump them all together as 'qualitative vs quantitative' as some interpretive methodologies are at least partially quantitative. It only cam einto more common usage in the late 1980s, and developed as a reaction against positivism, as it was suggested that studying the social world couldn't be achieved objectively.

The interpretive paradigm adopts an 'internalist idealist' ontology and a subjective epistemology and prefer an IDEOGRAPHIC methodology. For example, facts only exist within the context of a mental framework (construct); i.e. reality is only viewed through a window tinted by values. Knowledge is the consequence of human activity, i.e it is a human construct and can never be agreed as ultimately true. The paradigm takes a position of RELATIVISM - there are multiple realities existing within different individual's minds, and are open to subjective interpretation. The mind of the researcher doesn't "create" the findings, but colours the interpretation of the findings, assigning meaning and intentions makes the social reality.

The paradigm rejects the positivist idea of an independently existing reality that can be found through specific methods. Even 'objective' methods are value-laden and open to interpretation. It takes an EMIC approach. Sparkes discusses authors' views on ethnography, explaining social reality from within the group, as opposed to the positivist view, detachedly looking in.

Interpretive research and the 'researcher as instrument'
Sparkes discusses researchers viewing the role of the researcher within ethnographic studies as being *the* most important research instrument - it is "engagement of the self". Within ethnographic studies, the data that are not gathered, as well as those that are, shape the research - the framework within the researcher will affect the data gathered.

In interpretivism, truth/validity isn't a matter of correspondence, it is a matter of COHERENCE - what is true is what we can agree (within the confines of that time/context) is true. 2Within a coherence theory of truth, a proposition is judged to be true if it COHERES (connects consistently) with other propositions in a scheme or network at a particular time - COHERENCE is a matter of internal relations as opposed to the DEGREE OF CORRESPONDENCE with some external reality.

Popkewitz describes objectivity in this area is the result of inter-subjective consensus through social interaction. He also states that this also applies to the scientific community - knowledge of science is considered valid and truthful only insofar as it reflects the consensus of the scholarly community.

The "truth" of an ethnographic report depends on how well it "rings true" to natives and colleagues within the field. However, interpretation remains an INTERPRETATION of a set of events; credibility is not necessarily is altered by agreement or disagreement by the subjects. The subjects will have their own interpretation of their social world (first order constructs) which often differ from those of the researcher (second order constructs). It is still important for the researcher to check their findings and that certain words were used/events took place. It's also important to discuss interpretations even though there may be disagreements - these, in themselves, can be of use and provide further data for interpretation - REFLEXIVE ELABORATION. Furthermore, there may be disagreements amongst scholars studying the same group. These differences are generally those of emphasis or orientation. "qualitative research cannot be made researcher-proof". Because multiple interpretations can be made even on the same group, there can be 'many truths' available. So, with multiple interpretations possible, which is 'best' or 'most nearly correct'?

There can be multiple truths through multiple interpretations, but we must make sure that we don't lose our critical abilities - not all interpretations stand up to scrutiny. This is true, even though interpretivism is relativistic. There are three main views of relativism (Rorty, 1985):
  1. every belief is as good as any other;
  2. "true" is an equivocal term, with as many meanings as there are procedures of justification;
  3. There is nothing to be said about truth or rationality apart from descriptions of the familiar procedures or justifications which our society uses in areas of inquiry.
Most interpretivists adhere to the third view - judgements of truth are relative to a particular framework, paradigm or point of view. Therefore, not everything goes; researchers within an interpretive paradigm differentiate between 'good' and 'bad' research, according to their own framework.

Some researchers have discussed how constructivist research can best be judged, in this case. e.g. trustworthiness, credibility, transferability, dependability and authenticity. Under these come sub-lists - these checklists help to judge the 'goodness' of constructivist research.

Harris states that at least three levels of interpretation are needed for 'good' qualitative research:

  1. Grounded in shared understandings about the culture developed between the researcher and members of the group being examined.
  2. Must include the researcher’s insights of the culture that are not well-articulated by the members of the group;
  3. Must include theoretical generalisations that go beyond the group culture to link to relevant parts of other research.

Athens discusses here criteria for evaluating qualitative studies.
  1. Theoretical import – the contribution towards new theories or refinement of existing theories;
  2. Whether the scientific concepts are empirically grounded;
  3. “Scientific credibility” – the researcher must make the study credible by providing an account of the research along with a description of the results. An account is a story told by the researcher about how they performed the research in question.
It is important to remember that different researchers using different frameworks and theoretical underpinnings, when observing the same group, may well provide different interpretations using sound interpretive research.
To try to consider which research is ‘stronger’ we are tied by HERMENEUTICS – truth is constructed by the rhetoric of the researchers. “A good explanation makes sense of the behaviour, but to appreciate the good explanation, one must agree on what makes good sense. What makes good sense is a function of ones readings, and these in turn are based on the kind of sense one understands.” ”Truth is what we agree to be true at any one time”.
However, different interpretations can co-exist, providing a richer view of a culture.

The Critical Paradigm



There is no such thing as ‘critical theory’; it’s an umbrella term for a range of different theories. The key commonly shared assumption is EMANCIPATION – enabling people to gain knowledge and power in order to take control of their lives. There is a dismissal of positivism as a means of enforcing the status quo whilst producing theories that are “often trivial and useless”.

Critical theory developed from the work of the Frankfurt School; it is overtly political, highlighting substantive social issues, uncovering the oppression within these issues and, importantly, doing something about it.

With regard to ontology and epistemology, there are two strands running through the critical paradigm:
  1. Associated with positivism – a RADICAL STRUCTURALIST strand with an external realist ontology, an objectivist epistemology and a deterministic view of people. It concentrates on STRUCTURAL relationships within a realist social world via analysis of deepseated internal contradictions and the analysis of power relationships.
  2. Associated with radical humanism – similar to the interpretive paradigm, it has in internal-idealist ontology, a subjectivist epistemology and a voluntaristic view of people. It is believed that reality is socially constructed, knowledge is context specific and value-laden.
Critical theorists see interpretive paradigm research as having major weaknesses as they do not consider the wider sociohistorical, political and economic movements on the theories.

One major concern within education research (PE) is that interpretivist research has not taken into account the way individual and group behaviour is influenced by the way society is organised. Findings are determined within a social and organisational context permeated by the inequal power relationships (cf Foucault?) Anderson (1989a) suggests these following critical research questions regarding the nature of knowledge in organisations:
  1. What counts as knowledge?
  2. How is what counts as knowledge organised?
  3. How is what counts as knowledge transmitted?
  4. How is access to what counts as knowledge determined?
  5. What are the processes of control?
  6. What ideological appeals justify the system?
The central emphasis is the way human consciousness is shaped and controlled by existing social arrangements to serve some groups in society at the expense of others (INEQUAL POWER RELATIONSHIPS). Relational analysis is often used to analyse these relationships – viewing the historical, social and cultural constructs surrounding the practice under study.

Griffin (1990): “
  1. Society is made of groups with power and privilege and those without;
  2. Social institutions in a society perpetuate the status quo of this power imbalance;
  3. The powerful and privileged have a vested interest in maintaining the status quo (their power and privilege);
  4. The powerless and disadvantaged have a vested interest in social change;
  5. The competing interests result in conflict and tension, often below the surface of apparent harmony and consensus;
  6. The critical perspective brings to light the contradictions between apparent harmony/consensus and conflict/tension, in order to problematize the status quo;
  7. A critical perspective is concerned with ‘why/why not’ questions – whose interest is served? The intention is to change the world, not describe it;
  8. A critical perspective believes in the importance of changing individual and group consciousness in creating social change.
Critical researchers may use data collection techniques from the interpretive paradigm, but go further, to investigate the inequal power relationships – critical ethnography. This can be done in three ways:
  1. Consider the subject group in a wider social context. This is weak as the critical analysis can be omitted if it just analyses functional relationships between the subject group and the wider social milieu.
  2. Critical ethnography through focussing on the wider structural relations and examining how social processes in the subject group are mediated by structural relations.
  3. The strongest form is to incorporate ethnography into a DIALECTICAL analysis – the understandings from the ethnographic study are analysed in relation to the social structures that shape the lives of people.
This final method begins with structural relationships and then undertakes an ethnographic study in order to facilitate a structural analysis.
Where critical ethnographers differ is that they claim that the subject’s perceptions of social reality are permeated with meanings that sustain powerlessness. The conscious models used by people exist to ensure the continuance of the social phenomena.
Researchers investigate the process by which certain meaning structures become accepted as the status quo. They then consider whose interest the status quo benefits. They actively engage with the social group to elicit transformation through understanding and action. They seek to change the world. Examples of the critical paradigm are: feminist research and neo-Marxist critical ethnography.
For transformation to occur, research must be done with the full participation of the people under study. The researcher provides those researched with insights that might act as the basis for change. The researched are participants in the process not subjects to be studied.
Validity has a very different definition in critical research. It needs to be understood as being relevant to the practitioner’s situation, and potentially transformative, to be valid. “Validity in critical research relates not only to the trustworthiness and credibility of the interpretation but also how effective the research process has been in empowering the participants and enabling them to create change.” This is very different to the positivist view of researcher neutrality. Therefore, validity’s definition changes significantly depending on the paradigm within which you work.

Comment by Sparkes

Validity derives its meaning from different sets of assumptions, theories and purposes within the different paradigms. Each paradigm needs to be understood in its own terms. It is important to recognise that each paradigm can help develop our understanding of the social world, and also that new paradigms will be developed. You must judge research carried out within a paradigm according to that paradigm’s own criteria and assumptions. It is important to be aware of the assumptions within each paradigm, even if you do not work within it.

What does all this mean to me?

Not a lot at the moment…
Firstly, this paper was written a long time ago, now. I’m sure the arguments have moved on and that there are new paradigms that have taken over as the most in touch with the ‘zeitgeist’.
Secondly – the long words… my brain hurts…
Thirdly, whilst I get the gist of critical research, I doubt I could get involved in it. The thought of being transformative in the research seems such a huge jump, especially with my background in empiricism and the ‘detached scientist’ viewpoint. Interpretivism doesn’t seem quite such a huge leap, though. We’ll see. Lots more to read, and I need to go back to look at some of the terms I don’t understand.

Saturday 19 September 2015

Eisner - objectivity in educational research

Eisner, E. (1992). Objectivity in educational research. Curriculum Enquiry, 22(1), 9-15.

This is the first text I've read for the first Ed.D. weekend, which is in a fortnight. We've been given a list of texts to read, but no context with which to read them. I suppose I should look at it as an introductory and mind-widening process - learning to think and consider the frameworks within which we all think unless we are challenged to do otherwise. I'll read through all the texts and see what themes emerge.

This is going to be quite descriptive, as I have to get to grips with the ideas he's dismissing, before I can understand why he's dismissing them. Lots of new terms, which I'll describe first:

Ontology - pertaining to being

Ontologically objective - a belief is ontologically objective if the belief accurately describes external phenomena. In  other words, it is seeing the world as it is, unfiltered by our own conceptions. To see things the way the are is to experience or know them in their ontological state" (p. 10)- this is known as veridicality

Procedural objectivity - deals not with our beliefs but our methods of decision making. For example, a teacher is procedurally objective when she assigns grades based on the merit of the work and not her knowledge of the student.

Veridicality - the degree to which an experience, perception or interpretation accurately represents reality.

Correspondence theory of truth - the truth or falsity of a representation is determined solely by how it relates to a reality - whether it accurately describes that reality.

Who is Eisner?
A professor of art and education.

Description of the article

Eisner starts by asking us to consider what is meant by objective and what we mean by being 'as objective as we can'. He suggests that neither ontological objectivity nor procedural objectivity provide a meaningful measure of objectivity. He discusses the role of episteme (truth) and doxa (belief). What we know is truth - if it isn't true, we couldn't know it, but only believe it to be true, but belief is different. "Ontological objectivity gives an undistorted view of reality".

Procedural objectivity eliminates scope for personal judgement, for example a scored achievement test marked by computer. "Traditionally the aim of research, from a methodological perspective, is to use a procedurally objective set of methods in order to gain an ontologically objective understanding of the events and objects we study".

My thoughts: This makes sense in relation to my background in science. Procedural objectivity ensures reproducibility of results, for example of a microbiology test. With sufficient procedural detail, a microbiologist on the other side of the world should be able to replicate the tests and results that I achieve in the lab. This relates to SOPs, ISO standards and so on - procedural objectivity.

Eisner asks the question - how can we know whether our view of reality corresponds to the ontological reality? He quotes Popper (1959 - look up!) who says that "we can never verify the truth of a claim we can only refute it (and even refutation can be uncertain."

Perception is influenced by the framework within which we operate - we are influenced by our traditions, language, etc. we only see and interpret what we are able to see. If we do not have the means to interpret it, we will not seek it.

Another issue with ontological objectivity is the limitation of representation, be that through language, visual art or oral compositions etc. The medium we use to represent the world as it is both reveals and conceals reality, leading to only a partial view of the reality we seek to describe.

A further complication is the use of schemata which we use to structure our perception. Schemata are "structures of appropriation"; these schemata define how we interpret and understand the world. each group creates its own world, so which is the objective one?

In summary of this first part, Eisner proposes that it is effectively impossible to achieve ontological objectivity.

He posits that procedural objectivity can be achieved, but in a limited way, for example a multi-choice, computer-read assessment tool. However, this does not demonstrate an understanding of reality, merely an understanding of a general consensus.

So, why do we need objectivity? Intellectual traditions stemming from the Enlightenment - we crave an "objectively knowable world".

Eisner proposes that we should recognise that what we think we know is a function of a transaction between an ontologically objective view of the world, which we cannot know, and the frames of reference and personal histories we bring to them. These are moulded by the culture within which we operate. There is a "transaction" between objective conditions and personal frames of reference, of which we make sense- this is experience. Experience, therefore is made, not had - it is constructed.
The construction depends on the frameworks we are able to use and the skill with which we use them. Acculturation and education can be considered as psychosocial processes that provide frameworks to the young, to ensure a commonality of framework with others. If there is no communis - sharing of frameworks, there is no common ground and no understanding between people.

"Knowledge is always constructed relative to a framework, to a form of representation, to a cultural code, and to a personal biography". (p. 14).

So, what does this mean? Eisner suggests:
- there is no singular way to make sense of the world, but a participation in a variety/plurality of worlds, but with a common understanding.
- truth can be retained as an ideal but with the understanding that what we understand to be the truth will vary depending on our framework for perception and understanding (and also representation?)
- the history of science shows that what we understand as truth changes.
- All experience is transactive.
- "Belief, supported by good reasons, is a reasonable and realistic aim for inquiry".

What does this mean to me?
Well, it makes sense. How we see the world, and how we interpret it, is coloured by our own experiences. How I, as a white, female, middle-class, well-educated, British person see the world will differ from people experiencing the world through different frameworks. In this way, this was a useful text to pick to read first - I think it might inform how I understand some of the other texts.

Does it relate to my research ideas? Perhaps.
Having an understanding of my own perceptions and how they can cause me to interpret the world differently will hopefully give me greater awareness of the different frameworks of others. I think I'll have to read more about this to understand its relevance, as I get the feeling it's something I agree with but I need to read some other viewpoints. I believe (doxa!) that one vociferous critic was Denis C. Philips, so I should go away and read that. I can't find the original paper, but there's a discussion of that paper at the top link (below). I'll write more when I've had a chance to read it.

 
 
www.open.edu/openlearnworks/mod/resource/view.php?id=11426

Thursday 17 September 2015

Interview

Yesterday I attended an interview for admission onto the Ed.D. I had visions of being quizzed about Derrida, Bourdieu and my instinctive leanings towards positivism and how I can overcome this. In the morning, my brain had a slight existential meltdown as I considered my possible thesis questions, and decided that there was know way I could really 'know' anything, except through my white, middle-class 21st century eyes, so what was the point?

The interview, however, was a gentle chat about how I'd come to the decision that I wanted to do the Ed.D, and what possible areas my interests lay in for a thesis, to see if they aligned with expertise within the department. It was lovely. I couldn't stop talking. It sounds as though I've been accepted (hurrah!) but I have to wait for official confirmation. The cohort size will be around 10, so hopefully I will overcome my natural reticence to talking in groups pretty quickly and be able to contribute. That's something I will have to force myself to do, along with networking. wallflowers don't get anywhere - I need to make the most of this opportunity to develop myself not just academically but other skills, too.

Sunday 13 September 2015

"Just" an Ed.D.

I attended a session at the annual staff conference last week, on transition from school/FE to higher education. A couple of colleagues had been awarded £10,000 to investigate students' views of transition, and had used the money to video the students talking freely in answer to three questions on transitioning to HE.

The answers were interesting, not only because of the content and its implications to our practice as study skills advisers, but also in the language they used. The researcher drew attention to one student (not in the clips we were shown) who said that he'd "just" done a BTEC before coming to university.

So much to unpack in the one word - "just".
Does he feel that his qualification, which would have taken two years, is inferior to other options? If so, why? Are his feelings mirroring the media view of 'dumbed down' vocational qualifications - a legacy of Mr Gove's desire to denigrate these qualifications and raise 'A' levels onto a dais (albeit a dais he was happy to throw coconuts at)? Does he feel his academic background lacks in something that other students may have experienced? If so, what? Why this feeling? So many questions.

This would be an interesting topic to study in itself - perception of vocational qualifications on entry to HE. Having just *very* quickly searched on the topic, I came across this quantitative study. It's really interesting - young people are more likely to denigrate vocational quals than their parents, or anyone else. I suppose the parents have a vested interest in thinking that their child is doing a "worthwhile" qualification.
https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CD0QFjAEahUKEwitvLbslvLHAhVCORoKHZ8BC-o&url=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fuploads%2Fsystem%2Fuploads%2Fattachment_data%2Ffile%2F447683%2F2015-07-22-perceptions-of-a-levels-gcses-and-other-qualifications-in-england-wave-13.pdf&usg=AFQjCNFBVR97rYGWNUkeJK-SSRgRKdnKqQ&sig2=hw70WfsB3bbjVxwoT6kQtA

Why do I bring this subject up, apart from it being a really interesting question in itself? Well, I suppose I feel a little like that young man. Not so much with my entry qualifications (though, science .......). But, how is the EdD viewed? I have to explain to lots of people (e.g. my father) what an EdD is. He knows what a PhD is. He knows (in his mind, at least) that this is a pinnacle of academic endeavour, something he could never have imagined his daughter could have aimed at. But an EdD? What's that? Is this an opinion echoed by academics? Some literature seems to suggest a lack of parity of esteem between the two. If I were writing an essay, I'd put a few references in, but I'll just stick this in for now http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&uact=8&ved=0CEkQFjAFahUKEwi9xsTkx_THAhXDfRoKHSCjCLo&url=http%3A%2F%2Fwww.ukcge.ac.uk%2Fpdf%2FProfessional%2520Doctorates%2520in%2520the%2520UK%2520(2011).pdf&usg=AFQjCNEN5OfG-417jRc8y89EHx2XBFkQog&sig2=1FVcG-L1C5zZzN9DTKJD0A

What do I feel? I'm ambivalent. Academically, the two are assessed at the same level. However, one is of greater practicable use within the professional environment. But still, a little bit of me thinks like the BTEC student - "just" an EdD.

Friday 11 September 2015

Hippopotomonstrosesquipedaliophobia

Or, in slightly easier language, fear of long words. This is a fear that I am starting to exhibit, even before I am interviewed for a place on the Ed.D.

It is true that every sector has its jargon. Whether it's the ballcocks, flux and pipe-benders of plumbing or the JIT, kaizen and fishbone diagrams of lean management. Certainly, my background in food safety is full of jargon and also long words. Escherichia coli, Yersinia enterocolitica, and Staphylococcus aureus fair trip off my tongue, but to the uninitiated are complete gobbledegook.

But, combine the jargon with complex concepts from philosophers ranging back over 2000 years, and my heart starts to race a little. And not in the good way when you can't wait to get started on something. My biggest fear (well one of many) on starting the Ed.D is one of looking a fool because I don't know my Foucault from my elbow. Complex ideas which need to digested, understood and applied to new learning. O.M.G. as they say :-/

I'm going to have to do a lot of reading to get up to speed on these theories. Not only are the theories complex, but the literature uses words that I recognise but really don't understand. Soooo, reading with a dictionary close to hand, although a dictionary won't provide contextual understanding which is what I will really need. I think I will need to start some form of glossary, both for words and for ideas/theories. More gobbledegook! I really don't want to  mix up my epistemology with an episiotomy. That could be very painful.

This evening, I shall start reading Adams, Cochrane and Dunne's Applying Theory to Educational Research - one of the required texts for a first module. I better get that dictionary...

Thursday 10 September 2015

A new day, a new dawn, a new life and I feel...

...terrified?

A long, long, time ago, I was sort of given the opportunity to study for a PhD. I was coming to the end of my Master's and had done an assignment that had impressed a lecturer in poultry welfare. He suggested I might want to apply for a PhD studentship they had going in the subject of broiler welfare. I was flattered that someone thought I could cope with a PhD but I wanted to be with my partner, and not have to commute to see him for another three years. Besides, he'd been completing his PhD and I'd learnt all about the visitations from the 'elephant of despair' as he was writing it up in the final year. No. Goodbye academia.

Sort of.

As fate would have it, I ended up in academia. Firstly teaching (mostly) HE in FE, and then working solely in HE. As it happens, I also missed the challenge and excitement of learning new things, so have dipped in and out of formal education ever since. I suppose, if I were truthful, I've always had a little hope that some day I could go and complete that doctorate I gave up the opportunity for.

And so... panning forward to 2015. I um and ah about applying for a professional doctorate at my current workplace. So many things act as potential obstacles: a small child still at nursery; my own vacillation; concern over whether my husband would think it a silly thing to do - a 'vanity' qualification. So many times I talked myself out of applying. maybe next year when small child is at school. But.. new logistical problems would undoubtedly rear their heads. In the end, I enquired, and there was still time to apply. I have applied and will shortly be having an interview.

And now... mild panic. I don't feel I have a strong enough background in education. It may have been my 'career' (using the term loosely) for the past 13 or so years, but I've always felt an interloper. Never more so than in my current job, where most consider our roles as being on the fringes of academia, and where each  of us has 'fallen' into the job. There's no set way to become a study skills adviser, and so perhaps I feel my lack of confidence in the role stems from this. Others know more, can quote learning theory off pat, can discuss philosophical theories. I feel like an outsider. The EdD is, at some level, an attempt to show myself (and others) that I do have the knowledge to support others, that I can support students effectively. A little bit of it is that I would have the letters Dr in front of my name - that means quite a lot in students' eyes, and academic colleagues' eyes. Perhaps it shouldn't, but it does.

What I do have is an idea to develop. I know what I want to do. I want to investigate student transition from vocational FE to HE. What are the challenges? How can we overcome them, or help students and colleagues at FE colleges too overcome them? Is there a problem there at all? Anecdotally there seems to be. But in reality? I don't know. Is it a worthwhile topic? Again, I don't know. The research project is years away. There are many challenges to come before then.

Am I up to them? I don't know. It's a new day it's a new dawn, it's a new life, and I feel excited.