SOCI2000: 3.0 Qualitative Data Collection

Lecture Slides

Slides for Week 3 Lecture

Acknowledgement of Country

This class is taught on the lands of the Wattamattageal clan of the Darug nation.

We acknowledge the Wattamattageal clan as the traditional custodians of the land on which Macquarie University stands.

We pay our respects to Elders past, present, and emerging.

This land was never ceded. It always was and always will be Aboriginal land.

What is an acknowledgement of country?
Why is an acknowledgement of country important?


Concepts

Methods

Field work

In depth interviews

Focus groups

Open ended survey questions



Analytical techniques (not in textbook)

Ethnographic sandwich

Thematic analysis

Saturation

Triangulation



Field work

Ethnography

Participant-observation

Naturalism

Field site

Strategies for success

Gatekeeper

Informant

Normalisation

Building rapport

Continuous negotiation

Going native

Ernest novice

Appearance of interest

Attitude of strangeness

Leaving the field

Ethical + safety issues



Focus groups

Good for attitudes and experiences

Bad for detailed individual cases

Empowers marginalised groups

Allows debate/conflict



In depth interviews

Avoid leading questions

Avoid yes/no questions – give options (“Did x happen first or did y happen first?”)

You should be flexible, not them

Follow their train of thought

Linear stories are easier for participants – start at the beginning

Expect digressions

Double check timelines

Useful questions:

Can you give me an example?

How do you know that?

What exactly did he/she say?

Double check timelines

Be careful of assumptions



Sampling

Purposive

Events: Routine, special, unanticipated



Types of questions:

Descriptive

Structural

Contrast



Note taking

Types of field notes

Jotted notes

Analytic notes/memos

Direct observation notes

Tips for note taking

Do immediately

Write in complete sentences

Record exact quotes

Record chronologically

Tape recordings are no substitute (they break, are out of range, and you rarely have time to listen to them)

Be descriptive, not evaluative (or clearly separate the two).

Be as concrete as possible

Spend approximately same amount of time writing notes as you spend in the field


Introduction

1. Methods of Qualitative Data Collection

Field work

Field work is research conducted in a natural environment, rather than in a laboratory or other artificial setting.

For example, you might go and live with a community for six months, or two years. You would spend your time participating in the community, and also observing and interviewing people. Each evening you would likely write huge amounts of notes (field notes).

The archetypical example of field work in the social sciences is an anthropologist living with an indigenous community some where in the world, and documenting their culture, language, and way of life.

However, field work can be done in any community. For example, a family home in Macquarie Park, or an office in the CBD of Sydney.

In depth interviews

In depth interviews involve sitting down one to one with a respondent, generally for an hour or more. Sometimes over several days. A detailed interview is conducted, generally from a set of prepared questions. However, there is considerable flexibility in the structure of the interview, and the questions are more a checklist for the interviewer to make sure they covered important topics. The interview has the feeling of a conversation, rather than an interogation or questioning.

Focus groups

Focus groups assemble a group of people to discuss a product, issue, or topic as a group. They are often used in marketing, political campaigns, and finalisation of film and TV programs.

Open ended survey questions

One of the simplest and cheapest ways to get qualitative data is simply to include open ended survey questions in your survey. For example, if you were surveying migrant domestic workers about exploitative working conditions, you might ask this pair of quantitative and qualitative questions.

  1. Has your employer ever stopped you from leaving the house?

Yes No Not sure

  1. If you said Yes, or Not sure, please provide details. _______________________________________________________________

2. Analytical techniques (not in textbook)

Ethnographic sandwich

The ethnographic sandwich is a way of thinking about ethnographic research (field research) in a mixed method (i.e. mix of qualitative and quantitative data) study.

The key idea is that ethnography can be like the two peices of bread on a sandwich - coming before or after (or both) the quantitative research (such as a survey).

Each peice of the sandwich has very different purposes.

Before a quantitative study

Ethnograhic research before a quantiative study tends to be exploratory and designed to orient the researcher. The main research a quantitative researcher should do qualitative research before their main survey is because “they don’t want to look like an idiot.” It is remarkable how many people do research on a community they have not even talked to. This is not only foolish, but also unethical.

Ethnographic research before a quantitative study can also:

  • help you develop contacts,
  • help you become familiar with the community,
  • help you develop theories and hypotheses to test, and
  • help you to understand the language and norms of the community.

For example, in ‘Gang Leader for Day’, Sudhir Venkatesh, starts by doing a survey of low income black communities, where the survey contains the question “How does it feel to be poor and black? Very good, good, neutral, bad, very bad.” Clearly this is a very insensitive question that could have been avoided through better knowledge of the community.

After a quantitative study

Ethnographic research after a quantitative study helps you to better understand the social mechanisms and the social reality of any statistical findings you make. When we do a quantitative study, we generally end up with some type of statistical finding - a correlation of some kind. But this doesn’t necessarily tell us what this means in real life, or what underlying mechanism gave rise to this.

For example, in one quantitative paper I wrote on migrant workers, we presented data that showed that 66% of migrant workers with an injury or salary claim had been threatened with deportation (in Singapore). This compared to just 10% of workers without problems. But why were they being threatened? We didn’t know this. So we designed a second study where we conducted interviews of workers and asked exactly why they were threatened. This was a qualitative study (since we didn’t have a clear hypothesis we could simply measure with a closed ended question). In the end we found that there were seven themes which kept recuring in our interviews, themes that were reasons workers believed their employers were threating them were to:

  1. enforce workplace discipline;
  2. thwart complaints;
  3. force early return from medical leave;
  4. stop workers from seeking independent medical treatment;
  5. reduce medical treatment and medical leave costs;
  6. punish perceived ‘fake’ injuries; and
  7. remove ‘damaged’ injured workers.

So in summary:

  • ethnographic research before a quantitative study is exploratory, orients the researcher, and helps with development of theories and hypotheses;
  • ethnographic research after a quantitative studyallows you to understand more detailed mechanisms between correlated variables.

Thematic analysis

Most analysis of qualitative data focuses on identifying key themes that are repeated across the data (such as interview transcripts). If themes recur across interviews, then generally the researcher will draw conclusions and inferences from these.

For example, we can imagine a hypothetical study involving interviewing parents from a marginalized community. Researchers asked parents what were the major problems which were preventing their students from getting a quality education. Themes identified included:

  1. lack of food, as some students went to school without breakfast or packed lunch;
  2. attendance issues, as some students did not go to school regularly;
  3. falling behind and not being able to catch up, as once students were behind the learning stage of the majority of students they tended to not be able to follow class lessons;
  4. constraints on parents, such as shift work, long working hours, lack of English or formal education, which prevented parents of disadvantaged youth from helping their children with their studies and learning;
  5. **overcrowded home environment8*, which made it difficult for students to complete homework.

Saturation

When qualitative researchers from a positivist methodological orientation (i.e. those who see social science as simply applying the methods of the natural sciences to the social world) analyze data they generally aim to reach a point of ‘saturation’.

Saturation means that in the later interviews or data collection, no new themes emerge. The implication of saturation is that there is no point in further data collection.

For example, in the hypothetical case above (education in a disadvantaged community), the researchers might argue that saturation had been reached after 25 interviews because in the last 3 interviews, the interviewees only raised issues which had already been identified in the previous 22 interviews.

Triangulation

Triangulation is an analytical method used in many different settings, including qualitative research.

It involves using multiple methods, or multiple data sources, to test the validity of an idea, concept, theory, or claim.

The concept of triangulation comes from navigation and mapping, where one often determines the position of an object by measuring it from at least two different positions.

For example, in the ‘Hairifying Truth’ study in Week 1, the students argued that sex with men tended to drive pubic hair grooming in women by pointing to five separate findings:

  • sex with men was statistically significantly correlated with all types of gromming except bikini waxing;
  • sex with women was not significantly correlate with grooming;
  • qualitative interviews with lesbians indicated that there were no strong preferences or pressures to groom or not;
  • qualitative interviews with gay men indicated that there were strong normative pressures to groom;
  • qualitative data from hetrosexual men showed strong preferences against pubic hair in women.

In this case, the hypothesis appears to have strong support because it is supported by multiple independent data sources.

3. Field work

Ethnography

Ethnography is a detailed (sometimes called scientific) description of peoples and their cultures. It generally involves describing the culture from the view of the insiders in that culture.

An ethnography of a community is generally the result or end product of qualitative field work.

Participant-observation

Participant observation involves simply spending significant time immersed in a community as a participant and observer. It is a type of field work.

Naturalism

Naturalism is the philosophical belief that social scientists do the best research when they study societies in a natural setting, rather than an artificially created setting, such as a survey or a laboratory.

For example, when some qualitative researchers express a belief that you can only truly get to know a community by spending significant time emersed in it, they are expressing a commitment to (or belief in) naturalism.

Field site

Field site a place in which field research is done. Field sites are normally places where there is regular interaction between people who share a culture.

Examples of field sites include:

  • a soup kitchen where injured migrant workers are given breakfast and dinner
  • a Papua New Guinea village
  • a block of rental flats
  • a coffee shop
  • a boarding house for children with behavioural disorders or adjustment issues.

Strategies for success

Gatekeeper

Gatekeeper is a person who controls access to a community or a field site. This control can be formal or informal.

For example, a bartender may give you permission to stay and study a bar for several months, or they might make your life very difficult if they don’t want you conducting the study.

Informant

An informant is a person who a researcher obtains information about the culture of the field site. Normally they are someone with whom the researcher develops a strong relationship, and who provides the researcher with significantly detailed information about the field site.

Normalisation

Normalization is process or series of actions by a researcher, in a field site, where they help to make social science research non-threatening, unexceptional, and normal.

For example, a researcher might tell dirty jokes, or smoke, or drink, or admit to their own flaws to make themselves seem more ordinary and normal to participants.

Building rapport

Building rapport is the process of build a connection with participants. It means having a positive relationship.

Techniques for building rapport include:

  • doing small favours, such as buying someone coffee;
  • maintaining appropriate eye contact;
  • referring to shared interests or experiences;
  • self-disclosure.

Going native

Going native is when a researcher abandons their professional role - as a more or less objective researcher, and one who is committed to social scientific goals - and adopts the culture, norms, and commitments of those in the field site.

Generally a BAD strategy. Seen as unprofessional.

Ernest novice

A method for building rapport as a researcher is to adopt the persona of an ‘ernest novice’. This means that you treat the participants as experts, yourself as a novice, and show an ernest enthusiasm to learn from them.

This is in many ways the opposite of how many privileged people behave - where they treat the world as a place made up of people who have got a tonne to learn from their advice and wisdom.

However, most people don’t want your advice, and it turns out being the ‘wise guy’ is a way to alienate most people. The ernest novice is a good alternative persona that doesn’t have these problems.

Appearance of interest

Appearance of interest is a way of approaching interactions in a field setting, where the research gives the impression they are interested and engaged, even when they sometimes might be bored or uninterested.

Attitude of strangeness

Attitude of strangeness is technique in field research, where the researcher tries to “see” the field site - the events, the people, the culture - with the views of someone who is seeing it for the first time, or from the perspective of an outsider. It is generally easier to adopt the attitude of strangeness when one genuinely is an outsider to a field site. The attitude of strangeness is aimed at helping the researcher make explicit the various assumed social meanings, unspoken norms, and cultural practices which the members of the field site take for granted.

Leaving the field

If you spend a lot of time with people in a community, you get attach to them, and they get attached to you.

Given the privileged position of most researchers - they are studying at university, often from developed countries or wealther backgrounds - there can be inequality in the relationship.

There can be the tendency for researchers to simply disappear at the end of their research, leaving individuals and the community they studied feeling abandoned and used. This is one of the ways researchers exploit vulnerable communities.

The main thing to recommend is to be aware of this, and to consciously plan your exit, and take seriously any expectations or commitments you have made to the community - either explicitly or implicitly.

Ethical + safety issues

One of the most important ethical issues when doing field research is maintaining the privacy of participants. Often you are documenting and creating evidence for people’s actions and lives that would not other wise exist.

Your feild notes could damange relationships, or even get participants into legal trouble.

One simple way to avoid this problem is to engage in anonymisation of field notes from the very beginning. Simply use pseudonyms the whole time. This saves you from having to go back and anonymise later.

4. Focus groups

Focus groups can be an incredibly powerful method of research.

Good for attitudes and experiences

In particular, focus groups allow you to get a very wide range of opinions on a single topic is a short period of time.

Allows debate/conflict

You also get to see how different attitudes and beliefs interact in a conversation. So if you had a focus group with people of different backgrounds and politics, you could see how they debated and discussed a particular controversial issue.

Empowers marginalised groups

Focus groups can provide a space to empower marginalised groups. I have worked with researchers who have used focus groups as spaces to empower domestic workers, and sex workers. The experience of hearing other members of your marginalised community can be empowering. It can help participants see their problems as systemic, not individual. They can also see that they are not alone. They can even talk about solutions together.

Bad for detailed individual cases

One of the main problems with focus groups is that you have limited time, so they are bad for getting detailed information about individual cases. They are also not very useful when you want to cover a wide range of topics, as you quickly run out of time.

5. In depth interviews

Avoid leading questions

Leading questions assume a particular answer or assumption. This biases responses towards particular answers.

For example, “Is your employer a bad man?” is a biased question. Better would be “Would you say our employer is a bad man or good man or what?”

Avoid yes/no questions – give options (“Did x happen first or did y happen first?”)

Particularly with marginalised communities, it is important to avoid simple yes no questions. The problem is that the default answer is always “yes”. People try to be agreeable.

One of the best ways I have found to avoid this is to instead give a series of options.

I had a migrant workers who I interviewed give a very detailed case history, and everytime I asked if something happened he would say “yes”. In the end I found the only way I could get a straight answer out of him was to say “Did you go to the public hospital or private hospital first?”

You should be flexible, not them. Follow their train of thought

You should be the one who is flexible, not them. You will have your script, your list of questions. The temptation will be to run through them like a list. This is bad interview technique.

What you should be doing is opening up the conversation with the first question or two, and then try to let the participant talk freely, and slowly ask questions to make sure you get the details you want for your study.

The main way to do this well is to know your questions very well. This way you can make sure you can raise questions in whatever order best fits the participants train of thought.

Linear stories are easier for participants – start at the beginning

Remember that linear stories are easier for people to tell. They have a natural flow. And even when people get distracted or wander off, the timeline provides a place you can come back to an orient the conversation.

A famous study by Professor Martina Morris used qualitative interviews to obtain the entire sexual histories of hundreds of people in Thailand, Uganda, and the United States. And apparently the way she got very honest answers was by starting by asking about their first romantic or sexual experience, and then simply going through them in sequence.

Expect digressions

Simply expect digressions. People will get off topic. Some people will find it impossible to stay on topic. You need to use your judgement and humanity to decide how much and how often to steer the conversation back on topic.

Double check timelines

People’s memories are terrible. Double check timelines (if they really matter). I have heard people get major events completely out of sequences - incorrect years and orders of events.

Be careful of assumptions

This is a hard one to monitor, but just make sure you check some of your assumptions. Many migrant workers, for example, will tell you they have good boss, and give big smile and be genuinely happy. But then if you ask about some common problems, in simple ways, you find problems. “What time do you start work? Finish? How many days per week? Ever work overnight? Does boss deduct saving money?” I had a worker I interviewed who insisted he was happy at work and treated well. He had not had a day off in months, and regularly worked in a construction site for 24 hours straight so as to help his boss meet deadlines. His employer also kept a proportion of his pay as savings, which the worker would only receive when he left Singapore. This is actually an indicitator of forced labor (because it is a penalty held over the worker that makes them unable to say no to the employer). But his first answer was that he was happy, he had good boss, etc.

Interview transcript.

This is a transcript of an audio (or video) recording. Often these can take considerable time to make (4-5 hours per hour of audio tape). However, in recent years cheap online, automated methods can make this quite a lot easier (e.g. otter.ai trint.com and even youtube.com subtitles (though note privacy issues). Often automated transcripts require considerable further cleaning up to be usable.

Useful questions:

These are the simple questions I find most useful for in depth interviews.

Can you give me an example?

Often people require you to reask this question multiple times until they finally give you concrete example.

Examples are good because they help you understand exactly what they are talking about. They also make very vivid quotes when writing up research.

How do you know that?

Many times people express opinions or beliefs. If you ask “How do you know that?” it is a nice way of saying “Give me the evidence or I don’t believe you?”

It encourages them to keep talking, and to expand on their opinions, with evidence.

What exactly did he/she say?

Often people will talk in generalities. If you ask for what was specifically said, then there is less ambiguity about meaning. Less room for misunderstanding. More concrete evidence for you to cite in your report/article.

6. Sampling

Purposive

When we conduct qualitative research, we don’t just interview anyone. We try - like all reasearchers - to get an unbiased sample. We want to talk about a group of people, and use this to draw conclusions about a much large group or community they represent.

In qualitative research, we generally can’t do true random sampling.

Instead we tend to use what is called ‘purposive’ sampling, which simply means that you use your judgement to try to get a sample that contains signficant levels of variation and difference.

By choosing a sample that has significant variation and difference, it becomes harder for research to be accused of bias in sampling.

Events: Routine, special, unanticipated

One way qualitative researchers talk about appropriate sampling is to talk about sampling across different types of events.

Routine events are those which happen regularly. They are the normal parts of life. Sunday church. Doing the laundry. A dinner party with friends.

Special events are those which are more rare. A wedding. A funeral. An economic crisis.

Unanticipated events are those which we did not expect when designing our study. These can often be quite insightful, as they break our preconsceived ideas about a community or the world.

7. Types of questions:

Descriptive

Descriptive questions: these are questions that are very open ended and general. They might ask the participation to describe their typical day, or a particular event, example, or experience.

For example: “How would you describe the type of fashion (clothing) that you wear to university? How about your friends? And other people at this university?”

Structural

Structural questions: these are questions which use conceptual categories - normally from the participant or the field site - and which asks interviewees to provide more detail.

For example “Some people say that you can classify students based on their approach to fashion. What do you think are the major types of students, based on their fashion?” or “You say there are a group of female students who dress in ‘business’ attire. Can you tell me more about the clothes they wear? And who are these women? Why do they wear busines clothes?”

Contrast

Contrast questions: These are questions that ask the interviewee to compare and contrast cultural/conceptual categories. “What are the main differences between smart casual and business fashion at this university?”

8. Note taking

Types of field notes

Jotted notes

Jotted notes are brief notes written while in a focus group or out in the field. Because the researcher has little time, these notes are short, and their main purpose is to prompt the researchers memory when they are writing complete detailed fieldnotes later.

Analytic notes/memos

Analytic notes/memos: These are notes that include the researchers analysis and reflection on the field site, event, or interview. It might be relating the direction observations to an academic theory, it could be a hunch the researcher has about how various observed events fit together, or it could be a note about the methods the researcher used.

Direct observation notes

Direct observation notes: These are the detailed field notes written immediate after being in the field - generally on the same day - and provide a record of what took place. Jotted notes are used to prompt the researcher when writing these notes. These should take the form of standard prose - full sentences, use of paragraphs, and the logical and systematic organisation of ideas. They should be what is directly observed.

Tips for note taking

Do immediately

Write up your field notes immediately after an interview. And if you can’t really try to do it on the same day.

Ideally you would simply go for a coffee and write your fieldnotes immediately after an interview.

Every minute and hour that pass, your memory is degrading.

Also writing fieldnotes is a lot of work. And we tend to procrastinate on this. Just get it done. You will feel so much better. You won’t have this hanging over you.

Write in complete sentences

Field notes should be written in complete sentences, and in paragraphs.

Record exact quotes

Try to record exact quotes. They are great for writing up your article/report. They are powerful and memorable for readers.

Record chronologically

It is easiest if you just write up your field notes chronologically, or in the order in which they events were told to you in the interview.

Tape recordings are no substitute (they break, are out of range, and you rarely have time to listen to them)

Remember that tape recordings are no substitute for feild notes.

Tape recorders break. They are out of range. You forget to turn them on.

And they take a long long time (or money) to transcribe and listen to.

If you do feild notes, you will integrate your understanding, solidify it in your memory, and start to have text which you can use for your final write up.

Be descriptive, not evaluative (or clearly separate the two).

When you write your notes be descriptive not evaluative or if you are going to be evaluative separate the two because the descriptive counts as evidence. The evaluative is analysis, but not really evidence.

Be as concrete as possible

Concreteness is the essence of good quality writing and good quality research.

The more concrete you are the better examples you will have for your write ups.

Spend approximately same amount of time writing notes as you spend in the field

This is a very useful estimate of how long you should spend writing your field notes after being in the field.

Summary and Conclusion

The ethnographic sandwich is a conception of how ethnographic work can be important as a ‘bracket’ (come before and after) around quantitative research. Before quantitative research, qualitative research can be used to explore a topic and orient the researcher. After a quantitative study, it can be used to understand underlying social mechanisms, as quantitative studies often show no more than a correlation. So qualitative research can be both exploratory and explanatory.

We generally analyze qualitative data with thematic analysis, which basically involved trying to identify common themes that answer some research question.

When we do qualitative research we know we can stop data collection and analysis when we have reached saturation. Saturation is when new interviews (or focus groups or other sources of data) are not giving us any new themes

There are four main methods of qualitative data collection we studied today:

  • Field work: long term engagement with a real world place and community. Lots of skills you can acquire to be better at this.
  • Focus groups: small group facilitated discussions around a topic. Good for collecting opinions or experiences around a narrow topic.
  • Open ended survey questions: a simple, cheap way to get qualitative data when doing a survey.
  • In depth interviews: one-on-one detailed interviews. Good for detail and depth of information. Lots of skills you can acquire to be better at this. Most important are to follow their train of thought and ask them for concrete examples.

When we ask questions in interviews, we should try to move from exploratory, open ended questions – descriptive questions – to those that start to identify social categories and cognitive categories the participants use. We do this by moving from descriptive, to structural, and finally contrast questions.

We generally do purposive sampling in qualitative research and we try to make sure it is that we get a multiple different types of events not just the routine but also the special.

We generally take three types of notes: jotted notes (our quickly hand written notes during an interview or observation); direct observation notes (which we write up after the interview or observation, and which are done in full sentences); analytical notes (which are similar to direct observation notes, but differ in that direct observation is what we ACTUALLY OBSERVED; while analytical notes are our INTERPRETATION of events).

There are many useful tips for note taking. The main ones to be aware of are: (1) write up field notes immediately; (2) remember tape recordings are no substitute for field notes; (3) spend about as much time typing up fieldnotes as you do in the field (timetable this into your day!); (4) write your field notes in full sentences.


Last updated on 19 April, 2020 by Dr Nicholas Harrigan (nicholas.harrigan@mq.edu.au)