Design Research Methodologies

Friday, 2 August 2019

Girish Dalvi

Source Grid

Author	Definition	Notes
Athavankar (1990)	Design is…	First Definition
Balram (1995)	Design is…	Inspired by Ulm
	Design is…	Agrees with Athavankar, adds…
	Design is…	Disagrees with …

Referencing Styles

• LNCS

• APA

Manual Styles

• Chicago Manual of Style

• Oxford Manual of Style

Reporting the work of others

Paraphrasing (Writing another person’s idea in your own words)

Summary ()

Quotations (Avoid too many)

Synthesis (Taking multiple authors, paraphrase them, link their work in like one or two lines. If you have 10-15 authors but quoting each will make the text too long)

Introductory phrases

Reporting verbs

Integral 

According to Dalvi (2010) …

Non-integral

… Dalvi

Indicating agreement

Joshi (2003) points out that a far more effective approach is…

Foreshadowing for disagreement

Joshi (2003) claims that a far more effective approach is…

Direct quotation must be in quotes, author must be at beginning or end

“Hit me baby one more time” (Spears, 1998)

Long quotations (Mostly used for a narrative)

According to Lakhwani,

“When I was a child, I was … and that’s the story of how I got lost in the mall” (Lakhwani, 2019)

Patching

Only slightly changing the wording - using a thesaurus and swapping words. Even if you provide citation/reference, patching is considered a form of plagiarism.

Side-stepping (To show your awareness of the complexity of the issue) 

• Acknowledge a problem exists but mention that it is not your focus.

• Give references to the work done by other people in the side-stepped area for people to pivot towards.

“These issues are not the focus of the study, readers interested in postcolonial studies can refer, to mention a few, the works of (Alvares 1991), (Bhabha 1994) and the writings of Subaltern studies group”

Organising your review

• Topical

• Chronological

• Problem-cause-solution

• General-to-specific / specific-to-general

• Known-to-unknown

• Comparison-and-contrast

Suggested method for organising your review (general-to-specific)

• Broad topics

• Subtopics

• Studies like yours

Hedging (#humble)

Hypothesis =/= Opinion =/= Facts

Distinguish between absolutes (100% certain) and probabilities (<100% certain)

Provide traceable evidence and justification for any claims you make or any opinions you have formed as a result of your research.

Post hedge

With the evidence we have, in the context of …

Weasel words: Seems like, probably, perhaps, it appears
These sort of words are frowned upon in other types of writing but in academic writing you’re protecting yourself against making outlandish claims. You also sound humbler.

Assignment Number 1a

Due by next GD course - 21 August?

Prepare a plan for the Literature Review of your Project/Research

What will I cover?

What will I not cover?

Strategy for finding resources?

What will my conclusion be?

Assignment Number 1b

Due at end of semester

Execute the plan for Literature Review

Supplementary Reading

1. Hypotheses, Laws and Theories: A user’s guide
   Stephen Van Evera

2. The literature review: it’s role within research
   Booth, Sutton et al

3. How to Read a Paper
   S. Keshav

Wednesday, 7 August 2019

“What is research? Nobody knows.” - AJ, 2019

Creation of new knowledge

Reliable, valid and reproducible

Contextualising and referencing

• What was known, what is new, how is it different?

• Incremental - builds on previous work.

• ‘Good’ ideas are refined + advanced, ‘bad’ ideas are discarded

Articulation

• Writing papers

Friday, 9 August 2019

Correlational Research

Correlational is not necessary coincidental.

Confound

this is the third variable which muddles your data.

Causality -

Experimental Research

A ‘controlled’ experiment in a ‘lab’ setting

2 variables - manipulated variable and response variable.

AKA Dependent variable and independent variable.

Experimental research must have at least 1 IV.
Number of “effects” increases rapidly with number of IVs, try to keep them limited in an experiment 2 to 3? Eg. Drug trials 2 or more start interacting with each other also.

Independent Variable (IV)

Independent of user behaviour. It is manipulated in an experiment to elicit a change.

Eg. change in human response while using a design

“Look, I’m creating a theory.” - AJ 2019

• Must have at least 2 levels/values/settings

• Name both the factor (IV) and levels (test conditions)

• Human attributes (age, gender, height, etc.) could be IV but cannot be “manipulated”

• There may be confounding variables - a variable that affects IV & DV both, causing spurious association. They are often hidden, you need to guess at them.

• Usually correlational studies

Dependent Variable

• Use anything that is measurable and reproducible

• You should be able to collect the data

Control Variable

Circumstance (not under investigation) that is kept constant while testing effect of IV.

Random Variable

Circumstance (not under investigation) that is allowed to vary randomly while testing effect of IV.

Confounding Variables

Circumstance apart from DVs under investigation that has an effect on IV.

Operational values

When we’re doing research we need to operationalize values - eg. trying to study technology adoption we might use number of devices owned, time taken to learn a technology (whatsapp, net banking)

Research Questions

Is there a higher proportion of women in the profession of design as opposed to engineering?

Hypothesis

There is no difference in the proportion of women in the professions of engineering and design.

Research Design

Tension between external v/s internal validity

If you control some of the random variables, variablity comes down.

Some degree of randomness allows for better external validity.

Between Subjects

Each participant is tested on one condition only.

More participants, but shorter study each.

Variation in skills of participants can affect the study, needs balancing variables across test conditions.

Within Subjects

Each participant is tested on each condition. AKA repeated measures.

Fewer participants, but longer study each.

Less variation because of skills of participants, balancing not needed.

Order effects (eg. learning, practice, fatigue) are possible - counterbalancing reqd.

Asymmetric learning effects

Eg. Keyboard A teaches people how to use keyboard B and vice versa.

Use a training session to fix asymmetric learning effects.

Order effects - Counterbalancing if possible, randomize if not. 

• Learning

• Practice

• Fatigue

Balanced latin square when all P&C not possible

Wednesday, 14 August 2019

“Have fun with the data” - AJ 2019

Tapping Experiment

Design of experiment

• PC v/s Android - between subjects

• With and without feedback - within subjects, counter balanced

• Mixed design (since it used multiple things)

The sample population is all design students who are heavy users of technology.

Homework for today

Friday, 16 August 2019

Null hypothesis

Null Hypothesis

The hypothesis that there is no significant difference between specified populations, any observed difference being due to sampling or experimental error.

What would be the purpose of doing the phonebook experiment?

• Gender bias, argue for more diverse hiring

Dal factory (tur dal)

Dicot seed with outer shell, some oil. “Phatka” machine - not very sophisticated.

Sortex machine

Uses optical camera and sorts 16 tons of dal a day.

Good dal 70-80%

Broken pieces to make powder

Unbroken put back in the machine

Wednesday, 21 August 2019

Variables according to levels of measurement

Categorical variable

Categories/names of things

Nominal variable

2 or more categories with no intrinsic order

Binomial/Dichotomous variable

There are some special tests we can use, sometimes coded as 0/1

• Do you own smartphone? (yes/no)

• Success (successful/failed)

Ordinal variable

• How do you feel about govt’s performance? (approve, on the fence, disapprove)

• Ranking is special case of Ordinal

• Letter grades in IITB - they start as numerical, but you want consistency in measurement across subjects, and teaching methods and evaluation methods vary.

Continuous variable

Semantic differential scale

Use bipolar adjectives to measure the psychological meaning of an attitude object.

Visual continuous scale

Interval scale

• Measured along continuum

• Interval between 30°C and 40°C is same as 40°C and 50°C.

• 0 is arbitrary value at which water freezes, not lack of temperature

• Ratio is not meaningful - 10°C is not double of 5°C.

Ratio scale

• 10m is twice of 5m

• 0 is absolute lack of something

• All tests for interval can be used for ratio but not other way around.

Normal distribution

Bell curve

P-value

Chance that the result is by chance, given that null hypothesis is true.

“There are only 2 things in the world - samples and populations.” - AJ 2019

Friday, 23 August 2019

Girish Dalvi

Research paper

• Thesis/claim/data

• Method of collecting data

• Method of analysis

• Discussion
  What are reasons for getting results we are getting? What other areas are there for exploration?

Lit Review content

• Purpose of Lit rev

1. How will you use the material you describe? Chekhov’s gun.

2. Prove demand for research/product

3. Prove impact for research/product

4. Document/survey and identify gaps

5. Identify similiar/existing research/product

6. Analyse them, use theory/good features

7. Avoid disadvantages of existing products

8. Other uses

• Organisation of Lit rev

• Sources
  dssd

• Desired conclusion of Lit rev

What is a belief system?

“Research is a belief system.” - Girish Dalvi 2019

• Nature is orderly and regular system

• We can know/understand nature (human exceptionalism)

• Knowledge is better than ignorance

• All natural phenomena have natural causes

• Nothing is self-evident, but it must be verified or falsified

• Knowledge is derived from observation of nature

Use of research

Establish facts

To test (and establish) ‘causal’ explanations for established facts

Epistemology

Rationalist

Reasoning creates knowledge

Empiricism

Obtained through senses

Types of explanation

Idiographic

Seeks to explain particular situation
→ Limited to single case

Nomothetic

Seeks to explain a class of situations

→ Generalizability

Research strategies

Inductive

Linear, bottom up process

Argues from particular to general

Deductive

Also linear, but top down

Argues from general to particular

2 less common but more complex non linear processes
Spiral processes with many iterations

Abductive (Schütz)

Develop concepts and theories from actors’ everyday life and understanding; inference to the best explanation

//Humans are linguist abductors - everyone is guessing what people are trying to say.

Retroductive

Use reason and imagination to create an explanatory model and then test in real world.

Homework

Assignment 2a

Read ‘Strategies for answering research questions’ - Norman Blaikie.
Explain each strategy with 5 examples.

Assignment 2b

Examine and articulate through Writing your research/project beliefs and the evidence for it.

Girish Dalvi sir’s next class is on 18 September

Wednesday, 28 August 2019

AJ

Revision class, notes added in previous sections

Wednesday, 4 September 2019 

Dr. Pramod Khambete

Introduction to Research

Surveys and Questionnaires (focus on questionnaires.)

“Get all the information you can, we’ll think of a use for it later.” - common practice.

Pros

Popular in quant, can be used for qual also.

Enables coverage of large spread of population.

Cons

Results can be misleading unless done well.

Survey is the measurement process. Questionnaire is the highty structured instrument comprising of a set of ‘questions’.

How to set questions

Questions may be in the form of statements, W/H question not necessary.

• According to what we know

• Intangible/invisible things we want to know converted into ‘questions’

• Well organised - order, grouping and mix of structured and unstructured questions.

• Articulated in a manner to reduce biases and errors

• Answers must be analysable

• Appropriate communication approach

• Interview, self-administered

• Paper based, telephonic, web

Behaviour and Attitude measurement

Behaviour

What people do

Attitude

A learned, stable predisposition to respond to oneself, others, objects, events and issues in a consistently favourable or unfavourable way.

Types of Attitude

Cognitive

Memories, evaluations, beliefs

Affective

Feelings, emotions, intuitions, values

Behaviourally bases

Expectations, behavioural intentions

Attitude influences behaviour and vice-versa as well.

Developing a survey questionnaire

1 Research Question

What are things you want to find? What to conclude?

2 Investigative Questions

Specific questions that adequately cover research questions?

3 Measurement questions

Specific, analysable questions participants must answer.

4 Questionnaire development

Exact form and content of questionnaire.

Known x Unknown

Known Known - You know you know this.

Known Unknown - You know you don’t know this.

Unknown Known - You don’t know you know this - you have some partial information.

Unknown Unknown - You don’t know anything about this.

Friday, 6 September 2019 

Dr. Pramod Khambete

Circle of data collection activities

1. Locating site/individual

2. Gaining access/making rapport

3. Purposeful sampling

4. Collecting data

5. Recording information

6. Resolving field issues

7. Storing data

… and loop.

Creswell (2007)

Be flexible

Expect deviations from the plan, things can and will go wrong. Adapt to the situation.

Random sampling is not always the best sampling method.

Sampling is usually not random, sample composition may emerge during study.

Random sample is not 100% representative sample.

Recognise role of gatekeeper

Voluntary participation is a must, but role of gatekeeper can introduce filters in population, or influence people to talk to you or be more guarded.

Recording, collecting, storing data

Nuances v/s. Hard data - need to strike a balance

Use multiple methods/tools (eg. notes, audio rec, video rec)

Generally, you won’t be able to interview the same person again.

Ethics

Don’t record without consent

Participants must not face any risks, even inadvertently

Interviewer is in position of control, there is power asymmetry. Be conscious of this.

Interview

Purposeful conversation - not a Q&A. Needs to be flexible.

Your aim is to get information, other person has no obligation to you and is doing you a favor.

Generates verbal data, capturing nuances is important, capture other kinds of data if possible.

Empathise

“यह सब bullet point तो ठीक है, field में करके देखो” - Khambete, 2019

Types of interview

Open ended/unstructured

• Questions are tweaked to match unfolding situation

• Extensive use of probes - cultural probes can be used

Semi-structured

Structured

Preparatory stage of interview

• Give basic info about research

• Tell them role of interview in research

• Tell them their role

Wednesday, 18 September 2019 

Girish Dalvi

What do I want to know?

Try and be very specific

• What is a better way to find fonts for amateur font enthusiasts?

• What does better mean?

• How to measure improvement?

• Is one font better than another?

• Self reporting?

Why do I want to know?

What impact will it have?

• Furthering of the epistemological imagination.

What do I already know?

How sure am I? What is the evidence?

Anecdotal evidence is not enough. Hasty generalization. Evidence should be sustantial.

How will I know I want to know?

What is the process? Is this the best way to know what I want to know?

How will I test the new knowledge?

• Validation

• Comparing with previous literature

• Replication in time, space

• RCT

• Benchmark studies

Friday, 20 September 2019 

Girish Dalvi

Claims and Arguments

“The first part of this talk is a rant. It’s a personal rant, aimed mostly at the Mdes folks, but don’t feel too bad.”

Design exceptionalism

With no prior knowledge about the domain, there is this expectation that you will be able to do better than people in the domain. Eg. go to school, think you know better than someone who has been teaching for 30 years.

This leads to the designer as a tourist - expecting others to do things for you, your experience is at the center.

Othering

From postmodernism - never become them, but profit from them.

Designers claim more than what they do.

If you claim you will make a blackboard, as a product designer, that’s okay. If you claim you will radically change education in India, that’s a problem.

Informed designers

• Critically aware of own limitations
• Use evidence to evaluate, make judgements
• Are interested in opposing ideas
• Understanding, learning as a goal
• Avoid emotionalism, think before acting
• Open mind, open to criticism

Some words

• Fact
• Claim
• Opinion
• Assumptiom
• Argument
• Inference
• Plausible
• Validity
• Thesis
• Evidence
• Rationale
• Warrant

Every design is a claim

A claim is a statement that something is true.

• people might not believe it
• typically designers don’t provide evidence/proof
• evaluated as: accept, reject, suspend judgement (not enough proof)

“Designers are, जैसे Fight Club में कहते हैं, middle children of nowhere. We are not marketing, scientists, but we’d like to be both.” - Girish 2019

A claim needs Arguments

An arguments os the reason(s) why you support a claim.

The process of exmplaining these reasons is argumentation.

“If you accept arguments without evidence, others will ask for it. External will ask for it.” - Girish 2019

Thesis

A thesis is the main idea, opinion or theory.

• contains multiple claims
• includes evidence

Greek modes of persuasion

Logos

• Logical appeal/appeal to reason
• Use data to support claim

Ethos

• Appeal to character/appeal to credibility
• Use people’s character/trustworthiness to support claim

“If Narendra Modi says everyone should start designing for AR/VR…” - Girish 2019

Pathos

• Appeal to emotion/appeal to “values & beliefs”
• Use people’s ability toempathize to support claim

“Education in this country is terrible, think about the children…” - Girish 2019

Toulmin Model of Argumentation

• Informal method of reasoning
• Method for constructing, analyzing arguments
• Tool for diagramming arguments

Types of claims

• Fact: an empirically verifiable phenomena
• Judgement/value: opinions, attitudes, subjective evaluation
• Policy: advocating courses of action that should be undertaken
• Definition/Classification: indicates what criteria are being used to define a term or what category something falls into

Fallacies

Claims and warrants are prone to fallacies.

Loaded question

Question assumes an unjustified/unproved assumptions.

• How can teaching be made effective by using iPads?
• Are you still beating your wife?

Appeal to Popularity

AKA Bandwagon effect. Claim that something is good/true because it is popular.

• A lot of people want to come to IDC to do Mdes, so it is India’s #1 design school.
• A lot of people watch Sallu bhai’s movies, he must be a good actor.

Hasty Generalization

Claim on the basis of inadequate evidence.

• Two out of three VC students I asked don’t want a studio space.
• I benefitted from swimming, you should do it too.

Sweeping Generalization

Absolute statements that involve words such as all, always

• All children are creative
• School education destroys children’s creativity

“Children are, quite sorry to say, quite stupid.” - Girish 2019

“Creativity is measured by diverse ideas. In the study, they found children can only do one or two घिसा पिटा ideas.” - Girish 2019

“I did a study, around 80% of Bengalis love fish.”

Post Hoc Fallacy

Claiming cause-effect relationship simply because one event preceded another. (Co-occurence)

1. Students are denied studio spaces
2. Grades of students are poorer than earlier years
3. Denial of studio spaces caused a drop in grades

Non sequitur Fallacy

Linking two ideas that have no logical connection

• Girish is a farmer, so he must be depressed.
• Roma studied in America, so she must be smart.

False analogy

Two things share some characteristics, therefore they are alike in all respects. 1 VR/AR is being used in entertainment, and it is effective. 2 Education should contain aspects of entertainment…

Wednesday, 25 September 2019 

Girish Dalvi

Making and defending design claims

Chaining claims

Claim 1 becomes the basis for claim 2.

Claim 1 –> Claim 2 –> Claim 3

Formative evidence

Testing

Friday, 27 September 2019 

Girish Dalvi

Introduction to Research Design

Definintion

• Master plan specifying methods, procedures for collecting, analyzing the data.
• Overall plan for answering research questions and gathering evidence for your claims.

Empirical research

Based on quant.

Interpretive research

Eg. Analysis of Ruskin Bond and Shakespeare

Conceptual variables

Often expressed in general, theoretical, subjective or qualitative terms.

Operational/measured variables

Measuring concepts

How can we measure things like:

• Studiousness : (Hours spent studying?)
• Sincerity : (Num times submitted on times?)
• Intelligence : (Intelligence test/makrks/exam score)
• Aggression : (Num violent acts)

Claims

• Step outside emotions and marketing land
• Food/pain/feeling/fonts are subjective
• Enhance experience, effective etc. is vague
• Make claims precise and specific

Characteristics of a good design

• Objectivity - data, analysis should be free from bias
• Reliability - design (of the research) should deliver consistent result
• Validity - should deliver accurate/correct results
• Generalization - results for sample should hold for entire population

Fisher’s principles of experiment design

Principle of Replication

Repeat experiement more than once to increase accuracy

Principle of Randomization

Effect of extraneous variables gets clubbed under general factor of ‘chance’

Principle of Local Control

Try and control only 1 variable

Wednesday, 9 October 2019 

AJ

Writing Research

What is research?

“Creative + systematic work undertaken to increase the stock of knowledge, inclusing knowledge of hamns, culture, society, and the use of this stock to devise new applications.”

What is PhD?

“Keep pushing.” - AJ 2019

Whitepaper

• In theory: report/guide that informs readers concisely about complex issue
• Marketing: form of marketing presentation

Position paper

I feel that/in my opinion, if we tried this, it might lead to that, so we should do this.

Case Study

This is what i made, this is how it works, this is why it’s cool.

Research paper (“paper”)

• Describes learning from a particular research activity.
• I blind peer reviewed (often double- or triple-blind).

“It’s all real.” - AJ 2019