Difference between revisions of "Case Study Guidelines"
From COST Action FP0804: FORSYS
SeanGordon (Talk | contribs) m |
SeanGordon (Talk | contribs) m (→Forys Specific) |
||
| (33 intermediate revisions by one user not shown) | |||
| Line 1: | Line 1: | ||
| − | < | + | [[category:Case Studies]] |
| + | [[category:Meeting Nov 2011]] | ||
| + | <div style="float:right;">__TOC__</div> | ||
==What is a case study?== | ==What is a case study?== | ||
| − | Excerpted from Wikipedia | + | ;Excerpted from Wikipedia [http://en.wikipedia.org/wiki/Case_study Case Study]: |
| − | + | :Rather than using samples and following a rigid protocol (strict set of rules) to examine limited number of variables, case study methods involve an in-depth, longitudinal (over a long period of time) examination of a single instance or event: a case. They provide a systematic way of looking at events, collecting data, analyzing information, and reporting the results. As a result the researcher may gain a sharpened understanding of why the instance happened as it did, and what might become important to look at more extensively in future research. Case studies lend themselves to both generating and testing hypotheses. | |
| − | + | ;From [[Case Study Literature Methods Yin 2003 | Yin (2003)]]: | |
| + | :typically used to investigate "how" or "why" questions about a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident | ||
| + | ;Case studies vs literature syntheses | ||
| + | :There was discussion at our Leuven meeting about whether broadly defined cases (e.g. Fire behavior DSS in southern Europe) are really cases or more literature syntheses. It could be considered a broad case with "embedded" cases [[Case Study Literature Methods Yin 2003 | (Yin 2003)]], if it includes standardized descriptions (cases) for a number of DSS. If it does not contain such sub-cases, then it would be better to call it a synthesis. Either type of study could contribute to our overall purpose of defining guidelines for DSS development and use. | ||
| − | == | + | ==Logistics== |
| − | + | The first questions we should answer are how much time and capacity do we have to compile the case studies? | |
| − | + | ||
| − | + | ||
| − | == | + | ===Capacity for the work=== |
| − | === | + | * We must keep in mind our capacity to conduct case studies because this is a volunteer activity |
| − | * | + | * The case study committee will help coordinate the work but broad involvement from FORSYS participants is needed |
| − | * | + | * STSM's can support the work; we should discuss possibilities in Leuven |
| − | ===Specific=== | + | * We should consider how much to rely on existing materials/cases versus conducting new research |
| + | |||
| + | ===Timeline=== | ||
| + | * 18th November 2011 – FORSYS participants submit STSM opportunities to wiki | ||
| + | * 1st February 2012 – FORSYS participants submit case study (CS) abstracts to CS technical committee (CSTC) | ||
| + | * 1st March – CSTC provides reporting format for CS | ||
| + | * 1st March - CSTC responds to abstracts | ||
| + | * 1st June - CSTC follow up on cases | ||
| + | * 1st September – FORSYS participants submit first draft of CS to CSTC | ||
| + | * 1st October – CSTC responds to drafts | ||
| + | * 1st November – FORSYS participants submit final CS | ||
| + | * 20th December 2012 – CSTC synthesis | ||
| + | * 2013 CS synthesis informs development of FORSYS guidelines | ||
| + | * technically have until 3 months after end of project (4/2013) to complete all project work | ||
| + | |||
| + | ===Organization=== | ||
| + | How should we best organize ourselves to do the case study work ? | ||
| + | * By working groups: emphasis would be on WG topics (eg 3 cases on knowledge management in DSS) | ||
| + | * By mixed WG groups: try to cover all 4 topics for each case (more integrative but probably harder to organize) | ||
| + | * By regions or countries: each region would produce a number of cases | ||
| + | * By ad hoc case study groups based on individual interests | ||
| + | |||
| + | I don't believe we resolved this in Leuven, but the default seems to be | ||
| + | * ad hoc groups that build on existing research | ||
| + | * the working groups will probably play an important role in extracting/reviewing lessons related to their areas | ||
| + | |||
| + | ==Purpose(s) of FORSYS Case Studies== | ||
| + | The most important question is what are the main purposes of our case studies? These purposes will influence the many methodological choices outlined below. | ||
| + | ===General ideas=== | ||
| + | Three general types of case studies: | ||
| + | # Exploratory: explores a relatively unknown issue, generally unstructured, develop hypotheses for further research | ||
| + | # Descriptive: some important features of the case(s) have been identified beforehand and are used to structure the case write-up | ||
| + | # Explanatory: a cause-effect hypothesis is stated, and the case is structured to test this relationship | ||
| + | |||
| + | ===Forsys Specific Purposes=== | ||
| + | * <b>FORSYS general purpose: Produce guidelines for the development and use of forest decision support systems</b> | ||
| + | * Case studies provide lessons learned, which will provide evidence for our overall guidelines | ||
| + | * Exchange knowledge among FORSYS researchers | ||
| + | * In the FORSYS application for the COST Action the CS were said to be selected based on their relevance to the European forest sector and the [http://www.forestplatform.org/en/strategic-research-agenda Strategic Research Agenda] (SRA). In the SRA the Strategic Objective 4 "Meeting the multifunctional demands on forest resources and their sustainable management" is most relevant. It contains the research areas: <i>4-1 Forests for multiple needs, 4-2 Advancing knowledge on forest ecosystems and 4-3 Adapting forestry to climate change</i>. If we are turning to these areas we might enter into DSS that are quite new and not put into practices to any larger extent but it might be important as well. | ||
| + | |||
| + | ===Specific examples=== | ||
| + | ====Exploratory==== | ||
| + | * What are the concrete impacts of DSS in sustainable forest management? | ||
| + | * How can DSS support EU Strategic Research Agenda: | ||
| + | ** 4-1 Forests for multiple needs | ||
| + | ** 4-2 Advancing knowledge on forest ecosystems | ||
| + | ** 4-3 Adapting forestry to climate change. | ||
| + | |||
| + | ====Descriptive==== | ||
| + | * How do DSS contribute to different stages in decision making? | ||
| + | ** The [[Participatory planning case template]] describes the roles of participation and DSS in particular cases using a descriptive structure based on Simon's (1960) stages of decision making: organization-intelligence-design-choice-monitoring | ||
* Does the adoption and use of DSS vary by problem type? | * Does the adoption and use of DSS vary by problem type? | ||
| + | * What are lessons learned by WG themes (architecture, models, knowledge mgt, participation)? | ||
| + | |||
| + | ====Explanatory==== | ||
| + | * The participatory case study group extracted success criteria for participation (e.g. fairness, accessibility, transparency) from the literature and then assessed the degree to which DSS features might influence these participation criteria.<ref>http://fp0804.emu.ee/pdf/STSM_scientific_report_Nordstrom.pdf</ref> | ||
| + | |||
| + | * A more general example of an explanatory research question is: What factors make a forest DSS successful? | ||
| + | For a DSS to be successful, a simple model could consider three stages: 1) development of the DSS itself > 2) adoption of the DSS by users > 3) use of the DSS. A case could focus on one of these stages or try to cover all three. | ||
| + | ; DSS development factors | ||
| + | : The use of iterative prototyping and feedback from potential users is an often-cited success factor in DSS development. | ||
| + | ; DSS adoption (by users) factors | ||
| + | : A commonly used perspective on the adoption of technologies, such as DSS, relies on theories of innovation diffusion. Rogers (2003) text on innovation diffusion theory states that adoption rates depend on five attributes of innovations: relative advantage, compatibility, complexity, triability and observability. | ||
| + | ; DSS use factors | ||
| + | : Once the decision is made to use a DSS, both technical and social factors are likely to influence its success. DeLone and McLean (1992, 2003) have developed a widely cited framework for technical factors, which begins with: system quality, information quality, and service quality. Gordon (2006) reviewed social success factors, and reduced them to DSS effects on: participation, communication, translation, and mediation. | ||
==Methods - Case definition options== | ==Methods - Case definition options== | ||
| − | + | There are actually many ways in which "a case" can be defined - it is important to explicitly make a choice: | |
| − | * The application of a single DSS to a | + | * The application of a single DSS to a single problem (probably the most common type of case definition) |
| − | * The application of DSS (possibly multiple) | + | * A particular DSS over its lifetime (its application to multiple problems) |
| + | * The application of DSS (possibly multiple) to a particular problem domain (eg fire) or to a problem type | ||
==Methods - Selecting studies== | ==Methods - Selecting studies== | ||
===General=== | ===General=== | ||
| − | * | + | * Purpose will drive selection, examples: |
| − | * Replication strategy | + | ** Exploratory study of DSS for climate change would select DSS most relevant to climate change |
| + | |||
| + | * Replication strategy | ||
| + | ** similar to replicating an experiment | ||
| + | ** type of replication depends again on purpose | ||
| + | ***same DSS & same context OR same DSS & different contexts OR different DSS & same context, etc | ||
* How many cases do we undertake? | * How many cases do we undertake? | ||
** Depends on the detail needed (cases could be short or long, depending on purposes) | ** Depends on the detail needed (cases could be short or long, depending on purposes) | ||
| − | ===Specific=== | + | ===FORSYS Specific=== |
| − | * | + | * Representative sample of problem types |
| + | * For WG4 more interest on case specific studies (e.g. Valencia) while other groups looks for good general representation in terms of region, development approach, context etc. | ||
| + | * Most significant DSS identified by country reports | ||
| + | * Will we solicit/accept cases from non-Forsys members? | ||
| + | ** We could, but we'd need to figure out selection criteria and what would motivate a non-Forsys member to contribute (eg publication opportunities) | ||
| + | * Will we accept cases from other (non-forestry) domains? | ||
| + | ** Yes, if they are useful for contributing to our guidelines; will need to interpret their applicability to the forestry domain | ||
| + | * How do we handle DSS gaps, where there are no cases? | ||
| + | ** Gaps are important; we could try to build a matrix of gaps | ||
| + | ** May be able to address gaps using non-forest cases and more general literature syntheses | ||
==Methods – Data sources & collection methods== | ==Methods – Data sources & collection methods== | ||
| − | * six general sources for case studies: documents, archival records, interviews, direct observation, participant observation, physical artifacts | + | * There are six general sources for case studies: documents, archival records, interviews, direct observation, participant observation, physical artifacts |
| − | ==Methods - | + | ==Methods - Data Analysis== |
| − | * How to guarantee the | + | * Cover Forsys themes in description & tagging in semantic wiki |
| − | ** | + | ** Working group themes (architecture, models, knowledge mgt, participation) |
| + | ** Forsys problem dimensions | ||
| + | ** Forest resources: wood, fire, water, etc | ||
| + | ** Regions: regional issues | ||
| + | ** Stages of development: development, adoption, use, adaptation | ||
| + | ** Stages of decision making: organization-intelligence-design-choice-monitoring | ||
| + | ** Users: researchers, developers, users, stakeholders | ||
| + | |||
| + | * How can we capture lessons from cases and synthesize into final guidelines? | ||
| + | ** We need to develop a system, hopefully within our semantic wiki | ||
| + | * How can we use the existing literature to contribute to cases and the guidelines? | ||
| + | ** We need to find a good way to store literature citations and a way to annotate & tag them with Forsys concepts, lessons | ||
| + | |||
| + | ==Methods - improving validity== | ||
| + | * Case study is known as a triangulated research strategy. Snow and Anderson (cited in Feagin, Orum, & Sjoberg, 1991) asserted that triangulation can occur with data, investigators, theories, and even methodologies. Stake (1995) stated that the protocols that are used to ensure accuracy and alternative explanations are called triangulation. The need for triangulation arises from the ethical need to confirm the validity of the processes. In case studies, this could be done by using multiple sources of data (Yin, 1984). ([http://www.nova.edu/ssss/QR/QR3-2/tellis1.html Tellis 1997]) | ||
| + | ===Specific=== | ||
| + | * How to guarantee the "fairness" in evaluating case studies? | ||
| + | ** We should develop some form of collaboration and peer review, especially for cases where the case study writer is directly involved in the case | ||
==Reporting results== | ==Reporting results== | ||
| + | ===General=== | ||
| + | * Possible audiences: academic, DSS developers, forest managers, DSS funders | ||
| + | ===Forsys Specific=== | ||
* Add to wiki | * Add to wiki | ||
* Publish most developed case studies as journal articles (seek special issue?) | * Publish most developed case studies as journal articles (seek special issue?) | ||
| + | * Publicly accessible report for DSS users & managers | ||
==Other issues== | ==Other issues== | ||
| − | + | == References == | |
| − | + | <references/> | |
| − | + | ||
Latest revision as of 19:33, 24 November 2011
What is a case study?
- Excerpted from Wikipedia Case Study
- Rather than using samples and following a rigid protocol (strict set of rules) to examine limited number of variables, case study methods involve an in-depth, longitudinal (over a long period of time) examination of a single instance or event: a case. They provide a systematic way of looking at events, collecting data, analyzing information, and reporting the results. As a result the researcher may gain a sharpened understanding of why the instance happened as it did, and what might become important to look at more extensively in future research. Case studies lend themselves to both generating and testing hypotheses.
- From Yin (2003)
- typically used to investigate "how" or "why" questions about a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident
- Case studies vs literature syntheses
- There was discussion at our Leuven meeting about whether broadly defined cases (e.g. Fire behavior DSS in southern Europe) are really cases or more literature syntheses. It could be considered a broad case with "embedded" cases (Yin 2003), if it includes standardized descriptions (cases) for a number of DSS. If it does not contain such sub-cases, then it would be better to call it a synthesis. Either type of study could contribute to our overall purpose of defining guidelines for DSS development and use.
Logistics
The first questions we should answer are how much time and capacity do we have to compile the case studies?
Capacity for the work
- We must keep in mind our capacity to conduct case studies because this is a volunteer activity
- The case study committee will help coordinate the work but broad involvement from FORSYS participants is needed
- STSM's can support the work; we should discuss possibilities in Leuven
- We should consider how much to rely on existing materials/cases versus conducting new research
Timeline
- 18th November 2011 – FORSYS participants submit STSM opportunities to wiki
- 1st February 2012 – FORSYS participants submit case study (CS) abstracts to CS technical committee (CSTC)
- 1st March – CSTC provides reporting format for CS
- 1st March - CSTC responds to abstracts
- 1st June - CSTC follow up on cases
- 1st September – FORSYS participants submit first draft of CS to CSTC
- 1st October – CSTC responds to drafts
- 1st November – FORSYS participants submit final CS
- 20th December 2012 – CSTC synthesis
- 2013 CS synthesis informs development of FORSYS guidelines
- technically have until 3 months after end of project (4/2013) to complete all project work
Organization
How should we best organize ourselves to do the case study work ?
- By working groups: emphasis would be on WG topics (eg 3 cases on knowledge management in DSS)
- By mixed WG groups: try to cover all 4 topics for each case (more integrative but probably harder to organize)
- By regions or countries: each region would produce a number of cases
- By ad hoc case study groups based on individual interests
I don't believe we resolved this in Leuven, but the default seems to be
- ad hoc groups that build on existing research
- the working groups will probably play an important role in extracting/reviewing lessons related to their areas
Purpose(s) of FORSYS Case Studies
The most important question is what are the main purposes of our case studies? These purposes will influence the many methodological choices outlined below.
General ideas
Three general types of case studies:
- Exploratory: explores a relatively unknown issue, generally unstructured, develop hypotheses for further research
- Descriptive: some important features of the case(s) have been identified beforehand and are used to structure the case write-up
- Explanatory: a cause-effect hypothesis is stated, and the case is structured to test this relationship
Forsys Specific Purposes
- FORSYS general purpose: Produce guidelines for the development and use of forest decision support systems
- Case studies provide lessons learned, which will provide evidence for our overall guidelines
- Exchange knowledge among FORSYS researchers
- In the FORSYS application for the COST Action the CS were said to be selected based on their relevance to the European forest sector and the Strategic Research Agenda (SRA). In the SRA the Strategic Objective 4 "Meeting the multifunctional demands on forest resources and their sustainable management" is most relevant. It contains the research areas: 4-1 Forests for multiple needs, 4-2 Advancing knowledge on forest ecosystems and 4-3 Adapting forestry to climate change. If we are turning to these areas we might enter into DSS that are quite new and not put into practices to any larger extent but it might be important as well.
Specific examples
Exploratory
- What are the concrete impacts of DSS in sustainable forest management?
- How can DSS support EU Strategic Research Agenda:
- 4-1 Forests for multiple needs
- 4-2 Advancing knowledge on forest ecosystems
- 4-3 Adapting forestry to climate change.
Descriptive
- How do DSS contribute to different stages in decision making?
- The Participatory planning case template describes the roles of participation and DSS in particular cases using a descriptive structure based on Simon's (1960) stages of decision making: organization-intelligence-design-choice-monitoring
- Does the adoption and use of DSS vary by problem type?
- What are lessons learned by WG themes (architecture, models, knowledge mgt, participation)?
Explanatory
- The participatory case study group extracted success criteria for participation (e.g. fairness, accessibility, transparency) from the literature and then assessed the degree to which DSS features might influence these participation criteria.[1]
- A more general example of an explanatory research question is: What factors make a forest DSS successful?
For a DSS to be successful, a simple model could consider three stages: 1) development of the DSS itself > 2) adoption of the DSS by users > 3) use of the DSS. A case could focus on one of these stages or try to cover all three.
- DSS development factors
- The use of iterative prototyping and feedback from potential users is an often-cited success factor in DSS development.
- DSS adoption (by users) factors
- A commonly used perspective on the adoption of technologies, such as DSS, relies on theories of innovation diffusion. Rogers (2003) text on innovation diffusion theory states that adoption rates depend on five attributes of innovations: relative advantage, compatibility, complexity, triability and observability.
- DSS use factors
- Once the decision is made to use a DSS, both technical and social factors are likely to influence its success. DeLone and McLean (1992, 2003) have developed a widely cited framework for technical factors, which begins with: system quality, information quality, and service quality. Gordon (2006) reviewed social success factors, and reduced them to DSS effects on: participation, communication, translation, and mediation.
Methods - Case definition options
There are actually many ways in which "a case" can be defined - it is important to explicitly make a choice:
- The application of a single DSS to a single problem (probably the most common type of case definition)
- A particular DSS over its lifetime (its application to multiple problems)
- The application of DSS (possibly multiple) to a particular problem domain (eg fire) or to a problem type
Methods - Selecting studies
General
- Purpose will drive selection, examples:
- Exploratory study of DSS for climate change would select DSS most relevant to climate change
- Replication strategy
- similar to replicating an experiment
- type of replication depends again on purpose
- same DSS & same context OR same DSS & different contexts OR different DSS & same context, etc
- How many cases do we undertake?
- Depends on the detail needed (cases could be short or long, depending on purposes)
FORSYS Specific
- Representative sample of problem types
- For WG4 more interest on case specific studies (e.g. Valencia) while other groups looks for good general representation in terms of region, development approach, context etc.
- Most significant DSS identified by country reports
- Will we solicit/accept cases from non-Forsys members?
- We could, but we'd need to figure out selection criteria and what would motivate a non-Forsys member to contribute (eg publication opportunities)
- Will we accept cases from other (non-forestry) domains?
- Yes, if they are useful for contributing to our guidelines; will need to interpret their applicability to the forestry domain
- How do we handle DSS gaps, where there are no cases?
- Gaps are important; we could try to build a matrix of gaps
- May be able to address gaps using non-forest cases and more general literature syntheses
Methods – Data sources & collection methods
- There are six general sources for case studies: documents, archival records, interviews, direct observation, participant observation, physical artifacts
Methods - Data Analysis
- Cover Forsys themes in description & tagging in semantic wiki
- Working group themes (architecture, models, knowledge mgt, participation)
- Forsys problem dimensions
- Forest resources: wood, fire, water, etc
- Regions: regional issues
- Stages of development: development, adoption, use, adaptation
- Stages of decision making: organization-intelligence-design-choice-monitoring
- Users: researchers, developers, users, stakeholders
- How can we capture lessons from cases and synthesize into final guidelines?
- We need to develop a system, hopefully within our semantic wiki
- How can we use the existing literature to contribute to cases and the guidelines?
- We need to find a good way to store literature citations and a way to annotate & tag them with Forsys concepts, lessons
Methods - improving validity
- Case study is known as a triangulated research strategy. Snow and Anderson (cited in Feagin, Orum, & Sjoberg, 1991) asserted that triangulation can occur with data, investigators, theories, and even methodologies. Stake (1995) stated that the protocols that are used to ensure accuracy and alternative explanations are called triangulation. The need for triangulation arises from the ethical need to confirm the validity of the processes. In case studies, this could be done by using multiple sources of data (Yin, 1984). (Tellis 1997)
Specific
- How to guarantee the "fairness" in evaluating case studies?
- We should develop some form of collaboration and peer review, especially for cases where the case study writer is directly involved in the case
Reporting results
General
- Possible audiences: academic, DSS developers, forest managers, DSS funders
Forsys Specific
- Add to wiki
- Publish most developed case studies as journal articles (seek special issue?)
- Publicly accessible report for DSS users & managers
