- Journal List
- HHS Author Manuscripts
- PMC4941817
As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsem*nt of, or agreement with, the contents by NLM or the National Institutes of Health.
Learn more: PMC Disclaimer | PMC Copyright Notice
J Mix Methods Res. Author manuscript; available in PMC 2017 Jul 1.
Published in final edited form as:
Published online 2015 Feb 25. doi:10.1177/1558689815572023
PMCID: PMC4941817
NIHMSID: NIHMS716272
PMID: 27429602
Deborah Cragun,1 Tuya Pal,1 Susan T. Vadaparampil,1 Julie Baldwin,2 Heather Hampel,3 and Rita D. DeBate2
Author information Copyright and License information PMC Disclaimer
The publisher's final edited version of this article is available at J Mix Methods Res
Associated Data
- Supplementary Materials
Abstract
Qualitative comparative analysis (QCA) was developed over 25 years ago to bridge the qualitative and quantitative research gap. Upon searching PubMed and the Journal of Mixed Methods Research, this review identified 30 original research studies that utilized QCA. Perceptions that QCA is complex and provides few relative advantages over other methods may be limiting QCA adoption. Thus, to overcome these perceptions, this article demonstrates how to perform QCA using data from fifteen institutions that implemented universal tumor screening (UTS) programs to identify patients at high risk for hereditary colorectal cancer. In this example, QCA revealed a combination of conditions unique to effective UTS programs. Results informed additional research and provided a model for improving patient follow-through after a positive screen.
Keywords: configurational comparative method, effectiveness, evaluation, cross-case comparison, RE-AIM
Use of what is still sometimes dichotomized into qualitative and quantitative research methods in complimentary or comparative ways has become widely accepted in several social science disciplines (Bazeley, 2009). In contrast, the extent to which various disciplines accept and utilize approaches that fuse or blend qualitative and quantitative methods is less clear. Qualitative comparative analysis (QCA) is a hybrid method designed to bridge the qualitative (case-oriented) and quantitative (variable-oriented) research gap and to serve as a practical approach for understanding complex, real-world situations (Ragin, 1987; ). QCA was initially developed by Dr. Charles Ragin for use in small- or medium-N case study research (Ragin, 1987). QCA combines Boolean algebra and minimization algorithms to systematically compare cases and derive solutions consisting of one or more patterns of conditions that when present or absent are uniquely associated with the presence or absence of an outcome (Ragin, 1987). QCA therefore takes a set-theoretic approach originating from the idea that attributes of cases are often best evaluated in a holistic fashion using set relations (Ragin, 1987; ). In QCA, set membership is assigned based on whether or to what degree a case satisfies criteria for each outcome or condition. When QCA was originally developed, conditions and outcomes were dichotomized as either present or absent and cases were classified according to whether they belong in each set. This original technique is now typically referred to as crisp-set QCA (csQCA) in order to distinguish it from related techniques that were later developed (). Other QCA techniques include multi-value QCA (mvQCA) which allows outcomes to have more than two values and fuzzy-set QCA (fsQCA) which allows for wide variation in the extent to which cases satisfy set membership criteria for each outcome and condition (). Software programs are available to assist in performing QCA; one of these was developed by Charles Ragin and is freely available for download online at http://www.fsqca.com along with a user manual (Ragin et. al, 2006).
Some criticisms of QCA are based on its perceived complexity or lack of identified advantage over other methods (Hawley, 2007). Admittedly, more traditional qualitative approaches to performing multiple cross-case comparisons exist (). However, as the number of cases increases, systematic comparisons may not be logistically feasible without using QCA software. Another advantage of QCA stems from its mathematical approach to identify solutions and assess their overall merit, a quality valued by journals that publish primarily “quantitative” research.
The versatility of QCA is evidenced through its use in conjunction with various types of research designs (Kahwati et al., 2011; ; ). QCA can be used to analyze individual-level, institution-level, or country-level data from studies with small, medium, and large sample sizes. Furthermore, both unstructured data (e.g., interview transcripts) and structured data (e.g., responses to closed-ended survey questions) can be used to perform QCA.
The ability of QCA to identify combinations of conditions that are likely to be 'necessary' and/or 'sufficient' for a particular outcome of interest to occur is useful for developing or testing theories and models. For example, knowledge about a positive health behavior may be necessary, but it is rarely sufficient to ensure that individuals will perform the health behavior. According to the Health Belief Model (), individuals often require a combination of the following conditions in order to perform a positive health behavior: 1) knowledge about the behavior; 2) high level of perceived threat to their health if they fail to perform the behavior; 3) high-level of perceived benefits to performing the behavior; and 4) low-level of perceived barriers to performing the behavior. The ability to identify this type of “causal complexity” is one reason why QCA can be useful when generating or testing theoretical models (Ragin, 1987).
Structural equation modeling (SEM) is a more commonly used analytic technique that also allows researchers to incorporate multiple variables and test theoretical models. Although SEM may arguably be easier to use than QCA (Hawley, 2007), SEM requires large samples and the results are interpreted in a reductionist manner by considering the influence that one variable has on the outcome while holding all other variables in the model constant. Furthermore, unlike QCA, SEM and other inferential statistical techniques typically fail to consider the possibility of equifinality, whereby different combinations of conditions can lead to the same outcome (Ragin, 1987; ). For example, the combination of knowledge about how to perform a behavior along with a high level of perceived benefits may be sufficient to elicit a positive health behavior among a subset of women who do not face a particular barrier; however, additional or different conditions may be needed to elicit the behavior among other individuals. If a key factor is relevant to the outcome for only a subset of individuals, the correlation between the factor and outcome is weakened, potentially causing what may be a key factor to be deemed insignificant if inferential statistics are used. Additionally, inferential statistics assume that the influence of variables is symmetrical even though conditions that lead to the consistent performance of a health behavior may be different from conditions that cause poor adherence to the behavior.
Despite several relative advantages to QCA, the extent to which this hybrid analytic approach has diffused and been adopted across academic disciplines remains unclear. Thus, the first objective of this article is to explore the diffusion and adoption of QCA through health research channels and mixed methods researchers. To achieve this objective, results are presented from a literature search of articles indexed by PubMed and articles published in the Journal of Mixed Methods Research. The second objective is to discuss several potential reasons for the diffusion and adoption rates of QCA. Subsequently, to promote the broader goal of active QCA dissemination, the final objective is to increase knowledge of QCA and decrease perceived complexity. To achieve the final objective, data obtained as part of a multiple-case study are used to demonstrate how to perform csQCA and to illustrate benefits and limitations of this technique.
Diffusion and Adoption of QCA
In April of 2014, the index term “qualitative comparative analysis” was used for online searches of articles indexed by PubMed or published in the Journal of Mixed Methods Research (JMMR). Abstracts of all articles retrieved using the designated search term and published in or after 1987 (when QCA was developed) were reviewed. Articles were initially counted if the authors used any of the three QCA types mentioned previously (i.e., crisp-set, fuzzy-set, or multi-value) in an original research study or with hypothetical data. Given the paucity of articles, criteria were extended to include any articles where the authors described or mentioned QCA in order to evaluate contexts in which this method has been discussed.
Only 30 articles meeting the initial inclusion criteria had been indexed by PubMed as of April 2014, with 29 of them reporting data from an original research study and one that used hypothetical data. After expanding the criteria, two additional PubMed articles were identified. Among the latter articles, one mentioned QCA, along with a few other “new techniques”, as a potential way to help advance research in topic areas of stress, coping, and social support (Thoits, 1995); and the other described QCA and several other methods used in synthesizing qualitative and quantitative evidence ().
Only one article published in JMMR as of April 2014 met the initial inclusion criteria, but 8 met expanded criteria. The single JMMR article meeting initial criteria reported how a large-N survival analysis and small-N QCA yielded new insights about the reasons for project delay in various organizations (Krohwinkel, 2014). One of the articles meeting expanded criteria was a book review by Hawley (2007). An additional seven articles mentioned QCA during discussions on various topics including: integration, synthesis, and triangulation in mixed methods research (Bazeley, 2009; ; ; Wolf, 2010); qualitative data analysis tools (); data analysis as a process of interpretation (); or lack of experimentation with innovative methods such as QCA ().
Although this literature search was limited in scope, the articles reveal diverse contexts in which QCA has been utilized either as the sole analytic technique or less commonly to complement other analytic techniques. Articles posited contrasting views in terms of which end of a qualitative/quantitative spectrum researchers classify QCA. Finally, this review substantiates the assertion that QCA has been slow to diffuse into health research, but also suggests that the rate at which QCA is being adopted in health research may be increasing over time. Support for this latter assertion comes from the finding that half of the QCA articles identified in PubMed were published after 2011.
Potential Reasons for the Slow Diffusion of QCA
Diffusion of Innovations Theory (Rogers, 2003) provides several possible explanations for these findings. First, an innovation takes time to diffuse within and across social groups and the rate of diffusion is dependent on communication channels. QCA was developed in the late 1980's by Charles Ragin, a Sociologist who studies politics (Ragin, 1987). QCA therefore had to spread across members of Political Science and Sociology disciplines through a limited number of communication channels into other disciplines. Second, QCA may be viewed by some researchers as being incompatible with the methodological paradigm to which they may still subscribe (Barbour, 1998). “Qualitative” researchers might view QCA as incompatible because it is based on Boolean algebra and a computer program is typically used to aid the researcher in identifying solutions which are then evaluated using quantitative measures called solution consistency and coverage. Whereas “quantitative” researchers may view QCA as incompatible because it entails an iterative process of evaluating data, often uses a non-random sample, and requires researchers to use their substantive knowledge of the cases to make several 'subjective or interpretive' decisions at multiple points during the analysis (). Third, knowledge about how QCA works may be limited as there appear to be a relatively small number of researchers who have been trained to conduct QCA. Fourth, performing QCA was complex until computer software became widely available and automated much of the process. Nevertheless, Hawley (2007) has pointed out that the unique terminology used in QCA also makes learning this technique inherently difficult. Furthermore, additional complexities have arisen as researchers have developed several different types of QCA ().
Given that mixed methods researchers generally take a pragmatic approach that transcends the positivist/constructivist or quantitative/qualitative “paradigm wars” (Morgan, 2007), findings from the JMMR review, which suggested that few mixed methods researchers have adopted QCA, were unexpected. Hawley’s (2007) description of QCA in the book review published in JMMR suggests that high perceived complexity and lack of relative advantage over other techniques may explain the slow diffusion and low adoption rates. Therefore, to reduce complexity, the following section provides a stepwise account of how QCA was highly instrumental as an initial step in a multiple-case study designed to evaluate the implementation and effectiveness of universal colorectal tumor screening programs to identify Lynch syndrome.
QCA Example
Background on Universal Tumor Screening (UTS) for Lynch syndrome
Lynch syndrome, the most common cause of hereditary colorectal cancer (CRC), confers a 50–70% lifetime risk of colorectal cancer (CRC) (Barrow et al., 2008; Hampel, Stephens, et al., 2005; E. Stoffel et al., 2009) as well as increased risks for other cancers (Barrow et al., 2009; Hampel, et al., 2005; Stoffel et al., 2009; Watson et al., 2008). Universal tumor screening (UTS) is the process whereby tumors from all newly diagnosed CRC patients are screened to identify those patients who may have Lynch syndrome (Bellcross et al., 2012). Over 35 cancer centers and hospitals across the U.S have implemented UTS, but substantial variability in protocols and procedures exist across institutions (Beamer et al., 2012; Cohen, 2013). Outcomes also vary across institutions as noted by large differences in the percentage of patients with a positive screen who follow-through with genetic counseling and germline genetic testing (Beamer et al., 2012; Lynch, 2011; South et al., 2009; Cragun et al., 2014). Considering the critical importance of patient follow-through to the successful identification of family members with Lynch syndrome and subsequent prevention or early detection of cancers (Bellcross et al., 2012), a multiple-case study was initiated to identify institution-level conditions that might contribute to the wide variability in patient follow-through.
Study Design
Upon approval from the Institutional Review Board at the University of South Florida, a multiple-case study was initiated during the fall of 2012. The rationale for employing a multiple-case study design was based on the following (adapted from Yin, 2008): (a) the key objective was to provide a detailed understanding of a complex phenomenon (i.e. UTS program implementation and patient follow-through) for which there is limited data; (b) the purpose was to answer how and why questions; (c) the behavior of those involved could not be manipulated; and, (d) it was hypothesized that contextual conditions would be relevant to variations in patient follow-through. The current article uses data from an online survey of institutional representatives. However, additional data were collected through a six-month follow-up survey and interviews with institutional representatives and other personnel involved in UTS implementation at participating institutions. Details regarding the overall study design have been published elsewhere (Cragun et al., 2014).
Conceptual Frameworks
The RE-AIM evaluation framework and the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009; ) served as the conceptual framework for the multiple-case study. RE-AIM is comprised of five evaluation dimensions (Follow-through, Effectiveness, Adoption, Implementation, and Maintenance) that assist with identifying conditions for multi-level comprehensive evaluations (). In the current study the RE-AIM evaluation dimensions were defined as follows:
Reach: the absolute number, proportion, and representativeness of CRC patients screened.
Effectiveness: the impact of UTS on outcomes (including patient follow-through with genetic counseling and germline genetic testing after a positive screen and potential negative effects).
Adoption: the absolute number, proportion, and representativeness of institutions and staff who implement UTS.
Implementation: the consistency of delivery, time and cost of the UTS program and what adaptations are made in various settings.
Maintenance: the effects of UTS over time with regard to both the institution and patients.
RE-AIM was selected based on the expectation that it would increase the quality, speed, and impact of stakeholder efforts to more effectively translate UTS into practice. The CFIR provided a framework for exploring the Implementation dimension of RE-AIM and identify conditions that might influence Effectiveness. Table 1 lists the five CFIR domains and several conditions within each (Damschroder et al., 2009).
Table 1
Five Domains of the Consolidated Framework for Implementation Research (CFIR)
| CFIR Domain | Description and Examples of Associated Constructs |
|---|---|
| Intervention | Characteristics of the intervention such as complexity, cost, and relative advantage. |
| Inner setting | Structural, political, and cultural contexts through which implementation proceeds. Includes organizational structure, social architecture, communication/networks, and implementation climate & readiness. |
| Outer Setting | Economic, political, and social context in which an organization resides. Includes the extent to which the organization has an accurate knowledge of patient needs, billing & reimbursem*nt, funding constraints, and ties to external organizations. |
| Individuals involved | Individuals in the inner or outer setting can promote the implementation process and alter program effectiveness via their actions which are influenced by motivations, attitudes, etc. |
| Implementation Process | Processes include actions that lead to implementation, protocol and procedures, and ongoing reflection. |
Open in a separate window
Data analysis and interpretation were influenced by the RE-AIM and CFIR frameworks as well as by the following assumptions: 1) UTS implementation experiences will differ across institutions; 2) despite implementation heterogeneity, QCA can identify patterns of conditions that are consistently associated with high patient follow-through (i.e., program effectiveness); 3) Results of QCA, in conjunction with detailed knowledge of each institution’s unique experiences, can be used to propose a “causal model” explaining differences in patient follow-through across institutions; 4) this model can inform implementation recommendations that are expected to improve program effectiveness; 5) research is an iterative process and alternative ways to achieve high patient follow-through may later be identified, thereby necessitating changes to the model. Based on these assumptions, this study methodology is pragmatic rather than being rooted in any of the competing epistemological or methodological paradigms (i.e., positivist vs. constructivist or quantitative vs. qualitative) (Morgan, 2007).
Study Participants
Representatives for the Lynch Syndrome Screening Network (LSSN) who worked at various institutions that perform UTS were recruited using methods detailed in a previous manuscript (Cragun et al., 2014). After reviewing a consent form, participants completed an initial survey. Fifteen participants met the following a priori inclusion criteria: 1) represented institutions that had been performing Lynch syndrome screening on tumors from all newly diagnosed colorectal cancer (CRC) patients for at least six months; and 2) had access to institutional data on patient follow-through with genetic counseling and germline genetic testing. Patient follow-through data were collected by each institutional representative and provided in aggregate form to maintain strict patient anonymity. Names of institutions and institutional representatives were de-identified to maintain confidentiality.
Measures
The key study variables were derived from institution-level data that were collected through surveying institutional representatives. The initial online survey was developed, with input from several experts in cancer genetics and behavioral science, using the RE-AIM and CFIR frameworks as well as the researchers’ knowledge of institutional variations in UTS protocols. Information collected included: a) length of time UTS had been performed at the institution; b) details on the implementation process, protocol, and procedures; c) percentage of patients who undergo genetic counseling and percentage who undergo germline genetic testing; and, d) additional conditions within CFIR domains that may have helped facilitate or impede patient follow-through after a positive tumor screen. Details pertaining to survey content, validation, and piloting are reported elsewhere (Cragun et al., 2014).
Crisp-Set Qualitative Comparative Analysis (csQCA)
In the current study, QCA was used to identify facilitators, barriers or other conditions unique to institutions that reported relatively high patient follow-through; thus the unit of analysis was at the level of the institution. Crisp-set QCA (csQCA) was chosen for two main reasons: 1) the conditions assessed in the survey were dichotomous; and 2) csQCA is simpler to perform and interpret than other QCA techniques (). Steps used to perform csQCA in the current example are summarized in Table 2 and detailed in the next sections. In practice these steps are somewhat fluid as QCA is an iterative (rather than linear) process that allows modifications to be made as researchers gain additional information and insights into the cases (Ragin, 1987; ; ). Briefly, steps 1–3 are needed to prepare data for use in QCA. Step 4 involves deciding which type of analyses to perform. Steps 5–9 describe how to determine which conditions are sufficient for the outcome. Finally, solutions are interpreted to propose “causal models” in step 10. Screenshots illustrating steps to using fsQCA2.0 software are available online in supplementary figures.
Table 2
Summary of Steps Used to Perform Crisp-set Qualitative Comparative Analysis (csQCA)
| csQCA steps | Application of QCA steps in the current study |
|---|---|
| Step 1: (a) Determine, define, and operationalize the outcome of interest (b) Assign dichotomous set membership scores for the outcome | (a) Outcome=patient follow-through (PF) Defined as the percentage of patients who follow-through with genetic counseling and germline genetic testing following an abnormal tumor screen at each institution. Operationalized based on two survey questions as described in the manuscript text. (b) Cases naturally fell into three groups or sets: High-PF; Medium- PF, and Low-PF. Cases with a PF score ≥5 were included in the High-PF set and coded as High-PF=1. All other cases were coded High-PF=0 and are referred to with a tilde to indicate they are not in the High-PF set (i.e., ~High-PF). |
| Step 2: Select Cases | Several High-PF and several Low-PF institutions were needed. However, to maximize both sample size and diversity in contextual variables, all available cases that met the minimum a priori inclusion criteria were used in the analysis. |
| Step 3: (a) Identify key conditions (b) Assign dichotomous set membership scores for each condition (c) Create a data matrix of scores for conditions | (a) As part of the multiple-case study, data on many conditions were collected to gain an in-depth understanding of the cases. Based on theory and careful review of the cases, conditions for possible inclusion in QCA were selected as detailed in the manuscript text. (b) Although this is often not the case, all of the conditions were already dichotomized as either present=1 or absent=0 based on how they were asked on the survey. (c) A data matrix (Table 3) was created by listing membership scores for the outcome and key conditions for each case. |
| Step 4: Determine which analyses to run | To determine whether conditions are necessary for the presence of an outcome, a separate analysis is recommended. However, none of the conditions in this study were hypothesized to be necessary in all High-PF cases. Thus, only sufficiency analyses were conducted. |
| Step 5: Create a “truth table” | Although not necessary for the presence of High-PF, conditions may be sufficient for High-PF either when occurring alone or in combination with other conditions. Using freely available software (fsQCA 2.0), a truth table was created showing all possible configurations of selected conditions (Table 4). |
| Step 6: Examine the truth table and resolve contradictions | The first row of the truth table (Table 4) shows the configuration that contains 4 High-PF cases as well as 1 Medium-PF case (consistency =.8). The second row contains 1 High-PF and 1 Medium-PF case (consistency = .5) To resolve these contradictions, an additional condition (diff_contact_pt) was added to create a revised truth table (shown in abridged format in Table 5). |
| Step 7: Use computer software to generate solutions through multiple comparisons of case configurations in the truth table | Using fsQCA 2.0 software, a “Standard Analysis” was performed to identify conditions associated with High-PF. This software uses the Quine-McCluskey algorithm (which is based on Boolean simplification) to make multiple comparisons of case configurations represented in the truth table and logically simplify the data. During this process input from the researchers was required to select prime implicants and determine which simplifying assumptions were tenable. The software then used this information to generate three solutions (complex, parsimonious, and intermediate) for High-PF. |
| Step 8: Determine if the influence of conditions is symmetrical | To determine if conditions associated with High-PF are the same as those associated with the absence of the outcome (~High-PF), steps 4–6 were repeated using ~High-PF as the outcome. |
| Step 9: Evaluate the consistency and coverage of the solutions | The overall solution consistencies were 1 for each of the two outcomes evaluated (High-PF and ~High-PF), indicating that the respective combination of conditions were consistently associated with the respective outcome or absence thereof. The overall coverage for each solution was 1; indicating that all of the cases with the presence (or absence) of the outcome were explained (i.e., covered) by the respective solution. |
| Step 10: Interpret the resulting solutions and create causal models | Even when conditions are uniquely and consistently associated with an outcome, it does not necessarily mean they cause the outcome. However, these solutions in conjunction with theories, frameworks, and details about the cases can be used to develop a causal theoretical model that describes how the conditions might lead to the outcome. |
Open in a separate window
Step 1: Outcome operationalization and set membership scoring
The outcome (i.e., patient follow-through) was operationalized using two questions assessing the percentage of patients who follow-through with genetic counseling and percentage who follow-through with germline genetic testing after a positive tumor screening result. Response options were the same for both questions: 1 = <10%; 2 = 11–25%; 3 = 26–40%; 4 = 41–55%; 5 = 56–70%; 6 = 71–85%; and 7 =>85%. The ordered categorical response options for the two questions were averaged to create a patient follow-through (PF) score ranging from 1–7. After arranging cases in descending order by PF, two natural breaks were identified (Table 3, column 1). The first 5 cases were grouped into the High-PF set, the second 5 cases into the Medium-PF set, and the last 5 into the Low-PF set. Natural breaks were chosen to ensure that cases with very similar values were grouped together, as has been recommended by .
Table 3
Data Matrix of Conditions Considered for Inclusion in QCA
| Patient follow- through (PF) scorea | Set member -shipb | Out- come | Conditions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (High -PF)c | d | e | f | g | h | i | j | k | l | m | n | o | ||
| 6 | High-PF | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 |
| 5.5 | High-PF | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 5 | High-PF | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 |
| 5 | High-PF | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 |
| 5 | High-PF | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 |
| 4 | Med-PF | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 |
| 3.5 | Med-PF | 0 | 0 | - | - | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 |
| 3 | Med-PF | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| 3 | Med-PF | 0 | 0 | - | - | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 2.5 | Med-PF | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
| 1.5 | Low-PF | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
| 1 | Low-PF | 0 | 0 | - | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
| 1 | Low-PF | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
| 1 | Low-PF | 0 | 0 | - | - | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| 1 | Low-PF | 0 | 0 | - | - | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
Open in a separate window
Notes:
aPatient follow-through (PF) was calculated by averaging the ordinal response options from two questions estimating the percentage of patients who follow-through with genetic counseling and percentage who follow-though with germline testing after a positive screen.
bNatural break points in PF scores were used to initially categorize institutions into three sets (High-PF, Med-PF, Low-PF).
cThe outcome for the initial QCA was High-PF (presence=1, absence=0).
dGenetic professional discloses positive tumor screening results (presence=1, absence=0, ).
eScreening results disclosed by telephone (presence=1, absence=0, don’t know = “-“).
fScreening results disclosed at follow-up visit (presence=1, absence=0, don’t know = “-“).
gObtaining/receiving a referral from a non-genetics health care provider is primary mechanism for genetic testing (presence=1, absence=0)
hHealth care providers often fail to see the importance of genetic counseling after a positive screen (presence=1, absence=0).
iHealth care providers often fail to see the importance of germline testing after a positive screen (presence=1, absence=0).
jCombined condition based on Boolean addition “OR” (presence of condition “h” OR condition “i”=1, absence of both conditions=0).
kDifficulty contacting patients to set up genetic counseling after a positive tumor screen (presence=1, absence=0).
lDifficulty contacting patients to arrange germline genetic testing after a positive screen (presence=1, absence=0).
mCombined condition based on Boolean addition (presence of condition “k” OR condition “l”=1, absence of both=0).
nGenetic professional is responsible for pre-test discussion of germline testing with the patient (presence=1, absence=0).
oGenetic professional contacts patient directly to set up pre-test counseling and germline testing (presence=1, absence=0).
One key limitation of csQCA is that all variables, including the outcome, need to be dichotomized so that the case either belongs to the set (coded as 1) or does not belong to the set (coded as 0). In the current study the threshold for inclusion in the High-PF set was a PF score ≥5. All other cases did not belong in the High-PF set. Cases not in a set are referred to by placing a tilde before the abbreviation (e.g., ~High-PF).
Step 2: Case selection
Although QCA has been used to analyze data from random samples, it was developed to compare cases that are carefully chosen using one of a number of different selection procedures (Gerring, 2007). In the current study, institutions representing High-PF and Low-PF sets were needed to determine conditions contributing to wide variability in patient follow-through across institutions. To maximize both sample size and diversity in conditions, all cases that met minimal inclusion criteria were dichotomized according to membership in the High-PF set and used in the analysis.
Step 3: Selection of key conditions
Although many CFIR constructs were measured to assist in gaining an in-depth understanding of each case, only a relatively small number of conditions could be used in QCA for two main reasons. First, the number of possible configurations increases exponentially according to an increase in the number of conditions; and this increases the likelihood that there will be a number of configurations for which there are no cases (i.e., remainders). Second, when the ratio of conditions to cases is high, the probability of getting a solution that just by chance appears sound even when the model is misspecified increases (). Guidelines from a simulation study by Marx and Dusa (2011) were therefore followed by limiting analyses to no more than 4 conditions so that misspecification of the model would most likely lead to contradictory cases (i.e., cases with the same configuration of conditions, but different outcomes).
In the current study, processes related to disclosure of screening results and discussion of genetic testing as well as the individuals involved with these processes were hypothesized to have the most direct influence on patient follow-through. As a first step in narrowing down the number of conditions to consider for QCA, a data spreadsheet of responses from each institutional representative was created by the researchers with cases organized from highest to lowest PF. Frequencies of responses were then generated for each PF category (i.e., High-PF, Medium-PF, Low-PF). Conditions were evaluated by the researchers in terms of how each might relate to patient follow-through independently or in combination with other conditions. During the selection process the researchers created a data matrix (Table 3) of set membership scores for the conditions considered for inclusion in QCA. The data matrix was then reviewed by the researchers to narrow down the list of conditions. This process consisted of a series of decisions described in more detail below whereby similar pairs of conditions were combined to create composite conditions and several conditions were later deleted.
General differences between PF groups were found with regard to who discloses positive screening results to patients. All representatives of the five High-PF institutions reported that a genetics professional discloses abnormal screening results to patients. There were also two Medium-PF institutions where a genetics professional discloses positive results. This condition was included in QCA and is referred to as (gen_prof_disclose_screen). How positive results were first disclosed (i.e., by phone or at a follow-up visit) was mixed across the PF groups; and was subsequently deleted from the data matrix. Nearly all institutions have genetics professionals provide pretest counseling prior to germline genetic testing. Consequently, this condition was deleted from the data matrix due to lack of variability ()
Several conditions that could act as barriers to patient follow-through with genetic counseling and germline genetic testing were also considered. Most ~High-PF institutions reported that obtaining a referral from a healthcare provider was the primary mechanism for the patient to receive genetic testing and was coded as (referral_barrier) for use in QCA. Other barriers demonstrated similarities in response patterns. Therefore analogous pairs of barriers were combined using the Boolean operator “OR”, which indicates Boolean addition. As an example, the new composite condition (difficulty_contact_pt) was “present” and coded as “1” if either (a) the institutional representatives indicated that difficulty contacting patients to set up genetic counseling was a barrier “OR” (b) that difficulty contacting patients to set up germline genetic testing was a barrier. Whereas if neither of these barriers were reported, then the new composite condition was considered “absent” and coded “0”.
The revised data matrix contained three conditions selected for inclusion in QCA (gen_prof_disclose_screen, referral_barrier, and gen_directly_contacts_pt) as well as several additional barriers to consider. Once complete, the principal investigator saved the data matrix (which was in an Excel spreadsheet) as a .csv file because this type of file can be opened and read by fsQCA2.0 software using the point and click FILE menu (Ragin et. al, 2006).
Step 4: Decide which analyses to run
While the focus of QCA is often on identifying conditions that are sufficient for the presence of an outcome, researchers have suggested that sufficiency analysis be preceded by identifying potential necessary conditions (). A necessary condition is one that occurs in all cases that demonstrate the presence of the outcome. There are many instances where a theory or previous empirical observations would lead researchers to hypothesize that certain conditions may be either 1) necessary and sufficient for an outcome or 2) necessary but insufficient for an outcome. However, in the current study, none of the conditions were originally hypothesized to be necessary for achieving high PF. Therefore, only analyses to determine sufficiency were performed.
FsQCA 2.0 software developed by Charles Ragin was chosen to run the sufficiency analyses as it is freely available for download online at http://www.fsqca.com along with a user manual (Ragin et. al, 2006). However, to help decrease perceived complexity, basic steps performed in the current study are described below. Also, to reduce complexity, key QCA terms are defined and illustrated throughout the following step-by-step description, but QCA jargon is used sparingly.
Step 5: Create a truth table
Using fsQCA software, “Truth Table Algorithm” was selected under the ANALYSE > Crisp sets menu. The outcome and conditions were chosen as prompted in the pop-up window before clicking the “run” button. The software then created a truth table similar to the replica in Table 4. Each row of the truth table shows a configuration of conditions and lists the number of cases that share that configuration. As is often the case, several configurations had no case examples (rows E-H); and these are called remainders ().
Table 4
Initial Truth Table of All Potential Conditional Configurations
| Rowa | gen_prof_ disclose_ screen b | referral_ barrier c | gen_directly_ contacts_pt d | # cases fitting configuration | High-PF e (outcome) | Raw consistency |
|---|---|---|---|---|---|---|
| A | 1 | 0 | 1 | 5 | 0.8f | |
| B | 1 | 0 | 0 | 2 | 0.5f | |
| C | 0 | 1 | 0 | 8 | 0g | |
| D | 0 | 1 | 1 | (remainder)h | ||
| E | 0 | 0 | 0 | (remainder)h | ||
| F | 0 | 0 | 1 | (remainder)h | ||
| G | 1 | 1 | 0 | (remainder)h | ||
| H | 1 | 1 | 1 | (remainder)h |
Open in a separate window
Notes: This is a replica of the initial truth table generated using fsQCA 2.0 software. However, the first column was added to label configurations and several descriptors were added in parentheses.
aEach potential configuration of conditions is represented by a row. Since there are 3 conditions there are 23 (8) possible configurations.
bGenetics professional discloses positive screening results (presence=1, absence=0)
cReferral is primary mechanism for patient to receive genetic testing (presence=1, absence=0)
dGenetic professional contacts patient to set up counseling and testing (presence=1, absence=0)
eThe outcome column is blank because the software requires the researchers to fill in a 0 or 1 for each configuration (row) based on whether or not the cases that share that configuration have the outcome of interest (i.e., high patient follow-through; High-PF).
fThese configurations contain contradictions (as indicated by consistency scores). Consistency for row A is 0.8 because 4 of the 5 cases with this configuration have High-PF=1. Consistency for row B is 0.5 because only one of the two cases in this configuration belongs to the High-PF set. Contradictions must be resolved before assigning outcome scores.
gThe consistency score for row C is 0 because none of the cases with this configuration have PF.
hThere are no consistency scores for rows D-H because there are no cases in this sample that fit these configurations. These are called remainders.
Step 6: Examine the truth table and resolve contradictions
The objective when creating a truth table is to ensure that all cases that share a configuration also share the same outcome. The consistency score for each row indicates the proportion of cases in the respective configuration that belong to the High-PF set (i.e., outcome is present). When the consistency is 1 it indicates that the configuration of conditions is always associated with the presence of the outcome. In the initial truth table (Table 4) generated for the current study, rows A and B have consistency scores of 0.8 and 0.5, respectively. This suggests that these rows represent configurations where the outcome is inconsistent. Specifically, row A represents a configuration that is shared by 4 High-PF cases and 1 Medium-PF case; and row B represents a configuration that is shared by 1 High-PF case and 1 Medium-PF case. The need to resolve such contradictions often occurs in QCA (). Contradictions provide researchers an opportunity to gain additional understanding of the cases and serves as a mechanism for building models (Ragin, 2004). For example, contradictions could indicate that a key condition is missing from the model.
To resolve the contradictions, the research team went back to the reduced data matrix to examine the cases and then select another key barrier. Logic dictated that difficulty contacting patients after a positive screen (difficulty_contact_pt) would directly lower PF. Once this condition was added, the new truth table contained no contradictions (Table 5). The consistency scores for the first two configurations (rows A-B) were 1 and the consistency scores for the other configurations (rows C-F) were 0. Thus, the outcomes of the first two configurations (rows A-B) were coded 1 by the researchers and the outcomes of all the other configurations for which there were cases (rows C-F) were coded 0. Table 5 does not show configurations for which there were no cases (i.e., remainders), as these configurations were deleted before running a standard analysis.
Table 5
Revised Truth Table
| Rowa | gen_prof_ disclose_ screen b | referral_ barrier c | gen_directly_ contacts_pt d | difficulty_ contact_pt e | # cases fitting configure- ation | High-PF (outcome) | Raw consis- tency |
|---|---|---|---|---|---|---|---|
| A | 1 | 0 | 1 | 0 | 4 | 1 | 1f |
| B | 1 | 0 | 0 | 0 | 1 | 1 | 1f |
| C | 0 | 1 | 0 | 0 | 7 | 0 | 0g |
| D | 0 | 1 | 0 | 1 | 1 | 0 | 0g |
| E | 1 | 0 | 1 | 1 | 1 | 0 | 0g |
| F | 1 | 0 | 0 | 1 | 1 | 0 | 0g |
Open in a separate window
Notes: The revised truth table was created using fsQCA 2.0 software by adding a fourth condition to the original truth table, assigning outcome scores for each configuration, and deleting configurations with no cases (remainders).
aEach row represents a configuration of conditions. Although there are 24 (16) possible configurations, only those configurations for which there are cases are shown.
bGenetics professional discloses positive screening results (presence=1, absence=0)
cReferral is primary mechanism for patient to receive genetic testing (presence=1, absence=0)
dGenetic professional contacts patient directly to set up counseling and testing (presence=1, absence=0)
eDifficulty contacting patients after a positive tumor screen (presence=1, absence=0)
fThe consistency scores for rows A-B are 1 because all cases with these configurations have high patient follow-through (High-PF=1).
gThe consistency scores for rows C-F are 0 because none of the cases in those configurations have high patient follow-through (High-PF=0)
Step 7: Use software to generate solutions
Although the final truth table (Table 5) is quite revealing in terms of which contextual conditions are associated with High-PF, it can be helpful to have the computer software generate three solutions (complex, parsimonious, and intermediate), particularly when truth tables are large, multiple different configurations are associated with the same outcome, or fsQCA (in which outcomes and/or conditions are not dichotomized) is used instead of csQCA. As part of the current study, the researchers ran a “Standard Analysis” by clicking this option in the menu at the bottom of the window. The computer software used the Quine-McCluskey algorithm (which is based on Boolean simplification) to make multiple comparisons of case configurations and logically simplify the data (Ragin et. al, 2006). The idea behind this minimization procedure is that if two configurations differ in only one condition, yet produce the same outcome, then the condition that distinguishes the two configurations can be considered irrelevant to the outcome and removed to create a simpler expression.
The fsQCA2.0 software determines three solutions. The first is the complex solution, which is determined by the computer through minimizing only those configurations for which cases are available (i.e., remainders are not used to make simplifying assumptions). When there are multiple conditions or multiple configurations leading to the presence of the outcome, this solution may be so complex that it is not very useful. This is why the software generates a parsimonious and intermediate solution with input from the researchers.
To determine the most parsimonious solution, the software makes assumptions about what the outcome might be for the configurations that do not have cases (i.e., remainders) and uses these remainders to further simplify the expression (Ragin et. al, 2006). During the minimization process in the current study, a “prime implicant chart” appeared on the screen. A prime implicant chart appears when there are multiple ways of simplifying a solution (Ragin et. al, 2006). In order to obtain the most parsimonious solution, researchers must choose one prime implicant to cover each configuration in the chart. In the notation for prime implicants, the tilde (~) indicates the condition is absent. An asterisk (*) indicates Boolean “AND” (meaning that the conditions joined by * must both be present). The prime implicant chart in the current study showed that the configurations for the High-PF cases could be simplified in two different ways: (a) ~referral_barrier * ~difficulty_contact_pt; or (b) gen_prof_disclose_screen * ~difficulty_contact_pt. Despite an inability to make a compelling argument for choosing one prime implicant over the other, in the current study the researchers chose the first prime implicant so that the software would continue the analysis. In some instances (such as the current study) the prime implicant chosen to create the parsimonious solution does not influence the researchers’ final interpretation because they should reject the parsimonious solution if they cannot use logic and knowledge of the topic to substantiate all of the simplifying assumptions upon which the parsimonious solution is based (Ragin, 2004).
Even though all assumptions underlying the parsimonious solution cannot always be reasonably justified by the researchers, certain assumptions might be easy for the researchers to substantiate to create an intermediate solution; these are referred to as “easy counterfactuals” (Ragin, 2004). As part of the analytic process, the computer software automatically opens another window so that researchers can decide which simplifying assumptions are reasonable. In order for the software to generate the intermediate solution in the current study, the following logic-based assumptions were selected:
Absence of each barrier (i.e., ~difficulty_contact_pt and ~referral_barrier) will contribute to High-PF, but the presence of each barrier will not contribute to High-PF.
Involvement of a genetic professional in the disclosure of screening results (gen_prof_disclose_screen) and in directly contacting the patient to arrange genetic counseling and testing (gen_directly_contacts_pt) will contribute to High-PF, while lack of involvement by genetics professionals will not be associated with High-PF.
Step 8: Determine if the influence of conditions is symmetrical
The combinations of conditions associated with High-PF may differ from those associated with less successful outcomes. In the real world there are often more pathways that lead to the failure of a health program than there are leading to successful programs. Because QCA is not based on correlations, it does not assume that conditions will have a symmetrical influence. To illustrate this point, QCA steps 4–6 were repeated using the absence of High-PF (~High-PF) as the outcome. During this analytic process the latter of the following two prime implicants was chosen to be consistent with the initial analysis: (a) ~gen_prof_disclose_screen or (b) referral_barrier. Assumptions made to generate the intermediate solution were the inverse of the assumptions chosen for the first analysis (i.e., presence of barriers would contribute to ~High-PF, and absence of involvement by genetics professionals would contribute to ~High-PF).
Step 9: Evaluate consistency and coverage scores for the solutions
Consistency and coverage are interpreted differently when determining whether conditions are necessary versus when determining if they are sufficient. When performing sufficiency analyses, as in the current example, solution consistency should be close to 1 in order for researchers to conclude that the combination(s) of conditions in the solution is(are) almost always associated with the outcome of interest (Ragin, 2004). A solution coverage of 1 indicates that all cases with the outcome of interest are represented by at least one of the combinations of conditions in the solution. When there are multiple combinations of conditions within a solution, raw and unique coverage can be used by the researcher to assess the importance of each combination of conditions and the extent to which a case is covered by more than one combination of conditions.
Step 10: Interpret the resulting solutions and create causal models
Even if conditions are consistently associated with an outcome, it does not mean they cause the outcome. However, researchers can use solutions in conjunction with theory, conceptual frameworks, and detailed knowledge about the cases to develop causal models that help unpack potential mechanisms leading to the outcome (Ragin, 2004). In the current study the researchers used their substantive knowledge of UTS and conceptual framework (i.e., CFIR) to interpret the solutions and piece together key conditions to create tentative models that were intended to be modified as additional details about the cases were obtained.
QCA Results
Table 6 lists the complex, parsimonious, and intermediate solutions from the first csQCA analysis performed to determine conditions associated with High-PF. The parsimonious solution was rejected because all of the simplifying assumptions could not be substantiated. The model was based on the intermediate solution, which in this case, happened to be the same as the complex solution. This intermediate solution is interpreted as meaning that all of the following three conditions are together sufficient for High-PF: 1) a genetics professional discloses the results of positive tumor screening to patients; AND 2) a referral from another health care provider is not the primary mechanism for the patient to receive testing; AND 3) difficulty contacting patients is not a barrier. This combination of three conditions is unique only to the High-PF cases, which is why the consistency score is 1. The coverage score of 1 verifies that that this combination of three conditions characterizes (covers) all 5 cases that belong to the High-PF set.
Table 6
QCA Solutions, Consistency and Coverage
| Outcome | Solutions | Consis- tency | Raw Coverage | Unique Coverage | |
|---|---|---|---|---|---|
| High Patient Follow- through (High- PF) | Complex: | gen_prof_disclose_screen * ~referral_barrier * ~difficulty_contact_pt | 1.0 | 1.0 | 1.0 |
| Parsimonious: | ~referral_barrier * ~difficulty_contact_pt | 1.0 | 1.0 | 1.0 | |
| Intermediate:a | gen_prof_disclose_screen * ~referral_barrier * ~difficulty_contact_pt | 1.0 | 1.0 | 1.0 | |
| Overall consistency = 1.0 Overall coverage = 1.0 | |||||
| Absence of High Patient Follow- through (~High- PF) | Complex: | gen_prof_disclose_screen * ~referral_barrier* difficulty_contact_pt + ~gen_prof_disclose_screen * referral_barrier * ~gen_directly_contacts_pt | 1.0 1.0 | 0.2 0.8 | 0.2 0.8 |
| Parsimonious: | difficulty_contact_pt + referral_barrier | 1.0 1.0 | 0.3 0.8 | 0.2 0.7 | |
| Intermediate:b | difficulty_contact_pt + ~gen_prof_disclose_screen * referral_barrier * ~gen_directly_contacts_pt | 1.0 1.0 | 0.3 0.8 | 0.2 0.7 | |
| Overall consistency = 1.0 Overall coverage = 1.0 | |||||
Open in a separate window
Notes: A tilde (~) indicates the absence of the outcome or condition.
The intermediate solutions are bolded because they were determined to be the most theoretically sound and not overly simple or complex.
*The asterisk indicates Boolean multiplication (i.e. logical “AND”)
+The plus sign indicates Boolean addition (i.e. logical “OR”)
aThe following three conditions are sufficient for High-PF: 1) a genetics professional discloses the results of positive tumor screening; AND 2) obtaining a referral from another health care provider for the patient to receive genetic counseling and testing is not a barrier; AND 3) difficulty contacting patients is not a barrier.
bTwo distinct sets of conditions could both explain the absence of the outcome. Difficulty contacting patients who screen positive is sufficient but not necessary to prevent High-PF. Alternatively, the following three conditions are together sufficient but not necessary to prevent High-PF: 1) genetic professionals do not disclose positive screening; AND 2) genetic counselors do not contact patients directly to arrange genetic counseling and testing ; AND 3) the need for a health care provider to refer the patient for genetic counseling and testing is a barrier.
The bottom of Table 6 presents all three solutions for the absence of the outcome (i.e.,~High-PF). The three solutions were all different; thus, the causal model was based on the intermediate solution because it was not too simple, but made more logical sense than the complex solution. The intermediate solution for absence of High-PF (i.e., ~High-PF) revealed two distinct sets of conditions that were both associated with the absence of the outcome (Table 6). The intermediate solution can be interpreted as meaning that difficulty contacting patients who screen positive is sufficient but not necessary to prevent PF. Alternatively the following three conditions are together sufficient to prevent PF: genetic professionals do not disclose positive screening results, AND genetic counselors do not contact patients directly to arrange genetic counseling and testing, AND health care provider referral is the key mechanism for patients to receive genetic testing. The consistency of the intermediate solution was 1, indicating there were no contradictory cases. The coverage score of 1 indicates that all cases without high-patient follow-through (~High-PF) fit one or both of the combinations in the solution. The raw coverage for the first configuration (i.e., difficulty contacting patients) was 0.3, indicating that the presence of this barrier distinguished 3 of the 10 ~High-PF cases from the High-PF cases. The unique coverage for this configuration was lower (0.2) because 1 of the 3 institutions with difficulty contacting patients also shared the second combination of conditions that uniquely covered the other ~High-PF cases (Table 6).
Discussion
QCA was used in the current multiple-case study to formulate tentative causal models for explaining high variability in patient follow-through across institutions that have implemented a universal tumor screening program to identify patients with Lynch syndrome. QCA solutions provided key insights into how program implementation may contribute to program effectiveness. In other words, QCA identified conditions associated with relatively high levels of patient follow-through with genetic counseling and germline testing after a positive tumor screen. QCA was also useful in identifying additional questions to explore as part of the ongoing multiple-case study. For example, why did representatives from the five High-PF institutions report no difficulty contacting patients? In addition, what may prevent stakeholders at Low-PF or Medium-PF institutions from: (a) altering UTS procedures so that genetics professionals contact patients to disclose positive screening results; and (b) eliminating the need for a referral? Insights gained from QCA therefore informed the creation of semi-structured interview guides and follow-up surveys. Subsequently, follow-up data have been used, in conjunction with QCA results, to develop a more complete mechanistic model of how implementation conditions are likely influencing patient follow-through. This model has since been published and used as evidence to support changes in UTS procedures (Cragun et al., 2014). Furthermore, these changes have already led to improvement in patient follow-through at one institution based on personal communication with the institutional representative.
Despite their many uses, models created using QCA may be overly simplistic or incomplete. For example, findings from this case study do not preclude the possibility that other combinations of conditions could lead to high patient follow-through (High-PF) at institutions that were not studied. Indeed one advantage of QCA is that it can identify multiple different “recipes” for success. Subsequently, as more information is obtained and as additional institutions performing UTS are identified it is likely that the model will be expanded and modified further.
Several other criticisms that researchers level at QCA originate from what Morgan (2007) referred to as the “paradigm wars”. For instance, researchers who view QCA using a “quantitative” lens might consider performing multiple analyses on the same data to be problematic. However, multiple analyses are consistent with the iterative nature of QCA. Furthermore, determining which conditions are associated with both the presence and absence of the outcome is considered good practice by QCA researchers () as it can provide additional insights into the underlying mechanisms and can add to the credibility of the proposed models. Several other concerns that critics raise, such as the use of purposive sampling, are also unproductive from a pragmatic perspective. Nevertheless, several practical limitations are worth mentioning.
One limitation of QCA is the potential for measurement error and case misclassification. The current study was based on self-reported data from a single individual on behalf of their institution and may contain inaccuracies or bias. Furthermore, the use of natural breaks for set membership scoring may result in misclassification. For example, an open-ended survey response from the institutional representative of a Medium-PF institution revealed that this institution may instead belong in the High-PF set due to a unique difference in this institution’s protocol that may have led to an underestimation of patient follow-through. This institution had the highest patient follow-through among the Medium-PF set and was similar to High-PF institutions in several key ways. However, the representative reported difficulty contacting patients as a barrier. Given that difficulty contacting patients was sufficient to prevent High-PF under the current model, reclassification of this institution into the High-PF set would unveil a contradiction that would need to be resolved through modifications to the model based on additional information.
The measurement issue described above illustrates another limitation of csQCA, whereby conditions and outcomes must be dichotomized. In contrast, fsQCA overcomes this limitation by allowing the researcher to code the outcome and/or conditions on a calibrated scale from 0 to 1. This fuzzy-score represents the extent to which a case falls within the set rather than being fully in or fully out of a set (). The resulting advantages of fsQCA over csQCA include the ability to maintain variation and to more accurately represent social reality when outcomes and/or conditions are not truly dichotomous. Although bias and measurement error may remain a concern, using fsQCA may lead the researcher to assign a set membership score that is off by only a small degree rather than misclassifying it into the opposing set; and this is expected to have a smaller impact on the results. Unfortunately, the advantages of fsQCA also make it more complicated than csQCA.
There are other limitations to this study that do not result directly from the use of QCA. First, the use of aggregated institution-level data, rather than raw patient-level data, did not allow us to assess for associations between individual-level factors and patient follow-through. This is clearly a limitation of our model (Cragun et al., 2014). To tease out the relative influence of institution-level and individual-level factors on patient follow-through, it will be necessary to collect individual-level data in a systematic fashion from a larger number of institutions so that multi-level modeling can be employed. In fact, our anticipated sample size of fewer than 20 institutions would have provided insufficient statistical power for multi-level modeling even if individual-level data had been available, thus QCA was our best option for this study. Power limitations also prevented us from performing other types of inferential statistics including structural equation modeling.
Conclusion
Although rooted in a qualitative paradigm, QCA may appeal to researchers or journal editors that prefer “quantitative” methods because QCA: (a) takes a logical and mathematical approach; (b) can be used to analyze small, medium, and large data sets; (c) provides a tool for identifying causal complexity and equifinality; (d) allows the researcher to generate solutions (with the aid of a computer program); and (e) calculates measures to evaluate the merit of the solutions (i.e., solution consistency and coverage). Given that QCA confers several advantages over other techniques, one of the purposes of this article was to encourage its active diffusion across mixed methods research channels. This article has attempted to reduce perceived complexity of QCA by illustrating how to perform the simplest type of QCA (i.e., csQCA). The example presented demonstrates how QCA aids in systematically identifying and simplifying key conditions that are uniquely associated with an outcome of interest. Although the use of cross-sectional data inhibits the ability to demonstrate causation, QCA provides solutions that researchers can use to propose logical mechanisms by which key conditions may act together to facilitate or impede outcomes. The iterative nature of QCA allows the researcher to gain an in-depth understanding of multiple cases and modify “causal” models as additional information is discovered.
For those researchers who are new to QCA and/or mixed methods research, we recommend they review a broad array of prior studies that have used various techniques, regardless of whether the topic areas align with their own research interests. It is our opinion that examples from other researchers are a great way to learn and apply new techniques that can advance research across disciplines and topical areas.
QCA and other techniques that fuse qualitative and quantitative methods (Bazeley, 1999) provide an opportunity to help in bridging the gap that “paradigm wars” have created. Ultimately, we believe researchers should first consider how resources or other conditions may limit the type of data they can feasibly obtain to answer their research questions and then choose one or more of a wide variety of analytic tools based on how well-suited the tools are for answering their specific research questions. To that end, QCA is another tool that mixed methods researchers may find useful.
Supplementary Material
Supplemental File with Screenshots
Click here to view.(328K, doc)
References
- Barbour RS. Mixing qualitative methods: quality assurance or qualitative quagmire? Qualitative Health Research. 1998;8(3):352–361. [PubMed] [Google Scholar]
- Bazeley P. Editorial: Integrating data analyses in mixed methods research. Journal of Mixed Methods Research. 2009;3(3):203–207. [Google Scholar]
- Bazeley P, Kemp L. Mosaics, triangles, and DNA metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research. 2012;6(1):55–72. [Google Scholar]
- Beamer LC, Grant ML, Espenschied CR, Blazer KR, Hampel HL, Weitzel JN, MacDonald DJ. Reflex immunohistochemistry and microsatellite instability testing of colorectal tumors for Lynch syndrome among US cancer programs and follow-up of abnormal results. Journal of Clinical Oncology. 2012;30(10):1058–1063. [PMC free article] [PubMed] [Google Scholar]
- Bellcross CA, Bedrosian SR, Daniels E, Duquette D, Hampel H, Jasperson K, Khoury MJ. Implementing screening for Lynch syndrome among patients with newly diagnosed colorectal cancer: summary of a public health/clinical collaborative meeting. Genetics in Medicine. 2012;14(1):152–162. [PMC free article] [PubMed] [Google Scholar]
- Boeije H, Slagt M, van Wesel F. The contribution of mixed methods research to the field of childhood trauma: A narrative review focused on data integration. Journal of Mixed Methods Research. 2013;7(4):347–369. [Google Scholar]
- Cohen SA. Current Lynch syndrome tumor screening practices: A survey of genetic counselors. Journal of Genetic Counseling. 2014;23(1):38–47. [PubMed] [Google Scholar]
- Cragun D, Debate RD, Vadaparampil ST, Baldwin J, Hampel H, Pal T. Comparing universal Lynch syndrome tumor-screening programs to evaluate associations between implementation strategies and patient follow-through. Genetics in Medicine. 2014 [PMC free article] [PubMed] [Google Scholar]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science. 2009;4 50. [PMC free article] [PubMed] [Google Scholar]
- Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research & Policy. 2005;10(1):45–53. [PubMed] [Google Scholar]
- Gerring J. Case Study Research: Principles and Practices. Cambridge, UK: Cambridge University Press; 2007. [Google Scholar]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health. 1999;89(9):1322–1327. [PMC free article] [PubMed] [Google Scholar]
- Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Education Research. 2006;21(5):688–694. [PubMed] [Google Scholar]
- Hawley JD. [Review of the book Innovative comparative methods for policy analysis: Beyond the quantitative-qualitative divide, by B. Rihoux, & H. Grimm] Journal of Mixed Methods Research. 2007;1(4):390–392. [Google Scholar]
- Janz NK, Becker MH. The Health Belief Model: a decade later. Health Education Quarterly. 1984;11(1):1–47. [PubMed] [Google Scholar]
- Kahwati LC, Lewis MA, Kane H, Williams PA, Nerz P, Jones KR, Kinsinger LS. Best practices in the Veterans Health Administration's MOVE! weight management program. American Journal of Preventive Medicine. 2011;41(5):457–464. [PubMed] [Google Scholar]
- Krohwinkel A. A configurational approach to project delays: Evidence from a sequential mixed methods study. Journal of Mixed Methods Research. 2014 [Google Scholar]
- Lynch PM. How helpful is age at colorectal cancer onset in finding HNPCC? Diseases of the Colon & Rectum. 2011;54(5):515–517. [PubMed] [Google Scholar]
- Marx A, Dusa A. Crisp-set qualitative comparative analysis (csQCA): Contradictions and consistency benchmarks for model specification. Methodological Innovations Online. 2011;6(2):103–148. [Google Scholar]
- Miles MB, Huberman AM. Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: SAGE; 1994. [Google Scholar]
- Onwuegbuzie AJ, Bustamante RM, Nelson JA. Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research. 2010;4(1):56–78. [Google Scholar]
- Ragin CC. Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis. 2004 Retrieved from http://escholarship.org/uc/item/1z567tt. [Google Scholar]
- Ragin CC. The comparative method : moving beyond qualitative and quantitative strategies. Berkeley, CA: University of California Press; 1987. [Google Scholar]
- Rihoux B, De Meur G. Crisp-set qualitative comparative analysis (csQCA) In: Rihoux B, Ragin C, editors. Configurational comparative methods: qualitative comparative analysis (QCA) and related techniques. Thousand Oaks, CA: SAGE; 2009. pp. 33–68. [Google Scholar]
- Rihoux B, Marx A. QCA, 25 years after “the comparative method”: Mapping, challenges, and innovations—mini-symposium. Political Research Quarterly. 2013;66(1):167–235. [Google Scholar]
- Rihoux B, Ragin CC. Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Thousand Oaks, CA: SAGE; 2009. [Google Scholar]
- Rogers EM. Diffusion of Innovations, 5th Edition. New York, NY: Free Press; 2003. [Google Scholar]
- Sandelowski M, Voils CI, Leeman J, Crandell JL. Mapping the mixed methods–mixed research synthesis terrain. Journal of Mixed Methods Research. 2012;6(4):317–331. [PMC free article] [PubMed] [Google Scholar]
- Schneider CQ, Wagemann C. Standards of good practice in qualitative comparative analysis (QCA) and fuzzy-sets. Comparative Sociology. 2010;9(3):397–418. [Google Scholar]
- Shanahan MJ, Vaisey S, Erickson LD, Smolen A. Environmental contingencies and genetic propensities: social capital, educational continuation, and dopamine receptor gene DRD2. American Journal of Sociology. 2008;114(Suppl):S260–S286. [PubMed] [Google Scholar]
- South CD, Yearsley M, Martin E, Arnold M, Frankel W, Hampel H. Immunohistochemistry staining for the mismatch repair proteins in the clinical care of patients with colorectal cancer. Genetics in Medicine. 2009;11(11):812–817. [PubMed] [Google Scholar]
- Thoits PA. Stress, coping, and social support processes: Where are we? What next? Journal of Health and Social Behavior, Spec No, 53–79. 1995 [PubMed] [Google Scholar]
- Van Ness PH, Fried TR, Gill TM. Mixed methods for the interpretation of longitudinal gerontologic data: Insights from philosophical hermeneutics. Journal of Mixed Methods Research. 2011;5(4):293–308. [PMC free article] [PubMed] [Google Scholar]
- Weiner BJ, Jacobs SR, Minasian LM, Good MJ. Organizational designs for achieving high treatment trial enrollment: a fuzzy-set analysis of the community clinical oncology program. Journal of Oncology Practice. 2012;8(5):287–291. [PMC free article] [PubMed] [Google Scholar]
- Wolf F. Enlightened eclecticism or hazardous hotchpotch? Mixed methods and triangulation strategies in comparative public policy research. Journal of Mixed Methods Research. 2010;4(2):144–167. [Google Scholar]
- Yin RK. Case study research: Design and methods. 4th ed. Thousand Oaks, CA: SAGE; 2008. [Google Scholar]
