Coding comparison query
Compare coding done by two users to measure the 'inter-rater reliability' or degree of agreement for coding between the users.
A Coding Comparison query enables you to compare coding done by two users or two groups of users.
It provides ways of measuring 'inter-rater reliability' or the degree of agreement between the users: through the calculation of the percentage agreement and 'Kappa coefficient'.
- Percentage agreement is the number of units of agreement divided by the total units of measure within the data item, displayed as a percentage.
- Kappa coefficient is a statistical measure which takes into account the amount of agreement that could be expected to occur through chance.
Create a Coding Comparison query
- On the Query tab, in the Create group, click Coding Comparison.
- Next to Search in, choose the files that you want to compare.
- Next to Coded To, choose the selected codes that contain the coding you want to compare. Alternatively, you can choose all codes, codes in selected sets, or cases with selected classifications.
- For User Group A and User Group B, click to select the users whose coding you want to compare.
- Select whether you want the calculations to be based on character, sentence or paragraph.
- Click the Run Query button at the top of Detail View.
When the query has finished running, the query results are displayed in Detail View.
- Only text files (documents, PDFs, memos, and externals) and datasets are supported in NVivo for Mac. Other file types—for example, pictures —are not included.
- When selecting codes in the query criteria, you can select all descendent codes in a hierarchy by holding down the Option key when you select a code higher in the hierarchy.
- To save a coding comparison query, click Save Criteria and enter a name and description (optional). The file is saved under Queries / Query Criteria in the Navigation View.
- To see where there is agreement or disagreement in coding between the two user groups for a specific file or code, select the Show coding comparison content check box.
- Although you cannot save the query results within NVivo, you can export the results of a Coding Comparison query and then import them to other applications such as Excel. Export query results
Click the Undock icon (in the top right of the Detail View) to open the file in the Detail View into its own window, making more space to work. See Customize the work area
1 Overall Kappa coefficient for codes and files specified in the query. If the users are in complete agreement then the Kappa coefficient (K) = 1. If there is no agreement among the raters (other than what would be expected by chance) then the Kappa coefficient (K) ≤ 0.
2 The code that contains the coding that is being compared. You can expand the codes to see the files.
3 The file name.
4 The file size, which is based on number of characters, sentences, or paragraphs, depending on the query criteria.
5 The Kappa coefficient for each code or code/file combination.
6 These columns show percentage agreement:
- Agreement Column = sum of columns A and B and Not A and Not B
- A and B = the percentage of data item content coded to the selected code by both Project User Group A and Project User Group B
- Not A and Not B = the percentage of data item content coded by neither Project User Group A and Project User Group B
7 These columns show percentage disagreement:
- Disagreement Column = sums of columns A and Not B and B and Not A
- A and Not B = the percentage of data item content coded by Project User Group A and not coded by Project User Group B
- B and Not A = the percentage of data item content coded by Project User Group B and not coded by Project User Group A
8 You can display the results of a Coding Comparison query using either:
- Unweighted Values Files are treated equally (regardless of size) when calculating the overall results for each code.
- Weighted Values File size is taken into account when calculating the overall results for each code. File Size is the characters, sentences or paragraphs, depending on the options chosen in the query criteria. For example, a document with 1000 paragraphs would contribute more to the overall results than a document with only 20 paragraphs.
9 Select the Show coding comparison content check box if you want to see where there is agreement or disagreement in coding for a particular file or code.
You can dig deeper to see the content that was coded the same or differently between users or user groups.
1 Select a file or code in the query results and then select the Show coding comparison content check box to display the content pane. Alternatively, you can double click on a file or code in the query results to display the coding comparison content pane.
With the coding comparison content pane visible, click on another file or code to change the content that is displayed. To return to the tabular display of the query results, deselect the Show coding comparison content check box
2 Colored shading indicates the areas of the file that were coded.
The color of the shading identifies where there was agreement or disagreement. Content that is shaded:
- Green was coded by both groups. (Agreement)
- Yellow was coded by Group A only. (Disagreement)
- Blue was coded by Group B only. (Disagreement)
Passages that were not coded by either Group A or Group B are displayed as grey text and can help provide context when reviewing the results.
The shading may include more text than was actually coded—for example, if your query criteria calculations were based on sentence, then a partially coded sentence would be shaded in its entirety.
3 Adjust the width of the content pane—for example, make it bigger to fit more text in the pane or make it smaller to see more of the tabular query results.
4 Show or hide shading by selecting or deselecting the check boxes for a particular color. For example, if you want to focus your attention on the disagreement in coding, deselect the Both groups check box to hide the green shading.
5 You can quickly see where there is agreement or disagreement in the file or code using the green, yellow, and blue markers on the scroll bar. Click in the scroll bar (or drag) to navigate to a different area of the content—for example, click on a blue marker to move to a passage that was coded by Group B only.
How is the percentage agreement calculated?
NVivo calculates percentage agreement individually for each combination of code and file.
Percentage agreement is the percentage of the file’s content where the two users agree on whether the content may be coded to the code.
The calculations can be based on character, sentence or paragraph. Calculations based on character yield the most precise results.
|Calculations based on||Example|
If the file is a document with 1000 characters, where:
then the percentage agreement is calculated as (700 + 50) ÷ 1000 = 80%.
If the file is a document with 100 sentences, where:
then the percentage agreement is calculated as (80 + 5) ÷ 100 = 85%.
If the file is a document with 10 paragraphs, where:
then the percentage agreement is calculated as (4 + 5) ÷ 10 = 90%.
How is the Kappa coefficient calculated?
Cohen’s Kappa coefficient is a statistical measure of inter-rater reliability which many researchers regard as more useful than the percentage agreement figure, since it takes into account the amount of agreement that could be expected to occur through chance. For more information, refer to the Wikipedia article Cohen's kappa.
NVivo calculates the Kappa coefficient individually for each combination of code and file.
If the two users are in complete agreement about which content of the file should be coded to the code, then the Kappa coefficient is 1. If there is no agreement between the two users (other than what could be expected by chance), the Kappa coefficient is ≤ 0. A value between 0 and 1 indicates partial agreement.
The Kappa coefficient is calculated as follows. (Note that the units of measure used in this calculation depend on the file type. For example, for documents the units of measure are characters, while for audios and videos the units of measure are seconds of duration.)
- Calculate the expected frequency by which the agreement between users could have occurred by chance (ΣEF), by summing:
- The number of units of the file’s content coded to the code by user A, multiplied by the number of units coded to the code by user B, divided by the total number of units in the file (EF1)
- The number of units of the file’s content not coded to the code by user A, multiplied by the number of units not coded to the code by user B, divided by the total number of units in the file (EF2)
- Expected frequency (EF) of the agreement occurring by chance = EF1 + EF2
- Calculate the Kappa coefficient (K) as equal to:
- Total units of agreement between the two users (TA) minus the expected frequency (ΣEF) of the agreement occurring by chance, divided by the total units (TU) within the file minus the expected frequency (ΣEF) of the agreement occurring by chance: K = (TA – ΣEF) ÷ (TU – ΣEF)
- In the case where both users are in complete agreement as to how the file’s content should be coded to the code, then the value of Kappa will equal 1
For an example of how NVivo calculates Kappa coefficients, you can download the Coding Comparison Calculation Examples spreadsheet.
How should the value of Kappa be interpreted?
One approximate set of guidelines for interpreting the value of Kappa is:
0.40 – 0.75
Fair to good agreement
Why is my Kappa low?
Because the Kappa coefficient calculation takes into account the likelihood of the agreement between users occurring by chance, the value of Kappa can be low even though the percentage agreement is high.
For example, if most of a file has not been coded to the code by either user, but each user has coded completely different small sections of the file at the code, then the percentage agreement between the users will be high. But since this situation would be highly likely to occur by chance (i.e. if the two users had each coded a small section at random), the Kappa coefficient is low.
Conversely, if most of a file has not been coded to the code by either user, but each user has coded almost the same sections of the file to the code, then the percentage agreement between the users will again be high. But this situation would be highly unlikely to occur by chance, so the Kappa coefficient is also high.
These examples indicate why many researchers regard the Kappa coefficient as a more useful measure of inter-rater reliability than the percentage agreement figure.
What does a negative Kappa coefficient mean?
A Kappa coefficient less than or equal to zero indicates that there is no agreement between the two users (other than what could be expected by chance) on which content in the file may be coded to the code.
All my Kappa coefficients are 0 or 1. Is something wrong?
This most often indicates that one of the two users being compared has not coded any of the selected files to the selected codes.
In your Coding Comparison query results:
- If the columns “A and B (%)” and “A and Not B (%)” are both entirely full of zeros, then user A has not coded any of the files to the selected codes
- If the columns “A and B (%)” and “B and Not A (%)” are both entirely full of zeros, then user B has not coded any of the files to the selected codes