Queries > Coding comparison query

Coding comparison query

Compare coding done by two users to measure the 'inter-rater reliability' or degree of agreement for coding between the users.

A Coding Comparison query enables you to compare coding done by two users or two groups of users.

It provides ways of measuring 'inter-rater reliability' or the degree of agreement between the users: through the calculation of the percentage agreement and 'Kappa coefficient'.

  • Percentage agreement is the number of units of agreement divided by the total units of measure within the data item, displayed as a percentage.
  • Kappa coefficient is a statistical measure which takes into account the amount of agreement that could be expected to occur through chance.

Create a Coding Comparison query

  1. On the Explore tab, in the Query group, click Coding Comparison. The Coding Comparison Query dialog box opens.
  2. Select the users to include in user groups A and B in the Compare coding between box.
  3. In the At box, click Select to choose specific nodes. Click the arrow to see a menu of other options—for example, if you want to select nodes or cases in selected sets.
  4. In the Scope box, click Select to choose specific files.
  5. Select whether you want to show Kappa Coefficient and percentage agreement in the results.
  6. Click Run.

When the query has finished running, the query results are displayed in Detail View.

NOTE

  • To save the query, click the Add to Project button and enter the name and description (optional).
  • Although you cannot save the query results within NVivo, you can export the results of a Coding Comparison query and then import them to other applications such as Excel. Export query results
  • If you include an 'aggregate' node in the scope of the query, content coded at the node and all its direct children will be included in the query results. Aggregate nodes (gather all content in a parent node)
  • To view the content that has been coded, right-click on a selected row. You can select to Open Node or Open File to review the coding in detail.

Understand Coding Comparison query results

Coding comparison query results.

1 The node that contains the coding that is being compared.

2 The file name and file folder location.

3 The file size, which is calculated as follows:

  • Documents, datasets, memos and externals  = number of characters
  • PDFs = number of pages and number of characters
  • Media file = duration in minutes/seconds/10ths of a second
  • Picture = the total number of pixels expressed as height multiplied by width

4 The Kappa coefficient—this column is only available if you select Display Kappa Coefficient.  If the users are in complete agreement then the Kappa coefficient (K) = 1. If there is no agreement among the raters (other than what would be expected by chance) then the Kappa coefficient (K) ≤ 0.

5 The green columns show percentage agreement:

  • Agreement Column = sum of columns A and B and Not A and Not B
  • A and B = the percentage of data item content coded to the selected node by both Project User Group A and Project User Group B
  • Not A and Not B = the percentage of data item content coded by neither Project User Group A and Project User Group B

6 The red columns show percentage disagreement:

  • Disagreement Column = sums of columns A and Not B and B and Not A
  • A and Not B = the percentage of data item content coded by Project User Group A and not coded by Project User Group B
  • B and Not A = the percentage of data item content coded by Project User Group B and not coded by Project User Group A

How is the percentage agreement calculated?

NVivo calculates percentage agreement individually for each combination of node and file.

Percentage agreement is the percentage of the file’s content where the two users agree on whether the content may be coded at the node.

For example, if the file is a document with 1000 characters, where:

  • 50 of these characters have been coded by both users
  • 150 of these characters have been coded by only one of these users, and
  • the remaining 800 characters have not been coded by either user

then the percentage agreement is calculated as (800 + 50) ÷ 1000 = 85%.

For other types of files, different units are used instead of characters for this calculation. For example, for audios and videos the unit of measurement is seconds of duration, and for pictures the unit of measurement is image area in pixels.

How is the Kappa coefficient calculated?

Cohen’s Kappa coefficient is a statistical measure of inter-rater reliability which many researchers regard as more useful than the percentage agreement figure, since it takes into account the amount of agreement that could be expected to occur through chance. For more information, refer to the Wikipedia article Cohen's kappa.

NVivo calculates the Kappa coefficient individually for each combination of node and file.

If the two users are in complete agreement about which content of the file should be coded at the node, then the Kappa coefficient is 1. If there is no agreement between the two users (other than what could be expected by chance), the Kappa coefficient is ≤ 0. A value between 0 and 1 indicates partial agreement.

The Kappa coefficient is calculated as follows. (Note that the units of measure used in this calculation depend on the file type. For example, for documents the units of measure are characters, while for audios and videos the units of measure are seconds of duration.)

  • Calculate the expected frequency by which the agreement between users could have occurred by chance (ΣEF), by summing:
  • The number of units of the file’s content coded at the node by user A, multiplied by the number of units coded at the node by user B, divided by the total number of units in the file (EF1)
  • The number of units of the file’s content not coded at the node by user A, multiplied by the number of units not coded at the node by user B, divided by the total number of units in the file (EF2)
  • Expected frequency (EF) of the agreement occurring by chance =  EF1 + EF2
  • Calculate the Kappa coefficient (K) as equal to:
  • Total units of agreement between the two users (TA) minus the expected frequency (ΣEF) of the agreement occurring by chance, divided by the total units (TU) within the file minus the expected frequency (ΣEF) of the agreement occurring by chance:  K = (TA – ΣEF) ÷ (TU – ΣEF)
  • In the case where both users are in complete agreement as to how the file’s content should be coded at the node, then the value of Kappa will equal 1

For an example of how NVivo calculates Kappa coefficients, you can download the Coding Comparison Calculation Examples spreadsheet.

How should the value of Kappa be interpreted?

One approximate set of guidelines for interpreting the value of Kappa is:

Kappa value

Interpretation

Below 0.40

Poor agreement

0.40 – 0.75

Fair to good agreement

Over 0.75

Excellent agreement

Why is my Kappa low?

Because the Kappa coefficient calculation takes into account the likelihood of the agreement between users occurring by chance, the value of Kappa can be low even though the percentage agreement is high.

For example, if most of a file has not been coded at the node by either user, but each user has coded completely different small sections of the file at the node, then the percentage agreement between the users will be high. But since this situation would be highly likely to occur by chance (i.e. if the two users had each coded a small section at random), the Kappa coefficient is low.

Conversely, if most of a file has not been coded at the node by either user, but each user has coded almost the same sections of the file at the node, then the percentage agreement between the users will again be high. But this situation would be highly unlikely to occur by chance, so the Kappa coefficient is also high.

These examples indicate why many researchers regard the Kappa coefficient as a more useful measure of inter-rater reliability than the percentage agreement figure.

What does a negative Kappa coefficient mean?

A Kappa coefficient less than or equal to zero indicates that there is no agreement between the two users (other than what could be expected by chance) on which content in the file may be coded at the node.

All my Kappa coefficients are 0 or 1. Is something wrong?

This most often indicates that one of the two users being compared has not coded any of the selected files at the selected nodes.

In your Coding Comparison query results:

  • If the columns “A and B (%)” and “A and Not B (%)” are both entirely full of zeros, then user A has not coded any of the files at the selected nodes
  • If the columns “A and B (%)” and “B and Not A (%)” are both entirely full of zeros, then user B has not coded any of the files at the selected nodes

If the user’s work has been imported from another NVivo project, this may indicate that this user’s coding has not been imported correctly and needs to be re-imported.

When using the Import Project feature in NVivo to import coding:

  • Be sure to choose the option “Merge into existing item” for duplicate items
  • When selecting which data to import, do not choose the option “Selected (excluding content)”
  • If you choose the option “Selected (including content)”, be sure to select to import “Coding” in the Import Options dialog.

Calculate agreement across multiple files or nodes?

NVivo calculates the Kappa coefficient and percentage agreement individually for each combination of node and file.

However, the results of a Coding Comparison query can be exported from NVivo as a spreadsheet (using the Export List command) to allow you to perform further calculations.

If you wish to calculate an average Kappa coefficient or percentage agreement for a single node across multiple files, or for multiple files and nodes, you will need to consider how you want to weight the different files in the calculation. For example, do you want to treat each file equally, or do you want to give more weight to large files than small files?

For some examples of how average Kappa coefficients and percentage agreements can be calculated from Coding Comparison query results exported from NVivo, download the Coding Comparison Calculation Examples spreadsheet. This spreadsheet includes four examples (with the average Kappa coefficients and percentage agreements calculated using spreadsheet formulas):

  • Average figures for a single node across 3 files (weighting each file equally)
  • Average figures for a single node across 3 files (weighting each file according to its size)
  • Average figures for 5 nodes across 3 files (weighting each file equally)
  • Average figures for 5 nodes across 3 files (weighting each file according to its size)

If your project has different types of files (for example, documents and audios), you may need to give further consideration to how you want to weight these different files (since document size is measured in characters, while audio size is measured in seconds of duration).