Skip to main content
SpringerLink
Log in

On Agreement Indices for Nominal Data

  • Chapter
Sociometric Research

Abstract

In empirical research certain measurements are frequently performed. Measurement is the assigning of numbers to subjects to denote a property. These measurements take place at a level of measurement: nominal, ordinal, interval or higher. Especially in the social sciences, the assignment to a category is very often based on the judgement of an expert and not on an objective and clear criterium. Still it is desirable that the ‘scientists agree’, i.e. the assignment will not or scarcely differ if another rater had performed the assignment task. So it is of importance that there is agreement among the raters. There is agreement when raters assign an entity to the same category. Agreement is not only investigated within the social sciences, but in general also in the fields of biology and medicine.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 29.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • ANDERBERG, M. R., Cluster Analysis for Applications (New York: Academic Press, 1973).

    Google Scholar 

  • ARMITAGE, P., BLENDIS, L. M. and SMYLLIE, H. C., ‘The Measurement of Observer Disagreement in the Recording of Signs’, Journal of the Royal Statistical Society (A), 129 (1966) 98–109.

    Article  Google Scholar 

  • BARTKO, J. J. and CARPENTER, W. T., ‘Methods and Theory of Reliability’, Journal of Nervous and Mental Disease, 163 (1976) 307–17.

    Article  Google Scholar 

  • BENNETT, E. M., BLOMQUIST, R. L. and GOLDSTEIN, A. C, ‘Response Stability in Limited Response Questioning’, Public Opinion Quarterly, 18 (1954) 218–23.

    Article  Google Scholar 

  • BRENNAN, R. L. and LIGHT, R. J., ‘Measuring Agreement when Two Observers Classify People into Categories not Defined in Advance’, British Journal of Mathematical and Statistical Psychology, 27 (1974) 154–63.

    Article  Google Scholar 

  • CARTWRIGHT, D. S., ‘A Rapid Non-parametric Estimate of Multijudge Reliability’, Psychometrika, 21 (1956) 17–29.

    Article  Google Scholar 

  • CICCHETTI, D. V., ‘A New Measure of Agreement Between Rank-Ordered Variables’, American Psychological Association. Proceedings of the 80th Annual Convention, 7 (1972) 17–18.

    Google Scholar 

  • CLEMENT, P. G., ‘A Formula for Computing Interobserver Agreement’, Psychological Reports, 39 (1976) 257–8.

    Article  Google Scholar 

  • COHEN, J., ‘A Coefficient of Agreement for Nominal Scales’, Educational and Psychological Measurement, 20 (1960) 37–46.

    Article  Google Scholar 

  • CRITTENDEN, K. S. and HILL, R. J., ‘Coding Reliability and Validity of Interview Data’, American Sociological Review, 36 (1971) 1073–80.

    Article  Google Scholar 

  • DICE, L. R., ‘Measures of the Amount of Ecological Association between Species’, Ecology, 26 (1945) 297–302.

    Article  Google Scholar 

  • ELSTON, R. C., SCHROEDER, S. R. and ROJAHN, J., ‘Measures of Observer Agreement when Binomial Data are Collected in Free Operant Situations’, Journal of Behavioral Assessment, 4 (1982) 299–310.

    Article  Google Scholar 

  • FLANDERS, N. A., ‘Estimating Reliability’, in AMIDON, E. J. and HOUGH, J. B. (eds) Interaction Analysis: Theory, Research and Applications (Reading, Mass.: Addison-Wesley, 1967) 161–6.

    Google Scholar 

  • FLEISS, J. L., ‘Estimating the Accuracy of Dichotomous Judgements’, Psychometrika, 30 (1965) 469–79.

    Article  Google Scholar 

  • —— ’Measuring Nominal Scale Agreement Among many Raters’, Psychological Bulletin, 76 (1971) 378–82.

    Article  Google Scholar 

  • GALTUNG, J., ‘Measurement of Agreement’, in GALTUNG, J. (ed.), Papers on Methodology. Theory and Methods of Social Research, vol. II (Copenhagen: Christian Eijlers, 1979) 82–135.

    Google Scholar 

  • GARRETT, C. S., ‘Modification of the Scott Coefficient as an Observer Agreement Estimate for Marginal-form Observation Scale Data’, Journal of Experimental Education, 43 (1975) 4, 21–6.

    Article  Google Scholar 

  • GOODMAN, L. A. and KRUSKAL, W. H., ‘Measures of Association for Cross-classifications’, Journal of the American Statistical Association, 49 (1954) 732–64.

    Google Scholar 

  • HARRIS, F. C. and LAHEY, B. B., ‘A Method for Combining Occurrence and Non-occurrence Interobserver Agreement Scores’, Journal of Applied Behavior Analysis, 10 (1978) 523–7.

    Article  Google Scholar 

  • HAWKINS, R. P. and DOTSON, V. A., ‘Reliability Scores that delude: An Alice in Wonderland Trip Through the Misleading Characteristics of Interobserver Agreement Scores in Interval Recording’, in RAMP, E. and SEMB, G. (eds) Behavior Analysis: Areas of Research and Application (Englewood Cliffs, N.J.: Prentice-Hall, 1975) 359–76.

    Google Scholar 

  • HOLLEY, J. W. and GUILFORD, J. P., ‘A Note on the G-index of Agreement’, Educational and Psychological Measurement, 24 (1964) 749–53.

    Article  Google Scholar 

  • HOPKINS, B. L. and HERMANN, J. A., ‘Evaluating Interobserver Reliability of Interval Data’, Journal of Applied Behavior Analysis, 10 (1977) 121–6.

    Article  Google Scholar 

  • HOUSE, A. E., HOUSE, B. J. and CAMPBELL, M. B., ‘Measures of Interobserver Agreement: Calculation Formulas and Distribution Effects’, Journal of Behavioral Assessment, 3 (1981) 37–57.

    Article  Google Scholar 

  • HUBERT, L. J., ‘Nominal Scale Response Agreement as a Generalized Correlation’, British Journal of Mathematical and Statistical Psychology, 30 (1977) 98–103.

    Article  Google Scholar 

  • JANES, C. L., ‘Extension of the Random Error Coefficient of Agreement to N * N tables’, British Journal of Psychiatry, 134 (1979) 617–19.

    Article  Google Scholar 

  • JANSON, S. and VEGELIUS, J., ‘On the Generalization of the G-index and the Phi Coefficient to Nominal Scales’, Multivariate Behavioral Research, 14 (1979) 255–69.

    Article  Google Scholar 

  • —— and —— ‘The J-index as a Measure of Nominal Scale Response Agreement’, Applied Psychological Measurement, 6 (1982) 111–21.

    Article  Google Scholar 

  • KENT, R. N. and FOSTER, S. L., ‘Direct Observational Procedures: Methodological Issues in Naturalistic Settings’, in CIMINERO, A. R., CALHOUN, K. S. and ADAMS, H. E. (eds) Handbook of Behavioral Assessment (New York: Wiley, 1977) 279–328.

    Google Scholar 

  • KRIPPENDORFF, K., Content Analysis. An Introduction to its Methodology (Beverly Hills: Sage, 1981).

    Google Scholar 

  • LIGHT, R. J., ‘Issues in the Analysis of Categorical Data’, in TRAVERS, R. M. W. (ed.) Second Handbook of Research Teaching (Chicago: Rand-McNally, 1973) 318–81.

    Google Scholar 

  • LISCH, R. and KRIZ, J., Grundlagen und Modelle der Inhaltanalyse (Reinbek: Rororo, 1978).

    Google Scholar 

  • MAXWELL, A. E., ‘Coefficients of Agreement Between Observers and their Interpretation’, British Journal of Psychiatry, 130 (1977) 79–83.

    Article  Google Scholar 

  • MAXWELL, A. E. and PILLINER, A. E. G., ‘Deriving Coefficients of Reliability and Agreement for Ratings’, British Journal of Mathematical and Statistical Psychology, 21 (1968) 105–16.

    Article  Google Scholar 

  • MOKKEN, R. J., A Theory and Procedure of Scale Analysis: With Applications in Political Research (The Hague: Mouton, 1971).

    Book  Google Scholar 

  • MONTGOMMERY, A. C. and CRITTENDEN, K. S., ‘Improving Coding Reliability for Open-ended Questions’, Public Opinion Quarterly, 41 (1977) 235–43.

    Article  Google Scholar 

  • POPPING, R., ‘Traces of Agreement: On the Dot-product as a Coefficient of Agreement’, Quality and Quantity, 17 (1983a) 1–18.

    Article  Google Scholar 

  • POPPING, R., ‘Overeenstemmingsmaten voor Nominale Data’, unpublished PhD thesis, University of Groningen (1983b).

    Google Scholar 

  • POPPING, R., ‘AGREE, a package for Computing Nominal Scale Agreement’, Computational Statistics and Data Analysis, 2 (1984) 182–5.

    Article  Google Scholar 

  • RAE, D. W. and TAYLOR, M., The Analysis of Political Cleavages (New Haven: Yale U.P., 1970) 115–45.

    Google Scholar 

Download references

Authors
  1. R. Popping
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Amsterdam, Netherlands

    Willem E. Saris (Professor of Social Science Methods and Techniques) (Professor of Social Science Methods and Techniques)

  2. Sociometric Research Foundation, Amsterdam, Netherlands

    Irmtraud N. Gallhofer (Managing Director) (Managing Director)

Copyright information

© 1988 Willem E. Saris and Irmtraud N. Gallhofer

About this chapter

Cite this chapter

Popping, R. (1988). On Agreement Indices for Nominal Data. In: Saris, W.E., Gallhofer, I.N. (eds) Sociometric Research. Palgrave Macmillan, London. https://doi.org/10.1007/978-1-349-19051-5_6

Download citation

Publish with us

Policies and ethics

Search

Navigation