Abstract
The chapter discusses the role of simple and lightweight Web-based systems in promoting a different approach to the externalization of practice-related knowledge within communities of professionals. This approach exploits common online questionnaire systems to collect the preferences of large numbers of domain experts to interesting paradigmatic work cases and proposes a statistically sound evaluation of these responses to evaluate the agreement reached within the community. We tested this approach in a case study that involved a large international medical association, that we chose as an example of a large and highly distributed community of expert professionals; in this study we challenged more than 1,000 surgeons about some border-line clinical cases where tacit notions based on life-long practice and situated experiences coexist (and sometimes clash) with scientific evidences drawn from the specialistic literature. We make the point that a sound evaluation of the collective agreement is a necessary precondition to use such lean Web-based tools in bottom-up knowledge elicitation initiatives. To this aim, existing measures of agreement and survey-related heuristics can be exploited to get a more precise picture of the “opinion of the many” in collective settings like communities of practice.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
http://www.mmslists.com/.
- 2.
http://www.nhs.net.
- 3.
Obviously a doctor does not choose a treatment by chance; but it is the doctors that are (should be) recruited by chance.
- 4.
Not to be confused with the Cohen Kappa, suitable for multi-case two-rater assessments.
- 5.
In this latter case we compared the values of Chi with the Kendall’s coefficient of concordance, which is a normalized score between 0 and 1 as the above mentioned Alpha and Kappa.
- 6.
These are, respectively, evidences of level II-3 and III or level C and D according to the evidence ranking developed by either the U.S. Preventive Services Task Force or the Oxford Centre for Evidence-based Medicine.
- 7.
The logistic regression model we obtained is represented by the function y(x) = a/b + ce-ax with \(a \simeq 1.04,\,b \simeq 0.004,\,c \simeq 0.06\) and it is indicated with P(t) in Fig. 6.2.
- 8.
This holds in the assumption that responses from the second turn are representative of the non-responses, and that people that can get convinced by a single reminder end up by exhibiting similar opinions to those that, conversely, are refractory to any reminder at all.
- 9.
y′ = abce cx/(b + e cx)2.
References
Aday, L.: Designing and Conducting Health Surveys. Jossey-Bass, San Francisco (1996)
Alstyne, M.V., Brynjolfsson, E.: Global village or cyber-balkans? Modeling and measuring the integration of electronic communities. Manag. Sci. 51(6), 851–868 (2005)
Atkins, D., Eccles, M., et al.: Systems for grading the quality of evidence and the strength of recommendations: critical appraisal of existing approaches. BMC Health. Serv. Res. 4(1), 38 (2004)
Barclay, S., Todda, C., Finlayb, I., Grande, G., Wyattc, P.: Not another questionnaire! Maximizing the response rate, predicting non-response and assessing non-response bias in postal questionnaire studies of gps. Fam. Pract. 19(1), 105–111 (2002)
Braithwaite, D., Emery, J., de Lusignana, S., Sutton, S.: Using the internet to conduct surveys of health professionals: a valid alternative? Fam. Pract. 20(5), 545–551 (2003)
Burns, K., Duffett, M., Kho, M., Meade, M.: A guide for the design and conduct of self-administered surveys of clinicians. Can. Med. Assoc. J. 179(3), 245–252 (2008)
Cabitza, F., Simone, C., Sarini, M.: Knowledge artifacts as bridges between theory and practice: the clinical pathway case. In: KMIA’08: Proceedings of the International Conference on Knowledge Management in Action. Held in Conjunction with the 20th IFIP World Computer Congress, 7 Jan 2008, Milan, Italy (2008)
Dillman, D.: Mail and Internet Surveys: The Tailored Design Method, 2nd ed. Wiley, New York (2000)
Dobrev, A., Haesner, M., Hüsing, T., Korte, W., Meyer, I.: Benchmarking ict use among general practitioners in Europe. Tech. rep., European Commission—Information Society and Media Directorate General (2008)
Feinstein, A.R., Cicchetti, D.V.: High agreement but low kappa: I. the problems of two paradoxes. J. Clin. Epidemiol. 43(6), 543–549 (1990)
Ferlie, E., Wood, M., Fitzgerald, L.: Some limits to evidence-based medicine: a case study from elective orthopaedics. Qual. Health. Care. 8, 99–107 (1999)
Gandhi, R., Tsvetkov, D., Davey, J., Mahomed, N.: Survival and clinical function of cemented and uncemented prostheses in total knee replacement. J. Bone. Joint. Surg. 91-B(7), 889–895 (2009)
Grava-Gubins, I., Scott, S.: Effects of various methodologic strategies. Can. Fam. Physician. 54(10), 1424–1430 (2008)
Gwet, K.: Handbook of Inter-rater Reliability. STATAXIS Publishing Company, Gaithersburg (2001)
Hamilton, M.B.: Online survey response rates and times. background and guidance for industry. Technical report, Ipathia, Inc./SuperSurvey (2009)
Handoll H.H.G., Hanchard N.C.A., Goodchild L.M., Feary J.: Conservative management following closed reduction of traumatic anterior dislocation of the shoulder. Cochrane. Database. Syst. Rev. 25(1) (2006). (Art. No.: CD004962, doi:10.1002/14651858.CD004962.pub2)
Hayes, A.: Statistical Methods for Communication Science. Lawrence Erlbaum Associates, Inc., Mahwah (2005)
Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)
Headrick, D.R.: When Information Came of Age: Technologies of Knowledge in the Age of Reason and Revolution, pp. 1700–1850. Oxford University Press, New York (2000)
Heide, L.: Punched-card Systems and the Early Information Explosion, pp. 1880–1945. Johns Hopkins University Press, Baltimore (2009)
Hiltz, S., Turoff, M.: Network Nation: Human Communication via Computer (Revised Edition). The MIT Press, Cambridge (1993)
Itoi, E., Hatakeyama, Y., et al.: Immobilization in external rotation after shoulder dislocation reduces the risk of recurrence. a randomized controlled trial. J. Bone. Joint. Surg. Am. 89(10), 2124–2131 (2007)
Jacobs W, Clement D.J., Wymenga A.A.B.: Retention versus sacrifice of the posterior cruciate ligament in total knee replacement for treatment of osteoarthritis and rheumatoid arthritis. Cochrane. Database. Syst. Rev. 19(4) (2005). (Art. No.: CD004803, doi:10.1002/14651858.CD004803.pub2)
Jones, J., Hunter, D.: Consensus methods for medical and health services research. BMJ. 311, 376–380 (1995)
Kane, B., Luz, S.: Achieving diagnosis by consensus. Comput. Support. Coop. Work 18(4), 357–392 (2009)
Kapetanios, E.: Quo vadis computer science: from turing to personal computer, personal content and collective intelligence. Data. Knowl. Eng. 67(2), 286–292 (2008)
Kellerman, S., Herold, J.: Physician response to surveys. A review of the literature. Am. J. Prev. Med. 20(1), 61–67 (2001)
Kongsved, S., Basnov, M., Holm-Christensen, K., Hjollund, N.: Response rate and completeness of questionnaires: a randomized study of internet versus paper-and-pencil versions. J. Med. Internet. Res. 9(3), e25 (2007)
Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. Sage, Thousand Oaks (2004)
Landis, J., Koch, G.: The measurement of observer agreement for categorical data. Biometrics. 33, 159–174 (1977)
Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M.: Internet versus mailed questionnaires: a controlled comparison. J. Med. Internet. Res. 6(4), e39 (2004)
Lipton, R., Liberman, J., Cutrer, F., Goadsby, P.: Treatment preferences and the selection of acute migraine medications: results from a population-based survey. J. Headache. Pain. 5(2), 123–130 (2004)
Marx, R.G., et al.: Beliefs and attitudes of members of the American academy of orthopaedic surgeons regarding the treatment of anterior cruciate ligament injury. Arthroscopy. 19(7), 762–770 (2003)
McColl, E., Jacoby, A., Thomas, L., Soutter, J.: Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health. Technol. Assess. 5(31), 1–256 (2001)
Mosca, L., et al.: National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 111, 499–510 (2005)
Parasuraman, A.: More on the prediction of mail survey response rates. J. Mark. Res. 19(2), 261–268 (1982)
Porter, T.M.: Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton University Press, Princeton (1996)
Potter, W.J., Levine-Donnerstein, D.: Rethinking validity and reliability in content analysis. J. Appl. Commun. Res. 27, 258–284 (1999)
Randelli, P., Cabitza, F., Arrigoni, P., Cabitza, P., Ragone, V.: Current practice in shoulder pathology: results of a web-based survey among a community of 1,084 orthopedic surgeons. Knee. Surg. Sports. Traumatol. Arthrosc. 20(5), 803–815 (2012). (doi:10.1007/s00167-011-1673-z)
Randolph, J.J.: Free-marginal multirater kappa: An alternative to fleiss’ fixed-marginal multirater kappa. In: Proceedings of the Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, 14–15 Oct (2005)
Scheibel, M., Kuke, A., Nikulka, C., et al.: How long should acute anterior dislocations of the shoulder be immobilized in external rotation? Am. J. Sports. Med. 37(7), 1309–1316 (2009)
Seguin, R., Godwin, M., MacDonald, S., McCall, M.: E-mail or snail mail? Randomized controlled trial on which works better for surveys. Can. Fam. Physician. 50(5), 414–419 (2004)
Sobal, J., Ferentz, K.: Assessing sample representativeness in surveys of physicians. Eval. Health. Prof. 13, 367–372 (1990)
Thorpe, C., Ryan, B., Burt, A., Stewart, M., Brown, J.: How to obtain excellent response rates when surveying physicians. Fam. Pract. 26(1), 65–68 (2009)
Van De Belt, T.H., Engelen, L.J., Berben, S.A., Schoonhoven, L.: Definition of health 2.0 and medicine 2.0: a systematic review. J. Med. Internet. Res. 12(2), e18 (2010)
VanGeest, J., Johnson, T., Welch, V.: Methodologies for improving response rates in surveys of physicians: a systematic review. Eval. Health. Prof. 30(4), 303–321 (2007)
Webster, B.S., Courtney, T.K., et al.: Physicians’ initial management of acute low back pain versus evidence-based guidelines. Influence of sciatica. J. Gen. Intern. Med. 20, 1132–1135 (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag London
About this paper
Cite this paper
Cabitza, F. (2012). Harvesting Collective Agreement in Community Oriented Surveys: The Medical Case. In: Dugdale, J., Masclet, C., Grasso, M., Boujut, JF., Hassanaly, P. (eds) From Research to Practice in the Design of Cooperative Systems: Results and Open Challenges. Springer, London. https://doi.org/10.1007/978-1-4471-4093-1_6
Download citation
DOI: https://doi.org/10.1007/978-1-4471-4093-1_6
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-4092-4
Online ISBN: 978-1-4471-4093-1
eBook Packages: Computer ScienceComputer Science (R0)