Internet search analytics for shoulder arthroplasty: what questions are patients asking?

Article information

Clin Shoulder Elb. 2023;26(1):55-63

Publication date (electronic) : 2023 February 23

doi : https://doi.org/10.5397/cise.2022.01347

Johnathon R. McCormick, Matthew C. Kruchten, Nabil Mehta, Dhanur Damodar, Nolan S. Horner, Kyle D. Carey, Gregory P. Nicholson, Nikhil N. Verma, Grant E. Garrigues

Department of Orthopedic Surgery, Rush University Medical Center, Chicago, IL, USA

Correspondence to: Johnathon R. McCormick Department of Orthopedic Surgery, Rush University Medical Center, 1620 W Harrison St, Chicago, IL 60612, USA Tel: +1-312-563-6306, Fax: +1-312-942-2040, Email: Johnathon_mccormick@rush.edu

Received 2022 November 3; Revised 2022 December 12; Accepted 2023 January 2.

Abstract

INTRODUCTION

Prior to surgery, orthopedic surgeons routinely counsel their patients on the risks, benefits, expectations, and rehabilitation associated with any proposed procedure. This understanding is necessary for informed consent on the part of patients, and studies have shown that appropriately aligned patient expectations are positively correlated with higher postoperative satisfaction [1-5]. Thorough preoperative education has been found to reduce anxiety, postoperative pain, and length of hospital stays for patients [3,6]. Furthermore, numerous studies have shown that preoperative expectations are predictive of postoperative outcomes [5-11]. These findings highlight the importance of establishing appropriate expectations in patients, which includes addressing patients’ questions and concerns regarding all aspects of surgery. Despite the advantages of in-depth patient education prior to surgery, providers may not be able to answer all patient questions due to time constraints around office visits. Furthermore, patients may seek answers to their questions elsewhere if and when additional questions come to them outside of their appointments, or if they do not feel comfortable asking their physicians in the moment.

Orthopedic patients seek answers to their surgical questions from many sources other than doctors. Today, most orthopedic patients use the internet to ask medical questions [12]. A 2013 Pew Research report finds that 72% of internet users and 59% of all US adults had looked online for health information in the preceding year, while 60% of respondents had obtained information from friends and family. The same report finds that 77% of all online health inquiries started on a search engine [13]. Queries using Google Search have become one of the most ubiquitous ways people seek answers to their questions in the internet age. Google has developed an incredibly successful search algorithm to answer users’ inquiries and commonly related questions. Google Search results feature a tool called “People also ask,” which uses machine learning to aid searchers by suggesting related questions based on data gathered from other internet users and natural language processing. These suggested questions and answers are frequently engaged by users and are highlighted near the top of the results page [14]. Each suggested question is followed by a brief snippet of text attempting to briefly answer the question, along with a hyperlink to the webpage from which the text originated. Unfortunately, information promoted through the Google Search algorithm is not verified by experts in pertinent fields as accurate or of high academic standards. Therefore, the most common answers to internet searches that patients receive—including with regard to shoulder arthroplasty—are of unknown quality and accuracy. Despite the vastness of its data, the internet has been shown, across numerous studies, to be a source of potentially poor quality information in matters of health [15-19]. Specifically, assessment of the quality of information provided online about common otolaryngology procedures revealed that webpages returned by Google Searches contain only 50%–65% of critical information that patients must know prior to undergoing surgery, and that nearly 3% of webpages contain false or inaccurate information [20,21]. These studies highlight the need for surgeons to identify and answer any questions that patients may have while reducing the risk of patients gathering false information online. At the same time, the research demonstrates the need for surgeons and health care institutions to publish high quality online resources for patients to easily access instead of relying on incomplete and possibly inaccurate information from non-surgical third parties promoted through the Google Search algorithm.

The purpose of this study is to investigate what questions shoulder arthroplasty patients search online and to determine the types and quality of webpages provided to patients from the top results of each query. By understanding what questions patients are asking online regarding shoulder arthroplasty, surgeons can improve communication in preoperative education and prepare patients for their experiences and outcomes prior to shoulder arthroplasty. These steps will, in turn, increase patient satisfaction and improve clinical outcomes.

METHODS

This study was conducted using publicly available, de-identified data to protect individual privacy, and as such, did not require Institutional Review Board approval or informed consent. Therefore, no additional ethical clearance was necessary for this research.

The methods of our study were adapted for shoulder arthroplasty from the work of Shen et al. [22]. A total of nine search strings were conducted under three categories of total shoulder arthroplasty using Google Search. The queries were chosen to capture general total shoulder arthroplasty questions, as well as those pertaining to anatomic and reverse total shoulder arthroplasty. For the general group, the search terms were as follows: “shoulder replacement,” “total shoulder replacement,” and “total shoulder arthroplasty.” For anatomic total shoulder arthroplasty, the search strings were “anatomic shoulder replacement,” “anatomic total shoulder replacement,” and “anatomic total shoulder arthroplasty.” For reverse total shoulder arthroplasty, the terms were “reverse shoulder replacement,” “reverse total shoulder replacement,” and “reverse total shoulder arthroplasty.”

Search queries were performed on internet browser Google Chrome, version 90.0.4430.212. Searches were performed independently on May 13, 2021, by authors (MCK and KDC), with the search location set to the United States. To avoid the bias of personalized search results influenced by prior search history, searches were run on a newly installed application with no prior queries. The following steps were performed according to instructions found on support.google.com to ensure no personalized search bias was present [23]. Browsing history was confirmed to be cleared, and any previously installed Google Chrome application was uninstalled from the hard drive. The hard drive was subsequently searched for any remaining files containing Google Chrome data, which were deleted if encountered. The Google Chrome application was then reinstalled using another browser.

For each search query, the “People also ask” tab was expanded until approximately 130 suggested searches appeared on the page. Each suggested question was paired with a single hyperlink to a webpage. The suggested questions and webpage hyperlinks were collected using automated Google Chrome extension Scraper, version 1.7. Questions that were clearly unrelated to the topic of shoulder arthroplasty were removed from the dataset. The final dataset was ultimately limited to the first 100 relevant “People also ask” questions per search string.

Based on previous studies in internet health information, each question was categorized using the Rothwell classification into one of three themes—fact, policy, or value [22,24,25]. The questions were also subcategorized based on content into one of the following categories: specific activities, timeline of recovery, restrictions, technical details, cost, indications/management, risks/complications, pain, longevity, and evaluation of surgery [22]. Descriptions and examples of classifications for each question can be found in Table 1.

Table 1.

Rothwell classifications, subcategorizations, and examples

Each website hyperlink was visited, and the website information source was categorized as academic, commercial, government, journal, lawyer/law-related, medical information site, medical practice, non-medical media site, or single-surgeon personal. Descriptions and examples of classifications for each website can be found in Table 2.

Table 2.

Website categorizations and examples

Each website was scored on a four-point scale for quality of source according to the established Journal of the American Medical Association (JAMA) benchmark criteria, which include authorship, attribution, currency, and disclosure [15,17,19,22,26,27]. Requirements to receive points for each criterion can be found in Table 3. Question classification, website classification, and JAMA benchmark scores were compiled independently by two authors (MCK and KDC) following consensus for categorical definitions. Discrepancies were reviewed by a third author (JRM) as a tiebreaker to decide final categorization.

Table 3.

JAMA benchmark criteria

Cohen’s kappa coefficient was used to evaluate interobserver reliability of question classification and website classification. Pearson’s chi-square tests and Student t-tests were used to evaluate the results for significance. Statistical significance was set to P-values <0.05.

RESULTS

Nine hundred questions (comprising 300 questions for each of the three categories of total shoulder arthroplasty—general, anatomic, and reverse) were generated with nine hundred associated webpages. Cohen’s kappa coefficient for interrater reliability was found to be 0.94 for question categorization and 0.98 for website categorization.

Most questions were categorized as fact questions by the Rothwell classification (54.0%), followed by value (24.7%) and policy (21.3%) questions (Table 4, Fig. 1). Fact questions were most commonly about specific activities (31.3%), followed by timeline of recovery (24.7%), technical details (23.3%), restrictions (16.5%), and costs (4.3%) of shoulder arthroplasty. The majority of value questions were categorized as questions about pain (62.2%), followed by evaluation of surgery (34.7%) and longevity (3.2%). Policy questions were most commonly categorized as indications/management (77.1%), with the remainder being questions about risks/complications (22.9%) (Table 4, Fig. 2). The most popular suggested question was, “How long does it take to recover from shoulder replacement?” This question, or a variation on the wording, was the top suggested question in five of nine search queries and was in the top three questions of all nine strings of queries.

Table 4.

Question results breakdown by classification and subclassification

Fig. 1.

Rothwell classification for general, anatomic, and reverse shoulder arthroplasty. Questions for each of the three search groups as categorized by Rothwell classification: fact, policy, and value.

Fig. 2.

Subcategorization of questions for general, anatomic, and reverse shoulder arthroplasty. Breakdown of questions into subcategorizations based on topical content.

Of the 900 total websites analyzed in this study, most were categorized as medical practice (24.6%), followed by academic (23.2%), medical information (14.4%), commercial (9.3%), journal (8.9%), single-surgeon personal (8.0%), government (6.2%), non-medical media (4.9%), and lawyer/law-related (0.4%) sites (Table 5, Fig. 3).

Table 5.

Website results breakdown by classification

Fig. 3.

Breakdown of websites classified based on host webpages.

The mean JAMA score for all 900 webpages was 1.69. The websites with the highest mean JAMA scores were journal websites (mean, 3.91) in comparison to all other websites (P<0.001). The websites with the lowest mean JAMA scores were medical practice websites (mean, 0.89) in comparison to all other websites (P<0.001). Means for the remaining categories were as follows: academic=1.36, commercial=1.32, government=1.91, lawyer/law-related=1.25, medical information sites=2.65, non-medical media sites=1.93, and single-surgeon practice=1.07.

There was a significant relationship between search category (general, anatomic, or reverse) and Rothwell question classification (fact, policy, or value). Pertinent values were χ²=27.80, df=4, and P<0.001. Most questions categorized herein as policy questions by the Rothwell classification (n=192) were retrieved from searches in reverse total shoulder arthroplasty (n=94, 49.0%), in comparison to general (n=52, 27.1%) or anatomic (n=46, 24.0%) searches. There was also a significant relationship between search category and question subclassification with pertinent values of χ²=40.83, df=18, and P=0.002. Questions categorized as pertaining to indications/management (n=148) were more frequently related to reverse searches (n=74, 50%) than general (n=42, 28.4%) or anatomic (n=32, 21.6%) searches (Table 4, Fig. 2).

DISCUSSION

The results of this study highlight the questions most commonly asked online by patients undergoing shoulder arthroplasty, free from the biases present in traditional clinical surveys. The key findings of our study are (1) the most commonly asked question is, “How long does it take to recover from shoulder surgery?”, (2) by Rothwell classification, the most common questions are those classified as fact (54.0%), (3) the most common subcategories of questions are related to specific activities (16.9%), followed by questions about pain (15.3%), (4) the majority of answers to commonly asked questions are found on webpages hosted by medical practices (24.6%), academic institutions (23.2%), and medical information sites (14.4%), and (5) overall, the majority of webpages score poorly according to JAMA benchmark criteria for source and quality (1.69), with journals scoring the highest (3.91).

Knowing what questions shoulder arthroplasty patients are seeking answers to online is an important aspect of shoulder surgeons’ practices, allowing for improved counseling and education prior to surgery. Most questions asked by patients on Google were fact-based questions (54.0%), with the majority of these relating to specific activities and timeline of recovery. Overall, the most popular suggested question was, “How long does it take to recover from shoulder replacement?” This question, or slight iterations thereof, was the top suggested question in five of nine search queries and was in the top three questions of all nine strings of queries. The most common subcategorization of questions was specific activities, representing 16.9% of all questions. This finding highlights patients’ concerns about their ability to perform and continue performing specific activities postoperatively. The activities in question varied, with most being related to basic activities of daily living such as bathing and dressing, but also included more personalized and high-functioning activities such as bowling, golfing, and swimming. This illustrates the vast differences in patients’ preoperative goals. Generally speaking, while some patients are simply looking to perform activities of daily living without pain, others are hoping to return to higher-level functionality and recreational athletics. Physicians can answer a range of questions by providing patients with basic information regarding expectations for postoperative daily activities, as well as by asking each patient what activities are important to them to customize their preoperative education. It is critically important that surgeons determine which activities a patient wishes or expects to return to postoperatively in order to best educate patients and set realistic expectations for them. Numerous studies have shown that expectations regarding functional outcomes are correlated with reported postoperative outcomes [1,2,5,11]. Pain was the second most common subcategorization of questions, representing 15.3% of all questions. It comes as no surprise that patients are concerned about pain following surgery. Knowing that preoperative expectations are correlated with postoperative experiences, pain must also be discussed with the goal of setting realistic expectations for patients and explaining how pain will be managed, particularly given today’s wide array of analgesics used for shoulder arthroplasty [3,5-11]. Preoperative risk factors for severe postoperative pain should be identified so that appropriate management and referrals can be provided insofar as severe pain is associated with higher costs and longer hospitalization stays [28]. With pain protocols continuing to improve and shoulder arthroplasty now being performed as an outpatient procedure, it is important to make patients aware of what to expect in terms of multimodal analgesia and a plan for adequate pain control upon being discharged home [7]. Similarly, a prospective study in 4,709 patients undergoing total joint arthroplasty finds the best predictors of postoperative satisfaction in patients to be the meeting of patients’ preoperative expectations and satisfactory pain relief [29].

Medical information sites scored highly on JAMA benchmark criteria because they nearly always listed authorship and date of publication. Medical practice, single-surgeon practice, and academic webpages often did not cite authors, references, or date of publication, and thus scored poorly according to JAMA benchmark criteria. All categories besides journal sites scored poorly for disclosure criteria. The most linked webpages were medical practice and academic pages, comprising 24.6% and 23.2% of total searches, respectively. When combined with single-surgeon practice sites (8.0%), most webpages were hosted by organizations employing orthopedic surgeons. However, each of these three categories scored poorly on the JAMA benchmark criteria, earning scores of 0.89, 1.36, and 1.07, respectively. These informational webpages were likely authored or reviewed and/or editorialized by surgeons. Nevertheless, these sites rarely listed authors, publication dates, disclosures, or references. This highlights the need for improvement in informational publishing practices from orthopedic sources.

We observed that 14.4% of webpages belonged to medical information sites such as verywellhealth.com and healthline.com. These webpages host topical articles written by healthcare journalists and undergo editorial review by physicians. Such medical information sites score very well according to JAMA benchmark criteria, with an average score of 2.65, insofar as they nearly always list authors, publication dates, and references. Non-medical media sites such as news articles, social media, and Wikipedia articles accounted for only 4.9% of all webpages returned by Google Search. As social media is generally regarded as a poor source of information, it is encouraging to see that social media websites are rarely provided as educational resources by the Google algorithm. Commercial and lawyer/law-related webpages only accounted for 9.3% and 0.4% of total webpages, respectively. While it is reassuring that most webpages visited by patients are hosted by organizations employing physicians, the quality of these published resources would benefit from increased academic rigor and source transparency.

The internet has been shown by numerous studies to be a source of poor quality health information for patients with specific needs [15-19]. Oftentimes, answers to medical questions are complicated and nuanced, thus not amenable to explanation by non-professionals. We believe these questions are best answered in conversations between patients and surgeons, with adequate time for questions and clarification, or via educational materials produced and provided directly to patients by their surgeons. The results of this study allow shoulder arthroplasty surgeons to better anticipate and address their patients’ concerns, with potential improvement in patient comprehension, expectations, and ultimately, outcomes. In addition, this study highlights the need for orthopedic surgeons to improve the quality of information available to patients published by their practices and institutions.

Questions about reverse shoulder replacements were more likely to be policy questions and subcategorized as indications/management according to our findings. Patients want to know what makes them a candidate for reverse shoulder replacement, as well as when and why reverse shoulders are used instead of anatomic shoulder replacements.

A recent study published by Sudah et al. [30] extracts 300 questions from Google pertaining to shoulder arthroplasty. The results of their analysis support our findings above. Beyond the preliminary analysis of questions and webpages in that previous research, our current study analyzes a three-times larger dataset of questions and websites, utilizes objective JAMA benchmark criteria to assess source quality, and additionally examines the topics of reverse and anatomic shoulder arthroplasty.

A limitation of this study is that it relies on Google-generated questions that people also ask when seeking online information about shoulder arthroplasty. These questions do not come directly from known orthopedic patients, and it is impossible to confirm who is searching for these questions. It is assumed these are questions posed by patients. It is also impossible to know whether these questions are being searched preoperatively or postoperatively. While there may be some uncertainty in who is asking these questions, there is a notable advantage in seeing what questions are being asked by patients in the reassuring context of online anonymity. The “People also ask” function generates results based on search trends within individuals’ regions of search. In this study, the United States was set as the search country, thus results are less applicable to nationalities and patient populations outside of the United States. Similarly, searches were conducted in 2021; results will change over time as patient populations and available information changes. Another limitation of this study is that JAMA benchmark criteria are a better measure of transparency and publishing practices than a measure of the accuracy of content. This is a known limitation of the criteria and was acknowledged in its original publication [22,25]. Despite this limitation, JAMA criteria remain a valuable tool for objectively evaluating sources of online health information [15,22,27]. Future studies should examine what types of questions patients ask their surgeons in offices and clinics in comparison to the questions to which patients seek answers online.

The most common questions asked online by SA patients are related to performing specific postoperative activities and the timeline of recovery. The majority of information is provided by low quality, non-peer-reviewed websites. This study highlights the need for improvement in online resources available to patients undergoing SA. By understanding which questions patients have and the quality of online information provided in response, surgeons can improve preoperative education and postoperative outcomes for patients of all types of shoulder arthroplasty.

Article information Continued