Development of search strategies for systematic reviews in health using ChatGPT: a critical analysis

Dear editor

To guide clinical decision-making systematic reviews, need to present transparent, reproducible, and standardized methods for identifying, synthesizing, and describing all scientific literature based on the previously developed central question. After structuring the research question, the strategies for searching for the information are in elaborate sequence. However, it is observed that one of the biggest challenges in designing systematic reviews and meta-analyses in the face of the ascending body of scientific literature is how to be assertive in searching for all scientific information.

ChatGPT is a type of chatbot, developed by free artificial intelligence (AI) model that uses the Generative Pre-trained Transformer language model capable of translating formal and extremely technical information into a clear and simple text in a few minutes [1]. This AI has been trained to access an enormity of data, texts, newspapers and scientific articles been evaluated for validity and execution in scientific research [2,3,4]. Wang and collaborators [5] tested Boolean operators in 2023, but the structuring a search strategy in different databases has not been critically evaluated yet. In this letter, we present the critical evaluation of ChatGPT’s ability regarding decoding core questions to search the entire literature in three used around the world databases used to guide researchers and methodologists.

To perform the analysis comparative search strategy, we used the record available by the PROSPERO platform (#CRD42023391396) aiming to answer the central question: “When does weight regain occur in obese individuals after bariatric surgery?” The PICOT components of the question were: population—Obese individuals with age major than 18 years; intervention—bariatric surgery; comparator—diet, drugs or placebo; outcomes—time of weight regains and study type—trials.

After structuring the central question, we asked ChatGPT to create the search strategy for the MEDLINE database (Additional file 1: Fig S1) and, after this guidance, we requested the development of search strategies that reflected the central question adapted for two other databases widely used in systematic reviews, LILACS and Embase, with the specific inclusion of descriptor bases (MeSH, DeCS, and Emtree) (Additional file 1: Fig S2). We present the manual search performed by a methodologist and validated by the librarian (Additional file 1: Fig S3).

Despite the quick return, we observed as to the constitution of the search strategies created by ChatGPT that this AI does not insert the synonymous terms (Entry Terms) and the jargon used in the clinical practice of the researchers. As for the structuring, we observed that the search strategies created by ChatGPT do not organize, in a correct manner, the groups of acronyms in the same search key. For example, obese people could not be related as an alternative to weight regain. We also observed that the ChatGPT inserted additional points to the objective of the intervention that was structured by the central question, that is, bariatric surgery and other surgeries are not objects of the central question. The insertion of the search deadline was another important point observed since if not justified, we cannot insert it. At last, we observed that ChatGPT did not insert a validated filter for the limitation of randomized clinical trials. The problems evaluated and the guidelines to work around them are available in Table 1.

Table 1 Problems and guidance about the methodological errors observed in the search by ChatGPT

In conclusion, we recommend caution for conducting information search strategies using ChatGPT exclusively. Despite being a simple-to-run tool and having ease in response, content and structuring problems are reported and searchers should be aware of these problems.