A graph-learning based model for automatic diagnosis of Sjögren’s syndrome on digital pathological images: a multicentre cohort study

Abstract

Background

Sjögren’s Syndrome (SS) is a rare chronic autoimmune disorder primarily affecting adult females, characterized by chronic inflammation and salivary and lacrimal gland dysfunction. It is often associated with systemic lupus erythematosus, rheumatoid arthritis and kidney disease, which can lead to increased mortality. Early diagnosis is critical, but traditional methods for diagnosing SS, mainly through histopathological evaluation of salivary gland tissue, have limitations.

Methods

The study used 100 labial gland biopsy, creating whole-slide images (WSIs) for analysis. The proposed model, named Cell-tissue-graph-based pathological image analysis model (CTG-PAM) and based on graph theory, characterizes single-cell feature, cell-cell feature, and cell-tissue feature. Building upon these features, CTG-PAM achieves cellular-level classification, enabling lymphocyte recognition. Furthermore, it leverages connected component analysis techniques in the cell graph structure to perform SS diagnosis based on lymphocyte counts.

Findings

CTG-PAM outperforms traditional deep learning methods in diagnosing SS. Its area under the receiver operating characteristic curve (AUC) is 1.0 for the internal validation dataset and 0.8035 for the external test dataset. This indicates high accuracy. The sensitivity of CTG-PAM for the external dataset is 98.21%, while the accuracy is 93.75%. In comparison, the sensitivity and accuracy for traditional deep learning methods (ResNet-50) are lower. The study also shows that CTG-PAM’s diagnostic accuracy is closer to skilled pathologists compared to beginners.

Interpretation

Our findings indicate that CTG-PAM is a reliable method for diagnosing SS. Additionally, CTG-PAM shows promise in enhancing the prognosis of SS patients and holds significant potential for the differential diagnosis of both non-neoplastic and neoplastic diseases. The AI model potentially extends its application to diagnosing immune cells in tumor microenvironments.

Introduction

Sjögren’s Syndrome (SS) is a rare chronic multisystem autoimmune disorder, affecting only 8% of the global population [1,2,3]. It primarily affects adult females and is characterized by persistent inflammatory features [4]. The hallmark feature of SS is impaired functionality of salivary and lacrimal glands, resulting in significant dryness of the oral mucosa [5]. The clinical manifestations of SS encompass oral and ocular dryness, resemble those observed in other conditions characterized by dry oral symptoms, including Sjögren’s syndrome-like disease. These manifestations pose a diagnostic challenge due to their similarity and necessitate lip gland biopsy for accurate identification. A strong correlation exists between the onset of SS and systemic lupus erythematosus (15-36%) and rheumatoid arthritis (20-32%) [6]. SS may potentially involve systemic organ damage and the development of B-cell lymphomas, affecting the respiratory, digestive, urinary, and reproductive system [7,8,9,10,11]. For instance, it can lead to kidney damage and in severe cases progress to kidney failure and mortality. In comparison to isolated rheumatoid arthritis, the presence of Sjögren’s Syndrome intensifies joint inflammation and joint damage, leading to heightened extra-articular symptoms such as pain, fatigue, anemia, and pulmonary function alterations. Additionally, it escalates the mortality rate among rheumatoid arthritis patients [12]. Therefore, the early diagnosis of SS holds paramount significance in the treatment of patients with rheumatoid arthritis or systemic lupus erythematosus.

Presently, SS is assessed through four distinct diagnostic criteria: the European Classification (PEC) criteria [13], the American European Consensus Group (AECG 2002) classification criteria [14], the American College of Rheumatology (ACR 2012) criteria [15], and the 2016 criteria jointly established by the American College of Rheumatology (ACR) and the European League Against Rheumatism (EULAR) [16]. Among these four sets of criteria, the histopathological evaluation of salivary gland tissue plays an irreplaceable role in the diagnosis of SS, particularly in the assessment of lymphocyte foci. However, lymphocyte counts in SS necessitate pathologists to engage in subjective assessment and manual enumeration within each high-power field of whole-slide imaging (WSI). Nonetheless, SS poses practical challenges due to its time-consuming nature, subjective judgment, potential counting inaccuracies, difficulty in diagnosing, and susceptibility to diagnostic errors.

Artificial Intelligence (AI) has witnessed substantial application in pathology image diagnosis [17, 18]. Deep learning facilitates computers in learning and synthesizing tasks akin to human comprehension. Multiple research reports have attested to the high performance and precision of deep learning algorithms in pathology image diagnostics, with diagnostic outcomes paralleling or surpassing human performance, encompassing ailments afflicting organs, for example, the kidneys, liver, brain, and lungs [19,20,21,22]. The majority of the work is accomplished using convolutional neural networks, such as ResNet-50 [23]. However, due to the large scale of pathological images, pixel-level analysis using convolutional neural networks (CNNs) poses a computational challenge and training difficulties.

In recent years, graph-based approaches for pathological imaging have gained increasing attention due to their interpretability and ability to effectively extract valuable information from images. The most common method for constructing a graph structure is to consider cells as nodes in the graph [24,25,26,27]. In addition to cell graphs, another common approach is to construct graph models using superpixels extracted from pathological images to utilize tissues information [28, 29]. However, current graph-based methods typically perform image classification on the entire or local images for diagnostic purposes, making it challenging to achieve fine-grained analysis. Moreover, these methods often struggle to effectively leverage the valuable information from both individual cells and the surrounding tissue. To this end, we propose a cell-centric, tissue-assisted graph-based pathological image cell analysis model called Cell-tissue-graph-based pathological analysis model (CTG-PAM). CTG-PAM focuses on cells as the primary entities for feature extraction and classification, while incorporating the contextual information of tissues to complement cell features.

This study propounds a model based on the 2016 ACR-EULAR diagnostic standards for SS. Leveraging graph-based learning and WSI, this model enables expeditious and accurate scoring and diagnosis of SS, thereby furnishing clinical and pathologists with objective, precise, reproducible, and standardized diagnostic capabilities.

Materials and methods

The baseline characteristics of the patients in the two datasets were summarized in Table S1 in supporting information.

Training and internal validation data sets

Thirty-six labial gland biopsy patients were obtained from the Department of Pathology, Guanghua School of Stomatology, Hospital of Stomatology, Sun Yat-sen University (SYSU) in China from January 2009 to June 2023 and were selected chronologically. Patients with hematoxylin-eosin staining (H&E) biopsy slides of low quality, including slides with drying effects, significant air bubbles, or broken glass, were excluded. An automatic digitalslide scanner (NanoZoomer S210, HAMAMATSU, Japan) at $\:\times\:$40 magnification was used to create WSIs. Two out of the 36 WSIs, one of which is diagnosed as SS and the other as non-SS, were employed to train the network for lymphocyte classification. The remaining 34 WSIs were utilized for validation. Four experienced academic pathologists graded and quantified the biopsy slides. Pathologist 1 was involved with annotating portions of lymphocytes and non-lymphocytes on the slides, where the two slides in the training set for model training and the remaining slides in internal validation data set for testing in the experiments. We categorized the doctors mentioned above into two groups based on their experience in pathology: the junior group (less than three years of pathology experience) and the senior group (more than three years of pathology experience). When the pathologist disagrees with the diagnosis, they will discuss and agree on their opinion. Finally, we compared their accuracy, sensitivity, and time against the automatic method.

External test data set

Sixty-four labial gland biopsy patients with SS were randomly selected from the First Affiliated Hospital of SYSU from January 2012 to January 2023. Independent preparations by each hospital were used for hematoxylin and eosin staining and WSI scanning. In total, 64 WSIs were obtained by using the method already mentioned. Pathologist 1 was involved with annotating portions of lymphocytes and non-lymphocytes on the slides in the external data set for testing in the experiments.

Immunohistochemical (IHC) staining

Following rehydration of paraffin-embedded tissue sections, antigen retrieval was achieved by exposing the sections to high heat and pressure in an EDTA buffer (pH 9.0) for 10 min. After blocking endogenous peroxidase activity and applying tissue blocking agents, the sections were incubated with primary antibodies against CD45 (1:100, clone: PD7/26 + 2B11, MXB, Fuzhou, China) at room temperature for 4 h. Biotin-conjugated secondary reagents were then applied for 1 h. The immunohistochemistry procedure was completed with DAB (3,3′-diaminobenzidine) staining and hematoxylin counterstaining.

Fig. 1

The pipeline of the proposed CTG-PAM for Automatic diagnostic method for SS automatic diagnostic

Cell-tissue-graph-based pathological analysis model

We proposed a graph-based method to automated diagnosis for SS, referred to as Cell-tissue-graph-based pathological analysis model (CTG-PAM). Our method is primarily divided into five stages: preprocessing, Computation of single-cell features and single-tissue features, Peripheral group feature evolution for cell-cell features and cell-tissue features, Cell classification for identifying lymphocytes, and Lymphocyte group detection for SS diagnosis (Fig. 1). Through preprocessing, we detect cells and tissue structures in pathological images. Feature evolution based on a graph model is a key aspect of CTG-PAM. After computing single-cell features, two graph models are used to derive two types of group features, which describe the characteristics of surrounding cells and the surrounding tissue structures, respectively. By combining single-cell features with these two types of group features, we perform cell classification to identify lymphocytes. Based on the recognition of lymphocytes, the model detects lymphocyte groups in the images, thereby achieving automatic diagnosis of SS. Below, we will introduce these five stages one by one.

Preprocessing

The primary focus of the preprocessing stage is the detection of cells and tissue structures within the regions of interest in pathological images. We initially utilize two models from prior research, namely “Stardist” [30] and “Segment Anything” [31] to detect cells and tissue structures respectively. Specifically, the detected cells and tissue structures are represented as two sets:

$$\:C=\left\{{c}_{1},\dots\:,\:{c}_{{n}_{{c}}}\right\},\:T=\left\{{t}_{1},\dots\:,{t}_{{n}_{{t}}}\right\},$$

where $\:{n}_{{c}}$ and $\:{n}_{{t}}$ are the numbers of detected cells and tissue structures, respectively.

Computation of single-cell features and single-tissue features

Cell morphology is the foundation and gold standard for determining cell origin, type, and characteristics [32, 33]. Work based on handcrafted morphological features has achieved excellent performance in cell classification tasks [34,35,36,37,38,39]. Consequently, we use eight morphological features commonly employed in image processing to characterize the inherent geometric properties of cells and tissue structures. The following are the eight features:

Feature 1: mean of grayscale value,

$$\:\begin{aligned}{x}_{\text{m}\text{e}\text{a}\text{n}}=&\text{T}\text{h}\text{e}\:\text{m}\text{e}\text{a}\text{n}\:\text{o}\text{f}\:\text{g}\text{r}\text{a}\text{y}\text{s}\text{c}\text{a}\text{l}\text{e}\:\text{v}\text{a}\text{l}\text{u}\text{e}\:\\&\text{o}\text{f}\:\text{p}\text{i}\text{x}\text{e}\text{l}\text{s}\:\text{i}\text{n}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.\end{aligned}$$

Feature 2: variance of grayscale value,

$$\:\begin{aligned}{x}_{\text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e}}=\,&\text{T}\text{h}\text{e}\:\text{v}\text{a}\text{r}\text{i}\text{a}\text{n}\text{c}\text{e}\:\text{o}\text{f}\:\text{g}\text{r}\text{a}\text{y}\text{s}\text{c}\text{a}\text{l}\text{e}\:\text{v}\text{a}\text{l}\text{u}\text{e}\:\\&\text{o}\text{f}\:\text{p}\text{i}\text{x}\text{e}\text{l}\text{s}\:\text{i}\text{n}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.\end{aligned}$$

Feature 3: area,

$$\:{x}_{\text{a}\text{r}\text{e}\text{a}}=\text{T}\text{o}\text{t}\text{a}\text{l}\:\text{n}\text{u}\text{m}\text{b}\text{e}\text{r}\:\text{o}\text{f}\:\text{p}\text{i}\text{x}\text{e}\text{l}\text{s}\:\text{i}\text{n}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.$$

Feature 4: perimeter,

$$\:\begin{aligned}{x}_{\text{p}\text{e}\text{r}\text{i}\text{m}\text{e}\text{t}\text{e}\text{r}}=&\text{T}\text{o}\text{t}\text{a}\text{l}\:\text{n}\text{u}\text{m}\text{b}\text{e}\text{r}\:\text{o}\text{f}\:\text{p}\text{i}\text{x}\text{e}\text{l}\text{s}\:\text{a}\text{l}\text{o}\text{n}\text{g}\:\\&\text{t}\text{h}\text{e}\:\text{b}\text{o}\text{u}\text{n}\text{d}\text{a}\text{r}\text{y}\:\text{o}\text{f}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.\end{aligned}$$

Feature 5: major axis length,

$$\:\begin{aligned}{x}_{\text{m}\text{a}\text{j}\text{o}\text{r}}=\,&\text{T}\text{h}\text{e}\:\text{l}\text{e}\text{n}\text{g}\text{t}\text{h}\:\text{o}\text{f}\:\text{t}\text{h}\text{e}\:\text{l}\text{o}\text{n}\text{g}\:\text{a}\text{x}\text{i}\text{s}\:\text{o}\text{f}\:\text{e}\text{l}\text{l}\text{i}\text{p}\text{s}\text{e}\:\\&\text{f}\text{i}\text{t}\text{t}\text{i}\text{n}\text{g}\:\text{t}\text{o}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.\end{aligned}$$

Feature 6: minor axis length,

$$\:\begin{aligned}{x}_{\text{m}\text{i}\text{n}\text{o}\text{r}}=\,&\text{T}\text{h}\text{e}\:\text{l}\text{e}\text{n}\text{g}\text{t}\text{h}\:\text{o}\text{f}\:\text{t}\text{h}\text{e}\:\text{s}\text{h}\text{o}\text{r}\text{t}\:\text{a}\text{x}\text{i}\text{s}\:\text{o}\text{f}\:\text{e}\text{l}\text{l}\text{i}\text{p}\text{s}\text{e}\:\\&\text{f}\text{i}\text{t}\text{t}\text{i}\text{n}\text{g}\:\text{t}\text{o}\:\text{t}\text{h}\text{e}\:\text{c}\text{e}\text{l}\text{l}\:\text{o}\text{r}\:\text{t}\text{i}\text{s}\text{s}\text{u}\text{e}\:\text{r}\text{e}\text{g}\text{i}\text{o}\text{n}.\end{aligned}$$

Feature 7: eccentricity,

$$\:{x}_{\text{e}\text{c}\text{c}\text{e}\text{n}\text{t}\text{r}\text{i}\text{c}\text{i}\text{t}\text{y}}\:=\:\frac{\sqrt{{\left(\frac{{x}_{\text{m}\text{a}\text{j}\text{o}\text{r}}}{2}\right)}^{2}+{\left(\frac{{x}_{\text{m}\text{i}\text{n}\text{o}\text{r}}}{2}\right)}^{2}}}{\frac{{x}_{\text{m}\text{a}\text{j}\text{o}\text{r}}}{2}}.$$

Feature 8: compactness,

$$\:{x}_{\text{c}\text{o}\text{m}\text{p}\text{a}\text{c}\text{t}\text{n}\text{e}\text{s}\text{s}}=\:\frac{{\left({x}_{\text{p}\text{e}\text{r}\text{i}\text{m}\text{e}\text{t}\text{e}\text{r}}\right)}^{2}}{{x}_{\text{a}\text{r}\text{e}\text{a}}}.$$

In these features, area [34,35,36,37,38,39], perimeter [34,35,36], major axis length [35, 36, 38, 39], minor axis length [35, 36, 38, 39], eccentricity [35,36,37] and compactness [37, 39] are used for cell classification in existing researches. The features other than compactness are calculated using the “scikit-image” library [40], and compactness is calculated directly from the above formula. The feature computation process is denoted by the operator $\:\mathcal{F}$, which can be expressed as:

$$\:{x}_{i}^{\left({c}\right)}={\mathcal{F}}\left({c}_{i}\right),\:{x}_{j}^{\left({t}\right)}={\mathcal{F}}\left({t}_{j}\right),\:i=1,\dots\:,{n}_{{c}},\:j=1,\dots\:,{n}_{{t}},$$

where,

$$\begin{aligned}\:x_i^{\left( c \right)} & = \left[ {\left( {{x_{{\text{mean}}}}} \right)_i^{\left( c \right)},\:\:\left( {{x_{{\text{variance}}}}} \right)_i^{\left( c \right)},\:\:\left( {{x_{{\text{area}}}}} \right)_i^{\left( c \right)},\:\:\left( {{x_{{\text{perimeter}}}}} \right)_i^{\left( c \right)},\:\:\left( {{x_{{\text{major}}}}} \right)_i^{\left( c \right)},} \right. \\ \\&\left. {\quad\quad\left( {{x_{{\text{minor}}}}} \right)_i^{(c)},\:\:\left( {{x_{{\text{eccentricity}}}}} \right)_i^{\left( c \right)},\:\left( {{x_{{\text{compactness}}}}} \right)_i^{(c)}} \right],\:\:x_j^{\left( t \right)} \\ & = \left[ {\left( {{x_{{\text{mean}}}}} \right)_j^{\left( t \right)},\:\:\left( {{x_{{\text{variance}}}}} \right)_j^{\left( t \right)},\:\:\left( {{x_{{\text{area}}}}} \right)_j^{\left( t \right)},\:\:\left( {{x_{{\text{perimeter}}}}} \right)_j^{\left( t \right)},} \right. \\ \\&\quad\quad \left. {\left( {{x_{{\text{major}}}}} \right)_j^{\left( t \right)},\:\:\left( {{x_{{\text{minor}}}}} \right)_j^{\left( t \right)},\:\:\left( {{x_{{\text{eccentricity}}}}} \right)_j^{\left( t \right)},\:\left( {{x_{{\text{compactness}}}}} \right)_j^{\left( t \right)}} \right] \in \:{\mathbb{R}^8} \\ \end{aligned}$$

These single-cell features and single-tissue features will serve as the foundation for the subsequent feature evolution that derives peripheral group features. It is worth noting that the selection of these eight fundamental features is not fixed. The number and types of fundamental features can be adjusted according to different application scenarios or diseases.

Peripheral group feature evolution for cell-cell features and cell-tissue features

When classifying each cell, doctors consider not only the characteristics of the cell itself but also its surrounding context. In existing studies [24,25,26,27], cell features are evolved to obtain more complex features for cell classification. We establish two graph models to derive two types of peripheral group features, including the features of surrounding cells, named cell-cell features, and surrounding tissue structures, named cell-tissue features, based on single-cell features and single-tissue features. In this process, each cell is sequentially assessed to determine if it is a lymphocyte, with the cell under evaluation referred to as the target cell. During this process, we establish two corresponding graph structures for each cell: the cell-cell graph (CCG) and the cell-tissue graph (CTG), where the CCG representing the relationships between the cell and its surrounding cells, and the CTG representing the relationships between the cell and its surrounding tissue structures. In both graphs, cells or tissue structures are nodes and edges are formed between those with relatively small distances, with predefined parameters governing the number of cells/tissue structures included and the distance threshold for edge creation. This approach of establishing edges based on the spatial distance between nodes is known as the k-nearest neighbor algorithm and is commonly used in cell graph models [26, 34, 41]. To approximate the actual distances between cells or between cells and tissue, we calculate the Euclidean distance between the center coordinates of two objects, subtracting the sum of their radii. The radii are estimated based on the area. Specifically, for any two cells or tissue structures $\:u,v\in\:C\cup\:T$, the distance is defined as:

$$\:D\left(u,v\right)={D}_{\text{c}\text{e}\text{n}\text{t}\text{e}\text{r}}\left(u,v\right)-(\sqrt{\frac{{S}_{u}}{\pi\:}}+\sqrt{\frac{{S}_{v}}{\pi\:}}),$$

where $\:{D}_{\:\text{c}\text{e}\text{n}\text{t}\text{e}\text{r}}\left(u,v\right)$ represents the Euclidean distance between the centers of $\:u$ and $\:v$, with $\:{S}_{u}$ and $\:{S}_{v}$ representing the areas of $\:u$ and $\:v$, respectively.

Below is the specific process for constructing CCG and CTG. Let’s denote the node feature matrix, degree matrix and adjacency matrix of the graph model as $\:X,\:D,\:A$. Specifically, denote the corresponding matrices for CCG and CTG as $\:{X}_{\text{C}\text{C}\text{G}},\:{D}_{\text{C}\text{C}\text{G}},\:{A}_{\text{C}\text{C}\text{G}}$ and $\:{X}_{\text{C}\text{T}\text{G}},\:{D}_{\text{C}\text{T}\text{G}},\:{A}_{\text{C}\text{T}\text{G}}.$ The maximum numbers of nodes for CCG and CTG are given as $\:{n}_{\text{C}\text{C}\text{G}}$ and $\:{n}_{\text{C}\text{T}\text{G}}.$ And the distance thresholds are $\:{d}_{\text{C}\text{C}\text{G}}$ and $\:{d}_{\text{C}\text{T}\text{G}}.$ The node feature matrices represent the set of node features, i.e., $\:{X}_{\text{C}\text{C}\text{G}}\in\:{\mathbb{R}}^{8\times\:{n}_{\text{C}\text{C}\text{G}}},\:{X}_{\text{C}\text{T}\text{G}}\in\:{\mathbb{R}}^{8\times\:{n}_{\text{C}\text{T}\text{G}}}.$ First, from $\:C$ select the nearest $\:{n}_{\text{C}\text{C}\text{G}}-1$ elements to the target cell $\:{c}_{*}$ based on their distance, forming a node set $\:{N}_{{c}_{*}}=\{{u}_{1},\dots\:,{u}_{{n}_{\text{C}\text{C}\text{G}}}\},$ which contain $\:{n}_{\text{C}\text{C}\text{G}}$ cells. The edge connections are determined based on the computed $\:{X}_{\text{C}\text{C}\text{G}}$ and the threshold $\:{d}_{\text{C}\text{C}\text{G}},$ i.e., for $\:{A}_{\text{C}\text{C}\text{G}}({n}_{\text{C}\text{C}\text{G}}\times\:{n}_{\text{C}\text{C}\text{G}})$:

$$\begin{aligned}\:{A}_{\text{C}\text{C}\text{G},i,j}=\left\{\begin{array}{l}1,\:\:\text{if}\:D\left({u}_{i},{u}_{j}\right)<{d}_{\text{C}\text{C}\text{G}}\\\:0,\:\:\text{other}.\end{array}\right.\end{aligned}$$

$\:{D}_{\text{C}\text{C}\text{G}}$ can be obtained from $\:{A}_{\text{C}\text{C}\text{G}}$ as a diagonal matrix:

$${D_{{\text{CCG}},i,j}} = \left\{ {\begin{array}{*{20}{l}}{\sum\limits_{k = 1}^{{n_{{\text{CCG}}}}} {{A_{{\text{CCG}},i,k}}} ,}&{i = j} \\ {0,}&{i \ne j.} \end{array}} \right.$$

Similarly, for CTG, based on the predefined parameters $\:{n}_{\text{C}\text{T}\text{G}}$ and $\:{d}_{\text{C}\text{T}\text{G}}$, $\:{{A}_{\text{C}\text{T}\text{G}},X}_{\text{C}\text{T}\text{G}},\:{D}_{\text{C}\text{T}\text{G}}$ can be obtained. This completes the construction of CCG and CTG for the target cell $\:{c}_{*}$.

After constructing the two graphs, we utilize the LightGCN model [42] for unsupervised graph feature evolution based on the two graphs. LightGCN is a widely used model for graph embedding feature computation, achieving significant success in graph-based fields such as recommender systems [43, 44]. Specifically, given the adjacency matrix $\:A(n\times\:n)$, the degree matrix $\:D(n\times\:n),$ and the initial feature matrix $\:X(n\times\:q)$ of the graph, where $\:n$ represents the number of nodes in the graph and $\:q$ represents the number of initial features. By performing calculations, we obtain the final extracted embedding features

$$\:E=\sum\:_{k=0}^{K}{\alpha\:}_{k}{E}^{\left(k\right)},$$

where

$$\:{E}^{(k+1)}=\left({D}^{-\frac{1}{2}}A{D}^{\frac{1}{2}}\right){E}^{\left(k\right)},$$

and $\:{E}^{\left(0\right)}=X$. We denote the LightGCN as $\:\mathcal{L}$. According to the results in the paper, we set $\:{\alpha\:}_{k}$ to be uniformly distributed as $\:\frac{1}{K+1}$, and fix $\:K=3$. Thus, we can simplify the LightGCN as

$$\:E=\mathcal{L}(A,D,X).$$

Specifically, applying LightGCN to both the cell-cell graph and the cell-tissue graph can be expressed as

$$\:\left\{\begin{array}{l}{E}_{\text{cell}}=\mathcal{L}\left({A}_{\text{cell}},{D}_{\text{cell}},{X}_{\text{cell}}\right),\\\:{E}_{\text{tissue}}=\mathcal{L}\left({A}_{\text{tissue}},{D}_{\text{tissue}},{X}_{\text{tissue}}\right),\end{array}\right.$$

where $\:{A}_{\text{cell}}$, $\:{D}_{\text{cell}}$ and $\:{X}_{\text{cell}}$ represent the adjacency matrix, the degree matrix and the feature matrix of the cell-cell graph, $\:{A}_{\text{tissue}}$, $\:{D}_{\text{tissue}}$ and $\:{X}_{\text{tissue}}$ represent the adjacency matrix, the degree matrix and the feature matrix of the cell-tissue graph. The results of feature evolution $\:{E}_{\text{cell}}$ and $\:{E}_{\text{tissue}}$ encompass the evolved features of all nodes in two graphs. We extract the cell-cell features and cell-tissue features corresponding to the target cell from $\:{E}_{\text{cell}}$ and $\:{E}_{\text{tissue}}$. These features represent the information of the surrounding cells and tissue structures of the target cell.

Cell classification for identifying lymphocytes

For any cell $\:{c}_{i}\in\:C$, after concatenating cell-cell feature vector $\:{x}_{i}^{\left(cc\right)}$ from CCG and cell-tissue feature vector $\:{x}_{i}^{\left(ct\right)}$ from CTG with the original single-cell feature vector$\:\:{x}_{i}^{\left({c}\right)}$, we employ a pre-trained classifier for cell classification. Specifically, we utilize a double-hidden-layer fully connected neural network, i.e. multilayer perceptron (MLP), as the classifier, and the classification process can be expressed as

$$\:{p}_{i}={W}_{3}\sigma\:\left({W}_{2}\sigma\:\left({W}_{1}{x}_{i}+{b}_{1}\right)+{b}_{2}\right),\:$$

where $\:{x}_{i}={x}_{i}^{\left({c}\right)}\oplus\:{x}_{i}^{\left({c}c\right)}\oplus\:{x}_{i}^{\left({c}t\right)},\:{x}_{i}\in\:{\mathbb{R}}^{24}\:\:$ is the feature vector of the cell $\:{c}_{i}$, $\:{W}_{1}$, $\:{W}_{2}$ and $\:{W}_{3}$ are weight matrices, $\:{b}_{1}$ and $\:{b}_{2}$ are bias vectors, $\:\sigma\:$ is the activation function, and $\:{p}_{i}\:$represents the probability distribution over the final class labels for the cell $\:{c}_{i}$. The class label with the highest probability in $\:{p}_{i}\:$ is chosen as the final prediction. We use cross-entropy as the loss function for network training and employ Adam optimizer to optimize the loss function. Additionally, we ensure consistency in the types of cells sampled in each training batch by using a random sampling method.

Lymphocyte group detection for SS diagnosis

In the lymphocyte group detection stage after identifying lymphocytes, we detect lymphocyte groups based on connected components in the lymphocyte graph structure, as a cluster containing more than 50 lymphocytes in a pathological image is diagnosed as SS according to clinical practice. First, we construct a graph with all lymphocytes in the pathological image as nodes, and we establish edges between two lymphocytes if their distance is less than a given threshold. Using a connected component detection algorithm based on breadth-first search [45], we extract all connected subgraphs from this graph and count the number of nodes in each connected subgraph. If there exists a connected subgraph with a node count exceeding 50, which serves as a diagnostic criterion for SS in clinical practice, then the patient corresponding to that pathological image is diagnosed with SS.

Hyperparameters setting

Here, we set hyperparameters in CTG-PAM. The maximum numbers of nodes and distance thresholds for edges in CCG and CTG, i.e., $\:{n}_{\text{C}\text{C}\text{G}}$, $\:{n}_{\text{C}\text{T}\text{G}}$, $\:{d}_{\text{C}\text{C}\text{G}}$, $\:{d}_{\text{C}\text{T}\text{G}}$, are set to 20, 2, 50, 5, respectively. In the fully connected neural network for cell classification, the width of hidden-layers is set to 50, the activation function is ReLU, and the training learning rate is 0.001. The subgraph edge establishment threshold for lymphocyte groups detection is set to 30. The experimental results include a sensitivity analysis of the hyperparameters in both CCG and CTG.

Statistical analysis

Accuracy, Sensitivity, and Specificity are used to evaluate diagnostic performance, and the formula is shown below. Fisher exact test is used to compare the diagnostic Accuracy, Sensitivity, and Specificity between model and pathologists of different qualifications via GraphPad Prism 8.0 software. When values of p < 0.05, were considered statistically significant (* p < 0.05; *** p < 0.001).

$$\:Accuracy=\frac{TP+TN}{TP+EP+TN+FN}$$

,

$$\:Sensitivity=\frac{TP}{TP+FN}$$

,

$$\:Specificity=\frac{TN}{TN+FP}.$$

Result

Lymphocyte identification performance of CTG-PAM

Accurate identification of lymphocytes is crucial in the SS diagnosis process. Therefore, we evaluate the lymphocyte identification performance in CTG-PAM. The experiment is divided into three parts. First, we validate that whether the eight single-cell features effectively represent the image information of individual cells by comparing the classification performance based on single-cell features and images. Second, we conduct sensitivity analysis for the hyperparameters in the CCG and the CTG, including the maximum number of nodes: $\:{n}_{\text{C}\text{C}\text{G}}$ and $\:{n}_{\text{C}\text{T}\text{G}}$, and the distance threshold parameters for establishing edges between nodes: $\:{d}_{\text{C}\text{C}\text{G}}$ and $\:{d}_{\text{C}\text{T}\text{G}}$. Third, we perform feature ablation experiments to show the effects of the CTG, which taking the cell-tissue information into the model for cell classification.

Validation the reasonableness of single-cell features

Eight fundamental features’ effectiveness in representing cell information determines the effectiveness of subsequently evolved group features, thereby impacting the performance of cell classification. Therefore, we validated the reasonableness of the single-cell features by verifying whether the eight single-cell features can effectively represent individual cell information through the following experiment. It is generally believed that, compared to designing feature extraction methods for images followed by classification, automatic feature extraction based on deep learning, represented by CNNs, can fully utilize image information [46]. Therefore, we conducted cell classification on the internal validation dataset using an MLP based on the eight single-cell features and a CNN based on images of each cell. We compared the performance of these two methods to verify whether the eight single-cell features sufficiently extracted the information from the images. The MLP’s hyperparameters were kept consistent with the previous setting. The CNN comprises two convolutional layers and two fully connected layers. The convolutional layers have 16 and 32 $\:3\times 3$ kernels, respectively, each followed by a ReLU activation function and a $\:2\times 2$ max pooling layer. The two fully connected layers have the same widths and activation functions as those in the MLP.

The experimental results showed that both methods exhibited high sensitivity and low specificity for lymphocyte identification. Specifically, the accuracy, sensitivity, and specificity of the MLP based on the eight single-cell features were 0.9192, 0.9556, and 0.4902, respectively, while the accuracy, sensitivity, and specificity of the CNN based on images were 0.8894, 0.9225, and 0.5004, respectively.

On the one hand, the similar performance of the feature-based method to the image-based method suggests that the eight single-cell features adequately extract the image information of individual cells. On the other hand, the experimental results with unsatisfactory specificity also highlighted the need to utilize group features in addition to single-cell features for cell classification.

Sensitivity analysis for the hyperparameters in CCG and CTG

Among the three types of features (single-cell features, cell-cell features and cell-tissue features) used for cell classification, the calculation of single-cell features does not involve hyperparameters that need adjustment. In contrast, the features of surrounding cells and surrounding tissues each include two hyperparameters: the maximum number of nodes ($\:{n}_{\text{C}\text{C}\text{G}}$ and $\:{n}_{\text{C}\text{T}\text{G}}$) and the distance threshold for establishing edges between nodes ($\:{d}_{\text{C}\text{C}\text{G}}$ and $\:{d}_{\text{C}\text{T}\text{G}}$). Through the adjustment and experimentation of these four parameters on the training dataset, we ultimately determined an optimal set of parameters. Specifically, the maximum number of nodes in the cell graph, $\:{n}_{\text{C}\text{C}\text{G}}$, is 20; the maximum number of nodes in the tissue graph, $\:{n}_{\text{C}\text{T}\text{G}}$, is 2; the distance threshold for edge connections in the cell graph, $\:{d}_{\text{C}\text{C}\text{G}}$, is 50; and the distance threshold for edge connections in the tissue graph, $\:{d}_{\text{C}\text{T}\text{G}}$, is 5. This set of parameters demonstrated the best cell classification performance in our experiments, with the accuracy of 0.9723, the sensitivity of 0.9741, and the specificity of 0.9518 in the internal validation dataset and the accuracy of 0.8024, the sensitivity of 0.7780, and the specificity of 0.9402 in the external dataset.

Using this parameter combination, we conducted a sensitivity analysis for each parameter, examining how changes in the parameters affected cell classification performance in Fig. 2. The results indicate that within a certain range of parameter variations, the cell classification performance does not undergo significant changes.

Evaluating the impact of cell-tissue features

We argue that cell-tissue information is important for accurate cell classification. To evaluate the impact of cell-tissue features provided by the CTG on classification performance, we designed the following experiment. We compared CTG-PAM with a model that does not use cell-tissue features. The model that uses only cell-cell features derived from CCG for cell classification is referred to as “CCG w/o CTG.” The experimental results of cell classification performance for both CCG w/o CTG and CTG-PAM are shown in Table 1. Based on the experimental results, we found that adding cell-tissue features enhanced the model’s cell classification performance. In the next section, we will further discuss the diagnostic effectiveness of these features combinations.

Fig. 2

Sensitivity analysis of key parameters affecting cell classification performance. (a) maximum node number in cell graph ($\:{n}_{\text{C}\text{C}\text{G}}$), (b) maximum node number in tissue graph ($\:{n}_{\text{C}\text{T}\text{G}}$), (c) edge connection distance threshold in cell graph ($\:{d}_{\text{C}\text{C}\text{G}}$), and (d) edge connection distance threshold in tissue graph ($\:{d}_{\text{C}\text{T}\text{G}}$). The red numbers on the horizontal axises represent the optimal parameters we selected

Table 1 Performance of cell classification for identifying lymphocytes with and without cell-tissue features in the internal validation data set and the external data set

Diagnostic performance of CTG-PAM and other models

We employed a model, referred to cell group detection model (CGDM), focused solely on cell group detection without lymphocyte differentiation, specifically only utilizing the stage “Lymphocyte group detection for SS diagnosis” of CTG-PAM, for comparative purposes in subsequent experiments. We completed the qualitative classification of both the internal validation and external test data sets by using the ResNet-50 [23], CGDM, CCG w/o CTG, and CTG-PAM. The areas under the receiver operating characteristic curve from the internal validation and external test data sets of CTG-PAM were 1.0 and 0.8035 respectively, which indicated that this was the best model (Fig. 3, A and B). The respective sensitivities for the internal and external data sets were 57.14% and 64.28% for ResNet-50, 100% and 98.21% for CGDM, 100% and 91.07% for CCG w/o CTG, and 100% and 98.21% for CTG-PAM, respectively. The corresponding accuracy were 50% and 56.25%, 94.11% and 87.5%, 97.06% and 87.5%, and 100% and 93.75%, individually. Our study showed that internal authentication was far superior to external authentication (Fig. 4; Tables 2 and 3). The CTG-PAM was visualized to identify the areas of tissue infiltration by lymphocytes, and both H&E and IHC confirmed that the model was able to learn from the characteristics of lymphocytes only (Fig. 5, A and B).

Fig. 3

Performance of the CTG-PAM. (a) and (b), The ROC and the AUC for diagnostic. (a), Comparison of the AUC/ROC for ResNet-50, CGDM, CCG w/o CTG, and CTG-PAM by using the internal test data set. CTG-PAM had an AUC (1.0) significantly greater than those of the other two models. (b), Comparison of the AUC/ROC for ResNet-50, CGDM, CCG w/o CTG, and CTG-PAM in the independent external test data set. CTG-PAM also provided the best AUC (0.8035) compared with those of the other ones

Table 2 Model and algorithm performance evaluation in internal validation data set

Fig. 4

Confusion matrix of the models’ diagnosis performance. (a)-(d), Confusion matrices of the internal validation data set for models of ResNet-50, CGDM, CCG w/o CTG, and CTG-PAM, respectively. (e)-(h), Confusion matrix of the independent external test data set for models of ResNet-50, CGDM, CCG w/o CTG, and CTG-PAM, respectively

Table 3 Model and algorithm performance evaluation in the external data set

Fig. 5

The models’ classification result of patch with a focus lymphocyte. (a), lymphocyte focus beside duct. (b), cell aggregates identify images, where distinct cell types are denoted by varying colors. The red color is employed to denote lymphocytes, whereas blue indicates duct cells. (c), a lymphocytes group, denoted in red color, detected by the connected subgraph detection technique on the constructed lymphocyte graphs. (d), CD45 confirmed that CTG-PAM identified the areas of lymphocyte focus

Achieving specialist levels of SS detection performance: interpretation accuracy comparison among beginner pathologists, skilled pathologists and CTG-PAM

CTG-PAM had exceeded skilled pathologists on prediction accuracy scores (Fig. 6, A and B). The average accuracy rates of the validation data set were 79.41%, 97.05% and 100% for beginner pathologists, skilled pathologists, and CTG-PAM, respectively. The average accuracy rates of the external data set were 84.37%, 90.62% and 93.75% for beginner pathologists, skilled pathologists, and CTG-PAM. On the other hand, beginner pathologists demonstrated markedly worse prediction accuracy scores. The AUC from the internal validation and external test data sets were 1 and 0.8035 for CTG-PAM, 0.8035 and 0.6428 for beginner pathologists, and 0.975 and 0.7857 for skilled pathologists. The average accuracy rates of the validation data set were 79.41%, 97.05%, and 100% for beginner pathologists, skilled pathologists, and CTG-PAM, respectively (Fig. 7, A and B). The average accuracy rates of the external data set were 84.37%, 90.62%, and 93.75% for beginner pathologists, skilled pathologists, and CTG-PAM. The different qualifications on the validation data set and external data set are shown in Table 4.

Fig. 6

Comparisons between CTG-PAM and pathologists of different qualifications. (a). The diagnostic results between model and pathologists of different qualifications in the internal validation data set CTG-PAM, beginner pathologists, and skilled pathologists were significant differences among these three groups on the average accuracy rates. (b). The diagnostic results between model and pathologists of different qualifications in the external data set CTG-PAM, beginner pathologists, and skilled patho differed among these three groups on the average accuracy rates

Fig. 7

Comparisons of ROC between CTG-PAM and two groups of human pathologist. (a), Comparison of the AUC/ROC for CTG-PAM, beginner pathologists, and skilled pathologists by using the internal test data set. CTG-PAM had an AUC (1.0) significantly greater than pathologists. (b), Comparison of the AUC/ROC for CTG-PAM, beginner pathologists, and skilled pathologists in the independent external test data set. CTG-PAM also provided the best AUC (0.8035) compared with those of the other ones

Table 4 Diagnostic results between model and pathologists of different qualifications

The CTG-PAM significantly reduces the diagnosis time compared to pathologists, as pathologists also need to invest time and effort in manually counting lymphocytes and lymphocytes foci under the microscope. We compared the time required for diagnosis between the CTG-PAM and pathologist judgment. The result showed that the CTG-PAM (22.22s, 95%CI, 21.49–22.96) took less time than beginner pathologist judgment (248.82s, 245.90-251.73) and skilled pathologist judgment (129.91s, 127.70-132.12) (Table S2, in supporting information).

Discussion

Through single-cell features, cell cluster features, and cell-tissue features, our model can effectively accomplish cell classification, facilitating flexible subsequent research and analysis. To our knowledge, this is the first study to identify and evaluate a predictive model for SS. The disease of SS has well-established and specific diagnostic criteria in pathological clinical settings, specifically, the presence of cell clusters with a lymphocyte count greater than 50, which confirms the diagnosis of SS [16]. Therefore, detecting cell clusters for automated diagnosis is a natural approach, and this can be achieved directly using connected component analysis in the cell graph. However, during the implementation of this method, it was found that its accuracy depends on the accurate detection of lymphocytes. Consequently, enabling the model to understand the definition of lymphocytes assumes paramount significance.

Therefore, a cell-level classification model that describes the characteristics of surrounding cells and the structure of surrounding tissues is necessary. Cell morphology is the foundation and gold standard for determining cell origin, type, and characteristics. Initially, our model utilized a relatively simple set of eight features, combined with the clinical expertise of pathologists in cell identification, to classify cells. However, the experimental results for lymphocyte identification based on single cells show that neither the extracted single-cell features nor the image-level classification of individual cells can provide satisfactory lymphocyte identification performance. This suggests that relying solely on single-cell features to identify cell types is insufficient. In fact, in practical clinical applications, the differential diagnosis of non-neoplastic and neoplastic diseases often requires consideration of the characteristics of surrounding cells and tissues from multiple perspectives. Regarding SS, the disease’s characteristic lies in the identification of lymphocytes surrounding the duct, which serves as SS’s diagnostic criterion and foundation. This holds great significance for the condition. In contrast, for specific tumors such as breast cancer, esophageal cancer, and thyroid cancer, the diagnostic criteria primarily emphasize the overall morphology, size, type, and structural arrangement of tumor cells. Based on this, we further improved our model and developed CTG-PAM, a model that extracts cell features from these three perspectives. Training the model with a small amount of labeled data allows it to distinguish lymphocytes from other cell types. Classifying cells before detecting cell clusters effectively enhances detection accuracy and reduces false positives. CTG-PAM demonstrates high accuracy and sensitivity in cell classification and diagnosis, showing great potential in cases where the number of tumor cells is low and pathologists face diagnostic challenges.

Furthermore, the emergence of precision medicine has significantly transformed cancer management. Clinicians are increasingly interested in the relationship between tumors and immune cells in the surrounding microenvironment. Given CTG-PAM’s precise ability to identify and count lymphocyte foci, this algorithm can extend its accuracy to the identification and counting of immune cells in the tumor microenvironment. This has the potential to significantly impact the treatment and prognosis of patients with breast cancer, gastric cancer, and squamous cell carcinoma. While the accuracy of cell classification may vary between different diseases, this does not affect diagnostic outcomes.

This study has several limitations. Despite the use of a total of 100 WSIs, the number of patients is small to draw a conclusive outcome regarding the application of deep learning in the pathology diagnosis of SS since three images from a single patient may have similarities. Further analysis with more patients and control subjects using H&E stained sections from multiple hospitals is needed. The control group consisted of patients who underwent lip gland biopsy but were not diagnosed with SS, without normal lip gland tissue or other parts such as the parotid gland. Distinguishing SS or sialadenitis of the parotid gland may be crucial, as the parotid gland is also one of the common sites for SS or sialadenitis. In this regard, the model may be effective in differentiating between sialadenitis and SS, as the lymphocytes in the sialadenitis of the parotid gland are scattered in the fibrous tissue, while SS is characterized by lymphocytic infiltration around the ducts.

Conclusion

Overall, the CTG-PAM shows great potential in improving the prognosis of patients with SS and has significant potential for diseases requiring multi-perspective differential diagnosis. It also demonstrates exceptional accuracy in cell classification and diagnosing SS, providing clinical pathologists with objective, precise, reproducible, and standardized diagnostic results.