Leveraging Topic Specificity and Social Relationships for Expert Finding in Community Question Answering Platforms

A variety of methods have been proposed to address the EF task by leveraging similarities between current and previously answered questions. Dehghan and Ansell (Dehghan and Abin, 2019) introduced a model that clusters question terms based on semantic similarity and co-occurrence. Similarly, Liang et al. (Liang, 2019) proposed a semantic unsupervised generative adversarial network to assess similarities between word representations and experts. Moreover, Dehghan et al. (Dehghan et al., 2019) modeled user expertise by utilizing the tree structure of various domains, tags, and the temporal dimension of user response behavior. In a related vein, Zhang et al. (Zhang et al., 2020) considered the temporal dimension by proposing models with multi-shift and multi-resolution settings to capture temporal dynamics effectively. Fu et al. (Fu et al., 2020) introduced a novel approach using the Recurrent Memory Reasoning Network (RMRN), which employs different reasoning memory cells equipped with attention mechanisms to focus on various aspects of the question. Building on the concept of attention mechanisms, Peng et al. (Peng et al., 2022a, b; Liu et al., 2022b; Peng et al., 2023) implemented these mechanisms in multi-view, multi-grained, semi-supervised pre-trained, and pre-trained models with personalized fine-tuning for expert finding. Moreover, Qian et al. (Qian et al., 2022b) proposed a Multi-Hop Interactive Attention-based Classification Network (MIACN) that utilizes attention mechanisms to detect latent interactions among question subjects and bodies.

The second group of methods in EF relies on hand-crafted features to model the expertise of community members. Roy et al. (Roy et al., 2018) formalized a scoring function to capture various aspects of expertise, including the propensity to answer questions related to profile tags, the ability to provide accepted answers, and recent activity levels. Mumtaz et al. (Mumtaz et al., 2019) proposed a framework integrating activity, community, and time-aware features, with the temporal aspect also considered in (Fu, 2019) to track and incorporate the evolution of user roles, as well as in (Kundu et al., 2019). Similarly, Tondulkar et al. (Tondulkar et al., 2018) included features that favor experts providing high-quality answers to complex questions, utilizing a comprehensive set of features capturing user availability and knowledge, and applying Learning-to-Rank (LtR) methods for expert ranking. The quality and consistency of answers are further addressed by Faisal et al. (Faisal et al., 2019), who proposed a model based on an adaptation of the bibliometric g-index. LtR techniques are further utilized by Sorkhani et al. (Sorkhani et al., 2022), introducing 74 content-based and social-based features. An important aspect in addressing the EF task is considering the intimacy between the asker and answerer, studied by Fu et al. (Fu, 2020). As in text-based approaches, Tan et al. (Tang et al., 2020) employed attention mechanisms by proposing a method using Hierarchical Attentional Factorization Machines (HAFM). This approach combines factorization machines with hierarchical attention mechanisms to effectively model user-expert interactions and to highlight the importance of various features in expert recommendation tasks. The hierarchical attention mechanism helps in prioritizing relevant information at both the user and expert levels, enabling more accurate and personalized recommendations. Contrary to many studies focusing primarily on identifying the most senior platform users as experts, Roy et al. (Roy and Singh, 2024) aim to predict promising expert users at an early stage in community question answering sites.

Network-based methods integrate interactions and relationship information within networks. Kundu et al. (Kundu and Mandal, 2019) defined a framework that comprises a text-based component for assessing expert knowledge on specific topics, accompanied by a Competition Based Expertise Network (CBEN) (Aslay et al., 2013). This network uses link analysis techniques, such as AuthorRank (Liu et al., 2005) and Weighted HITS (Li et al., 2002). The CBEN approach is further utilized in (Kundu et al., 2020), where connections between two users are established if they have responded to the same question. Additionally, this study incorporates both intra-profile and inter-profile preferences of community users. In (Kundu et al., 2021b), the same authors propose a topic-sensitive hybrid expertise retrieval system (TSHER) that integrates assessments of knowledge, reputation, and authority. Le et al. (Le and Shah, 2018) measure the similarity between answerer and asker using social network techniques, employing Random Walks with Restart to calculate the proximity between two nodes.

Contrarily, Sun et al. (Sun et al., 2018a) propose a language-agnostic perspective of user expertise, constructing a competition graph with user and question nodes where edges represent increasing levels of question difficulty, thus depicting the hierarchical structure of questions. This concept of hierarchy is further explored in (Sun et al., 2019a). Addressing data sparsity, Sang et al. (Sang et al., 2019) introduced a Multi-modal Multi-view Semantic Embedding (MMSE) framework that learns semantic embeddings from both local and global perspectives, incorporating social structure information to enhance embedding quality. Ghasemi et al. (Ghasemi et al., 2021) focus on user embedding, proposing a joint model for text and node similarity.

Moreover, Li et al. (Li et al., 2019) represent the CQA platform as a Heterogeneous Information Network (HIN), aiming to learn embeddings for nodes representing question contents, raisers, and answerers. They employ an LSTM-equipped Metapath-based Embedding algorithm and a CNN scoring function to rank experts. The use of HIN and metapath-based algorithms is also noted in (Qian et al., 2022a). Liu et al. (Liu et al., 2022c) apply advanced semantic analysis and interest drift modeling to evaluate experts based on the relevance and depth of their contributions in specific domains. The concept of interest drift is similarly addressed by Krishna et al. (Krishna and Antulov-Fantulin, 2023), who propose a simple graph diffusion-based expert recommendation model that accounts for semantic and temporal information. Temporal dynamics in expertise assessment are further examined by Costa et al. in (Costa and Ortale, 2023b, a), who introduce temporally-discounted, tag-based models.

TUEF employs a tag clustering approach to identify the main topics discussed on the CQA platform based on past questions. The clustering approach, discussed in (Hoffa, 2019), utilizes the k-means algorithm (MacQueen et al., 1967) on a tag co-occurrence matrix $M$. Tags’ co-occurrence patterns are leveraged to group semantically related tags, as tags often associated with the same questions tend to be conceptually similar (Giannakidou et al., 2008). For each question $q\in Q$, the tags associated with $q$ are denoted as $tags(q)$. Let $T$ be the set of all unique tags across questions in $Q$, and let $F$ represent the set of the most frequent tags (top $\lambda$) that serve as clustering features. The co-occurrence matrix $M^{|T|\times|F|}$ is constructed as follows:

Here, $m_{i,j}$ indicates the number of questions in $Q$ where the $i_{th}$ tag and the $j_{th}$ feature co-occur. Finally, the matrix rows are normalized to express the relative frequency of tag co-occurrences with each feature. For a given tag $t_{i}$ and feature $f_{j}$, the normalized matrix $\hat{M}$ is defined as:

Clustering the tag co-occurrence matrix to devise macro topics is based on the observation that CQA users often pair broad, common tags with more specific ones to achieve greater precision in topic identification. By considering the co-occurrences of specific tags with broader tags, we can enhance the categorization of questions. Using the normalized matrix $\hat{M}$ and a specified value of k for the k-means algorithm, the clustering process yields k disjoint clusters of tags representing primary community domains. The optimal k value is determined based on silhouette maximization criteria (Rousseeuw, 1987), as described in Section 4. This comprehensive approach to topic identification and clustering ensures that TUEF effectively captures and represents the main topical areas discussed within the online community, facilitating a good understanding of community dynamics and interactions.

TUEF meticulously models user relationships within each layer by treating users $U$ as nodes in a multi-layer graph and establishing connections based on similar patterns of providing accepted answers within specific layers. Formally, the structure of TUEF is defined using a multi-layer graph $G=[L_{1},...,L_{k}]$, where each layer $L_{i}$ corresponds to a tag cluster. Each layer $L_{i}=(V_{i},E_{i})$ is an independent graph, with nodes $V_{i}$ representing users associated with the layer and edges $E_{i}$ denoting their relationships. To ensure consistent and accurate answers within a specific layer, TUEF includes in $V_{i}$ only those users who have provided a number of accepted answers equal to or exceeding a given $\epsilon$ percentile. Moreover, to characterize users’ knowledge in $V_{i}$, each user $u\in V_{i}$ is assigned a topic vector $b^{i}_{u}$. This vector represents the user’s contribution to various tags within the layer, normalized by their total contribution across all layers. Specifically, for a tag $t_{j}$ within $L_{i}$, the $j_{th}$ position of $b^{i}_{u}$ is calculated as:

Here, $accepted(L_{i},u,t_{j})$ denotes the number of accepted answers provided by user $u$ for questions labeled with tag $t_{j}$ in layer $L_{i}$. Finally, after computing the topic vectors for all the users in $V_{i}$, the cosine similarity is calculated between each pair of users within the layer. If the similarity between two users $u_{a}$ and $u_{b}$ exceeds a predefined threshold $\delta$, an edge $(u_{a},u_{b})$ weighted by the cosine similarity value is added to $E_{i}$.

It is crucial to emphasize that each question in TUEF is associated with tags assigned to specific layers within $G$. Consequently, a question can belong to multiple layers, and users answering these questions are represented across all associated layers. This multidimensional representation of user expertise and relationships across layers enables TUEF to capture a comprehensive view of users’ knowledge and interactions within the CQA platform. This approach ensures that users’ expertise and social connections transcend individual layers, reflecting the nature of community participation and expertise within the platform.

Experts in CQA platforms can be identified heuristically by considering the number of accepted answers they provided in the past. Another signal of user trust is given by the acceptance ratio $r_{u}$, i.e., the ratio between the number of accepted answers and the total number of answers a given user provides. As in (Dargahi Nobari et al., 2017), in TUEF, we adopt an expert selection criterion based precisely on these two measures: first, we select the set $C\subseteq U$ of candidate experts by considering all the users having a number of accepted answers greater or equal to a specified threshold $\beta$: next, we label as experts the set $E\subseteq C$ of users whose acceptance ratio $r_{u}$ is greater than the overall average $\overline{r}$:

Each layer within the multi-layer graph comprises nodes labeled as experts and non-experts. While the former are the primary targets of the Expert Selection process, the latter facilitates comprehensive exploration of the graph structure.

By implementing this process, we aim to identify users who consistently provide valuable and accepted solutions within the community, as they are likely to be proficient candidates for effectively addressing new questions posted on the platform.

TUEF adopts a comprehensive approach to identify candidate experts for a given question $q$ by leveraging content and social information. This process involves two key perspectives: the Network-based perspective, focusing on users’ centrality within the network, and the Content-based perspective, assessing users’ relevance to the newly posted query based on their past interactions. In the Network-based approach, TUEF utilizes Betweenness centrality (Freeman, 1977), which quantifies the centrality of nodes within a graph by evaluating their influence over information flow. Nodes with higher Betweenness centrality are Considered more central within the network. Each node $v_{j}^{i}$ in layer $L_{i}$ is assigned a Betweenness score $s_{j}^{i}$, which is used to sort nodes in descending order. On the other hand, the Content-based approach involves sorting layer nodes according to their similarity to the query $q$ based on questions previously answered by experts. This is achieved using Information Retrieval techniques and pre-built indexes, specifically the TextIndex and TagIndex, which index the text and associated tags of historical questions, respectively. When a new question is presented with its tags, the Content-based method employs a retrieval model (we specifically use BM25) independently on both indexes to retrieve a sorted list of relevant questions, along with information about the experts who provided accepted answers. The query for the TextIndex includes the concatenated question, title, and body, while for the TagIndex it consists of the concatenated question tags. Subsequently, the lists retrieved from each index are merged alternately, preserving the original order of elements.

This approach results in a query-specific ordering of nodes within each layer $L_{i}$, effectively leveraging network centrality and content relevance to identify and rank candidate experts tailored to the context of the new query $q$. The combination of these perspectives enhances the precision and relevance of expert selection within the TUEF framework.

In the candidate collection phase, the objective is to select a subset of experts $D_{i}\subseteq V_{i}$ from each layer $L_{i}$. This process balances the need for a sufficiently large candidate pool to ensure high recall (the probability of obtaining the correct answer) with the desire to include relevant experts for high precision. To achieve a balance between precision and recall, we estimate the probability $p$ of not receiving an answer from any user in $D_{i}$. Initially, $p$ is set to 1 when $D_{i}$ is empty and is incrementally reduced as experts are added to $D_{i}$. Each expert $u$ added to $D_{i}$ contributes to lowering $p$ based on their acceptance ratio $\mu_{u}$, defined as the ratio of accepted answers to total answers provided by the user. Furthermore, to refine the modeling of topic-based expertise, we smooth $\mu_{u}$ by considering the user’s activity within the specific layer: the smoothing factor is computed by adjusting $\mu_{u}$ based on the ratio of expert answers within the layer to the maximum number of answers provided by any user within that layer. The probability $p$ is updated iteratively using the formula:

This iterative process continues until $p$ becomes less than or equal to a predefined threshold $\alpha$. Once the threshold $\alpha$ is met, the inclusion of new candidates to $D_{i}$ is halted, and the exploratory phase of the expert selection process begins. This strategic approach ensures that the candidate pool $D_{i}$ comprises experts likely to provide valuable and accurate answers to the specific question while representing good starting points for MLG exploration.

In the Exploratory phase, TUEF explores the graph structure to identify additional candidate experts that may not have been identified with the previously described phases, specifically exploiting the implicit relationships between users, aiming to enhance recall. The starting point for this exploration is the set of experts $D_{i}$ from each layer $L_{i}$. Specifically, for each node $v_{i}\in D_{i}$, TUEF initiates a series of fixed-length Random Walks. At each step of the Random Walk, the next node to visit is selected randomly based on a probability distribution $d$, computed considering the neighbors of the current node $v_{i}$ and their link weights, representing the similarities with neighboring nodes.

More formally, let $\mathcal{N}i={v{i1},\ldots,v_{iN_{i}}}$ denote the neighbors of expert node $v_{i}$, where $N_{i}=|\mathcal{N}_{i}|$ is the total number of neighbors of $v_{i}$. The probability $d_{j}$ of visiting a neighbor node $v_{ij}$ is calculated as:

Here, $w_{ij}$ represents the weight of the edge between node $v_{i}$ and its neighbor $v_{ij}$. Nodes with stronger connections to the current node $v_{i}$ are more likely to be selected during the Random Walks. Whenever an expert node is encountered during the exploration, it is added to the current set of candidate experts if not already included.

This exploratory process is carried out independently for each layer to which the question belongs, ensuring a comprehensive exploration of both Network-based and Content-based perspectives within each layer. Combining Random Walks with tailored probability distributions enriches the candidate pool $D_{i}$ with potentially neglected experts, enhancing the overall recall and effectiveness of the expert selection process in the TUEF framework.

We pre-processed the data to ensure a good set of questions and answers. Moreover, the different steps of TUEF require setting some parameters detailed below. We applied established data cleaning procedures, as outlined in (Mumtaz et al., 2019), to ensure the quality of our dataset. We opted for an eight-year timespan for most communities, excluding StackOverflow and Mathematics. The decision to use a shorter timeframe for these two communities arises from their high daily question volumes, ensuring a more balanced representation across all communities. This strategy aims to maintain a roughly equivalent dataset size for each community. We removed questions and answers without a specified question’s ID and OwnerUserID (i.e., the ID of the user who posted the question). Subsequently, we retained questions with an AcceptedAnswerID (i.e., closed questions) and answers with a valid ParentID (i.e., the respective question ID). Questions where the asker and the best answerer were the same user were also excluded. We split the dataset, allocating 80% for training/validation and the remaining 20% for the test set. Notably, we maintained the chronological order of the questions to preserve temporal integrity. Comprehensive statistics for the resulting datasets are provided in Table 1.

To identify the primary topics discussed in the community, we employed the clustering technique outlined in Section 3.1, focusing on the tags associated with questions in the training set. Considering the top $\lambda=10$ most frequent tags as features, we determined the optimal number of clusters within the range $K=[2,10]$ that maximizes the Silhouette score. Each cluster represents a specific macro-area, corresponding to a layer in the MLG, which captures interactions among users who have provided a number of accepted answers equal to or exceeding the $\epsilon=90th$ percentile. Users are represented by their topic vectors used to compute the pair-wise cosine similarities (see Section 3.1). When modeling relationships, we retained edges with a similarity equal to or greater than $\delta=0.5$.

We identify as experts the users with a number of accepted answers greater than or equal to the $\omega=95th$ percentile of the distribution, following the procedure detailed in Section 3.2. The minimum number of accepted answers and the total number of users labeled as experts for each dataset are outlined in Table 1 under the columns MinAccAns and Experts, respectively. TagIndex and TextIndex return the 1,000 most similar past questions for each query in the expert selection process, respectively. For both Network and Content methods, after node sorting, the collection phase identifies the initial set of experts $D$ that reaches a probability threshold of $p=0.001$, representing the probability of not receiving an answer. Subsequently, starting for each selected expert in $D$, we conduct 5 Random Walks of, at most, 10 steps.

We utilize the LightGBM³³3https://lightgbm.readthedocs.io/en/stable/ (Ke et al., 2017) implementation of LambdaMART (Burges, 2010) to learn the TUEF ranking model. The LtR training set is constructed using queries from the training dataset where the accepted answerer is labeled as an expert, including $0<\zeta\leq 50,000$ queries ordered from the most recent to the oldest. It is important to note that this is a TUEF parameter: if the dataset contains more than 50,000 queries answered by experts, TUEF will take the latest 50,000; otherwise, it will consider only the available queries. After performing the MLG exploration for each query of the LtR training set, we excluded all the queries for which TUEF couldn’t include in the candidate set the expert who provided the accepted answer. For the remaining queries, we extracted features for each query-candidate expert pair, as outlined in Section 3.3. The training set was split into training and validation sets following an 80/20 split. Hyper-parameter tuning is carried out using MRR on the validation set, leveraging the HyperOpt library (Bergstra et al., 2013), and optimizing four learning parameters: $learning\_rate\in[0.0001,0.15]$, $num\_leaves\in[50,200]$, $n\_estimators\in[50,150]$, $max\_depth\in[8,15]$, and $min\_data\_in\_leaf\in[150,500]$.

Table 1 shows the number of queries used for the LtR and the average length of expert lists to rank under the columns LtR and AvgList, respectively.

TUEF is prototyped in Python 3.8.17. All experiments are conducted on an Intel(R) Xeon(R) Platinum 8164 CPU 2.00GHz processor with 503GB RAM on Linux 5.4.0-153-generic. The source code of TUEF is publicly available⁴⁴4https://github.com/maddalena-amendola/TUEF.

Table 2 reports the results of the ablation study. We mark statistically significant performance gains/losses with respect to TUEF with the symbols $\blacktriangledown$ and $\blacktriangle$ (paired t-test with p-value¡0.05 and Bonferroni correction), and we underline the performance figures numerically greater than those of TUEF.

TUEF consistently outperforms in all examined communities some baseline models, including $BC$, $BM25$, $TUEF_{NB}$, and $TUEF_{Lin}$. A recurring pattern emerges across communities, revealing $BC$ and $TUEF_{NB}$ as the least effective baselines, both relying exclusively on network aspects. In contrast, $BM25$’s performance highlights the significance of the content-based component, as evidenced by a minimum of 10% of queries across communities being best addressed by experts who have previously answered similar questions. $TUEF_{Lin}$ consistently exhibits lower performance compared to TUEF, underscoring the efficacy of TUEF’s LtR methodology in capturing hidden relationships among expert features.

$TUEF_{SL}$ shows slightly higher or equal performance across communities except for StackOverflow and AskUbuntu. In the case of StackOverflow, TUEF outperforms a statistically significant margin $TUEF_{SL}$, while on AskUbuntu, the opposite holds, hinting at community-specific influences. This observation aligns with the Silhouette values described in Table 1, which quantify the clustering effectiveness of layers within the MLG. StackOverflow’s superior Silhouette value of 0.805 indicates a robust subdivision of tags into clusters, benefiting TUEF’s exploitation of topic layers. Conversely, AskUbuntu, with the lowest Silhouette value of 0.391, experiences challenges, contributing to $TUEF_{SL}$’s higher statistical performance. Consequently, the MLG proves beneficial when distinct community topics are evident. Moreover, the values of $TUEF_{CB}$ underscore content supremacy over the network component, although incorporating both components in TUEF marginally enhances system performances for most communities. Finally, $TUEF_{NoRW}$ exhibits lower recall (R@5) without a decrease in precision (P@1). The similar P@1 scores between TUEF and $TUEF_{NoRW}$ result from TUEF’s random walk mechanism, which identifies a larger set of relevant experts. TUEF significantly improves recall (R@5) by consistently including the correct expert within the top five positions. However, this larger candidate set makes it more challenging to rank the best expert in the first position, thus keeping P@1 similar for both models. Overall, TUEF’s random walk enhances recall by expanding the candidate set, even though it complicates achieving the highest precision score.

$TUEF_{IlMart}$ consistently exhibits statistically lower performance than TUEF, with the most notable disparity occurring in the StackOverflow community, where the difference reaches 27.5% for P@1 and 12.43% for R@5. In other communities, the variations are still statistically significant, ranging from a maximum of 18.71% (AskUbuntu) to a minimum of 2.16% (Unix) in terms of P@1. However, the differences diminish significantly when considering R@5, with the highest value being 6.83% (AskUbuntu) and no statistical differences in Mathematics, indicating $TUEF_{IlMart}$ proficiency in placing the top respondent within the first five positions.

$TUEF_{IlMart}$ capacity to offer model result explanations may offset the observed performance decline. TUEF, on the other hand, employs a more sophisticated decision tree-based LtR approach, which contributes to higher performance but offers only minimal clarity regarding the significance of different factors influencing the ranking result. Figure 2 illustrates the feature importance of both models (LambdaMart and $TUEF_{IlMart}$) for the StackOverflow community. It reports LambdaMart’s top height features and all those considered by $TUEF_{IlMart}$, all of which are present among LambdaMart’s top eight features, albeit in a different order. For example, the ScoreIndexTag feature, most used by LambdaMart, is the least used by $TUEF_{IlMart}$. Most of the crucial features for LambdaMart are associated with the content-based method. Nevertheless, features from the network-based method, such as Closeness, PageRank, and AvgWeight, exhibit high importance values. On the other hand, all the features selected by $TUEF_{IlMart}$ are content-based, highlighting the importance of the text component for the EF task.

Figure 3 illustrates how $TUEF_{IlMart}$ can be used for interpretability purposes, showcasing the contribution to the predicted score of the three IlMart main effects with the highest feature importance values (i.e., FreqIndexTag, VisitCountContent, and FreqIndexText) and the most important interaction effect learned from $TUEF_{IlMart}$ for the StackOverflow community. High values of these features indicate proficient expert competence. The 2D plots and the heatmap are built by aggregating all the trees using the same features, i.e., by representing the contribution of each main effect $\tau(x_{j})$ and each interaction effect $\tau(x_{i},x_{j})$, where $\tau(x_{j})$ and $\tau_{ij}(x_{i},x_{j})$ are ensembles of trees. Through Figure 3, we can analyze instances where TUEF successfully places the best answerer in the first position and cases where it fails. Table 3 presents four question examples: the first two showcase instances where TUEF successfully identifies the best expert by placing them in the first position. For a better understanding, we report the features of the top two ranked experts. Conversely, for the last two examples where TUEF fails, we show the feature values of the experts in ranking order up to the position where the ranker placed the best expert. The Rank column indicates the expert’s ranking position in the final list. In contrast, the BestExpert column serves as a label, with a value equal to 1 for the ground truth expert and 0 otherwise.

A notable pattern emerges in both instances of TUEF’s success and failure: the expert occupying the first position consistently demonstrates higher values for the features evaluated by the $TUEF_{IlMart}$ ranker. This results in a greater contribution and, consequently, a higher final score, as shown in Figure 3. For example, consider the first instance in Table 3, with $QID=65438592$, and focus on the feature FreqIndexTag. The BestExpert (first row of Table 3) has a value of 45, corresponding to a contribution of approximately 1, as shown in the upper left figure. In contrast, the expert placed in the second position (second row of Table 3) has a value of 7, contributing to a measure of less than 0.5 (half of the first). By combining all feature contributions computed with the aid of Figure 3, we can comprehend the final ranking and understand why the model could or could not place the BestExpert in the first position.