Heterogeneous Graph Representation Learning for multi-target Cross-Domain Recommendation

This paper discusses the current challenges in modeling real world recommendation scenarios and proposes the development of a unified Heterogeneous Graph Representation Learning framework for multi-target Cross-Domain recommendation (HGRL4CDR). A shared graph with user-item interactions from multiple domains is proposed as a way to provide an effective representation learning layer and unify the modelling of various heterogeneous data. A heterogeneous graph transformer network will be integrated to the representation learning model to prioritize the most important neighbours, and the proposed model would be able to capture complex information as well as adapt to dynamic changes in the data using matrix perturbation. Using the real world Amazon Review dataset, experiments would be conducted on multi-target cross domain recommendation.


INTRODUCTION
The challenges in modelling real-world, dynamic, personalized recommendation scenarios have attracted significant research interest.Recommender systems, mainly used as an effective solution to overcome information overload, have been shown to be indispensable in various information access systems as well facilitating decisionmaking processes by providing suggestions which match users' interest [20].Recommendation systems retrieve information from various sources to output personalized suggestions [10].These systems have not only been beneficial in improving customer experience, but also helped improve business profit to the product/service providers [11] by influencing consumer behaviour [6].Graph neural networks have become a state of the art approach to represent complex interconnections between users and items in real world scenarios [25].Through this representation learning approach, interactions can be represented into lower dimensional vector spaces where multiple relations (edges) connect user/items (nodes) [20].Downstream tasks such as link prediction, node classification and personalised recommendations have benefited from this type of representation learning [19].
As shown in Figure 1, real-world recommendation scenarios involve multiple users, products and behaviour ("purchase", "mention" and "view" among others).The four scenarios can be represented by graph a single graph, with the users/items as nodes and the user actions as edges.This unified representation, while maintaining the rich structural and semantic information, and modelling for the subsequent applications, is a challenging task.Moreover, dynamic events such as new users, new items and new user behaviour trends also result in changes to the graph structure, and would need to be captured.Research study on this effective heterogeneous representation learning by leveraging on the existing works which include Deep Neural Network (DNN) models [30] and Graph Effective unified Heterogeneous Representation Learning in order to facilitate the efficient implementation of downstream personalized recommendation tasks is a challenging task.Image Credit [26].
Neural Network (GNN) based models [23] is essential in order to address limitations of the state-of-the art models for Recommendation [25].
This research aims to resolve the following challenges, which have been partly addressed or not yet been solved by the existing research works: C1: Effective Heterogeneous Information Representation.
Real world recommendation scenarios involve multiple connectivity consisting of variable multiple nodes and evolving relations [28].In the context of Figure 1, transactions in Case (1) to Case (4) can involve the same and/or different users.In addition, due to changing product trends, the user and actions also continually evolve, hence learning representations effectively while maintaining the rich semantic and robust structural information is challenging [25].
C2: Efficient preservation of the model structure and semantics at lower computational cost [13].
In addition to the complex and dynamic structure described in C1, the complex and dynamic structure of the model would involve many operations and retraining, which results in high computational cost, inefficient and/or not practical in many situations [23].There is therefore need for a fair trade-off between the model efficiency and complexity, at the same time preserving the rich structural and semantic information of the graph neural network [16].
C3: Cold-start and data sparsity issues.
Although the recommendation model may perform well on static data, infrequent scenarios may result in data sparsity and cold-start issues [32].In the context of the use cases in Figure 1, recommending shampoo products to a user who purchased a neck chain, for instance, is a new scenario which does not have sufficient data for recommendation.This issue would be resolved by multi-domain relation inference.
While most state of the art models are effective on homogeneous data, they have limited applicability when it comes to heterogeneous data [22], and when the dataset changes due to new features and new users [31].In heterogeneous datasets, cold-start and data sparsity are frequent phenomena, mainly due to new and/or infrequent situations.Moreover, many current models are inefficient when handling dynamic datasets [8], particularly involving user and product trends changes in multiple domains [1].
To resolve the highlighted challenges, this research project will focus on developing a unified end-to end framework, HGRL4CDR, which incorporates shared heterogeneous representation learning, and multi-graph attention to reduce computations by the removal of less important node relations for the recommendation tasks.To resolve sparsity and cold start issues, multi-target domain adaptation will be proposed.
The main contributions in resolving the highlighted challenges are summarized as follows: 1. Use of the Matrix perturbation [19] to construct a Heterogeneous Graph Learning model which captures structural, spatial, semantic, and dynamic events to address Challenge 1, whereby the changes are continually updated across the multi-domain graph structure.Previous work on this technique was mainly focused on a single domain.
2.Integration of graph attention mechanisms to the proposed representation learning model to address C2, whereby computations would be mainly focused on the sub-graph related to the particular recommendation task instead of the whole Graph Neural Network representation.Most of the prior research works on heterogeneous graph attention have not been applied to multi-domain scenarios.
3.Use of a multi-domain shared graph model for unified multi-domain information modelling and inductive graph reasoning to address C3 by the inter-domain user-product relations.Cui et al. [2] uses recurrent attention with this approach, whereas the proposed Graph Transformer Network approach would assign different weights based on the the node neighbour importance.

RELATED WORK 2.1 Heterogeneous Graph Representation Learning
Due to their ability to handle heterogeneous data, there is growing interest in the use of GNN methods in recommender systems [30].Heterogeneous Graph Representation Learning has been shown by recent studies to be highly useful in dimensionality reduction, particularly for tasks which involve abundant data [19].
For the effective use of GNNs for Heterogeneous Graph Representation Learning in recommendation, the following issues should be considered: proper graph construction, which requires the appropriate representation of nodes (entities) and edges (relations); adaptive design of the recommendation GNN information aggregation [16]; model optimisation including the loss functions and sampling for the particular tasks [14]; as well as the trade-off between the model efficiency and computational cost [4].
Dai et al. [3] performs studies on Heterogeneous graph representation learning for higher order connectivity in knowledge aware recommendation.The results study show that this is an efficient representation learning technique to alleviate bias in networks with limited sensitive information.Their model, FairGNN uses adversarial debiasing for fair node representations on the GNN.However, more investigation still needs to be done for inputs which include sensitive attributes such as gender for instance, in social media data.
The Dynamic Heterogeneous Network Embedding model, proposed by Yin et al. [28], uses the skip-gram model for dynamic heterogeneous network representation learning.This research direction has the potential to address the dynamic heterogeneous data representation challenges, and is applicable to the real-world heterogeneous recommendation scenarios.However, there is not much work in this direction and more exploration needs to be done, particularly on large-scale networks [28].

Graph Attention Networks
To improve the recommendation accuracy, and alleviate data sparsity in some cases, modelling multi-user behaviour such as clicking, comments, collections and adding to cart, apart from just purchases and reviews, is essential [1].While the goal is to recommend the products that users would be eventually purchase, there is also need to take into account the above mentioned multi-user behaviour that is directly or indirectly related to the ultimate recommendation [12], as well as accurately predict the behaviour changes due to changing trends [8].Wei et al. [24], investigates how user preference is influenced by information exchange across different modalities.Their work is an interesting research direction in improving recommendation diversity and accuracy.The main limitation in their proposed Multi-Modal Graph Convolutional Network model is that it does not capture noisy information and adaptive user preference [20].
Tao et al, 2020 [21], proposes the Knowledge Graph Attention framework in which they investigate adaptive user-item interaction and node representation update.Their model, the Multimodal Graph Attention Network (MGAT), uses gated attention to determine weights based on varying user preferences, thereby capturing complex interaction patterns and different user behaviours which are essential for recommendation accuracy improvement.While the work was mainly limited to video datasets, this would be a good foundation to represent complex real-world user-item interactions for improved recommendation accuracy.
Tang et al. [27] proposes a heterogeneous graph attention network, which samples data into a unified sub-space and uses graph attention for multi-target domain prediction.This is a promising research direction, as the work is applicable to real-world recommendation scenarios where complex user and item interactions are involved.
Other interesting research works include Heterogeneous Graph Attention Network models which have also been used to improve representation of complex graph structures.The heterogeneous graph attention network (HGAT), proposed by Wang et al. [22], is a semi-supervised attention network which maps multi-domain samples into a unified subspace and multi-domain relationship; while the authors in [9] studied the Heterogeneous Graph Transformer to model heterogeneous web graphs using heterogeneous attention.

Cross-Domain Recommendation
Cross-Domain Recommendation (CDR) has been shown to have great potential resolving cold start and data sparsity problems in recommendation systems [27].Depending on the scenarios involved, CDR can be classified into three main categories: single-target, dual-target and multi-target CDR [32].In CDR, information is transferred from the source domain(s) to the target domain(s); this is also commonly referred to as domain adaptation [27].Using information from multiple domains can improve the performance, hence cross-domain recommendation has been shown to be effective in dynamic and new OR new, dynamic situations [2].
While most of the research on cross-domain recommendation has mainly been on single target dual-target domain adaptation [32], the following works could be extended to multi-target real world scenarios.The authors in [15] propose Joint Adaptation Networks (JAN) to jointly align multiple layers distribution across domains, which greatly facilitates the Deep Adaptation Network ability, at the same time simplifying the training process.A novel Deep Reconstruction Classification Network (DRCN), which learns semantic alignment for multiple-target domain adaptation has also been used by Ghifary et al. [5].

RESEARCH OBJECTIVES
The main purpose of this research project is to address the limitations of state-of-the-art models in real-world recommendation scenarios.A novel unified, computationally efficient framework for recommendation would be developed, with a focus on Heterogeneous Graph Representation Learning, multi-behaviour attention and multi-target Cross-Domain recommendation.A unified model from these concepts would be proposed, with an aim to resolve the research questions outlined below: RQ1: How do we adaptively design node relations for multiple connections, behaviour and trend changes?RQ2: How do we select the most suitable neighbours to incrementally update the changing node/edge representations?
RQ3: How do we effectively represent concurrent events for downstream cross-domain recommendation?

PROPOSED APPROACH
The model design mainly involves the development of a multi-modal interaction graph framework HGRL4CDR, to effectively represent the dynamic and heterogeneous information for cross-domain recommendation.To resolve C1 and address RQ1, the scenarios in Figure 1 would be represented by a single heterogeneous graph network of the multi-user-product product interactions.To address C2 and RQ2, the model would also be able to capture the semantic and structural changes to the graph structure in the case of addition of new products and/or users, new reviews and new behaviours.The proposed model aims to prioritize the end-to-end stages of the recommendation pipeline by efficient representation of the static and dynamic/semantic and structural features, at the same time being computationally efficient and providing highly accurate multi-scenario recommendations.By unified representation modelling of the scenarios in Figure 1 to a shared graph of multi-related domains, linked domains consisting multiple-behaviours and products, the integration of the Heterogeneous Graph Transformer Network for attention is proposed.From the complex relations, the transformer is able to learn the most suitable meta-paths for the complex interactions and dynamics across the inter-linked domains, thus alleviating cold-start and data sparsity issues indicated in C3.
Feedback on the proposed model design and implementation would be highly appreciated at the Doctoral Symposium.

Dataset and Metrics
The Amazon Review Dataset [17] is proposed for the experiments and the domains would be classified by the different product categories which include Automotive, Fashion, Beauty and Electronics.The proposed recommendation performance evaluation is based on the Area Under the Curve (AUC) metric.

Baselines and Evaluation
The proposed baselines are as follows: Bayesian Personalized Ranking [18], a common item recommendation framework using implicit feedback such as purchases and views; GraphSAGE-pool [7] and HeroGRAPH [29] models, which are state-of the art graph-based frameworks.
In the work by Cui et al. [2], a shared graph model is used to alleviate data sparsity by a shared graph model of users and items.Following this approach, focus on the proposed framework would be on how well it learns new representations after adjustments to the items/relations.As these changes result in structural changes to the graph nodes and edges, there is need also to leverage on the work by the authors in [7] in order order for the proposed model to learn the best representations under different user behaviours.Across the different domains, the behaviors of the different users would be represented by through attention weight scores.
The proposed training approach extends to the HeroGRAPH model [2], and experiments will be conducted on at least 4 domains as well as based on different combinations of the product domains in order to determine the dynamic performance of the model.

EXPECTED RESULTS
The output from this work would be a unified, computationally efficient dynamic GNN-based recommendation model.While the main focus in this project is resolving the practical challenges on data representation, and dynamicity, the framework could be extended to resolve issues such as explainability, scalability and robustness to adversarial attacks in future work.

CONCLUSION
In this paper, the proposed design of a Heterogeneous Graph Representation Learning framework HGRL4CDR, for Cross-Domain Recommendation, and the integration of a Heterogeneous Graph Transformer Network is discussed.The expected results for the project is to resolve the current practical recommendation challenges such as cold-start, data sparsity, handling data dynamicity and reduce the computational cost.

Fig. 1 .
Fig. 1.Real world recommendation scenarios involve multiple users, different products and variable behaviour.Effective unified Heterogeneous Representation Learning in order to facilitate the efficient implementation of downstream personalized recommendation tasks is a challenging task.Image Credit[26].