To effectively utilize multi-level user intentions, Hong Kong University of Science and Technology, Peking University, etc. proposed a new session recommendation model Atten-Mixer

As an intelligent information filtering technology, the recommendation system has been widely used in actual scenarios. However, the success of recommendation systems is often based on a large amount of user data, which may involve users' private and sensitive information. In scenarios where user information is restricted by privacy protection or cannot be obtained, traditional recommendation systems often fail to perform well. Therefore, how to build a trustworthy recommendation system while ensuring privacy and security is an urgent problem to be solved.

In recent years, as users pay more and more attention to their own privacy, more and more users tend to use online platforms without Perform login operations, which also makes anonymous session-based recommendations an important research direction. Recently, researchers from Hong Kong University of Science and Technology, Peking University, Microsoft Asia Research and other institutions have proposed a new model Atten-Mixer that efficiently utilizes multi-level user intentions. The research paper received an honorable mention for Best Paper at WSDM2023.

##Paper link ：https://dl.acm.org/doi/abs/10.1145/3539597.3570445

##Research background

Session-based recommendation (SBR) is a method of recommendation based on the user's short, dynamic session (that is, the user's behavior sequence).

Compared with traditional user- or item-based recommendation systems, SBR focuses more on capturing the user’s immediate needs in the current session and can better The challenge of effectively adapting to the rapid evolution of user interests and the long tail effect.

In the evolution of the SBR model, from a model based on Recurrent Neural Network (RNN) to a model based on convolutional neural network (Convolutional Neural Network, CNN) model, and in recent SBR research, models based on Graph Neural Network (GNN) are widely used to better mine the complex transfer relationships between items.

##However, these models Performance improvements on benchmark datasets are limited compared to the exponential increase in model complexity. Faced with this phenomenon, this paper raises the following questions: Are these GNN-based models too simple or too complex for SBR?

##Preliminary analysis

##In order to answer this question, the author Try to deconstruct existing GNN-based SBR models and analyze their role in SBR tasks.

Generally speaking, a typical GNN-based SBR model can be decomposed into two parts:

(1) GNN module. The parameters can be divided into propagation weights for graph convolution and GRU weights for fusing the original embedding and graph convolution output.

(2) Readout module. Parameters include attention pooling weights for generating long-term representations and transformation weights for generating session representations for prediction.

#Next, the author discusses these two parts respectively. Sparse Variational Dropout (SparseVD) is used, a commonly used neural network sparsification technology, and the density ratio of parameters is calculated when training the model.

The density ratio of a parameter refers to the ratio of the number of elements greater than a certain threshold to the total number of elements in the weight of the parameter. Its value can be used to measure the importance of the parameter.

#GNN module.

Since GNN has many parameters, with random initialization, there will be many at the beginning Knowledge to be updated. Therefore, we can see that the density ratio of the graph convolution propagation weight will fluctuate in the first few batches of data. As training stabilizes, the density ratio will tend to 0.

## Readout module.

We can find that as training proceeds, the density ratio of the

attention pooling weight can be maintained at a higher level. We can also observe the same trend on other datasets and other GNN-based SBR models.

Therefore, the authors found that many parameters of the GNN module are redundant during the training process. Based on this, the author proposes the following simpler and more effective model design guidelines for SBR:

(1) Do not pursue excessive complexity GNN design, the author prefers to

delete the GNN propagation part and only retain the initial embedding layer;

(2) Model designers should be more

Focus on the attention-based Readout module.

Since the attention pooling weight parameters maintain a high density ratio, the author speculates that more advanced attention-based readout methods should be used. Architectural design will be more beneficial.

Since this article abandons dependence on the propagation part of GNN, the Readout module should assume more responsibility for model reasoning.

Considering that the existing Readout module based on instance-view has limited reasoning capabilities, this article needs to design a stronger reasoning Capabilities of the Readout module.

How to design a Readout module with stronger reasoning capabilities

##According to psychopathology research, human reasoning is essentially a multi-level information processing process.

For example, by comprehensively considering the underlying goods Alice interacts with, humans can obtain some higher-level concepts, such as whether Alice plans to prepare for a wedding or decorate new house. After determining that Alice is likely planning a wedding, the human then considers wedding items related to the bouquet, such as wedding balloons, rather than decorative items related to the bouquet, such as a wall mural.

Adopting this multi-level reasoning strategy in recommendation systems can help prune a large search space and avoid local optimal solutions, by considering the user The overall behavior trend converges to a more satisfactory solution.

Therefore, this article hopes to introduce this multi-layer reasoning mechanism

into the Readout module design .

However, obtaining these high-level concepts is not an easy task, because simply enumerating these high-level concepts is not realistic and is likely to introduce irrelevant concepts and interfere with the performance of the model.

To address this challenge, this article adopts two SBR-related inductive biases: local invariance and inherent priority. (inherent priority), to reduce the search space .

• Intrinsic priority refers to the last few items in the session that better reflect the user's current interests;
• Local invariance It means that the relative order of the last few items in the session does not affect the user's interest. Therefore, in practice, groups can be formed by different numbers of tail items, and related high-level concepts can be built through these groups.

Here the tail item corresponds to the inherent priority, the group corresponds to local invariance, and the different numbers represent the multi-layered high-level concepts that this article considers.

Proposed model

Therefore, this article proposes a model called Atten-Mixer. The model can be integrated with various encoders. For the input session, the model obtains the embedding of each item from the embedding layer. The model then applies linear transformation to the resulting group representation to generate multi-level user intent queries.

Where Q1 is the instance-view attention query, while the others are higher-level attention queries with different receptive field and local invariant information. Next, the model uses the generated attention queries to attend the hidden state of each item in the session and obtain the final session representation.

Experiment and results

In the offline experiment, this article uses data from three different fields Sets: Diginetica is a dataset for e-commerce transactions, Gowalla is a dataset for social networks, and Last.fm is a dataset for music recommendations.

Offline experimental results

(1) Overall comparison

The author compared Atten-Mixer with four baseline methods based on CNN, RNN-based, GNN-based and readout-based.

Experimental results show that Atten-Mixer surpasses baseline methods in terms of accuracy and efficiency on three datasets.

(2) Performance improvement analysis

In addition, the author also embedded the Atten-Mixer module into SR-GNN and SGNN-HN to verify the performance improvement effect of this method on the original model.

Offline experimental results show that Atten-Mixer significantly improves model performance on all data sets, especially when the K value in the evaluation index is small, indicating that Atten-Mixer can help The original model generates more accurate and user-friendly recommendations.

Online experimental results

The author also deployed Atten-Mixer into large-scale e-commerce online services in April 2021. Online experiments show that the multi-level attention mixing network (Atten-Mixer) performs well on various online business indicators. All have achieved significant improvements.

Experimental conclusion

To summarize, Atten-Mixer has multi-level inference capabilities and demonstrates excellent online and offline performance in terms of accuracy and efficiency. The following are some major contributions:

• Complex model architecture is not a necessary condition for SBR, and the innovative architecture design of the attention-based readout method is an effective solution plan.
• Multi-level concept correlation helps capture users’ interests, and using inductive bias is an effective way to discover information-rich high-order concepts.

Research process

Finally, it is worth mentioning that this article has a tortuous development behind it being nominated for the WSDM2023 Best Paper honor. Experience, as one of the authors of the article, Haohan Wang from UIUC, introduced, this article was actually rejected many times during the submission process because it was too simple. Fortunately, the author of the article did not go for the Chinese article. Pandering to the tastes of reviewers, I instead stuck to my own simple approach and ultimately got the article an honorable mention.

The above is the detailed content of To effectively utilize multi-level user intentions, Hong Kong University of Science and Technology, Peking University, etc. proposed a new session recommendation model Atten-Mixer. For more information, please follow other related articles on the PHP Chinese website!