A clever solution to the 'data scarcity” problem! Tsinghua open source GPD: using diffusion model to generate neural network parameters

Traditional spatiotemporal prediction models usually require a large amount of data support to achieve good results.

However, spatiotemporal data (such as traffic and crowd flow data) in many regions are limited due to differences in development levels of different cities and inconsistencies in data collection policies. Therefore, model transferability becomes particularly important when data is scarce.

Current research mainly relies on data from source cities to train models and apply them to data from target cities, but this approach often requires complex matching designs. How to achieve broader knowledge transfer between source and target cities remains a challenging issue.

Recently, pre-trained models have made significant progress in the fields of natural language processing and computer vision. The introduction of prompt technology narrows the gap between fine-tuning and pre-training, allowing advanced pre-trained models to adapt to new tasks more quickly. The advantage of this method is that it reduces reliance on tedious fine-tuning and improves the efficiency and flexibility of the model. Through prompt technology, models can better understand user needs and produce more accurate output, thereby providing people with better experiences and services. This innovative approach is driving the development of artificial intelligence technology, bringing more possibilities and opportunities to various industries.

Pictures

Paper link: https://openreview.net/forum?id=QyFm3D3Tzi

Open source code and data: https ://m.sbmmt.com/link/6644cb08d30b2ca55c284344a9750c2e

The latest achievement "Spatio-Temporal Few-Shot Learning via Diffusive Neural Network Generation" published at ICLR2024 by the Urban Science and Computing Research Center of the Department of Electronic Engineering of Tsinghua University The GPD (Generative Pre-Trained Diffusion) model was introduced to successfully achieve spatio-temporal learning in data sparse scenarios.

This method uses the parameters of the generative neural network to transform spatiotemporal sparse data learning into a generative pre-training problem for the diffusion model. Unlike traditional methods, this method no longer requires extracting transferable features or designing complex pattern matching strategies, nor does it need to learn a good model initialization for few-shot scenarios.

Instead, this method learns knowledge about neural network parameter optimization by pre-training on data from the source city, and then generates a neural network model suitable for the target city based on prompts.

The innovation of this method is that it can generate customized neural networks based on "prompts", effectively adapt to the differences in data distribution and characteristics between different cities, and achieve ingenious spatio-temporal knowledge transfer.

This research provides new ideas for solving the problem of data scarcity in urban computing. The paper's data and code are open source.

From data distribution to neural network parameter distribution

Figure 1: Data pattern level knowledge transfer vs. neural network level knowledge transfer

As shown in Figure 1(a), traditional knowledge transfer methods usually train the model on the data of the source city and then apply it to the target city. However, there may be significant differences in data distribution between different cities, which results in direct migration of the source city model that may not fit well with the data distribution of the target city.

Therefore, we need to get rid of our reliance on messy data distribution and seek a more essential and transferable way of sharing knowledge. Compared with data distribution, the distribution of neural network parameters has more "high-order" characteristics.

Figure 1 shows the transformation process from the data pattern level to the neural network level knowledge transfer. By training a neural network on data from a source city and converting it into a process of generating neural network parameters adapted to the target city, the data distribution and characteristics of the target city can be better adapted.

Pre-training prompt fine-tuning: achieving spatio-temporal few-shot learning

Figure 2 Overview of the GPD model

As shown in Figure 2, this study The proposed GPD is a conditional generation framework designed to learn directly from the model parameters of the source city and generate new model parameters for the target city. The method consists of three key stages:

1. Neural Network Preparatory phase: First, for each source city area, the study trains a separate spatiotemporal prediction model and saves its optimized network parameters. Model parameters for each region are optimized independently without parameter sharing to ensure that the model can best adapt to the characteristics of the respective region.

2. Diffusion model pre-training: This framework uses the collected pre-trained model parameters as training data to train the diffusion model to learn the process of generating model parameters. The diffusion model generates parameters through stepwise denoising, a process similar to the parameter optimization process starting from random initialization, and is therefore better able to adapt to the data distribution of the target city.

3. Neural network parameter generation: After pre-training, parameters can be generated by using regional cues of the target city. This approach leverages hints to facilitate knowledge transfer and precise parameter matching, taking full advantage of the similarities between inter-city regions.

It is worth noting that in the pre-training-cue fine-tuning framework, the selection of cues is highly flexible, as long as it can capture the characteristics of a specific area. For example, various static features such as population, regional area, functions, and distribution of points of interest (POIs) can be used to achieve this purpose.

This work utilizes regional cues from both spatial and temporal aspects: spatial cues come from node representations in urban knowledge graphs [1,2]. It only uses relationships such as regional adjacency and functional similarity. These Relations are readily available in all cities; temporal cues come from the encoder of a self-supervised learning model. See the original article for more details on prompt design.

In addition, this study also explored different cue introduction methods, and the experiment verified that the cue introduction based on prior knowledge has the optimal performance: using spatial cues to guide the generation of neural network parameters for modeling spatial correlation, and using time Prompt guides temporal neural network parameter generation.

Experimental results

The team described the experimental settings in detail in the paper to help other researchers reproduce their results. They also provided the original paper and open source data code, whose experimental results we focus on here.

To evaluate the effectiveness of the proposed framework, this study conducted experiments on two classic spatiotemporal prediction tasks: crowd flow prediction and traffic speed prediction, covering multiple city data sets.

Picture

Table 1 shows the comparison results against state-of-the-art baseline methods on four datasets. Based on these results, the following observations can be made:

1) GPD exhibits significant performance advantages over the baseline model and consistently outperforms in different data scenarios, indicating that GPD achieves an effective neural network parameter level knowledge transfer.

2) GPD performs well in long-term prediction scenarios. This significant trend can be attributed to the framework’s mining of more essential knowledge, which helps transfer long-term spatiotemporal pattern knowledge to target cities.

Figure 3 Performance comparison of different spatio-temporal prediction models

In addition, this study also verified the flexibility of the GPD framework for adapting to different spatio-temporal prediction models. In addition to the classic spatiotemporal graph method STGCN, this study also introduces GWN and STID as spatiotemporal prediction models and uses a diffusion model to generate their network parameters.

Experimental results show that the superiority of the framework will not be affected by model selection, so it can be adapted to various advanced models.

Further, this study conducts case analysis by manipulating pattern similarity on two synthetic datasets.

Figure 4 shows that regions A and B have highly similar time series patterns, while region C exhibits significantly different patterns. Meanwhile, Figure 5 shows that nodes A and B have symmetric spatial positions.

Therefore, we can infer that regions A and B have very similar spatiotemporal patterns, while there are obvious differences from C. The neural network parameter distribution results generated by the model show that the parameter distributions of A and B are similar, but significantly different from the parameter distribution of C. This further validates the ability of the GPD framework to effectively generate neural network parameters with diverse spatiotemporal patterns.

Figure 4 Visualization of time series and neural network parameter distribution in different regions

Figure 5 Simulation data set regional space Connection relationship

Reference:

//m.sbmmt.com/link/6644cb08d30b2ca55c284344a9750c2e

[1] Liu, Yu, et al. "Urbankg: An urban knowledge graph system." ACM Transactions on Intelligent Systems and Technology 14.4 (2023): 1-25.

[2] Zhou, Zhilun, et al. "Hierarchical knowledge graph learning enabled socioeconomic indicator prediction in location-based social network." Proceedings of the ACM Web Conference 2023. 2023.