Discuss the current status and development trends of autonomous driving trajectory prediction technology

##​http://www.interaction-dataset.com/​

(3)apolloscape

This is Apollo’s public autonomous driving data set, which contains data provided for trajectory prediction. The internal file is a 1min data sequence of 2fps. The data structure includes frame number ID, target ID, target category, position xyz, length, width, and height information, and heading. The target category includes small cars, large cars, pedestrians, bicycles/electric vehicles, and others.

​https://apolloscape.auto/trajectory.html​

(4) TRAF

This data set focuses on high-density traffic conditions. This condition can help the algorithm better focus on analyzing human driver behavior in uncertain environments. Each frame of data contains approximately 13 motor vehicles, 5 pedestrians and 2 bicycles

​https://gamma.umd.edu/researchdirections/autonomousdriving/ad​

There are many trajectory prediction projects using this dataset in the link.

(5) nuScenes

The big news is coming. This data set was proposed in April 2020. It collected 1,000 driving scenes in Boston and Singapore, two cities with heavy traffic and challenging driving conditions. Its data set has related papers, you can take a look to get a better understanding of this data set.

​https://arxiv.org/abs/1903.11027​

There are prediction-related competitions in this data set, you can pay attention to them.

​https://www.nuscenes.org/prediction?externalData=all&mapData=all&modalities=Any​

6 Evaluation Index

The currently mainly used evaluation index is the geometric metric.

Geometric measurement has many indicators, the main ones used are ADE, FDE, and MR.

ADE is the normalized Euclidean distance. FDE is the Euclidean distance between the final prediction points. MR is the miss rate. There are many different names. The main thing is to set a threshold. If the Euclidean distance between the predicted points is lower than this prediction, it will be recorded as a hit. If it is higher than this threshold, it will be recorded as a miss. Finally, a percentage will be calculated.

Geometric metric is an important indicator to measure the similarity between the predicted trajectory and the actual trajectory, and can well represent the accuracy. But for the purpose of trajectory prediction, it is meaningless to just evaluate the accuracy. There should also be probabilistic measures to evaluate uncertainty, especially for multi-modal output distributions; there should also be task-level measures, robustness measures, and efficiency evaluations.

Probability measure: KL divergence, predicted probability, and cumulative probability can be used as probability measure. For example, NLL, KDE-based NLL [17]. Task-level metrics: Evaluate the impact of trajectory prediction on back-end regulation (piADE, piFDE) [18]. Robustness: Consider the length or duration of the observed part of the trajectory before prediction; the size of the training data; input data sampling frequency and sensor noise; neural network generalization, overfitting and input utilization analysis; perception module If there is a problem with the input input, whether the function is guaranteed to be normal and other factors. Efficiency: Consider computing power.

As shown in the figure below, the main consideration of this paper is that based on the true value (blue), the purple and green trajectories predicted by the gray target car have the same ADE and FDE if geometric metrics are used, but different The prediction method will have an impact on the planning of the own car, and there is currently no such metric for evaluating the task level, so they proposed piADE and piFDE to do this.

7 Three questions

Question 1: Three different trajectory prediction methods : Where are the application scenarios of physical model-based, learning-based, and planning-based? What are the advantages and disadvantages?

Different modeling methods can combine and exploit different types of contextual information. All modeling methods can be expanded by using the target's contextual clues and dynamic and static environments. However, different modeling methods exhibit different levels of complexity and efficiency in combining different categories of semantic information.

1. Physical model-based method

Applicable scenarios: targets, static environments, and dynamic simulations can be Explicit transfer equation modeling.

Advantages:

• Physically model-based methods can be easily applied across environments by selecting appropriate transfer equations, while No training data is required, although some data for parameter estimation is useful. In the paper, a simple CV model also produces reasonable results.
• The physical model-based method can be easily extended by combining it with target agent cues.

Disadvantages:

• This approach to explicit modeling may not capture the complexity of the real world well .
• The transfer equation lacks global information in space and time, resulting in the possibility of obtaining a local optimal solution.

#Such shortcomings limit the use of physical methods to short-term prediction or obstacle-free environments.

2. Learning-based method

Applicable scenarios: Suitable for current environments with complex unknown information (such as Common areas with rich semantics), and this information can be used for a relatively large prediction range.

Advantages:

• Learning-based methods can potentially handle all types of contextual information encoded in the collected data set. Some of them are map-based, and others can be used to further extend context information.

Disadvantages:

• Needs to collect enough data at a specific location for training.
• Contextual information expansion may lead to involved learning, data efficiency, and generalization issues.
• Tends to be used in non-safety critical components. In ADAS, interpretability is more important, which cannot be achieved based on learning methods.

3. Planning-based method

Applicable scenario: The end point is determined and the environment map is Available scenes, well represented.

Advantages:

• If the above two conditions are met, it can achieve better accuracy than physical methods and better than learning-based methods The method has better generalization ability.

shortcoming:

• Traditional planning algorithms: Dijkstra, Fast Marching Method, optimal sampling-based motion planners will grow exponentially as the number of targets, the size of the environment, and the prediction range increase.
• Compared with simple physics-based models, the parameters of context-cue-based planning methods (such as reward functions for inverse programming and models for forward programming) are trivial and generally easier to learn , but in terms of reasoning, it is less efficient for high-dimensional (target) agent states.

Planning-based methods are essentially map-aware and abstacle-aware, and naturally extend using semantic clues. Typically, they encode situational complexity into the goal/reward equation, but this may not properly integrate dynamic line inputs. Therefore, the authors had to design specific modifications to incorporate dynamic input into the prediction algorithm (Jump Markov Processes, local adaptations of the predicted trajectory, game-theoretic). Unlike learning-based methods, target inputs can be easily merged because both forward and backward planning processes are based on the same target dynamic model.

Question 2: Has the problem of trajectory prediction been solved now?

#The need for trajectory prediction largely depends on the application domain and the specific use case scenarios therein. It may not be said that the problem of trajectory prediction has been solved in the short term. Take the automotive industry as an example. Because there are special standards and regulations that define maximum speeds, traffic rules, distribution of pedestrian speeds and accelerations, and specifications for comfortable acceleration/deceleration rates of vehicles, it seems to be the most powerful in formulating requirements and proposing solutions. Mature. It can be said that for the AEB function of smart cars, the solution has reached a performance level that allows industrial production of consumer products, and its required use cases have been solved. As for other use cases, more standardization and clear articulation of requirements will be needed in the near future. And robustness and stability still need to evolve.

So before answering the question of whether trajectory prediction has solved this problem, we should at least set the standard.

Currently in the field of robotics

• methods based on physical models and learning can be implemented in a short time (1-2s) Higher accuracy. It is very suitable for local motion planning and collision avoidance of crowds. The simplest CV model has a good effect on the local planning of the robot. If you consider the interaction between pedestrians and the impact of the presence of robots on pedestrian movements, there are many advanced algorithms.
• There are great challenges for global path planning that requires prediction of 15-20 seconds. The requirements can be relaxed appropriately, and it becomes very important to understand the dynamic and static context input (which affects the operation in the long term, reasoning on the environment map, and the intention inference of the target). For local and global path planning, position-independent methods are best suited for predicting motion in various environments.
• The current robot predicts that the ADE in 4.8s is 0.19-0.4m. A simple velocity model can also achieve an ADE of 0.53m. 9s predicts an ADE of 1.4-2m.

Currently in the field of autonomous driving:

• Most work considers pedestrians crossing the road : Start walking, keep walking, stop walking.
• Bicycle: A cyclist approaches an intersection with up to five different directions of road behind them.

# Question 3: Are current evaluation techniques for measuring trajectory prediction performance good enough?

There is currently a lack of systematic approaches to prediction algorithms, especially for trajectory prediction methods that consider contextual inputs and predict an arbitrary number of targets.

Now most authors only use geometric measures (AED, FDE) as an indicator to measure the quality of an algorithm. For long-term predictions, however, the predictions are often multimodal and associated with uncertainty, and performance evaluation of such methods should use metrics that take this into account, such as the negative log-likelihood or logarithm obtained from KLD loss.

There is also a need for probabilistic measures that better reflect the randomness of human motion and the uncertainty involved in perceptual imperfections.

There is also robustness evaluation, which needs to consider the stability of the system when detection errors, tracking defects, self-positioning uncertainty or map changes occur on the sensing side.

At the same time, although the currently used data sets include very comprehensive scenarios, these data sets are usually semi-automatically annotated, and therefore can only provide incomplete and noisy true value estimates. Furthermore, trajectory length is often insufficient in some application areas where long-term predictions are required. Finally, the interaction between targets in the data set is usually limited. For example, in a sparse environment, it is difficult for targets to influence each other.

In summary: In order to evaluate the prediction quality, researchers should choose more complex data sets (including non-convex obstacles, long trajectories and complex interactions) and complete metrics (geometric probability). A better method is to set different accuracy requirements based on different prediction times, different observation periods, and different scene complexities. And there should be robustness evaluation and real-time evaluation. In addition, there should be relevant indicators that can measure the impact of ADAS systems on the backend [18] and indicators that measure sensitivity to hazardous scenarios [1].

8 Future Directions

From the discussion in [2], quoted here.

The current trend is to use more complex methods to go beyond using a single model KF method

Direction:

1. Use enhanced contextual information: Deeper semantic information can be used, which should provide a better understanding of static environments. And the current use of semantic features for trajectory prediction still needs to be developed
2. Regarding socially-aware scenarios: ① Most current methods assume that the behaviors of all observed people are similar , their movements can be predicted by the same model and the same features, while the capture and reasoning of high-level social attributes are still in the early stages of development. ② Most feasible methods are based on the assumption that cooperative behavior between people, and real people may be more inclined to optimize individual goals rather than joint strategies, so methods combined with traditional AI game theory are very promising.
3. For long-term predictions, contextual information becomes particularly important as intent is considered based on context and surrounding circumstances. Many current learning-based methods treat individuals as particles and use them to learn to transfer information to determine the direction of future movement. Extending these models with more intent-driven predictions, similar to human goal-directed behavior, will benefit long-term predictions.
4. Most planning-based methods rely on a given set of goals, which makes them unusable or inappropriate without prior knowledge of the destinations or where the number of destinations may be too high. accurate. This makes automatic inference of destination endpoints based on semantic information important. Or it can dynamically identify possible destinations in the environment and perform trajectory prediction based on this. This allows planning-based methods to be used in unknown environments.
5. Current methods focus on solving a specific type of task, such as when there is an obvious movement pattern in the environment, or when the spatial structure of the environment and the destination of the target agent When known in advance. Trajectory prediction methods need to be able to adapt to undefined/changing environments and handle unexpected situations. This requires transfer learning and some methods of coping with new environments, where learning and reasoning about basic immutable rules, or general pedestrian behavior or collision avoidance is not appropriate. Domain adaptation can be used to learn generalization models.
6. Another direction to pay attention to: robustness and integrability.

To sum up: To put it simply, the context information should be used more deeply, it is best to have different behavioral models and game theory for different goals, and make more decisions based on more information. Excellent intent prediction, automatic inference of endpoints, generalization problems to new environments, robustness and integrability.