Under the "intelligent" development trend of the automotive industry, various L2-level assisted driving functions have become important configurations to attract consumers. On the other hand, in the new era of "software-defined cars", autonomous driving is even more important It has become an important strategy affecting the future development of car companies.
Against such a backdrop, car companies need to answer a question: Should they develop their own autonomous driving systems?
Let’s first take stock of the layout of several car companies in the field of autonomous driving:
Inventory of Autonomous Driving Layout of Automobile Companies
It can be seen that self-developed autonomous driving systems have become a major trend for many automakers. Automobile companies have also discovered that the core of autonomous driving is to fully leverage the value of data in the context of "software and hardware decoupling". Some automakers have even established independent subsidiaries because they attach great importance to the autonomous driving business and to facilitate the subsequent development of the business. , focusing on the development of intelligent driving. For example, FAW Group has established an artificial intelligence subsidiary, FAW (Nanjing) Technology Development Co., Ltd.; Great Wall Motors has established Hao Mo Zhixing; SAIC Group has established a software center, SAIC Zero-Shu, etc.
However, it is not easy to develop an autonomous driving system by yourself, because the autonomous driving system The development process and tool chain are particularly complex.
Traditional automotive software development uses the V model, including many ADAS functions, which are also developed using this process. For details, please refer to the figure below. The left side is the design and development process, and the right side is the testing and verification process.
V model development process
The design and development process on the left, At this stage, the development process is mainly based on MBD (Model Based Design). Most of the elements (models) are based on the standard toolboxes and block groups provided by MathWorks products (MATLAB and Simulink). On the graphical interface Build models and automatically generate code. The overall amount of code that engineers need to write is not much, the development speed is fast, and the development cost is also low.
The right side is the test verification process, that is, X in the loop (X in the loop), using different test methods at different stages, including MIL/SIL/PIL/HIL/DIL/ VIL and other test methods.
Traditional automotive software control logic, including some relatively simple ADAS functions of L2, can still meet the needs using the MBD It is becoming more and more complex, and the previous development process based on MBD has become a bit inadequate.
First of all, for more complex autonomous driving functions, the amount of software code and complexity of functions have also increased by several orders of magnitude. Structured toolboxes and block group modeling are capable of developing simple functional algorithms, but when faced with complex deep learning algorithms, MBD seems to be somewhat stretched in terms of flexibility.
Secondly, the artificial intelligence industry has developed for so many years. Whether it is architecture, tool chains, or various open source function libraries, a strong ecosystem has been formed, which is very important for today’s autonomous driving industry. For readers, implementing it directly with programming is much more efficient than modeling in Mathworks.
At the same time, traditional automotive software will no longer change after mass production, which is unrealistic for autonomous driving software. On the one hand, the development cycle of autonomous driving is long, and the time for development and testing within the vehicle development cycle is far from enough; on the other hand, OTA makes it possible to continuously upgrade the software, so that the life cycle of the software is also extended, and with deep learning The autonomous driving algorithm represented by the model needs to continuously collect long-tail "corner case" data as "fuel" to continuously iterate the algorithm system.
There is a saying that goes well, if you want to go to space, you must build a spaceship. In order to develop autonomous driving systems more efficiently, industry experts have found a suitable autonomous driving development process based on deep learning - that is, a data-driven end-to-end development process.
Forward-looking car companies and Tier 1 have also long been aware of this change in the software development process.
Chen Liming, President of Bosch Chassis Control Systems China, once mentioned in public: “Autonomous driving involves many scenarios, and it is impossible to continue in the traditional way, so it must be added to the actual situation. Road testing, especially the use of data-driven verification methods to verify autonomous driving safety, is the combination of V-model and data-driven closed loop to achieve safety verification."
Pan Asia Technology Center Lu Jianxiang, senior director of , said at the recent World Intelligence Conference: "Traditional car companies must transform from the original waterfall system integration development model of the car end to the agile scenario integration development model of cloud, pipe and end integration."
Of course, this does not mean that the traditional MBD approach is completely outdated. The idea of the V model is still very instructive. For example, system simulation that plays an important role in autonomous driving system testing is actually SIL (software changing), and the MBD development method uses the underlying logic algorithms of the car, such as vehicle control algorithms. will still be used in .
Although the data-driven software development process of each company is different in details, the general idea is the same, which basically follows the following idea: data collection->data storage-> ;Data preprocessing->Data mining->Data annotation->Model training->Simulation testing->Deployment and release.
Waymo’s data closed-loop platform, quoted from Huang Yu’s Zhihu article
The tools and platforms used in the above links (including but not limited to data collection, processing, annotation tools, model training platforms, simulation platforms, OTA tools and some other development tools) are called "tool chains" . The efficiency of the tool chain determines the efficiency of the entire system development.
Although it may seem like there are not many steps, the entire link is actually very complicated.
Take data processing as an example. There are many types of data, including camera data, millimeter wave radar data, and lidar point cloud data. These data need to be denoised first. This is the so-called "data cleaning". Taking pictures as an example, data processing requires first erasing the geographical location information of the picture, removing sensitive information such as faces and license plates, and then unifying the format before the processing is completed.
After the data processing is completed, the next step is to start data annotation. The types of annotation can be roughly divided into 2D, 3D target object annotation, joint annotation, lane line annotation, semantic segmentation, etc. It also involves specific annotation specifications and annotation quality inspection process, and the whole process is extremely cumbersome.
Every link in this complex process requires the support of corresponding tools and platforms.
Unlike the MBD development process, which already has a mature tool chain, the data-driven development process started late and the tool chain efficiency is not high, which brings problems to the autonomous driving developers of car companies. A lot of trouble.
Data-driven, the source is data. Faced with the problem of huge data volume but scarcity of high-value data, car companies do not have much experience to draw from.
Of course, different car companies have different levels of accumulation in the field of autonomous driving, and encounter different problems in the tool chain.
Some car companies started early and invested heavily. The (data-driven) pipeline has been fully run and they have accumulated a lot. In order to further improve efficiency, they have also done a lot of customized development of tool chains. A developer from a car company told "Jiuzhang Zhijia" that because the existing tool chains provided by different companies are "segmented development", they only focus on their own part of the function rather than paying attention to the overall situation, resulting in serious fragmentation and "data islands". . In order to meet their own efficiency needs, they have to develop their own tool chains or find companies in the ecosystem to provide them. Even the data annotation platform must be developed by themselves.
For car companies that have not accumulated much experience in this field, investing so many people in developing tool chains at this stage is not very cost-effective. On the one hand, the foundation is weak. The technology hasn’t been developed to that point yet, and on the other hand, there really aren’t that many people. Limited by resource investment and technical foundation, they still hope to integrate ready-made tool chains and run through the (data-driven) Pipeline as soon as possible. "Tool chains are not our competitiveness. Requirement definition, system integration and functional testing are ours." competitiveness," the person in charge of intelligent driving of a car company told "Nine Chapters Intelligent Driving".
Although different car companies have different development stages, they also have similarities, that is, they all have the pain point of "fragmentation" of the tool chain. Next, we will start from the two links of data processing and simulation respectively, and sort out the current situation and pain points of the tool chain in detail.
Data-driven, the core is data.
The traditional model-based development process is more about optimizing the model based on the developer's past experience, while data-driven optimization is based on massive data. For car companies, in order to establish a data-driven development process, they must learn how to handle huge data.
Data processing is the first and most complicated link in the entire development chain, including data collection, data preprocessing, data mining, and data annotation. The amount of data and the quality of the data directly determine the level of the entire model.
The picture below shows the link for Tesla’s autonomous driving data processing.
Tesla Autopilot data processing process (quoted from Huang Yu’s Zhihu article)
The data-related tool chain includes data collection, data upload, data cleaning, data annotation and other links.
Let’s talk about data collection first.
There are some open source data sets in the industry that can be used for basic algorithm training. Currently, the most commonly used data sets include KITTI, nuScenes, etc. However, most of these data come from overseas public tests. There are still relatively few data sets with Chinese local characteristics.
Summary of open source data sets for autonomous driving, citing the Zhihu article "Summary of open source data sets for autonomous driving"
In this case, to train an algorithm to match a specific scenario, you need to collect data for that scenario. Only when enough and high-quality data is collected can the follow-up process be implemented. As for this first link, the efficiency of the current tool chain is not very good.
An employee of a leading new car manufacturing force told "Jiuzhang Zhijia" that the company's automatic driving data collection trigger and data upload strategy cannot meet the requirements of subsequent problem analysis. For example, after a user crashes a car, the returned data cannot be used—either the amount of data is incomplete, or the collection frequency is incorrect, and developers are very inefficient in troubleshooting the problem.
Generally speaking, different Corner Cases require different data formats for subsequent analysis, and different time periods before and after. This is easy to understand. The takeover caused by different reasons, whether it is a problem with the perception or decision-making module, or an error in the high-precision map, requires different data of course. Even for some special corner cases, there will be some customized data collection requirements, allowing testers to carry out collection tasks for road testing.
Precisely because the collection requirements are complex and the link is difficult to open, in reality some engineers will choose to collect data themselves when they encounter problems.
In order to avoid the above problems, some L4 Robotaxi companies choose to use the most primitive "hard disk copy" method to send back the full amount of data and then conduct data mining.
In this way, there is no problem when the number of test vehicles is small. Once the number of subsequent vehicles reaches a certain level, the amount of data collected by autonomous driving will soon enter the PB era, so "massive" Using the data, we can truly find valuable and small corner cases, truly "finding the needle in the haystack".
To collect fragmented data that is truly useful for autonomous driving, a more intelligent data collection strategy is needed.
What is an intelligent data collection strategy? It is to collect data for specific scenarios.
When Huawei internal staff communicated with "Nine Chapters Smart Driving", they mentioned that "Huawei Octopus" has the function of intelligently labeling scenes: "For example, if artificial intelligence occurs, Take over, or encounter tunnels, roundabouts, unprotected left turns, etc., as well as scenarios such as express tricycles where the cloud needs to actively collect and accumulate data for learning. Developers can upload the pictures that need to be obtained by the vehicle, issue instructions through the cloud, and the vehicle will Adopting a method similar to 'Search pictures by pictures', similar scenes will be automatically intercepted. In this way, you only need to filter out the tagged 'valuable' data and upload it to the cloud, which can avoid the entire section of data Uploading can improve the efficiency of Corner Case mining."
After finding valuable data , it is necessary to perform data cleaning and data annotation.
In the perception model based on deep learning, the mainstream deep learning training method is still supervised learning. Training with this method requires "feeding" a large amount of "true values" to the model. (Ground Truth)" data.
So where do these “true value” data come from? Manually marked.
So the industry often jokes that artificial intelligence means "there is as much intelligence as there are artificial people." Because of the demand for massive data annotation, a new profession was born—— "Artificial Intelligence Trainer".
Although the name of the profession sounds nice, in fact, data annotation is essentially a labor-intensive industry. In order to obtain sufficiently cheap labor, companies gathered in certain areas of Xinjiang, Henan, and Shanxi, forming industrial clusters for data annotation.
As customers (labeling demanders), they are concerned about whether the labeling quality is good enough and whether the labeling price is cheap enough. In other words, “you want the horse to run fast, And the horses don’t eat grass.”
First of all, model training requires high quality of annotated data. Data quality directly determines the accuracy of the trained model. If the quality is not high, it is easy to "Rubbish In, Rubbish Out" . The annotation quality is closely related to the annotation cost. Whether the annotation quality of cheap labor in economically underdeveloped areas can meet the needs of developers is a big question.
Secondly, the amount of data that needs to be annotated is huge. For example, a new visual algorithm usually requires tens of thousands to hundreds of thousands of annotated images for training, and regular optimization also requires thousands of annotated images. The demand for pictures, the marked price of a single picture is slightly different, and it is a very large number when the volume of hundreds of thousands of pictures is considered. Therefore, the demand side is very sensitive to price.
High-quality labeling requirements will inevitably lead to an increase in labor costs, while low prices will affect labeling quality. High quality and low price seem to have become an irreconcilable contradiction.
For car companies, the labor cost of raising dozens of people to do data annotation will be too heavy. They generally prefer to outsource to professional data annotation platforms or data annotation teams. The more famous data annotation platforms include Baidu Zhongchai, JD Zhongzhi, My Neighbor Totoro Data, Datatang, etc.
However, outsourcing is also divided into two categories: the first category is human outsourcing, that is, it provides the annotation platform and annotation tools itself, and the outsourcing company only needs to provide manpower; the second category is services Outsourcing means that it does not provide the annotation platform and annotation tools itself, but directly provides the data to be annotated to the outsourcing company, and the outsourcing company provides the annotated data.
Some car companies have very high requirements for labeling efficiency and will choose to develop their own labeling platforms and labeling tools, so they will choose human outsourcing; while for other car companies, they will choose to develop their own labeling platforms and labeling tools. Developing an annotation platform is obviously not cost-effective. On the one hand, so many resources need to be invested to develop an annotation platform. On the other hand, the self-developed annotation platform has no advantage in price compared with external ones, so it is not cost-effective.
Due to the explosion of market demand, many start-up companies have emerged in the data annotation industry. Data.forge is one of them. Its founder and CEO Yang Yang introduced to "Nine Chapters Smart Driving": Customers are most concerned about the quality/price ratio. In order to improve the quality/price ratio, they have taken many measures, such as automated auxiliary annotation and optimizing the convenience of annotation tools, which has also formed the company's core competitiveness.
When Huawei insiders introduced it to "Nine Chapters Smart Driving", they mentioned that "Huawei Octopus" also provides data annotation services:
"First of all, 'Huawei Octopus' has spent a lot of time polishing its own pre-annotation algorithm. At present, the accuracy of Huawei's pre-annotation algorithm has reached the leading level. It has been tested in multiple international public datasets for autonomous driving such as nuScenes, COCO, and KITTI. Winning first place in the ranking, the pre-labeling algorithm can significantly reduce the time required to label each frame of data.
"Secondly, in order to optimize the operation of the labeling platform, we will combine specific business operations to Optimize human-computer interaction methods and improve staff’s operational efficiency.
“Again, we have a mature management system to ensure the quality of annotation. After the annotator completes the annotation, it will go through three stages of self-inspection by the annotator, random inspection by the quality inspector and random inspection by the annotation manager. It will be delivered to the customer only after the quality inspection process. Unlike other annotation teams where most annotators are located in Xinjiang, Henan, Shanxi and other places, Huawei's manual annotation team is in Shenzhen, right in Huawei's office. The reason for this is In order to facilitate communication and management, it can also better ensure the quality of annotation.
"Finally, in order to solve the problem of insufficient local open source data sets, 'Huawei Octopus' can not only provide customers with In addition to providing incremental data annotation services, it can also provide customers with 20 million annotated objects. This data set is continuously iterated and expanded. Customers can use these data for training and quickly build models. ”
As a very core part of the autonomous driving tool chain, the simulation system mainly consists of a scene library, a simulation platform and The evaluation system consists of three parts. The efficiency of the simulation system directly affects the efficiency of the entire development link, so it has always been a pain point for customers and a market targeted by many players.
That’s exactly what it is Because they saw the importance and immature status of the simulation system and felt that "the world is vast and there is a lot to do", many players entered this track one after another. According to the type of company, these players can be roughly divided into traditional simulation software companies and start-up simulation software companies. There are three categories of simulation software for companies and technology giants. Let’s take a look at the categories below.
(1) Traditional simulation software companies
Traditional simulation software is represented by PreScan of Siemens, VTD of VIRES of Germany, CarMaker of IPG of Germany, and CarSim of MSC of the United States. They may rely on their deep accumulation in certain fields, or because of certain The functions are excellent and are widely used by many car companies: CarMaker and CarSim have the deepest accumulation and strongest strength in the field of vehicle dynamics; VTD is famous for its high scene rendering capabilities and the first support for OpenX; PreScan is easy to operate and easy to get started. Attracting many users.
With existing customer resources and advantages accumulated in the past, they have become an important player in the field of autonomous driving simulation software.
(2) Start-up simulation software companies
have seen the huge market space for simulation software, and many new players from start-up companies have also entered, hoping to get a piece of the pie. For example, the domestic startup 51WORLD (formerly 51VR) launched the 51Sim-One autonomous driving simulation test platform; the Israeli startup Cognata provides different simulation solutions for each stage of intelligent driving products. In order to meet the needs of different customers, it even launched a local There are three versions: version, cloud version and hardware version.
Start-up companies are more sensitive to the market and have no historical baggage. The simulation platform they provide for car companies has begun to consciously open up all aspects of simulation and has become a force that cannot be ignored.
(3) Technology giant simulation software company
NVIDIA: Drive Constellation
NVIDIA launched Drive Constellation simulation in 2018 system. The simulation system is built by two different servers. The first server runs NVIDIA DRIVE Sim software for sensor simulation, such as cameras, lidar and millimeter wave radar. The second server is equipped with NVIDIA DRIVE Pegasus artificial intelligence vehicle computing platform. Used to process simulated sensor data.
Drive Sim is based on the Omniverse platform. According to NVIDIA officials, it can achieve "photorealistic and physically accurate" sensor simulation. In terms of scenarios, Drive Constellation can generate data streams, create various test environments, simulate various weather conditions, as well as different road surfaces and terrains, and can also simulate dazzling bright lights at different times during the day and limited vision at night.
Huawei: "Huawei Octopus" autonomous driving cloud service
In the field of autonomous driving development tool chains, Huawei has launched an autonomous driving cloud service, also known as " "HUAWEI Octopus" provides complete solutions from data collection, hard case mining, data annotation, algorithm training, simulation platform, etc., and provides a large number of data sets and scenario libraries for customers to use, helping Automobile companies build a data-driven closed-loop autonomous driving development platform.
In addition, based on Huawei's powerful cloud business, "Huawei Octopus" integrates cloud training and cloud parallel simulation, has rich simulation scenarios, high concurrent instance processing capabilities, and provides more than 200,000 simulation scenario instances; the system’s daily virtual test mileage can exceed 10 million kilometers and supports concurrent testing of 3,000 instances.
Baidu: Apollo Platform
Baidu Apollo provides developers with cloud-based decision-making system simulation services, building a cloud simulation platform on Baidu Cloud and Microsoft Azure. Easily create a virtual operating capacity of millions of kilometers per day. In terms of scenario library, the scenario library provided by Baidu Apollo platform covers regulatory standard scenarios, hazardous working condition scenarios and capability assessment scenarios, with a total of about 200 types.
Apollo also cooperated with Unity to develop a virtual simulation environment based on the Unity engine and proposed an end-to-end autonomous driving simulation system - the augmented reality autonomous driving simulation system AADS, by simulating traffic Streams to enhance real-world images to create realistic simulation scenes.
Baidu has opened the self-driving data set ApolloScape. It has now opened 147,000 frames of pixel-level semantic annotated images, including hundreds of thousands of frames of pixel-by-pixel semantics such as perceptual classification and road network data. Segment annotated high-resolution image data and its corresponding pixel-by-pixel semantic annotations. ”
Tencent: TAD Sim
Tencent released the simulation platform TAD Sim in 2018, which is a combination of professional game engine, industrial-grade vehicle dynamics model, virtual reality An autonomous driving simulation platform that combines virtual and real, and integrates online and offline, created by integrated traffic flow and other technologies, can realize the geometric restoration, logical restoration and physical restoration of the scene.
TAD Sim also supports cloud Operation includes two modes: scene-based cloud simulation and virtual city-based cloud simulation. City-based cloud simulation can achieve both accelerated simulation and high-concurrency simulation to meet various scenarios and driving possibilities in the real world and accelerate enterprise automation. Driving test process. There are more than 1,000 scene types in the scene library, with a daily test capability of more than 10 million kilometers.
These technology giants build simulation platforms, which are more based on their Some have rendering capabilities, cloud computing and other advantages to build autonomous driving simulation processes. They pay more attention to cloud parallel simulation and scene libraries. They also more consciously open up all upstream and downstream links to move forward the testing and verification of autonomous driving systems. One step forward.
(1) Simulation software: You must understand both simulation and cars
As a link for autonomous driving development As one link, simulation needs to be organically integrated with other links.
Traditional simulation software, although very professional in certain fields, is very troublesome when connecting with upstream and downstream links.
For example, for problems discovered during drive testing, developers certainly hope to include the scenario into the simulation scenario library so that regression testing can be done later, but many traditional software do not support this. For functions, you can only build the scene library manually. The efficiency of manually building the scene library is very low, and you can't build a few in a day.
For example, some traditional simulation software can only run in the WINDOWS environment, but now the environment for autonomous driving development is under the Ubuntu environment.
Another example is that the cloud parallel simulation functions of traditional simulation software are not very compatible, and some are only compatible with cloud simulation in recent versions. According to an industry expert, because traditional simulation software sells licenses, several licenses are needed to install the software on several computers.
As cloud parallel simulation plays an increasingly important role, the SaaS model that charges based on service is obviously more customer-friendly and is also a subsequent development trend. Traditional simulation software sells licenses. It also needs to be adjusted accordingly.
Cloud parallel simulation can undoubtedly greatly improve the efficiency of autonomous driving development. The simulation platforms of giants such as Huawei, Baidu, and Tencent can seamlessly connect to their cloud platforms. The products of the startup 51WORLD also support Simulate in parallel and support deployment on private and public clouds.
In addition to providing simulation software, ecological giants also integrate simulation platforms and other tool chains into their full-stack solutions. For example, "Huawei Octopus" provides a one-stop simulation and evaluation tool chain in the cloud to realize DevOps in the field of autonomous driving. From code warehouse access, version management, to simulation and evaluation, it can realize an automated closed loop. In this way, it will be easier for car companies to get started and the adaptation cost will be lower.
However, these giants also face a big challenge, that is, due to the lack of sufficient accumulation of hardware such as vehicle dynamics models and core automotive components, these companies need to use their own Research or cooperate to supplement relevant capabilities. For example, Baidu chose to develop its own vehicle dynamics model, and Apollo 5.0 added a vehicle dynamics model; the "Huawei Octopus" simulation system strategically cooperated with VTD and embedded CarMaker's vehicle dynamics model. It is understood that Huawei and Saimu Technology have also begun to establish a cooperative relationship and will develop efforts in the field of autonomous driving expected functional safety (SOTIF).
(2) Scenario library is the core
In the data-driven development link, data-driven is equivalent to the "question sea tactic", All the examiner can do is ask more and more difficult questions. In the system development link, the scenario library is equivalent to the test questions given by the examiner to evaluate the quality of the software. Therefore, the quantity and quality of the scenario library directly determine the level of the system.
Scene libraries generally come from several sources: scene libraries purchased from third parties. Most of the third-party scene libraries available on the market are based on standard regulations and expert experience data; Forwardly build a scene library based on scenarios. For example, if you want to make a parking function, you need to design a scene for parking, which is more labor-intensive. For a takeover event or corner case discovered during the road test, reversely generate a scene library, which is equivalent to the candidate's previous experience. The wrong questions were compiled into my own "wrong question book".
In addition to these scene libraries, car companies also continue to "expand" their own scene libraries through corner cases encountered in road tests. In response to this demand, some simulation software, such as "Huawei Octopus", provides the function of "converting real drive test scenarios into simulation scenarios with one click", and can be edited and generalized on this basis. For example, changing the weather environment, surrounding environment, mirroring and other means to generalize more scenarios, and Huawei also provides virtual and real hybrid simulation capabilities.
The so-called virtual and real hybrid simulation is to build a test scenario in the cloud and then load it into the vehicle for running. In this way, the vehicle can simulate various simulations on the open road or in a closed site. Virtual scenes, especially dangerous scenes such as pedestrian crossing and non-motor vehicle CUT-IN, can test the autonomous driving algorithm and the vehicle dynamics performance of the real vehicle, thereby improving testing efficiency.
(3) Simulation evaluation
Simulation evaluation may be the most easily overlooked part of the entire simulation system.
Simulation evaluation mainly includes two aspects. On the one hand, it refers to whether the current test can be judged to be passed. On the other hand, it refers to the consistency and repeatability of the actual vehicle test in the same scene corresponding to the current test. .
How to evaluate whether the system can successfully pass the exam of a scene library? The test questions have been published and the candidates have completed them. How to "mark" and set KPIs for the autonomous driving software system?
If you were an examiner, what evaluation criteria can you think of?
Has the target point been reached? Did it drive safely (no collisions)? Did you run a red light? Are you accelerating or decelerating rapidly? etc.
There are many evaluation criteria just thinking about it. What is even more troublesome is that different scenarios have different emphasis on algorithm inspection, and it is very likely that the evaluation criteria are also different. The scene library is full of strange things, and the evaluation standards naturally vary widely.
But generally speaking, the evaluation criteria can be divided into five major aspects: standard matching (whether it meets standard regulations), driving safety (whether it is safe enough), and driving efficiency (whether it is safe enough). Able to reach the destination efficiently enough, fuel economy), driving comfort (whether it is comfortable enough) and driving intelligence (whether it is smart enough).
According to industry experts, each scene library must be "tailor-made" to pass or fail the evaluation criteria when building it. At this time, the simulation software needs to provide diversified simulation evaluation standards. If not provided, it means that certain aspects cannot be assessed.
Therefore, each simulation software also pre-defines the evaluation standards of the scene library for customers in advance, such as "Huawei Octopus" from safety, comfort, reliability, human-computer interaction A total of 200 evaluation indicators have been opened in terms of experience, usability, compliance, energy consumption and traffic efficiency. According to Huawei insiders, in order to make simulation evaluation more flexible, we will also support the development of customized simulation evaluation standards by customers in the future.
The simulation platform mentioned above and the upstream and downstream tool chains are connected vertically. There is also a relatively large one in the industry. The biggest pain point is that when connecting horizontally, the formats of different simulation software are not compatible.
The same car company often uses several simulation software at the same time. For example, it may use both Prescan and VTD. Each simulation software will accumulate a series of scenario cases, but different The formats of scenario case libraries produced by simulation software are incompatible with each other and the files cannot be universal.
This is actually because the entire industry is not standardized enough.
In order to solve this problem, the simulation field standard OpenX released by ASAM has been recognized by many car companies, suppliers and scientific research institutions. Currently, most simulation software has begun to support the OpenX standard. ASAM is developing more standards.
ASAM simulation format standard (quoted from 2020 China Autonomous Driving Simulation Blue Book)
There are still some simulation software that currently do not support the OpenX format. According to industry insiders: "Some simulation software companies want to control all links in their own hands, making themselves irreplaceable and binding customers to them, and they cannot change them even if they want to. This is also the case for some simulation testing companies in the past. It is a common method used by giants. But it is definitely unacceptable to car companies. They really don’t want to be kidnapped. They hope to achieve standardization and reduce migration costs."
After all, those who do not support OpenX are still A few, but overall, standardization is the general trend. I believe that with the advancement of standardization, file compatibility between different software will no longer be a problem in the near future.
As we all know, many L2 autonomous driving functions now use high-precision maps, especially for L4 autonomous driving. High-precision maps will become important infrastructure. For autonomous driving simulation, high-precision maps are also an indispensable and important link. The construction of many simulation scenes, such as the conversion of the drive test scene mentioned above into a simulation scene, and virtual-real hybrid simulation, are inseparable from the support of high-precision maps.
(1) Compliance issues
However, high-precision maps also have many pain points. The first thing to solve is the compliance issue.
Currently, there are only more than 20 domestic companies with Class A surveying and mapping qualifications. Huawei, Alibaba, Tencent, Baidu, Xiaomi, and Didi all have this qualification. Among car companies, SAIC Zhonghaiting, Geely Yikatong and Xpeng Motors, which recently acquired Zhitu Technology, also have Class A surveying and mapping qualifications.
NavInfo CEO Bai Xinping once told the media: "High-precision maps must be participated by qualified companies. Qualifications are related to compliance and safety. In the early stage, the state regulated this field. It’s not too strict, it will get stricter in the future.”
In this context, in order to solve compliance problems for mass production plans, car companies will choose to cooperate with qualified map service providers. Map service providers need to build a high-performance, high-reliability infrastructure that meets safety and compliance requirements and can effectively support the secure storage of massive map data. They should also have powerful computing resources and intelligent algorithms to desensitize drive test data. and the processing of compliance applications. At the same time, the system must also effectively support third-party partners to carry out intelligent driving development and map data application services.
(2) Complex scene accuracy issues
Currently, leading map service providers have covered major highways and expressways across the country, but The quality of the map is still not optimistic, and there will still be missing and wrong labels.
Industry insiders told Jiuzhang Zhijia that a leading map service provider’s high-precision map coverage of highway sections is incomplete, especially for ramps in and out of highways, toll stations and services area, there may be deviations or under-coverage.
When communicating with the person in charge of high-precision maps of a certain car company, the person in charge told Jiuzhang Zhijia that when they were testing L4 Robotaxi, the main scene was on urban roads. There are fewer map service providers that can cover this part, and the quality and update frequency are not high. They have to collect and produce high-precision maps themselves.
Therefore, high-precision maps should not only strengthen the coverage of highways, but also focus on solving the coverage problem in urban commuting scenarios and improve the accuracy of complex road conditions. Only in this way can the support role of high-precision maps for autonomous driving be improved, and at the same time effectively support the simulation and testing of autonomous driving in complex urban scenes.
(3) Dynamic update problem
High-precision maps also need to solve the problem of dynamic update. Otherwise, once the data loses its timeliness, not only It cannot effectively support intelligent driving and may also cause safety hazards.
Currently, many industry insiders believe that the map crowdsourcing update model will become the mainstream technology model in the future because it has more advantages in terms of update timeliness and collection cost. Regarding technical routes, relevant domestic map service providers are also constantly exploring and conducting relevant technical tests. Crowdsourcing map updates not only faces many technical challenges, such as diversified data sources, uneven quality, inconsistent collection element standards, cloud-device-vehicle link interoperability, etc., but also faces constraints from national laws and regulations. , which involves risks such as filtering of sensitive geographical information, encryption of map data, and leakage of personal privacy, and requires further overall planning by relevant national departments.
In fact, solving the dynamic update of high-precision maps is a systematic project that requires the aggregation and integration of multiple resources and data, as well as the collaboration of cloud and edge devices. In the future, it will be through map service providers, Intelligent connected cars, various traffic participants, road infrastructure, edge computing and cloud collaboration, as well as traffic big data, road construction and maintenance data, road operating companies and other parties cooperate to achieve dynamic updates of high-precision maps and improve high-precision map data of freshness.
In the author’s opinion, the production and update of high-precision maps is a huge project. If a unified high-precision map element standard can be formed, multiple resources can be coordinated and coordinated, and repetitive work can be reduced to jointly draw a map of the country. It will be a blessing to greatly reduce industry costs, improve industry efficiency and data reliability, and reduce data security risks.
In data-driven system development, no matter The storage of massive data, model training, and parallel simulation testing all require a large amount of IT resources.
Industry insiders told "Nine Chapters Smart Driving" that when developing autonomous driving systems, they will suddenly encounter some sudden computing power requirements, such as model training, and local computing power cannot To meet the needs, and go through the process to purchase new servers, the approval process may take several months, which will greatly affect the development progress. It is understood that in order to cope with this demand, the intelligent driving development subsidiary of a certain car company planned an entire floor of the office building as a computer room when planning a new office building.
Whether it is storage or training, there is actually a very good way to deal with this sudden demand, which is to go to the cloud.
There are many benefits to moving to the cloud. For example, the cloud development environment has good compatibility, rapid elastic expansion can improve development efficiency, and it also has benefits in terms of cost and data security.
Compared with building a self-built computer room, the benefits of moving to the cloud are
Under the special background of the new coronavirus epidemic, digital transformation has become the way for enterprises to survive. In response to the epidemic, if enterprises want to realize real-time online business and move service scenarios from offline to online, they must undergo digital transformation and integrate employees, customers, services and processes through cloud meetings, cloud procurement, cloud sales, cloud signing, etc. Fully online.
The higher the degree of digital development, the more beneficial it is to the development of enterprises. IDC survey data found that the viability of enterprises with high digitalization index is even more than five times higher than the average level.
The industry generally believes that to achieve digital transformation, going to the cloud is the only way, or even the first step. "Digitalization must first go to the cloud."
Going to the cloud is a necessary option to establish a closed-loop development link for autonomous driving data. Take the optimized link of "Huawei Octopus" on Corner Case as an example. After manual takeover occurs on the car side, "Huawei Octopus" is automatically triggered and feedbacks online to the cloud. The cloud will track and play back and diagnose the cause. If confirmed If it is the vehicle's own responsibility (its own system problem), then the data collection service will upload the valid data before and after the takeover to the cloud and enter the data processing process.
If the sensing link needs to be optimized, data collection, cleaning, and labeling will be carried out. After processing, the sensing module will be trained on the cloud; if the planning control module needs to be optimized, the problem will be Scenes can be converted into a simulation scene library with one click. The optimized algorithm system needs to undergo parallel simulation testing and regression testing. If the simulation evaluation also passes, the OTA push service will be launched in the cloud and the vehicle-side system will be upgraded. In this way, a complete closed loop will be completed.
##On the "Huawei Octopus" data closed-loop link
The cloud is the only way for autonomous driving to move from the development and testing stage to commercialization.
At present, most car companies still focus on development and testing, with the number of test vehicles ranging from a few to dozens. However, as the number of test vehicles increases, the number of subsequent vehicles will increase. After the production of thousands of units, the amount of data generated every day will also increase from hundreds/thousands of TB to 10PB, and the GPU computing power required for training and parallel simulation will also expand from dozens to thousands. By then, the need to move to the cloud will become increasingly urgent.
After understanding the benefits of moving to the cloud, let’s take a look at the classification of cloud computing.
Clouds are generally divided into three categories: public cloud, private cloud and hybrid cloud.
Public cloud is built on non-user-owned infrastructure and can be allocated to a cloud used by multiple tenants. The most commonly mentioned cloud refers to the public cloud. Common Public cloud service providers include Amazon AWS, Alibaba Cloud, Huawei Cloud and Tencent Cloud.
Private cloud is generally created for a single customer. Access rights are exclusive to that customer. Customers can choose to build it in their own computer room (private deployment) or in the cloud. Hosting services (hosted private cloud) are performed in the service provider's computer room.
Hybrid cloud can generally be regarded as a combination of private cloud and public cloud, or a public cloud using different service providers.
It is generally believed that public clouds can be expanded quickly and are more suitable for workloads with large demand or fluctuations. However, in order to expand private clouds, new hardware and resources must be purchased or leased. It's much more complicated. In the process of autonomous driving development, on the one hand, as the number of vehicles increases, the demand for storage will increase exponentially. On the other hand, there are often sudden demands for large computing power during development (cloud training or parallel simulation, etc.). Faced with such needs, public cloud will be more suitable.
Judging from the development trend of cloud computing, the proportion of public cloud market is increasing year by year, while the proportion of private cloud is decreasing year by year. iiMedia Consulting data shows that in China's cloud computing market in 2020, the scale of public cloud surpassed private cloud in 2019 and became the first major market.
When communicating with "Nine Chapters Smart Driving", while recognizing the benefits of public cloud, car company personnel also raised Worry - data security, "If my data is placed on the public cloud, will it be stolen by others?" said a car company employee.
It is precisely because of such concerns that many car companies will choose to build their own servers or choose private clouds; some car companies will choose hybrid clouds, that is, the companies will only use some of the services that are not involved. Data security and data privacy services run on the public cloud, while other services run on the private cloud.
Although some leading car companies and new car-making forces choose public cloud, they choose a service provider that has an equity relationship with them when choosing a public cloud service provider. "After all, it is you." People, don’t have to worry about data security,” they explained.
The basis of trust is mutual understanding and familiarity. Many times, distrust is due to lack of understanding, such as cloud computing.
For enterprises that migrate to the cloud, proper protection of their cloud data is the most important and basic security requirement. This is also what cloud service providers need to win the trust of customers. lifeline".
According to the introduction in "Alibaba Economy Cloud Native Practice", customers' requirements for data security can be summarized by the three basic elements of information security "CIA", namely confidentiality ( Confidentiality), integrity (Integrity) and availability (Availability).
Confidentiality refers specifically to the fact that protected data can only be accessed by legitimate (or expected) users. Its main implementation methods include data access control, data leakage prevention, data encryption and Key management and other means;
Integrity is to ensure that only legal (or expected) users can modify the data, mainly through access control, and at the same time in the transmission and storage of data The integrity of user data can be ensured through verification algorithms;
The availability of data is mainly reflected in the overall security capabilities, disaster recovery capabilities, reliability of the cloud environment, and various aspects of the cloud environment. Guarantee the normal working of related systems (storage systems, network access, identity verification mechanisms, permission verification mechanisms, etc.).
Among these three aspects, the most important technical means to ensure confidentiality (Confidentiality) is data encryption, and it is a full-link data encryption capability.
"Full-link encryption" refers to end-to-end data encryption protection capabilities, as well as encryption of the entire life cycle of data, mainly referring to from off-cloud to on-cloud and on-cloud The transmission process between units, the calculation process (processing/exchange) of data when the application is running, and the encryption capability in the storage process where the data is finally persisted to disk.
Generally speaking, the data encryption operation process is that the plaintext data is calculated through internationally and domestically recognized security algorithms to obtain the data ciphertext. In cryptographic operations, securely protected and managed keys are a sufficient and necessary condition for cryptographic protection. In other words, controlling the key also controls the initiative of the overall encryption operation. Since the user brings his own master key as a user resource, and any call must be authorized by the user, the user has complete independent control and initiative over the use of encrypted data. At the same time, any calls to user resources will be fully displayed in log audits, so the transparency of cloud usage of encrypted data is better guaranteed.
The life cycle of data security, excerpted from Alibaba Economy Cloud Native Practice
Many people in the industry also mentioned one point when communicating with "Nine Chapters Smart Driving": Who can guarantee that internal employees or operation and maintenance personnel of the cloud service provider will not use their permissions to secretly use my data?
This actually involves compliance, which needs to be ensured through internal processes, and this internal process is often confirmed through compliance certification by an authoritative third party. Among them, the most authoritative, widely accepted and applied information security system certification in the world is ISO27001. The compliance certifications passed by each major cloud service provider can also be found on their official websites.
External compliance certification must also be implemented internally. Taking Huawei as an example, there are a series of regulations on the internal security redline from development to management. Once someone violates it, it will be dealt with very quickly. Strict, often demoted, punished, warned, or even fired. When talking about compliance issues, the Huawei insider also joked that after the United States began to sanction Huawei, it has been trying its best to find "hard" evidence of Huawei's "non-compliance" around the world. As a result, it has not been found for more than two years. Found, this also proves from the side how strict Huawei is in terms of compliance. Some time ago, Huawei’s smart car cloud service also passed the ASPICE L2 third-party certification and the Volkswagen Group APSICE (KGAS) PN (potential supplier) review. This also shows that the R&D quality and development process of Huawei's smart car cloud services have been recognized by mainstream international automobile manufacturers.
Perhaps it would be easier to understand from the perspective of business logic. For cloud service providers, customer data security is the lifeblood. Customers entrust their lives to you. Once a problem occurs, this trust no longer exists, and the business foothold is lost. And from the perspective of the architecture of cloud computing itself, data on the cloud will also be safer: on the one hand, cloud service providers will do data backup for off-site disaster recovery (to prevent data loss caused by natural disasters such as fires), on the other hand, the level of security protection will also be Higher (more security talent, more security measures in place).
Although moving to the cloud is a general trend for car companies, it will not happen overnight. It takes a process for car companies to understand and accept the public cloud.
A public cloud marketing staff told "Jiuzhang Zhijia" that relatively speaking, autonomous driving companies and foreign car companies with an Internet background are more willing to go to the cloud. Traditional car companies, especially Since it is a state-owned enterprise, it will have more worries about data.
From the perspective of cloud computing industry development trends, different industries have different understandings of cloud and have different usage rates of cloud computing. According to Frost & Sullivan data, China’s current cloud computing The main users are concentrated in the Internet, finance, government and other fields that have relatively early exposure to the cloud. Among them, Internet-related industries account for about one-third, government cloud currently accounts for about 29%, and traditional industry clouds such as transportation, logistics, and manufacturing The level of computing applications is rapidly improving. I believe that in the future, as car companies deepen their understanding of cloud computing and accelerate the process of digital transformation, their acceptance of the cloud will become higher and higher. In the near future, being unable to access the cloud may no longer be a problem.
The current biggest pain point for car companies in developing autonomous driving systems Yes, mutual fragmentation of tool chains and data silos.
Traditional tool chain companies and start-ups often focus on a certain link in the tool chain. For example, those who do simulation do simulation, and those who do annotation do annotation. When used by car companies, each part is used as a whole. One link in the development tool chain is used in series. If you only focus on a certain link in the middle, it will inevitably be "misaligned" with other links.
Moreover, the current tool chain lacks industry standards, and each company is very different. Customers have to spend a lot of time to adapt, so car companies hope that one supplier can integrate the tools. Connect multiple links in the chain to reduce your own adaptation costs. Seeing this opportunity, technology giants have entered the market with "tool chain ecology" and provided full-stack tool chains.
Let’s take a look at the ecology of technology giants:
(1) NVIDIA: Chip-based ecology
Chip giant NVIDIA has built a unified GPU hardware architecture and CUDA software architecture around the car, desktop and cloud. Developers can use very simple instructions to call complex deep learning models. "Nine Chapters Smart Driving" learned from the exchange with industry experts that the important reason why they chose NVIDIA is that NVIDIA has a stable tool chain and a rich software ecosystem. The advantage of a mature tool chain is that if something goes wrong, you can quickly locate the problem.
In 2017, NVIDIA released the autonomous driving platform NVIDIA DRIVE, which is also equipped with self-developed software architectures Drive AV and Drive IX. In-vehicle intelligent driving controller for NVIDIA DRIVE platform. The Xavier series is currently on the market, and the latest Orin is planned to be mass-produced in 2022 and can meet the functional safety standard of ISO 26262 ASIL-D.
In the field of simulation, NVIDIA launched the Drive Constellation simulation system and Drive Sim in 2018.
In 2019, NVIDIA also demonstrated its high-precision positioning solution DRIVE Localization. In addition, NVIDIA is also planning a high-precision map crowdsourcing solution NVIDIA MapWorks.
Currently, NVIDIA has established autonomous driving R&D cooperation with Mercedes-Benz, Audi, Toyota, Volvo, Bosch, Continental and other companies.
(2) Huawei: an open ecosystem of cloud, pipe, terminal and core combination
Huawei insists on “not building cars, focusing on ICT technology, and helping cars The strategy of "Building Good Cars" focuses on chips, cloud, software and hardware, tool chains, and high-precision maps to create an open ecosystem.
Huawei Intelligent Driving Computing Platform MDC integrates Huawei’s self-developed CPU, AI chip and other control chips, and integrates optimization of the underlying software and hardware to achieve the highest overall performance in the industry leading level. In addition, Huawei MDC also has a complete test platform and tool chain, providing a full-stack solution for MDC development. The MDC platform hardware runs the intelligent driving operating system AOS/VOS and MDC Core. In other words, MDC has car-grade software and hardware, which is convenient for car companies to select for mass-produced models.
##MDC overall architecture diagram - from Huawei MDC white paper
In automatic In the field of driving development tool chains, Huawei has launched an autonomous driving cloud service. In addition, Huawei has also launched Internet of Vehicles cloud services (intelligent driving, smart cockpit data collection and storage), three-power cloud services (cloud management and control of three power systems), and high-precision map cloud services. In addition to these, Huawei has also implemented both soft and hard measures to deploy autonomous driving sensors.
(3) Baidu: Apollo open platform
In 2017, Baidu released Apollo, an open platform for driverless driving, to the automotive industry and partners in the field of autonomous driving to provide an open, complete and secure software platform. The Apollo platform is a complete set of software, hardware and service systems, including vehicle platform, hardware platform, software platform and cloud data services. , which can help partners combine vehicles and hardware systems to quickly build their own autonomous driving system.
Apollo continued to upgrade, opening limited-area visual high-speed autonomous driving, autonomous parking (Valet Parking), unmanned operating vehicles (MicroCar), automatic shuttle buses (MiniBus), and complex urban roads. Autonomous driving and other solutions, and began to build its own Robotaxi fleet and conduct test operations in various places under the brand "Carrot Run".
It is worth mentioning that Apollo released the middleware Cyber RT to improve the safety of the autonomous driving system.
Apollo ecological developers provide cloud-based system simulation services and augmented reality autonomous driving simulation system AADS.
In early 2021, Baidu and Geely jointly established Jidu Automobile and announced their end of car manufacturing. Robin Li publicly stated that “the purpose of establishing Jidu Automobile is to combine Baidu’s autonomous driving technology and smart cockpit technology to the market.”
(4) Tencent: full-link cloud service and development platform
Tencent is also laying out a development platform for the autonomous driving cloud ecosystem. Tencent does not build cars or sensors, but only provides software and services.
On the car side, Tencent provides solutions that include perception, positioning, planning, decision-making, and control; on the cloud, based on cloud storage and computing power support, Tencent has built a data collection system Management, sample annotation, algorithm training and evaluation, diagnosis and debugging, cloud simulation (simulation platform TAD Sim), and real vehicle feedback closed-loop full-process cloud services provide a full-link cloud service and development platform that supports autonomous driving research and development.
Tencent’s autonomous driving business layout and positioning (quoted from Tencent Su Kuifeng’s online public sharing)
The full-stack tool chain has obvious improvements in efficiency, especially the ability to quickly build a Pipeline. "Huawei Octopus" insiders said: If each company's discrete tool chain solution is used, it may take several months to debug the link alone. However, "Huawei Octopus" has already done this for the entire link. Integration and adaptation are completed, which reduces duplication of work. In addition, Huawei also provides customers with a set of reference algorithms. Customers can debug and optimize based on this, which greatly reduces the difficulty of getting started. It only takes a few days at the fastest to run through the entire process. Complete link, very efficient.
The reason why many car companies choose to develop self-developed tool chains is, on the one hand, efficiency considerations, and on the other hand, For the sake of "safety", car companies also want to maintain their past control position in the ecosystem, but instinctively dislike the potential risk of being "stuck", so they often like to cooperate with small companies in the tool chain.
In terms of openness, different technology giants have different strategies. According to an autonomous driving developer at a certain car manufacturer, the ecology of a certain enterprise’s autonomous driving development platform is not decoupled. "If you want to choose it, you must 'accept it in full' and not accept the use of a single module." This is used to deeply bind customers; Huawei has chosen another path - decoupling each module.
According to Huawei insiders, the tool chain of "Huawei Octopus" is divided into four parts: data, training, simulation, and supervision. These four parts are completely open to decoupling and not binding. , customers can replace it at any time.
For car companies, if the existing technical reserves cannot support the mass production plan, they can only outsource mass production. , which seems to conflict with the self-research strategy.
When communicating with "Jiuzhang Smart Driving", the answers given by car company developers were surprisingly consistent: On the one hand, mass-produced cars are equipped with ADAS solutions purchased from outside. , because it is a black box procurement, the supplier does not open any data, but for the competitiveness and sales of the vehicle, the car companies can only tolerate this "immediate compromise"; on the other hand, the car companies have invested a lot of manpower and material resources at the same time Regarding the self-developed L2 solution, "once the self-developed solution matures, it will be gradually replaced and put into use", so self-developed became "Poetry and Distance".
Considering these demands of car company customers, "Huawei Octopus" provides customers with a variety of cooperation options. Huawei insiders introduced: "In the first option, Huawei is responsible for developing and providing complete mass production solutions. option; in the second option, Huawei is responsible for development, and customers can freely configure some parameters; in the third option, Huawei provides an autonomous driving development tool chain, and customers conduct self-development, and Huawei provides a full set of after-sales development consulting services.”
This article analyzes the current status and development trends of the industry from the perspective of the autonomous driving development tool chain.
The current development of the autonomous driving development tool chain industry is still immature, and the phenomenon of non-standardization and information islands is relatively serious. For the sake of development efficiency, the leading autonomous driving teams have to "invent their own wheels" .
However, with the entry of many new tool chain players, the overall industry is developing toward maturity, and subsequent tool chains will gradually become open, standardized, and standardized. In particular, giants such as Huawei and Nvidia have entered the game with their ecosystems, opening up the entire development chain, bringing examples to the industry, and promoting industry development. In the words of a Huawei insider, this is "pulling up China's autonomous driving industry." , keep running forward."
Autonomous driving on the cloud is a major trend. With high-level autonomous driving, it is gradually evolving from the technical research stage to the large-scale commercial stage. In addition to the requirements for storage, computing power and other resources, It also places stringent requirements on the high reliability, security and scalability of infrastructure services.
The traditional data center construction model will bring huge construction costs and operation and maintenance pressure to autonomous driving development companies. By supporting multiple computing power, the public cloud can meet the business needs of extreme computing power, safety, reliability, elasticity and flexibility for massive infrastructure resources during the autonomous driving development process, such as model training and parallel simulation, thereby achieving agile development of autonomous driving algorithms. with iteration. Therefore, although most companies currently have doubts about the public cloud approach, I believe that with the rapid development of the entire autonomous driving industry and the continued deepening of understanding of the public cloud, this service model will be further promoted.
The above is the detailed content of An in-depth discussion on the current status and future trends of autonomous driving development tool chains. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
