Autohome e-commerce system architecture evolution and platform architecture practice

##01

Preface

02

Architecture evolution

2.1 Starting stage

2.2 Microservice stage

2.3 Master data Stage

2.4 Platform Architecture Stage

## 3.2 Service Orchestration

3.3 Business Configuration 3.4 Development Toolization

3.5 Data Visualization

3.6 Knowledge Precipitation

4.1 Exploring new retail

4.2 Architecture upgrade

Preface

The Autohome e-commerce system was born in 2014 and grew from 2016 to 2019. It has experienced the peak test of Double 11 and 818 parties for many years, and has accumulated a stable, reliable, and Excellent online trading capabilities. With the rise of the wave of business middle platform construction, it will enter the middle platform construction stage in 2019, exporting its five-year accumulation of capabilities in the field of automotive e-commerce, assisting the development of the automotive e-commerce industry, and accelerating the digital transformation of enterprises!

Architecture evolution

This part mainly talks about the architecture development process of Autohome e-commerce system, the business status, technical challenges and technical system response strategies at each stage. .

Initial stage

The Internet environment experienced thousands of regiment wars and e-commerce from 2011 to 2013 After the war‍[1], e-commerce business has become another strategic high ground for monetizing Internet traffic in addition to the advertising model. During the “Double Eleven” period in 2013, Autohome launched a car purchase service and regarded the transaction link as an important development direction[2]. In the initial stage of business, the technical requirement is to quickly iterate and go online to verify the feasibility of the product. While meeting daily business needs, thinking on technical architecture has not stopped. Taking into account the scalability of the future e-commerce system and referring to Alibaba's technology system in the industry, the technology stack began to be developed in 2014 and gradually changed from the .NET system to the Java system, and all application switching was completed on May 30, 2015. Launched the complete online car shopping platform Car Mall .

Microservice stage

With the rapid development of e-commerce business, the number of technical personnel increased. By 2016, the technical team had hundreds of people. The pain of a monolithic architecture comes head-on. As for a front-end mall git project, almost 30 Maven sub-projects are developed in parallel when the requirements are met. Code merger conflicts, waiting for requirements to come online, and slow online development often occur. SQL and other issues, the development efficiency and system stability of the entire system have deteriorated.

#At this time, system support faces huge challenges, and the system architecture must be upgraded and evolved. We began to develop a distributed strategy, splitting the original single system into multiple centralized systems with high cohesion and low coupling. That is, the current user center, product center, order center, promotion center, coupon center, and merchant center. Each independent system can be independently designed, independently received, and independently released. The entire R&D efficiency and system stability have been improved. steps. At this stage, we have technically completed the construction of a million-level commodity system [3], order system [4], and coupon system [5] to support automotive e-commerce, and completed all applications on the cloud [ 6], Automated test platform construction [7].

At the same time, we have explored various business models such as self-operated vehicle e-commerce model, open platform model, B2B2C model, quotation model, consulting model, TPCC model, and parallel imported car sales.

Master data stage

The speed of e-commerce development is so fast. By 2019, the company already has a variety of online transaction models, such as travel, automobile Products and after-market services, points redemption, etc. Based on the development strategy, the company decided to build an e-commerce middle platform. On the one hand, it is to concentrate the company's high-quality product resources and operational resources to create an influential vertical e-commerce trading platform. On the other hand, it is also to reasonably manage and control technical resources and realize the e-commerce system. of unity. Against this background, my team took on the task of building an e-commerce middle platform. Since the business forms and technical architectures of each system are very different, the first problem we faced was how to achieve transactions. Integration of class systems. To this end, on the one hand, we have begun to become familiar with the current status of trading systems in different business scenarios, and on the other hand, we are also thinking and discussing technical solutions. In the end, we chose the solution of "based on data normalization, providing standardized middle-end services, and integrating system by system from the bottom to the top."

Data normalization

Data is the core asset of the company. In any system, especially trading systems, data is the most important. heavy. On the one hand, the construction of master data can unify the data model and break down system barriers; on the other hand, it can also conduct operational data analysis through centralized data and provide a basis for business decisions. Therefore, we regard the construction of master data as the first step in system integration. step. In the transaction process, the most important data is concentrated in the four fields of merchants, products, orders, and promotional activities. Based on the current situation of the company's transaction scenarios, we abstract the master data in these four fields and model them in a unified manner as much as possible. Suitable for most trading scenarios. The following is a schematic diagram of the core data model structure of the order master data:

#After completing the unified data model, the next step is to import the existing heterogeneous data into the master data model. In the database, we use the method of reading the database binlog (mysql, sqlserver) for data processing to complete the initial data synchronization import. This is also the smallest and fastest implementation solution with the least intrusion into the business.

API standardization

After completing the construction of master data, we will start the construction of API standardization based on master data in the next step. API can be regarded as the nerve of the system. Quality APIs can connect a high-quality system together, and unified APIs can achieve system closure to a certain extent. To this end, we follow the principle of single responsibility, distinguish by domain, clarify boundaries, and atomize all underlying API functions to facilitate upstream users to flexibly assemble APIs to complete business logic. At the same time, we unify the parameter structure of the API and the structure of the response results. Unified error codes, unified publishing and calling based on API gateway, API data statistics monitoring, degradation, and current limiting realize unified management and control.

API reading and writing switching

With a standardized API, it is natural for business parties to use it to reflect the value of the API. In order to prevent the step from taking too big a step, Based on the importance and magnitude of the business, we also adopted a phased read-write solution to call and switch business one by one. This seemingly reasonable step also exposed many problems during the actual execution process: 1) In the case of strong read-write dependencies Scenario, for example: after the user places an order, he will immediately jump to the order details to view the order. At this time, when the write API switch is not completed, reading data through the read API will fail due to the delay in data synchronization. At this time, there is no way to follow the instructions. Read first and then write and switch in stages. The best way is to switch between reading and writing at the same time. 2) The method with the least impact on business switching is of course compatible with the parameters and return results of the original interface. If we force the business side to switch according to our standard API, it will inevitably bring switching costs and unnecessary negative effects to the business side. At this time, we naturally have to make some trade-offs from the other party's perspective. The method we adopt is to add an adaptation layer on top of the standard API for the conversion of old and new protocols, so that the business party only needs to switch the domain name and requested URL, and the other logic remains unchanged, maximizing the Friendly to the business side. 3) Since the underlying APIs we provide are all atomic, in actual scenarios, especially in projects where the front and back ends are separated, the front end is not willing to call the interface multiple times to obtain the same result. Faced with this situation, we are also A facade layer is added to the backend. Based on the underlying atomic API, an API that meets the business scenario is provided to the outside world, and is moderately compatible with differentiated interface logic. 4) Reading and writing switching cannot be achieved overnight. In this process, there will inevitably be scenarios where the main data API and the original business API coexist. Since all API entrances will be provided by us, we also adopt a routing mechanism, through the routing layer Different locations are forwarded, and all APIs are transparent to the caller. 5) In the actual API switching process, there is a special scenario. Because the system must be integrated in the end, forcing API switching for those functions that will be integrated later is actually a waste of resources, so we are also ahead of schedule. After making a pre-judgment, you can moderately avoid switching and wait for the functions to be integrated before switching the overall functions.

System Function Integration

After completing the API read and write switching, the functions based on the master data have basically completed the aggregation. At this time, it is necessary to systematically unify the common functions, such as : Unified merchant management backend, unified operation backend, unified C-end transaction experience, etc. The purpose of unified integration at the system level is to give users a unified operating interface and reflect the professionalism of the platform. In the process of system integration, we adopted the principle of "precipitation of commonalities and trade-offs of differences". For those common capabilities, such as product release, order list and other functions, we will abstract the common capabilities and provide unified unitized capabilities. The operation interface meets the usage requirements of various business parties. For functions that are unique to the business side, we will recommend that the business side implement them. For those functions that cannot yet form general capabilities but may become general capabilities in the future, we will use the fastest way to implement small versions online in accordance with the MVP principle. , with the iteration of business, function precipitation is continuously realized. During the entire system integration process, new requirements will inevitably arise while integrating the original system functions. Faced with this scenario, our approach is to develop the new and old systems simultaneously, which seems to increase the cost. In fact, the integration of new systems is beneficial. On the one hand, it will not affect the development of business and will not cause stagnation to the business due to technical integration. On the other hand, possible problems in the new system can be found out by comparing the old and new systems. , which will also be the best way to verify the functionality of the new integrated system. After completing most of the system integration work, the core e-commerce transaction links have been operational and have undergone long-term verification online, from merchant entry, product release, product display, order placement, payment, contract fulfillment, From after-sales to final settlement, problems encountered during the process are also resolved one by one. At this stage, we completed the integration of the three major trading systems in the company, and carried out the structural upgrade of the e-commerce platform’s flash sale system[8], and the structural upgrade of the coupon system, supporting the 818 in 2020-2021 Flash sales and coupon issuance scenarios for large-scale events such as parties, Double 11, and Double 12. In addition, we are also actively exploring the theory and industry practice of domain-driven model DDD, and have implemented it in the reconstruction of the invoice general database system[9], which also provides information for subsequent platform architecture upgrades. technical support.

Platform architecture stage

In the process of "approaching" the e-commerce business center to the business, the abstraction of the system The difficulty of construction has also increased exponentially, and a series of new problems have emerged: 1) With the end of the construction middle platform project and the evacuation of personnel, after integrating the logic of so many business lines, the code of the e-commerce business middle platform is full of A large number of conditional judgments, the development cost and test regression cost of each demand iteration are very high. How to isolate the logic between different businesses and reduce the coupling between businesses? 2) How to abstract the common capabilities of multiple business lines that have been connected to the e-commerce business platform to avoid duplication of construction? 3) When a new business is connected to the e-commerce business center, how can it be quickly assembled and launched based on existing capabilities and solutions to support rapid business iteration and innovation? 4) How can we use technical means to help improve the efficiency of daily work in product operations? To sum up, it is particularly important to abstract the common capabilities of business lines and the reusable design and implementation of common capabilities in the e-commerce business middle platform during the construction process. The business middle platform must achieve capability reuse and flexibility in order to make the middle platform Construction plays a role in reducing costs and increasing efficiency in the development process of enterprises. The system architecture must be upgraded, which leads to the platform architecture stage.

Platform Architecture Practice

What is platform architecture? It is necessary to separate the basic capabilities from the characteristic services of each business party, and to isolate the logic between services. The core of platformization is the openness of business abstract modeling and system architecture. Business abstraction solves 80% of common problems, and the openness of system architecture solves 20% of personalized problems. After referring to the "Modern Enterprise Architecture White Paper" solution given by ThoughtWorks[10] and the middle-end solutions of the industry's Internet companies Meituan[11] and Youzan[12] , we have given a solution suitable for Zhijia e-commerce platform: through domain-driven modeling, we abstract the common capabilities of multiple business lines in the e-commerce business and reserve expansion points, and then use service orchestration to combine the common capabilities. The principle is as shown in the figure: Each business running in the e-commerce business can be understood as: business identity, business process, and rules. The business process is realized through process service orchestration, and the extension point is realized through the extension point mechanism. In the entire transaction process, the entry of merchants and product release on the B-side are relatively common. The main process differences of different businesses are reflected in the order fulfillment before order collection and after payment. These processes often require customized development. For this reason, the entire solution The core lies in the order platform architecture design.

As shown in the figure: the entire order platform architecture is divided into four layers, from bottom to top:

• Infrastructure layer: Provides middleware such as storage, messaging, RPC, etc.
• Basic service layer: Basic services organized by domain, and domain services provide extension points for differences in different businesses.
• Business capability layer: Connect different domain services to form business capabilities that can be used externally, such as ordering, payment, etc.
• Business process layer: arranges business capabilities, forms order transaction processes, and completes order transaction processes.
• Business layer: Develop business identity, extension point implementation and business process configuration to achieve different business differences.

The entire order platform architecture upgrade practice process is summarized in the following points:

Business identity

The concept of business identity is the earliest Alibaba proposed that when the business platform provides services to various businesses at the same time, it needs to be able to distinguish the business identity elements of each business service request in order to provide differentiated and personalized services; therefore, it is necessary to establish the identity and characteristics of each business of the enterprise. Model and differentiate, the output of which is business identity. Business identity is unique. It is similar to an ID number and must be unique across the entire business. With the business identity, the business center can abstract the business process and business rules, and implement service routing and business monitoring based on the business identity. Secondly, business identity is similar to the concept of tenants in the SAAS system. Different business parties use business identities to isolate data permissions in the middle office. This ensures that the operations of each business can only see the data of their own business part.

For example, in the field of automobile e-commerce, we abstract business identity through three dimensions: people, goods, and fields. The human dimension includes whether you have a membership, whether you are a certified car owner, what value-added services have been activated, etc.; the goods dimension includes product type (complete vehicle, physical goods, virtual goods, etc.), delivery method (write-off, exchange, 4S store delivery) etc.; the dimensions of the field include online ordering, offline ordering, which online mall the order was placed at, which delivery store the order was placed at, the source of the product delivery channel, etc. Once a unique business identity is determined based on these dimensions, the business process for each transaction is determined.

Service orchestration

The e-commerce business center adopts a microservice architecture as a whole and splits the entire e-commerce system into merchants Center, user center, product center, promotion center, transaction center, fulfillment center, after-sales center. Each center is logically divided into two layers: capabilities with business attributes and basic capabilities without business attributes. The basic capability layer focuses on the entity attributes, behaviors, and events of the domain model, which will not change with the adjustment of business needs. It focuses on industry common behaviors, converges business models, and ensures the stability of basic services. Capabilities with business attributes are based on the basic capability layer through technical means such as service combination and process orchestration to form business-oriented solutions and complete the transformation from common business to personalized. There are two common approaches: one is to use hard coding. As the logic of different business lines continues to increase, the basic capabilities called by each business capability will become intricate and complex, making it difficult to configure and flexibly implement them. When requirements change, it is difficult for testers to assess the scope of impact of modifications, and the cost cycle of regression testing is long, making it difficult to achieve truly agile development and rapid business response. The second is to use service orchestration. Perform service composition through service orchestration of existing microservices, and then return the information required by the front desk at once. The capabilities of different business lines execute different processes. Through the graphical, XML, and JSON orchestration framework, the process can be made clear and code details can be shielded. There is no need to go into details about the benefits of service splitting, but to realize business value, it is not about the capabilities of a single service, but about coordinating all services to ensure the success of the enterprise's end-to-end business process. The business middle platform is the integration platform for enterprise business. The integration technology must loosely couple the applications and resources that make up the process. Otherwise, the logic of the process will be hard-coded into a specific technology platform, and changes may be costly, thus violating business process reuse. the entire goal.

Service orchestration framework

In the field of service orchestration, there are already many industrial solutions. We refer to Service orchestration based on API gateway [13], based on workflow system Orchestration frameworks Flowable and Activiti[14], Netflix Conductor[16] and Zeebe[17] based on microservice architecture orchestration framework. After analyzing the technical principles, we found that they all have certain deficiencies and cannot be applied to the service orchestration of the e-commerce business center. In the end, we chose Apache Camel [18] as the service The underlying engine of orchestration is developed for secondary encapsulation. Apache Camel was born in 2007. Around 2009, it became a top-level Apache project and was renamed Apache Camel. The latest version is 3.0. The advantage of Apache Camel is that it has more than 300 extension components in more than ten years since its release; the extension mechanism is also extremely convenient and flexible; application integration problems are solved through best practices available out of the box; It is based on an event-driven architecture and has good performance and throughput [19]. The following is a simple service process orchestration example:

The business middle office uses service orchestration technology. On the one hand, it can automatically identify transaction capabilities as a visual presentation of components to form a capability map; on the other hand, it can implement service processes based on these basic capabilities. The orchestration can quickly build all or part of the transaction process suitable for the business through drag and drop. It is similar to building blocks, reusing basic components and flexibly matching them, thereby realizing the reuse of e-commerce enterprise-level capabilities, saving development costs and quickly empowering business goals.

Extension point framework

The full name of extension point is Service Provider Interface, or SPI for short. It is a set of mechanisms provided by Java for loading and running interface implementation classes of third-party extensions. It is generally used in component replacement and framework expansion scenarios. SPI separates service interfaces and service implementations to achieve decoupling and improve application scalability. In programming, interface-oriented programming is used between modules without specific implementation class references, and application plug-in is achieved by dynamically loading implementation classes. The COLA framework is an extension point framework for application architecture proposed by Alibaba technical experts [20]. The extension of the COLA framework is implemented through annotations. The Extension extension annotations use three attributes: use case (useCase), business (bizId), and scenario (scenario) to identify the identity. Using the extension points of the COLA framework can support the logical isolation of different business identities at the code level, because different logic is scattered in different implementation classes, which conforms to the opening and closing principle of software design. When the application Spring context is initialized, the COLA framework begins to scan beans with Extension annotations for extension point registration, which are stored in a Map structure. The key is a string concatenation of useCase, bizId, and scenario, and the value is the bean. At runtime, the extension point implementation class is located through the business identity, and then the logic implemented by the extension point is executed. When locating the extension point implementation, it supports three-layer routing. First, the extension point will be found according to the useCase bizId scenario. If it is not found, it will be searched according to the useCase bizId scenario default value. If it has not been found, it will be searched according to the useCase bizId default value scenario default value. The specific principle is As shown in the figure:

#For simple business scenarios, there are not many non-functional requirements for high scalability and business isolation of the application system. However, as the same application system supports more businesses and business scenarios become more complex, it is necessary to provide unified expansion solutions at the architectural level to solidify the changing business rules. This not only helps to unify technical specifications, but also reduces hard coding. The IF-ELSE, strategy pattern, etc. bring complexity in understanding and consistency in specifications due to inconsistent developer levels. Through the extension point mechanism, the business center can manage different services from the business identity and framework levels like SAAS management tenants. Different business identities can extend predefined extension points in different scenarios. Alibaba's business middle office is also based on the idea of ​​expansion points, which realizes the separation and decoupling of core business logic and technical details, so that the shared business unit can support hundreds of business lines within the group.

Service Orchestration Extension Point Application Example

After verifying the functions, the scenarios of the e-commerce transaction system were classified. First, the ones with low user awareness and minimal impact on users even if problems occurred were selected. The node undergoes a reconstruction trial. If unpaid orders time out and the user cancels the order, the node will be closed. Take the scenario of user canceling an order as an example. Before the modification, the logic of order cancellation by users in each business is to modify the order status to canceled status and then execute the same process. The execution order of the process is hard-coded. The pseudo code is as shown in the figure:

After the modification, it has been carefully arranged according to the characteristics of each business. If the second-hand car business does not use coupons, then there is no need to have this link; in the general purpose of points In terms of capabilities, Wanlitong points are expanded. The pseudo code is as shown in the figure:

The hotel and air ticket business of the traveler business line does not have the traditional concept of commodity inventory, so there is no need to return commodity inventory. , but abstracts a new general capability: canceling supplier orders, and presets two extension points for canceling hotel supplier orders and canceling air ticket supplier orders. The pseudo code is shown in the figure:

The application effect of the entire system is obvious, mainly reflected in the performance improvement and human efficiency improvement. The performance improvement is mainly reflected in the shortened response time of the system. The TP99 improvement percentage of the production environment of the interface for canceling orders after modification is approximately 30%. The improvement of human efficiency is mainly reflected in the comparison of the time spent in testing to cancel orders and add new process nodes. Before modification, the codes between each business process are coupled. Modifying the process to add new nodes requires regression testing of the previous businesses. After modification, there is no need to conduct regression testing of each business.

Business configuration

In the practice of platform architecture, we uniformly extract the core configurations that affect business flow, and configure attribute values ​​according to business identities. Ensure the unified standard of the entire transaction process link, reduce frequent modifications to the core transaction link code, and allow different businesses to flexibly switch between different nodes in the same transaction process based on different attribute values. For example: whether the product is automatically pushed to the resource pool, whether an ID card is required to place an order, whether a clue is pushed if the payment is successful, whether the receipt has not been confirmed for more than N days, whether the receipt is automatically confirmed, etc. All configuration items are unified through the configuration management background. maintain. In addition, for all metadata in the e-commerce platform, including business identities, we have also unified management through the configuration management backend and provided a unified API to provide external query services.

Development Toolization

Starting from the multi-dimensional aspects of business and technology, we have developed various practical and convenient tools for common business problems or technical problems that arise in daily work. Small tools can improve work efficiency and quickly locate problems, such as: message distribution and retrieval tools; product discount price quick calculation tools; product display price comparison and monitoring tools; cache management tools; one-click downgrade tools, etc. As everyone's awareness of tools continues to increase, such small tools continue to appear and are brought together to form an indispensable toolbox for R&D personnel.

Data Visualization

The e-commerce system’s performance indicators, resource utilization indicators, call volume and other system-dimensional indicators can be uniformly monitored through the company’s cloud platform. The same applies to business data. We need to provide unified business data visualization tools to provide business parties with a reference for making relevant decisions. To this end, we developed a large-screen order visualization system in a real-time and offline manner. Through this system, changes in order volume can be monitored in real time according to multiple dimensions such as business line, order status, and region. If the order volume fluctuation within a fixed period of time exceeds our pre-configured threshold, a DingTalk message will be sent to promptly notify the business party of attention.

In addition, for offline data, we also perform data statistical analysis from multiple dimensions according to day, week, and month, and will eventually send it to you in the form of email and office APP messages. On the business side, the purpose of these methods is to achieve visual management of e-commerce data and provide business users with more convenient tools to conduct comprehensive management and control of e-commerce business.

Knowledge Precipitation

The team I am in is in the e-commerce field within the company It is a professional team that has accumulated a lot of experience in technology and product operations over the years. Throughout the construction process of the e-commerce middle platform, we also used these experiences and solutions to daily problems as a continuous accumulation of wealth. In the past, we used document management tools such as wiki to summarize them. In order to generate value from this knowledge, we have also begun to build our own e-commerce knowledge base system, and enter all content that can be used as knowledge accumulation into the knowledge base system according to different fields. The entire knowledge base provides quick retrieval and The positioning function can serve technical personnel and product operation personnel, further cultivate everyone's awareness of knowledge accumulation, and improve everyone's work efficiency.

end

Twenty years ago, the Internet first became popular in China. Information was displayed in the form of information, and there were almost no online transactions. Ten years ago, the Internet developed rapidly and consumers could purchase it on Taobao, Tmall, and JD.com. You can buy the goods you need or like on online malls for online transactions; now, various e-commerce forms are emerging, and it has become a trend that a hundred flowers are blooming, such as content e-commerce Xiaohongshu, interest e-commerce Douyin Kuaishou, social e-commerce WeChat Shang, Pinduoduo, etc., member e-commerce Tmall 88vip, JD plus, etc. These online transaction forms fully prove that e-commerce is an important part of monetizing traffic in the Internet field, and has become the water, electricity and coal of Internet enterprise infrastructure. The construction of an e-commerce middle platform is not only the construction of a technical system, but also a process of reshaping the organizational structure. However, as time goes by, the space for the growth of the value of the middle office will become increasingly narrow. This requires consciously looking for innovation points, breaking through the boundaries of the existing system, and thinking across borders, so we also began to get closer to the front office business. , actively carry out new business exploration and technical architecture upgrade.

Exploring New Retail

In exploring the business model of automobile e-commerce in the past, we found that the core pain point is the inability to bypass 4S stores to provide services. In recent years, Tesla and new domestic car manufacturing forces have emerged. The emerging direct sales model has broken the ecology of the 4S distribution system of traditional car companies in one fell swoop. The domestic car buying group has also become increasingly younger, allowing us to see the new retail of online car ordering and offline delivery. Patterns are becoming possible. By upgrading the existing capabilities of the e-commerce system, products support SKU selection, orders support combination payment and refund of large and small deposits, and a new delivery system has been added to provide services for the customized car business of industrial associations and the business of new automobile retail offline stores. support. In the future, we will continue to create an industry-aligned floating price model for new energy options and a service package model for optional product packages.

Architecture Upgrade

In the original e-commerce transaction order process, the external services designed were atomic services with relatively small granularity, which resulted in The access cost for the business side is relatively high, and the user experience is not very good. In the future, we will improve the product strength and operational efficiency of the business through technical means such as adding BFF layers, streamlining the call chain, and e-commerce access scaffolding.