Sina Weibo platform architecture with billions of users

The monthly active users (MAU) announced by Sina Weibo in March 2014 have reached 143 million, and the number of Weibo posts sent in the first minute of the New Year in 2014 reached 808,298. Such a huge user scale and business volume require high availability (HA) ), high concurrent access, and powerful backend system support with low latency.

The first generation architecture of the Weibo platform is the LAMP architecture. The database uses MyIsam, the backend uses PHP, and the cache is Memcache.

With the growth of application scale, the derived second-generation architecture has modularized, service-oriented and componentized business functions. The back-end system has been replaced from PHP to Java, gradually forming an SOA architecture, which has supported micro-enterprises for a long time. Business development of the gambling platform.

On this basis, after a long period of reconstruction, online operation, thinking and precipitation, the platform has formed a third-generation architecture system.

Let’s first take a look at Weibo’s core business diagram (below). Is it very complicated? But this is already a business diagram that cannot be simplified anymore. The third-generation technology system is to ensure the rapid, efficient and reliable release of new products and functions on Weibo’s core business.

The third-generation technology system of the Weibo platform uses the orthogonal decomposition method to build a model: in the horizontal direction, a typical three-level hierarchical model is adopted, namely the interface layer, service layer and resource layer; in the vertical direction, it is further subdivided into Business architecture, technical architecture, monitoring platform and service governance platform. The following is the overall architecture diagram of the platform:

As shown in the figure above, the orthogonal decomposition method decomposes the entire figure into 3*4=12 areas. Each area represents the intersection of a horizontal dimension and a vertical dimension, and accordingly defines the core functional points of this area, such as area 5. Mainly completes the technical architecture of the service layer.

The design principles of horizontal and vertical directions will be introduced in detail below, with special emphasis on the technical components in 4, 5, and 6 and their role in the entire architecture system.

The division of horizontal dimensions is very basic in the design of large and medium-sized Internet back-end business systems, and is reflected in every generation of the platform's technology system. Here is a brief introduction to pave the way for the subsequent extension of the vertical dimension:

Horizontal layering has a characteristic that the dependencies are from top to bottom. The services of the upper layer depend on the lower layer, and the services of the lower layer do not depend on the upper layer, creating a simple and direct dependency relationship.

Corresponding to the layered model, the servers in the Weibo system mainly include three types: front-end machine (providing API interface services), queue machine (processing uplink business logic, mainly data writing) and storage (mc, mysql, mcq, redis, HBase, etc.).

With the development and optimization of business architecture, platform research and development has implemented many excellent middleware products to support core businesses. These middleware are driven by business. As the technical components become more and more abundant, a complete platform technology framework is formed. , greatly improving the platform’s product research and development efficiency and business operation stability.

Different from the relationship between the upper layer and the lower layer in the horizontal direction, the vertical direction uses the technical framework as the foundation support point to drive and influence the business architecture, monitoring platform, and service governance platform on both sides. The following is an introduction to the core components.

The interface framework simplifies and standardizes business interface development work, packages common interface layer functions into the framework, and adopts Spring's aspect-oriented (AOP) design concept. The interface framework is secondary developed based on Jersey, defining the interface (url, parameters) based on annotation, with built-in Auth, frequency control, access log, and downgrade functions, supporting the interface layer monitoring platform and service governance, and also has automated Bean-json/xml Serialization.

The service layer mainly involves the RPC remote calling framework and the message queue framework. These are the two most widely used frameworks in the service layer of the Weibo platform.

Message queue provides a first-in-first-out communication mechanism. Within the platform, the most common scenario is to write the data landing operation asynchronously to the queue, and the queue handler reads and writes in batches DB, the asynchronous mechanism provided by the message queue speeds up the response time of the front-end machine. Secondly, the batch DB operation also indirectly improves the DB operation performance. In another application scenario, the platform provides search, big data, and commercial operation departments through the message queue. Real-time data.

The MCQ (SimpleQueue Service Over Memcache) message queue service that is widely used within the Weibo platform is based on the MemCache protocol. The message data is persistently written to BerkeleyDB. There are only two commands, get/set, and it is also very easy to monitor (stats queue). There is a rich client library, which has been running online for many years, and its performance is many times higher than the general MQ.

Weibo's Motan RPC service, the underlying communication engine uses the Netty network framework, the serialization protocol supports Hessian and Java serialization, the communication protocol supports Motan, http, tcp, mc, etc., Motan framework It is widely used internally. In terms of system robustness and service governance, there are relatively mature technical solutions. In terms of robustness, it implements High Availability and Load Balance strategies based on the Config configuration management service (supports flexible FailOver and FailFast HA strategies, and Load Balance strategies such as Round Robin, LRU, and Consistent Hash). In terms of service governance, it generates complete service call chain data, service request performance data, response time (Response Time), QPS, and standardized Error and Exception log information.

There are many resource layer frameworks, including Key-List DAL middleware that encapsulates MySQL and HBase, customized counting components, and Proxy that supports distributed MC and Redis. The industry has a lot of experience sharing in these aspects. I Here we share the object library and SSD Cache components of the platform architecture.

The object library supports convenient serialization and deserialization of object data in Weibo: during serialization, the objects in the JVM memory are serialized and written into HBase and a unique ObjectID is generated. When you need to access the object, read it through ObjectID. The object library supports any type of object and supports PB, JSON, and binary serialization protocols. The largest application scenario in Weibo uniformly defines the videos, pictures, and articles referenced in Weibo. For objects, a total of dozens of object types are defined, and the standard object metadata Schema is abstracted. The content of the object is uploaded to the object storage system (Sina S3), and the download address of Sina S3 is stored in the object metadata.

With the popularity of SSD hard drives, superior IO performance makes them increasingly used to replace traditional SATA and SAS disks. There are three common application scenarios: 1) Replacing mysql database Hard disk, there is currently no MySQL version optimized for SSD in the community. Even so, directly upgrading the SSD hard disk can bring about an 8 times increase in IOPS; 2) Replace the Redis hard disk to improve its performance; 3) Use it in CDN to speed up Static resource loading speed.

The Weibo platform applies SSD in distributed caching scenarios, extending the traditional Redis/MC + Mysql method to Redis/MC + SSD Cache + Mysql method. The SSD Cache is used as an L2 cache, which first reduces the MC/Redis The problems of high cost and small capacity also solve the database access pressure caused by penetrating DB.

As service scale and business become more and more complex, it is difficult even for business architects to accurately describe the dependencies between services, and the management and operation of services becomes more and more difficult. In this context, refer to Google's dapper and twitter's zipkin, the platform implements its own large-scale distributed tracking system WatchMan.

Like other large and medium-sized Internet applications, the Weibo platform is composed of numerous distributed components. After each HTTP request from a user through a browser or mobile client reaches the application server, it will pass through many business systems or system components and leave behind Footprint. However, these scattered data are of limited help in troubleshooting or process optimization. For such a typical cross-process/cross-thread scenario, it is particularly important to collect and analyze such logs. On the other hand, collecting performance data of each footprint and performing flow control or downgrading of each subsystem according to policies are also important factors in ensuring the high availability of the Weibo platform. It is necessary to be able to track the complete call link of each request; collect performance data of each service on the call link; be able to track all Errors and Exceptions in the system; and provide feedback by calculating performance data and comparing performance indicators (SLA). Into the control flow, Weibo’s Watchman system was born based on these goals.

A core principle of the system design is non-invasiveness: as a non-business component, it should intrude as little or not into other business systems as possible, maintaining transparency to users, which can greatly reduce the burden on developers and access barriers. Based on this consideration, all log collection points are distributed in technical framework middleware, including interface framework, RPC framework and other resource middleware.

WatchMan is a framework built by the technical team and is applied in all business scenarios. The operation base is based on this system to improve the monitoring platform. Business and operation and maintenance use this system together to complete distributed service governance, including service expansion and reduction, service degradation, traffic Switching, service publishing and grayscale.

Nowadays, the technical framework plays an increasingly important role in the platform, driving the platform’s technology upgrade, business development, and system operation and maintenance services. Due to space limitations, this article does not introduce it. The design of core middleware will be introduced in the future. Principles and system architecture.

Wei Xiangjun (@伟向军_微博) graduated from Beijing University of Posts and Telecommunications and is currently a Weibo platform architect. He has been engaged in technology research and development work at Microsoft, Kingsoft Cloud, and Sina Weibo, focusing on system architecture design, audio and video Communication systems, distributed file systems, data mining and other fields.