What is the result of the emergence of big data?

#The emergence of big data is the result of the accumulation of quantitative changes in data to qualitative changes.

Big data (big data), a term in the IT industry, refers to a collection of data that cannot be captured, managed and processed with conventional software tools within a certain time range. It requires new processing models to have more accurate data. Massive, high-growth and diversified information assets with strong decision-making, insight and process optimization capabilities.

Reasons for the emergence of big data:

Most technological breakthroughs come from actual product needs. Big data was originally born in Google's search engine. With the development of the web 2.0 era, the amount of data on the Internet has shown explosive growth. In order to meet the needs of information search, there is a very strong need for large-scale data storage.

Based on cost considerations, it is increasingly impractical to upgrade hardware to search large amounts of data, so Google proposed a reliable software-based file storage system GFS, which uses ordinary PCs to parallelize Support large-scale storage.

The stored data is of low value, and only the data can be processed to meet actual application needs. So Google created the MapReduce computing model, which can use the power of clusters to convert complex operations into Split it into each ordinary PC, and after the calculation is completed, the final calculation results are obtained by summarizing, so that better computing power can be obtained by directly increasing the number of machines.

With GFS and MapReduce, the storage and operation of files have been solved, but new problems have arisen. GFS's random reading and writing capabilities are very poor, and Google needs a database to store formatted data. The problem that was originally solved by a stand-alone database was a tragedy for Google, so Google, the magic tool, developed another one. The BigTable system uses the GFS file storage system and a distributed lock management system Chubby to design a columnar database system like BigTable.

The above is the detailed content of What is the result of the emergence of big data?. For more information, please follow other related articles on the PHP Chinese website!