What is the big data application development process?

Big data project development steps:

Step one: Requirements: data input and data output;

The second step: data volume, processing efficiency, reliability, maintainability, simplicity;

The third step: data modeling;

The fourth step: architecture design: how the data Come in, how to display the output, the most important thing is the architecture for processing outflow data;

The fifth step: Think again about the interaction between the big data system and the enterprise IT system;

The sixth step:Finally Determine choices, specifications, etc.;

Step 7: Write basic service code based on data modeling;

Step 8:Formally write the first module;

Ninth Step: Implement other modules, and complete testing and debugging;

Step 10: Testing and acceptance;

Big data process:

From a process perspective, the entire big data processing can be divided into 4 main steps.

The first step is the collection and storage of data;

The second step is to conduct exploratory research on the data through data analysis technology, including the elimination of irrelevant data, that is, data cleaning, and finding data The model explores the value of the data;

The third step is to select and develop data analysis algorithms to model the data based on basic data analysis. Extracting valuable information from data is actually the real Alibaba Cloud big data learning process. This will involve many algorithms and technologies, such as machine learning algorithms, etc.;

The last step is the deployment and application of the model, that is, applying the researched model to the production environment.

1) Data collection: Customize and develop the collection program, or use the open source framework flume

2) Data preprocessing: Customize and develop the mapreduce program to run on the hadoop cluster

3) Data Warehouse technology: Hive based on hadoop

4) Data export: sqoop data import and export tool based on hadoop

5) Data visualization: custom development of web programs or use of products such as kettle

The above is the detailed content of What is the big data application development process?. For more information, please follow other related articles on the PHP Chinese website!