What are the big data tools and frameworks that Java developers must know?

1. MongoDB - the most popular, cross-platform, document-oriented database.

MongoDB is a database based on distributed file storage, written in C language. Designed to provide scalable, high-performance data storage solutions for web applications. Application performance depends on database performance. MongoDB is the most feature-rich among non-relational databases and is most like a relational database. With the release of MongoDB 3.4, its application scenario capabilities have been further expanded.

The core advantages of MongoDB are its flexible document model, highly available replica sets, and scalable sharded clusters. You can try to understand MongoDB from several aspects, such as real-time monitoring of MongoDB tools, memory usage and page faults, number of connections, database operations, replication sets, etc.

2. Elasticsearch - a distributed RESTful search engine built for the cloud.

ElasticSearch is a search server based on Lucene. It provides a distributed multi-user capable full-text search engine based on a RESTful web interface. Elasticsearch, developed in Java and released as open source under the terms of the Apache license, is a popular enterprise-level search engine.

ElasticSearch is not only a full-text search engine, but also a distributed real-time document storage, in which each field is indexed data and can be searched; it is also a distributed search engine with real-time analysis capabilities. And it can be expanded to hundreds of servers to store and process petabytes of data. ElasticSearch uses Lucene to complete its indexing function at the bottom level, so many of its basic concepts originate from Lucene.

3. Cassandra - an open source distributed database management system, originally developed by Facebook, designed to handle large amounts of data on many commodity servers, provide high availability, and no Single point of failure.

Apache Cassandra is an open source distributed NoSQL database system. It combines the data model of Google BigTable with the fully distributed architecture of Amazon Dynamo. It was open sourced in 2008. Since then, due to its good scalability, Cassandra has been adopted by Web 2.0 websites such as Digg and Twitter, and has become a popular distributed structured data storage solution.

Because Cassandra is written in Java, it can theoretically run on machines with JDK6 and above. The officially tested JDKs include OpenJDK and Sun's JDK. The operation commands of Cassandra are similar to the relational databases we usually operate. For friends who are familiar with MySQL, the operation will be easy to get started.

4. Redis - open source (BSD licensed) in-memory data structure storage, used as a database, cache and message broker.

Redis is an open source log-type Key-Value database written in ANSI C language, supports network, can be memory-based and persistent, and provides APIs in multiple languages. Redis has three main features that set it apart from many other competitors: Redis is a database that saves data entirely in memory, using disk only for persistence purposes; Redis has a relatively rich set of data types compared to many key-value data storage systems ; Redis can copy data to any number

5. Hazelcast - an open source memory data grid based on Java.

Hazelcast is an in-memory data grid that provides Java programmers with mission-critical transactions and trillion-scale memory applications. Although Hazelcast does not have a so-called "Master", it still has a Leader node (the oldest member). This concept is similar to the Leader in ZooKeeper, but the implementation principle is completely different. At the same time, the data in Hazelcast is distributed, and each member holds part of the data and corresponding backup data, which is also different from ZooKeeper.

The application convenience of Hazelcast is loved by developers, but if you want to put it into use, you need to consider it carefully.

6. EHCache - a widely used open source Java distributed cache. Mainly for general cache, Java EE and lightweight containers.

EhCache is a pure Java in-process caching framework, which is fast and capable. It is the default CacheProvider in hibernate. The main features are: fast and simple, with multiple caching strategies; cached data has two levels, memory and disk, so there is no need to worry about capacity issues; cached data will be written to disk during the restart of the virtual machine; it can be accessed through RMI and pluggable API Distributed caching in other ways; has a listening interface for cache and cache managers; supports multiple cache manager instances, as well as multiple cache areas for one instance; provides Hibernate cache implementation.

7. Hadoop - an open source software framework written in Java, used for distributed storage, and for very large data users can develop distributed programs without understanding the underlying details of distribution.

Make full use of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. The core design of the Hadoop framework is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides calculation for massive data.

8. Solr - an open source enterprise search platform, written in Java, from the Apache Lucene project.

Solr is an independent enterprise-level search application server that provides an API interface similar to Web-service. Users can submit XML files in a certain format to the search engine server through http requests to generate indexes; they can also make search requests through Http Get operations and get returned results in XML format.

Like ElasticSearch, it is also based on Lucene, but it extends it to provide a richer query language than Lucene, while being configurable, scalable and optimizing query performance.

9. Spark - the most active project in the Apache Software Foundation, is an open source cluster computing framework.

Spark is an open source cluster computing environment similar to Hadoop, but there are some differences between the two that make Spark superior in certain workloads. In other words That being said, Spark enables in-memory distributed datasets, which in addition to being able to provide interactive queries, can also optimize iterative workloads.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala are tightly integrated, and Scala can operate as easily as local collection objects.

10. Memcached - a general distributed memory cache system.

Memcached is a distributed caching system that was originally developed by Danga Interactive for LiveJournal, but is used by many software (such as MediaWiki). As a high-speed distributed cache server, Memcached has the following characteristics: simple protocol, event processing based on libevent, and built-in memory storage.

11. Apache Hive--Provides a SQL-like layer on top of Hadoop.

Hive is a data warehouse platform based on Hadoop. Through hive, ETL work can be easily performed. hive defines a query language similar to SQL, which can convert user-written SQL into corresponding Mapreduce programs for execution based on Hadoop. Currently, Apache Hive 2.1.1 version has been released.

12. Apache Kafka - a high-throughput, distributed subscription messaging system originally developed by LinkedIn.

Apache Kafka is an open source messaging system project written in Scala. The goal of this project is to provide a unified, high-throughput, low-latency platform for processing real-time data. Kafka maintains messages differentiated by classes, called topics. Producers publish messages to Kafka topics, and consumers register with topics and receive messages published to these topics.

13. Akka - A toolkit for building highly concurrent, distributed and resilient message-driven applications on the JVM.

Akka is a library written in Scala that simplifies writing fault-tolerant, highly scalable Java and Scala actor model applications. It has been successfully used in the telecommunications industry, and the system almost never goes down.

14. HBase - open source, non-relational, distributed database, modeled using Google's BigTable, written in Java, and runs on HDFS.

Different from commercial big data products such as FUJITSU Cliq, HBase is an open source implementation of Google Bigtable. Similar to Google Bigtable, which uses GFS as its file storage system, HBase uses Hadoop HDFS as its file storage system; Google runs MapReduce for processing. HBase also uses Hadoop MapReduce to process the massive data in Bigtable; Google Bigtable uses Chubby as a collaborative service, and HBase uses Zookeeper as a counterpart.

15. Neo4j - an open source graph database implemented in Java.

Neo4j is a high-performance NOSQL graph database that stores structured data on the network instead of in tables. It is an embedded, disk-based, fully transactional Java persistence engine.

The above is the detailed content of What are the big data tools and frameworks that Java developers must know?. For more information, please follow other related articles on the PHP Chinese website!