Hadoop pseudo-distributed cluster construction
Software preparation
I'm using a virtual machine with CentOS-6.6, with the host name repo. Refer to the steps to install a Linux virtual machine in Windows, I installed JDK in that virtual machine, refer to the guide to installing JDK in Linux. In addition, the virtual machine is configured with a key-free login itself, and the settings for configuring key-free login between each virtual machine are referenced. The download address of the Hadoop installation package is: //m.sbmmt.com/link/8485694bae96aebc7c4fe6119599d0e0 .
-
Upload the Hadoop installation package to the server and decompress it
[root@repo ~]# tar zxvf hadoop-2.6.5.tar.gz -C /opt/apps/
-
Configure environment variables
# You can directly locate the last line of the file [root@repo hadoop-2.6.5]# vi /etc/profile export HADOOP_HOME=/opt/apps/hadoop-2.6.5 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin [root@repo hadoop-2.6.5]# . /etc/profile
-
Modify the three configuration files of hadoop-env.sh, mapred-env.sh, and yarn-env.sh, and add JAVA_HOME
[root@repo hadoop]# pwd /opt/apps/hadoop-2.6.5/etc/hadoop [root@repo hadoop]# vi hadoop-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73 [root@repo hadoop]# vi mapred-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73 [root@repo hadoop]# vi yarn-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_73
-
Modify the core-site.xml and hdfs-site.xml configuration files to configure pseudo-distributed related content
[root@repo hadoop]# vi core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://repo:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/var/hadoop/pseudo</value> </property> </configuration> [root@repo hadoop]# vi hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>repo:50090</value> </property> </configuration>
-
Modify the slaves configuration file and specify the node where the DataNode is located
[root@repo hadoop]# vi slaves repo
-
Format file system
[root@repo hadoop]# hadoop namenode --format # Success message17/09/16 21:17:11 INFO common.Storage: Storage directory /var/hadoop/pseudo/dfs/name has been successfully formatted.
-
Start HDFS and YARN
[root@repo hadoop]# start-dfs.sh Starting namesnodes on [repo] repo: starting namenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-namenode-repo.out repo: starting datanode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-datanode-repo.out Starting secondary namesodes [repo] repo: starting secondarynamenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-repo.out [root@repo hadoop]# start-yarn.sh starting yarn daemons starting resourcemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-resourcemanager-repo.out repo: starting nodemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-nodemanager-repo.out [root@repo hadoop]# jps 4368 Jps 3957 ResourceManager 3512 NameNode 3641 DataNode 4058 NodeManager 3805 SecondaryNameNode
Visit the WEB page
Build successfully!
The above is the detailed content of Hadoop pseudo-distributed cluster construction. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undress AI Tool
Undress images for free

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

When Windows cannot detect a second monitor, first check whether the physical connection is normal, including power supply, cable plug-in and interface compatibility, and try to replace the cable or adapter; secondly, update or reinstall the graphics card driver through the Device Manager, and roll back the driver version if necessary; then manually click "Detection" in the display settings to identify the monitor to confirm whether it is correctly identified by the system; finally check whether the monitor input source is switched to the corresponding interface, and confirm whether the graphics card output port connected to the cable is correct. Following the above steps to check in turn, most dual-screen recognition problems can usually be solved.

The key steps to install PHP on Windows include: 1. Download the appropriate PHP version and decompress it. It is recommended to use ThreadSafe version with Apache or NonThreadSafe version with Nginx; 2. Configure the php.ini file and rename php.ini-development or php.ini-production to php.ini; 3. Add the PHP path to the system environment variable Path for command line use; 4. Test whether PHP is installed successfully, execute php-v through the command line and run the built-in server to test the parsing capabilities; 5. If you use Apache, you need to configure P in httpd.conf

The rise of meme coins reflects the key role of community power and social media influence in the cryptocurrency market. 1. Dogecoin was originally a satirical joke and was born in 2013; 2. Driven by tweets from celebrities such as Elon Musk, the attention soared; 3. The market value once reached tens of billions of dollars, becoming a mainstream digital asset. Shiba Inu Coin is positioned as a "dogcoin killer" and has rapidly risen through community-driven strategies, building a decentralized exchange ShibaSwap, and relies on low-priced units to attract a large number of users to participate. Its success also depends on circulation guarantees on mainstream platforms such as Binance, Coinbase, and OKX. The core driving forces of meme coins include: 1. Viral transmission mechanism, rapid spread of information; 2. Enhanced sense of community belonging

When encountering the problem of printing task stuck, clearing the print queue and restarting the PrintSpooler service is an effective solution. First, open the "Device and Printer" interface to find the corresponding printer, right-click the task and select "Cancel" to clear a single task, or click "Cancel all documents" to clear the queue at one time; if the queue is inaccessible, press Win R to enter services.msc to open the service list, find "PrintSpooler" and stop it before starting the service. If necessary, you can manually delete the residual files under the C:\Windows\System32\spool\PRINTERS path to completely solve the problem.

The "reverse reference" in the currency circle, as the name suggests, refers to those reference objects whose views or operations are often opposite to the actual market trend. When such people or groups are extremely optimistic, the market may face a decline; when they are extremely pessimistic, the market may instead rebound. This is not to say that these people deliberately provide wrong signals, but that their judgments may deviate from the mainstream trends in the market, or that their operating behavior happens to be a catalyst for market reversal in a specific situation.

When encountering the problem of "Windowscan'tconnecttothisnetwork", you should first check whether the network name and password are correct, including case, spaces and special characters. If it is a Chinese WiFi name, try to change it to English, and then forget the network and reconnect; secondly, check whether the router limits the number of devices or enables MAC address filtering, and restart the router or log in to the background to adjust the settings; thirdly, check the system network settings and drivers, update the wireless network card driver, reset the network settings or confirm that the IP is automatically obtained; finally check whether the network authentication method is compatible, especially in enterprise or campus networks, you need to consult the administrator to configure the correct EAP protocol and installation certificate.

Enable HSTS to force browsers to access websites through HTTPS, improving security. 1. To enable HTTPS in Apache, you must first configure HTTPS, and then add Strict-Transport-Security response header in the site configuration file or .htaccess; 2. To configure max-age (such as 31536000 seconds), includeSubDomains and preload parameters; 3. Make sure that the mod_headers module is enabled, otherwise run sudoa2enmodheaders and restart Apache; 4. You can optionally submit to the HSTSPreload list, but it must satisfy that both the main site and the subdomain support HTTPS.

Finding airdrop opportunities for cryptocurrency projects is the way many participants want to acquire tokens for early-stage projects. These airdrops are usually a means for project parties to promote brand, community construction, or inspire early users. To find this information effectively, you need to rely on multiple reliable channels and methods.
