Home System Tutorial Windows Series Hadoop pseudo-distributed cluster construction

Hadoop pseudo-distributed cluster construction

May 07, 2025 pm 04:45 PM
linux centos windows apache hadoop red

Software preparation

I'm using a virtual machine with CentOS-6.6, with the host name repo. Refer to the steps to install a Linux virtual machine in Windows, I installed JDK in that virtual machine, refer to the guide to installing JDK in Linux. In addition, the virtual machine is configured with a key-free login itself, and the settings for configuring key-free login between each virtual machine are referenced. The download address of the Hadoop installation package is: //m.sbmmt.com/link/8485694bae96aebc7c4fe6119599d0e0 .

  1. Upload the Hadoop installation package to the server and decompress it

     [root@repo ~]# tar zxvf hadoop-2.6.5.tar.gz -C /opt/apps/
  2. Configure environment variables

     # You can directly locate the last line of the file [root@repo hadoop-2.6.5]# vi /etc/profile
    export HADOOP_HOME=/opt/apps/hadoop-2.6.5
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    [root@repo hadoop-2.6.5]# . /etc/profile
  3. Modify the three configuration files of hadoop-env.sh, mapred-env.sh, and yarn-env.sh, and add JAVA_HOME

     [root@repo hadoop]# pwd
    /opt/apps/hadoop-2.6.5/etc/hadoop
    [root@repo hadoop]# vi hadoop-env.sh
    export JAVA_HOME=/usr/local/jdk1.8.0_73
    [root@repo hadoop]# vi mapred-env.sh
    export JAVA_HOME=/usr/local/jdk1.8.0_73
    [root@repo hadoop]# vi yarn-env.sh
    export JAVA_HOME=/usr/local/jdk1.8.0_73
  4. Modify the core-site.xml and hdfs-site.xml configuration files to configure pseudo-distributed related content

     [root@repo hadoop]# vi core-site.xml
    <configuration>
    <property>
     <name>fs.defaultFS</name>
     <value>hdfs://repo:9000</value>
    </property>
    <property>
     <name>hadoop.tmp.dir</name>
     <value>/var/hadoop/pseudo</value>
    </property>
    </configuration>
    [root@repo hadoop]# vi hdfs-site.xml
    <configuration>
    <property>
     <name>dfs.replication</name>
     <value>1</value>
    </property>
    <property>
     <name>dfs.namenode.secondary.http-address</name>
     <value>repo:50090</value>
    </property>
    </configuration>
  5. Modify the slaves configuration file and specify the node where the DataNode is located

     [root@repo hadoop]# vi slaves
    repo
  6. Format file system

     [root@repo hadoop]# hadoop namenode --format
    # Success message17/09/16 21:17:11 INFO common.Storage: Storage directory /var/hadoop/pseudo/dfs/name has been successfully formatted.
  7. Start HDFS and YARN

     [root@repo hadoop]# start-dfs.sh
    Starting namesnodes on [repo]
    repo: starting namenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-namenode-repo.out
    repo: starting datanode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-datanode-repo.out
    Starting secondary namesodes [repo]
    repo: starting secondarynamenode, logging to /opt/apps/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-repo.out
    [root@repo hadoop]# start-yarn.sh
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-resourcemanager-repo.out
    repo: starting nodemanager, logging to /opt/hadoop-2.7.4/logs/yarn-root-nodemanager-repo.out
    [root@repo hadoop]# jps
    4368 Jps
    3957 ResourceManager
    3512 NameNode
    3641 DataNode
    4058 NodeManager
    3805 SecondaryNameNode
  8. Visit the WEB page Hadoop pseudo-distributed cluster constructionHadoop pseudo-distributed cluster construction Build successfully!

The above is the detailed content of Hadoop pseudo-distributed cluster construction. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undress AI Tool

Undress AI Tool

Undress images for free

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Peak: How To Revive Players
1 months ago By DDD
PEAK How to Emote
3 weeks ago By Jack chen

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to fix second monitor not detected in Windows? How to fix second monitor not detected in Windows? Jul 12, 2025 am 02:27 AM

When Windows cannot detect a second monitor, first check whether the physical connection is normal, including power supply, cable plug-in and interface compatibility, and try to replace the cable or adapter; secondly, update or reinstall the graphics card driver through the Device Manager, and roll back the driver version if necessary; then manually click "Detection" in the display settings to identify the monitor to confirm whether it is correctly identified by the system; finally check whether the monitor input source is switched to the corresponding interface, and confirm whether the graphics card output port connected to the cable is correct. Following the above steps to check in turn, most dual-screen recognition problems can usually be solved.

How to Install PHP on Windows How to Install PHP on Windows Jul 15, 2025 am 02:46 AM

The key steps to install PHP on Windows include: 1. Download the appropriate PHP version and decompress it. It is recommended to use ThreadSafe version with Apache or NonThreadSafe version with Nginx; 2. Configure the php.ini file and rename php.ini-development or php.ini-production to php.ini; 3. Add the PHP path to the system environment variable Path for command line use; 4. Test whether PHP is installed successfully, execute php-v through the command line and run the built-in server to test the parsing capabilities; 5. If you use Apache, you need to configure P in httpd.conf

Meme Coin Mania: The Power of Dogecoin, Shiba Inu and Community Hype Meme Coin Mania: The Power of Dogecoin, Shiba Inu and Community Hype Jul 10, 2025 pm 07:48 PM

The rise of meme coins reflects the key role of community power and social media influence in the cryptocurrency market. 1. Dogecoin was originally a satirical joke and was born in 2013; 2. Driven by tweets from celebrities such as Elon Musk, the attention soared; 3. The market value once reached tens of billions of dollars, becoming a mainstream digital asset. Shiba Inu Coin is positioned as a "dogcoin killer" and has rapidly risen through community-driven strategies, building a decentralized exchange ShibaSwap, and relies on low-priced units to attract a large number of users to participate. Its success also depends on circulation guarantees on mainstream platforms such as Binance, Coinbase, and OKX. The core driving forces of meme coins include: 1. Viral transmission mechanism, rapid spread of information; 2. Enhanced sense of community belonging

How to clear the print queue in Windows? How to clear the print queue in Windows? Jul 11, 2025 am 02:19 AM

When encountering the problem of printing task stuck, clearing the print queue and restarting the PrintSpooler service is an effective solution. First, open the "Device and Printer" interface to find the corresponding printer, right-click the task and select "Cancel" to clear a single task, or click "Cancel all documents" to clear the queue at one time; if the queue is inaccessible, press Win R to enter services.msc to open the service list, find "PrintSpooler" and stop it before starting the service. If necessary, you can manually delete the residual files under the C:\Windows\System32\spool\PRINTERS path to completely solve the problem.

What does the counter-referential meaning of the currency circle? Why do some people specifically operate in reverse? Market sentiment indicators What does the counter-referential meaning of the currency circle? Why do some people specifically operate in reverse? Market sentiment indicators Jul 10, 2025 pm 09:27 PM

The "reverse reference" in the currency circle, as the name suggests, refers to those reference objects whose views or operations are often opposite to the actual market trend. When such people or groups are extremely optimistic, the market may face a decline; when they are extremely pessimistic, the market may instead rebound. This is not to say that these people deliberately provide wrong signals, but that their judgments may deviate from the mainstream trends in the market, or that their operating behavior happens to be a catalyst for market reversal in a specific situation.

Windows can't connect to this network Windows can't connect to this network Jul 10, 2025 pm 12:40 PM

When encountering the problem of "Windowscan'tconnecttothisnetwork", you should first check whether the network name and password are correct, including case, spaces and special characters. If it is a Chinese WiFi name, try to change it to English, and then forget the network and reconnect; secondly, check whether the router limits the number of devices or enables MAC address filtering, and restart the router or log in to the background to adjust the settings; thirdly, check the system network settings and drivers, update the wireless network card driver, reset the network settings or confirm that the IP is automatically obtained; finally check whether the network authentication method is compatible, especially in enterprise or campus networks, you need to consult the administrator to configure the correct EAP protocol and installation certificate.

How to enable HTTP Strict Transport Security (HSTS) in Apache? How to enable HTTP Strict Transport Security (HSTS) in Apache? Jul 13, 2025 am 01:12 AM

Enable HSTS to force browsers to access websites through HTTPS, improving security. 1. To enable HTTPS in Apache, you must first configure HTTPS, and then add Strict-Transport-Security response header in the site configuration file or .htaccess; 2. To configure max-age (such as 31536000 seconds), includeSubDomains and preload parameters; 3. Make sure that the mod_headers module is enabled, otherwise run sudoa2enmodheaders and restart Apache; 4. You can optionally submit to the HSTSPreload list, but it must satisfy that both the main site and the subdomain support HTTPS.

How to check the airdrop of currency circle projects? How to avoid fake airdrop scams? How to check the airdrop of currency circle projects? How to avoid fake airdrop scams? Jul 10, 2025 pm 09:12 PM

Finding airdrop opportunities for cryptocurrency projects is the way many participants want to acquire tokens for early-stage projects. These airdrops are usually a means for project parties to promote brand, community construction, or inspire early users. To find this information effectively, you need to rely on multiple reliable channels and methods.

See all articles