How does a Java crawler implement Jsoup using the dom method to traverse the Document object?-javaTutorial-php.cn

先给出网页地址：

https://wall.alphacoders.com/featured.php?lang=Chinese

Copy after login

主要步骤：

利用Jsoup的connect方法获取Document对象

String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get();

Copy after login

内容过长，就不再显示。

我们以这部分为例：

 About Us
 FAQ
 Privacy Policy
 Terms Of Service
 Acceptable Use
 Etiquette
 Advertise With Us
 Change Consent

Copy after login

我们先找到所有的ul：

Elements elements = doc.getElementsByTag("ul");

Copy after login

输出如下：

   提交 
   精美奖品 
 
     中文  
   登录 
   注册 
 
  < 上一页
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 ...
 319
 下一页 >
 
  < 上一页
 1
 2
 3
 4
 5
 6
 7
 ...
 319
 下一页 >
 
  << 
 < 上一页
 下一页 >
  >>
 
  1
 2
 3
 4
 5
 6
 7
 
  About Us
 FAQ
 Privacy Policy
 Terms Of Service
 Acceptable Use
 Etiquette
 Advertise With Us
 Change Consent

Copy after login

可以发现class为"nav nav-pills"的只有一个，我们找到它：

Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } }

Copy after login

循环遍历这个ul，输出其中每一个li里每一个a的href和rel属性：

Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } }

Copy after login

最终结果：

href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow

完整代码：

import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; import java.io.IOException; import org.jsoup.Jsoup; /** * @ClassName: Jsoup_Test * @description: * @author: KI * @Date: 2020年8月17日 下午8:15:14 */ public class Jsoup_Test { public static void main(String[] args) throws IOException { // TODO 自动生成的方法存根 String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get(); System.out.println(doc); Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } } System.out.println(tempElement); Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } } } }

Copy after login

The above is the detailed content of How does a Java crawler implement Jsoup using the dom method to traverse the Document object?. For more information, please follow other related articles on the PHP Chinese website!