Home > Java > Java Tutorial > body text

Method to implement Chinese word frequency statistics in Java (with code)

不言
Release: 2018-09-19 13:56:46
Original
5932 people have browsed it

 本篇文章给大家带来的内容是关于Java中实现中文词频统计的方法(附代码),有一定的参考价值,有需要的朋友可以参考一下,希望对你有所帮助。

昨日有个中文词频统计的需求, 百度一番后, 发现一大堆标题党文章, 讲的与内容严重不符, 这里就简单记录下自己实现的流程吧!

与英文单词的词频统计不同, 中文的难点在于如何分词, 不过好在有许多优秀的现成库供调用,这里就使用了 ansj_seg 插件.

首先添加依赖:

下载jar

访问 http://maven.nlpcn.org/org/ansj/ 最好下载最新版 ansj_seg/

同时下载nlp-lang.jar 需要和ansj_seg 配套..配套关系可以看jar包中的maven依赖,一般最新的ansj配最新的nlp-lang不会有错。

导入到eclipse ,开始你的程序吧。

maven
       
            org.ansj
            ansj_seg
            5.1.1
        
Copy after login

基本用法为:

 String str = "欢迎使用ansj_seg,(ansj中文分词)在这里如果你遇到什么问题都可以联系我.我一定尽我所能.帮助大家.ansj_seg更快,更准,更自由!" ;
 System.out.println(ToAnalysis.parse(str));
  欢迎/v,使用/v,ansj/en,_,seg/en,,,(,ansj/en,中文/nz,分词/n,),在/p,这里/r,如果/c,你/r,遇到/v,什么/r,问题/n,都/d,可以/v,联系/v,我/r,./m,我/r,一定/d,尽我所能/l,./m,帮助/v,大家/r,./m,ansj/en,_,seg/en,更快/d,,,更/d,准/a,,,更/d,自由/a,!
Copy after login

下面就贴上代码:

 public static void wordFrequency() throws IOException {
        Map map = new HashMap<>();

        String article = getString();
        String result = ToAnalysis.parse(article).toStringWithOutNature();
        String[] words = result.split(",");


        for(String word: words){
            String str = word.trim();
            // 过滤空白字符
            if (str.equals(""))
                continue;
            // 过滤一些高频率的符号
            else if(str.matches("[)|(|.|,|。|+|-|“|”|:|?|\\s]"))
                continue;
            // 此处过滤长度为1的str
            else if (str.length() < 2)
                continue;

            if (!map.containsKey(word)){
                map.put(word, 1);
            } else {
                int n = map.get(word);
                map.put(word, ++n);
            }
        }

        Iterator> iterator = map.entrySet().iterator();
        while (iterator.hasNext()){
            Map.Entry entry = iterator.next();
            System.out.println(entry.getKey() + ": " + entry.getValue());
        }
 
        List> list = new ArrayList<>();
        Map.Entry entry;
     
        while ((entry = getMax(map)) != null){
            list.add(entry);
        }

        System.out.println(Arrays.toString(list.toArray()));

    }


    /**
     * 找出map中value最大的entry, 返回此entry, 并在map删除此entry
     * @param map
     * @return
     */
    public static Map.Entry getMax(Map map){
        if (map.size() == 0){
            return null;
        }
        Map.Entry maxEntry = null;
        boolean flag = false;
        Iterator> iterator = map.entrySet().iterator();
        while (iterator.hasNext()){
            Map.Entry entry = iterator.next();
            if (!flag){
                maxEntry = entry;
                flag = true;
            }
            if (entry.getValue() > maxEntry.getValue()){
                maxEntry = entry;
            }
        }
        map.remove(maxEntry.getKey());
        return maxEntry;
    }

    /**
     * 从文件中读取待分割的文章素材.
   * 文件内容来自简书热门文章: https://www.jianshu.com/p/5b37403f6ba6
     * @return
     * @throws IOException
     */
    public static String getString() throws IOException {
        FileInputStream inputStream = new FileInputStream(new File("/home/as_/IdeaProjects/SpringMaven/article-txt"));
        BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
        StringBuilder strBuilder = new StringBuilder();

        String line;
        while((line = reader.readLine()) != null){
            strBuilder.append(line);
        }
        reader.close();
        inputStream.close();
        return strBuilder.toString();
    }
Copy after login

最后依旧附上图片:

The above is the detailed content of Method to implement Chinese word frequency statistics in Java (with code). For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Recommendations
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn:Public welfare online PHP training,Help PHP learners grow quickly!