java - 迁移数据效率问题
巴扎黑
巴扎黑 2017-04-18 10:11:27
0
3
383

问题描述如下

100万数据从A机房的一个表迁移到B机房的一个表,两个表结构不一样。我的实现方法如下:查出数据来,java层面进行数据的转义,然后开了30个线程,进行分页同步数据。即每次查询500条数据,然后插入B机房的表。线程池中有30个线程。

现在的问题是:

迁移到效率很慢,很慢。17万数据就用了一个多小时,还没有跑完数据,请问有什么方案可以提高效率的么???

巴扎黑
巴扎黑

reply all (3)
洪涛

There are 3 places where there may be efficiency issues

  1. Batch Update is not used when calling JDBC, resulting in low performance

  2. Checking 500 items each time is too few, checking a few thousand items each time should be fine

  3. The tables in computer room B have indexes and constraints. These should be turned off or dropped during migration, and then turned on again after the migration is completed

    巴扎黑

    Your speed is abnormal. Is the IO full? Or is there a problem with the network? I used to use Kettle to migrate millions of data an hour. Because I was used to using ETL tools, I never wrote a program for data migration.

      巴扎黑

      Ugh. . , why use java? Isn't 1 million data a lot? Wouldn't it be faster to back it up and then dump it on the same machine?

        Latest Downloads
        More>
        Web Effects
        Website Source Code
        Website Materials
        Front End Template
        About us Disclaimer Sitemap
        php.cn:Public welfare online PHP training,Help PHP learners grow quickly!