Home>Article>Backend Development> Big data batch insertion of several databases in c# (SqlServer, Oracle, SQLite and MySql)
I only knew that SqlServer supports batch insertion of data, but I did not know that Oracle, SQLite and MySql also support it. However, Oracle needs to use the Orace.DataAccess driver. Today I will post several batch insertion solutions for databases.
First of all, there is a plug-in service interface IBatcherProvider in IProvider for batch insertion. This interface has been mentioned in the previous article.
////// 提供数据批量处理的方法。 /// public interface IBatcherProvider : IProviderService { ////// 将 /// 要批量插入的的数据批量插入到数据库中。 /// 。 /// 每批次写入的数据量。 void Insert(DataTable dataTable, int batchSize = 10000); }
1. SqlServer data batch insertion
SqlServer batch insertion is very simple, just use SqlBulkCopy. The following is the implementation of this class:
////// 为 System.Data.SqlClient 提供的用于批量操作的方法。 /// public sealed class MsSqlBatcher : IBatcherProvider { ////// 获取或设置提供者服务的上下文。 /// public ServiceContext ServiceContext { get; set; } ////// 将 /// 要批量插入的的数据批量插入到数据库中。 /// 。 /// 每批次写入的数据量。 public void Insert(DataTable dataTable, int batchSize = 10000) { Checker.ArgumentNull(dataTable, "dataTable"); if (dataTable.Rows.Count == 0) { return; } using (var connection = (SqlConnection)ServiceContext.Database.CreateConnection()) { try { connection.TryOpen(); //给表名加上前后导符 var tableName = DbUtility.FormatByQuote(ServiceContext.Database.Provider.GetService (), dataTable.TableName); using (var bulk = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, null) { DestinationTableName = tableName, BatchSize = batchSize }) { //循环所有列，为bulk添加映射 dataTable.EachColumn(c => bulk.ColumnMappings.Add(c.ColumnName, c.ColumnName), c => !c.AutoIncrement); bulk.WriteToServer(dataTable); bulk.Close(); } } catch (Exception exp) { throw new BatcherException(exp); } finally { connection.TryClose(); } } } }
No transactions are used above. Using transactions will have a certain impact on performance. If you want to use transactions, you can set SqlBulkCopyOptions.UseInternalTransaction.
2. Oracle data batch insertion
System.Data.OracleClient does not support batch insertion, so you can only use the Oracle.DataAccess component as the provider.
////// Oracle.Data.Access 组件提供的用于批量操作的方法。 /// public sealed class OracleAccessBatcher : IBatcherProvider { ////// 获取或设置提供者服务的上下文。 /// public ServiceContext ServiceContext { get; set; } ////// 将 /// 要批量插入的的数据批量插入到数据库中。 /// 。 /// 每批次写入的数据量。 public void Insert(DataTable dataTable, int batchSize = 10000) { Checker.ArgumentNull(dataTable, "dataTable"); if (dataTable.Rows.Count == 0) { return; } using (var connection = ServiceContext.Database.CreateConnection()) { try { connection.TryOpen(); using (var command = ServiceContext.Database.Provider.DbProviderFactory.CreateCommand()) { if (command == null) { throw new BatcherException(new ArgumentException("command")); } command.Connection = connection; command.CommandText = GenerateInserSql(ServiceContext.Database, command, dataTable); command.ExecuteNonQuery(); } } catch (Exception exp) { throw new BatcherException(exp); } finally { connection.TryClose(); } } } /// /// 生成插入数据的sql语句。 /// /// /// /// ///private string GenerateInserSql(IDatabase database, DbCommand command, DataTable table) { var names = new StringBuilder(); var values = new StringBuilder(); //将一个DataTable的数据转换为数组的数组 var data = table.ToArray(); //设置ArrayBindCount属性 command.GetType().GetProperty("ArrayBindCount").SetValue(command, table.Rows.Count, null); var syntax = database.Provider.GetService (); for (var i = 0; i < table.Columns.Count; i++) { var column = table.Columns[i]; var parameter = database.Provider.DbProviderFactory.CreateParameter(); if (parameter == null) { continue; } parameter.ParameterName = column.ColumnName; parameter.Direction = ParameterDirection.Input; parameter.DbType = column.DataType.GetDbType(); parameter.Value = data[i]; if (names.Length > 0) { names.Append(","); values.Append(","); } names.AppendFormat("{0}", DbUtility.FormatByQuote(syntax, column.ColumnName)); values.AppendFormat("{0}{1}", syntax.ParameterPrefix, column.ColumnName); command.Parameters.Add(parameter); } return string.Format("INSERT INTO {0}({1}) VALUES ({2})", DbUtility.FormatByQuote(syntax, table.TableName), names, values); } }
The most important step above is to convert the DataTable into an array representation of an array, that is, object[][]. The superscript of the former array is the number of columns, and the latter array is the number of rows, so The loop Columns uses the last array as the value of Parameter, that is, the value of the parameter is an array. The insert statement is no different from a normal insert statement.
3. Batch Insertion of SQLite Data
SQLite batch insertion only requires opening a transaction. The specific principle of this is unknown.
public sealed class SQLiteBatcher : IBatcherProvider { /// /// 获取或设置提供者服务的上下文。 /// public ServiceContext ServiceContext { get; set; } /// /// 将 /// 要批量插入的 /// 生成插入数据的sql语句。 /// /// /// /// (); var names = new StringBuilder(); var values = new StringBuilder(); var flag = new AssertFlag(); table.EachColumn(column => { if (!flag.AssertTrue()) { names.Append(","); values.Append(","); } names.Append(DbUtility.FormatByQuote(syntax, column.ColumnName)); values.AppendFormat("{0}{1}", syntax.ParameterPrefix, column.ColumnName); }); return string.Format("INSERT INTO {0}({1}) VALUES ({2})", DbUtility.FormatByQuote(syntax, table.TableName), names, values); } }
4. MySql data batch insertion
////// 为 MySql.Data 组件提供的用于批量操作的方法。 /// public sealed class MySqlBatcher : IBatcherProvider { ////// 获取或设置提供者服务的上下文。 /// public ServiceContext ServiceContext { get; set; } ////// 将 /// 要批量插入的的数据批量插入到数据库中。 /// 。 /// 每批次写入的数据量。 public void Insert(DataTable dataTable, int batchSize = 10000) { Checker.ArgumentNull(dataTable, "dataTable"); if (dataTable.Rows.Count == 0) { return; } using (var connection = ServiceContext.Database.CreateConnection()) { try { connection.TryOpen(); using (var command = ServiceContext.Database.Provider.DbProviderFactory.CreateCommand()) { if (command == null) { throw new BatcherException(new ArgumentException("command")); } command.Connection = connection; command.CommandText = GenerateInserSql(ServiceContext.Database, command, dataTable); if (command.CommandText == string.Empty) { return; } command.ExecuteNonQuery(); } } catch (Exception exp) { throw new BatcherException(exp); } finally { connection.TryClose(); } } } /// /// 生成插入数据的sql语句。 /// /// /// /// ///private string GenerateInserSql(IDatabase database, DbCommand command, DataTable table) { var names = new StringBuilder(); var values = new StringBuilder(); var types = new List (); var count = table.Columns.Count; var syntax = database.Provider.GetService (); table.EachColumn(c => { if (names.Length > 0) { names.Append(","); } names.AppendFormat("{0}", DbUtility.FormatByQuote(syntax, c.ColumnName)); types.Add(c.DataType.GetDbType()); }); var i = 0; foreach (DataRow row in table.Rows) { if (i > 0) { values.Append(","); } values.Append("("); for (var j = 0; j < count; j++) { if (j > 0) { values.Append(", "); } var isStrType = IsStringType(types[j]); var parameter = CreateParameter(database.Provider, isStrType, types[j], row[j], syntax.ParameterPrefix, i, j); if (parameter != null) { values.Append(parameter.ParameterName); command.Parameters.Add(parameter); } else if (isStrType) { values.AppendFormat("'{0}'", row[j]); } else { values.Append(row[j]); } } values.Append(")"); i++; } return string.Format("INSERT INTO {0}({1}) VALUES {2}", DbUtility.FormatByQuote(syntax, table.TableName), names, values); } /// /// 判断是否为字符串类别。 /// /// ///private bool IsStringType(DbType dbType) { return dbType == DbType.AnsiString || dbType == DbType.AnsiStringFixedLength || dbType == DbType.String || dbType == DbType.StringFixedLength; } /// /// 创建参数。 /// /// /// /// /// /// /// /// ///private DbParameter CreateParameter(IProvider provider, bool isStrType, DbType dbType, object value, char parPrefix, int row, int col) { //如果生成全部的参数，则速度会很慢，因此，只有数据类型为字符串(包含'号)和日期型时才添加参数 if ((isStrType && value.ToString().IndexOf('\'') != -1) || dbType == DbType.DateTime) { var name = string.Format("{0}p_{1}_{2}", parPrefix, row, col); var parameter = provider.DbProviderFactory.CreateParameter(); parameter.ParameterName = name; parameter.Direction = ParameterDirection.Input; parameter.DbType = dbType; parameter.Value = value; return parameter; } return null; } }
The batch insertion of MySql is to write all the values in the values of the statement, for example, insert batcher(id, name) values(1, ' 1', 2, '2', 3, '3', ......... 10, '10').
5. Test
Next, write a test case to see the effect of using batch insertion.
public void TestBatchInsert() { Console.WriteLine(TimeWatcher.Watch(() => InvokeTest(database => { var table = new DataTable("Batcher"); table.Columns.Add("Id", typeof(int)); table.Columns.Add("Name1", typeof(string)); table.Columns.Add("Name2", typeof(string)); table.Columns.Add("Name3", typeof(string)); table.Columns.Add("Name4", typeof(string)); //构造100000条数据 for (var i = 0; i < 100000; i++) { table.Rows.Add(i, i.ToString(), i.ToString(), i.ToString(), i.ToString()); } //获取 IBatcherProvider var batcher = database.Provider.GetService(); if (batcher == null) { Console.WriteLine("不支持批量插入。"); } else { batcher.Insert(table); } //输出batcher表的数据量 var sql = new SqlCommand("SELECT COUNT(1) FROM Batcher"); Console.WriteLine("当前共有 {0} 条数据", database.ExecuteScalar(sql)); }))); }
The following table lists the time it takes for each of the four databases to generate 100,000 pieces of data
More big data for several c# databases For articles related to batch insertion (SqlServer, Oracle, SQLite and MySql), please pay attention to the PHP Chinese website!