Showing posts with label Bulk DML. Show all posts
Showing posts with label Bulk DML. Show all posts

Thursday, May 01, 2008

Slow When Bulk Inserting Records to Database Using Lotus Notes Agent

I don't do Lotus Notes program. But our Notes developers told me that it was extremely slow to insert records into Oracle (SQL Server as well) database. One example is that 1 million rows took 8 hours.

After some investigation, I found that they were constructing full SQLs, instead of using parameters. Below is a piece of their code:
    Dim con As ODBCConnection
   Dim qry As ODBCQuery
   Dim result As ODBCResultSet

   Set qry = New ODBCQuery
   Set result = New ODBCResultSet
   con.ConnectTo("")
   Set qry.Connection = con

   for each document loop
       qry.SQL = "insert into person (fname,lname) values( '" & v_fname & 
                  "','" & v_lname & "' )"
       Set result.Query = qry
   end loop
   ...
The high-lite is the trouble. Whenever this was called, I noticed so many queries on Oracle data dictionary views. It seems to me that Notes was parsing the SQL, and the second statement sometimes took about 2 seconds.

To solve it, I did some research and found the parameters was useful.
    qry.SQL = "insert into person (fname,lname) values( ?fname?, ?lname? )"
    Set result.Query = qry
    for each document loop
        Call result.SetParameter( fname, "'" & v_fname & "'" )
        Call result.SetParameter( lname, "'" & v_lname & "'" )
    end loop


The performance improved greatly, one example is 30K records reduced time from 4 hours to 10 minutes. Still high, but Notes spends most of the time preparing data.