Introduction of SQL LOADER

Introduction of SQL LOADER:-

--------------------------------------

SQL LOADER is an Oracle utility used to load data from the given data-file into table or tables created in the database. The data-file which has the records those need to be loaded. SQL*Loader takes data file, as well as a control file, to insert data into the table. When a Control file is executed, it may create 3 files called log file, bad file, and discard file.

LOADER utility is used to load data from other data source into Oracle. For example, if you have a table in FOXPRO, ACCESS or SYBASE or any other third party database, you can use SQL Loader to load the data into Oracle Tables. SQL Loader will only read the data from Flat files. So If you want to load the data from Foxpro or any other database, you have to first convert that data into Delimited Format flat file or Fixed length format flat file, and then use SQL loader to load the data into Oracle.

Log file will be created for sure in any case (Pass or fail). Bad file would only be created if any record validation fails and Discard file would be created only WHEN condition fails (If Control file has the WHEN condition).

Log file tells you the number of logical records already read from the input data-file and the state of the tables and indexes. This information can be used to resume the load where it left off.

bad file gives you the records that were rejected because of formatting errors or because they caused Oracle/validation errors .

discard file specifies the records that do not meet any of the loading criteria like when any of the WHEN clauses specified in the control file. These records differ from rejected records.

CONTROL file will have the path and name of the DATA-FILE. If both the CONTROL and DATA files are present in the same directory then you do not need to give any path of the data-

file (Name of the file will be enough)

PROCEDURE:

----------------------

Following is procedure to load the data from Third Party Database into Oracle using SQL Loader.

1. Convert the Data into Flat file using third party database command.
2. Create the Table Structure in Oracle Database using appropriate data types
3. Write a Control File, describing how to interpret the flat file and options to load the data.
4. Execute SQL Loader utility specifying the control file in the command line argument

To understand it better let us see the following case study(SQLLDR WITH POSITION:-):-

--------------------------------------------------------------------------------------------------

Suppose we have a fixed length format file containing employees data, as shown below, and wants to load this data into an Oracle table.

7782 CLARK      MANAGER   7839  2572.50          10

7839 KING       PRESIDENT       5500.00          10

7934 MILLER     CLERK     7782   920.00          10

7566 JONES      MANAGER   7839  3123.75          20

7499 ALLEN      SALESMAN  7698  1600.00   300.00 30

7654 MARTIN     SALESMAN  7698  1312.50  1400.00 30

7658 CHAN       ANALYST   7566  3450.00          20

7654 MARTIN     SALESMAN  7698  1312.50  1400.00 30

SOLUTION:

----------------

Steps :-

1.      First Open the file in a text editor and count the length of fields, for example in our fixed length file, employee number is from 1^st position to 4^th position, employee name is from 6^th position to 15^th position, Job name is from 17^th position to 25^th position. Similarly other columns are also located.

2.      Create a table in Oracle, by any name, but should  match columns specified in fixed length file. In our case give the following command to create the table.

SQL> CREATE TABLE emp (empno  NUMBER(5),

       name VARCHAR2(20),

       job  VARCHAR2(10),

       mgr. NUMBER(5),

       sal  NUMBER(10,2),

       comm NUMBER(10,2),

       deptno    NUMBER(3) );

                       

3.      After creating the table, now write a control file by using any text editor

$vi empfix.ctl

1)   LOAD DATA

2)   INFILE '/u01/oracle/fix.dat'

3)   INTO TABLE emp

4)   (empno         POSITION(01:04)   INTEGER EXTERNAL,

       name         POSITION(06:15)   CHAR,

       job          POSITION(17:25)   CHAR,

       mgr          POSITION(27:30)   INTEGER EXTERNAL,

       sal          POSITION(32:39)   DECIMAL EXTERNAL,

       comm         POSITION(41:48)   DECIMAL EXTERNAL,

5)   deptno         POSITION(50:51)   INTEGER EXTERNAL)

Notes:

1.       The LOAD DATA statement is required at the beginning of the control file.

2.       The name of the file containing data follows the INFILE parameter.

3.       The INTO TABLE statement is required to identify the table to be loaded into.

4.       Lines 4 and 5 identify a column name and the location of the data in the datafile to be loaded into that column. empno, name, job, and so on are names of columns in table emp. The data types (INTEGER EXTERNAL, CHAR, DECIMAL EXTERNAL) identify the datatype of data fields in the file, not of corresponding columns in the emp table.

5.       Note that the set of column specifications is enclosed in parentheses.

4.      After saving the control file now start SQL Loader utility by typing the following command.

$sqlldr userid=scott/tiger control=empfix.ctl log=empfix.log direct=y

After you have executed the above command SQL Loader will shows you the output describing how many rows it has loaded.

Loading Data into Multiple Tables using WHEN condition

-------------------------------------------------------------------------

You can simultaneously load data into multiple tables in the same session. You can also use WHEN condition to load only specified rows which meets a particular condition (only equal to “=” and not equal to “<>” conditions are allowed).

For example, suppose we have a fixed length file as shown below

7782 CLARK      MANAGER   7839  2572.50          10

7839 KING       PRESIDENT       5500.00          10

7934 MILLER     CLERK     7782   920.00          10

7566 JONES      MANAGER   7839  3123.75          20

7499 ALLEN      SALESMAN  7698  1600.00   300.00 30

7654 MARTIN     SALESMAN  7698  1312.50  1400.00 30

7658 CHAN       ANALYST   7566  3450.00          20

7654 MARTIN     SALESMAN  7698  1312.50  1400.00 30

Now we want to load all the employees whose deptno is 10 into emp1 table and those employees whose deptno is not equal to 10 in emp2 table. To do this first create the tables emp1 and emp2 by taking appropriate columns and datatypes. Then, write a control file as shown below

$vi emp_multi.ctl

Load Data
infile ‘/u01/oracle/empfix.dat’
append into table scott.emp1
WHEN (deptno=’10 ‘)
  (empno        POSITION(01:04)   INTEGER EXTERNAL,

   name         POSITION(06:15)   CHAR,

   job          POSITION(17:25)   CHAR,

   mgr          POSITION(27:30)   INTEGER EXTERNAL,

   sal          POSITION(32:39)   DECIMAL EXTERNAL,

   comm         POSITION(41:48)   DECIMAL EXTERNAL,

   deptno       POSITION(50:51)   INTEGER EXTERNAL)

    INTO TABLE scott.emp2
  WHEN (deptno<>’10 ‘)
  (empno        POSITION(01:04)   INTEGER EXTERNAL,

   name         POSITION(06:15)   CHAR,

   job          POSITION(17:25)   CHAR,

   mgr          POSITION(27:30)   INTEGER EXTERNAL,

   sal          POSITION(32:39)   DECIMAL EXTERNAL,

   comm         POSITION(41:48)   DECIMAL EXTERNAL,

   deptno       POSITION(50:51)   INTEGER EXTERNAL)

After saving the file emp_multi.ctl run sqlldr
$sqlldr userid=****/**** control=emp_multi.ctl

Conventional Path Load and Direct Path Load.

---------------------------------------------------------------------------

SQL Loader can load the data into Oracle database using Conventional Path method or Direct Path method. You can specify the method by using DIRECT command line option. If you give DIRECT=TRUE then SQL loader will use Direct Path Loading otherwise, if omit this option or specify DIRECT=false, then SQL Loader will use Conventional Path loading method.

Conventional Path

-------------------------------

Conventional path load (the default) uses the SQL INSERT statement and a bind array buffer to load data into database tables.

When SQL*Loader performs a conventional path load, it competes equally with all other processes for buffer resources. This can slow the load significantly. Extra overhead is added as SQL statements are generated, passed to Oracle, and executed.

The Oracle database looks for partially filled blocks and attempts to fill them on each insert. Although appropriate during normal use, this can slow bulk loads dramatically.

Direct Path

---------------------

In Direct Path Loading, Oracle will not use SQL INSERT statement for loading rows. Instead it directly writes the rows, into fresh blocks beyond High Water Mark, in datafiles i.e. it does not scan for free blocks before high water mark. Direct Path load is very fast because

• Partial blocks are not used, so no reads are needed to find them, and fewer writes are performed.
• SQL*Loader need not execute any SQL INSERT statements; therefore, the processing load on the Oracle database is reduced.
• A direct path load calls on Oracle to lock tables and indexes at the start of the load and releases them when the load is finished. A conventional path load calls Oracle once for each array of rows to process a SQL INSERT statement.
• A direct path load uses multiblock asynchronous I/O for writes to the database files.
• During a direct path load, processes perform their own write I/O, instead of using Oracle's buffer cache. This minimizes contention with other Oracle users.

Restrictions on Using Direct Path Loads

-----------------------------------------------------------------

The following conditions must be satisfied for you to use the direct path load method:

• Tables are not clustered.
• Tables to be loaded do not have any active transactions pending.
• Loading a parent table together with a child Table
• Loading BFILE columns

SQLLDR WITH DELIMITER FILE:-

---------------------------------------------

CREATE A CONTROL FILE  (c:\temp_file.ctl)--contol file

  load data

 Infile 'c:\test.csv'

  append ----------------------------------->use if already data is present in table.

  Into table <TABLE_NAME>

  fields terminated by "," OPTIONALLY ENCLOSED BY '"'

(a,b)

----1 way of calling----

calling ctl file in cmd:

go to command prompt:- sqlldr user_name/pwd@schema_name.

control file :  c:\temp_file.ctl

EX:

---

CREATE TABLE test (f1 varchar2(30), f2 varchar2(30), f3 varchar2(30));

Table created

$ cat test.ctl

load data

  infile *

  into table test

  fields terminated by "|" optionally enclosed by '"'

  (f1, f2, f3)

begindata

Field1|"Field2 containing | within"|Field3|

$ sqlldr scott/tiger control=test.ctl