sqoop data import 성능 테스트 ( 압축 , parquet ) [테스트 실패 진행불가]

sqoop , compress , parquet , textplain 을 사용하기 위한 테스트를 진행하기로 했다.

현재 프로젝트중인 고객사에서 사용하기로 한

snappy와 parquet에 대한 테스트를 위해서 해당 내용을 작성한다.

사용할경우 안할경우를 비교하기위해서

총 4가지의 케이스를 테스트한다.

MAIN Server 환경 :

- CPU : 2CPU (2Socket 24Core)

- RAM : 256GB

- HDD : PCX용 SSD + SATA HDD 조합 (hadoop, oracle data가 SATA HDD에 존재)

테스트상 Disk I/O가 영향이 많이가기때문에 이 내용도 기록함

- HDD는 전부 5400RPM - 버퍼 64MB , WD사의 데이터 저장용 HDD임.

- SSD는 PCI-E 에 장착하는 장비이며 INTEL SSD 700시리즈

- OS+APP+ENGINE은 SSD에 , DATA는 HDD에 있다

* ambari 를 이용한 하둡설치를 진행했으며 DATA NODE 는 총 4개 replica는 3

* namenode 2개 HA구성.

* Vmware 10

* Vmware 구성 Network 는 기본 100m 라인.

* Oracle은 Enterprise single node 12.1버전 (linux) - 이것역시 Vmware

oracle 테이블은 아래와 같다.

데이터 사이즈는 약 80GB정도 되는 테이블이며 150개의 컬럼에 1바이트안남기고 150바이트씩 전부 넣어서 데이터를 만들었다. 건수는 5,385,050건 이다.

최대한 비슷한 환경을 위해서 shared pool , db block buffer flush를 하면서 테스트를 진행했다.

case 1. 비압축 + text plain

명령어 실행

sqoop import --target-dir=/dev/data1_nc_txt --table HDFS_2 -direct --connect jdbc:oracle:thin:@192.168.0.117:1521:ORCL --username flashone --password 1234 --split-by ACOL1

실행 결과

[hdfs@amb2 ~]$ sqoop import --target-dir=/dev/data1_nc_txt --table HDFS_2 -direct --connect jdbc:oracle:thin:@192.168.0.117:1521:ORCL --username flashone --password 1234 --split-by ACOL1

16/08/10 14:01:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258

16/08/10 14:01:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

16/08/10 14:01:17 INFO manager.SqlManager: Using default fetchSize of 1000

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/08/10 14:01:18 INFO oracle.OraOopManagerFactory:

**************************************************

*** Using Data Connector for Oracle and Hadoop ***

**************************************************

16/08/10 14:01:18 INFO oracle.OraOopManagerFactory: Oracle Database version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

16/08/10 14:01:18 INFO oracle.OraOopManagerFactory: This Oracle database is not a RAC.

16/08/10 14:01:18 INFO tool.CodeGenTool: Beginning code generation

16/08/10 14:01:18 INFO manager.SqlManager: Executing SQL statement: SELECT "ACOL1","ACOL2","ACOL3","ACOL4","ACOL5","ACOL6","ACOL7","ACOL8","ACOL9","ACOL10","ACOL11","ACOL12","ACOL13","ACOL14","ACOL15","ACOL16","ACOL17","ACOL18","ACOL19","ACOL20","ACOL21","ACOL22","ACOL23","ACOL24","ACOL25","ACOL26","ACOL27","ACOL28","ACOL29","ACOL30","ACOL31","ACOL32","ACOL33","ACOL34","ACOL35","ACOL36","ACOL37","ACOL38","ACOL39","ACOL40","ACOL41","ACOL42","ACOL43","ACOL44","ACOL45","ACOL46","ACOL47","ACOL48","ACOL49","ACOL50","ACOL51","ACOL52","ACOL53","ACOL54","ACOL55","ACOL56","ACOL57","ACOL58","ACOL59","ACOL60","ACOL61","ACOL62","ACOL63","ACOL64","ACOL65","ACOL66","ACOL67","ACOL68","ACOL69","ACOL70","ACOL71","ACOL72","ACOL73","ACOL74","ACOL75","BCOL1","BCOL2","BCOL3","BCOL4","BCOL5","BCOL6","BCOL7","BCOL8","BCOL9","BCOL10","BCOL11","BCOL12","BCOL13","BCOL14","BCOL15","BCOL16","BCOL17","BCOL18","BCOL19","BCOL20","BCOL21","BCOL22","BCOL23","BCOL24","BCOL25","BCOL26","BCOL27","BCOL28","BCOL29","BCOL30","BCOL31","BCOL32","BCOL33","BCOL34","BCOL35","BCOL36","BCOL37","BCOL38","BCOL39","BCOL40","BCOL41","BCOL42","BCOL43","BCOL44","BCOL45","BCOL46","BCOL47","BCOL48","BCOL49","BCOL50","BCOL51","BCOL52","BCOL53","BCOL54","BCOL55","BCOL56","BCOL57","BCOL58","BCOL59","BCOL60","BCOL61","BCOL62","BCOL63","BCOL64","BCOL65","BCOL66","BCOL67","BCOL68","BCOL69","BCOL70","BCOL71","BCOL72","BCOL73","BCOL74","BCOL75" FROM HDFS_2 WHERE 0=1

16/08/10 14:01:18 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.2.0-258/hadoop-mapreduce

Note: /tmp/sqoop-hdfs/compile/9152233b807a9011bb4c6752ec771805/HDFS_2.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

16/08/10 14:01:21 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/9152233b807a9011bb4c6752ec771805/HDFS_2.jar

16/08/10 14:01:21 INFO mapreduce.ImportJobBase: Beginning import of HDFS_2

16/08/10 14:01:22 INFO manager.SqlManager: Executing SQL statement: SELECT "ACOL1","ACOL2","ACOL3","ACOL4","ACOL5","ACOL6","ACOL7","ACOL8","ACOL9","ACOL10","ACOL11","ACOL12","ACOL13","ACOL14","ACOL15","ACOL16","ACOL17","ACOL18","ACOL19","ACOL20","ACOL21","ACOL22","ACOL23","ACOL24","ACOL25","ACOL26","ACOL27","ACOL28","ACOL29","ACOL30","ACOL31","ACOL32","ACOL33","ACOL34","ACOL35","ACOL36","ACOL37","ACOL38","ACOL39","ACOL40","ACOL41","ACOL42","ACOL43","ACOL44","ACOL45","ACOL46","ACOL47","ACOL48","ACOL49","ACOL50","ACOL51","ACOL52","ACOL53","ACOL54","ACOL55","ACOL56","ACOL57","ACOL58","ACOL59","ACOL60","ACOL61","ACOL62","ACOL63","ACOL64","ACOL65","ACOL66","ACOL67","ACOL68","ACOL69","ACOL70","ACOL71","ACOL72","ACOL73","ACOL74","ACOL75","BCOL1","BCOL2","BCOL3","BCOL4","BCOL5","BCOL6","BCOL7","BCOL8","BCOL9","BCOL10","BCOL11","BCOL12","BCOL13","BCOL14","BCOL15","BCOL16","BCOL17","BCOL18","BCOL19","BCOL20","BCOL21","BCOL22","BCOL23","BCOL24","BCOL25","BCOL26","BCOL27","BCOL28","BCOL29","BCOL30","BCOL31","BCOL32","BCOL33","BCOL34","BCOL35","BCOL36","BCOL37","BCOL38","BCOL39","BCOL40","BCOL41","BCOL42","BCOL43","BCOL44","BCOL45","BCOL46","BCOL47","BCOL48","BCOL49","BCOL50","BCOL51","BCOL52","BCOL53","BCOL54","BCOL55","BCOL56","BCOL57","BCOL58","BCOL59","BCOL60","BCOL61","BCOL62","BCOL63","BCOL64","BCOL65","BCOL66","BCOL67","BCOL68","BCOL69","BCOL70","BCOL71","BCOL72","BCOL73","BCOL74","BCOL75" FROM "HDFS_2" WHERE 1=0

16/08/10 14:01:23 INFO impl.TimelineClientImpl: Timeline service address: http://amb3.local:8188/ws/v1/timeline/

16/08/10 14:01:23 INFO client.RMProxy: Connecting to ResourceManager at amb3.local/192.168.0.143:8050

16/08/10 14:01:25 WARN oracle.OraOopUtilities: System property java.security.egd is not set to file:///dev/urandom - Oracle connections may time out.

16/08/10 14:01:25 INFO db.DBInputFormat: Using read commited transaction isolation

16/08/10 14:01:25 INFO oracle.OraOopOracleQueries: Session Time Zone set to GMT

16/08/10 14:01:25 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL :

begin

dbms_application_info.set_module(module_name => 'Data Connector for Oracle and Hadoop', action_name => 'import 20160810140118KST');

end;

16/08/10 14:01:25 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session disable parallel query

16/08/10 14:01:25 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session set "_serial_direct_read"=true

16/08/10 14:01:25 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session set tracefile_identifier=oraoop

16/08/10 14:01:26 INFO oracle.OraOopDataDrivenDBInputFormat: The table being imported by sqoop has 11335664 blocks that have been divided into 739 chunks which will be processed in 4 splits. The chunks will be allocated to the splits using the method : ROUNDROBIN

16/08/10 14:01:26 INFO mapreduce.JobSubmitter: number of splits:4

16/08/10 14:01:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470728284233_0015

16/08/10 14:01:27 INFO impl.YarnClientImpl: Submitted application application_1470728284233_0015

16/08/10 14:01:27 INFO mapreduce.Job: The url to track the job: http://amb3.local:8088/proxy/application_1470728284233_0015/

16/08/10 14:01:27 INFO mapreduce.Job: Running job: job_1470728284233_0015

16/08/10 14:01:34 INFO mapreduce.Job: Job job_1470728284233_0015 running in uber mode : false

16/08/10 14:01:34 INFO mapreduce.Job: map 0% reduce 0%

16/08/10 14:01:55 INFO mapreduce.Job: map 1% reduce 0%

16/08/10 14:02:10 INFO mapreduce.Job: map 2% reduce 0%

16/08/10 14:02:29 INFO mapreduce.Job: map 3% reduce 0%

16/08/10 14:02:46 INFO mapreduce.Job: map 4% reduce 0%

16/08/10 14:03:01 INFO mapreduce.Job: map 5% reduce 0%

16/08/10 14:03:19 INFO mapreduce.Job: map 6% reduce 0%

16/08/10 14:03:39 INFO mapreduce.Job: map 7% reduce 0%

16/08/10 14:03:57 INFO mapreduce.Job: map 8% reduce 0%

16/08/10 14:04:17 INFO mapreduce.Job: map 9% reduce 0%

16/08/10 14:04:39 INFO mapreduce.Job: map 10% reduce 0%

16/08/10 14:04:56 INFO mapreduce.Job: map 11% reduce 0%

16/08/10 14:05:15 INFO mapreduce.Job: map 12% reduce 0%

16/08/10 14:05:32 INFO mapreduce.Job: map 13% reduce 0%

16/08/10 14:05:53 INFO mapreduce.Job: map 14% reduce 0%

16/08/10 14:06:10 INFO mapreduce.Job: map 15% reduce 0%

16/08/10 14:06:29 INFO mapreduce.Job: map 16% reduce 0%

16/08/10 14:06:54 INFO mapreduce.Job: map 17% reduce 0%

16/08/10 14:07:12 INFO mapreduce.Job: map 18% reduce 0%

16/08/10 14:07:30 INFO mapreduce.Job: map 19% reduce 0%

16/08/10 14:07:51 INFO mapreduce.Job: map 20% reduce 0%

16/08/10 14:08:21 INFO mapreduce.Job: map 21% reduce 0%

16/08/10 14:08:36 INFO mapreduce.Job: map 22% reduce 0%

16/08/10 14:09:01 INFO mapreduce.Job: map 23% reduce 0%

16/08/10 14:09:20 INFO mapreduce.Job: map 24% reduce 0%

16/08/10 14:09:37 INFO mapreduce.Job: map 25% reduce 0%

16/08/10 14:09:50 INFO mapreduce.Job: map 26% reduce 0%

16/08/10 14:10:15 INFO mapreduce.Job: map 27% reduce 0%

16/08/10 14:10:30 INFO mapreduce.Job: map 28% reduce 0%

16/08/10 14:10:55 INFO mapreduce.Job: map 29% reduce 0%

16/08/10 14:11:18 INFO mapreduce.Job: map 30% reduce 0%

16/08/10 14:11:33 INFO mapreduce.Job: map 31% reduce 0%

16/08/10 14:11:54 INFO mapreduce.Job: map 32% reduce 0%

16/08/10 14:12:09 INFO mapreduce.Job: map 33% reduce 0%

16/08/10 14:12:27 INFO mapreduce.Job: map 34% reduce 0%

16/08/10 14:12:49 INFO mapreduce.Job: map 35% reduce 0%

16/08/10 14:13:02 INFO mapreduce.Job: map 36% reduce 0%

16/08/10 14:13:32 INFO mapreduce.Job: map 37% reduce 0%

16/08/10 14:13:53 INFO mapreduce.Job: map 38% reduce 0%

16/08/10 14:14:12 INFO mapreduce.Job: map 39% reduce 0%

16/08/10 14:14:29 INFO mapreduce.Job: map 40% reduce 0%

16/08/10 14:14:48 INFO mapreduce.Job: map 41% reduce 0%

16/08/10 14:14:54 INFO mapreduce.Job: map 42% reduce 0%

16/08/10 14:15:15 INFO mapreduce.Job: map 43% reduce 0%

16/08/10 14:15:31 INFO mapreduce.Job: map 44% reduce 0%

16/08/10 14:15:43 INFO mapreduce.Job: map 45% reduce 0%

16/08/10 14:16:06 INFO mapreduce.Job: map 46% reduce 0%

16/08/10 14:16:13 INFO mapreduce.Job: map 47% reduce 0%

16/08/10 14:16:37 INFO mapreduce.Job: map 48% reduce 0%

16/08/10 14:16:43 INFO mapreduce.Job: map 49% reduce 0%

16/08/10 14:17:11 INFO mapreduce.Job: map 50% reduce 0%

16/08/10 14:17:27 INFO mapreduce.Job: map 51% reduce 0%

16/08/10 14:17:49 INFO mapreduce.Job: map 52% reduce 0%

16/08/10 14:17:55 INFO mapreduce.Job: map 53% reduce 0%

16/08/10 14:18:25 INFO mapreduce.Job: map 54% reduce 0%

16/08/10 14:18:43 INFO mapreduce.Job: map 55% reduce 0%

16/08/10 14:18:53 INFO mapreduce.Job: map 56% reduce 0%

16/08/10 14:19:16 INFO mapreduce.Job: map 57% reduce 0%

16/08/10 14:19:40 INFO mapreduce.Job: map 58% reduce 0%

16/08/10 14:19:47 INFO mapreduce.Job: map 59% reduce 0%

16/08/10 14:20:14 INFO mapreduce.Job: map 60% reduce 0%

16/08/10 14:20:22 INFO mapreduce.Job: map 61% reduce 0%

16/08/10 14:20:42 INFO mapreduce.Job: map 62% reduce 0%

16/08/10 14:20:55 INFO mapreduce.Job: map 63% reduce 0%

16/08/10 14:21:15 INFO mapreduce.Job: map 64% reduce 0%

16/08/10 14:21:29 INFO mapreduce.Job: map 65% reduce 0%

16/08/10 14:21:53 INFO mapreduce.Job: map 66% reduce 0%

16/08/10 14:22:23 INFO mapreduce.Job: map 67% reduce 0%

16/08/10 14:22:35 INFO mapreduce.Job: map 68% reduce 0%

16/08/10 14:23:00 INFO mapreduce.Job: map 69% reduce 0%

16/08/10 14:23:18 INFO mapreduce.Job: map 70% reduce 0%

16/08/10 14:23:34 INFO mapreduce.Job: map 71% reduce 0%

16/08/10 14:23:55 INFO mapreduce.Job: map 72% reduce 0%

16/08/10 14:24:15 INFO mapreduce.Job: map 73% reduce 0%

16/08/10 14:24:31 INFO mapreduce.Job: map 74% reduce 0%

16/08/10 14:24:54 INFO mapreduce.Job: map 75% reduce 0%

16/08/10 14:25:14 INFO mapreduce.Job: map 76% reduce 0%

16/08/10 14:25:56 INFO mapreduce.Job: map 77% reduce 0%

16/08/10 14:26:17 INFO mapreduce.Job: map 78% reduce 0%

16/08/10 14:26:37 INFO mapreduce.Job: map 79% reduce 0%

16/08/10 14:27:25 INFO mapreduce.Job: map 80% reduce 0%

16/08/10 14:27:39 INFO mapreduce.Job: map 81% reduce 0%

16/08/10 14:28:30 INFO mapreduce.Job: map 82% reduce 0%

16/08/10 14:28:58 INFO mapreduce.Job: map 83% reduce 0%

16/08/10 14:29:41 INFO mapreduce.Job: map 84% reduce 0%

16/08/10 14:29:59 INFO mapreduce.Job: map 85% reduce 0%

16/08/10 14:31:00 INFO mapreduce.Job: map 86% reduce 0%

16/08/10 14:31:39 INFO mapreduce.Job: map 87% reduce 0%

16/08/10 14:32:11 INFO mapreduce.Job: map 88% reduce 0%

16/08/10 14:32:35 INFO mapreduce.Job: map 89% reduce 0%

16/08/10 14:33:04 INFO mapreduce.Job: map 90% reduce 0%

16/08/10 14:33:51 INFO mapreduce.Job: map 91% reduce 0%

16/08/10 14:34:45 INFO mapreduce.Job: map 92% reduce 0%

16/08/10 14:35:14 INFO mapreduce.Job: map 93% reduce 0%

16/08/10 14:35:44 INFO mapreduce.Job: map 94% reduce 0%

16/08/10 14:36:16 INFO mapreduce.Job: map 95% reduce 0%

16/08/10 14:37:08 INFO mapreduce.Job: map 96% reduce 0%

16/08/10 14:37:44 INFO mapreduce.Job: map 97% reduce 0%

16/08/10 14:38:15 INFO mapreduce.Job: map 98% reduce 0%

16/08/10 14:38:45 INFO mapreduce.Job: map 99% reduce 0%

16/08/10 14:39:41 INFO mapreduce.Job: map 100% reduce 0%

16/08/10 14:40:05 INFO mapreduce.Job: Job job_1470728284233_0015 completed successfully

16/08/10 14:40:05 INFO mapreduce.Job: Counters: 30

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=619112

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=66021

HDFS: Number of bytes written=81583507500

HDFS: Number of read operations=16

HDFS: Number of large read operations=0

HDFS: Number of write operations=8

Job Counters

Launched map tasks=4

Other local map tasks=4

Total time spent by all maps in occupied slots (ms)=17659492

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=8829746

Total vcore-seconds taken by all map tasks=8829746

Total megabyte-seconds taken by all map tasks=13562489856

Map-Reduce Framework

Map input records=5385050

Map output records=5385050

Input split bytes=66021

Spilled Records=0

Failed Shuffles=0

Merged Map outputs=0

GC time elapsed (ms)=52264

CPU time spent (ms)=2221300

Physical memory (bytes) snapshot=1561075712

Virtual memory (bytes) snapshot=13249417216

Total committed heap usage (bytes)=847249408

File Input Format Counters

Bytes Read=0

File Output Format Counters

Bytes Written=81583507500

16/08/10 14:40:05 INFO mapreduce.ImportJobBase: Transferred 75.9806 GB in 2,322.6017 seconds (33.4987 MB/sec)

16/08/10 14:40:05 INFO mapreduce.ImportJobBase: Retrieved 5385050 records.

[hdfs@amb2 ~]$

case 2. 압축 + text plain

sqoop import --target-dir=/dev/data2_sn_txt --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --table HDFS_2 -direct --connect jdbc:oracle:thin:@192.168.0.117:1521:ORCL --username flashone --password 1234 --split-by ACOL1

실행결과

[hdfs@amb2 ~]$ sqoop import --target-dir=/dev/data2_sn_txt --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --tablHDFS_2 -direct --connect jdbc:oracle:thin:@192.168.0.117:1521:ORCL --username flashone --password 1234 --split-by ACOL1

16/08/10 14:52:34 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.2.0-258

16/08/10 14:52:34 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

16/08/10 14:52:34 INFO manager.SqlManager: Using default fetchSize of 1000

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/08/10 14:52:35 INFO oracle.OraOopManagerFactory:

**************************************************

*** Using Data Connector for Oracle and Hadoop ***

**************************************************

16/08/10 14:52:36 INFO oracle.OraOopManagerFactory: Oracle Database version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

16/08/10 14:52:36 INFO oracle.OraOopManagerFactory: This Oracle database is not a RAC.

16/08/10 14:52:36 INFO tool.CodeGenTool: Beginning code generation

16/08/10 14:52:36 INFO manager.SqlManager: Executing SQL statement: SELECT "ACOL1","ACOL2","ACOL3","ACOL4","ACOL5","ACOL6","ACOL7","ACOL8","ACOL9","ACOL10","ACOL11","ACOL12","ACOL13","ACOL14","ACOL15","ACOL16","ACOL17","ACOL18","ACOL19","ACOL20","ACOL21","ACOL22","ACOL23","ACOL24","ACOL25","ACOL26","ACOL27","ACOL28","ACOL29","ACOL30","ACOL31","ACOL32","ACOL33","ACOL34","ACOL35","ACOL36","ACOL37","ACOL38","ACOL39","ACOL40","ACOL41","ACOL42","ACOL43","ACOL44","ACOL45","ACOL46","ACOL47","ACOL48","ACOL49","ACOL50","ACOL51","ACOL52","ACOL53","ACOL54","ACOL55","ACOL56","ACOL57","ACOL58","ACOL59","ACOL60","ACOL61","ACOL62","ACOL63","ACOL64","ACOL65","ACOL66","ACOL67","ACOL68","ACOL69","ACOL70","ACOL71","ACOL72","ACOL73","ACOL74","ACOL75","BCOL1","BCOL2","BCOL3","BCOL4","BCOL5","BCOL6","BCOL7","BCOL8","BCOL9","BCOL10","BCOL11","BCOL12","BCOL13","BCOL14","BCOL15","BCOL16","BCOL17","BCOL18","BCOL19","BCOL20","BCOL21","BCOL22","BCOL23","BCOL24","BCOL25","BCOL26","BCOL27","BCOL28","BCOL29","BCOL30","BCOL31","BCOL32","BCOL33","BCOL34","BCOL35","BCOL36","BCOL37","BCOL38","BCOL39","BCOL40","BCOL41","BCOL42","BCOL43","BCOL44","BCOL45","BCOL46","BCOL47","BCOL48","BCOL49","BCOL50","BCOL51","BCOL52","BCOL53","BCOL54","BCOL55","BCOL56","BCOL57","BCOL58","BCOL59","BCOL60","BCOL61","BCOL62","BCOL63","BCOL64","BCOL65","BCOL66","BCOL67","BCOL68","BCOL69","BCOL70","BCOL71","BCOL72","BCOL73","BCOL74","BCOL75" FROM HDFS_2 WHERE 0=1

16/08/10 14:52:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.2.0-258/hadoop-mapreduce

Note: /tmp/sqoop-hdfs/compile/296e19dcfe9895e28bbc77da203ad202/HDFS_2.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

16/08/10 14:52:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/296e19dcfe9895e28bbc77da203ad202/HDFS_2.jar

16/08/10 14:52:39 INFO mapreduce.ImportJobBase: Beginning import of HDFS_2

16/08/10 14:52:40 INFO manager.SqlManager: Executing SQL statement: SELECT "ACOL1","ACOL2","ACOL3","ACOL4","ACOL5","ACOL6","ACOL7","ACOL8","ACOL9","ACOL10","ACOL11","ACOL12","ACOL13","ACOL14","ACOL15","ACOL16","ACOL17","ACOL18","ACOL19","ACOL20","ACOL21","ACOL22","ACOL23","ACOL24","ACOL25","ACOL26","ACOL27","ACOL28","ACOL29","ACOL30","ACOL31","ACOL32","ACOL33","ACOL34","ACOL35","ACOL36","ACOL37","ACOL38","ACOL39","ACOL40","ACOL41","ACOL42","ACOL43","ACOL44","ACOL45","ACOL46","ACOL47","ACOL48","ACOL49","ACOL50","ACOL51","ACOL52","ACOL53","ACOL54","ACOL55","ACOL56","ACOL57","ACOL58","ACOL59","ACOL60","ACOL61","ACOL62","ACOL63","ACOL64","ACOL65","ACOL66","ACOL67","ACOL68","ACOL69","ACOL70","ACOL71","ACOL72","ACOL73","ACOL74","ACOL75","BCOL1","BCOL2","BCOL3","BCOL4","BCOL5","BCOL6","BCOL7","BCOL8","BCOL9","BCOL10","BCOL11","BCOL12","BCOL13","BCOL14","BCOL15","BCOL16","BCOL17","BCOL18","BCOL19","BCOL20","BCOL21","BCOL22","BCOL23","BCOL24","BCOL25","BCOL26","BCOL27","BCOL28","BCOL29","BCOL30","BCOL31","BCOL32","BCOL33","BCOL34","BCOL35","BCOL36","BCOL37","BCOL38","BCOL39","BCOL40","BCOL41","BCOL42","BCOL43","BCOL44","BCOL45","BCOL46","BCOL47","BCOL48","BCOL49","BCOL50","BCOL51","BCOL52","BCOL53","BCOL54","BCOL55","BCOL56","BCOL57","BCOL58","BCOL59","BCOL60","BCOL61","BCOL62","BCOL63","BCOL64","BCOL65","BCOL66","BCOL67","BCOL68","BCOL69","BCOL70","BCOL71","BCOL72","BCOL73","BCOL74","BCOL75" FROM "HDFS_2" WHERE 1=0

16/08/10 14:52:42 INFO impl.TimelineClientImpl: Timeline service address: http://amb3.local:8188/ws/v1/timeline/

16/08/10 14:52:42 INFO client.RMProxy: Connecting to ResourceManager at amb3.local/192.168.0.143:8050

16/08/10 14:52:44 WARN oracle.OraOopUtilities: System property java.security.egd is not set to file:///dev/urandom - Oracle connections may time out.

16/08/10 14:52:44 INFO db.DBInputFormat: Using read commited transaction isolation

16/08/10 14:52:44 INFO oracle.OraOopOracleQueries: Session Time Zone set to GMT

16/08/10 14:52:44 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL :

begin

dbms_application_info.set_module(module_name => 'Data Connector for Oracle and Hadoop', action_name => 'import 20160810145235KST');

end;

16/08/10 14:52:44 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session disable parallel query

16/08/10 14:52:44 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session set "_serial_direct_read"=true

16/08/10 14:52:44 INFO oracle.OracleConnectionFactory: Initializing Oracle session with SQL : alter session set tracefile_identifier=oraoop

16/08/10 14:52:45 INFO oracle.OraOopDataDrivenDBInputFormat: The table being imported by sqoop has 11335664 blocks that have been divided into 739 chunks which will be processed in 4 splits. The chunks will be allocated to the splits using the method : ROUNDROBIN

16/08/10 14:52:45 INFO mapreduce.JobSubmitter: number of splits:4

16/08/10 14:52:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470728284233_0017

16/08/10 14:52:46 INFO impl.YarnClientImpl: Submitted application application_1470728284233_0017

16/08/10 14:52:46 INFO mapreduce.Job: The url to track the job: http://amb3.local:8088/proxy/application_1470728284233_0017/

16/08/10 14:52:46 INFO mapreduce.Job: Running job: job_1470728284233_0017

16/08/10 14:52:54 INFO mapreduce.Job: Job job_1470728284233_0017 running in uber mode : false

16/08/10 14:52:54 INFO mapreduce.Job: map 0% reduce 0%

16/08/10 14:53:16 INFO mapreduce.Job: map 1% reduce 0%

16/08/10 14:53:28 INFO mapreduce.Job: map 2% reduce 0%

16/08/10 14:53:43 INFO mapreduce.Job: map 3% reduce 0%

16/08/10 14:53:58 INFO mapreduce.Job: map 4% reduce 0%

16/08/10 14:54:15 INFO mapreduce.Job: map 5% reduce 0%

16/08/10 14:54:30 INFO mapreduce.Job: map 6% reduce 0%

16/08/10 14:54:58 INFO mapreduce.Job: map 7% reduce 0%

16/08/10 14:55:44 INFO mapreduce.Job: map 8% reduce 0%

16/08/10 14:56:31 INFO mapreduce.Job: map 9% reduce 0%

16/08/10 14:57:03 INFO mapreduce.Job: map 10% reduce 0%

16/08/10 14:57:31 INFO mapreduce.Job: map 11% reduce 0%

16/08/10 14:58:06 INFO mapreduce.Job: map 12% reduce 0%

16/08/10 14:58:45 INFO mapreduce.Job: map 13% reduce 0%

16/08/10 14:59:17 INFO mapreduce.Job: map 14% reduce 0%

16/08/10 14:59:57 INFO mapreduce.Job: map 15% reduce 0%

16/08/10 15:00:32 INFO mapreduce.Job: map 16% reduce 0%

16/08/10 15:01:24 INFO mapreduce.Job: map 17% reduce 0%

16/08/10 15:02:05 INFO mapreduce.Job: map 18% reduce 0%

16/08/10 15:02:36 INFO mapreduce.Job: map 19% reduce 0%

16/08/10 15:03:16 INFO mapreduce.Job: map 20% reduce 0%

16/08/10 15:03:45 INFO mapreduce.Job: map 21% reduce 0%

16/08/10 15:04:30 INFO mapreduce.Job: map 22% reduce 0%

16/08/10 15:04:58 INFO mapreduce.Job: map 23% reduce 0%

16/08/10 15:05:30 INFO mapreduce.Job: map 24% reduce 0%

16/08/10 15:06:05 INFO mapreduce.Job: map 25% reduce 0%

16/08/10 15:06:54 INFO mapreduce.Job: map 26% reduce 0%

16/08/10 15:07:24 INFO mapreduce.Job: map 27% reduce 0%

16/08/10 15:08:00 INFO mapreduce.Job: map 28% reduce 0%

16/08/10 15:08:44 INFO mapreduce.Job: map 29% reduce 0%

16/08/10 15:09:11 INFO mapreduce.Job: map 30% reduce 0%

16/08/10 15:09:55 INFO mapreduce.Job: map 31% reduce 0%

16/08/10 15:10:25 INFO mapreduce.Job: map 32% reduce 0%

16/08/10 15:11:40 INFO mapreduce.Job: map 33% reduce 0%

16/08/10 15:12:08 INFO mapreduce.Job: map 34% reduce 0%

16/08/10 15:12:42 INFO mapreduce.Job: map 35% reduce 0%

16/08/10 15:13:30 INFO mapreduce.Job: map 36% reduce 0%

16/08/10 15:14:05 INFO mapreduce.Job: map 37% reduce 0%

16/08/10 15:14:29 INFO mapreduce.Job: map 38% reduce 0%

16/08/10 15:14:59 INFO mapreduce.Job: map 39% reduce 0%

16/08/10 15:15:44 INFO mapreduce.Job: map 40% reduce 0%

16/08/10 15:16:12 INFO mapreduce.Job: map 41% reduce 0%

16/08/10 15:17:00 INFO mapreduce.Job: map 42% reduce 0%

16/08/10 15:17:35 INFO mapreduce.Job: map 43% reduce 0%

16/08/10 15:18:00 INFO mapreduce.Job: map 44% reduce 0%

16/08/10 15:18:45 INFO mapreduce.Job: map 45% reduce 0%

16/08/10 15:19:26 INFO mapreduce.Job: map 46% reduce 0%

16/08/10 15:19:52 INFO mapreduce.Job: map 47% reduce 0%

16/08/10 15:20:42 INFO mapreduce.Job: map 48% reduce 0%

16/08/10 15:21:11 INFO mapreduce.Job: map 49% reduce 0%

16/08/10 15:21:52 INFO mapreduce.Job: map 50% reduce 0%

16/08/10 15:22:37 INFO mapreduce.Job: map 51% reduce 0%

16/08/10 15:23:21 INFO mapreduce.Job: map 52% reduce 0%

16/08/10 15:23:56 INFO mapreduce.Job: map 53% reduce 0%

16/08/10 15:24:33 INFO mapreduce.Job: map 54% reduce 0%

16/08/10 15:25:10 INFO mapreduce.Job: map 55% reduce 0%

16/08/10 15:25:36 INFO mapreduce.Job: map 56% reduce 0%

16/08/10 15:26:14 INFO mapreduce.Job: map 57% reduce 0%

16/08/10 15:26:58 INFO mapreduce.Job: map 58% reduce 0%

16/08/10 15:27:40 INFO mapreduce.Job: map 59% reduce 0%

16/08/10 15:28:18 INFO mapreduce.Job: map 60% reduce 0%

16/08/10 15:28:49 INFO mapreduce.Job: map 61% reduce 0%

16/08/10 15:29:36 INFO mapreduce.Job: map 62% reduce 0%

16/08/10 15:30:08 INFO mapreduce.Job: map 63% reduce 0%

16/08/10 15:30:37 INFO mapreduce.Job: map 64% reduce 0%

16/08/10 15:31:06 INFO mapreduce.Job: map 65% reduce 0%

16/08/10 15:32:05 INFO mapreduce.Job: map 66% reduce 0%

16/08/10 15:32:39 INFO mapreduce.Job: map 67% reduce 0%

16/08/10 15:33:16 INFO mapreduce.Job: map 68% reduce 0%

16/08/10 15:33:53 INFO mapreduce.Job: map 69% reduce 0%

16/08/10 15:34:26 INFO mapreduce.Job: map 70% reduce 0%

16/08/10 15:34:59 INFO mapreduce.Job: map 71% reduce 0%

16/08/10 15:35:32 INFO mapreduce.Job: map 72% reduce 0%

16/08/10 15:36:26 INFO mapreduce.Job: map 73% reduce 0%

16/08/10 15:36:59 INFO mapreduce.Job: map 74% reduce 0%

16/08/10 15:37:40 INFO mapreduce.Job: map 75% reduce 0%

16/08/10 15:37:47 INFO mapreduce.Job: map 76% reduce 0%

16/08/10 15:38:28 INFO mapreduce.Job: map 77% reduce 0%

16/08/10 15:39:16 INFO mapreduce.Job: map 78% reduce 0%

16/08/10 15:39:43 INFO mapreduce.Job: map 79% reduce 0%

16/08/10 15:40:12 INFO mapreduce.Job: map 80% reduce 0%

16/08/10 15:40:43 INFO mapreduce.Job: map 81% reduce 0%

16/08/10 15:41:14 INFO mapreduce.Job: map 82% reduce 0%

16/08/10 15:42:01 INFO mapreduce.Job: map 83% reduce 0%

16/08/10 15:42:30 INFO mapreduce.Job: map 84% reduce 0%

16/08/10 15:43:24 INFO mapreduce.Job: map 85% reduce 0%

16/08/10 15:43:51 INFO mapreduce.Job: map 86% reduce 0%

16/08/10 15:44:27 INFO mapreduce.Job: map 87% reduce 0%

16/08/10 15:44:54 INFO mapreduce.Job: map 88% reduce 0%

16/08/10 15:45:31 INFO mapreduce.Job: map 89% reduce 0%

16/08/10 15:45:53 INFO mapreduce.Job: map 90% reduce 0%

16/08/10 15:46:48 INFO mapreduce.Job: map 91% reduce 0%

16/08/10 15:47:23 INFO mapreduce.Job: map 92% reduce 0%

16/08/10 15:48:02 INFO mapreduce.Job: map 93% reduce 0%

16/08/10 15:48:36 INFO mapreduce.Job: map 94% reduce 0%

16/08/10 15:49:07 INFO mapreduce.Job: map 95% reduce 0%

16/08/10 15:49:43 INFO mapreduce.Job: map 96% reduce 0%

16/08/10 15:50:10 INFO mapreduce.Job: map 97% reduce 0%

16/08/10 15:51:10 INFO mapreduce.Job: map 98% reduce 0%

16/08/10 15:51:54 INFO mapreduce.Job: map 99% reduce 0%

16/08/10 15:52:49 INFO mapreduce.Job: map 100% reduce 0%

16/08/10 15:53:13 INFO mapreduce.Job: Job job_1470728284233_0017 completed successfully

16/08/10 15:53:13 INFO mapreduce.Job: Counters: 30

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=619092

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=66021

HDFS: Number of bytes written=81593058054

HDFS: Number of read operations=16

HDFS: Number of large read operations=0

HDFS: Number of write operations=8

Job Counters

Launched map tasks=4

Other local map tasks=4

Total time spent by all maps in occupied slots (ms)=27988674

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=13994337

Total vcore-seconds taken by all map tasks=13994337

Total megabyte-seconds taken by all map tasks=21495301632

Map-Reduce Framework

Map input records=5385050

Map output records=5385050

Input split bytes=66021

Spilled Records=0

Failed Shuffles=0

Merged Map outputs=0

GC time elapsed (ms)=174416

CPU time spent (ms)=3340330

Physical memory (bytes) snapshot=1647304704

Virtual memory (bytes) snapshot=13287915520

Total committed heap usage (bytes)=879755264

File Input Format Counters

Bytes Read=0

File Output Format Counters

Bytes Written=81593058054

16/08/10 15:53:13 INFO mapreduce.ImportJobBase: Transferred 75.9895 GB in 3,632.0301 seconds (21.4242 MB/sec)

16/08/10 15:53:13 INFO mapreduce.ImportJobBase: Retrieved 5385050 records.

[hdfs@amb2 ~]$

case 3. 압축 + parquet

sqoop import --target-dir=/dev/data4_sn_pq --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --table HDFS_2 -direct --as-parquetfile --connect jdbc:oracle:thin:@192.168.0.117:1521:ORCL --username flashone --password 1234 --split-by ACOL1

실행 결과

실패 ( 어떤 상황이 생겼는지는 나만 알아야겠다;;; 끔찍하다;;; )

- 50만건으로 재시도하기로함.

저작자표시

'OpenSource(Bigdata&distribution&Network) > Sqoop' 카테고리의 다른 글

sqoop parquet snappy 테스트 (0)	2016.08.10
sqopp import format 별로 저장 (0)	2016.08.10

ToToRo

sqoop data import 성능 테스트 ( 압축 , parquet ) [테스트 실패 진행불가]

'OpenSource(Bigdata&distribution&Network) > Sqoop' 카테고리의 다른 글

티스토리툴바

sqoop data import 성능 테스트 ( 압축 , parquet ) [테스트 실패 진행불가]

'OpenSource(Bigdata&distribution&Network) > Sqoop' 카테고리의 다른 글

'OpenSource(Bigdata&distribution&Network)/Sqoop' Related Articles

티스토리툴바