Spark - SparkLauncher

□ Submit 테스트

SparkLauncher을 이용한 Job Submit 개인적 목적 : 웹에서 다수의 사용자가 Spark 연산 이용

□ Job

public class App {     private static final Logger logger = LogManager.getLogger(App.class);     public static void main( String[] args )     {         logger.info("launcher target Start");         SparkConf sconf = new SparkConf().setMaster("spark://192.168.0.XXX:7077")                                                            .setAppName(args[0])                                                            .set("spark.ui.port", args[1])                                                            .set("spark.storage.memoryFraction", "0.3")                                                            .set("spark.cores.max", "4");                 JavaSparkContext sc = new JavaSparkContext(sconf);                Configuration bsonDataConfig = new Configuration();         //파일 로딩         JavaPairRDD<Object, BSONObject> documents = sc.newAPIHadoopFile(                     "hdfs://192.168.0.190:9000/user/ksu/Odin201776",                     BSONFileInputFormat.class,                     Object.class,                     BSONObject.class,                     bsonDataConfig);         //카운트                logger.info(documents.count());     } }

□ Launcher

public class App {     public static void main( String[] args )     {         try {             test();         } catch (Exception e) {             e.printStackTrace();         }     }         static void test() throws Exception {         final String javaHome = "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el6_8.x86_64";         final String sparkHome = "/home/ksu/spark-2.1.0-bin-hadoop2.6";         final String appResource = "/home/ksu/Downloads/workspace/target/target/Server/target.jar";         final String mainClass = "com.launcher.target.App";         final String[] appArgs = new String[]{             "test",             "4045"         };                         SparkLauncher sparkas = new SparkLauncher()                 .setVerbose(true)                 .setJavaHome(javaHome)                 .setSparkHome(sparkHome)                 .setAppResource(appResource)                 .setMainClass(mainClass)                 .setMaster("spark://192.168.0.XXX:7077")                 .setConf(SparkLauncher.EXECUTOR_MEMORY, "512m")                 .addAppArgs(appArgs);         Process proc = sparkas.launch();         System.out.println(proc.waitFor());     } }

□ Add JAR

방법1) spark 설치 위치의 jars 폴더에 추가 후 기동 방법2) sc.addJar("/home/ksu/~~~path~~~/mongo-hadoop-core-1.5.2.jar");

□ 테스트 및 결과

▶SparkUI 확인 ▶Job Log 확인 - 실행 후 하둡에 저장된 데이터 518170개 카운팅   ▶정상 (0), 문제발생(1) 등 리턴