Splitで並列処理してみる
TweetPosted on Saturday Feb 15, 2014 at 08:19PM in Technology
「Partitioned Chunk Processingで遊ぶ」で使ったPartitionは一つの処理を手分けしてやる仕組みだが、splitは異なる処理を並列で行う仕組み。簡単なバッチを作って遊んでみる。
環境・前提条件
- Chunk方式のStepで例外発生時にスキップさせてみると同じ。ここで作ったプロジェクトが普通に動いているものとする
バッチを作ってみる
仕様
- 一番高レベルの要素は2つ。順番に動く
- split0 (Split)
- nextOfSplit (Step)
- split0にはflowが3つある
- 全てのflowは1つのstepを持つ
- 使っているartifactは全て同じ。単純なBatchletで、パラメータで与えられたミリ秒数だけThread.sleep()する
- それぞれ1, 2, 3秒スリープする
- nextOfSplitは特に何もしない。ちゃんとsplit0内のflowが全部終わってから遷移するか確かめるために置いた
資源
資源はこのへんにまとめて全部ある
動かしてみる
ログ
21:38:24,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5) entering process(): stepName=1sec, sleepInMills=1,000 21:38:24,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8) entering process(): stepName=3sec, sleepInMills=3,000 21:38:24,422 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7) entering process(): stepName=2sec, sleepInMills=2,000 21:38:25,424 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5) exiting process(): stepName=1sec, sleepInMills=1,000 21:38:26,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7) exiting process(): stepName=2sec, sleepInMills=2,000 21:38:27,424 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8) exiting process(): stepName=3sec, sleepInMills=3,000 21:38:27,430 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=nextOfSplit, sleepInMills=null 21:38:27,430 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) exiting process(): stepName=nextOfSplit, sleepInMills=null
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
129 | 124 | | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:27.433 | 2014-02-15 21:38:27.433 | COMPLETED | COMPLETED | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid =129 order by stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
147 | 129 | | 1sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:25.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
148 | 129 | | 2sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:26.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
149 | 129 | | 3sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:27.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
150 | 129 | | nextOfSplit | 2014-02-15 21:38:27.428 | 2014-02-15 21:38:27.431 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(4 rows)
備考
- ちゃんとパラで走っているみたい
- [1]によると、いずれかのFlowが死ぬと、親のSplitも死ぬ
- Flowは複数のStepを1つにまとめる場合に使う。うまく使うとStep間の遷移をすっきり表現できる気がする。Splitの子要素として以外でも使えるようだ
参考文献
Tags: jbatch
