Partitioned Chunk Processingで並列処理してみる
TweetPosted on Monday Feb 10, 2014 at 06:50PM in Technology
簡単な並列処理をさせて遊ぶ
Partitioned Chunk Processingとは
バルク処理で使うChunk-oriented Processingで並列処理をやるための仕組み。詳細は[1]の11.7 Partitioned Chunk Processingとかを読むと良い（各処理がどのような順序で、どのスレッドで、どのトランザクションで実行されるかが書いてある）。Chunk-oriented ProcessingについてはChunk方式のStepを使ってみるを参照
ここではChunk-oriented Processingで出てきた3つのインタフェースに加えて、以下の4つのインタフェースを使う。
- PartitionMapper
- PartitionReducer
- PartitionCollector
- PartitionAnalyzer
全てが必須ではないようだが、機能の確認のためにここでは4インタフェースとも使ってみる。それぞれの役割と実装の仕方はざっくり以下のような感じになるかと思われる
PartitionMapper
PartitionPlan mapPartitions()
- どのような条件で並列実行させるかの情報をPartitionPlanインタフェースを実装するオブジェクトに詰めて返す
- int partitions: 処理をいくつに分けて実行するか
- int threads: 何パラで実行するか。デフォルトは0で、この場合暗黙的にpartitionsと同じ数とみなされる
- Properties[] partitionProperties: 各並列処理単位（以下ワーカーと呼ぶ）に与えるProperties
- 各ワーカーに番号を割り当てたりとか
- boolean partitionsOverride: 再実行時に前回実行時の結果を捨てるか否か？試してないので不明。デフォルトはfalse
- PartitionPlanImplというクラスがAPIで用意されているのでこいつを使うと楽
- メインスレッドで実行される
- トランザクション外で実行される
PartitionReducer
- 処理開始終了前後の処理とかを書く
- 各メソッドはメインスレッドで実行される
void beginPartitionedStep()
- Partition*系のインタフェース中で一番最初に実行されるメソッド
- 何か並列処理に関連する前処理をしたいときはここに書いたらいいのかも。具体的に何を書いたらいいかはよくわからん
- トランザクション外で実行される
void beforePartitionedStepCompletion()
- 全ワーカーが処理を全件終了した後に実行される
- 全ての処理結果が揃ってはじめて出来る処理とかがあれば、ここに書けば良さげ
- トランザクション終了前（ワーカーのトランザクションとは別）に実行される
void rollbackPartitionedStep()
- 何か問題が起きてロールバックする時、その直前に呼ばれる？
- エラー発生時、並列処理に関連して必要な後処理的なものがあればここに書けばよいのかも
- これが呼ばれるときはbeforePartitionedStepCompletion()は呼ばれないらしい
void afterPartitionedStepCompletion(PartitionStatus)
- Partition*系のインタフェース中で一番最後に実行されるメソッド
- 何か並列処理に関連する後処理をしたいときはここに書いたらいいのかも。具体的に何を書いたらいいかはよくわからん
- トランザクション外で実行される
PartitionCollector
Serializable collectPartitionData()
- 各ワーカーが作る処理結果を返す処理を書く
- ここで返した値はメインスレッドのPartitionAnalyzer#analyzeCollectorData()で受け取れる
- ワーカースレッド（各ワーカーが走るスレッドで、メインスレッドとは別）で実行される
- ワーカーの他のクラス（ItemWriterとか）と情報をやり取りするにはStepContext#getTransientUserData() / setTransientUserData()とかが使える
- チェックポイント到達後のコミットが走った後に実行される。なのでコミット単位ごとに少しずつ処理結果を送れる
- トランザクション外で実行される
- 想像だが、ここで返したSerializableはRepositoryに記録されて再実行時とかに良きに計らってくれたりするのかも。また再実行時の挙動は調べる
PartitionAnalyzer
void analyzeCollectorData(Serializable)
- PartitionAnalyzer#collectPartitionData()で返したオブジェクトが引数で渡ってくる
- メインスレッドで実行される
- トランザクション内（ワーカーのトランザクションとは別）で実行される
- チェックポイントごとに呼ばれるので、受け取ったデータの件数を数えれば進捗状況の出力とか出来そう
各クラス・スレッド間のデータの受け渡しについて
- ワーカースレッドからメインスレッドへデータを受け渡すのはPartitionCollectorとPartitionAnalyzerの役割
- 同一スレッド内でのデータの受け渡しはStepContext#getTransientUserData() / setTransientUserData()を使う
- StepContext, JobContextはそれぞれのスレッドで別のインスタンスになる
- CDIをうまく使えばもう少し楽に書けそうだが、また調べてみる
- バルク処理の性質上、基本的には無理にメモリ上でデータをやり取りしないでテーブルやファイルを介した方が良さげではある
バッチを作ってみる
- Chunk方式のStepを使ってみるで作ったバッチを、並列処理対応させて簡単なバッチを作ってみる。新規か変更を加えた資源について述べる
- 資源の詳細はこのあたり
- エンティティ、テストデータは流用する
環境・前提条件
- Chunk方式のStepを使ってみると同じ。この記事で作ったプロジェクトや資源がすでに存在するものとする
バッチを構成する資源
partition.xml (ジョブXML)
- 入力と出力は前回と全く同じ。Chunk方式のStep1個だけ
- 並列実行用の記述を加えている
- PartitionMapperとItemReaderにパラメータでパラ実行の総数を与える
- ItemReaderにパラメータでワーカーの番号を与える
- ItemProcessorにパラメータでスリープするミリ秒数を与える
PartitionItemReader.java
- SELECT文でパラメータで受け取ったワーカーの番号とパラ実行の総数とChunkInputItemのidを使って、取ってくるレコードを絞っている
- WHERE MOD(c.id, :partitions) = :partitionNumberとかそんな感じ
PartitionItemProcessor.java
- 並列処理っぽさを出すためにパラメータで受け取ったミリ秒数だけThread.sleep()する
PartitionItemWriter.java
- ChunkOutputItemをEntityManager#persist()に渡す替わりに、StepContext#setPersistentUserData()に突っ込む
- 渡した値はMyPartitionCollectorが取り出す
MyPartitionMapper.java
- パラメータで受け取ったパラ実行数と、ワーカーの番号をPropertiesに詰めて、さらにPartitionPlanImplにまとめて入れて返す
MyPartitionReducer.java
- PartitionItemWriter→MyPartitionCollector→MyPartitionAnalyzerの順で渡ってきたChunkOutputItemを全部まとめてここでDBに書き込む
MyPartitionCollector.java
- PartitionItemWriterでStepContextに突っ込んだChunkOutputItemを取り出して返す
- ここで返した値はMyPartitionAnalyzerに渡る
MyPartitionAnalyzer.java
- MyPartitionCollectorから渡ってきたChunkOutputItemをStepContext#transientUserDataに貯めていく
- データ量が多くなるとメモリがいっぱいになるのであくまでサンプル
テスト用資源
- テストデータ、検証用データは前回と同じ
PartitionJobTest.java
- テストメソッドは1パラ実行と2パラ実行の2つ
動かしてみる
1パラ実行時のログ
12:25:45,520 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beginPartitionedStep()
12:25:45,521 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionMapper] (batch-batch - 9) mapPartitions(): partitions=1
12:25:45,521 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionMapper] (batch-batch - 9) mapPartitions(): partitionProperty={partitionNumber=0}
12:25:45,522 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 9) postConstruct(): divide=2, sleep=1000
12:25:45,523 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) open(): checkpoint=null, index starts at=0, partitions=1, partitionNumber=0
12:25:45,524 DEBUG [org.hibernate.SQL] (batch-batch - 1) select chunkinput0_.id as id1_0_, chunkinput0_.input as input2_0_, chunkinput0_.processed as processe3_0_ from ChunkInputItem chunkinput0_ where mod(chunkinput0_.id, ?)=? order by chunkinput0_.id
12:25:45,526 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) open(): checkpoint=null, index starts at=0
12:25:45,527 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=0, returning=ChunkInputItem [id=0, input=0, processed=false]
12:25:45,527 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=0, input=0, processed=false], output=ChunkOutputItem [id=0, result=0]
12:25:46,528 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=1
12:25:46,528 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=1, returning=ChunkInputItem [id=1, input=10, processed=false]
12:25:46,528 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=1, input=10, processed=false], output=ChunkOutputItem [id=1, result=5]
12:25:47,529 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=2
12:25:47,530 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=2, returning=ChunkInputItem [id=2, input=20, processed=false]
12:25:47,530 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=2, input=20, processed=false], output=ChunkOutputItem [id=2, result=10]
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=3
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=0, item=ChunkOutputItem [id=0, result=0]
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=0, item=ChunkOutputItem [id=1, result=5]
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=0, item=ChunkOutputItem [id=2, result=10]
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=3
12:25:48,531 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) checkpointInfo(): returns=1
12:25:48,533 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 1) collectPartitionData(): data=[ChunkOutputItem [id=0, result=0], ChunkOutputItem [id=1, result=5], ChunkOutputItem [id=2, result=10]]
12:25:48,533 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=3
12:25:48,533 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeCollectorData(): data=[ChunkOutputItem [id=0, result=0], ChunkOutputItem [id=1, result=5], ChunkOutputItem [id=2, result=10]], receivedDataCount=3
12:25:48,535 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=3, returning=ChunkInputItem [id=3, input=30, processed=false]
12:25:48,536 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=3, input=30, processed=false], output=ChunkOutputItem [id=3, result=15]
12:25:49,537 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=4
12:25:49,537 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=4, returning=ChunkInputItem [id=4, input=40, processed=false]
12:25:49,537 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=4, input=40, processed=false], output=ChunkOutputItem [id=4, result=20]
12:25:50,538 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=5
12:25:50,538 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=5, returning=ChunkInputItem [id=5, input=50, processed=false]
12:25:50,539 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=5, input=50, processed=false], output=ChunkOutputItem [id=5, result=25]
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=6
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=1, item=ChunkOutputItem [id=3, result=15]
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=1, item=ChunkOutputItem [id=4, result=20]
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=1, item=ChunkOutputItem [id=5, result=25]
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=6
12:25:51,540 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) checkpointInfo(): returns=2
12:25:51,542 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 1) collectPartitionData(): data=[ChunkOutputItem [id=3, result=15], ChunkOutputItem [id=4, result=20], ChunkOutputItem [id=5, result=25]]
12:25:51,542 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=6
12:25:51,542 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeCollectorData(): data=[ChunkOutputItem [id=3, result=15], ChunkOutputItem [id=4, result=20], ChunkOutputItem [id=5, result=25]], receivedDataCount=6
12:25:51,543 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=6, returning=ChunkInputItem [id=6, input=60, processed=false]
12:25:51,543 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=6, input=60, processed=false], output=ChunkOutputItem [id=6, result=30]
12:25:52,544 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=7
12:25:52,544 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=7, returning=ChunkInputItem [id=7, input=70, processed=false]
12:25:52,545 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=7, input=70, processed=false], output=ChunkOutputItem [id=7, result=35]
12:25:53,546 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=8
12:25:53,546 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=8, returning=ChunkInputItem [id=8, input=80, processed=false]
12:25:53,546 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=8, input=80, processed=false], output=ChunkOutputItem [id=8, result=40]
12:25:54,547 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=9
12:25:54,547 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=2, item=ChunkOutputItem [id=6, result=30]
12:25:54,548 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=2, item=ChunkOutputItem [id=7, result=35]
12:25:54,548 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=2, item=ChunkOutputItem [id=8, result=40]
12:25:54,548 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=9
12:25:54,548 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) checkpointInfo(): returns=3
12:25:54,550 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 1) collectPartitionData(): data=[ChunkOutputItem [id=6, result=30], ChunkOutputItem [id=7, result=35], ChunkOutputItem [id=8, result=40]]
12:25:54,550 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=9
12:25:54,550 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeCollectorData(): data=[ChunkOutputItem [id=6, result=30], ChunkOutputItem [id=7, result=35], ChunkOutputItem [id=8, result=40]], receivedDataCount=9
12:25:54,551 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=9, returning=ChunkInputItem [id=9, input=90, processed=false]
12:25:54,552 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 1) processItem(): input=ChunkInputItem [id=9, input=90, processed=false], output=ChunkOutputItem [id=9, result=45]
12:25:55,553 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=10
12:25:55,553 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) readItem(): index=10, returning=null
12:25:55,553 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) writeItems(): index=3, item=ChunkOutputItem [id=9, result=45]
12:25:55,553 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=11
12:25:55,553 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) checkpointInfo(): returns=4
12:25:55,555 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 1) collectPartitionData(): data=[ChunkOutputItem [id=9, result=45]]
12:25:55,555 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) checkpointInfo(): returns=11
12:25:55,555 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeCollectorData(): data=[ChunkOutputItem [id=9, result=45]], receivedDataCount=10
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) close()
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) close()
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 1) collectPartitionData(): data=[]
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeCollectorData(): data=[], receivedDataCount=10
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 1) close()
12:25:55,556 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 1) close()
12:25:55,557 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 9) analyzeStatus(): batchStatus=COMPLETED, exitStatus=COMPLETED
12:25:55,557 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion()
12:25:55,557 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=0, result=0]
12:25:55,558 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=1, result=5]
12:25:55,558 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=2, result=10]
12:25:55,558 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=3, result=15]
12:25:55,558 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=4, result=20]
12:25:55,559 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=5, result=25]
12:25:55,559 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=6, result=30]
12:25:55,559 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=7, result=35]
12:25:55,559 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=8, result=40]
12:25:55,560 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=9, result=45]
12:25:55,561 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,562 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,562 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,563 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,564 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,564 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,565 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,565 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,566 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,566 DEBUG [org.hibernate.SQL] (batch-batch - 9) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:55,594 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 9) afterPartitionedStepCompletion(): partitionStatus=COMMIT
2パラ実行時のログ
12:25:38,265 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beginPartitionedStep()
12:25:38,265 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionMapper] (batch-batch - 4) mapPartitions(): partitions=2
12:25:38,266 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionMapper] (batch-batch - 4) mapPartitions(): partitionProperty={partitionNumber=0}
12:25:38,266 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionMapper] (batch-batch - 4) mapPartitions(): partitionProperty={partitionNumber=1}
12:25:38,267 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 4) postConstruct(): divide=2, sleep=1000
12:25:38,269 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) open(): checkpoint=null, index starts at=0, partitions=2, partitionNumber=0
12:25:38,271 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 4) postConstruct(): divide=2, sleep=1000
12:25:38,273 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) open(): checkpoint=null, index starts at=0, partitions=2, partitionNumber=1
12:25:38,273 DEBUG [org.hibernate.SQL] (batch-batch - 2) select chunkinput0_.id as id1_0_, chunkinput0_.input as input2_0_, chunkinput0_.processed as processe3_0_ from ChunkInputItem chunkinput0_ where mod(chunkinput0_.id, ?)=? order by chunkinput0_.id
12:25:38,274 DEBUG [org.hibernate.SQL] (batch-batch - 8) select chunkinput0_.id as id1_0_, chunkinput0_.input as input2_0_, chunkinput0_.processed as processe3_0_ from ChunkInputItem chunkinput0_ where mod(chunkinput0_.id, ?)=? order by chunkinput0_.id
12:25:38,276 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) open(): checkpoint=null, index starts at=0
12:25:38,278 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=0, returning=ChunkInputItem [id=1, input=10, processed=false]
12:25:38,279 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 8) processItem(): input=ChunkInputItem [id=1, input=10, processed=false], output=ChunkOutputItem [id=1, result=5]
12:25:38,279 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) open(): checkpoint=null, index starts at=0
12:25:38,280 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=0, returning=ChunkInputItem [id=0, input=0, processed=false]
12:25:38,280 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 2) processItem(): input=ChunkInputItem [id=0, input=0, processed=false], output=ChunkOutputItem [id=0, result=0]
12:25:39,280 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=1
12:25:39,280 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=1, returning=ChunkInputItem [id=3, input=30, processed=false]
12:25:39,281 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=1
12:25:39,281 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 8) processItem(): input=ChunkInputItem [id=3, input=30, processed=false], output=ChunkOutputItem [id=3, result=15]
12:25:39,281 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=1, returning=ChunkInputItem [id=2, input=20, processed=false]
12:25:39,281 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 2) processItem(): input=ChunkInputItem [id=2, input=20, processed=false], output=ChunkOutputItem [id=2, result=10]
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=2
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=2, returning=ChunkInputItem [id=5, input=50, processed=false]
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 8) processItem(): input=ChunkInputItem [id=5, input=50, processed=false], output=ChunkOutputItem [id=5, result=25]
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=2
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=2, returning=ChunkInputItem [id=4, input=40, processed=false]
12:25:40,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 2) processItem(): input=ChunkInputItem [id=4, input=40, processed=false], output=ChunkOutputItem [id=4, result=20]
12:25:41,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=3
12:25:41,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) writeItems(): index=0, item=ChunkOutputItem [id=1, result=5]
12:25:41,282 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) writeItems(): index=0, item=ChunkOutputItem [id=3, result=15]
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) writeItems(): index=0, item=ChunkOutputItem [id=5, result=25]
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=3
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) checkpointInfo(): returns=1
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=3
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) writeItems(): index=0, item=ChunkOutputItem [id=0, result=0]
12:25:41,283 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) writeItems(): index=0, item=ChunkOutputItem [id=2, result=10]
12:25:41,284 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) writeItems(): index=0, item=ChunkOutputItem [id=4, result=20]
12:25:41,284 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=3
12:25:41,284 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) checkpointInfo(): returns=1
12:25:41,285 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 8) collectPartitionData(): data=[ChunkOutputItem [id=1, result=5], ChunkOutputItem [id=3, result=15], ChunkOutputItem [id=5, result=25]]
12:25:41,285 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 2) collectPartitionData(): data=[ChunkOutputItem [id=0, result=0], ChunkOutputItem [id=2, result=10], ChunkOutputItem [id=4, result=20]]
12:25:41,286 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=3
12:25:41,286 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=3
12:25:41,286 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[ChunkOutputItem [id=1, result=5], ChunkOutputItem [id=3, result=15], ChunkOutputItem [id=5, result=25]], receivedDataCount=3
12:25:41,286 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[ChunkOutputItem [id=0, result=0], ChunkOutputItem [id=2, result=10], ChunkOutputItem [id=4, result=20]], receivedDataCount=6
12:25:41,287 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=3, returning=ChunkInputItem [id=7, input=70, processed=false]
12:25:41,287 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=3, returning=ChunkInputItem [id=6, input=60, processed=false]
12:25:41,287 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 8) processItem(): input=ChunkInputItem [id=7, input=70, processed=false], output=ChunkOutputItem [id=7, result=35]
12:25:41,287 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 2) processItem(): input=ChunkInputItem [id=6, input=60, processed=false], output=ChunkOutputItem [id=6, result=30]
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=4
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=4
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=4, returning=ChunkInputItem [id=9, input=90, processed=false]
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=4, returning=ChunkInputItem [id=8, input=80, processed=false]
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 8) processItem(): input=ChunkInputItem [id=9, input=90, processed=false], output=ChunkOutputItem [id=9, result=45]
12:25:42,288 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemProcessor] (batch-batch - 2) processItem(): input=ChunkInputItem [id=8, input=80, processed=false], output=ChunkOutputItem [id=8, result=40]
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=5
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=5
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) readItem(): index=5, returning=null
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) readItem(): index=5, returning=null
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) writeItems(): index=1, item=ChunkOutputItem [id=7, result=35]
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) writeItems(): index=1, item=ChunkOutputItem [id=6, result=30]
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) writeItems(): index=1, item=ChunkOutputItem [id=9, result=45]
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) writeItems(): index=1, item=ChunkOutputItem [id=8, result=40]
12:25:43,290 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=6
12:25:43,291 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=6
12:25:43,291 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) checkpointInfo(): returns=2
12:25:43,291 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) checkpointInfo(): returns=2
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 2) collectPartitionData(): data=[ChunkOutputItem [id=6, result=30], ChunkOutputItem [id=8, result=40]]
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 8) collectPartitionData(): data=[ChunkOutputItem [id=7, result=35], ChunkOutputItem [id=9, result=45]]
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) checkpointInfo(): returns=6
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) checkpointInfo(): returns=6
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[ChunkOutputItem [id=6, result=30], ChunkOutputItem [id=8, result=40]], receivedDataCount=8
12:25:43,293 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[ChunkOutputItem [id=7, result=35], ChunkOutputItem [id=9, result=45]], receivedDataCount=10
12:25:43,294 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) close()
12:25:43,294 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) close()
12:25:43,294 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) close()
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) close()
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 8) collectPartitionData(): data=[]
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionCollector] (batch-batch - 2) collectPartitionData(): data=[]
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[], receivedDataCount=10
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 2) close()
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemReader] (batch-batch - 8) close()
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 2) close()
12:25:43,295 FINE [org.nailedtothex.jbatch.example.partition.PartitionItemWriter] (batch-batch - 8) close()
12:25:43,296 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeStatus(): batchStatus=COMPLETED, exitStatus=COMPLETED
12:25:43,296 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeCollectorData(): data=[], receivedDataCount=10
12:25:43,297 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionAnalyzer] (batch-batch - 4) analyzeStatus(): batchStatus=COMPLETED, exitStatus=COMPLETED
12:25:43,297 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion()
12:25:43,298 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=1, result=5]
12:25:43,298 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=3, result=15]
12:25:43,298 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=5, result=25]
12:25:43,298 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=0, result=0]
12:25:43,299 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=2, result=10]
12:25:43,299 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=4, result=20]
12:25:43,299 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=6, result=30]
12:25:43,299 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=8, result=40]
12:25:43,299 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=7, result=35]
12:25:43,300 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) beforePartitionedStepCompletion(): outputItem=ChunkOutputItem [id=9, result=45]
12:25:43,304 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,305 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,306 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,307 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,307 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,308 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,308 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,309 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,309 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,310 DEBUG [org.hibernate.SQL] (batch-batch - 4) insert into ChunkOutputItem (result, id) values (?, ?)
12:25:43,337 FINE [org.nailedtothex.jbatch.example.partition.MyPartitionReducer] (batch-batch - 4) afterPartitionedStepCompletion(): partitionStatus=COMMIT
job_execution
1パラ実行時
jbatch=# select * from job_execution where jobexecutionid=73;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------+-----------------
73 | 68 | | 2014-02-14 12:25:45.517 | 2014-02-14 12:25:45.517 | 2014-02-14 12:25:55.596 | 2014-02-14 12:25:55.596 | COMPLETED | COMPLETED | sleep = 1000 +|
| | | | | | | | | partitions = 1+|
| | | | | | | | | divide = 2 +|
| | | | | | | | | |
(1 row)
2パラ実行時
jbatch=# select * from job_execution where jobexecutionid=72;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+------------------------+------------------------+-------------+------------+----------------+-----------------
72 | 67 | | 2014-02-14 12:25:38.262 | 2014-02-14 12:25:38.262 | 2014-02-14 12:25:43.34 | 2014-02-14 12:25:43.34 | COMPLETED | COMPLETED | sleep = 1000 +|
| | | | | | | | | partitions = 2+|
| | | | | | | | | divide = 2 +|
| | | | | | | | | |
(1 row)
実行時間が半分になっている
step_execution
1パラ実行時
jbatch=# select * from step_execution where jobexecutionid=73;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+----------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
88 | 73 | | doChunk | 2014-02-14 12:25:45.519 | 2014-02-14 12:25:55.595 | COMPLETED | COMPLETED | | | 10 | 10 | 4 | 0 | 0 | 0 | 0 | 0 | |
(1 row)
2パラ実行時
jbatch=# select * from step_execution where jobexecutionid=72;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+----------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
87 | 72 | | doChunk | 2014-02-14 12:25:38.263 | 2014-02-14 12:25:43.339 | COMPLETED | COMPLETED | | | 10 | 10 | 4 | 0 | 0 | 0 | 0 | 0 | |
(1 row)
partition_execution
1パラ実行時
jbatch=# select * from partition_execution where stepexecutionid = 88;
partitionexecutionid | stepexecutionid | version | batchstatus | exitstatus | executionexception | persistentuserdata | readercheckpointinfo | writercheckpointinfo
----------------------+-----------------+---------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 | 88 | | COMPLETED | COMPLETED | | \xaced00057372001f6a6176612e7574696c2e436f6c6c656374696f6e7324456d7074794c6973747ab817b43ca79ede0200007870 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b02000078700000000b | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000004
(1 row)
jbatch=#
2パラ実行時
jbatch=# select * from partition_execution where stepexecutionid = 87 order by partitionexecutionid;
partitionexecutionid | stepexecutionid | version | batchstatus | exitstatus | executionexception | persistentuserdata | readercheckpointinfo | writercheckpointinfo
----------------------+-----------------+---------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 | 87 | | COMPLETED | COMPLETED | | \xaced00057372001f6a6176612e7574696c2e436f6c6c656374696f6e7324456d7074794c6973747ab817b43ca79ede0200007870 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000006 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000002
1 | 87 | | COMPLETED | COMPLETED | | \xaced00057372001f6a6176612e7574696c2e436f6c6c656374696f6e7324456d7074794c6973747ab817b43ca79ede0200007870 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000006 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000002
(2 rows)
jbatch=#
checkpointInfoをパラ実行単位で持っているようなので、再実行時は途中からの処理再開が可能なのだろうか。また試してみる
備考
- もう少し実践的な使い方がわかるようなサンプルが有れば良いのだが見つからない。もしSpring Batchに似たような機能があれば、そのサンプルを参考にできるかも
- そのうち異常終了時の再実行を試してみる
参考文献
Tags: jbatch
