Entries tagged [jbatch]
A way to resolve dependencies of jobs
TweetPosted on Sunday Feb 16, 2014 at 08:17AM in Technology
This is about a way to resolve dependencies of jobs that used only JSR 352 API.
Environment
- jBeret 1.0.1Beta-SNAPSHOT
- WildFly 8.0.0.Final
Why need it?
- For tasks that need to execute sequentially, usually we have to let all tasks in single job XML.
- I guess it would bring huge difficulty of testing.
- For example, I don't want to test such big job like EndOfTheDay.xml
- Rather than I would go with smaller jobs with external job control system.
How realize it?
- This needs to create parent job XML that defines dependencies of child jobs.
- Every step uses artifact named “jobLauncherBatchlet”
- This batchlet invokes child job. name of the job and job parameters are given by its parameter of the step.
- After invoke the job, then it starts polling BatchStatus, and wait till the job is finished.
Sample project
Whole resources are available in GitHub.
- Artifacts
- jobXMLs
- Test class
log
13:45:22,589 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: persistentUserData=null, waitForFinish=null
13:45:22,589 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: starting new job: jobXMLName=jobdependency-child1, properties={sleepInMillis=1000}
13:45:22,593 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job started: executionId=207
13:45:22,593 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: waiting for finish the job... executionId=207
13:45:22,595 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=child1-step1, sleepInMills=1,000
13:45:23,596 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) exiting process(): stepName=child1-step1, sleepInMills=1,000
13:45:24,595 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job finished: BatchStatus=COMPLETED
13:45:24,601 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: persistentUserData=null, waitForFinish=null
13:45:24,601 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: starting new job: jobXMLName=jobdependency-child2, properties={sleepInMillis=2000}
13:45:24,605 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job started: executionId=208
13:45:24,605 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: waiting for finish the job... executionId=208
13:45:24,606 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=child2-step1, sleepInMills=2,000
13:45:26,608 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) exiting process(): stepName=child2-step1, sleepInMills=2,000
13:45:27,608 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job finished: BatchStatus=COMPLETED
13:45:27,614 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: persistentUserData=null, waitForFinish=null
13:45:27,614 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: starting new job: jobXMLName=jobdependency-child3, properties={sleepInMillis=3000}
13:45:27,618 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job started: executionId=209
13:45:27,618 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: waiting for finish the job... executionId=209
13:45:27,619 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) entering process(): stepName=child3-step1, sleepInMills=3,000
13:45:30,620 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) exiting process(): stepName=child3-step1, sleepInMills=3,000
13:45:31,620 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job finished: BatchStatus=COMPLETED
Job repository tables
job_instance and job_execution
jbatch=# select * from job_instance i, job_execution e where i.jobinstanceid = e.jobinstanceid and i.jobname like 'jobdependency%' and i.jobinstanceid >= 194 order by e.jobexecutionid;
jobinstanceid | version | jobname | applicationname | jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
---------------+---------+----------------------+-----------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
194 | | jobdependency | jbatchtest | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | |
195 | | jobdependency-child1 | jbatchtest | 207 | 195 | | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED | COMPLETED | sleepInMillis = 1000+|
| | | | | | | | | | | | | |
196 | | jobdependency-child2 | jbatchtest | 208 | 196 | | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED | COMPLETED | sleepInMillis = 2000+|
| | | | | | | | | | | | | |
197 | | jobdependency-child3 | jbatchtest | 209 | 197 | | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED | COMPLETED | sleepInMillis = 3000+|
| | | | | | | | | | | | | |
(4 rows)
step_execution
jbatch=# select * from step_execution s, job_execution e, job_instance i where s.jobexecutionid = e.jobexecutionid and e.jobinstanceid = i.jobinstanceid and i.jobinstanceid >= 194 order by s.starttime;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo | jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition | jobinstanceid | version | jobname | applicationname
-----------------+----------------+---------+--------------+-------------------------+-------------------------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------+---------------+---------+----------------------+-----------------
277 | 206 | | step1 | 2014-02-19 13:45:22.587 | 2014-02-19 13:45:24.595 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000cf | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
278 | 207 | | child1-step1 | 2014-02-19 13:45:22.593 | 2014-02-19 13:45:23.596 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 207 | 195 | | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED | COMPLETED | sleepInMillis = 1000+| | 195 | | jobdependency-child1 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
279 | 206 | | step2 | 2014-02-19 13:45:24.599 | 2014-02-19 13:45:27.608 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
280 | 208 | | child2-step1 | 2014-02-19 13:45:24.605 | 2014-02-19 13:45:26.608 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 208 | 196 | | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED | COMPLETED | sleepInMillis = 2000+| | 196 | | jobdependency-child2 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
281 | 206 | | step3 | 2014-02-19 13:45:27.612 | 2014-02-19 13:45:31.621 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
282 | 209 | | child3-step1 | 2014-02-19 13:45:27.618 | 2014-02-19 13:45:30.62 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 209 | 197 | | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED | COMPLETED | sleepInMillis = 3000+| | 197 | | jobdependency-child3 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
(6 rows)
- We can see that they were executed in particular order.
Remarks
- I intent to care of stop and restart, and sample project contains some codes about it, but I haven't tested it yet. maybe I would write about it too later.
- It can use some functions of jobXML such as parallel executions (like split element) for control of job execution even.
Take care of the max number of threads in thread-pool
- Some implementations have limit of threads count.
- When you used this method or similar one with many parallel execution elements (split or partition), you must ensure to keep number of threads lower than the limit.
- If limit reaches, your batch will be frozen and it won't resume.
How to configure max threads
- On WildFly + jBeret, the max number of threads is set to 10 as default.
- This increases the limit to 100:
/subsystem=batch/thread-pool=batch:write-attribute(name=max-threads, value=100)
Tags: jbatch
StepContext#persistentUserDataで遊ぶ
TweetPosted on Sunday Feb 16, 2014 at 07:43AM in Technology
StepContext#persistentUserDataにはStepに紐づくデータを保存しておける。ここに入れたデータはJobRepositoryに保存され、後続のStepやジョブの再実行時に参照できる。何かしら使い道があると思われるので若干遊んでみる
環境・前提条件
- Chunk方式のStepで例外発生時にスキップさせてみると同じ。ここで作ったプロジェクトが普通に動いているものとする
関連メソッドの仕様を見てみる
[1]から引いてみる
javax.batch.runtime.context.StepContextインタフェース
Stepに関連する操作をする時に使うインタフェース。artifactで@Injectで注入して使える。
/**
* The getPersistentUserData method returns a persistent data object
* belonging to the current step. The user data type must implement
* java.util.Serializable. This data is saved as part of a step's
* checkpoint. For a step that does not do checkpoints, it is saved
* after the step ends. It is available upon restart.
* @return user-specified type
*/
public Serializable getPersistentUserData();
/**
* The setPersistentUserData method stores a persistent data object
* into the current step. The user data type must implement
* java.util.Serializable. This data is saved as part of a step's
* checkpoint. For a step that does not do checkpoints, it is saved
* after the step ends. It is available upon restart.
* @param data is the user-specified type
*/
public void setPersistentUserData(Serializable data);
javax.batch.runtime.StepExecutionインタフェース
Stepの実行履歴的な情報を含むインタフェース。JobOperatorを使って引っ張って来れたり、Deciderのメソッドの引数で受け取れたりする。
/**
* Get user persistent data
* @return persistent data
*/
public Serializable getPersistentUserData();
先行StepでセットしたpersistentUserDataを後続Stepで参照してみる
サンプルの仕様
- 第一レベルの要素は動作順に以下
- set (setPersistentUserDataBatchlet)
- get (getPersistentUserDataBatchlet)
- setでセットしたpersistentUserDataをgetで参照してみる
資源
資源はこのへんにまとめて全部ある
- バッチ本体
- テスト
動かしてみる
ログ
08:52:02,642 FINE [org.nailedtothex.jbatch.example.persistentuserdata.SetPersistentUserDataBatchlet] (batch-batch - 4) set: process() 08:52:02,649 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) get: process() 08:52:02,650 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) stepExecution: stepName=set, persistentUserData=* my step name is set * 08:52:02,650 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) stepExecution: stepName=get, persistentUserData=null
setステップで設定した「 my step name is set 」がgetステップから見えている（3行目）。うむ
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
131 | 126 | | 2014-02-16 08:52:02.637 | 2014-02-16 08:52:02.637 | 2014-02-16 08:52:02.653 | 2014-02-16 08:52:02.653 | COMPLETED | COMPLETED | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid = 131 order by stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+----------+-------------------------+-------------------------+-------------+------------+--------------------+----------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
153 | 131 | | set | 2014-02-16 08:52:02.639 | 2014-02-16 08:52:02.642 | COMPLETED | COMPLETED | | \xaced00057400172a206d792073746570206e616d6520697320736574202a | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
154 | 131 | | get | 2014-02-16 08:52:02.645 | 2014-02-16 08:52:02.65 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(2 rows)
persistentuserdataカラムに何かが入っている。うむ
前回実行時のデータを再実行時に参照してみる
サンプルの仕様
- Stepは1つだけ
- 何もpersistentUserDataに入ってなかったら
- persistentUserDataにデータを入れる
- 例外投げて死ぬ
- 何かpersistentUserDataに入っていたら
- 内容を表示して終了
- 初回実行はpersistentUserDataにデータ入れてFAILEDになる
- 再実行時はpersistentUserDataのデータを表示してCOMPLETEDになる
資源
資源はこのへんにまとめて全部ある
- バッチ本体
- テスト
動かしてみる
ログ
09:15:07,338 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 3) getAndSet: process()
09:15:07,339 WARN [org.jberet] (batch-batch - 3) JBERET000001: Failed to run batchlet org.jberet.job.model.RefArtifact@7e76d151: java.lang.RuntimeException: to confirm whether the data is visible or not when restart
at org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet.process(GetAndSetPersistentUserDataBatchlet.java:31) [classes:]
at org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:61) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:207) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149) [wildfly-jberet-8.0.0.Final.jar:8.0.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
09:15:08,351 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 2) getAndSet: process()
09:15:08,351 FINE [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 2) persistentUserData=* my job execution id is 134 *
初回実行時に入れた「 my job execution id is 134 」が再実行時に見えている。うむ
Repository
job_execution
jbatch=# select * from job_execution where jobexecutionid in (134, 135) order by jobexecutionid;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
134 | 128 | | 2014-02-16 09:15:07.335 | 2014-02-16 09:15:07.335 | 2014-02-16 09:15:07.343 | 2014-02-16 09:15:07.343 | FAILED | FAILED | |
135 | 128 | | 2014-02-16 09:15:08.347 | 2014-02-16 09:15:08.347 | 2014-02-16 09:15:08.354 | 2014-02-16 09:15:08.354 | COMPLETED | COMPLETED | |
(2 rows)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (134, 135) order by jobexecutionid, stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+-----------+-------------------------+-------------------------+-------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
157 | 134 | | getAndSet | 2014-02-16 09:15:07.336 | 2014-02-16 09:15:07.34 | FAILED | FAILED | java.lang.RuntimeException: to confirm whether the data is visible or not when restart +| \xaced000574001e2a206d79206a6f6220657865637574696f6e20696420697320313334202a | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| | | | | | | | at org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet.process(GetAndSetPersistentUserDataBatchlet.java:31)+| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:61) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:207) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58) +| | | | | | | | | | |
| | | | | | | | at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.FutureTask.run(FutureTask.java:262) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) +| | | | | | | | | | |
| | | | | | | | at java.lang.Thread.run(Thread.java:744) +| | | | | | | | | | |
| | | | | | | | at org.jboss.threads.JBossThread.run(JBossThread.java:122) +| | | | | | | | | | |
| | | | | | | | | | | | | | | | | | |
158 | 135 | | getAndSet | 2014-02-16 09:15:08.349 | 2014-02-16 09:15:08.352 | COMPLETED | COMPLETED | | \xaced000574001e2a206d79206a6f6220657865637574696f6e20696420697320313334202a | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(2 rows)
備考
- 再実行時にはpersistentUserDataがコピーされるようなので、データ量がでかい場合は気をつけた方がいいのかも
参考文献
Tags: jbatch
Decisionでフロー制御してみる
TweetPosted on Saturday Feb 15, 2014 at 08:20PM in Technology
フロー制御は、ExitStatusでフロー制御してみるでやったようにBatchletだけでもできるけど、Chunk要素などの後続処理のフロー制御はDecision要素を使うと楽にできそうなので、試してみる
例えば先行のChunk処理でスキップが多すぎる時に、一応ジョブは正常終了しつつも警告メール的なものを送っておくとか、そういうことをやりたい時にはDecisionを使うとよいかもしれない。引数でStepExecution[]が渡ってくるので、先行処理のMetricsを取り出して調べたりすることが楽にできる。
環境・前提条件
- Chunk方式のStepで例外発生時にスキップさせてみると同じ。ここで作ったプロジェクトが普通に動いているものとする
仕様を確認する
- [2]から引いてみる
public interface Decider
A Decider receives control as part of a decision element in a job. It is used to direct execution flow during job processing. It returns an exit status that updates the current job execution's exit status. This exit status value also directs the execution transition based on next, end, stop, fail child elements configured on the same decision element as the decider.
String decide(StepExecution[] executions) throws Exception
The decide method sets a new exit status for a job. It receives an array of StepExecution objects as input. These StepExecution objects represent the execution element that transitions to this decider as follows:
Step
When the transition is from a step, the decide method receives the StepExecution corresponding to the step as input.
Split
When the transition is from a split, the decide method receives a StepExecution from each flow defined to the split as input.
Flow
When the transition is from a flow, the decide method receives a StepExecution corresponding to the last execution element that completed in the flow. This will be a single StepExecution if the last element was a step and multiple StepExecutions if the last element was a split.
Parameters:
executions - specifies the StepExecution(s) of the preceding element.
Returns:
updated job exit status
- とりあえず上に書かれた3パターンで試してみる
Stepからの遷移の場合
資源はこのへんにまとめて全部ある
資源
仕様
- 第一レベルの要素は以下
- step1 (batchlet step)
- step2 (chunk oriented step)
- step2decision (decision)
- step3_NO_SKIP_FOUND (batchlet step)
- step4_SKIP_FOUND (batchlet step)
- step2でスキップがなかった場合はstep3_NO_SKIP_FOUNDに遷移する
- step2でスキップがあった場合はstep4_SKIP_FOUNDに遷移する
スキップが無い場合のテストを動かしてみる
ログ
13:53:59,614 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): executions=1 13:53:59,614 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=step2, stepExecutionId=248 13:53:59,614 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=10 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=4 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=10 13:53:59,615 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): exitStatus=SKIP_NOT_FOUND 13:53:59,617 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 10) step3_NO_SKIP_FOUND
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+------------------------+------------------------+-------------------------+-------------------------+-------------+----------------+-----------------------+-----------------
188 | 176 | | 2014-02-16 13:53:59.41 | 2014-02-16 13:53:59.41 | 2014-02-16 13:53:59.619 | 2014-02-16 13:53:59.619 | COMPLETED | SKIP_NOT_FOUND | itemReaderFailAt = -1+|
| | | | | | | | | divide = 2 +|
| | | | | | | | | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (188) order by jobexecutionid, stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+---------------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
247 | 188 | | step1 | 2014-02-16 13:53:59.412 | 2014-02-16 13:53:59.453 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
248 | 188 | | step2 | 2014-02-16 13:53:59.46 | 2014-02-16 13:53:59.61 | COMPLETED | COMPLETED | | | 10 | 10 | 4 | 0 | 0 | 0 | 0 | 0 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b02000078700000000a | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000004
249 | 188 | | step3_NO_SKIP_FOUND | 2014-02-16 13:53:59.615 | 2014-02-16 13:53:59.617 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(3 rows)
jbatch=#
スキップが有る場合のテストを動かしてみる
ログ
13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): executions=1 13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): stepName=step2, stepExecutionId=251 13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: WRITE_SKIP_COUNT=0 13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: PROCESS_SKIP_COUNT=0 13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: WRITE_COUNT=9 13:55:00,761 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: COMMIT_COUNT=4 13:55:00,762 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: READ_SKIP_COUNT=1 13:55:00,762 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: ROLLBACK_COUNT=0 13:55:00,762 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: FILTER_COUNT=0 13:55:00,762 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: READ_COUNT=9 13:55:00,762 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): exitStatus=SKIP_FOUND 13:55:00,763 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 5) step4_SKIP_FOUND
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
189 | 177 | | 2014-02-16 13:55:00.639 | 2014-02-16 13:55:00.639 | 2014-02-16 13:55:00.765 | 2014-02-16 13:55:00.765 | COMPLETED | SKIP_FOUND | itemReaderFailAt = 5+|
| | | | | | | | | divide = 2 +|
| | | | | | | | | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (189) order by jobexecutionid, stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+------------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
250 | 189 | | step1 | 2014-02-16 13:55:00.641 | 2014-02-16 13:55:00.642 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
251 | 189 | | step2 | 2014-02-16 13:55:00.644 | 2014-02-16 13:55:00.758 | COMPLETED | COMPLETED | | | 9 | 9 | 4 | 0 | 1 | 0 | 0 | 0 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b02000078700000000a | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000003
252 | 189 | | step4_SKIP_FOUND | 2014-02-16 13:55:00.762 | 2014-02-16 13:55:00.763 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(3 rows)
jbatch=#
- Decider#decide()に渡されるStepExecutionは、基本的に遷移元Stepのものだけのようだ
Splitからの遷移の場合
資源
仕様
- 第一レベルの要素は以下
- split1 (split)
- split1decision (decision)
- split1の下には同じようなflowが3つある
- flowの下にはstep (batchlet) が2つ
- decisionの後の分岐とスキップはもう試したので省略
テストを動かしてみる
ログ
13:42:04,217 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 3) flow3-step1 13:42:04,217 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step1 13:42:04,217 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 2) flow2-step1 13:42:04,221 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=flow2-step2, sleepInMills=2,000 13:42:04,221 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 3) entering process(): stepName=flow3-step2, sleepInMills=3,000 13:42:04,221 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 6) entering process(): stepName=flow1-step2, sleepInMills=1,000 13:42:05,222 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 6) exiting process(): stepName=flow1-step2, sleepInMills=1,000 13:42:06,222 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) exiting process(): stepName=flow2-step2, sleepInMills=2,000 13:42:07,222 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 3) exiting process(): stepName=flow3-step2, sleepInMills=3,000 13:42:07,225 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): executions=3 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow1-step2, stepExecutionId=243 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0 13:42:07,226 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow2-step2, stepExecutionId=242 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0 13:42:07,227 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow3-step2, stepExecutionId=244 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0 13:42:07,228 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): exitStatus=SKIP_NOT_FOUND
- 遷移元がSplitの場合、Splitの子Flowの数だけStepExecutionが渡される（Flowの最後の要素のStepExecution）。だから配列なのか。なるほどね
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+----------------+---------------+-----------------
186 | 174 | | 2014-02-16 13:42:04.198 | 2014-02-16 13:42:04.198 | 2014-02-16 13:42:07.229 | 2014-02-16 13:42:07.229 | COMPLETED | SKIP_NOT_FOUND | |
(1 row)
jbatch=#
step_execution
jbatch=# select * from step_execution where jobexecutionid in (186) order by jobexecutionid, stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
239 | 186 | | flow1-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
240 | 186 | | flow3-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
241 | 186 | | flow2-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
242 | 186 | | flow2-step2 | 2014-02-16 13:42:04.22 | 2014-02-16 13:42:06.223 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
243 | 186 | | flow1-step2 | 2014-02-16 13:42:04.22 | 2014-02-16 13:42:05.223 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
244 | 186 | | flow3-step2 | 2014-02-16 13:42:04.22 | 2014-02-16 13:42:07.222 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(6 rows)
Flowからの遷移の場合
資源
仕様
- 第一レベルの要素は以下
- flow1 (flow)
- flowdecision (decision)
- flow1の下にはstep (batchlet) が2つ
テストを動かしてみる
ログ
13:47:01,234 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step1 13:47:01,237 FINE [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step2 13:47:01,239 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): executions=1 13:47:01,239 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): stepName=flow1-step2, stepExecutionId=246 13:47:01,239 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: WRITE_SKIP_COUNT=0 13:47:01,239 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: PROCESS_SKIP_COUNT=0 13:47:01,239 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: WRITE_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: COMMIT_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: READ_SKIP_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: ROLLBACK_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: FILTER_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: READ_COUNT=0 13:47:01,240 FINE [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): exitStatus=SKIP_NOT_FOUND
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+------------------------+------------------------+-------------+----------------+---------------+-----------------
187 | 175 | | 2014-02-16 13:47:01.231 | 2014-02-16 13:47:01.231 | 2014-02-16 13:47:01.24 | 2014-02-16 13:47:01.24 | COMPLETED | SKIP_NOT_FOUND | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (187) order by jobexecutionid, stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
245 | 187 | | flow1-step1 | 2014-02-16 13:47:01.232 | 2014-02-16 13:47:01.234 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
246 | 187 | | flow1-step2 | 2014-02-16 13:47:01.236 | 2014-02-16 13:47:01.237 | COMPLETED | COMPLETED | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(2 rows)
jbatch=#
- 遷移元がFlowの場合はFlowの一番最後のStepのStepExecutionだけが渡される
備考
- Decider#decide()に渡されるStepExecutionは以下のようになる
- 遷移元がStepの場合: 遷移元StepのStepExecution (配列の要素数は1個)
- 遷移元がSplitの場合: 遷移元Splitの各子Flowの最後のStepのStepExecution (配列の要素数は複数個)
- 遷移元がFlowの場合: 遷移元Flowの最後のStepのStepExecution (配列の要素数は1個)
参考文献
Tags: jbatch
Splitで並列処理してみる
TweetPosted on Saturday Feb 15, 2014 at 08:19PM in Technology
「Partitioned Chunk Processingで遊ぶ」で使ったPartitionは一つの処理を手分けしてやる仕組みだが、splitは異なる処理を並列で行う仕組み。簡単なバッチを作って遊んでみる。
環境・前提条件
- Chunk方式のStepで例外発生時にスキップさせてみると同じ。ここで作ったプロジェクトが普通に動いているものとする
バッチを作ってみる
仕様
- 一番高レベルの要素は2つ。順番に動く
- split0 (Split)
- nextOfSplit (Step)
- split0にはflowが3つある
- 全てのflowは1つのstepを持つ
- 使っているartifactは全て同じ。単純なBatchletで、パラメータで与えられたミリ秒数だけThread.sleep()する
- それぞれ1, 2, 3秒スリープする
- nextOfSplitは特に何もしない。ちゃんとsplit0内のflowが全部終わってから遷移するか確かめるために置いた
資源
資源はこのへんにまとめて全部ある
動かしてみる
ログ
21:38:24,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5) entering process(): stepName=1sec, sleepInMills=1,000 21:38:24,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8) entering process(): stepName=3sec, sleepInMills=3,000 21:38:24,422 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7) entering process(): stepName=2sec, sleepInMills=2,000 21:38:25,424 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5) exiting process(): stepName=1sec, sleepInMills=1,000 21:38:26,423 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7) exiting process(): stepName=2sec, sleepInMills=2,000 21:38:27,424 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8) exiting process(): stepName=3sec, sleepInMills=3,000 21:38:27,430 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=nextOfSplit, sleepInMills=null 21:38:27,430 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) exiting process(): stepName=nextOfSplit, sleepInMills=null
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
129 | 124 | | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:27.433 | 2014-02-15 21:38:27.433 | COMPLETED | COMPLETED | |
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid =129 order by stepexecutionid;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
147 | 129 | | 1sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:25.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
148 | 129 | | 2sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:26.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
149 | 129 | | 3sec | 2014-02-15 21:38:24.42 | 2014-02-15 21:38:27.424 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
150 | 129 | | nextOfSplit | 2014-02-15 21:38:27.428 | 2014-02-15 21:38:27.431 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(4 rows)
備考
- ちゃんとパラで走っているみたい
- [1]によると、いずれかのFlowが死ぬと、親のSplitも死ぬ
- Flowは複数のStepを1つにまとめる場合に使う。うまく使うとStep間の遷移をすっきり表現できる気がする。Splitの子要素として以外でも使えるようだ
参考文献
Tags: jbatch
JBeret1.0.0 skip problem
TweetPosted on Friday Feb 14, 2014 at 10:07PM in Technology
Problem
- skippable-exception-classes not worked expectedly
- I declared a exception to my jobXML, but it seems just ignored. my batch job stopped at exception and failed.
Environment
- WildFly8.0.0.Final
- Oracle JDK7u51
Project to reproduce the problem
- Entire the project is available at GitHub
Spec
- ItemReader supplies int array which contains 3 items
- ItemProcessor throws RuntimeException at second item
- RuntimeException declared in skippable-exception-classes
How to reproduce
- Deploy the project
- Browse http://localhost:8080/jsr352-skip/
- servlet kicks the batch job
- Check log of application server and job repository
Log
22:03:49,505 INFO [stdout] (batch-batch - 1) readItem: 1 22:03:49,505 INFO [stdout] (batch-batch - 1) process: i=1, item=1 22:03:49,506 INFO [stdout] (batch-batch - 1) readItem: 2 22:03:49,506 ERROR [org.jberet] (batch-batch - 1) JBERET000007: Failed to run job skip, doChunk, org.jberet.job.model.Chunk@50e84eee: java.lang.RuntimeException: I want to skip this! at example.ExampleItemProcessor.processItem(ExampleItemProcessor.java:14) [classes:] at org.jberet.runtime.runner.ChunkRunner.processItem(ChunkRunner.java:396) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.ChunkRunner.readProcessWriteItems(ChunkRunner.java:295) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.ChunkRunner.run(ChunkRunner.java:193) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:204) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58) [jberet-core-1.0.0.Final.jar:1.0.0.Final] at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149) [wildfly-jberet-8.0.0.Final.jar:8.0.0.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Repository
job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
101 | 96 | | 2014-02-14 22:03:49.478 | 2014-02-14 22:03:49.478 | 2014-02-14 22:03:49.515 | 2014-02-14 22:03:49.515 | FAILED | FAILED | |
(1 row)
jbatch=#
step_execution
jbatch=# select * from step_execution where jobexecutionid =101;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo
-----------------+----------------+---------+----------+-------------------------+-------------------------+-------------+------------+--------------------------------------------------------------------------------------------------------------------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
116 | 101 | | doChunk | 2014-02-14 22:03:49.489 | 2014-02-14 22:03:49.508 | FAILED | FAILED | java.lang.RuntimeException: I want to skip this! +| | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| | | | | | | | at example.ExampleItemProcessor.processItem(ExampleItemProcessor.java:14) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.ChunkRunner.processItem(ChunkRunner.java:396) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.ChunkRunner.readProcessWriteItems(ChunkRunner.java:295) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.ChunkRunner.run(ChunkRunner.java:193) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:204) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) +| | | | | | | | | | |
| | | | | | | | at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58) +| | | | | | | | | | |
| | | | | | | | at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149)+| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.FutureTask.run(FutureTask.java:262) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) +| | | | | | | | | | |
| | | | | | | | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) +| | | | | | | | | | |
| | | | | | | | at java.lang.Thread.run(Thread.java:744) +| | | | | | | | | | |
| | | | | | | | at org.jboss.threads.JBossThread.run(JBossThread.java:122) +| | | | | | | | | | |
| | | | | | | | | | | | | | | | | | |
(1 row)
jbatch=#
Worked expectedly in GlassFish4.0
Log
2014-02-14T22:02:27.017+0900|情報: readItem: 1 2014-02-14T22:02:27.017+0900|情報: process: i=1, item=1 2014-02-14T22:02:27.017+0900|情報: readItem: 2 2014-02-14T22:02:27.017+0900|情報: readItem: 3 2014-02-14T22:02:27.017+0900|情報: process: i=3, item=3 2014-02-14T22:02:27.017+0900|情報: readItem: null 2014-02-14T22:02:27.017+0900|情報: write: [1, 3]
Repository
kyle-no-MacBook:bin kyle$ ./asadmin list-batch-jobs
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
JOBNAME INSTANCECOUNT
skip 1
Command list-batch-jobs executed successfully.
kyle-no-MacBook:bin kyle$ ./asadmin list-batch-job-steps -l 1
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
STEPNAME STEPID STARTTIME ENDTIME BATCHSTATUS EXITSTATUS STEPMETRICS
doChunk 1 Fri Feb 14 22:02:26 JST 2014 Fri Feb 14 22:02:27 JST 2014 COMPLETED COMPLETED METRICNAME VALUE
READ_COUNT 3
WRITE_COUNT 2
COMMIT_COUNT 1
ROLLBACK_COUNT 0
READ_SKIP_COUNT 0
PROCESS_SKIP_COUNT 1
FILTER_COUNT 0
WRITE_SKIP_COUNT 0
Command list-batch-job-steps executed successfully.
kyle-no-MacBook:bin kyle$
References
Tags: jbatch