Kohei Nozaki's blog 

A way to resolve dependencies of jobs

Posted on Sunday Feb 16, 2014 at 08:17AM in Technology

This is about a way to resolve dependencies of jobs that used only JSR 352 API.

Environment

  • jBeret 1.0.1Beta-SNAPSHOT
  • WildFly 8.0.0.Final

Why need it?

  • For tasks that need to execute sequentially, usually we have to let all tasks in single job XML.
  • I guess it would bring huge difficulty of testing.
    • For example, I don't want to test such big job like EndOfTheDay.xml
  • Rather than I would go with smaller jobs with external job control system.

How realize it?

  • This needs to create parent job XML that defines dependencies of child jobs.
  • Every step uses artifact named “jobLauncherBatchlet”
    • This batchlet invokes child job. name of the job and job parameters are given by its parameter of the step.
    • After invoke the job, then it starts polling BatchStatus, and wait till the job is finished.

Sample project

Whole resources are available in GitHub.

log

13:45:22,589 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: persistentUserData=null, waitForFinish=null
13:45:22,589 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: starting new job: jobXMLName=jobdependency-child1, properties={sleepInMillis=1000}
13:45:22,593 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job started: executionId=207
13:45:22,593 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: waiting for finish the job... executionId=207
13:45:22,595 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=child1-step1, sleepInMills=1,000
13:45:23,596 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2)  exiting process(): stepName=child1-step1, sleepInMills=1,000
13:45:24,595 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job finished: BatchStatus=COMPLETED
13:45:24,601 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: persistentUserData=null, waitForFinish=null
13:45:24,601 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: starting new job: jobXMLName=jobdependency-child2, properties={sleepInMillis=2000}
13:45:24,605 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job started: executionId=208
13:45:24,605 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: waiting for finish the job... executionId=208
13:45:24,606 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=child2-step1, sleepInMills=2,000
13:45:26,608 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4)  exiting process(): stepName=child2-step1, sleepInMills=2,000
13:45:27,608 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job finished: BatchStatus=COMPLETED
13:45:27,614 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: persistentUserData=null, waitForFinish=null
13:45:27,614 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: starting new job: jobXMLName=jobdependency-child3, properties={sleepInMillis=3000}
13:45:27,618 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job started: executionId=209
13:45:27,618 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: waiting for finish the job... executionId=209
13:45:27,619 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) entering process(): stepName=child3-step1, sleepInMills=3,000
13:45:30,620 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9)  exiting process(): stepName=child3-step1, sleepInMills=3,000
13:45:31,620 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job finished: BatchStatus=COMPLETED

Job repository tables

job_instance and job_execution

jbatch=# select * from job_instance i, job_execution e where i.jobinstanceid = e.jobinstanceid and i.jobname like 'jobdependency%' and i.jobinstanceid >= 194 order by e.jobexecutionid;
 jobinstanceid | version |       jobname        | applicationname | jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus |    jobparameters     | restartposition 
---------------+---------+----------------------+-----------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
           194 |         | jobdependency        | jbatchtest      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      | 
           195 |         | jobdependency-child1 | jbatchtest      |            207 |           195 |         | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED   | COMPLETED  | sleepInMillis = 1000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
           196 |         | jobdependency-child2 | jbatchtest      |            208 |           196 |         | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED   | COMPLETED  | sleepInMillis = 2000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
           197 |         | jobdependency-child3 | jbatchtest      |            209 |           197 |         | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED   | COMPLETED  | sleepInMillis = 3000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
(4 rows)

step_execution

jbatch=# select * from step_execution s, job_execution e, job_instance i where s.jobexecutionid = e.jobexecutionid and e.jobinstanceid = i.jobinstanceid and i.jobinstanceid >= 194 order by s.starttime;
 stepexecutionid | jobexecutionid | version |   stepname   |        starttime        |         endtime         | batchstatus | exitstatus | executionexception |                                                                           persistentuserdata                                                                           | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo | jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus |    jobparameters     | restartposition | jobinstanceid | version |       jobname        | applicationname 
-----------------+----------------+---------+--------------+-------------------------+-------------------------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------+---------------+---------+----------------------+-----------------
             277 |            206 |         | step1        | 2014-02-19 13:45:22.587 | 2014-02-19 13:45:24.595 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000cf |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             278 |            207 |         | child1-step1 | 2014-02-19 13:45:22.593 | 2014-02-19 13:45:23.596 | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            207 |           195 |         | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED   | COMPLETED  | sleepInMillis = 1000+|                 |           195 |         | jobdependency-child1 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
             279 |            206 |         | step2        | 2014-02-19 13:45:24.599 | 2014-02-19 13:45:27.608 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d0 |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             280 |            208 |         | child2-step1 | 2014-02-19 13:45:24.605 | 2014-02-19 13:45:26.608 | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            208 |           196 |         | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED   | COMPLETED  | sleepInMillis = 2000+|                 |           196 |         | jobdependency-child2 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
             281 |            206 |         | step3        | 2014-02-19 13:45:27.612 | 2014-02-19 13:45:31.621 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d1 |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             282 |            209 |         | child3-step1 | 2014-02-19 13:45:27.618 | 2014-02-19 13:45:30.62  | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            209 |           197 |         | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED   | COMPLETED  | sleepInMillis = 3000+|                 |           197 |         | jobdependency-child3 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
(6 rows)
  • We can see that they were executed in particular order.

Remarks

  • I intent to care of stop and restart, and sample project contains some codes about it, but I haven't tested it yet. maybe I would write about it too later.
  • It can use some functions of jobXML such as parallel executions (like split element) for control of job execution even.

Take care of the max number of threads in thread-pool

  • Some implementations have limit of threads count.
  • When you used this method or similar one with many parallel execution elements (split or partition), you must ensure to keep number of threads lower than the limit.
  • If limit reaches, your batch will be frozen and it won't resume.

How to configure max threads

  • On WildFly + jBeret, the max number of threads is set to 10 as default.
  • This increases the limit to 100:
/subsystem=batch/thread-pool=batch:write-attribute(name=max-threads, value=100)

Comments [0]

Tags: jbatch

StepContext#persistentUserDataで遊ぶ

Posted on Sunday Feb 16, 2014 at 07:43AM in Technology

StepContext#persistentUserDataにはStepに紐づくデータを保存しておける。ここに入れたデータはJobRepositoryに保存され、後続のStepやジョブの再実行時に参照できる。何かしら使い道があると思われるので若干遊んでみる

環境・前提条件

関連メソッドの仕様を見てみる

[1]から引いてみる

javax.batch.runtime.context.StepContextインタフェース

Stepに関連する操作をする時に使うインタフェース。artifactで@Injectで注入して使える。

    /**
     * The getPersistentUserData method returns a persistent data object 
     * belonging to the current step. The user data type must implement 
     * java.util.Serializable. This data is saved as part of a step's 
     * checkpoint. For a step that does not do checkpoints, it is saved 
     * after the step ends. It is available upon restart. 
     * @return user-specified type
     */
    public Serializable getPersistentUserData();

    /**
     * The setPersistentUserData method stores a persistent data object 
     * into the current step. The user data type must implement 
     * java.util.Serializable. This data is saved as part of a step's 
     * checkpoint. For a step that does not do checkpoints, it is saved 
     * after the step ends. It is available upon restart. 
     * @param data is the user-specified type
     */ 
    public void setPersistentUserData(Serializable data);

javax.batch.runtime.StepExecutionインタフェース

Stepの実行履歴的な情報を含むインタフェース。JobOperatorを使って引っ張って来れたり、Deciderのメソッドの引数で受け取れたりする。

    /**
     * Get user persistent data
     * @return persistent data 
     */
    public Serializable getPersistentUserData();

先行StepでセットしたpersistentUserDataを後続Stepで参照してみる

サンプルの仕様

  • 第一レベルの要素は動作順に以下
    1. set (setPersistentUserDataBatchlet)
    2. get (getPersistentUserDataBatchlet)
  • setでセットしたpersistentUserDataをgetで参照してみる

資源

資源はこのへんにまとめて全部ある

動かしてみる

ログ

08:52:02,642 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.SetPersistentUserDataBatchlet] (batch-batch - 4) set: process()
08:52:02,649 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) get: process()
08:52:02,650 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) stepExecution: stepName=set, persistentUserData=* my step name is set *
08:52:02,650 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetPersistentUserDataBatchlet] (batch-batch - 4) stepExecution: stepName=get, persistentUserData=null

setステップで設定した「 my step name is set 」がgetステップから見えている（3行目）。うむ

Repository

job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus | jobparameters | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
            131 |           126 |         | 2014-02-16 08:52:02.637 | 2014-02-16 08:52:02.637 | 2014-02-16 08:52:02.653 | 2014-02-16 08:52:02.653 | COMPLETED   | COMPLETED  |               | 
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid = 131 order by stepexecutionid;
 stepexecutionid | jobexecutionid | version | stepname |        starttime        |         endtime         | batchstatus | exitstatus | executionexception |                       persistentuserdata                       | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo 
-----------------+----------------+---------+----------+-------------------------+-------------------------+-------------+------------+--------------------+----------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
             153 |            131 |         | set      | 2014-02-16 08:52:02.639 | 2014-02-16 08:52:02.642 | COMPLETED   | COMPLETED  |                    | \xaced00057400172a206d792073746570206e616d6520697320736574202a |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             154 |            131 |         | get      | 2014-02-16 08:52:02.645 | 2014-02-16 08:52:02.65  | COMPLETED   | COMPLETED  |                    |                                                                |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
(2 rows)

persistentuserdataカラムに何かが入っている。うむ

前回実行時のデータを再実行時に参照してみる

サンプルの仕様

  • Stepは1つだけ
  • 何もpersistentUserDataに入ってなかったら
    1. persistentUserDataにデータを入れる
    2. 例外投げて死ぬ
  • 何かpersistentUserDataに入っていたら
    1. 内容を表示して終了
  • 初回実行はpersistentUserDataにデータ入れてFAILEDになる
  • 再実行時はpersistentUserDataのデータを表示してCOMPLETEDになる

資源

資源はこのへんにまとめて全部ある

動かしてみる

ログ

09:15:07,338 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 3) getAndSet: process()
09:15:07,339 WARN  [org.jberet] (batch-batch - 3) JBERET000001: Failed to run batchlet org.jberet.job.model.RefArtifact@7e76d151: java.lang.RuntimeException: to confirm whether the data is visible or not when restart
    at org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet.process(GetAndSetPersistentUserDataBatchlet.java:31) [classes:]
    at org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:61) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:207) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58) [jberet-core-1.0.1.Beta-SNAPSHOT.jar:1.0.1.Beta-SNAPSHOT]
    at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149) [wildfly-jberet-8.0.0.Final.jar:8.0.0.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

09:15:08,351 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 2) getAndSet: process()
09:15:08,351 FINE  [org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet] (batch-batch - 2) persistentUserData=* my job execution id is 134 *

初回実行時に入れた「 my job execution id is 134 」が再実行時に見えている。うむ

Repository

job_execution
jbatch=# select * from job_execution where jobexecutionid in (134, 135) order by jobexecutionid;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus | jobparameters | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
            134 |           128 |         | 2014-02-16 09:15:07.335 | 2014-02-16 09:15:07.335 | 2014-02-16 09:15:07.343 | 2014-02-16 09:15:07.343 | FAILED      | FAILED     |               | 
            135 |           128 |         | 2014-02-16 09:15:08.347 | 2014-02-16 09:15:08.347 | 2014-02-16 09:15:08.354 | 2014-02-16 09:15:08.354 | COMPLETED   | COMPLETED  |               | 
(2 rows)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (134, 135) order by jobexecutionid, stepexecutionid;
 stepexecutionid | jobexecutionid | version | stepname  |        starttime        |         endtime         | batchstatus | exitstatus |                                                                   executionexception                                                                   |                              persistentuserdata                              | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo 
-----------------+----------------+---------+-----------+-------------------------+-------------------------+-------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
             157 |            134 |         | getAndSet | 2014-02-16 09:15:07.336 | 2014-02-16 09:15:07.34  | FAILED      | FAILED     | java.lang.RuntimeException: to confirm whether the data is visible or not when restart                                                                +| \xaced000574001e2a206d79206a6f6220657865637574696f6e20696420697320313334202a |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
                 |                |         |           |                         |                         |             |            |         at org.nailedtothex.jbatch.example.persistentuserdata.GetAndSetPersistentUserDataBatchlet.process(GetAndSetPersistentUserDataBatchlet.java:31)+|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.BatchletRunner.run(BatchletRunner.java:61)                                                                       +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:207)                                             +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:131)                                                            +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:162)                                              +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)                             +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:58)                                                               +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.wildfly.jberet.services.BatchEnvironmentService$WildFlyBatchEnvironment$1.run(BatchEnvironmentService.java:149)                        +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)                                                                    +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at java.util.concurrent.FutureTask.run(FutureTask.java:262)                                                                                   +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)                                                            +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)                                                            +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at java.lang.Thread.run(Thread.java:744)                                                                                                      +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |         at org.jboss.threads.JBossThread.run(JBossThread.java:122)                                                                                    +|                                                                              |           |            |             |               |               |                  |             |                |                      | 
                 |                |         |           |                         |                         |             |            |                                                                                                                                                        |                                                                              |           |            |             |               |               |                  |             |                |                      | 
             158 |            135 |         | getAndSet | 2014-02-16 09:15:08.349 | 2014-02-16 09:15:08.352 | COMPLETED   | COMPLETED  |                                                                                                                                                        | \xaced000574001e2a206d79206a6f6220657865637574696f6e20696420697320313334202a |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
(2 rows)

備考

  • 再実行時にはpersistentUserDataがコピーされるようなので、データ量がでかい場合は気をつけた方がいいのかも

参考文献

  1. JSR-000352 Batch Applications for the Java Platform - Final Release

Comments [0]

Tags: jbatch

How to define a JavaMail session using CLI

Posted on Sunday Feb 16, 2014 at 07:30AM in Technology

  • Defining a JavaMail session for gmail.

Environment

  • WildFly8.0.0.Final
  • Oracle JDK7u51

Defining with jboss-cli

Define a outbound-socket-binding named “MyMailSMTP”

  • SMTP server hostname
  • SMTP server port
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=MyMailSMTP:add( \
host=smtp.gmail.com, \
port=465)

Define a JavaMail session named “MyMail”

  • JNDI name of JavaMail session
  • Email address of the sender
  • Debug flag
/subsystem=mail/mail-session=MyMail:add(jndi-name="java:/mail/myMail", from="jane.doe@gmail.example.com", debug=true)

Add a reference from “MyMail” to “MyMailSMTP”

  • SMTP server username
  • SMTP server password
  • TLS flag
  • SSL flag
/subsystem=mail/mail-session=MyMail/server=smtp:add( \
outbound-socket-binding-ref=MyMailSMTP, \
ssl=true, \
username=jane.doe@gmail.example.com, \
password=***)

Example Servlet sends a mail

@WebServlet("/")
public class MailServlet extends HttpServlet {
    @Resource(lookup = "java:/mail/myMail")
    Session session;

    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        MimeMessage message = new MimeMessage(session);
        try {

            InternetAddress[] address = { new InternetAddress("jane.doe@icloud.example.com") };
            message.setRecipients(Message.RecipientType.TO, address);
            message.setSubject("test");
            message.setSentDate(new Date());
            message.setText("testbody");
            Transport.send(message);
        } catch (MessagingException ex) {
            throw new ServletException(ex);
        }
    }
}

Mail which sent from WildFly

Debug Log

15:37:26,305 INFO  [stdout] (default task-1) DEBUG: JavaMail version 1.5.1
15:37:26,306 INFO  [stdout] (default task-1) DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
15:37:26,307 INFO  [stdout] (default task-1) DEBUG: Tables of loaded providers
15:37:26,307 INFO  [stdout] (default task-1) DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle]}
15:37:26,307 INFO  [stdout] (default task-1) DEBUG: Providers Listed By Protocol: {imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Oracle], imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Oracle], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Oracle], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Oracle], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle]}
15:37:26,307 INFO  [stdout] (default task-1) DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
15:37:26,309 INFO  [stdout] (default task-1) DEBUG: getProvider() returning javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Oracle]
15:37:26,309 INFO  [stdout] (default task-1) DEBUG SMTP: need username and password for authentication
15:37:26,309 INFO  [stdout] (default task-1) DEBUG SMTP: useEhlo true, useAuth true
15:37:26,309 INFO  [stdout] (default task-1) DEBUG SMTP: trying to connect to host "smtp.gmail.com", port 465, isSSL true
15:37:26,889 INFO  [stdout] (default task-1) 220 mx.google.com ESMTP *** - gsmtp
15:37:26,889 INFO  [stdout] (default task-1) DEBUG SMTP: connected to host "smtp.gmail.com", port: 465
15:37:26,889 INFO  [stdout] (default task-1) 
15:37:26,893 INFO  [stdout] (default task-1) EHLO ***
15:37:27,006 INFO  [stdout] (default task-1) 250-mx.google.com at your service, [27.114.70.39]
15:37:27,006 INFO  [stdout] (default task-1) 250-SIZE 35882577
15:37:27,006 INFO  [stdout] (default task-1) 250-8BITMIME
15:37:27,006 INFO  [stdout] (default task-1) 250-AUTH LOGIN PLAIN XOAUTH XOAUTH2 PLAIN-CLIENTTOKEN
15:37:27,007 INFO  [stdout] (default task-1) 250-ENHANCEDSTATUSCODES
15:37:27,007 INFO  [stdout] (default task-1) 250 CHUNKING
15:37:27,007 INFO  [stdout] (default task-1) DEBUG SMTP: Found extension "SIZE", arg "35882577"
15:37:27,007 INFO  [stdout] (default task-1) DEBUG SMTP: Found extension "8BITMIME", arg ""
15:37:27,007 INFO  [stdout] (default task-1) DEBUG SMTP: Found extension "AUTH", arg "LOGIN PLAIN XOAUTH XOAUTH2 PLAIN-CLIENTTOKEN"
15:37:27,008 INFO  [stdout] (default task-1) DEBUG SMTP: Found extension "ENHANCEDSTATUSCODES", arg ""
15:37:27,008 INFO  [stdout] (default task-1) DEBUG SMTP: Found extension "CHUNKING", arg ""
15:37:27,008 INFO  [stdout] (default task-1) DEBUG SMTP: Attempt to authenticate using mechanisms: LOGIN PLAIN DIGEST-MD5 NTLM 
15:37:27,008 INFO  [stdout] (default task-1) DEBUG SMTP: AUTH LOGIN command trace suppressed
15:37:27,959 INFO  [stdout] (default task-1) DEBUG SMTP: AUTH LOGIN succeeded
15:37:27,960 INFO  [stdout] (default task-1) DEBUG SMTP: use8bit false
15:37:27,960 INFO  [stdout] (default task-1) MAIL FROM:<jane.doe@gmail.example.com>
15:37:28,074 INFO  [stdout] (default task-1) 250 2.1.0 OK *** - gsmtp
15:37:28,074 INFO  [stdout] (default task-1) RCPT TO:<jane.doe@icloud.example.com>
15:37:28,194 INFO  [stdout] (default task-1) 250 2.1.5 OK *** - gsmtp
15:37:28,195 INFO  [stdout] (default task-1) DEBUG SMTP: Verified Addresses
15:37:28,195 INFO  [stdout] (default task-1) DEBUG SMTP:   jane.doe@icloud.example.com
15:37:28,195 INFO  [stdout] (default task-1) DATA
15:37:28,881 INFO  [stdout] (default task-1) 354  Go ahead *** - gsmtp
15:37:28,881 INFO  [stdout] (default task-1) Date: Thu, 6 Mar 2014 15:37:26 +0900 (JST)
15:37:28,881 INFO  [stdout] (default task-1) To: jane.doe@icloud.example.com
15:37:28,881 INFO  [stdout] (default task-1) Message-ID: <56980149.1.1394087846308.JavaMail.jane.doe@gmail.example.com>
15:37:28,882 INFO  [stdout] (default task-1) Subject: test
15:37:28,882 INFO  [stdout] (default task-1) MIME-Version: 1.0
15:37:28,882 INFO  [stdout] (default task-1) Content-Type: text/plain; charset=us-ascii
15:37:28,882 INFO  [stdout] (default task-1) Content-Transfer-Encoding: 7bit
15:37:28,882 INFO  [stdout] (default task-1) 
15:37:28,882 INFO  [stdout] (default task-1) testbody
15:37:28,883 INFO  [stdout] (default task-1) .
15:37:29,802 INFO  [stdout] (default task-1) 250 2.0.0 OK 1394087849 *** - gsmtp
15:37:29,803 INFO  [stdout] (default task-1) QUIT
15:37:29,916 INFO  [stdout] (default task-1) 221 2.0.0 closing connection *** - gsmtp
  • Debug logging can be turned off like this:
/subsystem=mail/mail-session=MyMail:write-attribute(name=debug, value=false)
/:reload

References

  1. How to configure a Mail Session in Wildfly? | Community
  2. JBoss AS 7.2.0 Email Session configurtion - Eng… | Community
  3. jboss7.x - Please help configure JBoss 7 “mail-session” via jboss-cli - Stack Overflow
  4. Configuring and Using mail service in JBoss AS7 « JBoss
  5. JBoss 7 JavaMail | Community

Comments [1]

Tags: wildfly

Decisionでフロー制御してみる

Posted on Saturday Feb 15, 2014 at 08:20PM in Technology

フロー制御は、ExitStatusでフロー制御してみるでやったようにBatchletだけでもできるけど、Chunk要素などの後続処理のフロー制御はDecision要素を使うと楽にできそうなので、試してみる

例えば先行のChunk処理でスキップが多すぎる時に、一応ジョブは正常終了しつつも警告メール的なものを送っておくとか、そういうことをやりたい時にはDecisionを使うとよいかもしれない。引数でStepExecution[]が渡ってくるので、先行処理のMetricsを取り出して調べたりすることが楽にできる。

環境・前提条件

仕様を確認する

  • [2]から引いてみる

public interface Decider

A Decider receives control as part of a decision element in a job. It is used to direct execution flow during job processing. It returns an exit status that updates the current job execution's exit status. This exit status value also directs the execution transition based on next, end, stop, fail child elements configured on the same decision element as the decider.

String decide(StepExecution[] executions) throws Exception

The decide method sets a new exit status for a job. It receives an array of StepExecution objects as input. These StepExecution objects represent the execution element that transitions to this decider as follows:

Step

When the transition is from a step, the decide method receives the StepExecution corresponding to the step as input.

Split

When the transition is from a split, the decide method receives a StepExecution from each flow defined to the split as input.

Flow

When the transition is from a flow, the decide method receives a StepExecution corresponding to the last execution element that completed in the flow. This will be a single StepExecution if the last element was a step and multiple StepExecutions if the last element was a split.

Parameters:

executions - specifies the StepExecution(s) of the preceding element.

Returns:

updated job exit status

  • とりあえず上に書かれた3パターンで試してみる

Stepからの遷移の場合

資源はこのへんにまとめて全部ある

資源

仕様

  • 第一レベルの要素は以下
    1. step1 (batchlet step)
    2. step2 (chunk oriented step)
    3. step2decision (decision)
    4. step3_NO_SKIP_FOUND (batchlet step)
    5. step4_SKIP_FOUND (batchlet step)
  • step2でスキップがなかった場合はstep3_NO_SKIP_FOUNDに遷移する
  • step2でスキップがあった場合はstep4_SKIP_FOUNDに遷移する

スキップが無い場合のテストを動かしてみる

ログ

13:53:59,614 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): executions=1
13:53:59,614 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=step2, stepExecutionId=248
13:53:59,614 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=10
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=4
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=10
13:53:59,615 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): exitStatus=SKIP_NOT_FOUND
13:53:59,617 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 10) step3_NO_SKIP_FOUND

Repository

job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime       |       starttime        |         endtime         |     lastupdatedtime     | batchstatus |   exitstatus   |     jobparameters     | restartposition 
----------------+---------------+---------+------------------------+------------------------+-------------------------+-------------------------+-------------+----------------+-----------------------+-----------------
            188 |           176 |         | 2014-02-16 13:53:59.41 | 2014-02-16 13:53:59.41 | 2014-02-16 13:53:59.619 | 2014-02-16 13:53:59.619 | COMPLETED   | SKIP_NOT_FOUND | itemReaderFailAt = -1+| 
                |               |         |                        |                        |                         |                         |             |                | divide = 2           +| 
                |               |         |                        |                        |                         |                         |             |                |                       | 
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (188) order by jobexecutionid, stepexecutionid;
 stepexecutionid | jobexecutionid | version |      stepname       |        starttime        |         endtime         | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount |                                                                         readercheckpointinfo                                                                         |                                                                         writercheckpointinfo                                                                         
-----------------+----------------+---------+---------------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
             247 |            188 |         | step1               | 2014-02-16 13:53:59.412 | 2014-02-16 13:53:59.453 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                                                                                                                                                                      | 
             248 |            188 |         | step2               | 2014-02-16 13:53:59.46  | 2014-02-16 13:53:59.61  | COMPLETED   | COMPLETED  |                    |                    |        10 |         10 |           4 |             0 |             0 |                0 |           0 |              0 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b02000078700000000a | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000004
             249 |            188 |         | step3_NO_SKIP_FOUND | 2014-02-16 13:53:59.615 | 2014-02-16 13:53:59.617 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                                                                                                                                                                      | 
(3 rows)

jbatch=#

スキップが有る場合のテストを動かしてみる

ログ

13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): executions=1
13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): stepName=step2, stepExecutionId=251
13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: WRITE_SKIP_COUNT=0
13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: PROCESS_SKIP_COUNT=0
13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: WRITE_COUNT=9
13:55:00,761 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: COMMIT_COUNT=4
13:55:00,762 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: READ_SKIP_COUNT=1
13:55:00,762 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: ROLLBACK_COUNT=0
13:55:00,762 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: FILTER_COUNT=0
13:55:00,762 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): metric: READ_COUNT=9
13:55:00,762 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 5) decide(): exitStatus=SKIP_FOUND
13:55:00,763 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 5) step4_SKIP_FOUND

Repository

job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus |    jobparameters     | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
            189 |           177 |         | 2014-02-16 13:55:00.639 | 2014-02-16 13:55:00.639 | 2014-02-16 13:55:00.765 | 2014-02-16 13:55:00.765 | COMPLETED   | SKIP_FOUND | itemReaderFailAt = 5+| 
                |               |         |                         |                         |                         |                         |             |            | divide = 2          +| 
                |               |         |                         |                         |                         |                         |             |            |                      | 
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (189) order by jobexecutionid, stepexecutionid;
 stepexecutionid | jobexecutionid | version |     stepname     |        starttime        |         endtime         | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount |                                                                         readercheckpointinfo                                                                         |                                                                         writercheckpointinfo                                                                         
-----------------+----------------+---------+------------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
             250 |            189 |         | step1            | 2014-02-16 13:55:00.641 | 2014-02-16 13:55:00.642 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                                                                                                                                                                      | 
             251 |            189 |         | step2            | 2014-02-16 13:55:00.644 | 2014-02-16 13:55:00.758 | COMPLETED   | COMPLETED  |                    |                    |         9 |          9 |           4 |             0 |             1 |                0 |           0 |              0 | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b02000078700000000a | \xaced0005737200116a6176612e6c616e672e496e746567657212e2a0a4f781873802000149000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000003
             252 |            189 |         | step4_SKIP_FOUND | 2014-02-16 13:55:00.762 | 2014-02-16 13:55:00.763 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                                                                                                                                                                      | 
(3 rows)

jbatch=#
  • Decider#decide()に渡されるStepExecutionは、基本的に遷移元Stepのものだけのようだ

Splitからの遷移の場合

資源

仕様

  • 第一レベルの要素は以下
    1. split1 (split)
    2. split1decision (decision)
  • split1の下には同じようなflowが3つある
    • flowの下にはstep (batchlet) が2つ
  • decisionの後の分岐とスキップはもう試したので省略

テストを動かしてみる

ログ

13:42:04,217 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 3) flow3-step1
13:42:04,217 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step1
13:42:04,217 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 2) flow2-step1
13:42:04,221 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=flow2-step2, sleepInMills=2,000
13:42:04,221 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 3) entering process(): stepName=flow3-step2, sleepInMills=3,000
13:42:04,221 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 6) entering process(): stepName=flow1-step2, sleepInMills=1,000
13:42:05,222 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 6)  exiting process(): stepName=flow1-step2, sleepInMills=1,000
13:42:06,222 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2)  exiting process(): stepName=flow2-step2, sleepInMills=2,000
13:42:07,222 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 3)  exiting process(): stepName=flow3-step2, sleepInMills=3,000
13:42:07,225 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): executions=3
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow1-step2, stepExecutionId=243
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0
13:42:07,226 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow2-step2, stepExecutionId=242
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0
13:42:07,227 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): stepName=flow3-step2, stepExecutionId=244
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_SKIP_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: PROCESS_SKIP_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: WRITE_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: COMMIT_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_SKIP_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: ROLLBACK_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: FILTER_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): metric: READ_COUNT=0
13:42:07,228 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 10) decide(): exitStatus=SKIP_NOT_FOUND
  • 遷移元がSplitの場合、Splitの子Flowの数だけStepExecutionが渡される（Flowの最後の要素のStepExecution）。だから配列なのか。なるほどね

Repository

job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus |   exitstatus   | jobparameters | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+----------------+---------------+-----------------
            186 |           174 |         | 2014-02-16 13:42:04.198 | 2014-02-16 13:42:04.198 | 2014-02-16 13:42:07.229 | 2014-02-16 13:42:07.229 | COMPLETED   | SKIP_NOT_FOUND |               | 
(1 row)

jbatch=#
step_execution
jbatch=# select * from step_execution where jobexecutionid in (186) order by jobexecutionid, stepexecutionid;
 stepexecutionid | jobexecutionid | version |  stepname   |        starttime        |         endtime         | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo 
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
             239 |            186 |         | flow1-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             240 |            186 |         | flow3-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             241 |            186 |         | flow2-step1 | 2014-02-16 13:42:04.211 | 2014-02-16 13:42:04.217 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             242 |            186 |         | flow2-step2 | 2014-02-16 13:42:04.22  | 2014-02-16 13:42:06.223 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             243 |            186 |         | flow1-step2 | 2014-02-16 13:42:04.22  | 2014-02-16 13:42:05.223 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             244 |            186 |         | flow3-step2 | 2014-02-16 13:42:04.22  | 2014-02-16 13:42:07.222 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
(6 rows)

Flowからの遷移の場合

資源

仕様

  • 第一レベルの要素は以下
    1. flow1 (flow)
    2. flowdecision (decision)
  • flow1の下にはstep (batchlet) が2つ

テストを動かしてみる

ログ

13:47:01,234 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step1
13:47:01,237 FINE  [org.nailedtothex.jbatch.example.decision.NopBatchlet] (batch-batch - 6) flow1-step2
13:47:01,239 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): executions=1
13:47:01,239 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): stepName=flow1-step2, stepExecutionId=246
13:47:01,239 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: WRITE_SKIP_COUNT=0
13:47:01,239 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: PROCESS_SKIP_COUNT=0
13:47:01,239 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: WRITE_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: COMMIT_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: READ_SKIP_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: ROLLBACK_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: FILTER_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): metric: READ_COUNT=0
13:47:01,240 FINE  [org.nailedtothex.jbatch.example.decision.MyDecider] (batch-batch - 6) decide(): exitStatus=SKIP_NOT_FOUND

Repository

job_execution
jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |        endtime         |    lastupdatedtime     | batchstatus |   exitstatus   | jobparameters | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+------------------------+------------------------+-------------+----------------+---------------+-----------------
            187 |           175 |         | 2014-02-16 13:47:01.231 | 2014-02-16 13:47:01.231 | 2014-02-16 13:47:01.24 | 2014-02-16 13:47:01.24 | COMPLETED   | SKIP_NOT_FOUND |               | 
(1 row)
step_execution
jbatch=# select * from step_execution where jobexecutionid in (187) order by jobexecutionid, stepexecutionid;
 stepexecutionid | jobexecutionid | version |  stepname   |        starttime        |         endtime         | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo 
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
             245 |            187 |         | flow1-step1 | 2014-02-16 13:47:01.232 | 2014-02-16 13:47:01.234 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             246 |            187 |         | flow1-step2 | 2014-02-16 13:47:01.236 | 2014-02-16 13:47:01.237 | COMPLETED   | COMPLETED  |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
(2 rows)

jbatch=#
  • 遷移元がFlowの場合はFlowの一番最後のStepのStepExecutionだけが渡される

備考

  • Decider#decide()に渡されるStepExecutionは以下のようになる
    • 遷移元がStepの場合: 遷移元StepのStepExecution (配列の要素数は1個)
    • 遷移元がSplitの場合: 遷移元Splitの各子Flowの最後のStepのStepExecution (配列の要素数は複数個)
    • 遷移元がFlowの場合: 遷移元Flowの最後のStepのStepExecution (配列の要素数は1個)

参考文献

  1. JSR-000352 Batch Applications for the Java Platform - Final Release
  2. Decider (Java™ EE 7 Specification APIs)

Comments [0]

Tags: jbatch

Splitで並列処理してみる

Posted on Saturday Feb 15, 2014 at 08:19PM in Technology

「Partitioned Chunk Processingで遊ぶ」で使ったPartitionは一つの処理を手分けしてやる仕組みだが、splitは異なる処理を並列で行う仕組み。簡単なバッチを作って遊んでみる。

環境・前提条件

バッチを作ってみる

仕様

  • 一番高レベルの要素は2つ。順番に動く
    1. split0 (Split)
    2. nextOfSplit (Step)
  • split0にはflowが3つある
    • 全てのflowは1つのstepを持つ
    • 使っているartifactは全て同じ。単純なBatchletで、パラメータで与えられたミリ秒数だけThread.sleep()する
    • それぞれ1, 2, 3秒スリープする
  • nextOfSplitは特に何もしない。ちゃんとsplit0内のflowが全部終わってから遷移するか確かめるために置いた

資源

資源はこのへんにまとめて全部ある

動かしてみる

ログ

21:38:24,423 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5) entering process(): stepName=1sec, sleepInMills=1,000
21:38:24,423 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8) entering process(): stepName=3sec, sleepInMills=3,000
21:38:24,422 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7) entering process(): stepName=2sec, sleepInMills=2,000
21:38:25,424 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 5)  exiting process(): stepName=1sec, sleepInMills=1,000
21:38:26,423 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 7)  exiting process(): stepName=2sec, sleepInMills=2,000
21:38:27,424 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 8)  exiting process(): stepName=3sec, sleepInMills=3,000
21:38:27,430 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=nextOfSplit, sleepInMills=null
21:38:27,430 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4)  exiting process(): stepName=nextOfSplit, sleepInMills=null

Repository

job_execution

jbatch=# select * from job_execution order by jobexecutionid desc limit 1;
 jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus | jobparameters | restartposition 
----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+---------------+-----------------
            129 |           124 |         | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:24.415 | 2014-02-15 21:38:27.433 | 2014-02-15 21:38:27.433 | COMPLETED   | COMPLETED  |               | 
(1 row)

step_execution

jbatch=# select * from step_execution where jobexecutionid =129 order by stepexecutionid;
 stepexecutionid | jobexecutionid | version |  stepname   |        starttime        |         endtime         | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo 
-----------------+----------------+---------+-------------+-------------------------+-------------------------+-------------+------------+--------------------+--------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------
             147 |            129 |         | 1sec        | 2014-02-15 21:38:24.42  | 2014-02-15 21:38:25.424 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             148 |            129 |         | 2sec        | 2014-02-15 21:38:24.42  | 2014-02-15 21:38:26.424 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             149 |            129 |         | 3sec        | 2014-02-15 21:38:24.42  | 2014-02-15 21:38:27.424 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
             150 |            129 |         | nextOfSplit | 2014-02-15 21:38:27.428 | 2014-02-15 21:38:27.431 | COMPLETED   | SUCCESS    |                    |                    |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      | 
(4 rows)

備考

  • ちゃんとパラで走っているみたい
  • [1]によると、いずれかのFlowが死ぬと、親のSplitも死ぬ
  • Flowは複数のStepを1つにまとめる場合に使う。うまく使うとStep間の遷移をすっきり表現できる気がする。Splitの子要素として以外でも使えるようだ

参考文献

  1. JSR-000352 Batch Applications for the Java Platform - Final Release

Comments [0]

Tags: jbatch

About Me

Java backend engineer obsessed with test driven development. Apache Roller committer. kohei at apache dot org

Archives

« February 2014 »
SunMonTueWedThuFriSat
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
       
Today

Hot tags

Here are the most frequently used tags in the blog. Hover over a tag to see the number of posts, click a tag to see the most recent posts with the tag.

Recent entries

Recent comments

Twitter