Kohei Nozaki's blog 

A way to resolve dependencies of jobs


Posted on Sunday Feb 16, 2014 at 08:17AM in Technology


This is about a way to resolve dependencies of jobs that used only JSR 352 API.

Environment

  • jBeret 1.0.1Beta-SNAPSHOT
  • WildFly 8.0.0.Final

Why need it?

  • For tasks that need to execute sequentially, usually we have to let all tasks in single job XML.
  • I guess it would bring huge difficulty of testing.
    • For example, I don't want to test such big job like EndOfTheDay.xml
  • Rather than I would go with smaller jobs with external job control system.

How realize it?

  • This needs to create parent job XML that defines dependencies of child jobs.
  • Every step uses artifact named “jobLauncherBatchlet”
    • This batchlet invokes child job. name of the job and job parameters are given by its parameter of the step.
    • After invoke the job, then it starts polling BatchStatus, and wait till the job is finished.

Sample project

Whole resources are available in GitHub.

log

13:45:22,589 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: persistentUserData=null, waitForFinish=null
13:45:22,589 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: starting new job: jobXMLName=jobdependency-child1, properties={sleepInMillis=1000}
13:45:22,593 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job started: executionId=207
13:45:22,593 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: waiting for finish the job... executionId=207
13:45:22,595 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=child1-step1, sleepInMills=1,000
13:45:23,596 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2)  exiting process(): stepName=child1-step1, sleepInMills=1,000
13:45:24,595 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job finished: BatchStatus=COMPLETED
13:45:24,601 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: persistentUserData=null, waitForFinish=null
13:45:24,601 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: starting new job: jobXMLName=jobdependency-child2, properties={sleepInMillis=2000}
13:45:24,605 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job started: executionId=208
13:45:24,605 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: waiting for finish the job... executionId=208
13:45:24,606 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=child2-step1, sleepInMills=2,000
13:45:26,608 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4)  exiting process(): stepName=child2-step1, sleepInMills=2,000
13:45:27,608 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job finished: BatchStatus=COMPLETED
13:45:27,614 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: persistentUserData=null, waitForFinish=null
13:45:27,614 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: starting new job: jobXMLName=jobdependency-child3, properties={sleepInMillis=3000}
13:45:27,618 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job started: executionId=209
13:45:27,618 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: waiting for finish the job... executionId=209
13:45:27,619 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) entering process(): stepName=child3-step1, sleepInMills=3,000
13:45:30,620 FINE  [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9)  exiting process(): stepName=child3-step1, sleepInMills=3,000
13:45:31,620 FINE  [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job finished: BatchStatus=COMPLETED

Job repository tables

job_instance and job_execution

jbatch=# select * from job_instance i, job_execution e where i.jobinstanceid = e.jobinstanceid and i.jobname like 'jobdependency%' and i.jobinstanceid >= 194 order by e.jobexecutionid;
 jobinstanceid | version |       jobname        | applicationname | jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus |    jobparameters     | restartposition 
---------------+---------+----------------------+-----------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
           194 |         | jobdependency        | jbatchtest      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      | 
           195 |         | jobdependency-child1 | jbatchtest      |            207 |           195 |         | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED   | COMPLETED  | sleepInMillis = 1000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
           196 |         | jobdependency-child2 | jbatchtest      |            208 |           196 |         | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED   | COMPLETED  | sleepInMillis = 2000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
           197 |         | jobdependency-child3 | jbatchtest      |            209 |           197 |         | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED   | COMPLETED  | sleepInMillis = 3000+| 
               |         |                      |                 |                |               |         |                         |                         |                         |                         |             |            |                      | 
(4 rows)

step_execution

jbatch=# select * from step_execution s, job_execution e, job_instance i where s.jobexecutionid = e.jobexecutionid and e.jobinstanceid = i.jobinstanceid and i.jobinstanceid >= 194 order by s.starttime;
 stepexecutionid | jobexecutionid | version |   stepname   |        starttime        |         endtime         | batchstatus | exitstatus | executionexception |                                                                           persistentuserdata                                                                           | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo | jobexecutionid | jobinstanceid | version |       createtime        |        starttime        |         endtime         |     lastupdatedtime     | batchstatus | exitstatus |    jobparameters     | restartposition | jobinstanceid | version |       jobname        | applicationname 
-----------------+----------------+---------+--------------+-------------------------+-------------------------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------+---------------+---------+----------------------+-----------------
             277 |            206 |         | step1        | 2014-02-19 13:45:22.587 | 2014-02-19 13:45:24.595 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000cf |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             278 |            207 |         | child1-step1 | 2014-02-19 13:45:22.593 | 2014-02-19 13:45:23.596 | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            207 |           195 |         | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED   | COMPLETED  | sleepInMillis = 1000+|                 |           195 |         | jobdependency-child1 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
             279 |            206 |         | step2        | 2014-02-19 13:45:24.599 | 2014-02-19 13:45:27.608 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d0 |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             280 |            208 |         | child2-step1 | 2014-02-19 13:45:24.605 | 2014-02-19 13:45:26.608 | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            208 |           196 |         | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED   | COMPLETED  | sleepInMillis = 2000+|                 |           196 |         | jobdependency-child2 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
             281 |            206 |         | step3        | 2014-02-19 13:45:27.612 | 2014-02-19 13:45:31.621 | COMPLETED   | COMPLETED  |                    | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d1 |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            206 |           194 |         | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED   | COMPLETED  |                      |                 |           194 |         | jobdependency        | jbatchtest
             282 |            209 |         | child3-step1 | 2014-02-19 13:45:27.618 | 2014-02-19 13:45:30.62  | COMPLETED   | SUCCESS    |                    |                                                                                                                                                                        |         0 |          0 |           0 |             0 |             0 |                0 |           0 |              0 |                      |                      |            209 |           197 |         | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED   | COMPLETED  | sleepInMillis = 3000+|                 |           197 |         | jobdependency-child3 | jbatchtest
                 |                |         |              |                         |                         |             |            |                    |                                                                                                                                                                        |           |            |             |               |               |                  |             |                |                      |                      |                |               |         |                         |                         |                         |                         |             |            |                      |                 |               |         |                      | 
(6 rows)
  • We can see that they were executed in particular order.

Remarks

  • I intent to care of stop and restart, and sample project contains some codes about it, but I haven't tested it yet. maybe I would write about it too later.
  • It can use some functions of jobXML such as parallel executions (like split element) for control of job execution even.

Take care of the max number of threads in thread-pool

  • Some implementations have limit of threads count.
  • When you used this method or similar one with many parallel execution elements (split or partition), you must ensure to keep number of threads lower than the limit.
  • If limit reaches, your batch will be frozen and it won't resume.

How to configure max threads

  • On WildFly + jBeret, the max number of threads is set to 10 as default.
  • This increases the limit to 100:
/subsystem=batch/thread-pool=batch:write-attribute(name=max-threads, value=100)



No one has commented yet.

Leave a Comment

HTML Syntax: NOT allowed