A way to resolve dependencies of jobs
TweetPosted on Sunday Feb 16, 2014 at 08:17AM in Technology
This is about a way to resolve dependencies of jobs that used only JSR 352 API.
Environment
- jBeret 1.0.1Beta-SNAPSHOT
- WildFly 8.0.0.Final
Why need it?
- For tasks that need to execute sequentially, usually we have to let all tasks in single job XML.
- I guess it would bring huge difficulty of testing.
- For example, I don't want to test such big job like EndOfTheDay.xml
- Rather than I would go with smaller jobs with external job control system.
How realize it?
- This needs to create parent job XML that defines dependencies of child jobs.
- Every step uses artifact named “jobLauncherBatchlet”
- This batchlet invokes child job. name of the job and job parameters are given by its parameter of the step.
- After invoke the job, then it starts polling BatchStatus, and wait till the job is finished.
Sample project
Whole resources are available in GitHub.
- Artifacts
- jobXMLs
- Test class
log
13:45:22,589 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: persistentUserData=null, waitForFinish=null
13:45:22,589 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: starting new job: jobXMLName=jobdependency-child1, properties={sleepInMillis=1000}
13:45:22,593 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job started: executionId=207
13:45:22,593 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: waiting for finish the job... executionId=207
13:45:22,595 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) entering process(): stepName=child1-step1, sleepInMills=1,000
13:45:23,596 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 2) exiting process(): stepName=child1-step1, sleepInMills=1,000
13:45:24,595 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step1: job finished: BatchStatus=COMPLETED
13:45:24,601 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: persistentUserData=null, waitForFinish=null
13:45:24,601 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: starting new job: jobXMLName=jobdependency-child2, properties={sleepInMillis=2000}
13:45:24,605 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job started: executionId=208
13:45:24,605 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: waiting for finish the job... executionId=208
13:45:24,606 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) entering process(): stepName=child2-step1, sleepInMills=2,000
13:45:26,608 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 4) exiting process(): stepName=child2-step1, sleepInMills=2,000
13:45:27,608 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step2: job finished: BatchStatus=COMPLETED
13:45:27,614 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: persistentUserData=null, waitForFinish=null
13:45:27,614 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: starting new job: jobXMLName=jobdependency-child3, properties={sleepInMillis=3000}
13:45:27,618 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job started: executionId=209
13:45:27,618 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: waiting for finish the job... executionId=209
13:45:27,619 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) entering process(): stepName=child3-step1, sleepInMills=3,000
13:45:30,620 FINE [org.nailedtothex.jbatch.example.split.SleepBatchlet] (batch-batch - 9) exiting process(): stepName=child3-step1, sleepInMills=3,000
13:45:31,620 FINE [org.nailedtothex.jbatch.example.jobdependency.JobLaunchBatchlet] (batch-batch - 8) step3: job finished: BatchStatus=COMPLETED
Job repository tables
job_instance and job_execution
jbatch=# select * from job_instance i, job_execution e where i.jobinstanceid = e.jobinstanceid and i.jobname like 'jobdependency%' and i.jobinstanceid >= 194 order by e.jobexecutionid;
jobinstanceid | version | jobname | applicationname | jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition
---------------+---------+----------------------+-----------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------
194 | | jobdependency | jbatchtest | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | |
195 | | jobdependency-child1 | jbatchtest | 207 | 195 | | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED | COMPLETED | sleepInMillis = 1000+|
| | | | | | | | | | | | | |
196 | | jobdependency-child2 | jbatchtest | 208 | 196 | | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED | COMPLETED | sleepInMillis = 2000+|
| | | | | | | | | | | | | |
197 | | jobdependency-child3 | jbatchtest | 209 | 197 | | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED | COMPLETED | sleepInMillis = 3000+|
| | | | | | | | | | | | | |
(4 rows)
step_execution
jbatch=# select * from step_execution s, job_execution e, job_instance i where s.jobexecutionid = e.jobexecutionid and e.jobinstanceid = i.jobinstanceid and i.jobinstanceid >= 194 order by s.starttime;
stepexecutionid | jobexecutionid | version | stepname | starttime | endtime | batchstatus | exitstatus | executionexception | persistentuserdata | readcount | writecount | commitcount | rollbackcount | readskipcount | processskipcount | filtercount | writeskipcount | readercheckpointinfo | writercheckpointinfo | jobexecutionid | jobinstanceid | version | createtime | starttime | endtime | lastupdatedtime | batchstatus | exitstatus | jobparameters | restartposition | jobinstanceid | version | jobname | applicationname
-----------------+----------------+---------+--------------+-------------------------+-------------------------+-------------+------------+--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+------------+-------------+---------------+---------------+------------------+-------------+----------------+----------------------+----------------------+----------------+---------------+---------+-------------------------+-------------------------+-------------------------+-------------------------+-------------+------------+----------------------+-----------------+---------------+---------+----------------------+-----------------
277 | 206 | | step1 | 2014-02-19 13:45:22.587 | 2014-02-19 13:45:24.595 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000cf | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
278 | 207 | | child1-step1 | 2014-02-19 13:45:22.593 | 2014-02-19 13:45:23.596 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 207 | 195 | | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:22.592 | 2014-02-19 13:45:23.599 | 2014-02-19 13:45:23.599 | COMPLETED | COMPLETED | sleepInMillis = 1000+| | 195 | | jobdependency-child1 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
279 | 206 | | step2 | 2014-02-19 13:45:24.599 | 2014-02-19 13:45:27.608 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
280 | 208 | | child2-step1 | 2014-02-19 13:45:24.605 | 2014-02-19 13:45:26.608 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 208 | 196 | | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:24.604 | 2014-02-19 13:45:26.611 | 2014-02-19 13:45:26.611 | COMPLETED | COMPLETED | sleepInMillis = 2000+| | 196 | | jobdependency-child2 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
281 | 206 | | step3 | 2014-02-19 13:45:27.612 | 2014-02-19 13:45:31.621 | COMPLETED | COMPLETED | | \xaced00057372000e6a6176612e6c616e672e4c6f6e673b8be490cc8f23df0200014a000576616c7565787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000000000d1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 206 | 194 | | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:22.586 | 2014-02-19 13:45:31.623 | 2014-02-19 13:45:31.623 | COMPLETED | COMPLETED | | | 194 | | jobdependency | jbatchtest
282 | 209 | | child3-step1 | 2014-02-19 13:45:27.618 | 2014-02-19 13:45:30.62 | COMPLETED | SUCCESS | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | | 209 | 197 | | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:27.617 | 2014-02-19 13:45:30.624 | 2014-02-19 13:45:30.624 | COMPLETED | COMPLETED | sleepInMillis = 3000+| | 197 | | jobdependency-child3 | jbatchtest
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
(6 rows)
- We can see that they were executed in particular order.
Remarks
- I intent to care of stop and restart, and sample project contains some codes about it, but I haven't tested it yet. maybe I would write about it too later.
- It can use some functions of jobXML such as parallel executions (like split element) for control of job execution even.
Take care of the max number of threads in thread-pool
- Some implementations have limit of threads count.
- When you used this method or similar one with many parallel execution elements (split or partition), you must ensure to keep number of threads lower than the limit.
- If limit reaches, your batch will be frozen and it won't resume.
How to configure max threads
- On WildFly + jBeret, the max number of threads is set to 10 as default.
- This increases the limit to 100:
/subsystem=batch/thread-pool=batch:write-attribute(name=max-threads, value=100)
Tags: jbatch
