Poor performance while raising/lowering task steps for many processes

[aetherfaerber]

As we try to automate a lot of steps in our Kitodo workflow it is often necessary to raise or lower the step for remaining manual checks. Of course we want to advance all processes at once.

Advancing 1177 processes takes roughly 3 minutes (this is reproducible after several tests)

By mere multiplication, advancing 4800 processes should not take more than 15 minutes. But it takes 7200 min or 2 hours.

Naturally, we would actually like to advance a lot more processes than that at once. But know it seems we are forced to break it down to tiny bits and repeat the action of selecting and advancing again and again.

The only alternative we could find so far was to use a Kitodo script to advance the processes. This takes less than 10 minutes for 10,000 processes but we need to regenerate the index, which takes between 30 to 45 minutes again.

We have looked at existing issues like #5131 but the indexing doesn't seem to be as slow as described during reindexing but the poor performance primarily occurs when raising/lowering task steps.

Changing ElasticSearch's refresh_interval to -1 or 10min did not solve the issue.

How do you all raise your steps in a timely manner?

[henning-gerhardt]

In SLUB we are getting issues while setting up or down set task state, see #5813. So we reset only 10 processes at once down until we reached the needed workflow task. Setting task state up is not our daily use as it could trigger other issues with our automatic tasks (starting them twice or three times then only once).

We did not need to set the task state of so many tasks up (or down) as this is done by the application itself. We have a lot of automatic tasks which in most cases execute a script which is exiting with state -1 so the task itself remain open and the task then will be closed after the work is done by using the ActiveMQ queue mechanism to close a task.
So manual setting up the task state did only happen when the executed external work was failed or the message from the message queue server get lost. But in most error cases we restart the this task and in most cases on a second execution the external execution is then fine and task get closed in the normal way.

[aetherfaerber]

Thank you for your prompt answer! I will see if we can find a solution from your example.

In SLUB we are getting issues while setting up or down set task state, see #5813. So we reset only 10 processes at once down until we reached the needed workflow task. Setting task state up is not our daily use as it could trigger other issues with our automatic tasks (starting them twice or three times then only once).

Going in steps of 10 processes at a time is not an option for imports of several thousand processes. How do you start your automatic tasks if not by mass-marking the previous step done? We would love for our automatic steps to just run (automatically!) without manual triggering but Kitodo does not allow the first step of a workflow to be automatic. So how do you reach your first automatic step after creating all the processes?

We did not need to set the task state of so many tasks up (or down) as this is done by the application itself. We have a lot of automatic tasks which in most cases execute a script which is exiting with state -1 so the task itself remain open and the task then will be closed after the work is done by using the ActiveMQ queue mechanism to close a task. So manual setting up the task state did only happen when the executed external work was failed or the message from the message queue server get lost. But in most error cases we restart the this task and in most cases on a second execution the external execution is then fine and task get closed in the normal way.

Do I understand correctly, that marking lots of steps done via ActiveMQ works swiftly? Does this mean we could use another application to create a view thousand processes at once and then finish their useless first step automatically? This other application would need to know the process ID again and this all seems quite cumbersome – why use Kitodo at all for process management, when processes are created and managed solely through another application?
Also, we do have steps that need manual actions but should be mass-closable nontheless.

[henning-gerhardt]

Going in steps of 10 processes at a time is not an option for imports of several thousand processes.

Understandable.

How do you start your automatic tasks if not by mass-marking the previous step done?

We have 3 manual tasks: creating the process from a catalogue itself, copying the picture data from any source into Kitodo.Production data storage and enrich the meta and structure data.

The workflow in SLUB is the following:

1. Create process (manual task or done by newspaper creating task for newspaper issues) from catalogue
2. Import picture data from any source into the Kitodo.Production data storage (manual task as the source differ over a lot of places, set done by human after data is transfered)
3. Validate the picture data (automatic task, executed extern, task get closed by ActiveMQ)
4. Generate preview pictures for the meta data editor (automatic task, executed extern, task get closed by ActiveMQ)
5. Enrich meta and structure data (manual task, set done by human after enrichment and maybe quality check is done)
6. Validate picture and meta data (automatic task, executed extern, task get closed by ActiveMQ)
7. Run OCR on processes where OCR should be executed (automatic task, executed extern, task get closed by ActiveMQ)
8. Export meta and pictures into a share for later presentation (automatic task, executed by application)
9. Archive pictures and OCR data in a long term persistence archive (automatic task, executed extern, task get closed by ActiveMQ)

As all of this 3 tasks are done by human the automatic tasks started after the previous manual task is set to done.

We would love for our automatic steps to just run (automatically!) without manual triggering but Kitodo does not allow the first step of a workflow to be automatic. So how do you reach your first automatic step after creating all the processes?

On SLUB we have a different work approach and work flow. We are doing a running a lot of automatic tasks but the start is every time started by human. Using Kitodo.Production in our way is new to me and yes you need the first task as automatic task except you would change your workflow.

Do I understand correctly, that marking lots of steps done via ActiveMQ works swiftly?

Its work for us in a very good way but we are not hammering thousand of messages per second into the message queue. Sending this messages is not the problem but maybe the handling inside the application. I don't know if the application can handle this in a performant way on a long time.

Does this mean we could use another application to create a view thousand processes at once and then finish their useless first step automatically?

In theory yes, but you need to know the task id to close this task over ActiveMQ as this task id is needed. How you can find them - I don't know. We got the task id by starting the automatic task, as the automatic task are executing a shell/python script and the task id is one of multiple parameters which are transferred to the called script.

This other application would need to know the process ID again

The other application must know at least the task id - the process id won't help as closing a task is depending on the task id.

Also, we do have steps that need manual actions but should be mass-closable nontheless.

Why are the persons who are interacting with this manual tasks are not closing the tasks if there work is done? Then it would not be necessary to close this manual tasks in a mass way.

[aetherfaerber]

Through further tests we have been able to link this issue to hierarchical processes. As it seems, a bunch of processes without a parent process can be pushed forward and backward much easier than processes that do have a parent processes. But leaving the processes without a parent makes it much harder and error prone to select all of them at once in order to move them forward.

[solth]

Through further tests we have been able to link this issue to hierarchical processes.

If this has something to do with hierarchical processes, perhaps it is related to #5934 and could be alliviated by #6185 ?

[henning-gerhardt]

You are using Kitodo.Production in a way which is currently not supported nor was a usage scenario in the past so development your needed features was not in any way of planing. If you needed this your features should be developed without breaking current working way or you must adept your workflow to the current supported way of using Kitodo.Production.

[henning-gerhardt]

If this has something to do with hierarchical processes, perhaps it is related to #5934 and could be alliviated by #6185 ?

I don't think you but I can not say that is may be related. I think that has something more to do how the state of a hierarchical process is calculated and maybe other checks which are be made on setting the process state forward or backward.

[BartChris]

Through further tests we have been able to link this issue to hierarchical processes.

If this has something to do with hierarchical processes, perhaps it is related to #5934 and could be alliviated by #6185 ?

I suppose those changes are not helping here as they are only related to the display of parent processes. I suspect we have different issues here. I looked at the code, and one thing that caught my attention is that when we close a task, we always call closeParent.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/workflow/WorkflowControllerService.java

Lines 477 to 486 in e68fd3c

	closeParent(process);
	}
	private void closeParent(Process process) throws DataException {
	if (Objects.nonNull(process.getParent()) && allChildrenClosed(process.getParent())) {
	process.getParent().setSortHelperStatus(ProcessState.COMPLETED.getValue());
	ServiceManager.getProcessService().save(process.getParent());
	closeParent(process.getParent());
	}
	}

In that method we call allChildrenClosed which will check all children for their status.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/workflow/WorkflowControllerService.java

Lines 263 to 273 in e68fd3c

	public static boolean allChildrenClosed(Process process) {
	if (!process.getChildren().isEmpty()) {
	boolean allChildrenClosed = true;
	for (Process child : process.getChildren()) {
	allChildrenClosed &= ProcessState.COMPLETED20.getValue().equals(child.getSortHelperStatus())
	|| ProcessState.COMPLETED.getValue().equals(child.getSortHelperStatus());
	}
	return allChildrenClosed;
	}
	return false;
	}

This might add a lot of overhead. One could definitely try to make improvements there:

• If the current process is still incomplete, there is no need to check the status of all other child processes. An incomplete process implies that at least one child process is also incomplete, so further checks can be skipped.
• If we have to do the check, we can probably offload the check to the database to not having to load all the processes into memory.

Something like that

public static boolean allChildrenClosed(Process process) {
    String hql = "SELECT COUNT(c) FROM Process c WHERE c.parent = :parent "
               + "AND c.sortHelperStatus NOT IN (:completedStates)";
    
    Map<String, Object> parameters = new HashMap<>();
    parameters.put("parent", process);
    parameters.put("completedStates", Arrays.asList(
        ProcessState.COMPLETED20.getValue(), 
        ProcessState.COMPLETED.getValue()
    ));
    
    List<Long> result = getByQuery(hql, parameters);  // Fetch the count
    
    Long count = result.get(0);
    return count == 0;
}

[aetherfaerber]

Through further tests we have been able to link this issue to hierarchical processes.

If this has something to do with hierarchical processes, perhaps it is related to #5934 and could be alliviated by #6185 ?

I suppose those changes are not helping here as they are only related to the display of parent processes. I suspect we have different issues here. I looked at the code, and one thing that caught my attention is that when we close a task, we always call closeParent.

Thank your for pointing in this direction. This could very well be at the heart of the issue.

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/workflow/WorkflowControllerService.java

Lines 263 to 273 in e68fd3c

	public static boolean allChildrenClosed(Process process) {
	if (!process.getChildren().isEmpty()) {
	boolean allChildrenClosed = true;
	for (Process child : process.getChildren()) {
	allChildrenClosed &= ProcessState.COMPLETED20.getValue().equals(child.getSortHelperStatus())
	|| ProcessState.COMPLETED.getValue().equals(child.getSortHelperStatus());
	}
	return allChildrenClosed;
	}
	return false;
	}

This might add a lot of overhead. One could definitely try to make improvements there:

• If we have to do the check, we can probably offload the check to the database to not having to load all the processes into memory.

Something like that

public static boolean allChildrenClosed(Process process) {
    String hql = "SELECT COUNT(c) FROM Process c WHERE c.parent = :parent "
               + "AND c.sortHelperStatus NOT IN (:completedStates)";
    
    Map<String, Object> parameters = new HashMap<>();
    parameters.put("parent", process);
    parameters.put("completedStates", Arrays.asList(
        ProcessState.COMPLETED20.getValue(), 
        ProcessState.COMPLETED.getValue()
    ));
    
    List<Long> result = getByQuery(hql, parameters);  // Fetch the count
    
    Long count = result.get(0);
    return count == 0;
}

As far as I understand this would change the lookup mechanism for the state of all child processes of the parent. Shouldn't another check be implemented as you suggested:

• If the current process is still incomplete, there is no need to check the status of all other child processes. An incomplete process implies that at least one child process is also incomplete, so further checks can be skipped.

and therefore

kitodo-production/Kitodo/src/main/java/org/kitodo/production/services/workflow/WorkflowControllerService.java

Lines 477 to 486 in e68fd3c

	closeParent(process);
	}
	private void closeParent(Process process) throws DataException {
	if (Objects.nonNull(process.getParent()) && allChildrenClosed(process.getParent())) {
	process.getParent().setSortHelperStatus(ProcessState.COMPLETED.getValue());
	ServiceManager.getProcessService().save(process.getParent());
	closeParent(process.getParent());
	}
	}

In that method we call allChildrenClosed which will check all children for their status.

include first a check on the current process and its status, then skip line 477 entirely if the status is not 'closed'?

[BartChris]

As far as I understand this would change the lookup mechanism for the state of all child processes of the parent. Shouldn't another check be implemented as you suggested and therefore include first a check on the current process and its status, then skip line 477 entirely if the status is not 'closed'?

Yes, i think that in many cases (during a workflow) we can skip the check for the closing of the parent entirely. This check has only to be executed if the current process is finished. And i suppose this will speed up things significantly.

[aetherfaerber]

Thank you, especially for pointing to possible changes that could resolve the problem. We have ceased using hierarchical processes for now and hope to come back to this later.

[BartChris]

I tried to come up with a draft Pull Request. #6228