Skip to content

Commit

Permalink
prosopopee: allow selecting number of threads used for thumbnail gene…
Browse files Browse the repository at this point in the history
…ration

Generating thumbnails is done in parallel threads via
multiprocessing.Pool. By default, Pool schedules tasks on as many
threads as there are cpu threads on the host machine.

Let's allow users to select the number of threads Pool can use.

Signed-off-by: Quentin Schulz <[email protected]>
  • Loading branch information
QSchulz committed Oct 10, 2021
1 parent 0de1e40 commit ba91bcb
Showing 1 changed file with 15 additions and 2 deletions.
17 changes: 15 additions & 2 deletions prosopopee/prosopopee.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,15 @@ def loglevel(string):
help="Configure the logging level",
)
subparser = parser.add_subparsers(dest="cmd")
subparser.add_parser("build", help="Generate static site")
parser_build = subparser.add_parser("build", help="Generate static site")
parser_build.add_argument(
"-j",
"--jobs",
default=None,
type=int,
help="Specifies number of jobs (thumbnail generations) to run simultaneously. Default: number "
"of threads available on the system",
)
subparser.add_parser("test", help="Verify all your yaml data")
subparser.add_parser("preview", help="Start preview webserver on port 9000")
subparser.add_parser("deploy", help="Deploy your website")
Expand Down Expand Up @@ -923,7 +931,12 @@ def main():
logger.info("Success: HTML file building without error")
sys.exit(0)

with Pool() as pool:
# If prosopopee is started without any argument, 'build' is assumed but the jobs parameter
# is not part of the namespace, so set its default to None (or 'number of available CPU
# treads')
jobs = args.jobs if args.cmd else None

with Pool(jobs) as pool:
# Pool splits the iterable into pre-defined chunks which are then assigned to processes.
# There is no other scheduling in play after that. This is an issue when chunks are
# outrageously unbalanced in terms of CPU time which happens when most galleries are
Expand Down

0 comments on commit ba91bcb

Please sign in to comment.