Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a threadpool for faster builds. #282

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions jekyll_picture_tag.gemspec
Original file line number Diff line number Diff line change
@@ -29,6 +29,8 @@ Gem::Specification.new do |spec|

# addressable is used to url-encode image filenames.
spec.add_runtime_dependency 'addressable', '~> 2.6'
# Needed for parallel excution of builds.
spec.add_runtime_dependency 'concurrent-ruby', '~> 1.1'
# Jekyll versions older than 4.0 are not supported.
spec.add_runtime_dependency 'jekyll', '~> 4.0'
# MIME types are needed for <source> tags' type= attributes.
37 changes: 35 additions & 2 deletions lib/jekyll_picture_tag/srcsets/basic.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
require 'mime-types'
require 'set'
require 'concurrent-ruby'

module PictureTag
# Handles srcset generation, which also handles file generation.
module Srcsets
@@ -62,16 +65,30 @@ def height_attribute

private

# Defining a threadpool takes 5 lines, but isn't really that complex
# of an operation, so we disable the rubocop method length warning.
# rubocop:disable Metrics/MethodLength
def build_files
# By 'files', we mean the GeneratedImage class.
return target_files if target_files.all?(&:exists?)

# This triggers GeneratedImage to actually build an image file.
files = checked_targets
files.each(&:generate)
files = unique_checked_targets
pool = Concurrent::ThreadPoolExecutor.new(
min_threads: 1, max_threads: Concurrent.processor_count,
max_queue: 2 * Concurrent.processor_count
)
files.each do |file|
pool.post do
file.generate
end
end
pool.shutdown
pool.wait_for_termination

files
end
# rubocop:enable Metrics/MethodLength

def checked_targets
if target_files.any? { |f| f.width > source_width }
@@ -85,6 +102,22 @@ def checked_targets
files || target_files
end

def unique_checked_targets
# Because we're processing in parallel, we can't use existince
# in the filestystem to prevent duplicate handling of files without
# a race condition, so we manually track generated files here so we
# don't attempt to generate the same image twice.
seen_files = Set[]
unique_targets = []
checked_targets.each do |target|
unless seen_files.include?(target.name)
seen_files.add(target.name)
unique_targets.push(target)
end
end
unique_targets
end

def source_width
source_image.width
end