diff --git a/.standard.yml b/.standard.yml index aea2b1c..61f239e 100644 --- a/.standard.yml +++ b/.standard.yml @@ -1,3 +1,5 @@ ignore: - 'test/**/*': - - Style/GlobalVars \ No newline at end of file + - Style/GlobalVars + +ruby_version: 3.0 diff --git a/CAVEATS.md b/CAVEATS.md index e74cffa..9586eec 100644 --- a/CAVEATS.md +++ b/CAVEATS.md @@ -2,19 +2,19 @@ This is a document that lists some caveats that you need to consider when dealing with Litestack and SQLite applications -## Single host (server) only +## Single host (server) only -By design, SQLite can be accessed only (in a performant and safe way) from a single host machine. That means that you cannot have things like auto-scaling of app servers since there is but a single one of them. This app server can be a huge one, with many processes, but only one app server at a time. +By design, SQLite can be accessed only (in a performant and safe way) from a single host machine. That means that you cannot have things like auto-scaling of app servers since there is but a single one of them. This app server can be a huge one, with many processes, but only one app server at a time. ## Containerization becomes a bit tricky Container technology was originally designed for stateless applications. Based on this, one of the core aspects of containers is that they are immutable, deploying an app to container means destroying one container and creating another. With statefull applications, like database servers and similarly SQLite based applications, this will potentially mean a down time will be perceived, which in some approaches is mitigate but not perfectly (yet). Expect some quirks when dealing with SQLite and containers. -## Single writer / multi reader +## Single writer / multi reader In its default configuration, Litestack allows one writer at a time to write to the database file, while simultaneously allowing any number of readers to access the database. Generally SQLite is pretty fast in writes, but you should be aware that a long running write operation will prevent any other writers from proceeding, for example creating an index on a very large table will stop other writers until the index creation is finished. There is no concurrent index creation facility in SQLite, yet. ## Litestack does not release the GVL -This also applies to the Ruby SQLite3 gem, when it is performing a query, it will not allow any other threads in the Ruby process to resume until it returns. Care should be taken when writing low latency multi-threaded applications with Litestack, it is generally recommended to use fibers over threads in that case, since everything is serialized anyway it makes sense to avoid the ovreheads of mult-threaded environments - +This also applies to the Ruby SQLite3 gem, when it is performing a query, it will not allow any other threads in the Ruby process to resume until it returns. Care should be taken when writing low latency multi-threaded applications with Litestack, it is generally recommended to use fibers over threads in that case, since everything is serialized anyway it makes sense to avoid the ovreheads of mult-threaded environments + diff --git a/CHANGELOG.md b/CHANGELOG.md index 5186056..af4a287 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,11 +5,11 @@ - [View Diff](https://github.com/oldmoe/litestack/compare/v0.4.1...master) - Add "sequel" as a development dependency - Diff links in CHANGELOG (thanks Weston Ganger) -- Fix deamonize type in liteboard (thanks Julian Rubisch) +- Fix daemonize type in liteboard (thanks Julian Rubisch) - Better Litecache schema (streamlined numeric value support) - Support for set_multi and get_multi in Litecache (read_multi and write_multi support for Rails Cache store) - More tests written for Litecache and Rails Litecache store -- Experimenting with removing the Rails LocalCache as it doesn't show enough improvement in perfromance to compensate for the memory overhead +- Experimenting with removing the Rails LocalCache as it doesn't show enough improvement in performance to compensate for the memory overhead - Switch Litecache to a FIFO eviction model vs LRU (thanks Julian Rubisch and Stephen Margheim) ## [0.4.2] - 2023-11-11 @@ -20,7 +20,7 @@ - Fix Litesearch tests - Suppress chatty Litejob exit detector when there are no jobs in flight - Tidy up the test folder -- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting propogated +- [#41](https://github.com/oldmoe/litestack/pull/41) - Fix bug in Litecable where the `connected` event was not getting propagated - Add Litemetric and Liteboard info to README.doc - Fix the testing rake task diff --git a/FILESYSTEMS.md b/FILESYSTEMS.md index 16cc6d3..8a03476 100644 --- a/FILESYSTEMS.md +++ b/FILESYSTEMS.md @@ -1,10 +1,10 @@ # Filesystems & Litestack -In the containerized world we live in, many layers of the hardware/software stacks are far abstracted that we no longer know they exist. For the filesystem enthusiasts out there this is a quick overview of how Litestack (and hence SQLite) can benefit from different filesytems +In the containerized world we live in, many layers of the hardware/software stacks are far abstracted that we no longer know they exist. For the filesystem enthusiasts out there this is a quick overview of how Litestack (and hence SQLite) can benefit from different filesystem ## XFS -A very stable and trusted filesystem with excellent performance charactersitcs +A very stable and trusted filesystem with excellent performance characteristics - Fast reads / writes @@ -14,7 +14,7 @@ Another stable and performant filesystem - Fast reads / writes -## F2FS +## F2FS Specially built for solid state storage, has an atomic write mode that is supported by SQLite @@ -32,7 +32,7 @@ Copy-on-write filesystem with a nifty set of features, very fast snapshotting bu - fast device cache - Compression -## Btrfs +## Btrfs Another CoW filesystem that delivers snapshotting and incremental send/recv at a much granular level. @@ -52,4 +52,4 @@ A new CoW filesystem built on the foundations of the bcache module. Improving ra - + diff --git a/ROADMAP.md b/ROADMAP.md index 2e4b456..9e0763a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -21,25 +21,25 @@ This is a list of functional/non-functional features that litestack targets to i - [ ] Automated testing matrix - [ ] Automated gem builds - [ ] Litedb improvements - - [ ] ActiveRecord Sqlite3 Adapter compatibility mode + - [ ] ActiveRecord Sqlite3 Adapter compatibility mode - [ ] Sequel Sqlite3 Adapter compatibility mode - - [ ] Extension bundling/loading + - [ ] Extension bundling/loading - [ ] Liteiob improvements - [ ] Persist jobs even during execution - [ ] Zombie job detection - - [ ] Better process exit handling + - [ ] Better process exit handling - [ ] Faster job dispatch for the no delay case - [ ] Database maintenance scripts - [ ] Online backup - [ ] Restore - [ ] Litestream integration - - [ ] CoW filesystems specific backup path + - [ ] CoW filesystem-specific backup path - [ ] Zero downtime deployment scripts - [ ] Rails - - [ ] Genereic Rack Applications + - [ ] Generic Rack Applications - [ ] Litemetric improvements - [ ] Rails performance module - [ ] Ruby Memory/GC module - [ ] Restore generic module - [ ] Better HTML/CSS -- [ ] Kredis replacement implementation? +- [ ] Kredis replacement implementation? diff --git a/bench/bench_cache_rails.rb b/bench/bench_cache_rails.rb index fe582e1..f56ec08 100644 --- a/bench/bench_cache_rails.rb +++ b/bench/bench_cache_rails.rb @@ -1,6 +1,6 @@ require "active_support" require_relative "../lib/litestack" -require_relative "./bench" +require_relative "bench" cache = ActiveSupport::Cache::Litecache.new @@ -32,17 +32,17 @@ end puts "== Multi Writes ==" - bench("litecache multi-writes", count/5) do |i| - idx = i * 5 - payload = {} - 5.times {|j| payload[keys[idx + j]] = values[idx + j] } + bench("litecache multi-writes", count / 5) do |i| + idx = i * 5 + payload = {} + 5.times { |j| payload[keys[idx + j]] = values[idx + j] } cache.write_multi(payload) end - bench("Redis multi-writes", count/5) do |i| - idx = i * 5 - payload = {} - 5.times {|j| payload[keys[idx + j]] = values[idx + j] } + bench("Redis multi-writes", count / 5) do |i| + idx = i * 5 + payload = {} + 5.times { |j| payload[keys[idx + j]] = values[idx + j] } redis.write_multi(payload) end @@ -56,21 +56,20 @@ end puts "== Multi Reads ==" - bench("litecache multi-reads", count/5) do |i| + bench("litecache multi-reads", count / 5) do |i| idx = i * 5 payload = [] - 5.times {|j| payload << random_keys[idx+j]} + 5.times { |j| payload << random_keys[idx + j] } cache.read_multi(*payload) end - bench("Redis multi-reads", count/5) do |i| + bench("Redis multi-reads", count / 5) do |i| idx = i * 5 payload = [] - 5.times {|j| payload << random_keys[idx+j]} + 5.times { |j| payload << random_keys[idx + j] } redis.read_multi(*payload) end - puts "==========================================================" keys = [] diff --git a/bench/bench_cache_raw.rb b/bench/bench_cache_raw.rb index 20b837b..14ed9c3 100644 --- a/bench/bench_cache_raw.rb +++ b/bench/bench_cache_raw.rb @@ -1,6 +1,6 @@ require "redis" require "sqlite3" -require_relative "./bench" +require_relative "bench" # require 'polyphony' require "async/scheduler" @@ -11,7 +11,7 @@ require_relative "../lib/litestack/litecache" # require 'litestack' -cache = Litecache.new #({path: "../db/cache.db"}) # default settings +cache = Litecache.new # ({path: "../db/cache.db"}) # default settings redis = Redis.new # default settings values = [] @@ -40,17 +40,20 @@ end puts "== Multi Writes ==" - bench("litecache multi-writes", count/5) do |i| - idx = i * 5 - payload = {} - 5.times {|j| payload[keys[idx + j]] = values[idx + j] } + bench("litecache multi-writes", count / 5) do |i| + idx = i * 5 + payload = {} + 5.times { |j| payload[keys[idx + j]] = values[idx + j] } cache.set_multi(payload) - end + end - bench("Redis multi-writes", count/5) do |i| - idx = i * 5 - payload = [] - 5.times {|j| payload << keys[idx + j]; payload << values[idx + j]} + bench("Redis multi-writes", count / 5) do |i| + idx = i * 5 + payload = [] + 5.times { |j| + payload << keys[idx + j] + payload << values[idx + j] + } redis.mset(*payload) end @@ -64,17 +67,17 @@ end puts "== Multi Reads ==" - bench("litecache multi-reads", count/5) do |i| + bench("litecache multi-reads", count / 5) do |i| idx = i * 5 payload = [] - 5.times {|j| payload << random_keys[idx+j]} + 5.times { |j| payload << random_keys[idx + j] } cache.get_multi(*payload) end - bench("Redis multi-reads", count/5) do |i| + bench("Redis multi-reads", count / 5) do |i| idx = i * 5 payload = [] - 5.times {|j| payload << random_keys[idx+j]} + 5.times { |j| payload << random_keys[idx + j] } redis.mget(*payload) end diff --git a/bin/liteboard b/bin/liteboard index 0b44bac..d24087b 100755 --- a/bin/liteboard +++ b/bin/liteboard @@ -11,10 +11,10 @@ DEFAULTS = { path: Litemetric::DEFAULT_OPTIONS[:path] } -options = { +options = { config_path: nil, path: nil, - deamonize: false, + daemonize: false, Port: 9292, Host: 'localhost', environment: 'production', @@ -29,9 +29,9 @@ OptionParser.new do |parser| parser.on("-s", "--server SERVER", "use SERVER (e.g. puma/falcon/iodine)") { |v| options[:port] = v } parser.on("-H", "--host HOST", "listen on HOST (default: #{options[:Host]})") { |v| options[:Host] = v } parser.on("-p", "--port PORT", "use PORT (default: #{options[:Port]})") { |v| options[:Port] = v.to_i rescue options[:Port] } - parser.on("-D", "--daemonize", "run in the background") { |v| options[:daemonize] = true } - parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v } - parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true } + parser.on("-D", "--daemonize", "run in the background") { |v| options[:daemonize] = true } + parser.on("-E", "--env ENVIRONMENT", "which environment to use (default: #{options[:environment]})") { |v| options[:environment] = v } + parser.on("-q", "--quiet", "turn off logging") { |v| options[:quiet] = true } parser.on("-h", "--help", "print this message") do puts parser exit @@ -39,14 +39,14 @@ OptionParser.new do |parser| end.parse! def check_database(path) - unless File.exist?(path) + unless File.exist?(path) puts "liteboard: missing database file, please ensure the db path is correct" puts "liteboard: exiting" exit end end -check_database(options[:path]) if options[:path] +check_database(options[:path]) if options[:path] # if there is a config file then we need to check it config = nil @@ -57,7 +57,7 @@ if options[:config_path] puts "liteboard: missing or bad config file, please ensure the config file path is correct" puts "liteboard: exiting" exit - end + end else # no config path! use the default config = YAML.load(ERB.new(File.read(DEFAULTS[:config_path])).result) rescue nil end @@ -66,13 +66,12 @@ if config if options[:path].nil? path = config['path'] || config[options[:environment]]['path'] options[:path] = path - end + end end # if still no path we assume a default db path options[:path] = DEFAULTS[:path] if options[:path].nil? - # check the validity of the path before starting the server check_database(options[:path]) Litemetric.options = options diff --git a/lib/active_support/cache/litecache.rb b/lib/active_support/cache/litecache.rb index 6540544..d88dcdd 100644 --- a/lib/active_support/cache/litecache.rb +++ b/lib/active_support/cache/litecache.rb @@ -9,7 +9,7 @@ module ActiveSupport module Cache class Litecache < Store - #prepend Strategy::LocalCache + # prepend Strategy::LocalCache def self.supports_cache_versioning? true @@ -27,9 +27,9 @@ def increment(key, amount = 1, options = nil) # todo: fix me # this is currently a hack to avoid dealing with Rails cache encoding and decoding # and it can result in a race condition as it stands - # @cache.transaction(:immediate) do - # currently transactions are not compatible with acquiring connections - # this needs fixing by storing the connection to the context once acquired + # @cache.transaction(:immediate) do + # currently transactions are not compatible with acquiring connections + # this needs fixing by storing the connection to the context once acquired if (value = read(key, options)) value = value.to_i + amount write(key, value, options) @@ -52,7 +52,7 @@ def cleanup(limit = nil, time = nil) @cache.prune(limit) end - def clear(options=nil) + def clear(options = nil) @cache.clear end @@ -74,11 +74,11 @@ def stats private - def serialize_entrys(entry, **options) + def serialize_entries(entry, **options) Marshal.dump(entry) end - def deserialize_entrys(entry) + def deserialize_entries(entry) Marshal.load(entry.to_s) end @@ -91,23 +91,23 @@ def read_multi_entries(names, **options) results = {} return results if names == [] rs = @cache.get_multi(*names.flatten) - rs.each_pair{|k, v| results[k] = deserialize_entry(v).value } + rs.each_pair { |k, v| results[k] = deserialize_entry(v).value } results end - + # Write an entry to the cache. def write_entry(key, entry, **options) write_serialized_entry(key, serialize_entry(entry, **options), **options) end - + def write_multi_entries(entries, **options) return if entries.empty? - entries.each_pair {|k,v| entries[k] = serialize_entry(v, **options)} + entries.each_pair { |k, v| entries[k] = serialize_entry(v, **options) } expires_in = options[:expires_in].to_i if options[:expires_in] if options[:race_condition_ttl] && expires_in > 0 && !options[:raw] expires_in += 5.minutes end - @cache.set_multi(entries, expires_in) + @cache.set_multi(entries, expires_in) end def write_serialized_entry(key, payload, **options) diff --git a/lib/generators/litestack/install/install_generator.rb b/lib/generators/litestack/install/install_generator.rb index e529e8c..8f4fac8 100644 --- a/lib/generators/litestack/install/install_generator.rb +++ b/lib/generators/litestack/install/install_generator.rb @@ -1,10 +1,10 @@ class Litestack::InstallGenerator < Rails::Generators::Base source_root File.expand_path("templates", __dir__) - # Force copy configuratioon files so Rails installs don't ask questions + # Force copy configuration files so Rails installs don't ask questions # that less experienced people might not understand. The more Sr folks. # will know to check git to look at what changed. - def modify_database_adapater + def modify_database_adapter copy_file "database.yml", "config/database.yml", force: true end diff --git a/lib/litestack.rb b/lib/litestack.rb index 81abdd1..6bd102f 100644 --- a/lib/litestack.rb +++ b/lib/litestack.rb @@ -1,23 +1,23 @@ # frozen_string_literal: true # load core classes -require_relative "./litestack/version" -require_relative "./litestack/litescheduler" -require_relative "./litestack/litesupport" -require_relative "./litestack/litemetric" -require_relative "./litestack/litedb" -require_relative "./litestack/litecache" -require_relative "./litestack/litejob" -require_relative "./litestack/litecable" +require_relative "litestack/version" +require_relative "litestack/litescheduler" +require_relative "litestack/litesupport" +require_relative "litestack/litemetric" +require_relative "litestack/litedb" +require_relative "litestack/litecache" +require_relative "litestack/litejob" +require_relative "litestack/litecable" # conditionally load integration with other libraries -require_relative "./sequel/adapters/litedb" if defined? Sequel -require_relative "./active_record/connection_adapters/litedb_adapter" if defined? ActiveRecord -require_relative "./railties/rails/commands/dbconsole" if defined?(Rails) && defined?(ActiveRecord) -require_relative "./active_support/cache/litecache" if defined? ActiveSupport -require_relative "./active_job/queue_adapters/litejob_adapter" if defined? ActiveJob -require_relative "./action_cable/subscription_adapter/litecable" if defined? ActionCable -require_relative "./litestack/railtie" if defined? Rails::Railtie +require_relative "sequel/adapters/litedb" if defined? Sequel +require_relative "active_record/connection_adapters/litedb_adapter" if defined? ActiveRecord +require_relative "railties/rails/commands/dbconsole" if defined?(Rails) && defined?(ActiveRecord) +require_relative "active_support/cache/litecache" if defined? ActiveSupport +require_relative "active_job/queue_adapters/litejob_adapter" if defined? ActiveJob +require_relative "action_cable/subscription_adapter/litecable" if defined? ActionCable +require_relative "litestack/railtie" if defined? Rails::Railtie module Litestack class NotImplementedError < RuntimeError; end diff --git a/lib/litestack/liteboard/views/litecable.erb b/lib/litestack/liteboard/views/litecable.erb index 86fa029..2a06399 100644 --- a/lib/litestack/liteboard/views/litecable.erb +++ b/lib/litestack/liteboard/views/litecable.erb @@ -61,7 +61,7 @@
diff --git a/lib/litestack/litecable.rb b/lib/litestack/litecable.rb index c2ef0be..2b2d16e 100644 --- a/lib/litestack/litecable.rb +++ b/lib/litestack/litecable.rb @@ -113,7 +113,7 @@ def create_pruner def create_listener Litescheduler.spawn do while @running - @last_fetched_id ||= (run_stmt(:last_id)[0][0] || 0) + @last_fetched_id ||= run_stmt(:last_id)[0][0] || 0 run_stmt(:fetch, @last_fetched_id, @pid).to_a.each do |msg| @logger.info "RECEIVED #{msg}" @last_fetched_id = msg[0] diff --git a/lib/litestack/litecache.rb b/lib/litestack/litecache.rb index 325d866..046f1a4 100644 --- a/lib/litestack/litecache.rb +++ b/lib/litestack/litecache.rb @@ -20,7 +20,7 @@ class Litecache include Litemetric::Measurable # the default options for the cache - # can be overriden by passing new options in a hash + # can be overridden by passing new options in a hash # to Litecache.new # path: "./cache.db" # expiry: 60 * 60 * 24 * 30 -> one month default expiry if none is provided @@ -71,7 +71,7 @@ def initialize(options = {}) # add a key, value pair to the cache, with an optional expiry value (number of seconds) def set(key, value, expires_in = nil) key = key.to_s - expires_in ||= @expires_in + expires_in ||= @expires_in @conn.acquire do |cache| cache.stmts[:setter].execute!(key, value, expires_in) capture(:set, key) @@ -82,30 +82,28 @@ def set(key, value, expires_in = nil) end true end - + # set multiple keys and values in one shot set_multi({k1: v1, k2: v2, ... }) def set_multi(keys_and_values, expires_in = nil) - expires_in ||= @expires_in + expires_in ||= @expires_in transaction do |conn| keys_and_values.each_pair do |k, v| - begin - key = k.to_s - conn.stmts[:setter].execute!(key, v, expires_in) - capture(:set, key) - rescue SQLite3::FullException - conn.stmts[:extra_pruner].execute!(0.2) - conn.execute("vacuum") - retry - end + key = k.to_s + conn.stmts[:setter].execute!(key, v, expires_in) + capture(:set, key) + rescue SQLite3::FullException + conn.stmts[:extra_pruner].execute!(0.2) + conn.execute("vacuum") + retry end end - true - end + true + end # add a key, value pair to the cache, but only if the key doesn't exist, with an optional expiry value (number of seconds) def set_unless_exists(key, value, expires_in = nil) key = key.to_s - expires_in ||= @expires_in + expires_in ||= @expires_in changes = 0 @conn.acquire do |cache| cache.transaction(:immediate) do @@ -132,7 +130,7 @@ def get(key) capture(:get, key, 0) nil end - + # get multiple values by their keys, a hash with values corresponding to the keys # is returned, def get_multi(*keys) @@ -140,11 +138,11 @@ def get_multi(*keys) transaction(:deferred) do |conn| keys.length.times do |i| key = keys[i].to_s - if (record = conn.stmts[:getter].execute!(key)[0]) + if (record = conn.stmts[:getter].execute!(key)[0]) results[keys[i]] = record[1] # use the original key format capture(:get, key, 1) else - capture(:get, key, 0) + capture(:get, key, 0) end end end @@ -162,13 +160,13 @@ def delete(key) end # increment an integer value by amount, optionally add an expiry value (in seconds) - def increment(key, amount=1, expires_in = nil) - expires_in ||= @expires_in + def increment(key, amount = 1, expires_in = nil) + expires_in ||= @expires_in @conn.acquire { |cache| cache.stmts[:incrementer].execute!(key.to_s, amount, expires_in) } end # decrement an integer value by amount, optionally add an expiry value (in seconds) - def decrement(key, amount=1, expires_in = nil) + def decrement(key, amount = 1, expires_in = nil) increment(key, -amount, expires_in) end @@ -225,7 +223,7 @@ def snapshot end # low level access to SQLite transactions, use with caution - def transaction(mode=:immediate) + def transaction(mode = :immediate) @conn.acquire do |cache| if cache.transaction_active? yield @@ -241,7 +239,7 @@ def transaction(mode=:immediate) def setup super # create connection - @bgthread = spawn_worker # create backgroud pruner thread + @bgthread = spawn_worker # create background pruner thread end def spawn_worker @@ -253,7 +251,7 @@ def spawn_worker retry rescue SQLite3::FullException cache.stmts[:extra_pruner].execute!(0.2) - rescue Exception => e # standard:disable Lint/RescueException + rescue Exception # standard:disable Lint/RescueException # database is closed end sleep @options[:sleep_interval] diff --git a/lib/litestack/litedb.rb b/lib/litestack/litedb.rb index a7ad488..eae4823 100644 --- a/lib/litestack/litedb.rb +++ b/lib/litestack/litedb.rb @@ -15,7 +15,7 @@ class Litedb < ::SQLite3::Database # add litesearch support include Litesearch - # overrride the original initilaizer to allow for connection configuration + # override the original initilaizer to allow for connection configuration def initialize(file, options = {}, zfs = nil) if block_given? super(file, options, zfs) do |db| diff --git a/lib/litestack/litejob.rb b/lib/litestack/litejob.rb index dbc3f6e..6446587 100644 --- a/lib/litestack/litejob.rb +++ b/lib/litestack/litejob.rb @@ -1,6 +1,6 @@ # frozen_stringe_literal: true -require_relative "./litejobqueue" +require_relative "litejobqueue" ## # Litejob is a Ruby module that enables seamless integration of the Litejobqueue job queueing system into Ruby applications. By including the Litejob module in a class and implementing the #perform method, developers can easily enqueue and process jobs asynchronously. diff --git a/lib/litestack/litejobqueue.rb b/lib/litestack/litejobqueue.rb index d6bd63e..f27a652 100644 --- a/lib/litestack/litejobqueue.rb +++ b/lib/litestack/litejobqueue.rb @@ -1,7 +1,7 @@ # frozen_stringe_literal: true -require_relative "./litequeue" -require_relative "./litemetric" +require_relative "litequeue" +require_relative "litemetric" ## # Litejobqueue is a job queueing and processing system designed for Ruby applications. It is built on top of SQLite, which is an embedded relational database management system that is #lightweight and fast. @@ -15,7 +15,7 @@ class Litejobqueue < Litequeue include Litemetric::Measurable # the default options for the job queue - # can be overriden by passing new options in a hash + # can be overridden by passing new options in a hash # to Litejobqueue.new, it will also be then passed to the underlying Litequeue object # config_path: "./litejob.yml" -> were to find the configuration file (if any) # path: "./db/queue.db" diff --git a/lib/litestack/litemetric.rb b/lib/litestack/litemetric.rb index 7824a21..1fda1d6 100644 --- a/lib/litestack/litemetric.rb +++ b/lib/litestack/litemetric.rb @@ -2,7 +2,7 @@ require "singleton" -require_relative "./litesupport" +require_relative "litesupport" # this class is a singleton # and should remain so @@ -276,7 +276,7 @@ def capture(topic, event, key, value = nil, time = nil) def capture_single_key(topic, event, key, value, time = nil) run_stmt(:capture_event, topic.to_s, event.to_s, key.to_s, time, 1, value) end - + def count run_stmt(:event_count)[0][0] end diff --git a/lib/litestack/litequeue.rb b/lib/litestack/litequeue.rb index 8965b02..5a46301 100644 --- a/lib/litestack/litequeue.rb +++ b/lib/litestack/litequeue.rb @@ -13,7 +13,7 @@ class Litequeue # the default options for the queue - # can be overriden by passing new options in a hash + # can be overridden by passing new options in a hash # to Litequeue.new # path: "./queue.db" # mmap_size: 128 * 1024 * 1024 -> 128MB to be held in memory @@ -80,7 +80,7 @@ def clear(queue = nil) run_sql("DELETE FROM queue WHERE iif(?1 IS NOT NULL, name = ?1, TRUE)", queue) end - # returns a count of entries in all queues, or if a queue name is given, reutrns the count of entries in that queue + # returns a count of entries in all queues, or if a queue name is given, returns the count of entries in that queue def count(queue = nil) run_sql("SELECT count(*) FROM queue WHERE iif(?1 IS NOT NULL, name = ?1, TRUE)", queue)[0][0] end diff --git a/lib/litestack/litescheduler.rb b/lib/litestack/litescheduler.rb index 7550d4f..1dcdfbd 100644 --- a/lib/litestack/litescheduler.rb +++ b/lib/litestack/litescheduler.rb @@ -82,9 +82,8 @@ def self.mutex @@mutex ||= Mutex.new end - private - def self.fiber_backed? backend == :fiber || backend == :polyphony end + private_class_method :fiber_backed? end diff --git a/lib/litestack/litesearch.rb b/lib/litestack/litesearch.rb index eb6a55a..778dc47 100644 --- a/lib/litestack/litesearch.rb +++ b/lib/litestack/litesearch.rb @@ -4,8 +4,8 @@ class Index; end class Schema; end end -require_relative "./litesearch/index" -require_relative "./litesearch/model" +require_relative "litesearch/index" +require_relative "litesearch/model" module Litesearch def litesearch_index_cache diff --git a/lib/litestack/litesearch/index.rb b/lib/litestack/litesearch/index.rb index bfc5dbf..de04fc3 100644 --- a/lib/litestack/litesearch/index.rb +++ b/lib/litestack/litesearch/index.rb @@ -1,5 +1,5 @@ require "oj" -require_relative "./schema" +require_relative "schema" class Litesearch::Index DEFAULT_SEARCH_OPTIONS = {limit: 25, offset: 0} @@ -71,7 +71,7 @@ def modify def rebuild! if @db.transaction_active? - do_rebuild + do_rebuild else @db.transaction(:immediate) { do_rebuild } end @@ -102,19 +102,20 @@ def search(term, options = {}) rs = @stmts[:search].execute(term, options[:limit], options[:offset]) generate_results(rs) end - - def similar(id, limit=10) + + def similar(id, limit = 10) # pp term = @db.execute(@schema.sql_for(:similarity_query), id) - if @schema.schema[:tokenizer] == :trigram + rs = if @schema.schema[:tokenizer] == :trigram # just use the normal similarity approach for now # need to recondisder that for trigram indexes later - rs = @stmts[:similar].execute(id, limit) + @stmts[:similar].execute(id, limit) # standard:disable Style/IdenticalConditionalBranches else - rs = @stmts[:similar].execute(id, limit) + @stmts[:similar].execute(id, limit) # standard:disable Style/IdenticalConditionalBranches end + generate_results(rs) end - + def clear! @stmts[:delete_all].execute!(id) end @@ -140,7 +141,7 @@ def generate_results(rs) else result = rs.to_a end - result + result end def exists?(name) @@ -175,7 +176,7 @@ def do_create(schema) def do_modify(new_schema) changes = @schema.compare(new_schema) - # ensure the new schema maintains feild order + # ensure the new schema maintains field order new_schema.order_fields(@schema) # with the changes object decide what needs to be done to the schema requires_schema_change = false diff --git a/lib/litestack/litesearch/model.rb b/lib/litestack/litesearch/model.rb index 0221795..4e4e79d 100644 --- a/lib/litestack/litesearch/model.rb +++ b/lib/litestack/litesearch/model.rb @@ -21,7 +21,7 @@ def search_models end module InstanceMethods - def similar(limit=10) + def similar(limit = 10) conn = self.class.get_connection idx = conn.search_index(self.class.send(:index_name)) r_a_h = conn.results_as_hash @@ -36,7 +36,6 @@ def similar(limit=10) end result end - end module ClassMethods @@ -77,7 +76,7 @@ def search_all(term, options = {}) else models_hash = search_models end - # remove the models from the options hash before passing it ot the query + # remove the models from the options hash before passing it to the query options.delete(:models) models_hash.each do |name, klass| selects << "SELECT '#{name}' AS model, rowid, -rank AS search_rank FROM #{index_name_for_table(klass.table_name)}(:term)" diff --git a/lib/litestack/litesearch/schema.rb b/lib/litestack/litesearch/schema.rb index c73b006..dcc8078 100644 --- a/lib/litestack/litesearch/schema.rb +++ b/lib/litestack/litesearch/schema.rb @@ -1,4 +1,4 @@ -require_relative "./schema_adapters" +require_relative "schema_adapters" class Litesearch::Schema TOKENIZERS = { diff --git a/lib/litestack/litesearch/schema_adapters.rb b/lib/litestack/litesearch/schema_adapters.rb index e9afc6b..6262de3 100644 --- a/lib/litestack/litesearch/schema_adapters.rb +++ b/lib/litestack/litesearch/schema_adapters.rb @@ -1,4 +1,4 @@ -require_relative "./schema_adapters/basic_adapter" -require_relative "./schema_adapters/standalone_adapter" -require_relative "./schema_adapters/contentless_adapter" -require_relative "./schema_adapters/backed_adapter" +require_relative "schema_adapters/basic_adapter" +require_relative "schema_adapters/standalone_adapter" +require_relative "schema_adapters/contentless_adapter" +require_relative "schema_adapters/backed_adapter" diff --git a/lib/litestack/litesearch/schema_adapters/backed_adapter.rb b/lib/litestack/litesearch/schema_adapters/backed_adapter.rb index 56deabe..2fe24a5 100644 --- a/lib/litestack/litesearch/schema_adapters/backed_adapter.rb +++ b/lib/litestack/litesearch/schema_adapters/backed_adapter.rb @@ -92,7 +92,7 @@ def enrich_schema if field[:target] && !field[:target].start_with?("#{table}.") field[:target] = field[:target].downcase target_table, target_col = field[:target].split(".") - field[:col] = "#{name}_id".to_sym unless field[:col] + field[:col] = :"#{name}_id" unless field[:col] field[:target_table] = target_table.to_sym field[:target_col] = target_col.to_sym field[:sql] = "(SELECT #{field[:target_col]} FROM #{field[:target_table]} WHERE id = NEW.#{field[:col]})" diff --git a/lib/litestack/litesupport.rb b/lib/litestack/litesupport.rb index 3d91b2d..0405e72 100644 --- a/lib/litestack/litesupport.rb +++ b/lib/litestack/litesupport.rb @@ -8,7 +8,7 @@ require "fileutils" require "erb" -require_relative "./litescheduler" +require_relative "litescheduler" module Litesupport class Error < StandardError; end @@ -175,7 +175,7 @@ def init(options = {}) end def configure(options = {}) - # detect enviornment (production, development, etc.) + # detect environment (production, development, etc.) defaults = begin self.class::DEFAULT_OPTIONS rescue @@ -187,7 +187,7 @@ def configure(options = {}) rescue {} end # an empty hash won't hurt - config = config[Litesupport.environment] if config[Litesupport.environment] # if there is a config for the current enviornment defined then use it, otherwise use the top level declaration + config = config[Litesupport.environment] if config[Litesupport.environment] # if there is a config for the current environment defined then use it, otherwise use the top level declaration config.keys.each do |k| # symbolize keys config[k.to_sym] = config[k] config.delete k @@ -232,7 +232,7 @@ def run_stmt_method(stmt, method, *args) end def create_pooled_connection(count = 1) - count = 1 unless count and count.is_a? Integer + count = 1 unless count&.is_a?(Integer) Litesupport::Pool.new(count) { create_connection } end diff --git a/lib/sequel/adapters/shared/litedb.rb b/lib/sequel/adapters/shared/litedb.rb index 7cba2e9..c0d58ea 100644 --- a/lib/sequel/adapters/shared/litedb.rb +++ b/lib/sequel/adapters/shared/litedb.rb @@ -715,7 +715,7 @@ def select(*cols) # Handle uniqueness violations when inserting, by using a specified # resolution algorithm. With no options, uses INSERT OR REPLACE. SQLite - # supports the following conflict resolution algoriths: ROLLBACK, ABORT, + # supports the following conflict resolution algorithms: ROLLBACK, ABORT, # FAIL, IGNORE and REPLACE. # # On SQLite 3.24.0+, you can pass a hash to use an ON CONFLICT clause. @@ -877,7 +877,7 @@ def _returning_values(values) end # Use from_self for aggregate dataset using VALUES. - def aggreate_dataset_use_from_self? + def aggregate_dataset_use_from_self? super || @opts[:values] end diff --git a/litestack.gemspec b/litestack.gemspec index 2976ecc..aed6152 100644 --- a/litestack.gemspec +++ b/litestack.gemspec @@ -36,13 +36,13 @@ Gem::Specification.new do |spec| spec.add_dependency "rackup" spec.add_dependency "tilt" spec.add_dependency "erubi" - - spec.add_development_dependency "activerecord" + spec.add_development_dependency "rake" + spec.add_development_dependency "activerecord" spec.add_development_dependency "railties" spec.add_development_dependency "minitest" - spec.add_development_dependency "rack", "~> 3.0" spec.add_development_dependency "simplecov" spec.add_development_dependency "standard" spec.add_development_dependency "sequel" + spec.add_development_dependency "debug" end diff --git a/scripts/build_metrics.rb b/scripts/build_metrics.rb index 6c7c291..1d56ab7 100644 --- a/scripts/build_metrics.rb +++ b/scripts/build_metrics.rb @@ -17,14 +17,14 @@ log: nil }) -$time = Time.now.to_i #- 10800 +$time = Time.now.to_i # - 10800 $start_time = Time.now.to_i class NormalJob include Litejob self.queue = "normal" def perform(time) - $time = time #-= (rand * 100).to_i #10 seconds in the past + $time = time # -= (rand * 100).to_i #10 seconds in the past sleep 0.001 warn "performing some normal action" end diff --git a/scripts/test_cable.rb b/scripts/test_cable.rb index 970e2bb..0a8ec89 100644 --- a/scripts/test_cable.rb +++ b/scripts/test_cable.rb @@ -11,7 +11,7 @@ def initialize(channel) end def call(*args) - warn "[#{Process.pid}]:#{object_id} recieved #{args} from #{@channel}" + warn "[#{Process.pid}]:#{object_id} received #{args} from #{@channel}" end end diff --git a/test/test_helper.rb b/test/helper.rb similarity index 100% rename from test/test_helper.rb rename to test/helper.rb diff --git a/test/test_ar_search.rb b/test/test_ar_search.rb index b259b85..f1ac6b8 100644 --- a/test/test_ar_search.rb +++ b/test/test_ar_search.rb @@ -1,7 +1,7 @@ require "minitest/autorun" require_relative "../lib/litestack/litedb" -require_relative '../lib/active_record/connection_adapters/litedb_adapter' +require_relative "../lib/active_record/connection_adapters/litedb_adapter" require "active_record" @@ -83,9 +83,9 @@ def setup Book.create(title: "In a middle of a night", description: "A tale of sleep", published_on: "2008-10-01", state: "available", active: true, publisher_id: 1, author_id: 1) Book.create(title: "In a start of a night", description: "A tale of watching TV", published_on: "2006-08-08", state: "available", active: false, publisher_id: 2, author_id: 1) end - + def test_similar - newbook = Book.create(title: "A night", description: "A tale of watching TV", published_on: "2006-08-08", state: "available", active: true, publisher_id: 2, author_id: 2) + newbook = Book.create(title: "A night", description: "A tale of watching TV", published_on: "2006-08-08", state: "available", active: true, publisher_id: 2, author_id: 2) book = Book.find 1 books = book.similar assert_equal 1, books.length diff --git a/test/test_backed_index.rb b/test/test_backed_index.rb index 258082f..784bdd3 100644 --- a/test/test_backed_index.rb +++ b/test/test_backed_index.rb @@ -204,7 +204,7 @@ def test_update_schema_add_field_remove_another_and_rebuild end assert_equal @idx.search("computer").length, 0 assert_raises do - @idx.search("reciever: sarah") + @idx.search("receiver: sarah") end @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body, urgency) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer', 'high')") assert_equal @idx.search("computer").length, 1 @@ -260,19 +260,19 @@ def test_adding_fields_with_zero_weight end def test_update_schema_change_tokenizer_auto_rebuild - @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") - @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") - assert_equal @idx.search('computer').length, 2 - @idx.modify do |schema| - schema.fields [:body] - schema.field :sender, {target: "person.name"} - schema.field :receiver, {target: "person.name"} - schema.field :subject, {weight: 10} - schema.tokenizer :trigram - schema.rebuild_on_modify true - end - assert_equal @idx.search('puter').length, 2 - @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") - assert_equal @idx.search('puter').length, 3 + @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") + @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") + assert_equal @idx.search("computer").length, 2 + @idx.modify do |schema| + schema.fields [:body] + schema.field :sender, {target: "person.name"} + schema.field :receiver, {target: "person.name"} + schema.field :subject, {weight: 10} + schema.tokenizer :trigram + schema.rebuild_on_modify true + end + assert_equal @idx.search("puter").length, 2 + @db.execute("INSERT INTO email(sender_id, receiver_id, subject, body) VALUES (1, 2, 'How are the girls?', 'I wanted to ask about the girls and the computer')") + assert_equal @idx.search("puter").length, 3 end end diff --git a/test/test_cache.rb b/test/test_cache.rb index 830939c..83f1ea4 100644 --- a/test/test_cache.rb +++ b/test/test_cache.rb @@ -6,7 +6,7 @@ def setup @cache = Litecache.new({path: ":memory:", sleep_interval: 1}) @cache.clear end - + def test_caceh_set @cache.set("key", "value") assert_equal "value", @cache.get("key") @@ -20,7 +20,7 @@ def test_caceh_set_unless_exists @cache.set_unless_exists("key", "new_value") assert_equal "value", @cache.get("key") end - + def test_cache_set_multi data = {k1: "v1", k2: "v2", k3: "v3"} @cache.set_multi(data) @@ -31,7 +31,7 @@ def test_cache_set_multi def test_cache_get_multi data = {k1: "v1", k2: "v2", k3: "v3"} - data.each_pair{|k, v| @cache.set(k, v) } + data.each_pair { |k, v| @cache.set(k, v) } results = @cache.get_multi(*data.keys) assert_equal data, results end @@ -40,25 +40,23 @@ def test_cache_expiry @cache.set("key", "value", 1) assert_equal "value", @cache.get("key") sleep 1.1 - assert_nil @cache.get("key") + assert_nil @cache.get("key") end - + def test_increment_decrement - @cache.increment("key") + @cache.increment("key") assert_equal 1, @cache.get("key") - @cache.increment("key", 5) + @cache.increment("key", 5) assert_equal 6, @cache.get("key") - @cache.decrement("key", 4) + @cache.decrement("key", 4) assert_equal 2, @cache.get("key") end def test_increment_decrement_expiry - @cache.increment("key", 2, 1) + @cache.increment("key", 2, 1) assert_equal 2, @cache.get("key") sleep 1.1 - @cache.increment("key", 5) + @cache.increment("key", 5) assert_equal 5, @cache.get("key") end - end - diff --git a/test/test_cache_rails.rb b/test/test_cache_rails.rb index 72e1d0a..d20bbae 100644 --- a/test/test_cache_rails.rb +++ b/test/test_cache_rails.rb @@ -1,7 +1,7 @@ require "minitest/autorun" require_relative "../lib/active_support/cache/litecache" -class TestCache < Minitest::Test +class TestCacheRails < Minitest::Test def setup @cache = ActiveSupport::Cache::Litecache.new({path: ":memory:", sleep_interval: 1}) @cache.clear @@ -15,9 +15,9 @@ def test_caceh_write end def test_cache_fetch - result = @cache.fetch("key") {"value"} + result = @cache.fetch("key") { "value" } assert_equal "value", result - result = @cache.fetch("key") {"new_value"} + result = @cache.fetch("key") { "new_value" } assert_equal "value", result end @@ -31,7 +31,7 @@ def test_cache_write_multi def test_cache_read_multi data = {k1: "v1", k2: "v2", k3: "v3"} - data.each_pair{|k, v| @cache.write(k, v) } + data.each_pair { |k, v| @cache.write(k, v) } results = @cache.read_multi(*data.keys) assert_equal data, results end @@ -47,25 +47,23 @@ def test_cache_expiry @cache.write("key", "value", expires_at: 1.second.from_now) assert_equal "value", @cache.read("key") sleep 1.1 - assert_nil @cache.read("key") + assert_nil @cache.read("key") end - + def test_increment_decrement - @cache.increment("key") + @cache.increment("key") assert_equal 1, @cache.read("key") - @cache.increment("key", 5) + @cache.increment("key", 5) assert_equal 6, @cache.read("key") - @cache.decrement("key", 4) + @cache.decrement("key", 4) assert_equal 2, @cache.read("key") end - + def test_increment_decrement_expiry - @cache.increment("key", 2, expires_at: 1.second.from_now) + @cache.increment("key", 2, expires_at: 1.second.from_now) assert_equal 2, @cache.read("key") sleep 1.1 - @cache.increment("key", 5) + @cache.increment("key", 5) assert_equal 5, @cache.read("key") end - end - diff --git a/test/test_contentless_index.rb b/test/test_contentless_index.rb index c555e1f..e360ebb 100644 --- a/test/test_contentless_index.rb +++ b/test/test_contentless_index.rb @@ -24,7 +24,6 @@ def test_similar assert_equal 4, rs[1]["id"] end - def test_search rs = @idx.search("Hamada") assert_equal rs.length, 1 diff --git a/test/test_jobqueue.rb b/test/test_jobqueue.rb index b551bcf..eae0cdf 100644 --- a/test/test_jobqueue.rb +++ b/test/test_jobqueue.rb @@ -12,13 +12,16 @@ class MyJob def perform(time) # puts "performing" - raise "An error occured" if Time.now.to_i < time + raise "An error occurred" if Time.now.to_i < time end end class TestJobQueue < Minitest::Test def setup @jobqueue = Litejobqueue.new({path: ":memory:", logger: nil, retries: 2, retry_delay: 1, retry_delay_multiplier: 1, queues: [["test", 1]]}) + end + + def teardown @jobqueue.clear end @@ -36,7 +39,6 @@ def test_delete @jobqueue.count @jobqueue.delete(id[0]) assert @jobqueue.count == 0 - @jobqueue.clear end def test_push_with_delay diff --git a/test/test_litejob.rb b/test/test_litejob.rb index 3fa0799..9acee4f 100644 --- a/test/test_litejob.rb +++ b/test/test_litejob.rb @@ -1,6 +1,6 @@ # frozen_string_literal: true -require_relative "./test_helper" +require_relative "helper" require_relative "../lib/litestack/litejob" class NoOpJob diff --git a/test/test_litescheduler.rb b/test/test_litescheduler.rb index 8b562d9..59c8f19 100644 --- a/test/test_litescheduler.rb +++ b/test/test_litescheduler.rb @@ -1,11 +1,10 @@ # frozen_string_literal: true -require_relative "./test_helper" +require_relative "helper" require_relative "../lib/litestack/litescheduler" describe Litescheduler do describe "#backend" do - before do Litescheduler.instance_variable_set(:@backend, nil) end diff --git a/test/test_sequel_search.rb b/test/test_sequel_search.rb index c7cc971..5eb35ca 100644 --- a/test/test_sequel_search.rb +++ b/test/test_sequel_search.rb @@ -79,7 +79,7 @@ def setup end def test_similar - newbook = Books.create(title: "A night", description: "A tale of watching TV", published_on: "2006-08-08", state: "available", active: true, publisher_id: 2, author_id: 2) + newbook = Books.create(title: "A night", description: "A tale of watching TV", published_on: "2006-08-08", state: "available", active: true, publisher_id: 2, author_id: 2) book = Books[1] books = book.similar assert_equal 1, books.length diff --git a/test/test_standlone_index.rb b/test/test_standalone_index.rb similarity index 99% rename from test/test_standlone_index.rb rename to test/test_standalone_index.rb index cb47a9f..ab95f72 100644 --- a/test/test_standlone_index.rb +++ b/test/test_standalone_index.rb @@ -176,5 +176,4 @@ def test_update_schema_change_tokenizer_auto_rebuild @idx.add({sender: "Kamal", receiver: "Layla", subject: "How are the girls?", body: "I wanted to ask how are the girls doing with the computer?"}) assert_equal @idx.search("puter").length, 3 end - end