-
Notifications
You must be signed in to change notification settings - Fork 186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix jdbc_fetch_size usage with postgresql (#103) #200
base: main
Are you sure you want to change the base?
Changes from 2 commits
e6de88e
f66b2f4
ef38c77
7c5265c
30bed65
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -143,6 +143,7 @@ def prepare_jdbc_connection | |
require "java" | ||
require "sequel" | ||
require "sequel/adapters/jdbc" | ||
require "sequel/adapters/jdbc/transactions" | ||
load_drivers(@jdbc_driver_library.split(",")) if @jdbc_driver_library | ||
|
||
begin | ||
|
@@ -157,6 +158,7 @@ def prepare_jdbc_connection | |
raise LogStash::ConfigurationError, "#{e}. #{message}" | ||
end | ||
@database = jdbc_connect() | ||
@database.extend(Sequel::JDBC::Transactions) | ||
@database.extension(:pagination) | ||
if @jdbc_default_timezone | ||
@database.extension(:named_timezones) | ||
|
@@ -205,26 +207,29 @@ def execute_statement(statement, parameters) | |
query = @database[statement, parameters] | ||
sql_last_value = @use_column_value ? @sql_last_value : Time.now.utc | ||
@tracking_column_warning_sent = false | ||
@logger.debug? and @logger.debug("Executing JDBC query", :statement => statement, :parameters => parameters, :count => query.count) | ||
@logger.debug? and @logger.debug("Executing JDBC query", :statement => statement, :parameters => parameters) | ||
|
||
if @jdbc_paging_enabled | ||
query.each_page(@jdbc_page_size) do |paged_dataset| | ||
paged_dataset.each do |row| | ||
@database.transaction do | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for describing this change in the PR. We really need this documented here as well. Do you know how this will play with other databases? I'm not anticipating problems ATM, but I'm curious what your thoughts are. I'm wondering if this codepath should be PG only. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think its ok with other dbs cause used standard JDBC transaction mechanic with Sequel::JDBC::Transactions, and all dbs that i knew uses transaction mechanic. I can rewrite code for use only with PG driver if u wish. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @izhigalko no, I think that makes sense. I do think it needs a comment to explain why we wrap with a transaction then rollback for seemingly no reason however. If you put that comment in we can move forward :) |
||
if @jdbc_paging_enabled | ||
query.each_page(@jdbc_page_size) do |paged_dataset| | ||
paged_dataset.each do |row| | ||
sql_last_value = get_column_value(row) if @use_column_value | ||
if @tracking_column_type=="timestamp" and @use_column_value and sql_last_value.is_a?(DateTime) | ||
sql_last_value=Time.parse(sql_last_value.to_s) # Coerce the timestamp to a `Time` | ||
end | ||
yield extract_values_from(row) | ||
end | ||
end | ||
else | ||
query.each do |row| | ||
sql_last_value = get_column_value(row) if @use_column_value | ||
if @tracking_column_type=="timestamp" and @use_column_value and sql_last_value.is_a?(DateTime) | ||
sql_last_value=Time.parse(sql_last_value.to_s) # Coerce the timestamp to a `Time` | ||
end | ||
yield extract_values_from(row) | ||
end | ||
end | ||
else | ||
query.each do |row| | ||
sql_last_value = get_column_value(row) if @use_column_value | ||
if @tracking_column_type=="timestamp" and @use_column_value and sql_last_value.is_a?(DateTime) | ||
sql_last_value=Time.parse(sql_last_value.to_s) # Coerce the timestamp to a `Time` | ||
end | ||
yield extract_values_from(row) | ||
end | ||
raise Sequel::Rollback | ||
end | ||
success = true | ||
rescue Sequel::DatabaseConnectionError, Sequel::DatabaseError => e | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was
:count
removed? Did it execute an extraCOUNT
or was there some other reason?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Count removed cause it slow down huge queries execution when debug enabled. My query have more than 5 subqueries in fields and i wait count execution about 5 hours before remove it.