Releases · jbsouthe/appd-etl-export-tool

30 Jan 21:41

jbsouthe

release-v1.14

77b0386

bug fixs and security updates Latest

Latest

some bugs were fixed in casting overflow on really large requests for data
moved from apache digester to jaxp for config file parsing (security scans were hitting on the old digester required libs)

Please let me know in an issue if any problems are found, the latest docker image is updated as well to v1.14

Assets 4

07 Feb 22:51

jbsouthe

release-v1.12

fcd40aa

Now with a max duration of data to query configuration

The default settings are shown:

<Scheduler enabled="false">
    <PollIntervalMinutes>60</PollIntervalMinutes>
    <FirstRunHistoricNumberOfDays>2</FirstRunHistoricNumberOfDays>
    **<MaxNumberOfDaysToQueryAtATime>14</MaxNumberOfDaysToQueryAtATime>**
    <ControllerThreads>50</ControllerThreads>
    <DatabaseThreads>50</DatabaseThreads>
    <ConfigurationRefreshEveryHours>12</ConfigurationRefreshEveryHours>
</Scheduler>

enabled=false causes the Scheduler to exit after one run, true is the default, when this option is missing.
PollIntervalMinutes 60, if enabled, causes the Scheduler to sleep for 60 minutes and run again, continuously
FirstRunHistoricNumberOfDays 2, causes the extract to default in pulling the last 48 hours of data for a given query, if no previous run is detected in the control table, specified in the database section later
MaxNumberOfDaysToQueryAtATime 14, forces the run to only pull a maximum of 14 days of data at a time, this means it will always have work to do grabbing 14 days of data each scheduled interval until caught up.
ControllerThreads 50, the number of threads to run pulling data from the controllers concurrently
DatabaseThreads 50, the number of worker threads watching the data queue to insert into the database
ConfigurationRefreshEveryHours 12, after this many hours, all the applications with the configuration setting to pull all metrics, will refresh the metrics in case new ones are registered since start.

Assets 5

06 Feb 18:11

jbsouthe

release-v1.11

3f24b6b

Now with a command parser and data management

Added data managmenet and a command parser to the utility.

Assets 3

17 Jan 21:24

jbsouthe

release-v1.10

c721fa0

Data Consistence Assurance Release

Many subtle changes to help ensure that the data exported is in line with exactly what a user would get from our analytics queries:

Set a delay timestamp on the end time for a query to -5 minutes so we can allow agents->controller->data store time to sync really slow data ingestion
Retry a failed data load later in the scheduler, rather than reporting the error an abandoning the entire load
Added a wire trace debug option to print the HTTP request and response entirely
Printing GC configuration for some troubleshooting and tuning in the future, kubernetes is non intuitive in jvm ergonomics (thank you Bruno Borges)
Updated README to include recommended java options for kubernetes deployment.

We have a couple of open issues that will begin to be worked for the next release, please keep the feedback coming.
Thanks, John

Assets 3

06 Jan 17:31

jbsouthe

release-v1.9

d674352

v1.9 scroll on large data export

This release adds the scroll fallback when analytics data is truncated, and fixed an encoding issue when special spanish characters are used in the analytics query. Also some optimizations in the analytics data load to the database. No longer downloading all data before starting database inserts.

As always, please create issues for anything that needs attention. Many people are using this now, and I may start refactoring the database sections to be easier to maintain. I wasn't sure what this would look like when I started, and I now regret some of that code.
Thanks, John

Assets 3