Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins looses connection to SQS queues #4

Open
ayashjorden opened this issue Oct 8, 2017 · 21 comments
Open

Jenkins looses connection to SQS queues #4

ayashjorden opened this issue Oct 8, 2017 · 21 comments
Labels

Comments

@ayashjorden
Copy link

Hi,
First, this plugin is great! helped me build a cool automated flow for handling event.

Now, for my issue,
My Jenkins setup is composed of a master that spawns slaves ontop of DC/OS (using Marathon).
I have three jobs that are configured to trigger based on events flowing from three queues.

Today is my second time that I discovered that Jenkins stopped triggering jobs from the queues. I've verified the AWS side flow of events to be working.
The first time it happened was two months ago (IMMSR).

Looked at Jenkins logs, its a mess, didn't find anything.

Is there someone who experienced that?
If I want to setup a monitor for that, what should I look for in the logs?

This plugin is part of our critical path event handling and I would like to know that I can count on it.

I'd be happy to provide more information, just ask 👍
Yarden

@ayashjorden
Copy link
Author

Hi @nickgrealy
Can you please point me to the right person (if not you)?

Thank you in advance,
Yarden

@lifeofguenter
Copy link

@ayashjorden can you replicate this on a vanilla installation? I did experience issues with long running tasks and jenkins, but that was due to faulty/interfering plugins.

@mvk
Copy link
Collaborator

mvk commented Nov 22, 2017

@ayashjorden The current mainline of the plugin is using relatively old AWS SDK.
I'm trying to merge a couple of new features, and with them, as the cherry on top we make some dependencies update to be more recent.

If @nickgrealy has the time to release the following 2-3 weeks will render some good changes,
And hopefully remedy the problem you're experiencing.

Are you using ALL functionality - SQS, SNS, and CodeCommit, or just SQS part ?

I've been using SQS part solely, and the only problems it really had were things related to easier automation of configuration.

@mvk mvk added the wontfix label Nov 28, 2017
@ayashjorden
Copy link
Author

Hello @mvk,
I'm also using the SQS feature only.
In my setup, I've created 3 triggering jobs, one per SQS queue that it consumes.

Several times I saw the an email that corresponds to an event in SQS, but it didn't get to Jenkins.
Also, no errors on the Jenkins side that relates to SQS plugin.

@ayashjorden
Copy link
Author

ayashjorden commented Jan 15, 2018

Hello @mvk @nickgrealy,
I've upgraded our Jenkins server and all plugins, SQS among them.
The connection last for the past two weeks.

On January 14th 2018, 05:18:14.000 UTC, the SQS plugin lost its connection:

SEVERE: 000048 Unknown error, monitor for queue QUEUE_NAME_HERE (sqs.us-west-2.amazonaws.com) {ce53fffb-ed82-4ec7-9f33-9f9fc699d2c2} stopped
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sqs.us-west-2.amazonaws.com:443 [sqs.us-west-2.amazonaws.com/52.94.210.243] failed: connect timed out
    at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1447)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1989)
    at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2013)

We have AWS related job running all the time, so its not connectivity issues.

Strange....

@ayashjorden
Copy link
Author

@lifeofguenter see ^^^
After two weeks Jenkins started experiencing Timeouts

@mvk
Copy link
Collaborator

mvk commented Jan 15, 2018

@ayashjorden just to make sure, can you maybe print the list of your plugins with:

  • name
  • version

I am very very reluctant to debug it right now :-)))))
I want to look and maybe see a problematic plugin....

@ayashjorden
Copy link
Author

Name  ↓ Version
Amazon EC2 plugin 1.38
Amazon Web Services SDK 1.11.248
This plugin provides AWS SDK for Java for other plugins.
AnsiColor 0.5.2
Adds ANSI coloring to the Console Output
Ant Plugin 1.7
Adds Apache Ant support to Jenkins
Apache HttpComponents Client 4.x API Plugin 4.5.3-2.0
Bundles Apache HttpComponents Client 4.x and allows it to be used by Jenkins plugins.
Artifactory Plugin 2.14.0
This plugin allows deploying maven artifacts and build info to Artifactory.
Async Http Client 1.9.40.0
This plugin provides a shared dependency on the async-http-client library so that other plugins can co-operate when using this library.
Authentication Tokens API Plugin 1.3
This plugin provides an API for converting credentials into authentication tokens in Jenkins.
Autofavorite for Blue Ocean 1.2.1
Automatically favorites multibranch pipeline jobs when user is the author
AWS SQS Build Trigger Plugin 2.0.1
This plugin triggers builds on all events received via Amazon Web Services Simple Queue Service (AWS SQS) on a specified Queue.
Azure Commons Plugin 0.2.4
Common APIs for Azure related Jenkins plugins
Azure Credentials 1.4.0
Manage Azure Service Principal credentials using Jenkins Credentials API
Azure Slave Plugin 0.3.4
Azure plugin to provision and deprovision slaves
Azure VM Agents 0.6.0
Provisions agents on Azure cloud
Bitbucket Branch Source Plugin 2.2.8
Allows to use Bitbucket Cloud as a source for multi-branch projects. It also provides the required connectors for Bitbucket Cloud Team Folder (also known as repositories auto-discovering).
Bitbucket Pipeline for Blue Ocean 1.3.5
BlueOcean Bitbucket pipeline creator
Blue Ocean 1.3.5
BlueOcean Aggregator
Blue Ocean Pipeline Editor 1.3.5
The Blue Ocean Pipeline Editor is the simplest way for anyone wanting to get started with creating Pipelines in Jenkins
bouncycastle API Plugin 2.16.2
This plugin provides an stable API to Bouncy Castle related tasks.
Box UK - JSLint 0.8.2
 
JSLint plugin that allows you to run a sensible subset of jslint rules against your project. You can include and exclude files and specify an output file.
 
Outputs in the CheckStyle format so you can integrate the JSLint results right into your build! You should probably use the Checkstyle plugin to include the jslint results in your Jenkins project,
 
Produced by Box UK
 
Branch API Plugin 2.0.18
This plugin provides an API for multiple branch based projects.
Build Authorization Token Root Plugin 1.4
Lets build and related REST build triggers be accessed even when anonymous users cannot see Jenkins.
Build Timeout 1.19
This plugin allows builds to be automatically terminated after the specified amount of time has elapsed.
build-name-setter 1.6.7
This plug-in sets the display name of a build to something other than #1, #2, #3, ...
Cloud Statistics Plugin 0.16
Report Jenkins dynamic provisioning activities and statistics
CloudBees Amazon Web Services Credentials Plugin 1.23
CloudBees Docker Build and Publish plugin 1.3.2
This plugin enables building Dockerfile based projects, as well as publishing of the built images/repos to the docker registry.
Command Agent Launcher Plugin 1.2
Allows agents to be launched using a specified command.
Common API for Blue Ocean 1.3.5
This plugin is a part of Blue Ocean UI
Conditional BuildStep 1.3.6
A buildstep wrapping any number of other buildsteps, controlling their execution based on a defined condition (e.g. BuildParameter).
Config API for Blue Ocean 1.3.5
BlueOcean Analytics Tools plugin
Config File Provider Plugin 2.16.4
Ability to provide configuration files (e.g. settings.xml for maven, XML, groovy, custom files,...) loaded through the UI which will be copied to the job workspace.
Copy Artifact Plugin 1.39
Adds a build step to copy artifacts from another project.
Credentials Binding Plugin 1.13
Allows credentials to be bound to environment variables for use from miscellaneous build steps.
Credentials Plugin 2.1.16
This plugin allows you to store credentials in Jenkins.
CVS Plug-in 2.13
Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient.
Dashboard for Blue Ocean 1.3.5
Blue Ocean Dashboard
Display URL API 2.2.0
Provides the DisplayURLProvider extension point to provide alternate URLs for use in notifications
Display URL for Blue Ocean 2.2.0
This plugin generates BlueOcean specific URLs for the Display URL plugin.
Docker Commons Plugin 1.11
Provides the common shared functionality for various Docker-related plugins.
Docker Pipeline 1.14
Build and use Docker containers from pipelines.
Durable Task Plugin 1.17
Library offering an extension point for processes which can run outside of Jenkins yet be monitored.
Dynamic Parameter Plug-in 0.2.0
This plugin allows build parameters with dynamically generated default values.
Email Ext Recipients Column Plugin 1
This plugin is a sample to explain how to write a Jenkins plugin.
Email Extension Plugin 2.61
This plugin is a replacement for Jenkins's email publisher. It allows to configure every aspect of email notifications: when an email is sent, who should receive it and what the email says
embeddable-build-status 1.9
This plugin adds the embeddable build status badge to Jenkins so that you can easily hyperlink/show your build status from elsewhere.
Environment Injector Plugin 2
This plugin makes it possible to set an environment for the builds.
Events API for Blue Ocean 1.3.5
Blue Ocean Events
Extensible Choice Parameter plugin 1.4.1
Extensible Choice Parameter provides several ways to retrieve choices, including the way to share choices among all jobs.
External Monitor Job Type Plugin 1.7
Adds the ability to monitor the result of externally executed jobs
Favorite 2.3.1
This plugin allows users to favorite a job.
Folders Plugin 6.3
This plugin allows users to create "folders" to organize jobs. Users can define custom taxonomies (like by project type, organization type etc). Folders are nestable and you can define views within folders. Maintained by CloudBees, Inc.
Gerrit Trigger 2.27.1
This plugin integrates with Gerrit code review.
Git client plugin 2.7.0
Utility plugin for Git support in Jenkins
Git Pipeline for Blue Ocean 1.3.5
BlueOcean Git SCM pipeline creator
Git plugin 3.7.0
This plugin integrates Git with Jenkins.
GIT server Plugin 1.7
Allows Jenkins to act as a Git server.
GitHub API Plugin 1.9
This plugin provides GitHub API for other plugins.
GitHub Branch Source Plugin 2.3.2
Multibranch projects and organization folders from GitHub. Maintained by CloudBees, Inc.
GitHub Organization Folder Plugin 1.6
The functionality that was provided by this plugin has been moved to the GitHub Branch Source plugin. You can safely delete this plugin once there are no plugins that depend on this plugin installed.
GitHub Pipeline for Blue Ocean 1.3.5
BlueOcean GitHub organization pipeline creator
GitHub plugin 1.28.1
This plugin integrates GitHub to Jenkins.
GitHub Pull Request Builder 1.39.0
GitLab Plugin 1.5.2
This plugin integrates GitLab to Jenkins by faking a GitLab CI Server.
Gradle Plugin 1.28
This plugin allows Jenkins to invoke Gradle build scripts directly.
Green Balls 1.15
Because green is better than blue! For color blind support configure user property.
HTML Publisher plugin 1.14
This plugin publishes HTML reports.
Hubot Pipeline Steps 1.1.0
Hubot Pipeline Steps
i18n for Blue Ocean 1.3.5
Blue Ocean Internationalization (i18n) Plugin. This plugin is a part of the Blue Ocean Plugin set.
Icon Shim Plugin 2.0.3
Allows plugins make full use of the <l:icon> layout tag when running on newer versions of Jenkins, while still being compatible with older versions.
Ivy Plugin 1.28
This plugin allows Jenkins to automatically configure project dependencies using the Ivy dependency management system.
Jackson 2 API Plugin 2.8.10.1
This plugin exposes the Jackson 2 JSON APIs to other Jenkins plugins.
Javadoc Plugin 1.4
JavaScript GUI Lib: ACE Editor bundle plugin 1.1
JavaScript GUI Lib: ACE Editor bundle plugin.
JavaScript GUI Lib: Handlebars bundle plugin 1.1.1
JavaScript GUI Lib: Handlebars bundle plugin.
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin 1.2.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin.
JavaScript GUI Lib: Moment.js bundle plugin 1.1.1
JavaScript GUI Lib: Moment.js bundle plugin.
JIRA Integration for Blue Ocean 1.3.5
JIRA plugin 2.5
This plugin integrates Jenkins to Atlassian JIRA.
jira-ext Plugin 0.7
A plugin for Jenkins CI to update JIRA tickets in an extensible way: both what to update and how to up date are exposed as Extension Points
Job Configuration History Plugin 2.18
Job history plugin for Jenkins.
Job DSL 1.61
This plugin allows Jobs and Views to be defined via DSLs
jQuery plugin 1.12.4-0
This plugin provides an stable version of jQuery Javascript Library
JSch dependency plugin 0.1.54.1
Jenkins plugin that brings the JSch library as a plugin dependency, and provides an SSHAuthenticatorFactory for using JSch with the ssh-credentials plugin.
JUnit Plugin 1.23
Allows JUnit-format test results to be published.
JWT for Blue Ocean 1.3.5
BlueOcean JWT plugin: Enables JWT based BlueOcean API authentication
LDAP Plugin 1.18
Adds LDAP authentication to Jenkins
Mailer Plugin 1.2
This plugin allows you to configure email notifications for build results
MapDB API Plugin 1.0.9.0
This plugin provides a shared dependency on the MapDB library so that other plugins can co-operate when using this library.
Marathon Deployment 1.6.0
Deploy applications to Marathon after a build.
Matrix Authorization Strategy Plugin 1.7
Offers matrix-based security authorization strategies (global and per-project).
Matrix Project Plugin 1.12
Multi-configuration (matrix) project type.
Maven Integration plugin 3
This plug-in provides, for better and for worse, a deep integration of Jenkins and Maven: Automatic triggers between projects depending on SNAPSHOTs, automated configuration of various Jenkins publishers (Junit, ...).
Mercurial plugin 2.2
This plugin integrates Mercurial SCM with Jenkins. It includes repository browsing support for hg serve/hgweb, Google Code, Bitbucket, FishEye, KilnHG and RhodeCode. Features include guaranteed clean builds, named branch support, module lists, Mercurial tool installation, and automatic caching.
mesos 0.15.0
This plugin can be used to connect Jenkins to a Mesos cluster and dynamically launch Jenkins slaves based on the work load.
Metrics Plugin 3.1.2.10
This plugin exposes the Metrics API to Jenkins plugins.
Monitoring 1.70.0
Jenkins' monitoring with JavaMelody. Open report after installation.
Naginator 1.17.2
This plugin reschedules failed jobs.
NAnt Plugin 1.4.3
This plugin is a NAnt Builder for building .NET projects.
Node Iterator API Plugin 1.5.0
This plugin provides support for iterating through all the Node instances that are in use by Jenkins, even those Node instances that are not traditionally attached to Jenkins. The API exposed by this plugin can be used by cloud provider plugins to identify unused provisioned resource.
OWASP Markup Formatter Plugin 1.5
Uses the OWASP Java HTML Sanitizer to allow safe-seeming HTML markup to be entered in project descriptions and the like.
PAM Authentication plugin 1.3
Adds Unix Pluggable Authentication Module (PAM) support to Jenkins
Parameterized Trigger plugin 2.35.2
Personalization for Blue Ocean 1.3.5
Blue Ocean Personalization
Pipeline 2.5
A suite of plugins that lets you orchestrate automation, simple or complex. See Pipeline as Code with Jenkins for more details.
Pipeline Graph Analysis Plugin 1.6
Provides a REST API to access pipeline and pipeline run data.
Pipeline implementation for Blue Ocean 1.3.5
This plugin is a part of BlueOcean Plugin
Pipeline SCM API for Blue Ocean 1.3.5
This plugin is a part of BlueOcean Plugin
Pipeline Utility Steps 1.5.1
Utility steps for pipeline jobs.
Pipeline: API 2.24
Plugin that defines Pipeline API.
Pipeline: Basic Steps 2.6
Commonly used steps for Pipelines.
Pipeline: Build Step 2.6
Adds the Pipeline step build to trigger builds of other jobs.
Pipeline: Declarative 1.2.6
An opinionated, declarative Pipeline.
Pipeline: Declarative Agent API 1.1.1
Replaced by Pipeline: Declarative Extension Points API plugin.
Pipeline: Declarative Extension Points API 1.2.6
APIs for extension points used in Declarative Pipelines.
Pipeline: GitHub Groovy Libraries 1
Allows Pipeline Grrovy libraries to be loaded on the fly from GitHub.
Pipeline: Groovy 2.42
Pipeline execution engine based on continuation passing style transformation of Groovy scripts.
Pipeline: Input Step 2.8
Adds the Pipeline step input to wait for human input or approval.
Pipeline: Job 2.16
Defines a new job type for pipelines and provides their generic user interface.
Pipeline: Milestone Step 1.3.1
Plugin that provides the milestone step
Pipeline: Model API 1.2.6
Model API for Declarative Pipeline.
Pipeline: Multibranch 2.16
Enhances Pipeline plugin to handle branches better by automatically grouping builds from different branches.
Pipeline: Nodes and Processes 2.17
Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection.
Pipeline: REST API Plugin 2.9
Provides a REST API to access pipeline and pipeline run data.
Pipeline: SCM Step 2.6
Adds a Pipeline step to check out or update working sources from various SCMs (version control).
Pipeline: Shared Groovy Libraries 2.9
Shared libraries for Pipeline scripts.
Pipeline: Stage Step 2.3
Adds the Pipeline step stage to delineate portions of a build.
Pipeline: Stage Tags Metadata 1.2.6
Library plugin for Pipeline stage tag metadata.
Pipeline: Stage View Plugin 2.9
Pipeline Stage View Plugin.
Pipeline: Step API 2.14
API for asynchronous build step primitive.
Pipeline: Supporting APIs 2.16
Common utility implementations to build Pipeline Plugin
Plain Credentials Plugin 1.4
Allows use of plain strings and files as credentials.
Pub-Sub "light" Bus 1.12
A simple Publish-Subscribe light-weight event bus for Jenkins
Python Plugin 1.3
Adds the ability to execute python scripts as build steps.
Python Wrapper Plugin 1.0.3
This plugin provides the runtime library for plugins written in Python.
Rebuilder 1.27
This plugin is for rebuilding a job using the same parameters.
Resource Disposer Plugin 0.8
Dispose resources asynchronously. Utility plugin for resources that require more retries or take a long time to delete.
REST API for Blue Ocean 1.3.5
This plugin is a part of Blue Ocean UI
REST Implementation for Blue Ocean 1.3.5
This plugin is a part of Blue Ocean UI
Role-based Authorization Strategy 2.6.1
Enables user authorization using a Role-Based strategy. Roles can be defined globally or for particular jobs or nodes selected by regular expressions.
Run Condition Plugin 1
Define conditions for the execution of build steps
S3 publisher plugin 0.10.12
This is a plugin to upload files to Amazon S3 buckets.
Safe Restart Plugin 0.3
This plugin allows you to restart Jenkins safely.
SAML Plugin 1.0.4
This plugin enables use of a SAML 2.0 authentication source for single sign-on support.
SCM API Plugin 2.2.6
This plugin provides a new enhanced API for interacting with SCM systems.
Script Security Plugin 1.4
Allows Jenkins administrators to control what in-process scripts can be run by less-privileged users.
Scriptler 2.9
Scriptler allows you to store/edit/execute groovy scripts on any of the slaves/nodes... no need for copy paste groovy code anymore. Beside administer your scripts, Scritpler also provides a way to share scripts between users via hosted script catalogs on the internet.
Secure Requester Whitelist Plugin 1.2
Allows an administrator to specify sites trusted to make JSONP or primitive-XPath REST API requests.
Server Sent Events (SSE) Gateway Plugin 1.15
Server Sent Events (SSE) Gateway.
Slack Notification Plugin 2.3
This plugin is a Slack notifier that can publish build status to Slack channels.
SSH Agent Plugin 1.15
SSH Credentials Plugin 1.13
Allows storage of SSH credentials in Jenkins
SSH Slaves plugin 1.25
Allows to launch agents over SSH, using a Java implementation of the SSH protocol.
Structs Plugin 1.1
Library plugin for DSL plugins that need names for Jenkins objects.
Subversion Plug-in 2.10.2
Support Core Plugin 2.44
This support plugin allows generation of a support request bundle that contains diagnostic information to aid with resolving issues.
Timestamper 1.8.9
Adds timestamps to the Console Output
Token Macro Plugin 2.3
This plug-in adds reusable macro expansion capability for other plug-ins to use.
Translation Assistance plugin 1.15
user build vars plugin 1.5
This plugin is used to set user build variables: jenkins user name and id.
Variant Plugin 1.1
This user-invisible library plugin allows other multi-modal plugins to behave differently depending on where they run.
vSphere Plugin 2.16
This plugin provides tools to implement "cloud" like functionality in Jenkins for those using a vSphere Virtualization infrastructure.
Web for Blue Ocean 1.3.5
Blue Ocean core
Windows Slaves Plugin 1.3.1
Allows you to connect to Windows machines and start slave agents on them.
Workspace Cleanup Plugin 0.34
This plugin deletes the project workspace after a build is finished.

@elbuo8
Copy link

elbuo8 commented Feb 27, 2018

We experienced the error too. Version 2.0.1 of the plugin. Single job. Same error log @ayashjorden posted above.

It stops working for a few hours and then it resumes the queue consumption without any intervention 🤔

@ayashjorden
Copy link
Author

ayashjorden commented Feb 28, 2018 via email

@metacyclic
Copy link

metacyclic commented Nov 5, 2018

@ayashjorden - have you figured out what is happening? I am seeing the same issue with plugin version 2.0.1. If I go into Manage Jenkins -> Configure System and click on Test Access buttons below each queue ( which always seems to work ), jenkins starts picking messages off of the queue. The timing seems to also be every two weeks.

After clicking on Test Access and saving the page, I see log lines like the following for each of our listeners:

INFO: 000C83 Create new SQSTriggerQueue

@abeeskau
Copy link

abeeskau commented Nov 5, 2018

We were seeing issues similar to this when our SQS queues were setup as FIFO Queue type. We converted all of our queues to Standard Queue type and no longer see this behavior.

@metacyclic
Copy link

All of our queues were set up as Standard Queue.

@metacyclic
Copy link

metacyclic commented Nov 7, 2018

With help from AWS support, I think the culprit is the JVM and DNS caching. AWS suggests setting the TTL to 60, so following this example, I've added the following to the JAVA_OPTS of the container.

value: '-Dsun.net.inetaddr.ttl=60'

🤞 for 2-3 weeks

@metacyclic
Copy link

28 days later, and our jenkins master has not lost connection to SQS. 👍

Maybe this setting should be mentioned in the docs?

@metacyclic
Copy link

Still losing connections randomly. Going to try 0s. If that doesn't work, we'll need to monitor our sqs for backed up messages.

@ayashjorden
Copy link
Author

Hi @metacyclic, any update?
We're about to migrate to poll the queues with a time-based job triggering.

@metacyclic
Copy link

Hi @ayashjorden. I haven't had an issue yet. If it creeps up, I will post back.

@metacyclic
Copy link

@ayashjorden - the problem persists. Have had it happen a few times over the last couple of weeks. Looking at other solutions

@ayashjorden
Copy link
Author

ayashjorden commented May 5, 2020

@ayashjorden - the problem persists. Have had it happen a few times over the last couple of weeks. Looking at other solutions

We migrated to polling the queues (not using this plugin), stability is much higher now....

@AlexanderRainchikArammeem

@ayashjorden - the problem persists. Have had it happen a few times over the last couple of weeks. Looking at other solutions

We migrated to polling the queues (not using this plugin), stability is much higher now....

What instrument do you use for polling queues from sqs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants