Skip to content
This repository has been archived by the owner on Jun 17, 2024. It is now read-only.

Streaming API Disconnects after period of time #131

Open
Kampe opened this issue Dec 5, 2016 · 19 comments
Open

Streaming API Disconnects after period of time #131

Kampe opened this issue Dec 5, 2016 · 19 comments

Comments

@Kampe
Copy link

Kampe commented Dec 5, 2016

Not sure if this is a SF issue itself, but the streaming API seems to lose connection every few hours or so, is there a config to circumvent this and set the connection to keep-alive?

@kevinohara80
Copy link
Owner

Others have reported this but I haven't been able to reproduce in the past. There isn't a config in nforce to retry or reconnect but that is really the responsibility of the faye package under the covers. If you want to fork this and try messing with the faye version and/or configuration, I'd be open to a PR if you got it fixed.

@rgoldiez
Copy link

rgoldiez commented Feb 6, 2017

FYI.... I'm seeing the same thing.

@rgoldiez
Copy link

rgoldiez commented Feb 6, 2017

@Kampe - Have you tried setting the timeout when calling createStreamClient()? It looks like this package passes timeout down to Faye. And Faye's documentation seems to indicate that setting a timeout will keep alive an idle connection.

@akutta
Copy link

akutta commented Apr 10, 2017

Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"87"}
Faye incoming {"channel":"/meta/disconnect","successful":true}
Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","advice":{"interval":0,"reconnect":"none"},"channel":"/meta/connect","id":"87","error":"403::Unknown client","successful":false}

This is logged by adding an extension to the FayeClient:

      incoming: function(message, callback) {
        console.log(`Faye incoming ${JSON.stringify(message)}`);
        callback(message);
      },
      outgoing: function(message, callback) {
        console.log(`Faye outgoing ${JSON.stringify(message)}`);
        callback(message);
      }
    };

    str.client._fayeClient.addExtension(Logger);

where str is _org.stream();

This is with autoRefresh set to true as well.

No notification has been sent up to the client in terms of str.on('disconnect', ...);

@akutta
Copy link

akutta commented Apr 10, 2017

450 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5n"}
451 Faye incoming {"clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","channel":"/meta/connect","id":"5n","successful":true}
452 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5o"}
452 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5o"}
453 Faye incoming {"advice":{"interval":0,"reconnect":"handshake"},"channel":"/meta/connect","id":"5o","error":"403::Unknown client","successful":false}
454 Faye outgoing {"channel":"/meta/handshake","version":"1.0","supportedConnectionTypes":["in-process","websocket","long-polling"],"id":"5p"}
455 Faye incoming {"ext":{"replay":true},"minimumVersion":"1.0","clientId":"6o1uymp0ydsollqtbyf5js55mj3","supportedConnectionTypes":["long-polling"],"channel":"/meta/handshake","id":"5p","version":"1.0","successful":true}
456 Faye outgoing {"channel":"/meta/connect","clientId":"6o1uymp0ydsollqtbyf5js55mj3","connectionType":"long-polling","id":"5q"}
457 Faye outgoing {"channel":"/meta/subscribe","clientId":"6o1uymp0ydsollqtbyf5js55mj3","subscription":"/topic/xxx","id":"5r"}
458 Faye incoming {"clientId":"6o1uymp0ydsollqtbyf5js55mj3","advice":{"interval":0,"reconnect":"none"},"channel":"/meta/connect","id":"5q","error":"403::Unknown client","successful":false}
459 Faye incoming {"advice":{"interval":0,"reconnect":"handshake"},"channel":"/meta/subscribe","id":"5r","error":"403::Unknown client","successful":false}
460 Faye outgoing {"channel":"/meta/handshake","version":"1.0","supportedConnectionTypes":["in-process","websocket","long-polling"],"id":"5s"}
461 Faye incoming {"ext":{"replay":true},"minimumVersion":"1.0","clientId":"6p13gpcil0mrn281li95v1kqac7r","supportedConnectionTypes":["long-polling"],"channel":"/meta/handshake","id":"5s","version":"1.0","successful":true}
462 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5t"}
463 Faye outgoing {"channel":"/meta/subscribe","clientId":"6p13gpcil0mrn281li95v1kqac7r","subscription":"/topic/xxx","id":"5u"}
464 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","advice":{"interval":0,"timeout":110000,"reconnect":"retry"},"channel":"/meta/connect","id":"5t","successful":true}
465 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/subscribe","id":"5u","subscription":"/topic/xxx","successful":true}
466 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5v"}
467 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5v","successful":true}
468 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5w"}
469 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5w","successful":true}
470 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5x"}
471 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5x","successful":true}
472 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5y"}
473 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5y","successful":true}
474 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5z"}```

For reference, this is earlier in the connection lifetime.  It appears the server is telling us to disconnect but either nforce isn't being notified by Faye, or nforce is not reacting to the disconnect.  I have an event listener on: connect, disconnect, and error.  Only connect is fired, and that's just on the very initial startup. 

@kevinohara80
Copy link
Owner

Has anyone tried upgrading Faye? Perhaps it's a bug in the packaged version.

@akutta
Copy link

akutta commented Apr 11, 2017

Currently 1.2.3 is running in my app, with a current release of 1.2.4 available. It does not seem to address this kind of bug.

My guess at this point, is that it randomly takes too long to respond to the reconnect request and the server says 'Go Away (disconnect).'

I have some extensive logging that is added in to try and validate this theory. Of course, the issue isn't easily reproducible, so it is a matter of time.

@akutta
Copy link

akutta commented Apr 13, 2017

this is how i intercepted the disconnect event and pass it up to our implementation. It's not an ideal solution, I think this is actually an issue with Faye not properly sending the disconnect event up.

    const Logger = {
      incoming: function(message, callback) {
        console.log(`FAYE - incoming - ${new Date()} - ${JSON.stringify(message)}`)
        if ( message.channel === '/meta/disconnect' ) {
          errorHandler('disconnected');
        }
        callback(message);
      },
      outgoing: function(message, callback) {
        console.log(`FAYE - outgoing - ${new Date()} - ${JSON.stringify(message)}`)
        callback(message);
      }
    };

    str.client._fayeClient.addExtension(Logger);

My errorHandler right now does a reconnect to salesforce + streaming

@kevinohara80
Copy link
Owner

@akutta Great work here. This is really appreciated. Could I see the implementation of the errorhandler? I'm assuming that this is what you're using to force a reconnect. I'm willing to put it into nforce as a workaround. We'd just need to strip out the console logs and build it into the stream method.

@akutta
Copy link

akutta commented Apr 13, 2017

My error handler just logs in again (probably unnecessary) and then starts a new streaming connection:

    var str = this._org.stream({
      topic: topic,
      oauth: oauth
    });

I feel like this should just be propagated up as a 'disconnect' event from the stream so whoever is integrating with nforce can decide the appropriate action.

    str.on('disconnect', () => { console.log('disconnected from stream'); })

Maybe have a helper function on the stream object that can do the reconnect.

@kevinohara80
Copy link
Owner

One thing to ask...have you tried setting different timeout and retry settings in the client? It's possible that there are defaults set by Faye, but the nforce options hash is sending in undefineds and blowing over the internal defaults.

Here is where that could be happening: https://github.com/kevinohara80/nforce/blob/master/lib/fdcstream.js#L47

@akutta
Copy link

akutta commented Apr 13, 2017

I havn't changed the defaults, but I saw that the Faye client defaults were still setup properly while debugging in other portions of the client.

@swati410
Copy link

swati410 commented Jan 17, 2018

@kevinohara80 I am also facing the same issue as above . I have subscribed to platform events and for some few hours I am able to receive events but as soon as my token expires I don't receive any event . If I refresh my token and then make new connection and with that connection if I subscribe I will receive the event for new connection and my old one died . So my question is does nforce auto refreshes the access token for platform events.

On more observation :
If I my system (listener) disconnects for few hours , it will stop listening to events . Why is this so ?

Below is my implementation :
For making connection :

var connObj = {
            clientId: self.dataSource.settings.clientId,
            clientSecret: self.dataSource.settings.clientSecret,
            redirectUri: 'http://localhost:3000/oauth/_callback',
            environment: self.dataSource.settings.environment,
            apiVersion: 'v41.0',
            mode: self.dataSource.settings.mode || 'single',
            autoRefresh: true,
            onRefresh: function(newOauth, oldOauth, cb) {
                self.oauth = newOauth;
                cb();
                return;
            }
        };

        self.oauth = {};
        self.oauth.access_token = self.settings.accessToken;
        self.oauth.refresh_token = self.settings.refreshToken;
        self.oauth.instance_url = self.settings.instanceUrl;
       connObj.oauth = self.oauth;
        var org = self.org = nforce.createConnection(connObj);

For subscribing to platform events :

var channel = this._channel = this.org.stream({
               topic: topic,
               oauth: this.oauth,
               isEvent: true,
               apiVersion: 'v41.0',
           });

@swati410
Copy link

swati410 commented Jan 29, 2018

@kevinohara80
With salesforce api version 37 and more client is getting disconnected from endpoint and hence we don't get event after that .
Following is the log from faye client after which process silently disconnects .

incoming
{ channel: '/meta/disconnect' }
incoming
{ clientId: '5rw1bxn4jzyed3vhbcjnpkwb2fsd',
  advice: { interval: 0, reconnect: 'none' },
  channel: '/meta/connect',
  id: 'k1',
  error: '403::Unknown client',
  successful: false }

@srihakum
Copy link

srihakum commented Feb 6, 2018

@kevinohara80 We tried with Faye client version 1.2.4, but still end up the client being disconnected after sometime.

@srihakum
Copy link

srihakum commented Feb 6, 2018

@kevinohara80 there is an issue raised in faye client repo faye/faye#492 but they suggest since server is responding with reconnect: none they do not attempt again. However I have posted that in such scenarios do they send a transport:down event which we are handling as disconnect. So that any further code on this could be handled appropriately.

@kevinohara80
Copy link
Owner

I think I may have a solution in the works for this. It seems as if the reason for the disconnects is oauth access token expirations. I'm going to play with an autoRefresh function this week similar to how auto-refreshes work in normal nforce calls.

@dhulke
Copy link

dhulke commented Feb 28, 2019

I have tried implementing that auto refresh feature but I had to disable it. Sometimes Faye Client keeps a connection open and for some reason, salesforce kept that open too and I ended up with several active subscribers. The best solution I found was to close the process and start it with pm2 and autorestart.

@rakeshsm91
Copy link

@kevinohara80 Just curious to know if the version 2.0 is coming soon?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants