Streaming API Disconnects after period of time

[Kampe]

Not sure if this is a SF issue itself, but the streaming API seems to lose connection every few hours or so, is there a config to circumvent this and set the connection to keep-alive?

[kevinohara80]

Others have reported this but I haven't been able to reproduce in the past. There isn't a config in nforce to retry or reconnect but that is really the responsibility of the faye package under the covers. If you want to fork this and try messing with the faye version and/or configuration, I'd be open to a PR if you got it fixed.

[rgoldiez]

FYI.... I'm seeing the same thing.

[rgoldiez]

@Kampe - Have you tried setting the timeout when calling createStreamClient()? It looks like this package passes timeout down to Faye. And Faye's documentation seems to indicate that setting a timeout will keep alive an idle connection.

[akutta]

Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"87"}
Faye incoming {"channel":"/meta/disconnect","successful":true}
Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","advice":{"interval":0,"reconnect":"none"},"channel":"/meta/connect","id":"87","error":"403::Unknown client","successful":false}

This is logged by adding an extension to the FayeClient:

      incoming: function(message, callback) {
        console.log(`Faye incoming ${JSON.stringify(message)}`);
        callback(message);
      },
      outgoing: function(message, callback) {
        console.log(`Faye outgoing ${JSON.stringify(message)}`);
        callback(message);
      }
    };

    str.client._fayeClient.addExtension(Logger);

where str is _org.stream();

This is with autoRefresh set to true as well.

No notification has been sent up to the client in terms of str.on('disconnect', ...);

[akutta]

450 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5n"}
451 Faye incoming {"clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","channel":"/meta/connect","id":"5n","successful":true}
452 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5o"}
452 Faye outgoing {"channel":"/meta/connect","clientId":"2v1ozicqq5hbjy11wiv2c7puqxf7","connectionType":"long-polling","id":"5o"}
453 Faye incoming {"advice":{"interval":0,"reconnect":"handshake"},"channel":"/meta/connect","id":"5o","error":"403::Unknown client","successful":false}
454 Faye outgoing {"channel":"/meta/handshake","version":"1.0","supportedConnectionTypes":["in-process","websocket","long-polling"],"id":"5p"}
455 Faye incoming {"ext":{"replay":true},"minimumVersion":"1.0","clientId":"6o1uymp0ydsollqtbyf5js55mj3","supportedConnectionTypes":["long-polling"],"channel":"/meta/handshake","id":"5p","version":"1.0","successful":true}
456 Faye outgoing {"channel":"/meta/connect","clientId":"6o1uymp0ydsollqtbyf5js55mj3","connectionType":"long-polling","id":"5q"}
457 Faye outgoing {"channel":"/meta/subscribe","clientId":"6o1uymp0ydsollqtbyf5js55mj3","subscription":"/topic/xxx","id":"5r"}
458 Faye incoming {"clientId":"6o1uymp0ydsollqtbyf5js55mj3","advice":{"interval":0,"reconnect":"none"},"channel":"/meta/connect","id":"5q","error":"403::Unknown client","successful":false}
459 Faye incoming {"advice":{"interval":0,"reconnect":"handshake"},"channel":"/meta/subscribe","id":"5r","error":"403::Unknown client","successful":false}
460 Faye outgoing {"channel":"/meta/handshake","version":"1.0","supportedConnectionTypes":["in-process","websocket","long-polling"],"id":"5s"}
461 Faye incoming {"ext":{"replay":true},"minimumVersion":"1.0","clientId":"6p13gpcil0mrn281li95v1kqac7r","supportedConnectionTypes":["long-polling"],"channel":"/meta/handshake","id":"5s","version":"1.0","successful":true}
462 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5t"}
463 Faye outgoing {"channel":"/meta/subscribe","clientId":"6p13gpcil0mrn281li95v1kqac7r","subscription":"/topic/xxx","id":"5u"}
464 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","advice":{"interval":0,"timeout":110000,"reconnect":"retry"},"channel":"/meta/connect","id":"5t","successful":true}
465 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/subscribe","id":"5u","subscription":"/topic/xxx","successful":true}
466 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5v"}
467 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5v","successful":true}
468 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5w"}
469 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5w","successful":true}
470 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5x"}
471 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5x","successful":true}
472 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5y"}
473 Faye incoming {"clientId":"6p13gpcil0mrn281li95v1kqac7r","channel":"/meta/connect","id":"5y","successful":true}
474 Faye outgoing {"channel":"/meta/connect","clientId":"6p13gpcil0mrn281li95v1kqac7r","connectionType":"long-polling","id":"5z"}```

For reference, this is earlier in the connection lifetime.  It appears the server is telling us to disconnect but either nforce isn't being notified by Faye, or nforce is not reacting to the disconnect.  I have an event listener on: connect, disconnect, and error.  Only connect is fired, and that's just on the very initial startup. 

[kevinohara80]

Has anyone tried upgrading Faye? Perhaps it's a bug in the packaged version.

[akutta]

Currently 1.2.3 is running in my app, with a current release of 1.2.4 available. It does not seem to address this kind of bug.

My guess at this point, is that it randomly takes too long to respond to the reconnect request and the server says 'Go Away (disconnect).'

I have some extensive logging that is added in to try and validate this theory. Of course, the issue isn't easily reproducible, so it is a matter of time.

[akutta]

this is how i intercepted the disconnect event and pass it up to our implementation. It's not an ideal solution, I think this is actually an issue with Faye not properly sending the disconnect event up.

    const Logger = {
      incoming: function(message, callback) {
        console.log(`FAYE - incoming - ${new Date()} - ${JSON.stringify(message)}`)
        if ( message.channel === '/meta/disconnect' ) {
          errorHandler('disconnected');
        }
        callback(message);
      },
      outgoing: function(message, callback) {
        console.log(`FAYE - outgoing - ${new Date()} - ${JSON.stringify(message)}`)
        callback(message);
      }
    };

    str.client._fayeClient.addExtension(Logger);

My errorHandler right now does a reconnect to salesforce + streaming

[kevinohara80]

@akutta Great work here. This is really appreciated. Could I see the implementation of the errorhandler? I'm assuming that this is what you're using to force a reconnect. I'm willing to put it into nforce as a workaround. We'd just need to strip out the console logs and build it into the stream method.

[akutta]

My error handler just logs in again (probably unnecessary) and then starts a new streaming connection:

    var str = this._org.stream({
      topic: topic,
      oauth: oauth
    });

I feel like this should just be propagated up as a 'disconnect' event from the stream so whoever is integrating with nforce can decide the appropriate action.

    str.on('disconnect', () => { console.log('disconnected from stream'); })

Maybe have a helper function on the stream object that can do the reconnect.

[kevinohara80]

One thing to ask...have you tried setting different timeout and retry settings in the client? It's possible that there are defaults set by Faye, but the nforce options hash is sending in undefineds and blowing over the internal defaults.

Here is where that could be happening: https://github.com/kevinohara80/nforce/blob/master/lib/fdcstream.js#L47

[akutta]

I havn't changed the defaults, but I saw that the Faye client defaults were still setup properly while debugging in other portions of the client.

[swati410]

@kevinohara80 I am also facing the same issue as above . I have subscribed to platform events and for some few hours I am able to receive events but as soon as my token expires I don't receive any event . If I refresh my token and then make new connection and with that connection if I subscribe I will receive the event for new connection and my old one died . So my question is does nforce auto refreshes the access token for platform events.

On more observation :
If I my system (listener) disconnects for few hours , it will stop listening to events . Why is this so ?

Below is my implementation :
For making connection :

var connObj = {
            clientId: self.dataSource.settings.clientId,
            clientSecret: self.dataSource.settings.clientSecret,
            redirectUri: 'http://localhost:3000/oauth/_callback',
            environment: self.dataSource.settings.environment,
            apiVersion: 'v41.0',
            mode: self.dataSource.settings.mode || 'single',
            autoRefresh: true,
            onRefresh: function(newOauth, oldOauth, cb) {
                self.oauth = newOauth;
                cb();
                return;
            }
        };

        self.oauth = {};
        self.oauth.access_token = self.settings.accessToken;
        self.oauth.refresh_token = self.settings.refreshToken;
        self.oauth.instance_url = self.settings.instanceUrl;
       connObj.oauth = self.oauth;
        var org = self.org = nforce.createConnection(connObj);

For subscribing to platform events :

var channel = this._channel = this.org.stream({
               topic: topic,
               oauth: this.oauth,
               isEvent: true,
               apiVersion: 'v41.0',
           });

[swati410]

@kevinohara80
With salesforce api version 37 and more client is getting disconnected from endpoint and hence we don't get event after that .
Following is the log from faye client after which process silently disconnects .

incoming
{ channel: '/meta/disconnect' }
incoming
{ clientId: '5rw1bxn4jzyed3vhbcjnpkwb2fsd',
  advice: { interval: 0, reconnect: 'none' },
  channel: '/meta/connect',
  id: 'k1',
  error: '403::Unknown client',
  successful: false }

[srihakum]

@kevinohara80 We tried with Faye client version 1.2.4, but still end up the client being disconnected after sometime.

[srihakum]

@kevinohara80 there is an issue raised in faye client repo faye/faye#492 but they suggest since server is responding with reconnect: none they do not attempt again. However I have posted that in such scenarios do they send a transport:down event which we are handling as disconnect. So that any further code on this could be handled appropriately.

[kevinohara80]

I think I may have a solution in the works for this. It seems as if the reason for the disconnects is oauth access token expirations. I'm going to play with an autoRefresh function this week similar to how auto-refreshes work in normal nforce calls.

[dhulke]

I have tried implementing that auto refresh feature but I had to disable it. Sometimes Faye Client keeps a connection open and for some reason, salesforce kept that open too and I ended up with several active subscribers. The best solution I found was to close the process and start it with pm2 and autorestart.

[rakeshsm91]

@kevinohara80 Just curious to know if the version 2.0 is coming soon?