Slow communication between faye processes

[nilbus]

On a single machine I have a background daemon publishing faye messages and a separate rails process that allows clients to subscribe. With faye-redis, the rails server does pick up all the jobs, but there's a long delay ranging between roughly 5 and 25 seconds before the rails faye server will pick up all the messages and send them to the clients. When I publish messages from the rails server, there is virtually no delay before the client receives it, so I am assuming the delay is in the faye-redis engine.

Shouldn't this be faster? If the delay is by design, how can I make it poll more frequently? I'd love if this were configurable.

Thanks!

[nilbus]

It seems this can be resolved by decreasing the faye server timeout.

[jcoglan]

Though you've closed this I'd like some more details because your description seems to misattribute the problem.

As I understand it, you have an app server running Rails and a Faye server in the same process, presumably with Faye added as middleware. You're also running a daemon that publishes messages.

Does the daemon publish by using a Faye client to connect to your Rails/Faye instance, or does it connect directly to Redis and put messages there?

If the former, what happens is that the incoming messages are received by the Faye server, and it figures out which clients should get them. It stores the message in a queue for each recipient, and emits a Redis pubsub event that lets any other connected Faye servers connected to Redis know there are new messages for the client(s).

When a Faye server gets that pubsub notification, it checks to see if any of the recipient clients are connected to it at that instant. If a client is connected, the server flushes the message queue from Redis and delivers them to the client.

The reason I'm confused is because you mention that changing the timeout fixes things. This timeout controls communication between the clients and the server, not anything in the backend. If there is delay, it can arise in several places:

• The Faye server writing messages to queues when it receives them
• The Redis pubsub notification not being processed quickly enough by other servers
• Taking too long to read a client's queue after receiving a pubsub notification

There are probably other places as well. faye-redis does not poll, it uses Redis pubsub to push messages around. My first thought was that maybe something is blocking the EventMachine loop and stopping I/O events from being processed quickly, but it could be something else.

It would be worth monitoring the /notifications key in Redis yourself, in a separate process from Rails -- that's the pubsub channel we use for notifying servers. See if events are emited from there promptly when new messages are published.

[nilbus]

Thanks for explaining that - using Redis pubsub makes much more sense than polling. Given your explanation, I'm not sure why changing the timeout helps either; it seems that the other clients should catch up instantly. I am using the ruby faye client on both the daemon and the Thin Rails server. When I get a chance, I'll look at the /notifications key in redis before the other server gets the message and report back.

[jcoglan]

Oh, so the Faye server and Rails server are separate processes? When the Faye client inside your Rails app gets a message, how does it forward that to the in-browser clients?

[nilbus]

Setup

I'll try to explain my setup a little better. If you're interested to look at the code for specifics, it's all public at nilbus/pinewood-derby.

A Rails application runs in a Thin server. It uses the faye-redis and faye-rails gems. Browser clients connect to faye through a rack mount defined in routes.rb.

In a separate process the same Rails application runs some code as a daemon, polling a hardware device for updates. See lib/daemons/sensor_watch.rb, which instantiates a SensorWatch, which calls Faye.ensure_reactor_running! in #initialize.

An AnnounceController uses ActiveRecord observers to add callbacks to certain models to watch for updates, which then publishes to faye.

Since the daemon takes some time to edit/stop/start to try things out, I also often use the Rails console to test publishing to faye. I've tested from both the daemon and the console and saw the same delays when constantly publishing the time. To publish messages in the console, I'll do something like:

Faye.ensure_reactor_running!
AnnounceController.publish('/announce', 'test message')

Testing

I did what you suggested and opened the redis-cli to monitor the /notifications pubsub channel. First I increased the server timeout to 25 sec. I used AnnounceController.publish in the console to push messages out and timed the duration before I got the pubsub notifications in redis-cli. The pubsub notifications were delayed 5-25 seconds after I published them.

I added a before_filter on every request that publishes on every request, and I always get those pubsub notifications in redis-cli immediately on loading a page.

When I publish from the browser, I also see these notifications instantly in redis-cli. I see a notification from the browser client and then immediately after a notification from the client (same jid)

var c = new Faye.Client('http://localhost:3001/faye', {timeout: 5});
c.publish('/announce', 'from browserland')

I noticed that publishing from the browser immediately triggers any "queued up" notifications from the console that had not been received by redis yet. If I publish a message in the console and then publish a message from the browser client, I then see the console's published message in both redis and in the browser immediately, rather than having to wait for the normal delay.

The last thing that I noticed that I found odd is that the Rails web server (thin) and the Rails console both share the same Faye client id. Could it be that the thin process is picking up all the messages for that client id before the daemon/console process can receive it? If that's the case, then it would seem that the ping message that happens on the timeout interval also causes it to check its messages queue, which is why decreasing the timeout helped me.

What do you make of all this?

[nilbus]

Sorry, I was definitely wrong about the rails thin server and rails console having the same client ID - I was reading the redis-cli output incorrectly. The client ID that shows there is the client that should receive the notification, not the sender. In this case, I think I might have been seeing the redis-cli client id as a recipient of all messages? No matter, the client IDs as reported by the channel monitor are unique for all publishers.

[GauravShetty1016]

Hi, I am facing a similar issue.
My Setup is as follows:
I have a rails server that publishes messages to redis.
I have a thread running that has access to the faye server instance and is subscribed to redis queues.
When I publish a message to redis i see that the thread receives it immediately. I then use faye_server.get_client.publish to publish to the appropriate channels. This is the point at which i notice a 5 second delay.

I do not notice any delays at all when i publish from browser to the same channel.
I am using the latest faye build.

Any ideas?

[jcoglan]

Sorry for taking so long to reply. I was wondering if you can reproduce this outside of Rails, since if you're running Faye in the same process and Rails there's all sorts of things that could be causing problems that will take a long time to rule out.

I've probably misunderstood something here but are you running Faye and Rails in separate processes or in the same process? i.e. is there one server hosting both your app and the Faye server, or are they separate servers?

It sounds to me like something is blocking the event loop. EventMachine is known not to work very well when you're in an irb session, and will cause events to be hugely delayed and possible not delivered. You're better off writing a little script to publish messages and running that over and over, rather than trying to publish from irb. Let me know if that script shows delay.

Likewise, if your daemon is showing delay it's probably something blocking the event loop.

Have you found out whether this problem is specific to Redis? Have you tried running Faye with the default engine? Does it have the same problems or not?

[GauravShetty1016]

Faye is running in an independent process. Right now I am running them both on my local machine as my application is still under development.

Also I am running faye as a rack application and the reason why I chose to implement it is to decouple my push server from my rails server. And also to avoid the delay associated with sending an http request via my rails server to the faye server.

The subscription to redis in the faye rack application is running in an independent thread. Do you think this is the reason for the block? Since its a separate thread I did not think this should be the case. Can you suggest an alternative architecture.

I am using faye-redis as the backend for faye. Is there any way to directly access the faye-redis queues and add the tasks in that directly from an external process?

[nybblr]

Having this same issue: using faye-rails and I'm running faye_server within my routes.rb (less than optimal, but I'll switch over to a separate process ASAP because I dislike thin for Rails dev); as everyone else, messages published via the main Rails server (e.g. via a model observer) propagate instantly to the Redis backend, and thus the clients. On the other hand, updating a model in the Rails console and saving triggers the observers instantly and calls Faye to publish the message, but it is about 5 seconds later before the event is actually processed and clients receive the message.

I'd switched to Redis so I could test publishing messages from the Rails console with a running app, so switching back to the default store isn't an option for me...so I'm not sure if it is a Redis issue??

~ Jonathan

[jcoglan]

Could one of you still experiencing this post a demo Rails app that reproduces the problem? There's any number of potential causes and I'm going to be totally guessing if I try to reproduce it myself.

Also, I'd remind everyone that EventMachine does not work very well in the Rails console.

You do not need the Redis backend just to test your app. You can use HTTP to send messages to your Faye/Rails server so if you need to test stuff from the command line, use curl. I'd still recommend running Faye and Rails as separate processes, though.

[nilbus]

This app is the one I was having the issue with. Note in the readme the specific version of rails you have to use. Because of that, it's not the easiest to set up, but it's otherwise documented how to set it up in the readme.

https://github.com/nilbus/pinewood-derby

If anyone else has an app with an easier setup, feel free to throw it out here too.