Aggregator and Auto-Submit?

[seichner]

Hello! If i read the code (Aggregator, Queue) correctly, i cannot have Aggregators submit their data automatically, right?

So either i use an auto-submitting queue OR i use an aggregator?

The Ruby-Lib seems to handle this differently, Aggregator supports an autosubmit_interval-option.

[drio]

Hi @seichner,

They do: (https://github.com/librato/python-librato/blob/master/librato/queue.py#L47).

Doc: https://github.com/librato/python-librato#client-side-aggregation and
https://github.com/librato/python-librato#sending-measurements-in-batch-mode.

Let me know if that is not what you were looking for.

[phoet]

@drio if i understand the python code correctly, the only way to use the aggregator and the queue together is by calling queue.add_aggregator() with an aggregator as an argument.

within add_aggregator the events are added to the queue and the autosubmit option will submit them when the limit is reached. this assumes, that the aggregators need to be added to a queue on a regular basis, right?

please have a look at our thumbor addon, that is going to provide librato-metrics support. within that code, how would you implement the auto-submission?

[drio]

@phoet Yes, you are right. We'll have to make some tweaks to accommodate what you are trying to do. I'll take a look to the addon and get back to you with solutions.

[phoet]

awesome, thx!

[drio]

please have a look at our thumbor addon, that is going to provide librato-metrics support. within that code, how would you implement the auto-submission?

Here is what we can do. You want to have a timer within your system that periodically
wakes up and:

• creates an aggregator
• adds all the measurements available in your metrics/queue to the aggregator
• submits the aggregation
• flushes your metrics/queue
• sleeps

Something similar to what we are doing in this ruby example.

[phoet]

I think what we want is this https://github.com/librato/librato-metrics/blob/master/lib/librato/metrics/aggregator.rb#L32

[drio]

I am working on porting that over. Stay tuned.

[seichner]

Cool, Thanks a lot @drio !

[drio]

@chancefeick @phoet @seichner

We have to put this on hold until we finish the process of consolidating features across the librato bindings.

I was looking at the option in the ruby bindings that you suggested and I don't think that is what you need. The flushing happens only when the # of seconds have passed and a new measurement is .added to the aggregator. You want to automatically flush the aggregator periodically.

In the meantime we work on all this, here is an example (in ruby) that you can use as a reference to implement what you need at user level.

[seichner]

@drio Thanks for your investigation - just to clarify our use case: checking the interval on every .add, like the Ruby version does, is completely fine for us. We are running Thumbor with quite a huge amount of requests.

I think this will always be the case for people using the Aggregator, as it's purpose is to reduce the number of requests to Librato. It usually will not be used on a low-traffic site where the .add dependency would be a problem.