Find a more efficient way to pass the Tripod config to the process queue jobs

[rsinger]

We're currently passing in a Tripod config array with every background job which places a massive overhead for the amount of jobs Resque can handle.

If, rather than passing the config as part of the job, we stored the config, keyed by a hash of the config, in redis, we would have a much smaller memory footprint and could subsequently queue a lot more jobs.

[rsinger]

I don't think we'd need to change the footprint of the job data: if the tripodConfig value is an array, assume that the config is being passed directly, if it's a string, assume that's a key to look up in redis.

[rsinger]

Just a really quick investigation into this found that for regular Tripod job, we pass in a JSON string that's around 33,602 characters long (depending on the store).

If we gzcompress that JSON string, it's 5,087 characters long. I'm actually going to recommend that we Base64 that gzcompressed string (6,785 characters) so we don't gum up the Resque web interface too badly. It's a longer string, but I think it would pay off in the end.

Anyway, if my math is right (it's probably not!), we should be able to put about 5x as many jobs on the queue for the same memory footprint.

[rsinger]

I kind of feel like there must be some way to get that original config size down, as well.

[scaleupcto]

Yup agree, original config is massive and I have been thinking about that for a while now.

Just a thought - does converting it to YAML and then compressing that buy us anything?

Also is it possible to cherry pick - at least on the Apply jobs - just enough config to get the work done?

[rsinger]

Given that there's no native YAML support in PHP, anyway, shoot for the moon. Open the possibilities to Thrift or protobuf.

[scaleupcto]

Filesize is > if I convert to YAML anyway, due to whitespace, so scratch that. Partial config could work although gzipping the JSON seems like a good start.

What do we gain from the Base64'ing?

[scaleupcto]

Actually, I just cat'ed a gzip, see what you mean!