Dynamic Parameters

[z-vr]

Hi, I've been working on the idea of dynamic parameters related to #51. The config could be extended as follows:

input {
    http_poller {
        urls => {
            papertrail => {
                method => "get"
                url => "https://papertrailapp.com/api/v1/events/search.json"
                headers => {
                    Accept => "application/json"
                }
                use_dynamic_params => true
                dynamic_params_map => {
                    min_id => "max_id"
                }
                last_dynamic_params => "./.last_dynamic_params"
            }
        }
        schedule => { every => "5s" }
        request_timeout => 60
        codec => "json"
    }
}

This means that min_idparameter of query will be populated with max_id field from the parsed event. I've never written Ruby code before, but it's quite fun. I know the tests are missing, but if you confirm that this can be potentially merged (seems to be quite popular feature), I will write tests.

One of the potential problems is that the request's spec now becomes populated with some extra variables, such as

{:request=>[:get, "https://papertrailapp.com/api/v1/events/search.json", {:headers=>{"Accept"=>"application/json"}, :query=>{"q"=>"sdk", "min_id"=>"741444678347702338"}, :dynamic_params_map=>{"min_id"=>"max_id"}, :last_dynamic_params=>"./.last_dynamic_params", :dynamic_params=>{"min_id"=>"741444678347702338"}}]

which might not be desirable, but I could not put them in the request object directly because logstash/plugin_mixins/http_client expects only 2 elements.

Another issue is that at the moment dynamic parameters are hard-coded to go into query property (and creating one if it does not exist), and not params which would be required for POST / other methods. Also they will override original query parameters specified in the config, but I guess this can be desirable.

Some implementation details had been taken almost directly from logstash-jdbc-input.

So yeah, let me know if it's cool and I'll write some tests / make suggested improvements. Cheers!

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

[andrewvc]

Thanks for the PR! I'll take a look at this on Monday :)

[jordansissel]

I haven't looked at the code, but your example has me asking one question -- why would this be preferred over having a papertrail input?

(Other examples of HTTP-using inputs: Elasticsearch, Heroku, S3, couchdb_changes, SQS, etc)

[buxel]

@jordansissel i think papertail is just an example for any HTTP input.
I'am currently in the same poisition, where I need to fetch data from a proprietary service that requires me to set GET parameters for timespans.
I was thinking about tailoring the http-poller plugin to my needs (just getting started with ruby) but a generic solution that benefits everyone would be better, of course.

@andrewvc , whats the status of this PR?