Lot's of retry_later errors, how to increase maximum sessions and pipline size?

[vfsoraki]

I see a lot of retry_later errors in production.

Searching a bit, I found that I need to set maximum number of sessions and pipeline size (right?).

How can I set that using HTTPotion? I found the api to do that in ibrowse here, but I'm not sure where to put them.

Also, those functions let me increase it for a specific domain/port. How can I set globally?

[valpackett]

There's an example of passing ibrowse options in the readme:

HTTPotion.get "https://check-tls.akamaized.net", [ ibrowse: [ ssl_options: [ versions, [:'tlsv1.1'] ] ] ]

And you can use the default_ibrowse config option to set them globally:

https://github.com/myfreeweb/httpotion/blob/c79314b91df2cc8e43835b509efd682820d87fb4/config/config.exs#L5

[vfsoraki]

Using config :httpotion, :default_ibrowse, [max_sessions: 5000, max_pipline_size: 5000] in my config file.

Unfortunately, this does not fix my retry_later errors.

Am I using the options right?

[valpackett]

Yes, that should be how you set them.

max_pipline_size is a typo though.

You can also try direct mode. Or throttle your requests with something like ex_rated.

references: #21 cmullaparthi/ibrowse#147

[vfsoraki]

I don't get how direct mode works. I spawn a worker for myself, and use it whenever I want. Right?

Does this worker have a life span or something? How do I know when it is working, or I should spawn another one?

Also, can I somehow not use pool or sessions or anything from ibrowse or HTTPotion? I already have a pool of workers, and I am load balancing my requests (and parsing responses) between them. It would be good if I can somehow bypass session and pipeline limit here.

[valpackett]

not use pool or sessions

That's what direct mode does. It skips ibrowse's pool.

You probably need to spawn a worker on every request.