Suspect limits on payload elements (CONFIRMED)

[massimo1962]

After some tests on api, we noticed that the service seems to have a limit on the number of payload elements, whatever parameter you use.
for example:

# 20 years
wget -O test.json "http://webservices.ingv.it/eidaws/wfcatalog/1/query?network=IV&sta=ACER&start=2004-01-01&end=2024-01-07"

return: 19646

# 5 years
wget -O test.json "http://webservices.ingv.it/eidaws/wfcatalog/1/query?network=IV&sta=ACER&start=2010-01-01&end=2015-01-07"

return: 19646

So at this point we assume that we have some limitation on the node-js webservice code.
what should we do? insert an error code into the response (something like "error exceeded query limit") or modify the node-js code?
thanks in advance.

[petrrr]

Just two additional notes for clarifications on this issue:

• We did various testing with very different queries (or expected number of expected documents) and the limit is always rather close to 20,000 documents but not exactly.
• We did the same tests on selected other nodes and the limit is practically the same, so probably most (if not all nodes) are affected by this limitation.

We believe this limitation is not being introduced intentionally.

Should we conclude that some limitation of the resulting output is necessary for technical reasons, this limit should become configurable and needs to be handled differently. If the configured limit is eceeded the service must not reply with status code 200, but 413 Payload Too Large.

[Jollyfant]

Hi all, this behavior may potentially be caused by the MAXIMUM_BYTES_RETURNED field specified in the configuration. The service keeps streaming documents until it hits this limit. The request by Massimo is roughly ~10MB in size which is the default limit (10E6 bytes). If you set this value to 0 it should keep streaming indefinitely (or increase it).

[petrrr]

@Jollyfant:
Do you see a possibility to impose some limit (max. payload or number of documents) but handle it with the correct status code when it is hit, instead of returning a code 200?

I suspect currently the 200 status code is send before the limit is actually reached and detected, i.e. this mechanism may not allow to impose a well-defined limit. right?

If so, we would need to impose no limit (zero) to ensure consistency with the status code 200.

[Jollyfant]

A HTTP status code is always set before the body payload arrives unfortunately. So it returns 200 first, then streams until hitting the limit. Potentially there is an option to first load all documents in memory, calculate the size, and determine the service should return a 413 Payload Too Large but this was (is) perhaps not so simple to implement efficiently especially with the continuous segments in MongoDB. I'm not sure what the best solution is ATM but the limit can be disabled by setting it to 0 and implementing an alternative (perhaps by calculating expected payload size on number of days)..

[petrrr]

@Jollyfant

I agree that the immediate solution is probably to disable the limit.

If we want to control the maximum size of the payload, we could set some limits to the number of documents, maybe both daily and segments. Here, I am assuming that counting the number of resulting documents for a query before serving data could be feasible and not too costly. For the segments it could be optained by agregating / summing the nseg field. Am I missing something?

[petrrr]

Hi @s-schneider!

I am assigning this issue to you, so it does not get stranded.

We can decide also with @jschaeff how to continue with it one. Basically the suspected limit has been confermed. It can be disabled.

However, we probably should:

• Document this behavior better and provide some guidance for installation
• Add the possibility do define a service limit, which is then correctly reported to the user.

[jppzanetti]

Fixed by #46. Starting at version 1.0.3, wfcatalog returns 413 if the response is larger than MAXIMUM_BYTES_RETURNED.