When I try to get data by days the memory went high.

[wudixiaotie]

The query is

<<"SELECT avg('057345b6-76b1-4a40-aac9-139818'.'cpu'.'usage' BUCKET 'zone', 1d) AS 'cpu-usage', avg('057345b6-76b1-4a40-aac9-139818'.'cpu'.'effective' BUCKET 'zone', 1d) AS 'cpu-effective', avg('057345b6-76b1-4a40-aac9-139818'.'cpu'.'nwait' BUCKET 'zone', 1d) AS 'cpu-nwait', avg(divide('057345b6-76b1-4a40-aac9-139818'.'mem'.'usage' BUCKET 'zone', 1048576), 1d) AS 'memory-usage', avg(divide('057345b6-76b1-4a40-aac9-139818'.'mem'.'value' BUCKET 'zone', 1048576), 1d) AS 'memory-value', avg(divide('057345b6-76b1-4a40-aac9-139818'.'swap'.'usage' BUCKET 'zone', 1048576), 1d) AS 'swap-usage', avg(divide('057345b6-76b1-4a40-aac9-139818'.'swap'.'value' BUCKET 'zone', 1048576), 1d) AS 'swap-value', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'net'.'net0'.'opackets64' BUCKET 'zone'), 1d) AS 'net-send-ops-net0', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'net'.'net0'.'ipackets64' BUCKET 'zone'), 1d) AS 'net-recv-ops-net0', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'net'.'net0'.'obytes64' BUCKET 'zone'), 1d) AS 'net-send-b-net0', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'net'.'net0'.'ibytes64' BUCKET 'zone'), 1d) AS 'net-recv-b-net0', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'zfs'.'nread' BUCKET 'zone'), 1d) AS 'zfs-read-b', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'zfs'.'nwritten' BUCKET 'zone'), 1d) AS 'zfs-write-b', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'zfs'.'reads' BUCKET 'zone'), 1d) AS 'zfs-read-ops', avg(derivate('057345b6-76b1-4a40-aac9-139818'.'zfs'.'writes' BUCKET 'zone'), 1d) AS 'zfs-write-ops' LAST 30d">>.

Then the memory of the beam suddenly goes extramly high, Almost 1G.
I saw the code. Maybe because ddb sends all the data to the dflow, and dfow will create many temporary data terms in memory during data gathering and processing. So there alway a Wave crest of the memory usage of beam, then all the sub-query processes will terminated, the beam will GC itself.
What can I do if these long-term query is very frequently to be use? Just expanded memory?
Is there a way to limit the memory usage? Or I need to cache the result in Redis or something like that?

[Licenser]

Hi,
1G doesn't seem unreasonable hight, the total data you compute over would take 4,423,680,000 bytes (or a bit over 4G) in total if held in memory (not including intermediate steps). You probably can try to reduce the chunk size and see if it's limits the size.

[wudixiaotie]

haha,1G because the vm I query was just created for one week or less. Thanks.

[Licenser]

No that's not how things work, it will still work over 4G

[wudixiaotie]

Thanks for answer me.
So if we query data like above very frequently, I need to set the physical memory to 32G, or cache the result, right?

[Licenser]

What is it you are trying to achive and what is the issue you run into? 1G of memory really doesn't strike me as bad not sure how you get from 1 to 32 either I think I'm missing part of the storry

[wudixiaotie]

What I want to ask is:
If the vm which running dflow set physical memory to 4G, then the dflow should handle one request at the same time. Or the erlang node which running dflow will crash because it can not allocate more memory. So I have to upgrade memory to 32G, so the erlang node can handle 8 requests simultaneously when there are too many people want to query.
Or I cache the result everyday.

The whole story is our application always cashed and did not restarted after running for a while. The crash log shows erlang node can not allocate any memory from the system. Then I found out the erlang node alway had a Wave crest of memory usage when I try to send a query request.

But the aim of our application is to show vms metrics data to our client. So the traffic of the request will be very heavy as we expect.