quickstart to see anomalies

[pathcl]

Hello there!

Thanks for making this project open source. I'm trying to figure what would be a quick way to get anomalies as it is shown here
. I couldn't find it on the quickstart. Could you please give me some guidelines? I tried to run a few jobs using dagster but didn't have luck so far

Thank you,

[andrewm4894]

howdy!

easiest is probably either local python to get started or docker.

(if you press the open in codespaces button on the readme it will start a github codespace that should just work once it loads and runs the docker image)

https://github.com/andrewm4894/anomstack?tab=readme-ov-file#local-python-env

https://github.com/andrewm4894/anomstack?tab=readme-ov-file#docker

ps deepwiki from devin is pretty good for chatting to the repo and might be useful

https://deepwiki.com/search/how-can-i-deploy-this_59ec8f83-257a-490a-8df3-c32503971023

really you would just need to make a new folder in /metrics/ with your own metrics you want and then just deploy that

i'd suggest playing around with it a little first to get the python example running and then you can go from there.

[pathcl]

howdy!

easiest is probably either local python to get started or docker.

(if you press the open in codespaces button on the readme it will start a github codespace that should just work once it loads and runs the docker image)

https://github.com/andrewm4894/anomstack?tab=readme-ov-file#local-python-env

https://github.com/andrewm4894/anomstack?tab=readme-ov-file#docker

ps deepwiki from devin is pretty good for chatting to the repo and might be useful

https://deepwiki.com/search/how-can-i-deploy-this_59ec8f83-257a-490a-8df3-c32503971023

really you would just need to make a new folder in /metrics/ with your own metrics you want and then just deploy that

i'd suggest playing around with it a little first to get the python example running and then you can go from there.

thanks for the quick reply! should I enable all dagster jobs in the dagster ui for python as well?.

[pathcl]

btw I'm following this: https://github.com/andrewm4894/anomstack/tree/main/metrics/examples/python/python_ingest_simple but I see more jobs in the dagster ui:

python_ingest_simple_alerts
python_ingest_simple_change
python_ingest_simple_delete
python_ingest_simple_ingest
python_ingest_simple_llmalert_job
python_ingest_simple_plot_job
python_ingest_simple_score
python_ingest_simple_train

• are all of them needed to be enabled?
• how much data or how long should I wait until an anomaly is detected? in the end I'm trying to figure howto quickly either backfill or have a local poc running. I'm looking at these options now: https://github.com/andrewm4894/anomstack/blob/main/metrics/defaults/defaults.yaml

[andrewm4894]

yeah you need to manually enable in dagster ui.

you would only need to do this the first time and then thier state would be saved in the dagster db.

its a little annoying - especially for dev when need to spin up and down new envs etc - there is a graphql way to make like an api call to dagster to enable them via an api call so no need for clicking in the ui - i was using it before - https://andrewm4894.github.io/anomstack/docs/graphql/examples/start_schedule

would be good if there was like the opposite of the disable_ingest type params here - eg something like on_by_default_ingest to have flag you want a job to be on by default. i think this would be possible to do.

... oh wait knew i added something for this

anomstack/anomstack/jobs/ingest.py

Line 306 in ac36698

if spec.get("ingest_default_schedule_status", "STOPPED") == "RUNNING":

param like ingest_default_schedule_status=RUNNING should have them toggled on by default.

ps yep - each "metric batch" will have n seperate dagster jobs - see jobs section here https://github.com/andrewm4894/anomstack/tree/main?tab=readme-ov-file#concepts

[andrewm4894]

how much data or how long should I wait until an anomaly is detected? in the end I'm trying to figure howto quickly either backfill or have a local poc running

can depend on how often its running - i think python job is like every 2 mins to try fill up quickly but backfill is good idea too.

have been thinking about adding some scripts to backfill in some way maybe to make it quicker to sort of even seed with some starter data.

[pathcl]

how much data or how long should I wait until an anomaly is detected? in the end I'm trying to figure howto quickly either backfill or have a local poc running

can depend on how often its running - i think python job is like every 2 mins to try fill up quickly but backfill is good idea too.

have been thinking about adding some scripts to backfill in some way maybe to make it quicker to sort of even seed with some starter data.

What I ran into now is that I have a lot of dagster jobs queued but not running. Some of them have been running for >4h but still not finishing. I'm using docker-compose.yaml to bootstrap everything (https://github.com/andrewm4894/anomstack/blob/main/docker-compose.yaml) did you tweak any config? I only enabled all python_ingest_* jobs. They ran fine for the 3 or 4 initial jobs but then started to get stuck

[pathcl]

how much data or how long should I wait until an anomaly is detected? in the end I'm trying to figure howto quickly either backfill or have a local poc running

can depend on how often its running - i think python job is like every 2 mins to try fill up quickly but backfill is good idea too.
have been thinking about adding some scripts to backfill in some way maybe to make it quicker to sort of even seed with some starter data.

What I ran into now is that I have a lot of dagster jobs queued but not running. Some of them have been running for >4h but still not finishing. I'm using docker-compose.yaml to bootstrap everything (https://github.com/andrewm4894/anomstack/blob/main/docker-compose.yaml) did you tweak any config? I only enabled all python_ingest_* jobs. They ran fine for the 3 or 4 initial jobs but then started to get stuck

ahh there're some resource constraints:

anomstack/docker-compose.yaml

Line 64 in ac36698

deploy:

[andrewm4894]

Yeah I think the GitHub code spaces is a bit under powered and so the every 2 mins dagster jobs back up a bit.

Probably should adjust them to 5 to 10 minutes or something.

[andrewm4894]

I think it's this dagster run coordinator that I switched to last week

It can become a bottleneck with the limit of 10 and if resources constraint which the default GitHub code space can be a bit under powered.

anomstack/dagster_docker.yaml

Line 7 in ac36698

class: QueuedRunCoordinator

[andrewm4894]

I have a guide on using Google cloud as a VM in Google cloud is a bit more easy, but needs setting up a GCP project etc.

https://andrewm4894.github.io/anomstack/docs/deployment/gcp

I'm actually planning on deploying the demo app to fly.io and was going to add more docs around that soon too.

[pathcl]

how much data or how long should I wait until an anomaly is detected? in the end I'm trying to figure howto quickly either backfill or have a local poc running

can depend on how often its running - i think python job is like every 2 mins to try fill up quickly but backfill is good idea too.

have been thinking about adding some scripts to backfill in some way maybe to make it quicker to sort of even seed with some starter data.

I've ran it for the last 2 hours (https://github.com/andrewm4894/anomstack/tree/main/metrics/examples/python) but so far I still don't see an anomalie as in this picture

Should I increase threshold or score config here? I was also thinking to use netdata examples if that's faster

[andrewm4894]

can you see the blue line just of the data? eg if you enable the plot_job is will give you a plot in the dagster logs or you should be able to see the fasthtml dashboard on port 5001

[pathcl]

can you see the blue line just of the data? eg if you enable the plot_job is will give you a plot in the dagster logs or you should be able to see the fasthtml dashboard on port 5001

This is what I see:
plots
jobs

I would expect to see those dots in orange/yellow at some point. When should they show up? they look as the anomaly to me

btw I tried to access discord for anomstack but the invite has expired. Is there still a discord available?

[andrewm4894]

ah great - that looks normal - just seems like the random python anomlaies have maybe not happened yet

only about 1% of the time will it generate an anomaly https://github.com/andrewm4894/anomstack/blob/main/metrics/examples/python/python_ingest_simple/ingest.py#L18

but fact you have blue line means ingest is working and fact you have the dashed red line means training and scoring is working.

just seems that the red line - the score - has not gone over 0.8 yet which iirc is default

yeah this param - https://github.com/andrewm4894/anomstack/blob/main/metrics/defaults/defaults.yaml#L56

you could drop it to 0.5 for example if want to get some anomalies triggered

[andrewm4894]

oh wait it looks like alerts job is not toggled on - its actuay that job that will check the anomaly score and trigger an alert