gets spotted by stockx bot detection

[KfirGez123]

Hey, I'm also a student and wanted to create a similar crawling program for my final's project.
I tried your program and it gets stuck on the "long press" bot authentication.
If you could find the time to hit me back and explain to me what to do
you'll be a lifesaver.

[JacobJustice]

That's pretty normal in my experience. The only way to fix this is by adjusting the sleep time delays and increasing them, I had them increased to very large numbers and I could never get it to scrape through the whole website unfortunately as it would have just taken too much time. The scraper used to be able to detect this page and would wait 3600 seconds and try to load the page again, doing this continuously until the page went away. But it's possible that it fails to detect the page correctly.

Maybe you could look into bot detection systems and try to prevent this somehow, but I imagine it would be too tedious for a final project.

Also just to make sure, other than this, the data is collecting normally still? I imagine the html layout might be slightly different so the code might not be collecting all the data.

Good luck with your project!

[JacobJustice]

Also worth noting that just adding more time delays isn't a perfect fix either, and I'm sure they'll just recognize that the scraper is going to the pages at regular intervals and auto-ban it.

[KfirGez123]

Alright, will look into it Thanks for the fast response man!