While we do assume some programming knowledge (i.e, knowing just "Hello World" won't be enough), the entry bar is deliberately set low so that students with a wide programming background can attempt the contest and get a non-zero score.
One can usually get a reasonable score with just CS1002 or equivalent (or at least the Python that you'll learn from 2000-level math modules at St Andrews). High-school programming knowledge could also be enough. With that, one should be able to solve at least one question in most contests.
Doing better would usually require you to have taken a data structures or algorithms class (so in St Andrews that would be CS2001/CS2101). At this point, knowledge of (especially) sorting algorithms and string manipulation, data structures and searching algorithms (like DFS/BFS) would help. Examples include Lily's Homework and Sherlock and the Valid String (Q3 and 4) in Contest 3, and Fruit Baskets 1 (Q4) in Contest 2. At this stage, attempting to solve questions in naive/non-optimal time complexity will usually not gain many points.
The hardest questions usually involve graphs (one example being Influenced Sum (Q4) in Contest 4), a very important concept which is only briefly taught in CS2001/MT2504 and is mostly expanded upon in CS3052. Another class of problems in the contest which usually end up being the hardest to solve are dynamic programming problems (examples include Q5 and 6 in Contest 3), something which is barely (if at all) covered at St Andrews (but is a very important skill to have in technical interviews). We recommend searching the internet (and practicing similar LeetCode problems) for learning dynamic programming.
Most questions do not require advanced mathematical knowledge, though knowledge of discrete maths is useful in general.
And a note about the programming language: we don't care, and the problem setters usually keep most (if not all) languages open, including estoric languages like Brainf**k (though there is no guarantee that the problem can be solved in such a language). Most contestants use Java, Python and C++, as all of these three languages offer powerful library functions which would greatly help in solving problems under a time constraint.
Note that none of what was mentioned above stops you from learning the material yourself, and that is infact the highly recommended approach. This will give you a significant headstart in future modules you take as well. There are a variety of resources to pick from; good places include LeetCode, HackerRank and GeeksForGeeks (for learning about a new concept). It is also strongly recommended to take part in contests (this gives you a good way of seeing where you are and helps you to learn from past mistakes); all of us do it. Popular places to do that are LeetCode's Biweekly contests and Google's trio of competitions (HashCode, Kickstart and CodeJam).