prevention

I've gotten control of a server that is having some issues.  I've located udp.pl, iroffer, and doze4 in the /tmp/ directory.  I've removed all of these files and disabled all compilers on the system right now.  My question is can I change permissions in /tmp/ so it's not executable?  To stop these files from being run.  I know mysql.sock file is in there, which hasn't worked when I tried this before, but any suggestions?  Yes I know I need to find out how they are putting these files in /tmp/ but the question right now is to stop the attacks/bandwidth stealing from happening while I figure out the rest.

Thanks.