Pre-emptively catching out-of-control processes?

I've been having a recurring problem with Amarok as of late. It seems that every now and then the program decides to enter "memory consuming vortex of doom" mode, wherein it starts to demand more RAM at a rate of a few megs a second. This is a problem for me, as while the program is doing this, my system starts swapping furiously, and there's almost no resources left to do anything other than feed Amarok RAM and write to disk. So this means as soon as Amarok goes balls-up, I almost completely lose control of my mouse, my keyboard, everything (until my system goes totally non-responsive and I just have to reboot). I saved my system from the vortex of doom recently thanks to a KVM and SSH, but it was a close thing.

So my question for the community is this: Is there anything I can use to set a watch on a process so that I can either deny it memory, or kill it when it starts acting erratically? I think something like dtrace on Solaris would be perfect for this, but Linux doesn't have a functioning dtrace equivalent yet. Is there anything less exotic I can use?