Cost of using vectors

I'm wondering how much extra RAM a vector uses up; would it be silly to have an array of *millions* of small vectors, each containing anywhere from zero to about 3 or 4 elements?