You could try a potentiometer with a higher resistance like this:
http://www.mouser.com/ProductDetail/ALPS/RK27112A-S20-C0-A254/?qs=hm0oVamVljsxpd34aFZJ/A%3d%3dor add an in line resistor to the output of the potentiometer. Try different values until the lowest volume setting is suitable. You may want to start with something like 10K ohms.
Once you find the right value, replace with a good quality resistor like a Mills. You will also need the correct wattage, not sure what that will be.