We used to do this before investing in a ceramic filter with activated carbon. While the up-front cost is a little high, we found that the entire solution was cheaper than 6 months of bottled water, much more convenient & had a much lower environmental impact.
Obviously the reasons behind the water not being suitable are critical - ceramic filters wouldn't take out salinity, etc.
If you are don't want to connect to the mains supply, there are a number of more affordable solutions too, there are a selection of example links in this discussion: http://knowledgepoint.org/questions/838/water-purification-plants/?answer=1048#post-id-1048
If, as the question implies you do have piped water then ceramics are a very simple solution (if your water quality is suitable). I use one of these in a locally available filter housing: http://www.faireyceramics.com/filter-candles-cartridges/
This thread is public, all members of KnowledgePoint can read this page.