Solitary wasps are hard to miss. Their long slender abdomens and unique flying style make them very apparently different from other flying stinging insects, like hornets and bees. Mud daubers (family Sphecidae), potter wasps (family Eumenidae), and pollen wasps (family Masaridae) are three of the most common solitary wasp families.
Mud dauber
Potter wasp
Pipe organ nest

There is a wide variety of wasps that are commonly referred to as solitary wasps. All this means is that singular females mate with male wasps and then build nests and lay their eggs on their own. This is very different from the mating and rearing habits of the more familiar (and dangerous) social wasps. Once the solitary wasps have made their nests and laid their eggs, the females abandon them. This too is unlike the social wasps and may be partly why the solitary wasps lack that aggressive protectiveness. So why control them? If you don't like where they are building their nest or have an extreme allergy to stings, controlling these wasps seems very reasonable. Otherwise, I would just let them be.
What kind of wasp is it? One of the easiest ways to tell is to look at their nests. If the nest is small, gray, umbrella-shaped, and made of paper, it is a paper wasp of some kind. If it is larger, gray, round or football-shaped, and made of paper, it is some kind of hornet (run away). If it is gray and looks like a blob of mud or concrete, it is probably just a solitary wasp. If you can remove that nest without crumbling it, you'll see that that blob actually contains one or more carefully constructed chambers. It's probably packed with a bunch of paralyzed spiders, and depending on the level of development, a wasp in its egg or larval form.
Physically removing the nest is the best control method. There really isn't a way to prevent these wasps from building their nest somewhere. The easiest way to take care of them is to destroy the nest with some kind of scraping tool as soon as you notice it. It destroys the next generation and helps to cut down on the number of potential wasps around your house. If the nest isn't completed and the mother wasp is still around, there's a risk that she will get mad at you. But they are not nearly as aggressive as their more sociable cousins. And since there is just the one, it isn't so hard to just deal with her and be done with it.
Sure, you can kill the wasp with a poison spray. As mentioned above, the solitary wasps won't normally come after you, so you could just as well ignore them. However, if you dislike them building nests on your house, killing them is one option for control. The thing to remember is that when that wasp has built her nest and laid her eggs, she doesn't come back. So make sure you are destroying the nest along with the mother to ensure that there aren't any future infestations—at least from those particular wasps. You could use just about any wasp spray, but often the long-range stuff is the easiest so you can keep your distance. A flyswatter or rolled-up newspaper will also work.
Wasps inside of holes require a slightly different tactic. Not all solitary wasps make their nests out of mud. Some dig, some scrape or chew holes in the ground or wood, and some use ready-made holes like those of a carpenter bee or rodent. Once they are inside the hole, spraying a liquid insecticide inside might not work so well because it's hard to say which direction the passage leads. A lot of professionals use a fumigator, which gets the poison in the air so it penetrates deeper. Another option is to spray a dust-based insecticide (like Sevin) into the hole. The powder will hang in the air and coat surfaces. Seal off the hole after application and watch for future activity.
Eliminate their food source. In the case of most solitary wasps, as adults they feed off of nectar and pollen, or sugars and proteins, like most wasps. That means that keeping your yard free from excessive sources of sugar, like hummingbird feeders, garbage, or decomposing fruit from fruit trees, should help deter wasps from moving in. Another thing to consider is that solitary wasps like to paralyze and store spiders and other insects in their nests for their young to eat. It is nearly impossible to get rid of all spiders (encouraging solitary wasps is one way to cut down the population), but you can try to make spider-prone areas, like barns and garages, less accessible to wasps.
If you've been stung by one of these solitary wasps, you were most likely doing something you shouldn't have done, like trying to touch or otherwise molest the poor creature. They mostly use their stinger for paralyzing and gathering food and some species even skewer and carry their prey on the stinger. However, they will also use it in defense if excessively provoked. Generally speaking, if you leave them alone, they will leave you alone. They are a beneficial part of the ecosystem in that they eat other flying insects and small spiders. One species even targets the dangerous black widow spider specifically! The point is, if at all possible, give solitary wasps a break. Just let them do their thing. They are not the threat that their social wasp cousins are—and remember you can always scrape their nest off in the colder months when they aren't using it. This allows them to propagate, but it also allows you to maintain whatever aesthetic you might be seeking for your home.

Power washer. One easy way to remove mud or clay wasp nests from your wall is to use a power washer or hose. This eliminates the need for poisonous sprays and also allows you to maintain a fairly safe distance, just in case the mother wasp decides to take offense to your activities.

Scraping tools. Another easy nest removal method just involves grabbing a trowel, crowbar, chisel, or screwdriver and carefully prying or scraping the mud from the exterior of your house. If you're the scientific type, open up the cylindrical chambers and look at what you find inside. Another option would be to relocate the nest.

Insect screen. Some mud dauber species like to build inside of vents, tubes, or holes in your house. They've even been blamed for major airline accidents caused by mudding up aircraft instrumentation. If you don't want them going inside something, cover it with insect mesh. Just make sure that the mesh doesn't interfere with the function of the vent.