A clean and simple way to switch power on small loads can be achieved with a reed relay and a small magnet.
The white thing is the actual reed switch. It is encapsulated in a convenient watertight plastic body. This cheap component is normally used to detect opening of doors in burglar alarms, or to count wheel revolutions for bicycle computers... See also: en.wikipedia.org/wiki/Reed_switch
The reed switch has a smal piece of iron glued to it. This is for the magnet to stick to it.
The small metal disc in the right picture is a magnet. It closes the reed switch when close to it. The magnet is thus used as 'power on key'. The magnet stays there even if the switch is shaken.
To power of: remove the magnet (and place it on another iron place not to loose it)
Note: I got as set of 10 switches and 10 magnets for 12$ on eBay, from China with free shipping :-)