There are some inherent drawbacks to nuclear and antimatter propulsion (unfortunately), and not all of them technical. The largest hurdle to overcome in nuclear propulsion is political and environmental. To date, manufacturers and governments with space probes and satellites utilizing SNAP generators and their descendants have kept the power source as low key as possible and generally off the radar of environmentalists. Commercialization of nuclear propulsion will bring with it a backlash from constituencies citing radiation concerns, be they founded or not, particularly associated with a potential accident during boost phase or an unplanned reentry of a propulsion core. Fear drives governmental decisions far more than reality.
Antimatter propulsion would generate the necessary energy to allow us to journey to all the planets in the solar system but has a couple of technical drawbacks that make it less than an ideal fuel. The first limitation is our ability to create antimatter. In the 40 or so years that accelerators have been creating antimatter the total mass stored (at CERN and a few other labs) is on the order of 10[sup]-12[/sup]kg. There is no practical way at this time to make antimatter in quantity. Furthermore, most of the mass of antimatter has to remain as anti-protons or positrons so they can be magnetically isolated from common matter. This brings me to the other technical limitation of antimatter. Keeping large quantities of antimatter contained together is a recipe for disaster particularly under accidental circumstances. If you want to know how much energy is available in a kilogram of say, anti-protons, simply run Einsteins famous equation E=mc[sup]2[/sup]. Your answer will be in Joules. Keep in mind that m or mass must be doubled as a kilogram of anti-protons requires a kilogram of protons to annihilate. Figure that a nuclear weapon using a 5-8 kilogram core is only 1-2% efficient when detonated. Antimatter annihilation is 100% efficient. It does not represent a practical fuel.
Photon pressure and ion pressure rockets seems to be the next generation of propulsion but we will still rely on chemical rockets to reach escape velocity for many, many years to come.