One of the central issues for small government people, such as classical liberals, libertarians, and many conservatives, is government failure. While the standard arguments for big government purport to identify market failures that justify government intervention, small government people emphasize government failure. For small government people, the government regularly engages in activities that constitute government failure. The government program does bad things, causing harmful results or producing beneficial results, but at excessive costs.
Ordinary Democratic voters and many Republican voters see government agencies as basically doing a good job. When they see the FDA, many people simply picture an agency that protects us from unsafe and ineffective drugs. Many other people, while more skeptical, still see the FDA as basically doing good, even if at times it makes mistakes or pursues inappropriate policies. It is the rare person who sees the FDA as fundamentally misconceived, as requiring a radical restructuring where it has far less coercive power and is much more of a certifier than a regulator.