Why Are Complementary and Alternative Therapies Harder to Evaluate?

The treatments are assumed to be safe

Many people choose complementary or alternative therapies because they think there are no harmful side effects from them. This is not always true. One big concern is that, with alternative treatments, the delay in mainstream treatment can allow the cancer to grow and spread to other parts of the body. Another is that some complementary and alternative therapies have been reported to cause serious problems or even deaths. Even so, most of these problems are not reported to the FDA by the patient or family, so no one else hears about them. Sometimes, if the patient is treated afterward by a doctor who writes up the problem for a medical journal, there may be reports of some of the more serious effects of these treatments. But it’s clearly impossible to be sure that all of the side effects are reported and published.

We do know that certain vitamins and minerals can increase the risk of cancer or other illnesses, especially if too much is taken. But when it happens to one person, it’s very easy to miss any link between the illness and the supplement. Large groups of people must be studied to find out about small increases in risk.

Some companies don’t follow the FDA’s rules about making claims and labeling supplements properly. In many cases, when herbal supplements have been tested, they were found to contain very little or none of the listed ingredient. Although studies aren’t done very often, when they are, they invariably find that many herbal supplements don’t contain what’s on the label. Some even contain contaminants that could be harmful to certain people.

It has long been known that if a company does not carefully control the growing, harvesting, and manufacturing processes for their product, it’s possible for harmful contaminants to get into dietary supplements. Serious illnesses and even deaths have resulted from these kinds of problems. For more information on dietary supplements, see Dietary Supplements: What is Safe?

Finally, some of these companies know that they’re operating illegally. Some will put man-made drugs into an “all natural” supplement, and when discovered, will move to a country where regulations are more lax than the United States. Or they will change names and continue their practices until they are discovered again.

If you’ve experienced a serious side effect from an herb or dietary supplement, you can report it to the FDA’s Medwatch program. (See the “To learn more” section.)

In contrast to dietary supplements and alternative therapies, many of the complementary mind-body methods are very safe. It’s rare for people to have problems with activities like meditation or music therapy.

At the other extreme, some alternative biological therapies are no less toxic than chemotherapy, so safety studies are needed. Even though the details of clinical trials for testing drugs and complementary or alternative methods may differ, the basic principles are the same.

Some treatments are assumed to be effective

There are those who think that treatments derived from folk remedies that have been used for thousands of years must work. Bloodletting, for example, was enthusiastically used for more than 2000 years until data showed that it didn’t work against fevers and pneumonia.

It’s important to keep in mind that just because a treatment method has been used a long time does not mean that it works. In historical times when these treatments were all that was available, people of all ages died of illnesses that can now be prevented, treated, or cured. Still, it’s very common for people who think you should try these methods to remind you of their long history.

When scientific studies are not done, it’s hard to tell what is caused by the illness and what is caused by the treatment. Herbal treatments that are given for illnesses that go away on their own may be given credit for curing the person. Or the treatment might make the person feel better for a short time but have no lasting effect.

The expectation effect

It’s quite common for people to feel better after almost any kind of treatment that they expect to help them. This is called the placebo effect, and it’s one form of the expectation effect. The placebo effect means that if the person expects the treatment to help, he or she may feel better after getting it – even if the treatment does nothing for the underlying problem. This effect usually lasts only a short time, and seems to have something to do with the body’s own chemical ability to relieve pain or certain other symptoms for up to a few hours.

The expectation effect can also work in a less pleasant way. A person who expects a strong treatment to have side effects may notice a headache, fatigue, nausea, or other symptom even though he or she got a sham (inactive) treatment. This has been named the nocebo effect. This effect is one reason why, in the most careful scientific studies, side effects are listed for the placebo group as well as the treatment group.

There are also many other ways in which clinical trials (human research studies) can appear to show effects when there’s no actual effect. For example, studies that are very small, have high dropout rates, or aren’t randomized can seem to show effects that really don’t exist. This can happen in all kinds of research studies, not just studies of complementary and alternative treatments.

When scientific studies are not carefully designed and reported (such as when there is no control group or the participants aren’t randomized), it’s impossible to separate these kinds of effects from some of the short-term treatment effects. Even studies that have a control group but don’t use a placebo may have different outcomes than those with a placebo group.

The placebo effect, among other effects, may explain one of the reasons that people keep using certain types of complementary or alternative methods that don’t help the underlying disease. If they feel better for a few hours, it may be worthwhile for them to keep using the method as long as it does not hurt them. It does not mean that it will have the same effect on everyone who tries it, though, and it’s not expected to affect the underlying illness. 

Testing is not required by law

Although there are requirements about how dietary supplements must be made and labeled, no one requires that they be tested to find out whether they actually help, or whether they are safe to use.

This is opposite from the way mainstream drugs are handled by the FDA. With medicines, even ones that are sold over the counter, the maker of the drug must show that it works before they can ever sell it. Each drug must also be carefully tested to find out about its risks and side effects to be sure it is safe before it can be sold.

By contrast, the FDA does not require proof that dietary supplements have been tested before they are sold. The sellers are allowed to make certain general health claims without showing proof, as long as they don’t say that the product can diagnose, treat, cure, or prevent a specific disease. And the FDA regards most dietary supplements as safe until proven otherwise. The burden of proof is on the FDA to show that a dietary supplement isn’t safe, which can only happen after the product is on the market.

One of the reasons for these differences is that, unlike drugs, supplements are not intended to treat, diagnose, prevent, or cure diseases. This means supplements are not allowed to make claims such as “reduces arthritic pain” or “treats heart disease.” Claims like these can only be made for drugs that have been proven to do what they claim. Products that are proven to have a significant effect on any disease are considered drugs by the FDA and are strictly regulated.

Other methods, such as massage therapy, acupuncture, and naturopathy have come into wide use with no requirement for testing to see how well they work. Some complementary and alternative providers or methods may be licensed or regulated by individual states, but nationally there are no specific rules governing the providers or practices of a particular treatment. Of course, regulations, licenses, or certificates do not guarantee safe and effective treatment from any provider, but they can give you information and may offer more options if something does go wrong.

Good research studies take time and care

It’s hard for some people to believe that the treatments their family and friends recommend have no proof that they work. They may hear convincing stories of a person who was told they had cancer, then, after using this treatment, the patient was cured and healthy again.

Everyone wants to believe these hopeful stories. But stories about amazing cures cannot be thought of as evidence that a treatment works. Most of the time, there’s no way to be sure the stories are true. It’s often not possible to know that the person described in the story had the disease which treatment helped, or whether he or she got sick again later – it’s hard to find out if the person is still alive.

Controlled human studies ( clinical trials) are the best way to find out if a treatment works. Some of the clinical trials for complementary methods are done differently than those for drugs and other mainstream treatments. This is especially true of older studies that took place when these methods first started to be tested, although many studies have similar problems even today. Common problems include things like:

  • Sometimes the study does not have a placebo group or even a control group. In studying some methods (for example, acupuncture and some other hands-on methods), it’s almost impossible to come up with a good placebo method for the control group.
  • If there is a placebo group, the people in the study may not be randomized (that is, randomly put into one group or the other), which often leads to biased results.
  • There might not be enough people in the study to show any effect.
  • Sometimes even when there is a placebo group, the studies aren’t blinded, meaning that the researchers know who is getting placebo or standard treatment. Knowing this information can lead to biased results.

It takes time and money to get careful clinical trials done. How long a clinical trial takes depends partly on what’s being tested. For instance, it may take several years to show that a treatment helps survival, but only a few months to show that it helps nausea related to chemotherapy. Since the companies that make supplements or offer complementary therapies don’t have to carefully test them, and often don’t before selling them, it’s left to other researchers to look into their claims.

Because there are so many types of complementary therapies, it may be a long time after a treatment is first offered before studies are published that can show whether it actually helps. Because of the way studies are set up, the small early studies often suggest that such treatments work, but then later, better-constructed studies show that they really don’t. This can be confusing and frustrating when a person wants information to make a decision right now. And even for treatments that have been studied, a person trying to look it up may find old studies that seem to show it helps, and newer studies that show it doesn’t. The old information is still around, and sometimes gets much more attention. Some sellers even promote the older studies and don’t mention the new ones, which can make it hard to know what to believe.

Finding reliable studies can be a challenge

The good news is that more and more doctors and scientists are now studying complementary methods with the same careful methods used to study drugs. Results from many of these studies are often published in reliable, mainstream medical journals. As more of these studies are completed, patients and healthcare professionals will have better information to use when making decisions about these treatments.

The bad news is that some of the main sources health professionals use to look up this kind of information are increasingly tainted by journals that aren’t carefully reviewed. Articles may be accepted and published even though they may contain inaccurate information. The journal might not have the means for expert review, and may overlook certain kinds of errors. When news outlets pick up on these kinds of studies, they may not be aware of or recognize these problems.

If it’s hard to find information from a reliable expert source about a cancer treatment method, it might mean that the method has not yet been studied enough to show that it works in people.

Some researchers publish results of low-quality studies in journals with names that may sound very much like respected, well-known journals. When these journals get research studies, they might not have qualified reviewers or the rigorous evaluation process that should be used to decide if the study is credible and worth publishing. They might take the researcher’s data at face value, rather than carefully checking all of the methods and numbers. They might not require careful scientific methods for studies they publish. Mistakes can be missed because of a lax or rushed review process, and end up being published as truth. In fact, it’s been learned that a number of journals have accepted and published papers submitted by “researchers” who bought their scientific papers from commercial companies or writers. Not only does this suggest lack of careful review, it raises the possibility of bias in favor of commercial products.

Some journals that do try to publish good information may still end up accidentally distorting rather than clarifying the scientific picture. For example, studies that show a treatment gets good results are much more interesting than studies that show no difference between the treatment and control groups. Researchers often don’t bother to take the time to finalize manuscripts and submit them to journals when their studies showed no effect, and even if they do, journals are less eager to publish these results. This means that most of the studies that have useful information showing that a treatment doesn’t work are never published, and the few that do show some difference are. It also means that even the methods that don’t actually work usually will have a few small studies in the literature suggesting that they do. Even when well-designed studies are later published showing no effect, the older low-quality studies are still around. Sometimes people don’t know what to believe.

In addition, there are unscrupulous people who set up internet or even printed journals just to promote certain types of treatment. They may offer pseudo-science – statements that look and sound scientific, but aren’t. This makes it harder to learn about these treatments, since the glowing information put out by the fake journal makes it sound like it works and has few or no side effects. And many companies that offer products for sale will have “research” on their website that sounds good but was never published in peer-reviewed journals. Sometimes companies will even link their claims to real research studies in the National Library of Medicine, knowing that most people won’t read the studies – but also knowing that even if they do, most people won’t understand that the studies weren’t about their product.

With hundreds of new journals cropping up in the past few years, and the number growing every day, it can be a challenge to know which ones are reporting reliable results.

The bottom line is that, if you can’t find reliable information from researchers who don’t have a stake in the product, it’s very likely that the research proving that it works in humans has not been done. That means there’s no proof that the method works in people.

For more information on the research and testing of cancer treatments, see Learning About New Cancer Treatments.

The American Cancer Society medical and editorial content team

Our team is made up of doctors and oncology certified nurses with deep knowledge of cancer care as well as journalists, editors, and translators with extensive experience in medical writing.

Last Revised: March 31, 2015

American Cancer Society medical information is copyrighted material. For reprint requests, please see our Content Usage Policy.