Rigorous Documentation

Script / Documentation


Welcome to Just Facts Academy, where you learn how to research like a genius.

Remember, you don’t have to be an Einstein to be a great researcher—you simply need to put in the effort and apply the 7 Standards of Credibility that we share in this video series.

In today’s lesson, we look at Rigorous Documentation.

I’ll explain its importance, provide a real-world example, and show you how to apply it in your own research.

As we begin, let’s take a slight philosophical turn. I’m sure you’ve pondered great mysteries like: If a tree falls in the forest, but no one is around, does it make a sound?

Now, let me ask you this: If you share a fact, but no one believes you, does it matter?

Of course, facts matter regardless of who accepts them, but when people don’t know or refuse to believe facts, bad things can happen. To borrow the words of Harold Bowden, the illustrious CEO of the Raleigh Bicycle Company,[1] “Facts that are not frankly faced have a habit of stabbing us in the back.”[2]

So, to inform others effectively, you must provide evidence that the facts you are sharing are—in fact—facts. It matters whether you are writing for a teacher, your boss, your subordinates, a public official, or your colleagues.

And this is where rigorous documentation comes in. It helps you get the facts right and quickly proves to others that you are accurately representing your sources.

Anyone can make a claim, but proving it separates real research from posing. This applies regardless of who you are.

Consider, for example, a New York Times column by Nobel Prize-Winning Ph.D. economist Paul Krugman, in which he wrote that total government spending fell in 2009 and 2010. He said this occurred because cities and states underwent “drastic spending cuts, more than offsetting the modest increase at the federal level.”[3] [4]

Krugman provided no numbers to support that claim, but if he took the time to document them, he would have found the exact opposite to be true. In fact, the only comprehensive source of such data at the time showed that total government spending rose by 10% over this timeframe,[5] while inflation was only 4%.[6]

You may not learn this from your professor, but a degree or prestigious award cannot give people the magical ability to create real data out of thin air.

Everyone should pay attention to this lesson: Don’t make claims that you haven’t documented. And make it easy for others to check your work by mentioning and preferably linking to the source—which Krugman failed to do in this case.

This simple step makes us accountable to both ourselves and others.

That’s only basic documentation. Rigorous documentation takes it up a notch and makes your research truly believable and credible. This involves quoting the actual numbers or words that support your claim. It’s not complicated. We just did it—by showing the exact figures for inflation and government spending from the sources we cited.

Now, why should you do this? Because people sometimes cite a source to support their point, and the source doesn’t actually provide that support. This is called a “citation bluff.”

Check out this revealing example: Dr. David Katz, Director of the Yale Prevention Research Center, wrote the following statement in an op-ed for the Huffington Post: “Semi-automatic weapons with high-capacity magazines almost never figure in effective self-defense.”[7]

As you can see, he included a link to support that claim. However, what you can’t see is that the link leads to Just Facts’ core research on gun control, which said nothing about this topic at the time. In fact, it didn’t even mention magazine capacity, much less how it relates to self-defense. Moreover, Just Facts’ research was about 100 pages long, making it very difficult for anyone to discover that the source Katz cited does not support his claim.[8]

There’s an easy way to quickly show your readers that you are accurately representing your sources and not using citation bluffs: Directly quote the sources.

These don’t need to be lengthy quotes, because even a snippet of several words allows your readers to search your sources and see if you got it right.

This also obliges you to double check your own work and avoid embarrassing errors—or worse yet, spreading misinformation that can harm yourself and others.

So, how do you apply the Rigorous Documentation standard in your own research?

First, document every claim you make, and don’t trust the undocumented claims of others. If someone makes a statement without offering proof of it, your antennas should go up, regardless of their credentials.

Similarly, your Spidey Sense should tingle when you hear phrases like “experts say,” because this can actually mean “the experts I have chosen say.” The same goes for phrases like “science says.” Real science is about empirical facts, not the claims of selected scientists or slogans that use the word “science.”[9] [10] [11] [12]

Second, don’t use citation bluffs—and don’t fall for them. How? Watch the Just Facts Academy video on Primary Sources and use this hack: When information is in a digital format, use the FIND function (Crtl+F on PCs and Command+F on Macs) to quickly search through lengthy documents and find what you are looking for—or see if you’re sources are accurately representing their sources.

Third, directly quote your sources. This not only proves that you read the source, but it helps people quickly find the context of your quote. It’s okay to use short quote snippets, but make sure you paraphrase the rest of the words accurately. A good way to do this is to temporarily place the words of the source right next to your words. Then carefully compare them to make sure that you’re communicating the same thing.

Fourth, if you perform calculations, do them in a spreadsheet and make it available. It’s quick and easy to share spreadsheets via online file-sharing services.

Fifth, save copies of your sources. Even if your assignment doesn’t require this, it keeps the source material handy, and it provides evidence if the source disappears from the internet, which is not uncommon.

And lastly, use credible sources. For more about this, check out Just Facts Academy videos about Primary Sources, Comprehensiveness, and Raw Data.

If you apply these practices, you will increase your ability to accurately inform others. Furthermore, others will quickly see that your research is worth paying attention to.

So use the Rigorous Documentation standard and the rest of Just Facts’ Standards of Credibility, so you can research like a genius.


Footnotes

[1] Webpage: “Our History.” Raleigh Bicycle Company. Accessed June 15, 2021 at <community.raleighusa.com>

Back in the late 1880’s, co-founder of the Raleigh Bicycle Company, Frank Bowden wanted everyone to find the simple happiness that came with riding a bike. He made good on that wish by transforming a small shop on Raleigh Street in Nottingham, England into the largest bicycle manufacturer on the planet. …

1932

Sir Harold Bowden, son of Sir Frank Bowden, and chairman/chief executive of Raleigh Bicycle Company retires after mean [many] years of strong leadership and devotion to the company and the world of cycling. Production had reached a staggering 62,000 bicycles a year.

[2] Article: “In the Public Eye.” The Guardian (Catholic Diocese of Little Rock, Arkansas), March 16, 1929. <arc.stparchive.com>

Page 2: “Facts that are not frankly faced have a habit of stabbing us in the back.—Sir Harold Bowden”

[3] Commentary: “Hey, Small Spender.” By Paul Krugman. New York Times, October 10, 2010. <www.nytimes.com>

Here’s the narrative you hear everywhere: President Obama has presided over a huge expansion of government, but unemployment has remained high. And this proves that government spending can’t create jobs.

Here’s what you need to know: The whole story is a myth. There never was a big expansion of government spending. …

… Of the roughly $600 billion cost of the Recovery Act in 2009 and 2010, more than 40 percent came from tax cuts, while another large chunk consisted of aid to state and local governments. Only the remainder involved direct federal spending.

And federal aid to state and local governments wasn’t enough to make up for plunging tax receipts in the face of the economic slump. So states and cities, which can’t run large deficits, were forced into drastic spending cuts, more than offsetting the modest increase at the federal level.

[4] Webpage: “Paul Krugman.” New York Times. Accessed June 14, 2021 at <www.nytimes.com>

Paul Krugman joined The New York Times in 1999 as an Op-Ed columnist. He is distinguished professor in the Graduate Center Economics Ph.D. program and distinguished scholar at the Luxembourg Income Study Center at the City University of New York. In addition, he is professor emeritus of Princeton University’s Woodrow Wilson School.

In 2008, Mr. Krugman was the sole recipient of the Nobel Memorial Prize in Economic Sciences for his work on international trade theory.

[5] Calculated with the dataset: “Table 3.1. Government Current Receipts and Expenditures [Billions of Dollars].” U.S. Department of Commerce, Bureau of Economic Analysis. Revised September 30, 2010. Accessed 10/26/2010 at <www.bea.gov>. Currently located at <apps.bea.gov>

Total expenditures …

2008 [=] $5,020.2

2009 [=] $5,344.9

2010Q1 [=] $5,471.6

2010Q2 [=] $5,591.9

CALCULATIONS:

  • 2010 first half average = ($5,471.6 + $5,591.9) / 2 = $5,531.8
  • Increase at time of Krugman’s claim: ($5,531.8 – $5,020.2) / $5,020.2 = 10.2%

[6] Dataset: “CPI—All Urban Consumers (Current Series).” U.S. Department of Labor, Bureau of Labor Statistics. Accessed October 26, 2010 at <www.bls.gov>. Currently located at <www.bls.gov>

Consumer Price Index …

December 2007 [=] 210.036 …

June 2010 [=] 217.965

CALCULATION: (217.965 – 210.036) / 210.036 = 3.8%

[7] Commentary: “Bullets for Gun Control.” By David Katz, M.D (Director, Yale Prevention Research Center). Huffington Post, January 8, 2013. <www.huffingtonpost.com>

If a nation of reasonable citizens—of loving parents and grandparents—can get organized, we really should have a shot at overcoming the NRA/gun-seller alliance and achieving a semblance of sane gun control in this country. We really should have a shot!

We certainly have the bullets …

• Semi-automatic weapons with high-capacity magazines almost never figure in effective self-defense.

[8] An archive of Just Facts’ gun control research at the time the claim above was published is available at <bit.ly>

[9] Entry: “science.” Random House Kernerman Webster’s College Dictionary, 2010. <www.thefreedictionary.com>

1. a branch of knowledge or study dealing with a body of facts or truths systematically arranged and showing the operation of general laws.

2. systematic knowledge of the physical or material world gained through observation and experimentation.

[10] Article: “Richard Feynman.” Encyclopædia Britannica Ultimate Reference Suite 2004.

“American theoretical physicist who was widely regarded as the most brilliant, influential, and iconoclastic figure in his field in the post-World War II era.”

NOTE: See the next two footnotes for Feynman’s pragmatic definition of science and his description of how people misuse this word.

[11] Book: Six Easy Pieces: Essentials of Physics Explained By Its Most Brilliant Teacher. Addison-Wesley, 1995. (Comprised of six chapters from the book Lectures on Physics by Richard Feynman. Addison-Wesley, 1963.

Page 2: “The principle of science, the definition, almost, is the following: The test of all knowledge is experiment. Experiment is the sole judge of scientific ‘truth’.”

[12] Lecture: “What is Science?” By Richard Feynman. Presented at the fifteenth annual meeting of the National Science Teachers Association, 1966. <www.feynman.com>

When someone says, “Science teaches such and such,” he is using the word incorrectly. Science doesn’t teach anything; experience teaches it. If they say to you, “Science has shown such and such,” you might ask, “How does science show it? How did the scientists find out? How? What? Where?”

It should not be “science has shown” but “this experiment, this effect, has shown.” And you have as much right as anyone else, upon hearing about the experiments—but be patient and listen to all the evidence—to judge whether a sensible conclusion has been arrived at.