Editor’s Note: We usually think of Stephen Downes as a philosopher, scholar, and editor of OLDaily, a rich source of current information and blogs related to teaching, learning, technology, and whatever else is important for education and training. We accept as normal his professional five-day-a-week newsletter and resource-full (sic) Web Page. This article is one of his more down-to-earth essays about something 0f special importance for teachers, learners, and editors of websites.

Tutorial Paper

Principles for Evaluating Websites

Stephen Downes
July 16, 2005

How do you know whether something you read on the web is true? You can’t know, at least, not for sure. This makes it important to read carefully and to evaluate what you read.

This guide will tell you how….
 

1. There Are No Authorities

Authorities used to be people you could trust. When you read it in the newspaper, for example, it was probably true. When a scientist reported a finding, you could count on it. But today, you can’t trust the authorities.

Why not? There are many reasons, but here are some of the major ones:

• Authorities lie. Not all authorities, and not all the time, but frequently enough to mean you can’t simply trust them.

• People impersonate authorities. A site may look like a newspaper or a government publication, but it might not be.

• Authorities are sometimes fooled. They may rely on bad data. They may be reporting something they heard.

Even if you trust the authority you are reading, you need to evaluate what they say for yourself. People don’t always mean to mislead you, but they do.

This is the most important principle of reading on the internet. You must determine for yourself whether or not something is true.

2. What You Know Matters

If you saw the local grocery this morning, and then someone told you it burned down last night, you would know they were wrong because of what you saw. And you would probably say so.

You can depend on your own knowledge. And you should use this knowledge when you read websites.

That doesn’t mean that you cannot be wrong. But most people don’t give themselves enough credit. They are too quick to assume that they must have been wrong.

Your own experiences matter. If someone says some software is easy to install, and you found that it wasn’t that easy to install at all, don’t simply assume that you can’t install software. If it wasn’t easy for you to install, it wasn’t easy, and someone who says it is easy is wrong.

3. Keep Count

You can’t check everything for yourself. Eventually, you will have to depend on what other people say. You can’t simply assume that what they say is true.

The key here is trust. You need to learn who to trust.

The way you learn to trust someone is through repeated contact. They not only say things you know are true, they don’t say things you know are not true. You need to keep track of this for yourself.

When a website says something, you need to ask yourself, have they misled me before? Websites usually follow a pattern; sites that are trustworthy generally stay trustworthy, while sites that mislead you once will likely mislead you again.

That doesn’t mean you never question what they say. Always check what they say against your own experience. But if you don’t know, depend on the sites you already trust rather than the ones you don’t.

4. Facts and Appearances

Many people are very careful appearances. Governments and businesses especially take great care to manage their image. Individual people, too, try to cast themselves in the best light possible.

They do this because people trust people who look good. Politicians always take care to dress nicely. Con artists are often dressed in suits. Businesses spend a lot of money to make their buildings and their websites look nice.

People create appearances in words as well. For example, they often use adjectives and adverbs to suggest how you should feel about something. They also use loaded terms to suggest that something is good or bad. Compare the following:

• "This respected software reliably saves your data in the most efficient format."

• "This suspicious software misleadingly saves your data in a common format."

The first software sounds a lot better than the second software. But in fact, they do exactly the same thing!

In your mind, remove the adjectives and adverbs from any sentence you read. Convert any loaded terms to neutral terms (for example, convert a sentence like "He claimed..." to "He said…").

In other words, practice distinguishing the facts in a sentence from how they appear.

You may be tempted to distrust things that use a lot of adjectives, adverbs and loaded terms. And certainly you should be suspicious. But sometimes people just write that way; it doesn’t mean they’re lying. And sometimes people try to fool you by writing in plain and straightforward language.

The main thing is, find the facts. You can check facts. And just ignore the appearances.

5. Generalizations Are Often Untrustworthy

When you look at facts, you will see that there are two types: specifics and generalizations.

• A specific is a statement about one thing, one person or one event. "John went to the store yesterday" is a specific.

• A generalization talks about a group of things, many people, or a number of events. "John always goes to the store."

People use generalizations because generalizations help them predict the future. If you know that John always goes to the store, then you can predict that he will go to the store tomorrow. Generalizations also often explain why something happens. John knows the shopkeeper because he always goes to the store.

There are two types of generalizations:

• A universal generalization talks about everything. When someone says "All dogs are animals", for example, they are talking about every single dog.

• A statistical generalization talks about a number of things, but not all of them. When someone says "Most dogs are brown," they are talking about a large number of dogs, but not all of them.

It is important to keep in mind that most universal generalizations are false. Not always – after all, it is true that all dogs are animals.

But people often make universal generalizations that are false. And in fact, when you read universal generalizations on a website, you should be very skeptical.

Watch for the following words: all, none, only, never, always, completely. And words that mean the same sort of thing. These indicate a universal generalization. When people use them, ask yourself, is this true? Are there no exceptions? And if you know that there are exceptions, then the source is less trustworthy.

6. Absolutes Are Hidden Generalizations

People often make generalizations without realizing that they are doing it. And they might fool you into thinking that something is a fact, when it is actually a questionable generalization.

"The Chinese cannot be trusted." This looks like a statement of fact, doesn’t it? But ask yourself, how many Chinese people is this person talking about? All of them? Most of them? There are a billion Chinese – how could this person possibly know that they cannot be trusted?

And of course, they can’t. You have no reason to trust such a statement. And a person who makes such a statement is less trustworthy.

7. Statistics Are Often Misleading

As the truism says, "There are lies, damned lies, and statistics." People are often skeptical of statistics, and for good reason. There are many ways statistics can be used to mislead.

Statistics must be based on data. For example, for somebody to say that "most dogs are brown" they would have had to go out and actually count some dogs to see how many of them are brown. Statistics that are not supported with data should not be trusted at all.

Even if there is data, statistics can still mislead. There are two major ways statistics can mislead:

• The sample size is too small. If you know five Americans, and four of them are crooks, is that sufficient to conclude that most Americans are crooks? Of course not. There are 330 million Americans; you need to meet more than five before you can start making generalizations.

• The sample is unrepresentative. If you wanted to know about Americans, and took your sample from a prison population, would you get a good result? Of course not – most Americans are not in prison, and are very different from prisoners.

Remember at the beginning of this article where I said that there are no authorities? When you look at the statistics produced by authorities, many of them break one of these two rules. What would you say about a scientist who surveyed 21 graduate studies and drew a conclusion about all people? Not much – but many papers that do exactly this are published.

Statistics are often misleading in ordinary writing as well. Often, they are disguised: a person might use words like 'most', 'often', 'many' or 'usually'. And their data will be suspect. A person might say, for example, "Most people are generous." How does he know? Because most of the people he knows are generous. But that’s not good data at all!

Think about the generalizations you believe. Are they based on good data? What is the data? I said above that you should trust yourself – but you should always review your own beliefs, to make yourself more trustworthy.

8. Go to the Source

People say things about other things and other people. That’s no surprise; you can’t talk about yourself all the time. For example, a person might report about what someone else said, or about what some data shows.

They may not mean to mislead you, though sometimes they do:

• They might have misread or misunderstood the original document. Heck, I do that myself.

• They may have quoted something out of context. For example, I may have written, "If people vote the wrong way then we'll have private health care" and be quoted as saying "We’ll have private health care."

• They may be misrepresenting the original. People sometimes pretend that someone said something that they didn’t, so they can make the other person look bad (that’s called a straw man).

When you read something you always need to ask, are they talking about something else and especially what somebody else said or reported. If so, go to the source to find out for yourself what the other person really said.

If there's no link or reference to the source, don't believe it. And even more importantly, websites that don't offer links or references are less trustworthy.

If you can't find the original source, try searching for the same information. Other people may have seen the same source and reported on it themselves. They may have described it differently. You may never know exactly what was said, but if people on different sides of the same issue agree on what was said, then it's more likely to be true.

9. Motives and Frames Matter

Most content on the web is trying to convince you that something is true. That’s why it’s on the web in the first place.

Usually, what they want you to believe isn’t just some isolated fact or data, but rather a whole collection of facts and data. They want you to see the world in a certain way. In philosophy, this is sometimes called a ‘world view’ while in linguistics this is called a ‘frame’.

Here are some examples of frames:

• It's a dangerous world and we have a lot to fear

• Microsoft products cannot be trusted

• Our country is the best (most free, most democratic, most advanced, etc.)

Think about all the sorts of things that could lead you to believe any of these three statements. Think about other sorts of things that might also be frames. Think about the way you look at the world – you probably view it from a certain frame, whether or not you recognize it.

That's not bad in itself – we all have to have a way of looking at the world. But we need to choose this way of looking at the world for ourselves. That’s why we need to understand what frames other people believe, so we know when we are being persuaded to look at the world one way or another.

That's why motives matter. A person’s motive is the frame or worldview he or she wants you to accept. You need to know why somebody is telling you something as well as what they are telling you.

Websites that hide their motives are untrustworthy. They are trying to convince you of something, but they are trying to do it in a sneaky way, so that you can’t make your decision for yourself. They think that if you just hear something over and over, and it all points to a certain way of looking at the world, that you will start seeing the world that way too.

If a website is sponsored by the government, but they hide this sponsorship, then they are hiding their motives. If a study is financed by a software company, but this financing is not revealed, then they are hiding their motives. If a news site is secretly sponsored by a religious organization, then the news site is untrustworthy. If an activist group is funded by the industry they are trying to change, then this group is untrustworthy.

They are not untrustworthy because what they are saying is false. They are untrustworthy because they are not being honest about why they are saying what they are saying.

10. Beware Misdirection

Have you even seen a political ad for one candidate that talks about the other candidate? Have you ever seen an advertisement about one product that only talks about another product?

These are cases of misdirection – they are trying to get you believe one thing by talking about another thing.

Misdirection is very common on the web. Sometimes it consists of misrepresenting the source, as discussed above. Very often, though, it consists of merely attacking the source.

You see this not only on discussion lists (where it is very common) but also on personal websites, corporate websites, political websites and even academic websites.

If a website is trying to convince you to believe one thing but actually talks about another thing, then the website is not trustworthy.

Summary

As I said in the second point, determining what to believe – or to not believe – is a matter of trust. You need to determine for yourself who to trust about what.

This is something you have to determine for yourself. Each time you look at a website, think of yourself as keeping score. When a website does something untrustworthy, take some trust away. When a website does something well, add some trust.

And it’s something very personal. The better you get to know a website, the more easily you can determine whether or not to trust it. The website gradually acquires a track record with you. Just like a friend or an associate.

And finally, this is something that works best if you use diverse sources. Try to read points of view from different frames – after all, every frame has an element of truth to it. Don’t just go with the flow, be ready to challenge and question everything – even yourself.

Examples

40 Things That Only Happen In Movies

Should you trust this site? The title should let you know that this is intended as humour. But if not, you should be alerted by the universals in this title. They are probably exaggerating to make a point.

Look at some of the assertions. “(In movies) any lock can be picked with a credit card or paperclip in seconds.” Well you know that this isn’t true. People don’t always pick locks in movies. Sometimes they can’t even break the door down.

This site is funny. But you shouldn’t trust it to tell you true things about the world.

Top Chinese general warns US over attack

This news article is offered by the Financial Times, a British news source with strong links to the British and American financial communities. The story reports that a Chinese general said that China would use nuclear arms if attacked.

Did the general say this? Probably. The general is named - Zhu Chenghu – and the place where he made the remark is also named - a function for foreign journalists (it would be better if they actually named the function and told us who else, in addition to the Chinese government, sponsored it). And a one-minute search in Google for ‘Zhu Chenghu’ links to other reports – from the BBC and the Times of India, for example – with the same information.

Is what the general said true? We have no way of knowing. Even the Financial Times article notes that Zhu is not a high-ranking official and that “Gen Zhu probably did not represent the mainstream People's Liberation Army view.” Coverage elsewhere, for example in the BBC, reports that the Chinese government is ‘downplaying” the remark.

So now the key question is, why did the Financial Times run the article? The article is intended to shape our views even if we cannot know whether what was said was true. Does it make us fear China more? Do the British and American financial communities stand to gain if readers fear China or become more concerned about nuclear war? Does this article fit a pattern in Financial Times coverage of China?

In my opinion, this article, although an accurate report, makes the Financial Times a bit less trustworthy.

Iraqis March Against Terror

This article is found in a blog titled BlackFive. It tells us that about 1000 Iraquis in the city Qayarrah, Iraq, marched against terror, and that “you probably haven't heard about it from Peter Jennings or Dan Rather.” The post includes a number of photographs of the demonstration taken by "Army Specialist David Nunn."

As one person commented, "Rather retired early in the year and Jennings has been off battling lung cancer for months." However, a search in Google shows that the protest was not covered by any major news outlet.

That a protest did happen seems evident from the pictures. Examination of the pictures, however, shows the banners to read "The juboor's tribe and its allies ask the coalition forces to release the highly-ranked officer Farhan Muthallak who was imprisoned by the coalition forces" in both English and Arabic.

A Google search for "Army Specialist David Nunn" reveals no citations not associated with this particular story.

This story is very untrustworthy. It reports a protest for one thing as a protest for something else. The source of the photographs cannot be verified. It attempts, further, to discredit the news media, thus engaging in misdirection. The site (and other sites, for many sites ran this item) is much less trustworthy as a result of running this item.

It is worth noting – as demonstrated in the trackbacks – that this story has been widely circulated. This is common, even for untrustworthy stories. That is why it is important to read, not only numerous source, but also diverse sources. And to check the data for yourself.

Again, one should ask why such a blatantly misleading story achieved such wide circulation.

Secure RSS Syndication

This site suggests that there is a need for encrypted RSS feeds and demonstrates how it is done. The need expressed is the author’s own, and two potential solutions are considered and rejected. The code used to generate the encryption is provided, along with samples of the encrypted data.

This article is very trustworthy. Very specific information is given, and in a form (via computer code) that can be directly verified by the reader. It should be noted that one argument ("Atom isn’t finished") will cease to be true at a future point; if you were reading this article after Atom is finished you would want to check to see whether it satisfies the need as well.

This article is supportive of the idea that encrypted content syndication is a good idea. This suggests that the author may have an interest in promoting commercial applications of content syndication. But such a conclusion should not be drawn without looking at a large number of other items written by the same author.

Bastille Day

This is a Wikipedia article about Bastille Day. Readers should note that Wikipedia articles frequently change. This article was current at 11:40 a.m. EDT, July 16, 2005.

The article begins, "Bastille Day is the French national holiday, celebrated on 14 July each year" and provides some background. This information can be verified from numerous sources using a quick Google search on ‘Bastille Day’. Much of the background and information is substantiated by other sources.

The article next contains the comment, "Margaret Thatcher once said of the French 'who can trust a people who celebrate, as their national event, a jailbreak'." This statement does not tell us about Bastille Day. It is derogatory to the French. The source of the quotation is not given. This statement may be disregarded as vandalism. (It is worth noting that as of 11:47 a.m. the statement had been removed.)

This article, with the exception of the one item noted, is trustworthy.

The Price is Right pricing games

This is a Wikipedia article about The Price is Right. Readers should note that Wikipedia articles frequently change. This article was current at 11:51 a.m. EDT, July 16, 2005.

The article lists a number of ‘minigames’ played on The Price is Right. Each game is described, with in formation about when it was played, how frequently it was played, and records, if applicable. Three external sources, including one from CBS, the producer of The Price Is Right, and one with screen shots of the games, are provided.

Readers who have seen The Price is Right can verify the game descriptions for themselves from their own experience. From my perspective (having seen many of the games) this article is very trustworthy.

The Flight of the Bumblebee

This is a video of a person playing Flight of the Bumblebee solo on guitar. The video is sufficiently detailed to show the fingering. The sound is a guitar sound. The tune is recognizable as Flight of the Bumblebee (people who have not heard this piece of music before should consult alternative sources to verify the title).

This video is trustworthy.

About the Author