Ranking the rankers: Just what do those Top 10 lists mean?

By  |  0 Comments



By Patti F. Smith

Ann Arbor is a great place to live, as many residents will agree. And if you read the “best of” lists that are put out every year, you’ll see that the people behind them think Ann Arbor is a very great place to live. 

Just this year, Ann Arbor has already been on at least 10 “best of” lists. But what do these lists really mean? How are they compiled? Is the data qualitative or are these things just listicles using subjective opinion? Can you trust any of them?

The Ann has just the lists to help guide you through the lists.


Top 5 reasons to be skeptical of lists

How can we be tops on a list one year, then fall several places — or  completely off the list — the next year? Are we the best city to live in, or is it really Seattle, San Francisco, Madison, Wis. …? 

The rankings depend on the methodology and sources used by each outlet. Richard Eisenberg, who created Money’s first “best places” compilation, offered an inside look at the lists in an article he wrote in 2014 on nextavenue.org: 

1. They only include certain cities. And the criteria can change from year to year. Money, for example, sometimes ranks cities having between 50,000 and 300,000 residents. Other years, it ranks populations of 20,000 to 350,000. In other words, if a city has 35,000 people, it might not even be considered.

2. They exclude other cities. Some list-makers want to spread the geographic love, and thus will eliminate cities located near municipalities already on the list. And there are other factors for exclusion; Money, for example, omits cities with incomes 80 percent or more below the state average, those with “poor educational scores,” those where more than 95 percent of residents are of a single race, and cities located more than an hour from the airport. After the first round of cuts, it then omits a city if its income is 210 percent or more above the state average, or where the median home price is $1 million or more. After these eliminations, generally about 100 cities remain to be ranked, out of almost 800. 

Another list-maker, 24/7 Wall Street, uses a system similar to Money’s. It eliminates communities with unemployment rates of 9.8 percent or more, crime rates 25 percent higher than the national rate, and where one racial group comprises more than 90 percent of population. It also excludes a city if it is too close to another ranking city. 

3. There’s a lack of consistency. While most websites explain their methodology, they typically don’t explain why some factors are weighted higher than others, or why some cities were eliminated. For example, many outlets consider many of the same criteria — economics, crime rate, employment — but might use different sources of data and weight them differently. 

They also get to subjectively decide if some results simply don’t count. Websites such as Livability use their own surveys in addition to standard types of data. Livability adds a weighted score to its own national survey that asks people what makes their communities better places to live, and what factors they would consider if moving to another community. 

4. The sources may be questionable. A number of the lists that ranked Ann Arbor use the same sources — U.S. News & World Report, the Census Bureau and the Bureau of Labor Statistics. Some, including Forbes, rely on the FBI’s Uniform Crime Report, which even the FBI says should not be used to rank locales, because cities voluntarily report the data and many choose not to participate.

5. The ranker may have less than stellar motives. It’s important to look at where the sources get their funding and how long they’ve been around. Forbes, for example, has been around since 1917, whereas WalletHub is three years old. Redfin is a real estate brokerage and makes its money when users buy or sell using Redfin’s agents.


Top 4 legitimate lists
(or at least semi-legit)

That said, The Ann found some of the lists did well enough using the above criteria to be called legitimate — or at least semi-legitimate.

 1. WalletHub is a personal finance website that ranked Ann Arbor as the most educated city in America. Its main focus is producing surveys and reports on issues relating to personal finance. It provides users with free credit report scores and comparisons of credit cards, checking and saving accounts, CDs, insurance and loans. 

In determining the most educated cities, WalletHub looked at educational attainment (80 percent of the overall score) and quality of education and attainment gap (the remaining 20 percent). For educational attainment, it looked at the percentage of degrees held by adults older than 25. For quality of education and attainment gap, it examined public school rankings, the racial and gender gap in educational attainment, and the number of students enrolled in the top 200 universities.

Data was accumulated from the U.S. Census Bureau, the Bureau of Labor Statistics, GreatSchools.org and U.S. News & World Report. In 2015, WalletHub rated Ann Arbor the most educated city and No. 1 among best college cities. 

2. Money magazine is published by Time Inc. and has been in existence since 1972. It covers personal finance topics such as saving for college and investing. It primarily relies on ad sales and subscriptions. This year, it ranked the University of Michigan No. 2 in its “best colleges” survey.

Money began by narrowing its search to colleges with at least 500 students, a graduation rate at or above the median for its type (public or private), and the financial standing of the school. More than 700 schools were ranked in three categories which were each given equal weight: quality of education, affordability and outcomes. To determine the quality of education, Money looked at the six-year graduation rate, peer quality, student-to-faculty ratio, and difference between the actual graduation rate and the expected graduation rate. In calculating affordability, Money examined the net price of a degree, the debt taken on by the student and parent PLUS loans, student loan repayment and default and affordability for low-income students. To compute outcomes, Money looked at the earnings of the graduates as self-reported to PayScale.com, the 10-year earnings of students, an analysis from the Brookings Institution regarding the market value of the skills most commonly listed by alumni on LinkedIn profiles, comparative value earnings, career services and the answers given by alumni on PayScale.com survey questions. 

The major sources used were the U.S. Department of Education, PayScale.com and Peterson’s, a college information website. 

3. Two rankings came from the same parent company, Niche.com. The website says it helps people find the niche where they want to live or go to school. Nichebeta ranked Ann Arbor the No. 1 place to live in Michigan, while Niche ranked Ann Arbor the No. 4 city to live in the United States. 

The websites looked at cost of living, percentage of residents with a bachelor’s degree or higher, the average Niche grade for public schools, the strength of the local real estate market, diversity, health and fitness, nightlife, shortest commutes, crime and safety statistics, the amenities for families, jobs, outdoor activities and weather.

Sources used by Niche and its affiliates include the Department of Education, the Census, the FBI, the Brookings Institution, Niche’s parent company and student surveys, among others.

4. This year, Forbes, a business magazine, ranked Ann Arbor the fifth best place for millennials and the 20th happiest city to work in. 

For the millennial survey, Forbes used Niche.com’s list of “factors of concern” to millennials, such as the percentage of residents in the 25-34 age group, what percentage of them had moved to the city in the past year, the crime rate, median rent, unemployment rate, diversity and quality of life considerations such as proximity to bars, coffee shops and restaurants. Census data were also used.

In the happiest ranking, Forbes used the “fulfillment-focused career site,” CareerBliss. It looked at eight factors affecting one’s happiness with work, such as work-life balance, the work environment, company culture, access to resources, opportunities for advancement and average salaries. No other metrics were given. 

Justin Fenwick, a millennial and resident of Detroit, disagrees with the rankings. While Ann Arbor can be great if you get a job, he said, it can be tough “if you have to change careers, because then you will likely have to accept a lower-than-market rate of pay. Otherwise, the job market is just so saturated, and I noticed a bias against early career individuals.” 

On the other hand, Fenwick said, “It was the area that kept me in Michigan long enough to discover Detroit.”


Top 3 reasons lists
favor Ann Arbor

These factors favor Ann Arbor and increase our chances of being included on lists: 

1. The unemployment rate. It’s currently 2.6 percent, well below the state average of 5.4 percent, and it’s remained steady during the past year. This factor alone gives us an advantage over similarly situated cities. 

2. The United States Census education statistics. The most recent data indicates that 96.4 percent of our residents have a high school degree and almost 71 percent have a bachelor’s degree or higher. These facts help us jump to the top of the lists that heavily weight education. 

3. Our public schools. According to GreatSchools.org, a popular source which allows anyone to search for schools based on test scores, academic growth, college readiness and other publicly available data, Ann Arbor Public Schools are “above average.” 


Top 2 clickbait articles

1. Real-estate brokerage Redfin recently rated Ann Arbor’s downtown as the most walkable neighborhood in the Midwest (mid-size city edition). 

Redfin used Radical Cartography’s definition of “Midwest,” and only considered cities of fewer than 300,000 people but with neighborhoods of more than 1,000. Otherwise, no methodology or primary data sources are given. The article hypes downtown at least partly because of places like the Michigan Theater, Arbor Brewing Company, Zingerman’s and because of its proximity to Michigan football games. It’s not clear whether Redfin even visited the city, and clicking on the links just brings readers to real estate listings in our area. 

2. Esquire magazine, which caters to the “interests, curiosity and passions of men,” listed Ann Arbor as one of the top towns worthy of a day trip. No methodology or data sources were listed. The short entry just shows a generic picture of the city and touts Zingerman’s, the Ann Arbor Art Center and the annual Ann Arbor Film Festival.

Resident Nancy Shore says, “I feel like (Ann Arbor’s high rankings) are useful in that they affirm what many of us already know to be true. But I am not sure I put much weight on them because many of them seem created to generate publicity for the publication that features them.”


No. 1 List You Don’t
Want to Land Upon

One list a city definitely does not want to land on is Forbes’ “most dangerous cities,” and Detroit has been included from time to time.

However, as stated above, Forbes uses the FBI Uniform Crime Statistics, which the FBI itself cautions should not be used to rank cities. Also, Data Driven Detroit (is.gd/dddetroit) gave a detailed explanation as to how nine other cities in the top 10 “most dangerous cities” were ranked based in part on their metropolitan statistical areas. Detroit’s MSA includes six counties surrounding it; however, Forbes used only metro division data, which is just Wayne County. If the metro statistical area around Detroit — including Macomb, Oakland and Washtenaw counties — was included, as it was for every other city in the top 10, then Detroit drops to No. 17 on the list.

Data Driven Detroit warns against simplifying data. The creator of the Money list says that if you don’t trust a survey’s methodology, then you shouldn’t trust the survey. So what do we do? Whom do we trust? The Ann obtained quotes from actual locals who live or have lived in Ann Arbor. Maybe we should start there. 


Caveat emptor

Ann Arbor resident Jean Henry reminded The Ann that despite A2’s recent accolades, we were ranked eighth in income segregation by The New York Times.

Other residents cited a few different categories the list-makers might want to look at: “most inflated rent,” “most residents ignoring the natural beauty while they look for Pokémon,” “most flooded market for teachers” … and Ypsilanti resident Matt Quirk suggested “best place to come for college and never leave.”

Ann Arbor resident Linh Song says she appreciates A2’s high rankings but she’s concerned that all of the praise might make us a little too comfortable: “We may become convinced that it’s perfect here, so nothing should change. This is where maps showing ‘brain drain’ — from graduates to young families who leave the city — would be more helpful.”


Usually, this just means Jim, one of our partners, uploaded the article. There's an off chance it could be an intern, though, so be nice.