The guiding philosophy we adopt is well encapsulated by a famous quote from Lord Kelvin: “Knowledge that is not quantifiable is of a meager and uninteresting kind.”
But another, slightly less well known quote, should also be kept firmly in mind: “Any figure that looks interesting is probably wrong.” (Sir Claus Moser, Presidential Address to the Royal Statistical Society).
The purpose of this post is not to be exhaustive or even particularly analytical. The main purpose is to help students, and others with an interest, find some useful data sources on the economies and real estate markets for a wide range of countries.
This page is still under construction. I'm posting it in rough form so that students from several courses can use these first links without delay. Check back later in May for a much improved version.
It's hard to say when it's going to be appropriate to remove the "Work in Progress" sign. A complete accounting of all available housing, urban and real estate data is beyond the scope of any blog entry, even one as long as this one. The world of data is fast moving; you'll see I've spent a lot of time describing some older data sources, with less attention to recent data sources and the emerging world of real time "big data." I do have some introductory comments and hope to expand on these in future posts. Please feel free to use the comments section to present your suggestions for additions, as well as other comments and corrections.
In due course, this page will be your point of entry to several related pages on international indicators. And eventually we'll post similar links and information on domestic U.S. indicators.
This "first edition" begins with a few pages on global urban indicators. We begin with an overview of an important international effort. But first..
An Important Caveat
In this post we’ll present a number of two-way plots of data. Remember that car causality in correlation not the same thing. Consider these simple exploratory data analyses and don’t make too much of them.
On the other hand, simple plots and correlations can motivate more careful modeling and econometric efforts, get us thinking:
Housing and Urban Development Indicators
Circa 1990, an international group of researchers focused on housing and urban development undertook to develop a set of comparable indicators on urban development across countries. Early efforts focused on housing and housing finance; these were later expanded to include other aspects of urban development.
Some History
Urban indicators have been around for many years, but there was a real impetus given circa 1990 by a partnership among the World Bank, the United Nations Centre for Human Settlements (Habitat), and numerous local partners. Spearheaded by World Bank economist Steve Mayo, with urban planner Shlomo Angel, the program had many antecedents including empirical analyses of housing markets under the Housing Demand research project.
Circa 1990, the World Bank and UN Habitat (a.k.a. UNCHS not to be confused with Habitat for Humanity International) began a data collection project called Housing and Urban Development Indicators (HUDI). Wave I, carried out under the direction of Mayo and Angel at the World Bank, with UNCHS, examined 52 cities, one city per country (usually the capital city). Mayo and Angel developed a questionnaire and instructions and recruited a team of researchers from over 50 countries to carry out the survey. Most data were dated 1990, but the basic publications began to appear in 1993, so this is sometimes referred to as HUDI Wave I and sometimes HUDI 1993.
HUDI came out of two ideas. The first was a realization of the importance of careful international comparisons, as demonstrated by the success of the World Bank's World Development Indicators (WDI). (NB: "careful" and "success" does not imply "perfect." WDI and other international comparisons are always subject to error.)
The second inspiration for HUDI was an increase in comparative research on housing markets, initially that carried out by a group of World Bank affiliated researchers. The aforementioned Housing Demand project was the most direct progeniture but several others deserve mention. One of the first studies at scale was the "City Study" of Bogota and Cali, directed by Greg Ingram and carried out by a large team of World Bank and Columbian researchers. The best single introduction to the City Study is a summary book by Rakesh Mohan. Another important influence was World Bank research on housing finance spearheaded by Bertrand Renaud and later Robert Buckley. Comparative work on rental markets, focusing on rent controls, was another result.
The HUDI effort in turn fed into a number of housing market and policy products, some of which will be discussed later. But first and perhaps foremost would be the World Bank's 1993 policy paper, Housing: Enabling Markets to Work, coauthored by Mayo and Angel.
Let us now turn to some details of the various "waves" of the Housing and Urban Development Indicators data collection.
Wave I HUDI
Data, again mainly from circa 1990, were collected and first published circa 1993. The first wave collected some 50 variables, comprising data on, among other things, demographic basics, incomes, housing rents and prices, size and quality of housing, financing, and transportation. The variables described in more detail in the documentation below.
A good starting point is to examine this short list of 10 key indicators from 52 countries (mainly the capital city, often also the largest). These ten key indicators are briefly defined in Table 1, below. They are described more fully in Malpezzi, Stephen and Stephen K. Mayo. Housing and Urban Development Indicators: A Good Idea Whose Time Has Returned.
 |
| Exhibit 1: From Malpezzi and Mayo, using data from Angel and Mayo and many country level team members. |
But this short list of ten indicators, and the Malpezzi-Mayo paper, only scratch the surface. A more complete listing of final variables from Wave I, and descriptions, can be found here.
The most complete analysis of the Wave I 1993 version of the indicators is by Shlomo Angel, Housing Policy Matters: A Global Analysis (Oxford University Press, 2000, available from fine bookstores everywhere).
See also the detailed review of Angel's book by Michael Murray, Journal of Housing Economics, June 2001. Murray provides a number of suggestions for applied research that could be undertaken with such data.
There are many analyses that can be made of these indicators, here's a taste. How does floorspace consumed vary with income, as proxied by GDP per capita?
 |
| Exhibit 2, from Wave 1 Urban Indicators |
Exhibit 2 presents the Wave 1 city-specific estimates of average floor space per person, in square meters, for a large city (usually the capital city) in each of 52 countries. The horizontal axis is country level GDP per capita in $1990; HUDI Wave 1 did not succeed in collecting reliable city-specific GDP or income measures in many cities, so think of these as a rough proxy for income.
The vertical axis is of course space per person in m². (For readers use to the English system, i.e. you are from the U.S., Liberia or Myanmar, 1 m² is 10.7 ft.²) GDP per capita is presented using a log scale; moving along that horizontal axis represents percentage changes in that variable. Space per capita is linear; if I used logs for the vertical axis also, you’d see a stronger linear correlation between the two logarithms; but I left the vertical axis linear to highlight how much variance there is in space per capita, especially among richer countries.
The U.S. (represented with data from Washington DC) is the most spacious at 70 m² per person followed by Melbourne, Australia at 50 m² and Oslo, Toronto and Stockholm at around 40 m². But Tokyo, about as rich as Washington DC by this measure, has space per capita of 15 m², less than poorer cities like Santiago (Chile), Caracas (Venezuela), Bangkok (Thailand) and Istanbul(Turkey).
Hong Kong is another country that “punches below its weight” in housing space; with 1990 GDP per capita of about $20,000 of GDP per capita in 1990, Hong Kong’s 7 m² per capita puts it below much poorer cities like Dakar (Senegal), New Delhi (India) and Bogota (Colombia).
Thus the countries with the most space per person are all rich countries; but the relationship is not linear; and there some rich countries with remarkably low space per capita. Economists call this pattern of data “heteroskedastic,” that is, the variance of the dependent variable is changing with the level of the independent variable.
There’s much more we can do to understand International differences in space per capita. We might examine how this varies with the overall population density of the country, the urbanization level, and various policy measures among many other possible determinants.
And of course it goes without saying that 1990 data, and rankings, can give us some interesting information about structural relationships, but are not reliable guides to current conditions. To give one outstanding example, space per capita in Beijing has more than doubled since 1990.
Each one of the indicators included in the data set can be similarly plotted and analyzed. Shlomo Angel’s book provides a good starting point on these analyses. Murray provides a guide to extending Angel’s work.
Many other interesting variables include rents asset prices and all of our top 10 at least a dozen more from the date full data set which you can download conveniently here. The data set is also replicated in Angel’s book, as an appendix.
Here are a few of the original data collection protocols developed by Steve Mayo, Solly Angel and others for the Wave I effort.
Housing Indicators Program Extensive Survey: Part I Introduction
Housing Indicators Program: Volume II: Indicator Tables
Wave II and Wave III HUDI
After this first effort HUDI were taken over by UN Habitat, as the World Bank reduced its support for urban research. Two additional waves of data collection were carried out and collated under Habitat's umbrella, in 1996 and 1998. Advantages of these waves include a larger sample size (more countries, and especially multiple cities from many of the countries). A number of variables were also added. But the second and third waves did not present useful data on household income, which is important for many analyses.
Also, the second and third waves concentrated on housing characteristics and outcomes, largely to the exclusion of housing policies. Research from the first wave, and much other research including the precursor housing market research at the World Bank, demonstrated the importance of understanding the policy environment.
As of this writing, I have not been able to successfully link to the correct pages to download at the UN site. But I have access to the data from past research.
Here is my modified version of the Wave II (Habitat Version 1) data
Here is my modified version of the Wave III (Habitat Version 2) data
Here are some related pdf documents.
Global Urban Indicators Selected Statistics includes a number of indicators, mainly demographic and public health, comparing Wave II (circa 1993 data) and Wave III (circa 1998 data).
Global Urban Indicators Database Version 2 is based on the same data as the preceding publication, but with a bit more analysis and regional and city comparisons.
The Original Indicators Effort Seems to Have Run Out of Gas
So, in brief -- the original Housing and Urban Development Indicators project lost World Bank institutional support when Steve Mayo and Solly Angel left the Bank. For some time the project continued under the aegis of UN Habitat, but eventually that support also waned.
The original Waves I, II and III were themselves preliminary in many respects, with numerous issues raised and some lessons learned about how best to collect and organize such data. But the vision of an eventual analytically sound, repeated data collection effort that would provide researchers, policymakers and others with a basis for cross-city and cross-country comparisons of economic and social outcomes related to cities has, sadly, not been realized.
From time to time there is discussion of reviving the effort, at one institution or another. Perhaps I'll be able to revise this entry at some point with better news. In the meantime, there are some sources of urban data that can be useful. Let's examine some of those below.
In 2000, 189 countries agreed to support a set of objectives called Millennium Development Goals. The aim was to develop a set of goals with corresponding indicators. A year later, the UN Secretary General unveiled a set of specific goals, The Millennium Development Goals (MDGs)
If 8 are good, 17 are better? UN Sustainable Development Goals (2015 to 2030). For the official line, click here.
During the transition from the MDGs to the longer SDGs, I wrote a short critique of the approach, available here.
Maybe I shouldn't throw too many stones. I too have written some long lists of things I thought needed to be done to improve housing markets, the growth and fairness of cities. Mea culpa, I admit I've gotten carried away. Recently, partly inspired by an interesting book by Dani Rodrik, I've tried to switch metaphors, from "checklists" to "recipes." If you want to see an early effort, focusing on recipes for China housing policy, click here.
Of the 17 goals, one is focused on cities and housing:
Goal 11: Make cities and human settlements inclusive, safe, resilient and sustainable.
Each goal comes with targets. Specifically, the targets for UN SDG 11: Sustainable Cities and Communities Targets are (in brief):
By the way, the term "slums," like "sustainable," "sprawl," "resilient" and so on is one that is vague and ambigous on a good day, and poorly and inconsistently measured as well. Full disclosure: I've used the word "slum" myself more than I should in the past, and probably will in the future. But see this trenchant comment by Alan Gilbert for one of many critiques.
Here are a few other UN publications regarding the collection and analysis of urban indicators.
Urban Indicators Guidelines: Monitoring the Habitat Agenda and the Millennium Development Goals
Solomon Greene and Brady Meixell, "Hacking the Sustainable Development Goals: Can the U.S. Measure Up? (Urban Institute)
Future posts will discuss other sources of indicators in more detail. But we can't wait -- let's link to a few key sources here.
First, some general sources, that include some urban data and a lot of important contextual data (demographics, GDP, and so on). Then we'll look at some of the available sources for housing and related data.
This is a "first edition;" future versions will include some additional links and sources.
The United Nations has a handy webpage that provides links to a few dozen key international databases from a number of UN agencies (including World Bank and IMF, which are specialized agencies of the UN).
We have already mentioned the World Bank's World Development Indicators as a source of basic country-level data.
Angus Maddison, (1926-2010) University of Groningen has compiled the most widely used comparative data on historical GDP (purchasing power parity) as well as ancillary data like population. His website, including links to his publications and data, can be found here.
The Groningen Growth and Development Centre carries on Maddison’s work, updating his historical data etc. Click here to go to that source.
Hans Rosling was a physician and professor of public health, from Uppsala, Sweden, who fronted the Gapminder and Dollar Street websites. Collaboratorson the posthumously published Factfulness, his son Ola Rosling and daughter-in-law Anna Rosling Ronnlund, are very good at programming innovative graphics. Go to YouTube and you can easily find some of his talks.
World City Pop, a database of city populations around the world over time, including a lot of historical data, is maintained by Dr Andrea Biguzzi.
But the mother of all urban population data is the United Nation's regularly updated World Urbanization Prospects. Vern Henderson's World Cities Database is another source.
Exhibit 3 presents the well-known correlation betwen urbanization and economic output, discussed in many studies such as Malpezzi (2006), World Bank (1993) and Henderson (2003), to give but three of hundreds of possible references. This literature provides the underlying theory of urban development, caveats about data, and more than a few puzzles.
Land use is another key urban indicator. Shlomo (Solly) Angel and colleagues have developed the Atlas of Urban Expansion. Solly and still more colleagues extended this work and are carrying it out under the rubric of the Marron Institute Land and Housing Survey in a Global Sample of Cities.
This effort focuses on 200 cities rigorously selected from a universe of 4231 cities (metro areas) with populations over 100,000. The sample is stratified by region, city population size, and number of cities in each country. There are actually two related surveys. One focuses on the regulatory environment, and the other focuses on housing market conditions (rents, prices, tenure type, household mobility…). Angel and colleagues have also calculated the percent of each country's total land area apparently covered by urban development, according to their satellite data. This figure shows their country-level estimates of this percentage.
Angel et al.'s data focuses on average densities for metro areas. For density patterns within cities, see Alain Bertaud's data on density. I've been honored to work with Alain on some of this project; a presentation that reviews our joint work to date, and presents some of the data, can be found here.
Primary density data have been collected by Bertaud, over 2+ decades. The primary unit of observation is the city's census tracts, or closest equivalent. The data are processed in GIS by Alain along with his collaborator Marie-Agnes Bertaud. Bertaud's method measures density in built-up areas only.
Exhibit 5 presents three-dimensional representations of Bertaud's analysis for two cities of then-similar size (10 million), Paris and Moscow. Each small polygon within a city represents the average density of a census-tract-like unit. The data are to the same scale horizontally, and the same scale vertically. With some exceptions (near La Defense), the densest locations in Paris are near the center of the city; in Moscow, the densest locations are near the periphery.
Paris is not exactly the pure laissez-faire city! There’s a lot of regulation (including height restrictions) that could reduce central density; and public development in the suburbs that could raise it farther out. Nevertheless, Paris looks reasonably close to the “standard urban model” of Alonso, Muth and Mills.
But by American standards, Paris is very dense. The peak density is half again that of New York, over four times that of Chicago, six times L.A.
Moscow, analyzed more deeply by Bertaud and Renaud (1997), shows the outcome when there is no real land market, but rather housing development by Soviet kombinats operating with no market information to guide them. Each vintage of Soviet housing in Moscow was simply put up at the then-edge of the city. On top of that, old housing, obsolete factories from the twenties, and so on, were never taken down, so land was never recycled as the city grew.
Several papers by Alain Bertaud, and Bertaud and Malpezzi, discuss how these population patterns aren't simply academic curiosities, but place large welfare costs on individuals, especially transport costs, as well as costs born by city governments.
Penn's Marja Hoek-Smit has spearheaded a fantastic resource, Housing Finance Network, or HOFINET. HOFINET is a web portal with access to a wide range of research on data on international housing finance, and related topics. Data are organized by both topic and country. This is my go-to site when I want to learn about housing finance around the world, as well as in-depth discussion of many social and economic aspects of housing.
Specifically, HOFINET's objective is to provide a wide variety of global housing and financial information, focusing on:
Exhibit 7 presents some basic data gleaned from HOFINET. The vertical axis is the ratio between mortgage debt and GDP, the standard way of measuring the overall size of financial markets across countries. It’s unsurprising that this measure of mortgage market depth increases with per capita output. Richer countries tend to have more developed financial markets in general, and mortgage markets are of course an important segment of those financial markets. Further discussion, and references, can be found at a three-part set of blog posts I've published at Rutgers' Center for Real Estate, headed by Morris Davis: Part 1, Part 2, Part 3. I've also published a companion piece on the U.S. mortgage market here.
Before we examine data sources, let us discuss all too briefly and important and somewhat complex question: What do we mean by the “price” of housing, anyway? Every house is more or less unique, in terms of its size, quality, associated services, and of course its location.
Consider a simple thought experiment. If I pay $1,000/ month rent and you pay $1,200, who’s paying a higher price? If our units are nearly identical, you do. If your apartment is 1500 square feet and mine is 500, maybe I do. If your apartment is in midtown Manhattan and mine is in Madison, maybe I do.
Other questions arise, e.g. how do we handle many characteristics at once? Are we looking at “flow” prices (per unit of time) or “stock” (“asset”) prices? And just to keep us on our toes, the way economists usually define "price" is different from common usage, especially in the housing market. Exhibit 8 presents a brief summary.
Even economists misuse these terms on occasion. I try to remember to say “value” or “asset price” when I mean the value of a unit. I try to say “price” when I’ve somehow controlled for the quantity of housing services produced by a unit. But I often fail, especially in casual conversation, or when speaking to a real estate audience, where one often needs to adapt to common usage.
There is a large literature on how housing economists grapple with these measurement issues. Green and Malpezzi (2003) present a review of the basic methods, including:
Further discussion is beyond the scope of this already long post, here we simply want to note the issue. See Green and Malpezzi or other sources that discuss these issues in greater detail.
A good starting point for studying housing prices across countries is the IMF's Global Housing Watch. Spearheaded by Prakash Lougani and now run by Hites Ahir, Global Housing Watch provides quarterly data on house prices as well as metrics used to assess valuation in housing markets, such as house price‑to‑rent and house-price‑to‑income ratios. Global Housing Watch also contains many informative analyses of recent price trends, and links to relevant research.
See also the excellent blog where Hites and Prakash provide commentary on housing as well as a wider range of issues, The Unassuming Economist.
The Federal Reserve Bank of Dallas International House Price Database provides quarterly house price and personal disposable income (PDI) series by country, beginning Q1 1975. They are provided as index numbers (2005=100), as well as income per working age population. The Dallas Fed also produces an aggregate global index weighted by 2005 purchasing power parity GDP. Detailed description of the sources and methodology can be found in Mack and MartÃnez-GarcÃa (2011).
Exhibit 8 presents a chart I created using the Dallas Fed database, and discussed briefly here. National real house price indexes are rebased to 1980=100. Notice the U.S. "average" boom and bust is evident in this index, but France, Ireland and the UK are among countries that are more volatile still. In contrast, notice how flat Japan and Germany are over this period. There's much to discuss about sources and technical details of measuring prices, but first and foremost we note that national indexes, however constructed, typically conceal large within-country variances in prices. Consider the last few decades of house price behavior in, say, Newcastle compared to London; or Detroit compared to San Francisco.
OECD has a database of housing price data, city populations and other related data. Limited to OECD countries, but a useful source nevertheless. Data are annual, for 45 countries. Data for the most complete countries can go back as far as 1970. Documentation can be found at the OECD Handbook on Residential Property Prices .
The Economist Global House Price Database presents housing price indexes, normalize by rent (price-to-rent) and by income (price-to-income) ratios for 28 countries and 40 large cities. Exhibit 10 highlights London and New York against the background of other city trends. Data duration varies but goes back 30 years for a number of locations.
A more recently available but longer run data source is due to several economists from Bonn. Katharina Knoll, Moritz Schularick and Thomas Steger present very long run global house price indexes from 14 countries. The data are described in their paper appearing in the American Economic Review (107(2), 2017, pp. 331-53); the data are also downloadable from that site. Exhibit 11 presents their basic country data. The shaded bars represent the two world war periods.
Exhibit 12 presents some of the house price and rent data from the Deloitte Property Index.
Real Capital Analytics is a leading data source for commercial real estate prices, capital flows and other basic data. Since 2000, RCA records individual transaction-level data on most property types, in 172 countries:
Exhibit 13 presents RCA's map of late 2019 commercial real estate transactions around the world. Exhibit 14 presents some representative data on commercial real estate price changes in 20 major metro areas. RCA aims to cover all transactions involving properties valued north of $2.5 million, in real time, both and pending deals of all types: sales, recapitalizations, partial interests; financing, defaults, and foreclosures.
Jones Lang Lasalle (JLL) is another leading source of global real estate data and analysis. Exhibit 15 presents some representative data from their end-2019 report on global real estate markets on 16 major countries.
Global REIT associations provide information on listed public real estate markets. These are more or less similar to U.S. Real Estate Investment Trusts. Currently about 38 countries have adopted such vehicles, with perhaps 10 considering future adoption. Exhibit 16 presents some summary data on the global scope of these vehicles. Real Estate Equity Securitization Alliance (REESA) members include:
The Commercial Real Estate Data Alliance (CREDA) is a consortium of leading real estate academics and professionals; the name is somewhat self-explanatory.
One lens through which to see the consortium’s motivation is through their repeated reference to the desire to put real estate on an even footing with traditional financial markets. Analysis of equity markets in particular was greatly enhanced when the University of Chicago’s CRSP (Center for Research in Security Prices) data became available. CRSP made a large variety of equity pricing data available on a timely, consistent basis, pre-cleaned and ready for analysis.
So one way of thinking of CREDA is CRSP for real estate. Headquartered at the University of North Carolina’s Wood Center for Real Estate Studies and UNC’s Institute for Private Capital, CREDA involves leading academics as well as private firms and professional organizations (Exhibit 17).
To date, the focus of CREDA has been primarily on US commercial real estate data. But we include it here because many of the participating academics in organizations are also internationally focused, and none of the data issues CREDA attacks are unique to the United States. I would speculate that CREDA’s efforts will eventually “go global,” whether formally or informally, to help improve the standardization and availability of data across countries.
CREDA is sponsoring several pilot studies that focus on the U.S. at the moment but that will have utility to global providers. One project is developing a platform to merge and scrub multiple property level data sources including providing an open source property address matching methodology as well as a one-stop shop for documenting commercial real estate data sources. Another CREDA sponsored study involves mapping permit data to capital expenditure data. CapEx data is not currently widely available; but most capital improvements require a building permit, which are available. To date this data is not well matched and of inconsistent quality so the research project is analyzing statistical relationships between permit data and actual CapExs which will allow extrapolation to any property or portfolio.
CREDA also sponsors an annual research symposium at UNC bringing together academic industry and government researchers, investors, and data providers. In addition to data issues the forums provide panels that discuss current policy issues such as housing affordability in light of new data.
Another strand of international urban indicators revolves around measuring the urban policy environment. We begin with an early effort at measuring the general economic policy environment. Ramgopal Agarwala created a classic index of economic policies for 32 "developing countries" for the 1983 World Development Report. The index is based on policies re:
They then constructed a simple additive index. Minimum (best) score is 7, maximum (worst) score is 21. In practice: Malawi scored 8, Thailand 10, Cameroon, Korea, Malaysia, Philippines, Tunisia scored 11. At the other extreme, Argentina and Chile scored 17, Tanzania and Bangladesh scored 18, Nigeria scored 19, Ghana scored 20. (Of course, these policy indexes are 40 years old. There is substantial serial correlation in policies. But policies do and have changed substantially over time.) Exhibit 18 presents the basic Agarwala data.
Research by Angel and Mayo (described below) finds that house prices-to-income don’t vary terribly systematically with income. But that does not mean there are no systematic patterns. Exhibit 19, from Malpezzi (1990) compared housing prices, normalized by income, to the Agarwala index. At the time we had no purpose-build index of urban policies, but Malpezzi (1990) argued that countries that distorted their labor and financial markets in general terms would likely be following policies that would also distort the housing market. The data are consistent with such an interpretation of the Agarwala index as a proxy for a general policy environment. Circa 1990, countries that "did the right thing" in economic policies experienced house price-to-income ratios averaging 4 or 5, while those with distorted policies averaged 7 or 8.
More data became available in the early 90s from reforming socialist economies after the breakup of the Soviet Union, and the imposition of post-Mao reforms in China. Other analyses (not shown here, see Buckley and Malpezzi, and Buckley and Tsenkova) found that in the early days of reform, back-of-the-envelope applications of the Agarwala framework and data on house prices found that many of these countries, placed closer to Ghana on the policy framework, with prices even more distorted that we observe in Exhibit 19.
The correlation of the Agarwala Index with various housing market outcomes motivated several attempts to construct measures that were more directly based on urban and housing policies. One of the first efforts focused on rent controls, motivated in part by research by Richard Arnott among others on the importance of analyzing specific features of rent regulation regimes. (Arnott's original paper with Nigel Johnston dates from 1981; a later, more readily accessible version is available here). A survey of some 50 countries was undertaken as part of a World Bank research project on rent control described in Malpezzi and Ball (1991, 1993).
Countries which have no controls receive a zero rating. Other countries are rated as follows. There are nine elements; for all but one, each element receives a rating of 0 (permissive), 1 (medium), 2 (restrictive). The first two elements are:
Some countries, such as the United States, and Canada, have many rent control regimes which vary greatly from place to place. In such cases, when there was substantial divergence from place to place in an element, we graded the element 1.
Key findings, which can be found in Malpezzi and Ball, include the following.
First, consistent with priors, the rent control index was correlated with the Agarwala index and other measures of general economic policies, such as contemporaneous measures of the overvaluation of a nation's currency.
Second, while many countries had some form of rent regulation in the late 1980s, the most stringent regulations were usually found in poor countries. Other research, described in the research project's final report, found that there were often ways both households and landlords evaded the rules, including illegal side payments (key money) and accelerated depreciation of units. Nevertheless, there was evidence, unsurprisingly, that rent burdens were lower, on average, in countries with the most stringent controls. On the other hand, asset prices were higher, relative to incomes, in those distorted markets, plausibly because stringent controls were present in markets that had other distortions on the supply side of the market. And markets with the most stringent controls had lower rates of housing investment, ceteris paribus.
So circa 1990, findings using the Agarwala index, on the one hand, and the rent control index, on the other, gave us confidence that such "policy indexes" could be informative. It was, and is, however, important to remember the limitations of any such index. In brief.... Any policy index perforce contains information on only a small fraction of interventions that affect housing, or any other urban market under consideration. Taken with the observed correlation among a number of diverse such indexes, both across countries and across cities within a country, we should never interpret any coefficient literally. For example, "the coefficient of my regulatory index is 0.1 in a logarithmic housing price model -- so if we change a regulation that theoretically reduces our index by a value of 1, we expect a 10 percent reduction in housing prices...." Well, maybe not, because that regulation, zoning or FAR or whatever, may well represent itself but also represent some other regulations that are not included in the index but which are correlated with the included measures. The problem is analogous to the interpretation of individual coefficient estimates in a hedonic regression model, as explained in papers by Butler and by Ozanne and Malpezzi.
In the event, some of us think of such regulatory indexes as akin to the "stupid pet tricks" readers of a certain age will remember from a popular late night comedy show. It's surprising, really, that they work at all.
Attempts to improve on these early efforts led to their inclusion in the Angel-Mayo World Bank-Habitat Urban Indicators Project, described generally near the beginning of this post. Wave I did the most thorough job of attempting to measure the policy environment. Angel and Mayo constructed policy indexes in six areas: property rights, housing finance, housing subsidies, residential infrastructure, the regulatory regime, and the industrial organization of the housing market. Subindexes were constructed for each area, and they were also combined in an overall "enabling index." Exhibit 20 presents a schematic summary of the effort.
Details of the indexes and their construction can be found in Angel and Mayo (1996) and especially in Angel (2000). Exhibit 21 presents the six subindexes, and the aggregated enabling index. Higher values of the indexes represent better policy environments.
A number of analyses were carried out by Angel and Mayo in the aforementioned publications, examining a range of incomes including rents, asset prices, investment levels, and housing tenure. Here we will present two simple graphics that can motivate a more careful reading of their original research.
Angel and Mayo (1996) group countries into three categories: low enablers, medium enablers, and high enablers, depending on the level of the overall enabling index. Exhibit 22 presents the supply elasticities they estimated by group. As expected, estimated supply elasticities were neatly ordered by the level of the enabling index.
Exhibit 23 presents to more data elements from Angel and Mayo: each capital citiy’s house price to income ratio, and the corresponding rent to income ratio. Exhibit 24 plots the house price to income ratio by the enabling index. Taking the plot at face value, better enablers (lower values)) perform better, though results are hardly tight. The regression line through the city specific points shows the expected positive relationship but is heavily influenced by China and the fit is not particularly impressive.
Let’s wrap up our discussion of the classic urban indicators of Angel and Mayo and their associates by noting that in addition to the World Bank’s 1993 housing policy paper Enabling Housing Markets to Work, they produced several examples separately and together of country and regional discussions of housing policy reforms that brought indicators data in as diagnostic tools. See for example Angel’s study of Panama, and Mayo’s study of South Africa.
Another effort at measuring the urban policy environment was carried out for Korea’s Center for Private Enterprise by Kim Malpezzi and Kim. Exhibit 25 presents a summary of their data collection for regulatory elements; with help from local experts they collected these for some 20 cities. In the event, Kim Malpezzi Kim found that housing rents and prices were correlated in the expected manner with the regulatory index and Kim found that in Asia supply elasticities were lower in markets with more restrictive environments. But in both cases the results were far from robust. A recurring bottom line of this and other studies is that it’s often difficult to get consistent information even with carefully designed questionnaires using a range of partners in countries with very different institutional environments and development traditions. Such indexes seem to work better (provide more reliable information) within countries than across them.
We’ve discussed Hernando’s De Soto’s pathbreaking (no pun intended) analysis of Peruvian development timelines detailed in his book El Otro Sendero (The Other Path) in our previous post on the so-called Malaysia model. Exhibit 27 presents De Soto’s original schematic for the procedures required to formalize a legally obtained home in Peru. As reviewed in our previous post on the Malaysia model e Malaysian developer MK Sen independently provided us with a very similar analysis for the Malaysian market, as detailed in Hannah Bertaud Malpezzi and Mayo (1989).
Lex Mundi project. This network has been used to collect indicators of the legal framework for real estate and other economic activities for range of countries. A series of papers by Simeon Djankov and several associates details these efforts.
Collects data from a network of law firms on the nature of property rights and contract enforcement procedures across countries. See Djankov, La Porta, Lopez-de-Silanes and Shleifer (2004)
Exhibit 28 provides a schematic of the start-up procedure for a small firm in France. Procedures are lined up sequentially on the horizontal axis. The time required to complete each procedure is described by the height of the bar and measured against the left scale. Cumulative costs are plotted using a line and measured against the right scale.
Labels on the horizontal axis are hardly legible so we repeat them here:
1. Check name uniqueness.
2. Notarize lease and obtain home office approval.
3. Certificate of clean criminal record.
4. Certificate of marital status.
5. Power of attorney.
6. Open bank account and deposit capital.
7. Notarize company deeds.
8. Publish headquarters address in Journal.
9. Register articles of association at tax office.
10. Request registration.
11. Form K for business.
12. Unblock capital.
13. Notify post office.
14. Designate bondsmen and guarantee taxes.
15. Initial firm ledgers and registers.
16. File for Social Security.
See also https://www.lawyersfrance.eu/starting-a-business-in-france
In France, according to the Lex Mundi data, it takes 66 days and money equivalent to 24% of per capita French GDP to navigate the regulatory regime for starting a firm in France. In contrast, in New Zealand (not represented graphically here, see the paper) requires only 10 days and a monetary cost equivalent to only 2% of GDP per capita.
Djankov et al. went beyond New Zealand and France to collect these and other indicators of the regulatory process for some 40 countries. Exhibit 29 presents the loglinear relationship between the number of procedures to register a firm, and the percentage of the labor force employed in the unofficial or informal economy. Unsurprisingly, the denser the regulatory thicket, the larger the fraction of the labor force employed outside the strictly legal framework. This is no small matter since the costs imposed on citizens by informality has been detailed many times before including in De Soto’s book.
The next step in the evolution of these indexes was to institutionalize this kind of cross country data collection in the World Bank’s Doing Business surveys. Exhibit 30 presents an overview of the kinds of data collected.
The Doing Business survey has been subject to several criticisms. One is that the data collected are about the procedures as laid out within the legal and regulatory framework on paper, not necessarily as actually implemented. In a number of countries, the actual regime hardly follows the official legal framework and may include additional payments; these may be legal (such as developer provision of public goods such as parks or traffic signals, and/or campaign contributions common in the United States) as well as illegal and/or corrupt payments.
Another problem related to the caveat we made above regarding care not to over interpret a single element or coefficient in such index elements is the charge that as Doing Business became more prominent, countries might “game the index” by focusing attention on one or a few elements that give them a chance to move up in the rankings published annually by the World Bank, but make a limited contribution to easing total regulatory burdens.
Yet another set of criticisms focuses on potential downsides of labor market flexibility. One man’s or woman’s labor inflexibility is another’s protection from arbitrary dismissal or unsafe working conditions. More on this in the next version of this blog post.
These elements are reported as national data, but are actually collected for the largest city in each country. Data are obtained from local experts, and refer to the rules de jure. Actual experience/enforcement may vary de facto.
India’s jump on WB’s Doing Business 2018 Index: A real estate analysis
JLL’s India Managing Director Shubhranshu Pani
Housing.com, November 7, 2017
Real estate-specific improvements cited by Pani include:
The Doing Business survey itself has four data elements related to real estate development:
As part of ongoing research with Alain Bertaud, I constructed a simple index from Doing Business real estate elements. First I undertook simple standardization of each of four components; then summed the four components to construct a simple index. For housing price data I used Global Property Guide house price per square meter, mainly because it was conveniently available. That data, as we noted above, is focused on the high-end. A future version of this blog may present more representative prices. In the event I normalize these prices by dividing them by each country’s World Bank Atlas method per capita GDP.
As the data are plotted we find a few surprises among the indexes. Taking the data at face value, according to the Doing Business data, Russia appears to be one of the easiest places to do real estate business. That would be a surprise to President Trump who has not yet managed to build his longed-for Moscow hotel. At the other extreme, Portugal is no easy place to do business – in fact it was a World Bank debate on over Portugal’s housing policies that help motivate the World Bank research program on rent control discussed above. Nevertheless, we might be surprised to see Portugal as such an outlier on the regulatory index.
Taking all this data at face value there is not much correlation between these crudely normalized housing prices and the simple regulatory index. The regression line has the expected sign but only 3% of the variance in the house price to per capita GDP ratio is explained.
Jones Lang LaSalle Real Estate Transparency Index JLL's Index aims to measure the quality and quantity of each country's real estate data. Historically real estate markets have been largely private and opaque. But a tradeoff exists. Increasing transparency may reduce excess returns to selected agents, eroding the value of private information; but in general this will be more than offset by the gains associated with attracting more capital, a wider range of investors, and overall increased efficiency and fairness of a market. See Gordon (2000) for elaboration of these arguments.
As Exhibit 34 illustrates, more transparent markets according to the JLL index (lower values of the index) are associated with greater transaction volumes. Again, correlation or causality? More work to do here.
Fisher (2015) presents a cogent introduction to big data and related topics. . Fisher quotes Gartner’s definition “big data is high-volume, high velocity, high variety information assets that require new forms of processing to enable enhanced decision-making, insight discovery and process optimization.”
Fisher begins by setting up a brief discussion of what he calls traditional data such as prices, rents, rates of return, transaction volume and so on; from traditional sources like RCA, NCREIF, and other sources discussed in this blog post as well as many others. He points out that traditional data are generally well structured, set up in time series and cross-sectional; and can be analyzed with traditional statistical and other analytical tools. Big data is, well, “big,” often nearly contemporaneously collected, and often analyzed using primarily descriptive black box modeling.
Big data are often noisy and unstructured from a wide range of sources. Data are often scraped from the web, from both traditional (e.g. Zillow’s scraping of property tax records) but also from nontraditional sources such as social media (Twitter, Facebook, LinkedIn) as well as informal documents published on the web in a variety of physical formats as well as blogs, discussion forums etc. Data can include video, audio and images as well as traditional numbers and words. The proliferation of the Internet of Things (IoT) presages another threshold leap in the availability of data, in subject matter, quantity, and form.
As Fisher points out these data require new ways of storing and analyzing them as well as collection protocols. Signaling his concern with data quality, Fisher points out that many observers are adding “veracity” to the measured characteristics of big data. Big data analysts often eschew traditional statistical techniques like regression analysis in favor of machine learning and various systems of classification including heavy reliance on visual methods.
As a cautionary tale Fisher (2017) does a quick analysis of correlates of housing starts with Google searches. He finds strong correlations with a number of searches that are hard to sensibly link to housing starts including “Copacabana New York,” “Ramada Plaza downtown,” and “Tower Records.com.” Fisher is making a point with a quick analysis of trends, and doesn’t provide much detail about the exact timing or strategy for his analysis. But the example is sufficient as a cautionary tale, pointing out among other things the ephemeral nature of these correlations. Tower Records, years ago one of my favorite stops for music, went out of business in 2006 after earlier bankruptcies. Perhaps the trending phrase was connected to the redevelopment of some of Tower’s better-known locations in cities like Boston and San Francisco, but it is doubtful Tower has much to tell us about housing starts today, if it ever truly did.
Next steps might be to impose more structure on big data analyses, that is to get away from the black box syndrome. Correlation is still not causality. The future may well see current data mining techniques more effectively combined with old-fashioned techniques attacking old-fashioned issues like endogeneity, robustness, and more or less reliable prediction out of sample.
Lots do to here, including discussion of many housing and urban indicators.
The blog directly preceding this one contains my arguably obsessive notes and teaching materials on the coronavirus and its effects on economies, housing and real estate markets, and people’s lives. Yet we’ve been through some 40 exhibits and even more links to economic and real estate data without a word on the coronavirus.
Unsurprisingly, in due course these thoughts on indicators will have to include some discussion of this particular elephant. That will come in time. Meanwhile I hope this version with its introduction and links helps students and researchers and other readers.
Digression: one of the world's recent successes is an increase in access to electricity. But an estimated 13 percent of global population is still without access, and others have intermittent service.
Be glad you probably have a much better reading environment. Click here to download a list of references.
Future Prospects for UN Urban Indicators May Be Linked to "Sustainable Development Goals"
In 2000, 189 countries agreed to support a set of objectives called Millennium Development Goals. The aim was to develop a set of goals with corresponding indicators. A year later, the UN Secretary General unveiled a set of specific goals, The Millennium Development Goals (MDGs)
For further details of this first Development Goals effort, including links to the indicators, click here.
If 8 are good, 17 are better? UN Sustainable Development Goals (2015 to 2030). For the official line, click here.
During the transition from the MDGs to the longer SDGs, I wrote a short critique of the approach, available here.
Maybe I shouldn't throw too many stones. I too have written some long lists of things I thought needed to be done to improve housing markets, the growth and fairness of cities. Mea culpa, I admit I've gotten carried away. Recently, partly inspired by an interesting book by Dani Rodrik, I've tried to switch metaphors, from "checklists" to "recipes." If you want to see an early effort, focusing on recipes for China housing policy, click here.
Of the 17 goals, one is focused on cities and housing:
Goal 11: Make cities and human settlements inclusive, safe, resilient and sustainable.
Each goal comes with targets. Specifically, the targets for UN SDG 11: Sustainable Cities and Communities Targets are (in brief):
- Adequate, safe and affordable housing, upgrade slums
- Safe, affordable, accessible and sustainable transport
- Inclusive and sustainable urbanization, sustainable human settlement planning and management
- Safeguard the world’s cultural and natural heritage
- Reduce deaths and other losses from disasters
- Reduce the adverse environmental impact of cities, paying special attention to air quality and waste management
- Provide universal access to green and public spaces
- Strengthen national and regional development planning
- Tackle climate change and resilience to disasters, improve risk managements
- Provide financial and technical assistance for building sustainable and resilient buildings utilizing local materials
So far, the UN's data collection on this is rudimentary at best. On housing, there's nothing approaching the HUDI data on rents, asset prices, and other details, merely a map from "Our World in Data" of country estimates of slum populations. You can see that and other first cut tracking data here.
By the way, the term "slums," like "sustainable," "sprawl," "resilient" and so on is one that is vague and ambigous on a good day, and poorly and inconsistently measured as well. Full disclosure: I've used the word "slum" myself more than I should in the past, and probably will in the future. But see this trenchant comment by Alan Gilbert for one of many critiques.
Here are a few other UN publications regarding the collection and analysis of urban indicators.
Urban Indicators Guidelines: Monitoring the Habitat Agenda and the Millennium Development Goals
Accelerating SDG11 Achievement
Solomon Greene and Brady Meixell, "Hacking the Sustainable Development Goals: Can the U.S. Measure Up? (Urban Institute)
Other Notable Indicator Efforts
Future posts will discuss other sources of indicators in more detail. But we can't wait -- let's link to a few key sources here.
First, some general sources, that include some urban data and a lot of important contextual data (demographics, GDP, and so on). Then we'll look at some of the available sources for housing and related data.
This is a "first edition;" future versions will include some additional links and sources.
Basic Data on Demographics, Economies, and Urbanization
The United Nations has a handy webpage that provides links to a few dozen key international databases from a number of UN agencies (including World Bank and IMF, which are specialized agencies of the UN).
We have already mentioned the World Bank's World Development Indicators as a source of basic country-level data.
Angus Maddison, (1926-2010) University of Groningen has compiled the most widely used comparative data on historical GDP (purchasing power parity) as well as ancillary data like population. His website, including links to his publications and data, can be found here.
The Groningen Growth and Development Centre carries on Maddison’s work, updating his historical data etc. Click here to go to that source.
Our World in Data is a data aggregator, based at Oxford University, that pulls together data from a large number of national and international sources; creates terrific and informative charts, along with documentation and explanation. Highly recommended! BTW, Our World in Data is carried out by a small team headed by German economist Max Roser.
World City Pop, a database of city populations around the world over time, including a lot of historical data, is maintained by Dr Andrea Biguzzi.
But the mother of all urban population data is the United Nation's regularly updated World Urbanization Prospects. Vern Henderson's World Cities Database is another source.
 |
| Exhibit 3: From World Bank World Development Indicators and UN World Urbanization Prospects |
Exhibit 3 presents the well-known correlation betwen urbanization and economic output, discussed in many studies such as Malpezzi (2006), World Bank (1993) and Henderson (2003), to give but three of hundreds of possible references. This literature provides the underlying theory of urban development, caveats about data, and more than a few puzzles.
Land Use and Density
Land use is another key urban indicator. Shlomo (Solly) Angel and colleagues have developed the Atlas of Urban Expansion. Solly and still more colleagues extended this work and are carrying it out under the rubric of the Marron Institute Land and Housing Survey in a Global Sample of Cities.
This effort focuses on 200 cities rigorously selected from a universe of 4231 cities (metro areas) with populations over 100,000. The sample is stratified by region, city population size, and number of cities in each country. There are actually two related surveys. One focuses on the regulatory environment, and the other focuses on housing market conditions (rents, prices, tenure type, household mobility…). Angel and colleagues have also calculated the percent of each country's total land area apparently covered by urban development, according to their satellite data. This figure shows their country-level estimates of this percentage.
 |
| Exhibit 4, from Solly Angel and colleagues |
Angel et al.'s data focuses on average densities for metro areas. For density patterns within cities, see Alain Bertaud's data on density. I've been honored to work with Alain on some of this project; a presentation that reviews our joint work to date, and presents some of the data, can be found here.
Primary density data have been collected by Bertaud, over 2+ decades. The primary unit of observation is the city's census tracts, or closest equivalent. The data are processed in GIS by Alain along with his collaborator Marie-Agnes Bertaud. Bertaud's method measures density in built-up areas only.
 |
| Exhibit 5: Tract average densities at identical scales, Paris and Moscow, from Alain Bertaud |
Exhibit 5 presents three-dimensional representations of Bertaud's analysis for two cities of then-similar size (10 million), Paris and Moscow. Each small polygon within a city represents the average density of a census-tract-like unit. The data are to the same scale horizontally, and the same scale vertically. With some exceptions (near La Defense), the densest locations in Paris are near the center of the city; in Moscow, the densest locations are near the periphery.
We can see this even more clearly with a re-expression of the data, Exhibit 6. Bertaud computes average density in each 1km annulus from the center of the city. There are a few data decision rules and tricks of the trade that are explained in the PowerPoint deck of an ongoing study Alain and I are undertaking, available here. We then compute several measures, including traditional density gradients. And don't forget our previous discussion of Alain's instant classic, Order Without Design.
 |
| Exhibit 6: Average population densities from Ex. 5, at each 1km annulus |
But by American standards, Paris is very dense. The peak density is half again that of New York, over four times that of Chicago, six times L.A.
Mortgages and Finance, including HOFINET
 |
| Exhibit 7: Country Level Mortgage Data from HOFINET and GDP Per Capita from WB WDI |
Penn's Marja Hoek-Smit has spearheaded a fantastic resource, Housing Finance Network, or HOFINET. HOFINET is a web portal with access to a wide range of research on data on international housing finance, and related topics. Data are organized by both topic and country. This is my go-to site when I want to learn about housing finance around the world, as well as in-depth discussion of many social and economic aspects of housing.
Specifically, HOFINET's objective is to provide a wide variety of global housing and financial information, focusing on:
- A standardized set of initial housing market, housing finance and policy measures that is tested internationally and can be expanded and adjusted over time.
- Housing finance laws and regulations for each country and pertinent research and consulting reports provided copyright laws allow posting.
- Short country assessments that focus on the unique features of each country’s system.
- A data download and analysis capability to facilitate research.
- Up-to-date international housing sector scholarship through theme pages.
- An ongoing forum for discussion with international experts.
Exhibit 7 presents some basic data gleaned from HOFINET. The vertical axis is the ratio between mortgage debt and GDP, the standard way of measuring the overall size of financial markets across countries. It’s unsurprising that this measure of mortgage market depth increases with per capita output. Richer countries tend to have more developed financial markets in general, and mortgage markets are of course an important segment of those financial markets. Further discussion, and references, can be found at a three-part set of blog posts I've published at Rutgers' Center for Real Estate, headed by Morris Davis: Part 1, Part 2, Part 3. I've also published a companion piece on the U.S. mortgage market here.
Housing Prices and Rents
Before we examine data sources, let us discuss all too briefly and important and somewhat complex question: What do we mean by the “price” of housing, anyway? Every house is more or less unique, in terms of its size, quality, associated services, and of course its location.
Consider a simple thought experiment. If I pay $1,000/ month rent and you pay $1,200, who’s paying a higher price? If our units are nearly identical, you do. If your apartment is 1500 square feet and mine is 500, maybe I do. If your apartment is in midtown Manhattan and mine is in Madison, maybe I do.
Other questions arise, e.g. how do we handle many characteristics at once? Are we looking at “flow” prices (per unit of time) or “stock” (“asset”) prices? And just to keep us on our toes, the way economists usually define "price" is different from common usage, especially in the housing market. Exhibit 8 presents a brief summary.
 |
| Exhibit 8: Economists' and Industry Terminology for Real Estate |
Even economists misuse these terms on occasion. I try to remember to say “value” or “asset price” when I mean the value of a unit. I try to say “price” when I’ve somehow controlled for the quantity of housing services produced by a unit. But I often fail, especially in casual conversation, or when speaking to a real estate audience, where one often needs to adapt to common usage.
- Simple medians, averages
- “Traditional” Laspeyres, Paasche price indexes
- Hedonic price models
- Repeat sales models
- User cost models
- Hybrid methods
Further discussion is beyond the scope of this already long post, here we simply want to note the issue. See Green and Malpezzi or other sources that discuss these issues in greater detail.
A good starting point for studying housing prices across countries is the IMF's Global Housing Watch. Spearheaded by Prakash Lougani and now run by Hites Ahir, Global Housing Watch provides quarterly data on house prices as well as metrics used to assess valuation in housing markets, such as house price‑to‑rent and house-price‑to‑income ratios. Global Housing Watch also contains many informative analyses of recent price trends, and links to relevant research.
See also the excellent blog where Hites and Prakash provide commentary on housing as well as a wider range of issues, The Unassuming Economist.
 |
| Exhibit 9: Malpezzi chart of real house price indices, annual country level data collected by Dallas Fed |
Exhibit 8 presents a chart I created using the Dallas Fed database, and discussed briefly here. National real house price indexes are rebased to 1980=100. Notice the U.S. "average" boom and bust is evident in this index, but France, Ireland and the UK are among countries that are more volatile still. In contrast, notice how flat Japan and Germany are over this period. There's much to discuss about sources and technical details of measuring prices, but first and foremost we note that national indexes, however constructed, typically conceal large within-country variances in prices. Consider the last few decades of house price behavior in, say, Newcastle compared to London; or Detroit compared to San Francisco.
Global Property Guide is a data and analytics site created by journalist Matthew Montagu-Pollock in 2005. Rent and price data are based on web scans for offerings of good quality houses and apartments. Thus they can be expected to over-estimate actual transactions prices/rents for the overall broader market. But many users (e.g. foreign investors) are interested in this market segment, and trends can still be indicative. They collect and present aggregates, and significant collateral data (financing terms, elements of the regulatory environment) for free at their website; more detailed data are offered for sale.
Bank for International Settlements Property Price Statistics. BIS is sometimes called "the central bank's central bank," and plays an important role in setting global standards for bank supervision and regulation. Given the well-known role housing and real estate plays in business cycles and the transmission of monetary policy to the real economy, it is no surprise that BIS has long collected both housing and commercial real estate prices.
BIS Database on Housing Policies A new database for policy actions on housing markets covers 60 economies worldwide from January 1990 (or earliest date available) to June 2012.
Demographia International Housing Affordability Survey is led by Wendell Cox. Demographia focuses on data from upper income countries, and China, specifically Australia, Canada, China, Ireland, Japan, New Zealand, Singapore, UK, USA. Demographia presents metropolican housing price data for 92 cities over 1 MM population. Data are annual, 2005 through 2018.
 |
| Exhibit 10: Screenshot of The Economist's House Price Database Chart |
The Economist Global House Price Database presents housing price indexes, normalize by rent (price-to-rent) and by income (price-to-income) ratios for 28 countries and 40 large cities. Exhibit 10 highlights London and New York against the background of other city trends. Data duration varies but goes back 30 years for a number of locations.
A more recently available but longer run data source is due to several economists from Bonn. Katharina Knoll, Moritz Schularick and Thomas Steger present very long run global house price indexes from 14 countries. The data are described in their paper appearing in the American Economic Review (107(2), 2017, pp. 331-53); the data are also downloadable from that site. Exhibit 11 presents their basic country data. The shaded bars represent the two world war periods.
 |
| Exhibit 11: Long-run house price indexes for 14 countries from Knoll et al. |
 |
| Exhibit 12: European residential data from Deloitte |
Exhibit 12 presents some of the house price and rent data from the Deloitte Property Index.
Commercial Real Estate
 |
| Exhibit 13: Mapping Q3 2019 real estate transactions from RCA |
Real Capital Analytics is a leading data source for commercial real estate prices, capital flows and other basic data. Since 2000, RCA records individual transaction-level data on most property types, in 172 countries:
- Office
- Industrial
- Retail
- Hotel
- Multifamily Residential
- Senior Housing/Healthcare
- Self Storage
- Development Sites
Exhibit 13 presents RCA's map of late 2019 commercial real estate transactions around the world. Exhibit 14 presents some representative data on commercial real estate price changes in 20 major metro areas. RCA aims to cover all transactions involving properties valued north of $2.5 million, in real time, both and pending deals of all types: sales, recapitalizations, partial interests; financing, defaults, and foreclosures.
 |
| Exhibit 14: RCA changes in CRE prices for selected global cities |
Jones Lang Lasalle (JLL) is another leading source of global real estate data and analysis. Exhibit 15 presents some representative data from their end-2019 report on global real estate markets on 16 major countries.
 |
| Exhibit 15: Real estate return summary from JLL Investment Strategy Annual 2020 |
 |
| Exhibit 16: Basic data on global REIT-like vehicles |
Global REIT associations provide information on listed public real estate markets. These are more or less similar to U.S. Real Estate Investment Trusts. Currently about 38 countries have adopted such vehicles, with perhaps 10 considering future adoption. Exhibit 16 presents some summary data on the global scope of these vehicles. Real Estate Equity Securitization Alliance (REESA) members include:
- National Association of Real Estate Investment Trusts (NAREIT)
- Asian Pacific Real Estate Association, (APREA)
- Association for Real Estate Securitization (ARES)
- British Property Federation (BPF)
- European Public Real Estate Association (EPRA)
- Property Council of Australia (PCA)
- Real Property Association of Canada (Realpac)
 |
| Exhibit 17: CREDA participants as of 2019 |
The Commercial Real Estate Data Alliance (CREDA) is a consortium of leading real estate academics and professionals; the name is somewhat self-explanatory.
One lens through which to see the consortium’s motivation is through their repeated reference to the desire to put real estate on an even footing with traditional financial markets. Analysis of equity markets in particular was greatly enhanced when the University of Chicago’s CRSP (Center for Research in Security Prices) data became available. CRSP made a large variety of equity pricing data available on a timely, consistent basis, pre-cleaned and ready for analysis.
So one way of thinking of CREDA is CRSP for real estate. Headquartered at the University of North Carolina’s Wood Center for Real Estate Studies and UNC’s Institute for Private Capital, CREDA involves leading academics as well as private firms and professional organizations (Exhibit 17).
To date, the focus of CREDA has been primarily on US commercial real estate data. But we include it here because many of the participating academics in organizations are also internationally focused, and none of the data issues CREDA attacks are unique to the United States. I would speculate that CREDA’s efforts will eventually “go global,” whether formally or informally, to help improve the standardization and availability of data across countries.
CREDA is sponsoring several pilot studies that focus on the U.S. at the moment but that will have utility to global providers. One project is developing a platform to merge and scrub multiple property level data sources including providing an open source property address matching methodology as well as a one-stop shop for documenting commercial real estate data sources. Another CREDA sponsored study involves mapping permit data to capital expenditure data. CapEx data is not currently widely available; but most capital improvements require a building permit, which are available. To date this data is not well matched and of inconsistent quality so the research project is analyzing statistical relationships between permit data and actual CapExs which will allow extrapolation to any property or portfolio.
CREDA also sponsors an annual research symposium at UNC bringing together academic industry and government researchers, investors, and data providers. In addition to data issues the forums provide panels that discuss current policy issues such as housing affordability in light of new data.
Measuring Institutions and Policies
Another strand of international urban indicators revolves around measuring the urban policy environment. We begin with an early effort at measuring the general economic policy environment. Ramgopal Agarwala created a classic index of economic policies for 32 "developing countries" for the 1983 World Development Report. The index is based on policies re:
- Exchange rates
- Interest rates
- Labor pricing (e.g. minimum wages)
- Trade (tariffs, etc.)
- Agricultural pricing
- Infrastructure pricing
- Inflation
Agarwala and team surveyed a range of studies and data, put each country into one of three “buckets:”
- “Do the right thing,” more or less (e.g. market exchange rate), score 1.
- Significant price distortion, e.g. exchange rate overvalued by 10 percent, score 2.
- Serious price distortion, e.g. Ghana’s exchange rate overvalued by several multiples, score 3.
They then constructed a simple additive index. Minimum (best) score is 7, maximum (worst) score is 21. In practice: Malawi scored 8, Thailand 10, Cameroon, Korea, Malaysia, Philippines, Tunisia scored 11. At the other extreme, Argentina and Chile scored 17, Tanzania and Bangladesh scored 18, Nigeria scored 19, Ghana scored 20. (Of course, these policy indexes are 40 years old. There is substantial serial correlation in policies. But policies do and have changed substantially over time.) Exhibit 18 presents the basic Agarwala data.
 |
| Exhibit 18: Agarwala's (1983) coding of economic policies |
 |
| Exhibit 19 |
Research by Angel and Mayo (described below) finds that house prices-to-income don’t vary terribly systematically with income. But that does not mean there are no systematic patterns. Exhibit 19, from Malpezzi (1990) compared housing prices, normalized by income, to the Agarwala index. At the time we had no purpose-build index of urban policies, but Malpezzi (1990) argued that countries that distorted their labor and financial markets in general terms would likely be following policies that would also distort the housing market. The data are consistent with such an interpretation of the Agarwala index as a proxy for a general policy environment. Circa 1990, countries that "did the right thing" in economic policies experienced house price-to-income ratios averaging 4 or 5, while those with distorted policies averaged 7 or 8.
More data became available in the early 90s from reforming socialist economies after the breakup of the Soviet Union, and the imposition of post-Mao reforms in China. Other analyses (not shown here, see Buckley and Malpezzi, and Buckley and Tsenkova) found that in the early days of reform, back-of-the-envelope applications of the Agarwala framework and data on house prices found that many of these countries, placed closer to Ghana on the policy framework, with prices even more distorted that we observe in Exhibit 19.
The correlation of the Agarwala Index with various housing market outcomes motivated several attempts to construct measures that were more directly based on urban and housing policies. One of the first efforts focused on rent controls, motivated in part by research by Richard Arnott among others on the importance of analyzing specific features of rent regulation regimes. (Arnott's original paper with Nigel Johnston dates from 1981; a later, more readily accessible version is available here). A survey of some 50 countries was undertaken as part of a World Bank research project on rent control described in Malpezzi and Ball (1991, 1993).
Countries which have no controls receive a zero rating. Other countries are rated as follows. There are nine elements; for all but one, each element receives a rating of 0 (permissive), 1 (medium), 2 (restrictive). The first two elements are:
- Enforcement: if controls are not enforced or rarely enforced, the country receives a zero score. Selective or partial enforcement scores a point; strict enforcement, two points.
- Coverage: if coverage is restricted to a very small part of the market, the country receives a zero score. If a significant part of the market is covered, the country receives a score of 1. If more than half the market is covered, the country receives a score of 2.
- Fair Rents: Countries which do not set rent levels for units receive a zero rating; those with some units so covered or no information, 1; those with stringent rent setting, 2.
- Indexation: If rents are indexed and closely tied to inflation, the country receives a zero rating. If rents are partially indexed or no information, 1; if rents are frozen or rarely revalued, 2.
- Cost Pass-Through: Are upgrading, maintenance, and tax increases passed through to tenants? If often, 0; if some items are passed through, or no information, 1; if no or little pass through, 2.
- Treatment of New Construction: If newly constructed units are exempt, score zero. If newly constructed units have a temporary exemption, or some other differential treatment, or there is no information, score 1. If new construction is controlled as other rental housing, score 2.
- Rents Reset On New Tenancy: If rents reset to market on new tenancy, 0; if revalued but below market, or no information, 1; if no change, 2.
- Tenure Security: If tenure security is more or less covered by private agreement (leases) and normal grounds for eviction, 0; if more stringent security of tenure or no information, 1; if strict security of tenure, 2.
Some countries, such as the United States, and Canada, have many rent control regimes which vary greatly from place to place. In such cases, when there was substantial divergence from place to place in an element, we graded the element 1.
Key findings, which can be found in Malpezzi and Ball, include the following.
First, consistent with priors, the rent control index was correlated with the Agarwala index and other measures of general economic policies, such as contemporaneous measures of the overvaluation of a nation's currency.
Second, while many countries had some form of rent regulation in the late 1980s, the most stringent regulations were usually found in poor countries. Other research, described in the research project's final report, found that there were often ways both households and landlords evaded the rules, including illegal side payments (key money) and accelerated depreciation of units. Nevertheless, there was evidence, unsurprisingly, that rent burdens were lower, on average, in countries with the most stringent controls. On the other hand, asset prices were higher, relative to incomes, in those distorted markets, plausibly because stringent controls were present in markets that had other distortions on the supply side of the market. And markets with the most stringent controls had lower rates of housing investment, ceteris paribus.
So circa 1990, findings using the Agarwala index, on the one hand, and the rent control index, on the other, gave us confidence that such "policy indexes" could be informative. It was, and is, however, important to remember the limitations of any such index. In brief.... Any policy index perforce contains information on only a small fraction of interventions that affect housing, or any other urban market under consideration. Taken with the observed correlation among a number of diverse such indexes, both across countries and across cities within a country, we should never interpret any coefficient literally. For example, "the coefficient of my regulatory index is 0.1 in a logarithmic housing price model -- so if we change a regulation that theoretically reduces our index by a value of 1, we expect a 10 percent reduction in housing prices...." Well, maybe not, because that regulation, zoning or FAR or whatever, may well represent itself but also represent some other regulations that are not included in the index but which are correlated with the included measures. The problem is analogous to the interpretation of individual coefficient estimates in a hedonic regression model, as explained in papers by Butler and by Ozanne and Malpezzi.
In the event, some of us think of such regulatory indexes as akin to the "stupid pet tricks" readers of a certain age will remember from a popular late night comedy show. It's surprising, really, that they work at all.
Attempts to improve on these early efforts led to their inclusion in the Angel-Mayo World Bank-Habitat Urban Indicators Project, described generally near the beginning of this post. Wave I did the most thorough job of attempting to measure the policy environment. Angel and Mayo constructed policy indexes in six areas: property rights, housing finance, housing subsidies, residential infrastructure, the regulatory regime, and the industrial organization of the housing market. Subindexes were constructed for each area, and they were also combined in an overall "enabling index." Exhibit 20 presents a schematic summary of the effort.
 |
| Exhibit 20: Schematic of the Angel-Mayo Housing Enabling Policy Index |
Details of the indexes and their construction can be found in Angel and Mayo (1996) and especially in Angel (2000). Exhibit 21 presents the six subindexes, and the aggregated enabling index. Higher values of the indexes represent better policy environments.
 |
| Exhibit 21: Six housing policy sub-indexes, and the aggregated Enabling Index from Angel and Mayo |
A number of analyses were carried out by Angel and Mayo in the aforementioned publications, examining a range of incomes including rents, asset prices, investment levels, and housing tenure. Here we will present two simple graphics that can motivate a more careful reading of their original research.
 |
| Exhibit 22: Average supply curves for countries grouped by values of the Enabling Index |
Angel and Mayo (1996) group countries into three categories: low enablers, medium enablers, and high enablers, depending on the level of the overall enabling index. Exhibit 22 presents the supply elasticities they estimated by group. As expected, estimated supply elasticities were neatly ordered by the level of the enabling index.
 |
| Exhibit 23: Angel and Mayo (1996) House Price-to-Income (HPI) and Rent-to-Price (RPI) ratios |
Exhibit 23 presents to more data elements from Angel and Mayo: each capital citiy’s house price to income ratio, and the corresponding rent to income ratio. Exhibit 24 plots the house price to income ratio by the enabling index. Taking the plot at face value, better enablers (lower values)) perform better, though results are hardly tight. The regression line through the city specific points shows the expected positive relationship but is heavily influenced by China and the fit is not particularly impressive.
 |
| Exhibit 24 |
Let’s wrap up our discussion of the classic urban indicators of Angel and Mayo and their associates by noting that in addition to the World Bank’s 1993 housing policy paper Enabling Housing Markets to Work, they produced several examples separately and together of country and regional discussions of housing policy reforms that brought indicators data in as diagnostic tools. See for example Angel’s study of Panama, and Mayo’s study of South Africa.
 |
| Exhibit 25 Kim Malpezzi and Kim survey questions on housing policy |
Another effort at measuring the urban policy environment was carried out for Korea’s Center for Private Enterprise by Kim Malpezzi and Kim. Exhibit 25 presents a summary of their data collection for regulatory elements; with help from local experts they collected these for some 20 cities. In the event, Kim Malpezzi Kim found that housing rents and prices were correlated in the expected manner with the regulatory index and Kim found that in Asia supply elasticities were lower in markets with more restrictive environments. But in both cases the results were far from robust. A recurring bottom line of this and other studies is that it’s often difficult to get consistent information even with carefully designed questionnaires using a range of partners in countries with very different institutional environments and development traditions. Such indexes seem to work better (provide more reliable information) within countries than across them.
 |
| Exhibit 26: Kim-Malpezzi-Kim key data for 15 cities including Regulation Index |
We’ve discussed Hernando’s De Soto’s pathbreaking (no pun intended) analysis of Peruvian development timelines detailed in his book El Otro Sendero (The Other Path) in our previous post on the so-called Malaysia model. Exhibit 27 presents De Soto’s original schematic for the procedures required to formalize a legally obtained home in Peru. As reviewed in our previous post on the Malaysia model e Malaysian developer MK Sen independently provided us with a very similar analysis for the Malaysian market, as detailed in Hannah Bertaud Malpezzi and Mayo (1989).
 |
| Exhibit 27: De Soto's original study of the timeline to formalize a home in Peru |
Lex Mundi project. This network has been used to collect indicators of the legal framework for real estate and other economic activities for range of countries. A series of papers by Simeon Djankov and several associates details these efforts.
Collects data from a network of law firms on the nature of property rights and contract enforcement procedures across countries. See Djankov, La Porta, Lopez-de-Silanes and Shleifer (2004)
 |
| Exhibit 28: Example of complex procedures to register a firm |
Exhibit 28 provides a schematic of the start-up procedure for a small firm in France. Procedures are lined up sequentially on the horizontal axis. The time required to complete each procedure is described by the height of the bar and measured against the left scale. Cumulative costs are plotted using a line and measured against the right scale.
Labels on the horizontal axis are hardly legible so we repeat them here:
1. Check name uniqueness.
2. Notarize lease and obtain home office approval.
3. Certificate of clean criminal record.
4. Certificate of marital status.
5. Power of attorney.
6. Open bank account and deposit capital.
7. Notarize company deeds.
8. Publish headquarters address in Journal.
9. Register articles of association at tax office.
10. Request registration.
11. Form K for business.
12. Unblock capital.
13. Notify post office.
14. Designate bondsmen and guarantee taxes.
15. Initial firm ledgers and registers.
16. File for Social Security.
See also https://www.lawyersfrance.eu/starting-a-business-in-france
 |
| Exhibit 29: Regulatory complexity, and labor market informality |
Djankov et al. went beyond New Zealand and France to collect these and other indicators of the regulatory process for some 40 countries. Exhibit 29 presents the loglinear relationship between the number of procedures to register a firm, and the percentage of the labor force employed in the unofficial or informal economy. Unsurprisingly, the denser the regulatory thicket, the larger the fraction of the labor force employed outside the strictly legal framework. This is no small matter since the costs imposed on citizens by informality has been detailed many times before including in De Soto’s book.
The next step in the evolution of these indexes was to institutionalize this kind of cross country data collection in the World Bank’s Doing Business surveys. Exhibit 30 presents an overview of the kinds of data collected.
 |
| Exhibit 30: Summary of Doing Business elements |
Another problem related to the caveat we made above regarding care not to over interpret a single element or coefficient in such index elements is the charge that as Doing Business became more prominent, countries might “game the index” by focusing attention on one or a few elements that give them a chance to move up in the rankings published annually by the World Bank, but make a limited contribution to easing total regulatory burdens.
Yet another set of criticisms focuses on potential downsides of labor market flexibility. One man’s or woman’s labor inflexibility is another’s protection from arbitrary dismissal or unsafe working conditions. More on this in the next version of this blog post.
 |
| Exhibit 31: JLL Managing Director comments on India's Doing Business score improvement |
India’s jump on WB’s Doing Business 2018 Index: A real estate analysis
JLL’s India Managing Director Shubhranshu Pani
Housing.com, November 7, 2017
Real estate-specific improvements cited by Pani include:
- Single window approval system for building plans.
- Streamlining business incorporation processes.
- Easing tax compliance procedures (via online filing, consolidation through GST, etc.).
- Bankruptcy and insolvency.
- Easing export-import border compliance procedures.
- Establishment of debt recovery tribunals in India, associated with a reduction in non-performing loans of 28 per cent.
The Doing Business survey itself has four data elements related to real estate development:
- Time required to register a property in days.
- Number of procedures to register a property.
- Time required to build a warehouse in days
- Number of procedures to build a warehouse.
 |
| Exhibit 32: Exploratory Data Analysis of house prices and World Bank Doing Business data |
As part of ongoing research with Alain Bertaud, I constructed a simple index from Doing Business real estate elements. First I undertook simple standardization of each of four components; then summed the four components to construct a simple index. For housing price data I used Global Property Guide house price per square meter, mainly because it was conveniently available. That data, as we noted above, is focused on the high-end. A future version of this blog may present more representative prices. In the event I normalize these prices by dividing them by each country’s World Bank Atlas method per capita GDP.
As the data are plotted we find a few surprises among the indexes. Taking the data at face value, according to the Doing Business data, Russia appears to be one of the easiest places to do real estate business. That would be a surprise to President Trump who has not yet managed to build his longed-for Moscow hotel. At the other extreme, Portugal is no easy place to do business – in fact it was a World Bank debate on over Portugal’s housing policies that help motivate the World Bank research program on rent control discussed above. Nevertheless, we might be surprised to see Portugal as such an outlier on the regulatory index.
Taking all this data at face value there is not much correlation between these crudely normalized housing prices and the simple regulatory index. The regression line has the expected sign but only 3% of the variance in the house price to per capita GDP ratio is explained.
Jones Lang LaSalle Real Estate Transparency Index JLL's Index aims to measure the quality and quantity of each country's real estate data. Historically real estate markets have been largely private and opaque. But a tradeoff exists. Increasing transparency may reduce excess returns to selected agents, eroding the value of private information; but in general this will be more than offset by the gains associated with attracting more capital, a wider range of investors, and overall increased efficiency and fairness of a market. See Gordon (2000) for elaboration of these arguments.
 |
| Exhibit 33: Elements of JLL's Transparency Index |
As Exhibit 34 illustrates, more transparent markets according to the JLL index (lower values of the index) are associated with greater transaction volumes. Again, correlation or causality? More work to do here.
 |
| Exhibit 34: Real estate transparency index and transparency volumes, from Jacques Gordan et al. JLL |
Is "Big Data" the Future? How Will Traditional Real Estate Productivity Measures Evolve in the World of Emerging "PropTech"?
 |
| Exhibit 35: Exponential growth of global data, broadly defined, over time |
Gigabyte: a thousand megabytes (10^9 bytes)
Terabyte a thousand gigabytes (10^12 bytes)
Petabyte:a thousand terabytes (10^15 bytes)
Exabyte: a thousand petabytes (10^18 bytes)
Zettabyte: a thousand exabytes (10^21 bytes)
Yottabyte: a thousand zettabytes (10^24 bytes)
To get a sense of scale, 10 terabytes would hold all the printed books in the Library of Congress; 2 petabytes would hold all U.S. academic research libraries; 5 exabytes would hold all the words ever spoken by human beings.
 |
| Exhibit 36: Some terms encountered in the world of "Big Data" |
Fisher (2015) presents a cogent introduction to big data and related topics. . Fisher quotes Gartner’s definition “big data is high-volume, high velocity, high variety information assets that require new forms of processing to enable enhanced decision-making, insight discovery and process optimization.”
Fisher begins by setting up a brief discussion of what he calls traditional data such as prices, rents, rates of return, transaction volume and so on; from traditional sources like RCA, NCREIF, and other sources discussed in this blog post as well as many others. He points out that traditional data are generally well structured, set up in time series and cross-sectional; and can be analyzed with traditional statistical and other analytical tools. Big data is, well, “big,” often nearly contemporaneously collected, and often analyzed using primarily descriptive black box modeling.
Big data are often noisy and unstructured from a wide range of sources. Data are often scraped from the web, from both traditional (e.g. Zillow’s scraping of property tax records) but also from nontraditional sources such as social media (Twitter, Facebook, LinkedIn) as well as informal documents published on the web in a variety of physical formats as well as blogs, discussion forums etc. Data can include video, audio and images as well as traditional numbers and words. The proliferation of the Internet of Things (IoT) presages another threshold leap in the availability of data, in subject matter, quantity, and form.
As Fisher points out these data require new ways of storing and analyzing them as well as collection protocols. Signaling his concern with data quality, Fisher points out that many observers are adding “veracity” to the measured characteristics of big data. Big data analysts often eschew traditional statistical techniques like regression analysis in favor of machine learning and various systems of classification including heavy reliance on visual methods.
As a cautionary tale Fisher (2017) does a quick analysis of correlates of housing starts with Google searches. He finds strong correlations with a number of searches that are hard to sensibly link to housing starts including “Copacabana New York,” “Ramada Plaza downtown,” and “Tower Records.com.” Fisher is making a point with a quick analysis of trends, and doesn’t provide much detail about the exact timing or strategy for his analysis. But the example is sufficient as a cautionary tale, pointing out among other things the ephemeral nature of these correlations. Tower Records, years ago one of my favorite stops for music, went out of business in 2006 after earlier bankruptcies. Perhaps the trending phrase was connected to the redevelopment of some of Tower’s better-known locations in cities like Boston and San Francisco, but it is doubtful Tower has much to tell us about housing starts today, if it ever truly did.
 |
| Exhibit 37: One survey of how real estate operators are currently using Internet-of-Things data |
 |
| Exhibit 38: PropTech investment boomed in recent years |
 |
| Exhibit 39: Examples of PropTech companies/startups from Jacques Gordon (need his original source) |
Next steps might be to impose more structure on big data analyses, that is to get away from the black box syndrome. Correlation is still not causality. The future may well see current data mining techniques more effectively combined with old-fashioned techniques attacking old-fashioned issues like endogeneity, robustness, and more or less reliable prediction out of sample.
Coming Later: A Separate Blog Post on Urban Indicators with a Domestic U.S. Focus
Lots do to here, including discussion of many housing and urban indicators.
The You-Know-What in the Room
 |
We just went through dozens of sources of economic and real estate data without reference to the Elephant in the Room. |
The blog directly preceding this one contains my arguably obsessive notes and teaching materials on the coronavirus and its effects on economies, housing and real estate markets, and people’s lives. Yet we’ve been through some 40 exhibits and even more links to economic and real estate data without a word on the coronavirus.
Unsurprisingly, in due course these thoughts on indicators will have to include some discussion of this particular elephant. That will come in time. Meanwhile I hope this version with its introduction and links helps students and researchers and other readers.
References and Links
 |
| Students in Guinea reading under streetlight |
Be glad you probably have a much better reading environment. Click here to download a list of references.
