Richard Davies



UK prices


economics, business, big data, firms, wages, finance

This page houses my work on prices and inflation, using 'micro data' to investigate inflationary puzzles and implications. I update the page monthly, my latest discussion note is here, and working paper is here.

The UK has great price data, with official CPI 'micro data' released with a lag of only one month, allowing work on prices, inflation and real wages to be conducted in (close to) real time. I have used this to create a long-run prices database (LRPD), which make available here.

To get started, select an item using the drop-down box below. The prices that underly our official inflation measures behave in interesting, and very different, ways.

Papers and ongoing research

Some papers and blogs using this data include:

This is ongoing research, if you have any comments or would like to use the data, please get in touch.

Research and teaching - examples

Some examples of courses and websites that have used the data either as a teaching resource or as an analytical tool:

/// LSE. The Economics for Python weekend designed and taught by Rahat Siddique. In the latest iteration of this, "Team BoE" put together some intersting new indices. You can get their presentation here, and Google Colab code (Pyhton code, running in the browswer) here.

/// Imperial. The Data Stories course with Ralf Martin.

/// Bristol. The Communicating Economics course run with Sarah Smith and Christian Spielmann.

/// UCL. The Data Skills Lab with Parama Chaudhury.

/// Economics Observatory. Analysis of the UK's 'Tampon Tax' by Sarah Smith, and related piece by Gemma Williams.

The long run prices database - using the data

I update the LRPD monthly, the latest update was 21st April 2021. The data is stored in the following files (all csv):

/// Prices. (Full database). This contains the price data. It is a very large file (there are currently ~41m observations.) To conserve memory all supplementary details are kept in other databases that can be linked to when needed. Update: lots of people are opening this in Excel and thereby truncating the data to 1m observations - please get in touch if you want the file and I will send it directly.

/// Prices. (5% sample). This is a 5% sample of the data, created to ease sharing and to get a feel for the data. If you get stuck with the large file above, please run your code on this. The variable names and definitions are identical, so you can work on it while I get you the big file. Too large to open in Excel.

/// Deciles. This file allows you to track the distribution of prices. It is the 10th to 90th percentiles inclusive in decile steps, for each item in each month. Around 200k observations - can be used in Excel.

/// Item. Information on the consumer products: their full names, descriptions, when they appear and exit the CPI, and the number of observations for each. Small file.

/// Region. The location the price quote was collected. Small file.

/// Date. The date on which the price quote was collected. Date is recorded in many formats (dd/mm/yyyy, yyyy-mm-dd, etc) to make it compatible with different types of software. Small file.

If you use the data and have comments on how to make it more useful, please get in touch.

Examples: interactive charts

Number of observations - by region

The ONS collects prices in proportion to the population in each of the NUTS1 areas. The pattern of useable prices (i.e post cleaning) that are available in the LRPD are below.

The Covid-19 pandemic disrupted price collection severely. Another version of the same chart, this time drawn for 2018-2021 allows us to focus on the recent volatility. Click on the legend to select a region and isolate its data.

Example - interactive chart: inflation vs volatility

The chart below plots cumulative inflation, by item, against the volatility of that item's prices. Some items change price a lot but the pattern of rises and falls generates little long-run inflation (e.g. lettuce, cucumber, sprouts, strawberries). Others change price infrequently, but these changes build over time to create large inflationary changes (e.g. haircuts, fish and chips, cheese).

The chart below is interactive. Try clicking on the legend to select the type of good you are interested in. Then hovering over dots to get more information. You can zoom ina and out on croweded areas of dots using your mouse or tracker ball. Coded in Vega Lite.

[Chart currently being updated]

Example: animation: pints of bitter

The humble pint of bitter is one of the most sampled prices in the UK CPI. Here is how we went from 80p a pint in 1988 to a huge range of prices, inlcuding £6 a pint today. Note the peak that emerges at £2, which a knowledgeable drinker tells me is the 'Wetherspoons Effect'. Creating using simple stop frame animation (running a loop to create 395 monthly charts).