Contrast:

The issue

Monitoring and managing activities under Local Area Agreements (LAA) requires the ability not just to gather but to analyse data and make sense of it. A variety of steps in the LAA process require data analysis skills.

• To build the evidence required to support the development of strategic priorities it is necessary to establish baselines that genuinely reflect conditions across the area.
• To convert this evidence into relevant interventions means identifying causal links that explain how and why the issue arose – without understanding the cause it is hard to work out an appropriate response.
• The monitoring needed for effective performance management requires analysis to identify what improvements are needed and how they should be implemented.

The Treasury’s Sub-national economic development and regeneration review, published in 2007, suggested that (subject to consultation) the government wants to introduce a statutory duty for ‘…all upper tier authorities to carry out an assessment of the economic circumstances and challenges of their local economy’ (though lower tier authorities are likely to want to undertake their own to feed into the upper tier assessment). This will present a new and specific challenge for local authorities to develop analytical skills.

No matter how good the data, its usefulness will depend on how well it is analysed and interpreted.  Analysis is a process for transforming a collection of data into a body of information that has meaning and significance.

This document is part of a series in the library looking at how to gather, analyse and use data to provide an evidence base to underpin your policy making and decision taking.  It looks at how to analyse and interpret the data you have collected.

Practitioners in the public sector have no always undertaken data analysis as effectively as data gathering. Reports present an undigested mass of figures and facts, but do not tell the reader what they mean, or what their significance is. There is an absence of clear links between the statistical information about an area, a population group, a problem, and the kind of interventions that are proposed: the data is not used to demonstrate need or demand for a project.

The results of poor data analysis are seen in a number of forms.

• Services are poorly targeted, leaving un-met needs amongst vulnerable populations.
• There is a poor understanding of the needs and opportunities in an area, with resources wasted on projects that fail to bring benefits.
• Interventions do not focus on the cause of the problem.
• Monitoring and evaluation are poor.
• Performance management is also poor, with no ability to monitor effectively the delivery of targets and priorities or to demonstrate convincingly the impact of an initiative on an area.

Performance management has become an increasingly important task for which data needs be regularly collected and analysed. Key activities are monitoring a baseline, benchmarking and monitoring progress in achieving targets and priorities.

Whilst good data is of little use if it is not analysed and interpreted, it is also the case that poor quality data will not yield an accurate understanding or useful insights. Most people will be familiar with the expression ‘garbage in – garbage out!’ At one time it was difficult to get detailed statistical information for small areas but this has improved dramatically in recent years with the introduction of on-line sources of data. So there is now no excuse for not collecting the sort of high quality data that will enable you to do analysis that will develop your own understanding, and enable you to communicate this understanding to others.

How to analyse data

The purpose of analysing data is to understand the issue or topic on which you have gathered the data, and to provide you with the evidence you need to communicate this understanding to others. Analysing data will enable you to understand your area – its distinctive characteristics, what is happening there, and why, and the kind of interventions that lead to improvements.

Good data analysis involves a mixture of ‘asking questions’ of the data and investigating hypotheses and theories, whilst also being alert to the new insights that the data reveals to you. You need to examine the data to see what strikes you about the figures; what looks interesting or unusual, what questions are raised, what assumptions are questioned?

The type of analysis you will need to produce depends on the purpose for which the data is required and the task you need it for. However certain processes are common to most of the kinds of analysis of quantitative data you are likely to do, involving:

• measuring: establishing scale
• identifying differences: establishing distinctive characteristics of sub-sets of the total
• making comparisons with other areas and contextualising
• measuring change
• noting correlations (distinguishing between those that are causally linked and those that are simply associated)
• quality checking the data, and looking at data from a number of sources where possible.

The processes of identifying differences, measuring change and making comparisons, especially those between different groups in the population, different geographical areas and different time periods, are key to good analysis of quantitative data, and are also relevant to qualitative analysis.

Different groups within the population

When you are looking at data across the main policy themes (for example worklessness, education, health, crime, and housing and so on), you will need to establish whether different groups within your target area are affected differently. You will also want to compare the experience of people in your area with that of those in other areas.

Key ways of breaking down data are by ethnicity, gender and age, called variables. Within an overall figure for the number of people unemployed, for example, you will need to look at the figures for different population groups, by age, gender, ethnicity, qualifications etc. But it may also be illuminating to break groups down further, for example by separating the figures for young males in the 16-24 year old group separately from those for young women, or to look at the experience of members of ethnic minorities by gender and age, not just ethnicity. There are many other variables that are likely to be relevant: tenure type, educational attainment, health/disability status, dependents, being just a few examples.

As well as looking at how different population groups are affected within a theme such as housing, health etc, good analysis should also look at how one specific group, for example children, or the elderly, are affected across all the issues to build up a comprehensive picture of the experience of that group. This approach helps to identify the sort of links, (for example people experiencing poor health because of being poorly housed,) that tend not to emerge if data is analysed in ‘silos’ (that is, monolithic blocks of data that are treated as being self contained.)

Different geographical areas 

This involves either comparing your area with other areas; or looking at differences within the area. Comparing your area with others helps to put the situation in your area in context. This allows you to see where your area resembles others, and where it is distinctive, for example in relation to unemployment, single parents, rates of sickness and ill health. Those responsible for the development of plans and strategies often neglect to do this, presenting a mass of data about the key features of an area without providing comparative information. This would show whether the situation is worse or better than elsewhere, and whether the rate of change is the same, slower, or faster.

You should have obtained data for a number of potential comparative areas when collecting the data:  the analysis stage is where you review the results and decide which areas offer the most illuminating comparisons, and then make the comparisons and explain their significance. You need to be careful when selecting the area(s) against which to compare your own. You might choose, for example, an actual area, such as another similar one, or the rest of the borough or sub-region. Alternatively you can select the average for a number of areas. It can be useful to use a number of points of comparison in your analysis, and averages for a number of areas usually provide the most valid comparisons. For example, a comparison with the average for other similar areas will show whether your area is better or worse than others, but it will not show how much worse a particular situation is in these areas than in the rest of the country. You need a comparison with the regional or the national average to show this.

It can also be informative to compare your area with one(s) that share the same strongly marked or significant characteristics. You might want to make comparisons with an area where the population has a similar ethnic make-up or a similar industrial and economic history. Comparing your area to ones that have experienced similar drivers of change should provide valuable insights. The drivers of decline are very different in an area of industrial decline where there is a falling population and housing abandonment and an old seaside resort where the collapse of English seaside tourism has seen redundant hotels turned into hostels for the homeless, care homes and cheap bedsits.

Different points in time

Snapshots describing a situation or problem at a particular moment in time can be useful.  However making comparisons between the same situation or problem at different points in time reveals trends and allows you to measure change. Data which describes the same phenomenon at different points in time is referred to as time series data.

Trend data enable you to identify the direction of change, and comparing data about changes in one problem with data about another can be illuminating and identify possible causal links. For example, are the numbers of all crimes increasing, or is one sort of crime increasing whilst another is falling? You can compare the trend in your area with those in other areas, up to the national level, to see whether the trend locally is rising or falling faster than elsewhere.

Analysing quantitative and qualitative data

Different approaches are needed for analysing quantitative and qualitative data.

Quantitative data measures variables, expressing their quantity, extent or strength numerically. Statistics are of course, a form of quantitative data, and are particularly useful in that they enable information to be compared in a meaningful way between populations or groups of very different sizes or groups with different characteristics or attributes. Much of the information you will be using will be statistical. You will generally be doing a secondary analysis – dealing with previously analysed data (for example the Census) – rather than primary analysis – dealing with raw data produced by a study. It is important that you understand what the statistics mean if you are to interpret your analysis, and guidance is given in the next paper in this series which look at using evidence.

Qualitative data usually involves interpreting words, though this data could also take the form of pictures, photographs or videos. Information about local perceptions of problems, needs, progress and so on, are important to the development of LAAs, and regarded by the government as essential. Some information about these can be obtained through quantitative methodologies like surveys. However, much useful information can be gathered through methods that produce qualitative data, like focus groups, case studies, or workshops. This information can be very valuable: it can capture information and give in-depth insights that cannot be obtained from quantitative data. Used well, qualitative data can deepen the understanding of the problem or situation being described. However, qualitative information needs to be treated carefully.

• If the information has come from a small number of people, you cannot assume they are representative.
• Consider who recorded the information – might there have been any bias (conscious or unconscious) in the way the information was recorded?
• Is the information of a sufficiently recent date to be valid?
• The scope for comparing information may be limited, as you cannot be sure that you are comparing ‘like with like.’

You can turn qualitative data into quantitative, by coding responses. You may have verbatim records of lengthy verbal replies to open questions. This sort of data tends to be unmanageable, but can be made useful by identifying common themes within the responses, and allocating numbers to each commonly occurring response. You can then count how many times each type of response occurs. Only quantify qualitative data where you are sure this is valid and legitimate. Do not impose a statistical analysis where this is inappropriate, or if the numbers involved are low.

Performance management

Performance management has become an increasingly important task for which data needs be collected and analysed. The recent LAA Operational Guidance pointed out that performance management requires the ‘…effective collection and use of local management data so that it is timely, appropriate quality, captures the views of local people and can be shared between partners.’ This further underlines the importance of good data analysis. Some of the key activities are:

• Measuring change against the baseline: most organisations have developed their baseline: up-dating it at regular intervals is essential to measure how the area is changing and thus assess whether progress is being made towards objectives.
• Benchmarking: benchmarks, which are reference points against which progress and performance are measured, set a standard you aim towards. Regular detailed analysis of performance in relation to benchmarks identifies where action is needed.
• Measuring performance against targets is central, of course, for LAAs. LAAs give rise to a need for cross partner performance management systems which will make demands on you in terms of data analysis. 

Actions summary

• Never leave data to speak for itself. Through analysis you have to transform a collection of data into a body of information that has meaning and significance.
• You need to be confident that the data you are going to analyse is of a good quality (reliable, robust, accurate.) If you did not collect it yourself, review the sources and collection methods and form a judgement. Check any data that you have doubts about, and be prepared to exclude any data where you have serious doubts about its quality.
• Use the data to test hypothesis and theories, and to seek answers to specific questions, but be alert to any new insights that emerge from the data. Be prepared to question your assumptions or the received wisdom if the data is disproving them, however inconvenient this might be.
• Relate the type of analysis you carry out to the purpose for which you need the data and your product. Think about who will be your audience, and their needs and capacity (for example to understand complex statistics.)
• Contextualise data about areas, population groups etc by comparing it with data from elsewhere. Select area(s) for comparison where the exercise will help you to understand better what is happening in your area. The average for a number of areas can be a useful starting point. 
• Looking at trends over time rather than a simple snapshot of a single moment will nearly always yield more useful information.
• Use different and appropriate techniques to analyse quantitative and qualitative information.
• Analyse data from a number of different sources, as a cross check. Think what the explanation for any differences might be. Check whether there are differences in the way that the different sources collected or analysed their information, and be very careful how you handle data not on the same basis. 
• Analyse associated data in parallel, for example look at all health problems side by side. This will enable you to get a fuller picture – are all the problems getting better or worse, or are some better and some worse, and what does this tell you?
• Make linkages between different bits of data. Look for cross-cutting issues within and across themes. Avoid having a silo mentality towards the different regeneration themes.
• Avoid assuming a causal relationship between associated data.
• Are there any gaps in the data? Can you fill these without having to do original research?  If not, is the missing information important enough to warrant the effort?
• Give some thought to how you will want to present the information, as this will have some impact on how you carry out the analysis, for example the format in which you analyse data (for example in tables, graphs or maps, etc.)

Want to know more?

Benchmarks

For information about benchmarking for performance improvement see:

• Public Sector Benchmarking Service 
• DTI

Variables

Srantakos, S., Social Research, 3rd edition, Macmillan, Hampshire, 2005

Analysis techniques

There are many different ways of analysing data. This paper has focused on the type of analysis those involved with LAA initiatives are most likely to need to do.

Cabinet Office - A good source of guidance to many of the techniques and approaches, for example modelling, forecasting and scenario development.