Home
News
Publications
Meeting Talks
Reports
Deliverables
Data
Quicklooks
Instruments
Links
Contacts

    

How to interpret this figure...

Cloud fraction comparison statistics

This document describes how to interpret figures summarising the comparison of model and observed cloud fraction for a month or year period at a single site.
More information on this product

Mean cloud fraction

Panel (a) shows mean cloud fraction versus height for the observations and the model. The blue line shows the observed mean cloud fraction, calculated from the radar and lidar data on the vertical grid of the model. Cloud fraction was calculated as the volume of each grid box occupied by cloud, rather than as the area obscured by cloud when viewed from above. Above a certain rainfall rate there is a danger that, due to strong attenuation, the radar may underestimate cloud fraction. Therefore this mean does not include situations with a surface rainrate above 8 mm/hr in the case of 35-GHz radars, and 2 mm/hr in the case of 94-GHz radars.

The magenta line shows the corresponding cloud fraction taken directly from the model. Periods when the rain rate exceeded the thresholds given above have been excluded. The radar can have problems in detecting tenuous ice clouds so for a fairer comparison, the red line shows the mean of the model values after filtering to remove ice clouds estimated to be undetectable by the radar, based on the known radar sensitivity and an estimate of the variation of radar reflectivity on a scale equivalent to the horizontal gridbox size of the model (Hogan and Illingworth 2003, J. Atmos. Sci., 60, 756-767). There is considerable uncertainty in this procedure, as indicated by the red error bars, which show the effect of assuming the radar to be 3 dB more or 3 dB less accurate when performing this filtering. Generally where the red and magenta lines diverge significantly, it is difficult to make a confident comparison with the observations. Fortunately, this tends to occur only above 7-8 km.

(ECMWF only) The ECMWF model treats ice cloud and snow separately, with snow not contributing to cloud fraction or ice water content, and hence not playing a role in radiative transfer. The dashed red line shows filtered model cloud fraction but including the contribution from snow, following Hogan et al. (2001, J. Appl. Meteorol., 40, 513-525). This often produces a better agreement between radar and model in mid-levels.

Frequency of occurrence and amount when present

Panels (c) and (e) show mean cloud fraction split into frequency of occurrence and amount when present. Frequency of occurrence is defined as the fraction of time that cloud fraction on the model grid exceeded 0.05. It is plotted for the observations and for the model, using the various representations from the model as were used in panel (a). Amount when present depicts the corresponding mean cloud fraction but averaged only over those times when it exceeded 0.05. The combination of frequency of occurrence and amount when present can help to diagnose the source of errors in mean cloud fraction; commonly the model frequency of occurrence is accurate but amount when present is not. This indicates that the model carries some cloud about the right amount of the time (indicating that the humidity field is reasonable), but has trouble diagnosing the right amount of cloud when some is present (indicating that the problem lies with the cloud scheme).

The data files contain these two parameters calculated for a whole range of threshold cloud fractions between 0.05 and 0.95.

Probability density functions

Panels (b), (d) and (f) show the probability density functions (PDFs) of cloud fraction in the height ranges 7-12 km, 3-7 km and 0-3 km, respectively (the data files also contain the 12-18-km PDF but this contains significant data only at tropical sites). The blue bars depict the observed PDF while the red bars depict the model PDF after filtering to remove undetectable ice clouds. Where this filtering has removed significant cloud, the magenta bars are visible, indicating the unmodified model. Note that the leftmost bar (cloud fractions from 0 to 0.1) is ten times smaller than the actual value; this is mostly clear-sky events. Points to note in interpreting these panels:

  • Panel (b): 7-12 km - in this range the sensitivity of the radar to tenuous cirrus is often a problem. If the red and magenta bars are different then the blue bars of the observations should be compared to the red bars. If the red and magenta bars are very different then there is likely to be a considerable amount of cloud undetected by the radar, so the process of filtering the model is less certain and one should avoid drawing definitive conclusions about the performance of the model in this height range.
  • Panel (d): 3-7 km - in this range the radar tends not to have a problem in detecting the clouds present and the comparison should be reliable.
  • Panel (f): 0-3 km - this range encompases the boundary layer and errors here are likely to be related to errors in the boundary-layer scheme.

Skill scores

The quantities discussed so far evaluate the climatology of the model, but pay no attention to whether clouds were predicted in the right place at the right time, i.e. the quality of the forecast. Panels (g) and (h) depict two skill scores that present a measure of how well the individual cloud features were predicted, the Equitable threat score and Yule's Q. Both have the property that a perfect forecast scores 1 while a random forecast scores 0. These two skill scores have been chosen because they are known to be relatively insensitive to the frequency of occurrence of the property being assessed (unlike scores such as hit rate or false alarm rate).

The scores are calculated as follows. Firstly a threshold cloud fraction is chosen. A contingency table is defined, such that A is the number of times that cloud fraction exceeded the threshold in both the model and the observations, B is the number of times that cloud fraction exceeded the threshold in the model but not the observations, C is the number of times that cloud fraction exceeded the threshold in the observations but not the model and D is the number of times that cloud fraction exceeded the threshold in neither the model nor the observations. The scores are defined by:

  • Equitable threat score ETS = (A-E)/(A+B+C-E), where E is the number of hits that occurred by chance, given by E=(A+B)*(A+C)/(A+B+C+D). This score is an improvement on the Threat score TS = A/(A+B+C), which does not have the property of a score of 0 corresponding to a random forecast.
  • Yule's Q skill score Q = (A*D-B*C)/(A*D+B*C). It is closely related to the odds ratio but conveniently ranges between -1 and 1.
The plots show the results for thresholds of 0.05, 0.25 and 0.45. If there is low cloud fraction in the period of interest then the scores may be rather noisy. Skill scores for an individual model can be difficult to interpret, but are much more useful in a comparative sense, i.e. to compare the performance of one model against another or the dependence of skill with season.


These pages are maintained by Ewan O'Connor.
Return to Radar Group | Department of Meteorology | University of Reading