Australian Temperature Data Are Garbage

From the Bureau’s hastily published “Fast Facts”:

“This means that each one second temperature value is not an instantaneous measurement of the air temperature but an average of the previous 40 to 80 seconds.”

That is complete nonsense.

At the end of each minute, the following data are recorded:

1. Lowest one second reading of the previous 60 seconds
2. Highest one second reading of the previous 60 seconds
3. Reading at the final second of the minute.

Firstly, 40 seconds is not one minute, the integration period recommended by the WMO in 2014 and by the Bureau’s own officers in 1997.  Anything less than 60 seconds is not compliant.

Secondly, consider this plot, which is from actual 1 minute temperatures recorded at Hervey Bay Airport on 22 February 2017.  (Data purchased by me from the Bureau).

Fig. 1:

Sunrise was at about 5:40 a.m.  Temperatures do not increase until about 6:30 a.m.  Note the strangely low temperature- the daily minimum- which was reported as occurring sometime in the 60 seconds before 06:00:00.  The BOM would have us believe that each of the values in Figure 1, including the low of 23.2C, are “averages” of the previous 40 to 80 seconds.

Next consider what happens in that minute from 5:59 to 6:00, as per the following plot.

Fig. 2:

We don’t know in which seconds the high and low readings for that minute occurred, so I have shown them for each of 59 seconds.  I have shown the 5:59 and 6:00 readings: both were 25.3C.

Consider how the value at 06:00 was obtained:

If by an “average” (however derived) of less than 60 seconds, the methodology is non-compliant.

If by an “average” of the previous 60 seconds, it must include values that contributed to the High of 25.4C and the Low of 23.2C.

If by an “average” of anything greater than 60 seconds, it must include values that contributed to both the Low and High values, and as well, values that contributed to the 5:59 reading- which is the same as the 06:00 reading.

Similar logic applies to the Low and High readings.

It follows that the intermediate instantaneous atmospheric temperatures that contributed to all three reported “average” values must have ranged from much higher than 25.4C to very much lower than 23.2C.

Look at Figure 1 again.  The air temperature at Hervey Bay on 22 February must have spiked down very much lower than the 23.2C plotted.

Really?

In the early morning there is very little near ground turbulence so temperatures do not fluctuate from one minute to the next by very much.  In How Temperature Is Measured in Australia Part 2 I showed that 91% of low temperatures vary from final second temperatures in the same minute by 0.2C or less.  A difference of 2.1C is extraordinary.  Fluctuations greater than that are difficult to believe.

However, in a comment at How Temperature Is Measured In Australia Part 1, Tony Banton, a retired meteorologist, says that the BOM explanation of cooler ground level air mixing upwards is correct.  If we accept that explanation, we must then face the problem of “comparability”.

In 61 seconds, the Hervey Bay AWS has reported temperatures of 25.3, 25.4, 23.2, and finally 25.3 degrees.  The BOM asserts that a liquid-in-glass thermometer will be able to respond as quickly and show similar temperatures- and remember, 23.2C was the morning’s official minimum.

My response: rubbish.  The data for 22 February at Hervey Bay show that no averaging is used at all, and the Low Temperature of 22.3C  23.2C is an instantaneous one second recording from a rogue downwards spike, whatever the cause, whether a natural event or other (e.g. electrical) factor.

Temperatures reported by the BOM are not fit for purpose of accurate reporting of maxima and minima, identifying records, or identifying warming or cooling by comparison with historic liquid-in-glass data.

Advertisements

Watch an AWS Fail

(With thanks to Lance, Phill, and others)

A week ago, a colleague alerted me to strange behaviour at an Automatic Weather System at Borrona Downs in NSW.  This is a brand new weather station, with its first observation on 21 July.

Phill writes in an email:  Do you ever wonder why you get a shiver down your spine?  Pity the poor folks in the NSW far west.  

 From this mornings (20^th  August) NSW observation list: The minimum temperature at Borrona Downs AWS was -62.5C at 9:59pm last night.  Probably some clowns with a bucket of dry ice or liquid nitrogen.  Perhaps Odin’s host crossed the night sky or maybe death just walked on by…  The individual reads don’t show anything lower than -37.5C also at 9:59 so the cold spike was quite sudden.  It went from -62.5C sometime between 21:58:00 and 21:59:00 to -37.5C at exactly 21:59:00 to -4.4C at 22:00:00.

I was too busy and preoccupied until now to follow this up, but I have a few days now.

Borrona Downs Station is in sandhill and claypan country in the far northwest of NSW:

Here is the Climate Data Online minima record (note minima indicated on two days):

The following plots show the deterioration in the performance of the AWS.  Firstly, the comparison with Tibooburra, 110km away, showing a sudden change at 29 July:  Subtracting Borrona Downs data from Tibooburra shows that Borrona Downs Tmin is too high from this date.  The whole (brief) record should be scrapped.

But the devil, as Phill found, is in the detail.  Here is part of the record for the 19^th:  Note the Low Temp at 9.59 pm, and I have indicated the official minimum for the day which would have occurred early that morning.

The Bureau has the minimum at 4.6C, but how was this value obtained?  The erroneous values, (including that of liquid nitrogen), are flagged, then manually removed, and the previous lowest temperature is retrieved from the one minute data for the day.  This also happened on the 26^th:

Things got much worse on August 27^th:

Why could no minimum be found?  Did the BOM realise that none of the data were reliable, and were essentially random errors?  Remember that the AWS records values every second, and the highest, lowest, and final second values for each minute are stored.  My guess is that many of these values were unreliable as well, even though many of the final second half hour values seem reasonable- for example 4.4C at 5.30 am.

This continued on August 28^th   with an all time low of -69.5C:

And the BOM ceased reporting values at 3:30 pm.

This description of events was confirmed by the Bureau’s response to a query:

“Do you know what is causing the very low temperature recordings?

There is a hardware fault within the AWS which is generating spurious values. The Bureau’s technicians are investigating but a site visit will be required.

Why was the August 19 low temperature recording not left blank?

Manual quality checking confirmed that the spiking on 19 August did not occur near the minimum  temperature for that day, as a result, the minimum temperature was recorded.”

This begs the question: is this what happened at Goulburn Airport on 2 July ? The initially reported figure of -10.4C was flagged as suspicious, so the previous low temperature of -10C was then reported, then this was removed , then the initial -10.4C was reinstated.  Perhaps.

-10.4C certainly should not have been flagged as too low for that location, as many other  values below 10C have been observed, including the record -10.9C recorded on 17 August 1994.  However, perhaps it was flagged as suspicious by comparison with the series of values before and after: too large a change in temperature from second to second.  But if so, why didn’t the BOM CEO just say so, instead of getting tangled in a web of conflicting explanations?

The AWS at Borrona Downs has failed.  So has the Bureau of Meteorology.

 

Garbage In, Garbage Out

(By Ken Stewart, assisted by Bill Johnston and Phill Goode; and cross-posted with Jo Nova)

Early ABC Radio news bulletins last Wednesday morning were led by this item, which you can read in full at ABC Online.

More climate scientists needed to avoid expensive mistakes, review urges

Apparently we urgently need 77 climate scientists to predict the future of areas like the Murray-Darling Basin with climate modelling.

Interestingly, Professor McDougall of the Australian Academy of Science points out that one of those “expensive mistakes” was the $2 billion desalination plant built in Queensland as a response to the millennium drought, “which really wasn’t an indication of climate change at all”.   Why didn’t the good professor raise his voice before the money was wasted?

But I digress.

Reliable modelling and projections for the future are surely desirable.

But such modelling must be based on reliable data, and the reliability of temperature data in Australia is demonstrably poor.

Example 1:  As has been widely reported in The Australian, and by Jennifer Marohasy and Jo Nova, cold temperatures at two separate sites (and possibly many others) were altered to appear warmer, then changed back, then deleted.  The BOM gave two conflicting explanations, both of which cast grave doubt on the reliability of “raw” temperature data from an unknown number of stations.

Example 2:  After enquiring why there are frequently two different temperature readings for exactly the same minute at various weather stations, a Bureau spokesperson told me that:

Firstly, we receive AWS data every minute. There are 3 temperature values:
1. Most recent one second measurement
2. Highest one second measurement (for the previous 60 secs)
3. Lowest one second measurement (for the previous 60 secs)

(See here and here.)

In other words, Australian maximum and minimum temperatures are taken from ONE SECOND readings from Automatic Weather Stations.  Spikes due to localised gusts of hot air, or instrument error, become the maximum for the day.  (This rarely has a large effect on minima, as night time temperatures are fairly smooth, whereas during the day temperature bounces rapidly up and down.  This is shown in this plot of temperatures at Thangool Airport in Queensland on Australia Day this year.)

And this is for the same day between 3.00pm and 4.00pm.

As you can see the temperature spikes up and down in the heat of the day by up to one degree between one minute and the next.  But these are the temperatures at the final second of each minute: during the intervening 59 seconds the temperature is spiking up and down as well, which we know because occasionally the highest or lowest temperature for the day occurs in the same minute as a final second recording on the BOM database (usually on the hour or half hour).  This can be up or down by two or three degrees in less than 60 seconds.

This is in contrast to the rest of the world.  The WMO recommends 1 minute (60 second) averages of temperature to be recorded to combat this very problem of noisy data, and this is followed in the UK.  In the USA 5 minute (300 second) averages are calculated.

From THE WEATHER OBSERVER’S HANDBOOK by Stephen Burt (Cambridge University Press, 2012):

Even without software or human interference as in Example 1, this means Australian temperature data, in particular maxima, are not reliable.

Example 3:  Historically, temperatures were observed from Liquid In Glass (LIG) thermometers.  From the 1990s, Automatic Weather Stations (AWS) were installed using Platinum Resistance Thermometers (PRT) and are now the source for daily data.  AWS thermometers are very precise, but as I showed in Example 2, their data is used idiosyncratically to record 1 second spikes, frequently resulting in higher maxima and less often slightly lower minima than a 1 or 5 minute average.

One would think that with such a major change in technology there would be comparative studies reported in the BOM’s meteorological journal or other “peer reviewed” literature.  Apparently not.

Dr Bill Johnston has investigated this and says:

Parallel data were collected all over Australia for over a decade, some until last year when thermometers were removed, at manned sites, mainly major airports (Ceduna, Sydney, Hobart, Perth, Darwin, Alice Springs, Albany, Norfolk Island, Wagga to name a few) and also met-offices such as Cobar and Giles. However, comparisons between screens were done at one site only (Broadmeadows, Melbourne, which is not even an official weather station) using PRT only and reported as a “preliminary report”, which is available (https://www.wmo.int/pages/prog/www/IMOP/WebPortal-AWS/Tests/ITR649.pdf) however, after AWS became primary instruments, as I’ve reported before, the Bureau had an internal policy that parallel liquid-in-glass thermometer data were not databased. Furthermore, they had another policy that paper-data was destroyed after 2-years. So there is nothing that is easily available…. there is also no multi-site replicated study involving screen types and thermometers vs. PRT probes ….

Deliberate destruction of data is scandalous; the only way now to compare Automatic Weather Stations (AWS) and Liquid in Glass, is to hunt for sites where there is overlap between two stations; where the AWS is given a new number. This is possible BUT the problem is that the change-over is invariably confounded with either a site move or the change to a small screen.

Therefore we suspect that the introduction and reliance on AWS has led to artificially higher maxima (and thus record temperatures) than in the past, but we have no way of knowing for sure or how much.

So we now have (1) temperatures that are altered before they even become ‘raw’ data; (2) use of one second spikes for recording daily maximum and minimum temperatures, very probably resulting in artificially high maxima and slightly lower minima; and (3) no way of telling how the resulting data compare with those from historical liquid-in-glass thermometers.

How can the CSIRO hope to produce reliable climate modelling with any number of climate scientists when the BOM cannot produce reliable temperature data?  Garbage in, garbage out.

How Temperature is “Measured” in Australia: Part 2

By Ken Stewart, ably assisted by Chris Gillham, Phillip Goode, Ian Hill, Lance Pidgeon, Bill Johnston, Geoff Sherrington, Bob Fernley-Jones, and Anthony Cox.

In the previous post of this series I explained how the Bureau of Meteorology presents summaries of weather observations at 526 weather stations around Australia, and questioned whether instrument error or sudden puffs of wind could cause very large temperature fluctuations in less than 60 seconds observed at a number of sites.

The maximum or minimum temperature you hear on the weather report or see at Climate Data Online is not the hottest or coldest hour, or even minute, but the highest or lowest ONE SECOND VALUE for the whole day.  There is no error checking or averaging.

A Bureau officer explains:

Firstly, we receive AWS data every minute. There are 3 temperature values:
1. Most recent one second measurement
2. Highest one second measurement (for the previous 60 secs)
3. Lowest one second measurement (for the previous 60 secs)

Relating this to the 30 minute observations page: For an observation taken at 0600, the values are for the one minute 0559-0600.

Automatic Weather Station instruments were introduced from the late 1980s, with the AWS becoming the primary temperature instrument at a large number of sites from November 1 1996.  They are now universal.

An AWS temperature probe collects temperature data every second; there are 60 datapoints per minute.  The values given each half hour (and occasionally at times in between) at each station’s Latest Weather Observations page are samples: spot temperatures for the last second of the last minute of that half hour, and the Low Temp or High Temp values on the District Summary page are the lowest and highest one second readings within that minute of reporting.  The remaining seconds of data are filtered out.  There is no averaging to find the mean over say one minute or ten minutes.  There is NO error checking to flag rogue values.  The maximum temperatures are dutifully reported in the media, especially if some record has been broken.  Quality Control does not occur for two or three months at least, which then just quietly deletes spurious values, long after record temperatures have been spruiked in the media.

In How Temperature is “Measured” in Australia: Part 1 I demonstrated how this method has resulted in large differences recorded in the exact same minutes at a number of stations.

What explanation is there for these differences? 

The Bureau will insist they are due to natural weather conditions.  Some rapid temperature changes are indeed due to weather phenomena.  Here are some examples.

In semi-desert areas of far western Queensland, such as in this example from Urandangi, temperatures rise very rapidly in the early morning.

Fig. 1:  Natural rapid temperature increase

For 24 minutes the temperature was increasing at an average of more than 0.2C per minute.  That is the fastest I’ve seen, and entirely natural- yet at Hervey Bay on 22 February the temperature rose more than two degrees in less than a minute, before 6 a.m., many times faster than it did later in the morning.

Similarly, on Wednesday 8 March, a cold change with strong wind and rain came through Rockhampton.  Luckily the Bureau recorded temperatures at 4:48 and 4:49 p.m., and in that minute there was a drop of 1.2C.

Fig. 2:  Natural rapid temperature decrease

That was also entirely natural, and associated with a weather event.

For the next plots, which show questionable readings, I have supplemented BOM data with data from an educational site run by the UK Met Office, WOW (Weather Observations Worldwide).  The Met gets data from the BOM at about 10 minutes before the hour, so we have an additional source which increases the sample frequency.  The examples selected are all well-known locations in Queensland, frequently mentioned on ABC TV weather.  They have been selected purely because they are examples of large one minute changes.

This plot is from Thangool Airport near Biloela, southwest of Rockhampton, on Friday 10 March.  The weather was fine, sunny, and hot, with no storms or unusual weather events.

Fig. 3:  Temperature spike and rapid fall at Thangool

This one is for Coolangatta International Airport on the Gold Coast on 20^th February.

Fig. 4:  Temperature spike and rapid fall at Coolangatta

And Maryborough Airport on 15^th February:

Fig. 5:  Temperature spike and rapid fall at Maryborough (Qld)

Figure 5(b):  The weirdest spike and fall:  Coen Airport 21 March 

Thanks to commenter MikeR for finding that one.

All of these were in fine sunny conditions in the hottest part of the day.  It is difficult to imagine a natural meteorological event that would cause such rapid fluctuations- in particular rapid falls- as in the above examples.  It is possible they were caused by some other event such as jet blast or prop wash blowing hotter air over the probe during aircraft movement, quickly replaced by air at the ambient surrounding temperature.  It is either that or random instrument error.  Either way, the result is the same: rogue outliers are being captured as maxima and minima.

How often does this happen?

Over one week I collected 200 instances where the High Temps and Low Temps could be directly checked as they occurred in the same minute as the 30 minute observation.

The results are astounding.  The differences occurring in readings in the same minute are scattered across the range of temperatures.  Most High Temp discrepancies are of 0.1 or 0.2 degrees, but there is a significant number (39% of the sample) with 0.3C to 0.5C decreases in less than one minute, and five much larger.

Fig. 6:  Temperature change within one minute from maximum

Notice that 95% of the differences were from 0.1C to 0.5C, which suggests that one minute ranges of up to 0.5C are common and expected, while values above this are true outliers.  The Bureau claims (see below) that in 90% of cases AWS probes have a tolerance of +/-0.2C, whereas the 2011 Review Panel mentioned the “the present +/- 0.5 °C”.  Is the tolerance really +/-0.5C?

Fig. 7:  Temperature change within one minute from minimum

There was one instance where there was no difference.  The vast majority have a -0.1C difference, which is within the instruments’ tolerance.

This next plot shows the differences (temperature falls in one minute from the second with the highest reading to that of the final second) ordered from greatest to least.

Fig. 8:  Ordered count of temperature falls

The few outliers are obvious.  More than half the differences are of 0.1C or 0.2C.

One minute temperature rises:

Fig. 9:  Ordered count of temperature rises

Note the outlier at -2.1C: that was Hervey Bay Airport.  Also note only one example with no difference, and the majority at -0.1C.

Is there any pattern to them? 

The minimum temperature usually occurs around sunrise, although in summer this varies, but very rarely when the sun is high in the sky.  Therefore rapid temperature rise at this time will be relatively small, as the analysis shows: 80% of the differences between the Low Temps and corresponding final second observations were zero or one tenth of a degree, and 91% two tenths of a degree or less.  As the instrument tolerance of AWS sensors is supposed to be +/- 0.2C, the vast majority of Low Temps are within this range.  Therefore, the Low Temps are not significantly different from the Latest Observation figures.  Yet as it is the Lowest Temperature that is being recorded, all but one example have the Low Temp, and therefore daily minimum, cooler than the final second observation.  9% are outside the +/-0.2C range and show real discrepancy, i.e. very rapid temperature rise within one minute, that is worth investigating.  Remember, the fastest morning rise I’ve found averaged about 0.2C per minute.

The High Temps have 56% of discrepancies within the +/-0.2C tolerance range.  Day time temperatures are much more subject to rapid rise and fall of temperatures.  The 44% of discrepancies of 0.3C or more are worth investigation.  Many are likely due to small localized air temperature changes, the AWS probes being very sensitive to this, but the rapid decreases shown in the examples above, as well as the rapid rises in the Low Temp examples, mean that random noise is likely to be a factor as well.

Have they affected climate analysis? 

Comparison of values at identical times has shown that out of 200 cases, all but one had higher or lower temperatures at some previous second than at the last second of that minute, with a significant number of High Temp observations (39% of the sample) with 0.3C to 0.5C decreases in less than one minute, and five much larger.  There is a very high probability that similar differences occur at every station in every state, every day.

In more than half of the sample of High Temps, and over 90% of the Low Temps, the discrepancy was within the stated instrumental tolerance range, and therefore the values are not significantly different, but the higher or lower reading becomes the maximum or minimum, with no tolerance range publicised.

This would of course be an advantage if greater extremes were being looked for.

Nearly 10 percent of minimum temperatures were followed by a rise of more than 0.2C, and 44 percent of maxima were followed by a fall of more than 0.2C.  While many of these may have entirely natural causes, none of the very large discrepancies examined had an identifiable meteorological cause.   It is questionable whether mercury-in-glass or alcohol-in-glass thermometers used in the past would have responded as rapidly as this.  This must make claims for record temperatures questionable at best.

If you think that the +/- 0.2C tolerance makes no difference in the big picture, as positives will balance negatives and errors will resolve to a net of zero, think again.  Maximum temperature is the High Temp value for the day, and 44% of the discrepancies were more than +0.2C.  If random instrument error is the problem causing the apparent temperature spikes, (and downwards spikes in the hot part of the day are not reported unless they show up in the final second of the 30 minute reporting period), only the highest upwards spike, with or without positive error, is reported.  Negative error can never balance any positive error.

Further, these very precise but questionable values then become part of the climate monitoring system, either directly if they are for ACORN stations, or indirectly if they are used to homogenise “neighbouring” ACORN stations. They also contribute to temperature maps, showing for example how hot New South Wales was in summer.

Again, temperature datasets in the ACORN network are developed from historic, not very precise, but (we hope) fairly accurate data from slow response mercury-in-glass or alcohol-in-glass thermometers observed by humans, merged with very precise but possibly unreliable, rapid response, one second data from Automatic Weather Systems.  The extra precision means that temperatures measured by AWS probes are likely to be some tenths of a degree higher or lower than LIG thermometers in similar conditions, and the higher proportion of High Temp differences shown above, relative to Low Temp differences, will lead to higher maxima and means in the AWS era.  Let’s consider maxima trends:

Fig. 10:  Australian maxima 1910-2016

There are no error bars in any BOM graph.  Maxima across Australia as a whole have increased by about 0.9 C per 100 years according to the Bureau, based on analysis of ACORN data.  Even if across the whole network of 526 automatic stations the instrument error is limited to +/- 0.2C, that is 22.2% of the claimed temperature trend.  In the past, indeed as recently as 2011 (see below), instrument error was as high as +/-0.5C, or about half of the 107 year temperature increase.  No wonder the Bureau refuses to show error bands in its climate analyses.

There have been NO comparison studies published of AWS probes and LIG thermometers side by side.  Can temperatures recorded in the past from liquid-in-glass thermometers really be compared with AWS one second data?  The following quotes are from 2011, when an Independent Review Panel gave its assessment of ACORN before its introduction.

Report of the Independent Peer Review Panel p8 (2011)

Recommendations: The Review Panel recommends that the Bureau of Meteorology should implement the following actions:

A1 Reduce the formal inspection tolerance on ACORN-SAT temperature sensors significantly below the present ±0.5 °C. This future tolerance range should be an achievable value determined by the Bureau’s Observation Program, and should be no greater than the ±0.2 °C encouraged by the World Meteorological Organization.

A2 Analyse and document the likely influence if any of the historical ±0.5 °C inspection tolerance in temperature sensors, on the uncertainty range in both individual station and national multidecadal temperature trends calculated from the ACORN-SAT temperature series.

And the BoM Response: (2012)

… … …   An analysis of the results of existing instrument tolerance checks was also carried out. This found that tolerance checks, which are carried out six-monthly at most ACORN-SAT stations, were within 0.2 °C in 90% of cases for automatic temperature probes, 99% of cases for mercury maximum thermometers and 96% of cases for alcohol minimum thermometers.

These results give us a high level of confidence that measurement errors of sufficient size to have a material effect on data over a period of months or longer are rare.

This confirms LIG thermometers have more reliable accuracy than automatic probes, and that 10% of AWS probes are not sufficiently accurate, with higher error rates.  That is, at more than 50 sites.  If they are in remote areas, their inaccuracy will have an additional large effect on the climate signal.   It is to be hoped that Alice Springs, which contributes 7-10% of the national climate signal, is not one of them.

Conclusion

It is very likely that the 199 one minute differences found in a sample of 200 high and low temperature reports are also occurring every day at every weather station across Australia.  It is very likely that nearly half of the High Temp cases will differ by more than 0.2 degree Celsius.

Maxima and minima reported by modern temperature probes are likely to be some tenths of a degree higher or lower than those reported historically using Liquid-In-Glass thermometers.

Daily maximum and minimum temperatures reported at Climate Data Online are just noise, and cannot be used to determine record high or low temperatures.

These problems are affecting climate analyses directly if they are at ACORN sites, or indirectly, if they are used to homogenise ACORN sites, and may distort regional temperature maps.

Instrument error may account for between 22% and 55% of the national trend for maxima.

A Wish List of Recommendations (never likely to be adopted):

That the more than 50 sites at which AWS probes are not accurate to +/- 0.2 degree Celsius be identified and replaced with accurate probes as a matter of urgency.

That the Bureau show error bars on all of its products, in particular temperature maps and time series, as well as calculations of temperature trends.

That the Bureau of Meteorology recode its existing three criteria filter, to zero-out spurious spikes and preferably send them as fault flags into a separate file in order to improve Quality Control.

That the Bureau replace its one second spot maxima and minima  reports with a method similar to wind speed reports: the average over 10 minutes.  That would be a much more realistic measure of temperature.

How Temperature Is “Measured” in Australia: Part 1

By Ken Stewart, ably assisted by Chris Gillham, Phillip Goode, Ian Hill, Lance Pidgeon, Bill Johnston, Geoff Sherrington, Bob Fernley-Jones, and Anthony Cox.

The Bureau of Meteorology maintains the Southern Oscillation Index (SOI), one of the most useful climate and weather records in the world.  In About SOI,  the Bureau says:

 Daily or weekly values of the SOI do not convey much in the way of useful information about the current state of the climate, and accordingly the Bureau of Meteorology does not issue them. Daily values in particular can fluctuate markedly because of daily weather patterns, and should not be used for climate purposes.

It is a pity that the BOM doesn’t follow this approach with temperature, and in fact goes to the opposite extreme.

Record temperatures, maximum and minimum temperatures, and monthly, seasonal, and annual analyses are based not on daily values but on ONE SECOND VALUES.

The Bureau reports daily maximum and minimum temperatures at Climate Data Online,   but also gives a daily summary for each site in more detail on the State summary observations page , and a continuous 72 hour record of 30 minute observations (examples below), issued every 30 minutes, with the page automatically refreshed every 10 minutes, also handily graphed .  These last two pages have the previous 72 hours of readings, after which they disappear for good.  However, the State summary page, also refreshed every 10 minutes, is for the current calendar day only.

This screenshot shows part of the Queensland observations page for February 26, showing the stations in the North Tropical Coast and Tablelands district.

Fig. 1:  District summary page

Note especially the High Temp of 30.5C at 01:26pm.  Clicking on the station name at the left takes us to the Latest Weather Observations for Mareeba page:

Fig. 2:  Latest Observations for Mareeba

Notice that temperature recordings are shown every 30 minutes, on the hour and half hour.

In Figure 1 I have circled the Low Temp and High Temp for Mareeba.  Except in unusual circumstances, High Temp and Low Temp values become the maximum and minimum temperatures and are listed on the Climate Data Online page, and for stations that are part of the ACORN network, become part of the official climate record.  It is most important that these High Temp and Low Temp values, the highest and lowest recorded temperatures of each day, should be accurate and trustworthy.

But frequently they are higher or lower than the half hourly observations, as in the Mareeba example (0.6C higher), and I wanted to know why.  In this post I show some recent examples, with the explanation from the Bureau.

Perhaps the difference between the Latest Weather Observations and maximum temperature reported at Climate Data Online is due to brief spikes in temperature in between the reported temperatures of the latest observations, such as in this example from Amberley RAAF on February 12.

Fig. 3:  Amberley RAAF temperatures, 12 February 2017

A probable cause would be that the Automatic Weather Station probe is extremely sensitive to sudden changes in temperature as breezes blow warmer or cooler air around or a cloud passes over the sun.

However, this may not be the whole story.

Occasionally the report time for the High Temp or Low Temp is exactly on the hour or half hour, and therefore can be directly compared with the temperature shown for that time at the station’s page.

These progressive Low and/or High Temps on the half hour or hour occur and can be observed throughout the day at various times, as well as at the end of the reporting period.

For example, here is a mid-afternoon screenshot of the Queensland- Wide Bay and Burnett district summary for Wednesday 15^th February.  I have highlighted the High Temp value for Maryborough at 1:00pm.

Fig. 4:  District summary at 2:00pm for Maryborough 15 February 2017

In the Latest Observations for Maryborough, I have highlighted the 1:00pm reading.

Fig. 5: Latest Observations at Maryborough at 01:00pm on 15 February

The difference is +1.5 degrees.  Here I have graphed the results.

Fig. 6:  Maryborough 15 February

That’s a 1.5 degree difference at the exact same minute.

Here is a screenshot of Latest Observations values at Hervey Bay Airport on Wednesday 22 February.  Low Temp for the morning of 23.2C was reached at 6.00 a.m.

Fig. 7:  Hervey Bay, 06:00am  22 February 2017

Note that at 6.00am, just after sunrise, the Latest Observations page shows that the temperature was 25.3 degrees.  The daily Low Temp was reported as 23.2 degrees at 6.00am – 2.1 degrees cooler.  This graph will show the discrepancy more plainly.

Fig. 8:  Hervey Bay temperatures 22 February

What possible influence would cause a dawn temperature to drop 2.1 degrees?

I sent a query to the Bureau about Hervey Bay, and the explanation from the Bureau’s officer was enlightening:

Firstly, we receive AWS data every minute. There are 3 temperature values:
1. Most recent one second measurement
2. Highest one second measurement (for the previous 60 secs)
3. Lowest one second measurement (for the previous 60 secs)

Relating this to the 30 minute observations page: For an observation taken at 0600, the values are for the one minute 0559-0600.

I’ve looked at the data for Hervey Bay at 0600 on the 22nd February.
25.3, 25.4, 23.2 .

The temperature reported each half hour on the station Latest Observations page is the instantaneous temperature at that exact second, in this case 06:00:00, and the High Temp or Low Temp for the day is the highest or lowest one second temperature out of every minute for the whole day so far.  There is no filtering or averaging.

The explanation for the large discrepancy was that “Sometimes the initial heating from the sun causes cooler air closer to the ground to mix up to the temperature probe (1.2m above ground).”

However, in Figure 7 above it can be seen that the wind was south east at 17 km/hr, gusting to 26 km/hr, and had been like that all night, over flat ground at the airport, so an unmixed cooler surface layer mixing up to the probe seems very unlikely.

You will also note that the temperatures in the final second of every half hour period from 12.30 to 6.30 ranged from 25C to 25.5C, yet in some second in the final minute before 6.00 a.m. it was at 23.2C.  I have shown these values in the graph below.

Fig. 9:  Hervey Bay 05:59 to 06:00am

The orange row shows the highest temperature for this last minute at 25.4C at some unknown second, the blue row the lowest temperature for this minute (and for the morning) at 23.2C at some unknown second, and the spot temperature of 25.3C at exactly 06:00:00am.  The black lines show the upper and lower values of half hourly readings between 12:30 and 06:30: the high temp and 06:00am readings are within this range.

23.2C looks a lot like instrument error, and not subject to any filtering.

Further, there are only two possibilities:  either from a low of 23.2C, the temperature rose 2.2 degrees to 25.4C, then down to 25.3C; or else from a high of 25.4C it fell 2.2 degrees to 23.2C, then rose 2.1 degrees to 25.3C, all in the 60 seconds or less prior to 06:00:00 a.m.

How often does random instrument error affect the High and Low Temps reported at the other 526 stations?  Like Thargomindah, where on February 12 the High Temp was 2.3 degrees to 2.5 degrees higher than the temperatures 15 minutes before and after?

Fig. 10:  Thargomindah temperatures 12 February 2017

Or was this due to a sudden rise and fall caused by a puff of wind, even a whirl-wind?

Who knows?  The Bureau certainly doesn’t.

 

In Part 2, I will look at patterns arising from analysis of 200 High and Low Temps occurring in the same minute as the half hourly values, and implications this has for our climate record.

Putting Temperature in Context: Pt 2

To show how handy my Excel worksheet is, here’s one I did in the last 15 minutes.

Apparently Sydney has had its warmest December minimum on record at 27.1 C.  The record before that was Christmas Day, 1868 at 26.3C.

The following seven plots show this in context.

Fig. 1:  The annual range in Sydney’s minima:

Extremes in minima can occur any time between October and March.

Fig. 2:  The first 2 weeks of December

Plainly, a new record was set this morning, but apart from Day 340 the other days are within the normal range.

Fig. 3:  7 day mean of Tmin in this period

Extreme, but a number of previous years had warmer averages.

Fig. 4:  Consecutive days above 20C Tmin.

But there have been longer periods of warm minima in the past.

Now let’s look at the same metric, but for all of December.

Fig. 5:  All Decembers (including leap years).

A record for December, with 1868 in second place.

Fig. 6:  7 day mean of Tmin for Decembers

Seven day periods of warm nights are not new.  The horizontal black line shows the average to this morning (20.6C) is matched or exceeded by a dozen other Decembers.  (Of course this December isn’t half way through yet.)  Also note what appears to be a step change about 1970.

Fig. 7:  Consecutive days above 20C Tmin in December.

I doubt if 15 December will be as warm as today, but could still be over 20C.

This is weather, not global warming.

 

Putting Daily Temperature in Context

In this post I demonstrate a simple way of comparing current temperatures for a particular location with those previously recorded.  In this way it is possible to show the climatic context.

Using data from Climate Data Online, I plot maximum temperature for each day of the year, and then for a particular short period: in this case the last week of November and the first week of December, which coincides with the recent very warm spell here in Queensland.  To account for leap and ordinary years this period is 15 days.  In ordinary years 24^th November is Day 328 and 7^th December is Day 341, while in leap years this same calendar period is Day 329 to 342.  I also calculate the running 7 day mean TMax for this period, and the number of consecutive days above 35C.

To put the recent heatwave in context, I have chosen six locations from Central and Southern Queensland which regularly feature on ABC-TV weather: Birdsville, Charleville, Roma, Longreach, Ipswich (Amberley RAAF), and Rockhampton.

Birdsville:

Fig. 1

The Police Station data are from 1954 to 2005, and the Airport from 2000.  This shows the range of temperatures throughout the year.  The red arrow indicates the current period.   The next plot shows data only for the period in question.

Fig. 2:  24 November- 7 December: Airport data

Note there were three days where the temperature this year was the highest for those days since 2000, but didn’t exceed the highest in this time period, which was in November.  The other days were well within the historic range.

For interest, let’s now see how this year compares with the Police Station record.  (The average difference in TMax during the overlap period was 0.0 to 0.3C.)

Fig. 3:  24 November- 7 December: Police Station data

In a similar range.

Fig. 4

This heatwave was the third hottest since 2000 and fifth overall.

Fig. 5

Five previous periods had more consecutive days above 35C.  2006 had 22.

Charleville:

Fig. 6: Charleville Aero since 1942

Temperatures in this period reached the extremes of the range on three days.

(Although the Post Office record begins in 1889, there are too many errors in the overlap period so the two records can’t be compared.)

Fig. 7:

A new record for early December was set, but note this was the same temperature as 29^th November 2006.

Fig. 8:

Definitely the hottest for this period since 1942.

Fig. 9:

Note this was not the longest warm spell by a mile: there were many previous periods with up to 26 consecutive days above 35C.

Roma

Fig. 10:

Although there is not one day of overlap so the two records can’t be compared, you can see that Airport (from 1992) and Post Office records are similar.

Fig. 11:

A new record for this time of year was set: 44.4C, and six days in a row above 40C.  Pretty hot….

Fig. 12:

…but there were longer hot periods in the past (since 1992).

Longreach

Fig. 13:  Longreach Aero since 1966.

Fig. 14:

Hot, but no record.

Although there is good overlap with the Post Office, temperatures for this period differ too much: from -1 to +0.7C.

Fig. 15:

Fifth hottest period since 1966.

Fig. 16:

And in the past there have been up to 47 consecutive days above 35C at this time of year.

Ipswich (Amberley RAAF):

Fig. 17:

Fig. 18:

Not unusually hot for this time of year.

Fig. 19:

Ninth hottest since 1941.

Fig. 20:

Hotter for longer in the past.

Rockhampton:

Fig. 21:

Fig. 22:

Very hot, but no records.  (The heat lasted another two days, with 36.6 and 37.3 on 8^th and 9^th.)

Fig. 23:

Fourth hottest 7 day average on record (since 1939).

Fig. 24:

Again, a number of hot days, but there were as many and more in the past.

To conclude: the recent heatwave was very hot certainly, and was extreme in southern inland Queensland.  While Charleville had the highest seven day mean temperature on record, NO location had as many consecutive hot days (above 35C) as in the past.

This is a handy method for showing daily data in context.  It can used for any period of the year, can be tuned to suit (I chose TMax above 35C, but temperatures below a set figure could be found), and can be used for any daily data.

If you would like a comparison done for a location that interests you, let me know in comments including time period and parameters of interest (e.g. Sydney, first 2 weeks of December, TMax above 30C say, or Wangaratta, September, daily rainfall over 10mm say.)

Land and Sea Temperature: South West Australia Part II: TMin

This is a quick follow up to my last post, as an update:  I’ve been reminded to show Tmin as well.   My apologies.

In this post I examine minimum temperature for Winter in South-Western Australia, and Sea Surface Temperature data for the South West Region, all straight from the Bureau of Meteorology’s Climate Change time series page .

All temperature data are in degrees Celsius anomalies from the 1961-90 average.

Fig. 1:   Southwestern Australia Winter TMin Anomalies & SST

Note that TMin roughly matches SSTs, but there are differences from TMax.  CuSums will show this:

Fig. 2:  CuSums of Winter TMin and SST compared:

Note that TMin has completely different change points, marked in red.  The major different ones are at 1949, 1956, 1964, 1990, 2000, and 2010.  There is a barely discernible point at 1976 (not 1975), so the next plots will use 1976 to show trends since then.

Fig. 3:  Trends in TMin:

Cooling since 1976 at -0.36C/100 years.

Detrending the data allows us to see where any of the winters “bucks the trend”.  In the following plots, the line at zero represents the trend as shown above.

Fig. 4:  TMin Detrended:

2016 winter TMin is 0.5C below trend, and 0.38C below average, however winter this year in southwest WA was not as cold as 1986, 1990, 2001, 2006, 2008, or 2010- according to Acorn of course.

The action is with TMax.

Land and Sea Temperature: South West Australia

This year, the south-west of Western Australia has recorded some unexpectedly low temperatures.  Has this been due to rainfall, cloud, winds, or the cooler than normal Leeuwin Current and Sea Surface Temperatures in the South West Region?

In this post I examine maximum temperature and rainfall data for Winter in South-Western Australia, and Sea Surface Temperature data for the South West Region, all straight from the Bureau of Meteorology’s Climate Change time series page .

All temperature data are in degrees Celsius anomalies from the 1961-90 average.

Figure 1 is a map showing the various Sea Surface Temperature monitoring regions around Australia.

Fig. 1

The Southwest Region is just to the west and southwest of the Southwest climate region, and winter south westerlies impact this part of the continent first.  2016’s winter has seen maxima drop sharply.  In fact, it was the coldest winter since 1993:

Fig. 2:  Southwestern Australia Winter TMax Anomalies

There is a relationship between rainfall and Tmax- as rain goes up, Tmax goes down, so here south west rainfall is inverted and scaled down by 100:

Fig. 3:  TMax and Rain:

The next plot shows TMax and the South West Region’s Sea Surface Temperature anomalies (SST):

Fig. 4:  TMax & SST:

Again, related: both have strong warming from the 1970s.  Next I check for whether there was a real change in direction in the 1970s, and if so, when.  To do this I use CuSums.

Fig. 5:  CuSums of Winter TMax and SST compared:

Both have a distinct change point: 1975, with SST warming since, but TMax appears to have a step up, with another change point at 1993 with strong warming since.  Rainfall however shows a different picture:

Fig. 6:  CuSums of Winter Rainfall

Note the major change at 1968 (a step down: see Figure 3), another at 1975 with increasing rain to the next change point at 2000, after which rain rapidly decreases.

I now plot TMax against rainfall and SST to see which has the greater influence.  First, Rain:

Fig. 7:  TMax vs Rain:

100mm more rain is associated with about 0.5C lower TMax, but R-squared is only 0.22.

Fig. 8:  TMax vs SST:

A one degree increase in SST is associated with more than 1.1C increase in TMax, and R-squared is above 0.51- a much closer fit, but still little better than fifty-fifty.

TMax is affected by rain, but more by SSTs.

I now look at data since the major change points in the 1975 winter.  The next three figures show trends in SST, Rain, and TMax.

Fig. 9:  Trends in SST:

Warming since 1975 of +1.48C/ 100 years.

Fig. 10:  Trends in Rainfall:

Decreasing since 1975 at 89mm per 100 years (and much more from 2000).

Fig. 11:  Trends in TMax:

Warming since 1975 at +2.14C per 100 years.

Detrending the data allows us to see where any of the winters “bucks the trend”.  In the following plots, the line at zero represents the trend as shown above.

Fig. 12:  SST Detrended:

Fig. 13:  Rainfall Detrended:

Fig. 14:  TMax Detrended:

Note that SST in 2016 is just below trend, but still above the 1961-90 average.  Rainfall is only slightly above trend, and still below average.  However TMax is well below trend, and well below average, showing the greatest 12 month drop in temperatures of any winter since 1975.

My conclusions (and you are welcome to comment, dispute, and suggest your own):

• Maximum temperatures in winter in Southwestern Australia are affected by rainfall, but to a much larger extent by Sea Surface Temperature of the South West Region.

• The large decrease in winter temperature this year cannot be explained by rainfall or sea surface temperature.  Cloudiness may be a factor, but no 2016 data are publicly available.  Stronger winds blowing from further south may be responsible.

Poles Apart

Satellite data from UAH (University of Alabama- Huntsville) are estimates of temperature in the Lower Troposphere, and thus a good indicator of whether greenhouse warming is occurring.  My next post about the length of The Pause in various regions will be ready in a few days’ time.  Meanwhile, I’ve been looking at the data in a different way.

In this post I will be examining how and when temperatures have changed in discrete regions of the globe, including over land and over oceans.  There are no startling revelations, but a different approach reinforces the need to understand climate variability in different regions.  The important regions of course are the Tropics and the Poles, and fortunately UAH data is available separately for just these three regions.

Firstly, Figure 1 shows the regions for which UAH has atmospheric data.

Fig. 1:  UAH Data Regions

The Northern and Southern Extra-Tropics include the Polar regions, so there are three discrete regions which do not overlap: Tropics, North Polar, and South Polar.  It would be very helpful if Dr Spencer provided data for the Extra Tropical regions excluding the Polar Regions.

For this analysis I use CuSum, which is a simple test of data useful for detecting linearity or otherwise, and identifying sudden changes in trend, or step changes.  It can be used for any data at all- bank balance, car accidents, rainfall, GDP, or temperature.  It is simple to use:  find the mean of the entire data, calculate differences for every data point from this mean, then calculate the running sum (Cumulative Sum) of the differences.  If done correctly, the final figure will be zero.  Plot the CuSum usually by time and identify points of any sudden change in direction.  A generally straight or smoothly curving line indicates linearity, but points of sudden change mean a change in trend or a step change.  (Further, data series with identical start and end points, exactly the same number of data points, and anomalies from the same period- such as UAH- should produce directly comparable CuSums.)  These points, and ranges between them, are then checked in the original data. The usefulness of CuSums will become obvious as we go, especially as they are compared.

The next figures show CuSum plots for various regions.

Fig. 2:  UAH CuSums for all regions

Points to note:

The brown line at the top is the South Polar region.  The line wobbles about zero, indicating little relative change in temperature from the mean.  Contrast this with the North Polar region (the blue line at the bottom.)  The Polar regions are conspicuously different from the other regions and from each other.

The spaghetti lines clustered in the middle are CuSums for (in order from top to bottom): Southern Extra-Tropics; Southern Hemisphere; Tropics; Globe; Northern Hemisphere; Northern Extra-Tropics.

The red arrows point to wobbles coinciding with major ENSO events.  These changes in direction indicate trend changes or step changes in the original data.  There are other changepoints, notably 2002-2003.

The vertical red line joins changepoints in all the CuSums in mid-1991 following the eruption of Mt Pinatubo.

Fig. 3: UAH CuSums for the Tropics, South Polar, and North Polar regions

Note there is little similarity between CuSums for the only regions with discrete data, and you have to look carefully to see North Polar CuSums changing some months after Tropics, but not always.

The next plots show the differing responses of Land and Ocean areas.

Fig. 4:   UAH CuSums for the Globe, Land and Ocean

Note that Land areas have greater relative temperature changes than the Oceans, and that the Global mean closely mirrors the Ocean CuSums (as the Globe is mostly Ocean).  The major turning point is in 1997-98.

Fig. 5:  UAH CuSums for the Tropics, Land and Ocean

Note once again the mean CuSums closely follow that of the Ocean as 20 degrees North to 20 degrees South is mostly water.  The changepoints are very distinct.

Fig. 6:  UAH CuSums for the North Polar region, Land and Ocean

Note that all CuSums are close, but after 1982 Ocean CuSum changes relatively more than Land- the blue line has switched to below the mean.  The main changepoints are 1991, 1993-94, 2002, 2009, and 2015.

Fig. 7:  UAH CuSums for the South Polar region, Land and Ocean

Now that is interesting.  Note all three CuSums have similar changepoints, but Land varies more than Ocean and after 1992 Land is largely negative, Ocean is largely positive.  The Land CuSum range is about half of the North Polar equivalent.

Remember CuSums in Figure 4 showed Land temperatures must vary more than Ocean (though not in the North Polar region).  The next figures show plots of UAH original data (not CuSums).

Fig. 8:  UAH original data for the Globe, Land and Ocean

I find a visual representation demonstrates greater relative variation in Land temperatures well.

Fig. 9:  UAH original data for the Tropics, Land and Ocean

Note much greater fluctuation with ENSO, and Land varying a little more that Ocean.

Fig. 10:  UAH original data for the North Polar region, Land and Ocean

Note the much greater variation, but Land is more often than not masked by Ocean.

Fig. 11:  UAH original data for the South Polar region, Land and Ocean

Note the much greater range in Land data, with large non-linear multi-year swings- calculate a linear trend for Land at your peril.

Having found changepoints, we can now analyse periods between them.  One way is to calculate means, and step changes between periods.

Fig. 12:  UAH original data for the Tropics based on CuSum changepoints

I deliberately ignored the 2001 changepoint- it made very little difference to means and appears to be a continuation of the series starting in 1997.  Note the step changes are very small, and the final step change is reliant on current data and will change.  While I have shown means and steps, the data are decidedly non-linear with sharp spikes and multi-year rises and falls.

Fig. 13:  UAH original data for the North Polar region based on CuSum changepoints

Note the large step change in the mid-1990s occurs before the 1997-98 El Nino.  The range is much greater than the Tropics.

As the Land data for the South Polar region looks more interesting, I decided to use Land instead of the mean.

Fig. 14:  UAH original data for the South Polar region (Land data) based on CuSum changepoints

Up and down like a toilet seat!

Conclusions:

The data series are characterised by step changes and multi-year rises and falls.

The Polar regions are “poles apart” in their climate behaviours.  Explanations might include: different geography (an ocean almost surrounded by land but subject to warming and cooling currents vs a continent isolated from the rest of the world by a vast ocean); different snow and ice albedo responses; different cloud influences.

The Global mean combines data from regions with very different climatic behaviour.  Averaging hides what is really going on.  The Tropics are governed by ENSO events, and the Poles are completely different.

Please Dr Spencer can you provide separate data for 20-60 degrees North and South?

Comments and interpretations are most welcome.