The Nile’s gifts for understanding climate – Part 3
Good news for reasonable people
[There is also a Greek version of the post—Υπάρχει και ελληνική έκδοση της ανάρτησης]
”On the one hand, a prophet is he who foretells the future by revelation of the Spirit; on the other hand, a stochastes is he who infers the future by prudence, comparing similar states, and by the experience of forefathers” (Basilius Caesariensis, Enarratio in prophetam Isaiam, 3.102.26)
[Original: «Οὐκοῦν Προφήτης μέν ἐστιν, ὁ κατὰ ἀποκάλυψιν τοῦ Πνεύματος προαγορεύων τὸ μέλλον· στοχαστὴς δὲ, ὁ διὰ σύνεσιν ἐκ τῆς τοῦ ὁμοίου παραθέσεως, διὰ τὴν πεῖραν τῶν προλαβόντων, τὸ μέλλον συντεκμαιρόμενος.» (Ἅγιος Βασίλειος ὁ Μέγας, Ερμηνεία εις τον προφήτην Ησαΐαν, 3.102.26)]
The second part of this trilogy had a subtitle “Bad news for forecast freaks and determinists”. The same news is good news for reasonable people. This was touched upon in the earlier part:
The uncertainly, else known as entropy, rules. That’s the most important physical law (in thermodynamics it’s known as the Second Law). […] Entropy makes life as fair as possible: even if a controligarch was able to build the DREAM machine, we may conjecture that its secret code would leak by some unpredictable action (cf. wikileaks, climategate, etc.). And it also makes life fascinating. Without the reign of entropy, i.e. uncertainty, life would be a universal boredom, and concepts such as hope, will (especially free will), freedom, expectation, optimism, etc., would hardly make sense.
I will explain it further in this post and I will try to show that uncertainty can be dealt with in a scientific manner, using stochastics.
A first hint about what stochastics is can be gained by the epigram due to Basilius Caesariensis1. Basilius contrasts a prophet with a ‘stochastes’ (‘στοχαστής’)—a noun usually and mistakenly translated into English as ‘diviner’. In fact, a stochastes is the opposite of a diviner—he is a prudent man who infers the future from similar states in the past.
In modern terms, what a traditional stochastes used to do empirically, is done through the scientific discipline of stochastics. Stochastics is a superset of probability theory, statistics and stochastic processes.2 The objective of a stochastic prediction is to assign a probability to a certain value of a variable, so as to know how likely the occurrence of this value is in a certain time interval. The exact time that such a value would occur is not included in the objective and is not relevant to most practical problems.
Suppose that a city is planning to build a bridge on its river and wants to find the water depth in the river, so that the floods do not destroy the bridge for a design lifespan, say the next fifty years. Taking into account the risk (which can never be eliminated) and the cost, the engineers suggest that the design discharge should be 1000 m³/s, which corresponds to a water depth of 10 m. This means that, with an acceptably low probability, we expect that some day in the next 50 years there will occur, with that probability, a flood in the river in which the water depth will be 10 m or higher. In all other days during that 50-year period, we expect that the water level would be lower than 10 m, with high probability. Does anybody care which the single day of that imaginary exceptional event (with low probability) would be? Would it be 1 February 2050, 23 March 2063 or another date among 18 260? A stochastes or an engineer cannot answer this question. Nor does he asks it at all. This is a question to be dealt with by a prophet—or a climate modeller. Actually, if it were possible to answer this question using reason—not using the supernatural skills of prophets and climodellers—our life would be very problematic. (Think about it.)
The above reasoning is probabilistic. In reality, when the 50 years have passed, the exceptional design water depth might have occurred once or even twice, etc., or, most probably, might have not occurred at all—because its probability is chosen to be low.
How can we assign probability and risk to the different flood or river level values? As Basilius correctly states, by studying past states.
Now enters the nilometer time series, the longest instrumental data set available, with a length of 849 years. The graph below reproduced from the booklet3, shows the annual minimum and annual maximum water depths of the Nile from 622 to 1470 AD.
Observing the lower graph, we see that the maximum water level each year usually varies between 8.5 and 10 m, but there are some rare exceptions with much higher and much lower values. Such diversions from the usual variation have been termed the Noah effect by Mandelbrot and Wallis, and a black swan by Taleb.4 Both terms aim to popularize those exceptions from regular variation. The latter became really popular generating tens of thousands of citations in the scientific and non-scientific literature. Even the politicians love to use it. But I do not like either of these terms. I prefer the more scientific and banal term heavy tail of the probability distribution. The purpose of this term is to contrast the probability distributions of natural processes with those that have light tails, like the popular normal distribution. This is not to say that the probability distributions of natural processes are abnormal. They are just natural.
Another important revelation from the nilometer graphs, is the dominance of "climate change". In particular, the graph of the minimum water levels shows that the 30-year climatic values exhibit high variability. The British hydrologist Harold Edwin Hurst, who devoted his lifetime to measuring and studying the Nile, and was the first to discover this behaviour in natural processes, did not use the term “climate change”. He simply wrote:
Although in random events groups of high or low values do occur, their tendency to occur in natural events is greater. This is the main difference between natural and random events.5
Mandelbrot and Wallis used the term Joseph effect for this behaviour, inspired by the biblical story of the seven fat and the seven lean cows. Later, this behaviour was frequently referred to as the Hurst phenomenon. Together with Tim Cohn6, we coined the terms Hurst-Kolmogorov behaviour and Hurst-Kolmogorov dynamics with a two-fold purpose: (a) to discourage the use of the word phenomenon as the behaviour is not phenomenal—it’s the rule in natural processes; (b) to make the link with Soviet mathematician Andrey Kolmogorov, who invented the mathematics of this dynamics7 (notably, a decade earlier before Hurst discovered the natural behaviour).
The difference of the natural processes from the random processes is illustrated in the following graph, reproduced from the booklet: Taking the annual nilometer minimum values (upper panel) and rearranging them in time at random (lower panel), we get a series which does not show climatic changes. In other words, nature produces climatic changes, while a random process (like in dice throws or roulette wheels) would produce a stable climate.
Both the above graphs suggest precisely the same uncertainty for the annual values, as the two series contain precisely the same data but temporally redistributed. The natural series additionally suggests uncertainty in climate. Hence, nature produces enhanced uncertainty: uncertainty in both short and long term.
Is it difficult to model the natural behaviour within stochastics? Not at all. We only need the concept of variance. From the annual series (849 values) we construct the series of 2-year averages (424 values), that of 3-year averages (283 values) and so on, up to 84-year averages (10 values). We don’t go beyond the 84-year time scale as the available data values would be too few to estimate the variance. The set of values of variance γ(κ) as a function of the time scale κ (= 1, 2, …, 84 years) is termed the climacogram. Its double-logarithmic plot is shown below, reproduced from the booklet.
If the process were purely random (white noise), the slope of the arrangement of points in the climacogram would be —1. (It is very easy to prove that.) In real-world processes, the slope is different from –1, designated as 2H – 2 where H is the Hurst parameter, taking values from 0 to 1, with the value of 1/2 corresponding to the purely random process.
The reality in the nilometer time series departs substantially from the purely random behaviour and is consistent with the Hurst-Kolmogorov behaviour with H = 0.85 and 0.82 for the minimum and maximum water depths, respectively.
Essentially, the Hurst-Kolmogorov behaviour manifests that long-term changes are much more frequent and intense than commonly perceived and, simultaneously, that the future states are much more uncertain and unpredictable on long time horizons than implied by pure randomness. So, a high value of H indicates enhanced multi-scale change and, hence, enhanced uncertainty.
The typical conceptualization and the standard statistical methodologies do not take into account the two important natural behaviours seen in the long nilometer series: the heavy tails and the Hurst-Kolmogorov dynamics. As a result, the standard methodologies underestimate substantially the probability and duration of extreme events. The inappropriateness of neglecting these behaviours has not been widely known because they are hidden if the time series of observations are not long enough. I believe that just acknowledging these two behaviours and reproducing them in modelling resolve most of the underestimation problems in the probability of occurrence of extremes and hence the risk. And this is done scientifically, without resorting to poorly performing deterministic climate models.
By the way, the Greek Orthodox church celebrates him on the first of the year (1 January) as the Santa who brings the gifts to children. I wish that in a few days, when we celebrate him, Santa bring to all of us, children and adults, intellectual gifts, which are the most important.
See additional explanations in Digression 1.A in my book:
D. Koutsoyiannis, Stochastics of Hydroclimatic Extremes - A Cool Look at Risk, Edition 3, ISBN: 978-618-85370-0-2, 391 pages, doi: 10.57713/kallipos-1, Kallipos Open Academic Editions, Athens, 2023.
B.B. Mandelbrot and J.R. Wallis, 1968. Noah, Joseph, and operational hydrology. Water Resources Research, 4(5), 909-918.
N.N. Taleb, 2007. The Black Swan: The Impact of the Highly Improbable. Random House, USA.
H.E. Hurst, 1951. Long term storage capacities of reservoirs. Trans. Am. Soc. Civil Engrs, 116, 776–808.
Kolmogorov, A.N., 1940. Wiener spirals and some other interesting curves in a Hilbert space. Dokl. Akad. Nauk SSSR, 26, 115-118. (English edition: Kolmogorov, A.N., 1991, Selected Works of A. N. Kolmogorov - Volume 1, Mathematics and Mechanics, ed. by Tikhomirov, V.M., Kluwer, Dordrecht, The Netherlands, 324-326).
As a non-scientist and hopeless mathematician, I am thankful to you for your clear explanation of stochastics. A chink of light has opened for me. I think that the deterministic way of analysing climate (past, present and future) is based on an overwhelming anthropomorphic arrogance. This arrogance imposes itself on sensible science. Sensible science collects the data nature provided, provides and will provide and allows the resulting evidence to speak for itself (via stochasitics for the mathematically inclined) and via common sense for the rest!