Exponential Functions and Scientific Progress

A rant

Here is a small rant. An often touted and rarely challenged expression is that scientific progress happens at an exponential rate. For example, in these days it is not uncommon to see chains of reasoning of the form since (1) scientific progress is exponential, (2) a popular chatbot is approaching human level intelligence on certain tasks, then it must be the case that (3) general artificial intelligence is just around the corner.

First, if we disregard the precise mathematical meaning of these concepts then one could argue that scientific progress is non-linear, in the sense that breakthroughs happen in surges rather than as a mere linear function of time and resources. Thus, we sometimes see years of relatively small (but important) improvements followed by rapid breakthroughs, again followed by relatively minor improvements stabilising the new discoveries. However, non-linearity does not imply exponentiality since there are many possible functions which would fit this distribution.

Second, even if we are currently in a scientific surge spearheaded by machine learning, this does certainly not imply that (the improvement of) large language models must advance at the same rate. In fact, the current data suggests that improvements are logarithmic rather than exponential, in the sense that one needs to vastly increase the training data to see a relatively modest increase in performance, and we currently have no idea how far one can push these models. Note that if improvements were indeed exponential then adding a single new text to the training data would significantly improve the performance of the model, which is of course non-sensical.

Conclusion: do not use mathematical terms in natural language to sound fancy. Make up new words instead.