Infodynamics, Information Entropy and the Second Law of Thermodynamics

Information and Energy are related. The Second Law of Thermodynamics states that entropy continuously increases, applies to changes in energy and heat, but it does not apply to information dynamics. Changes in energy and information are coupled but have completely different dynamics. Infodynamics has made clear that Thermodynamic Entropy and Information Entropy are distinct concepts. Total Energy contains Free Energy and Thermodynamic Entropy, whereas Total Information or Information Entropy contains Useful Information and Noise, both of which may be gained or lost in irreversible processes. Increases in Free Energy of open systems require more Useful Information, reducing or increasing Thermodynamic Entropy. Empirical data show that the more Free Energy is created, the more Useful Information is required; and the more Useful Information is produced the more Free Energy is spent. The Energy – Information relationship underlies all processes where novel structures, forms and systems emerge. Although science cannot predict the structure of information that will produce Free Energy, engineers have been successful in finding Useful Information that increases Free Energy. Here I explore the fate of information in irreversible processes and its relation with the Second Law of Thermodynamics, showing that distinguishing between Thermodynamic Entropy and Information Entropy, and disentangling its interactions, is fundamental in advancing our understanding of thermodynamics of irreversible processes.


Introduction
The Second Law of Thermodynamics summarizes empirical experience showing that heat always flows from warmer Qeios, CC-BY 4.0 • Article, June 18, 2024 Qeios ID: T13JP9.4 • https://doi.org/10.32388/T13JP9. 4  1/10 bodies to colder ones and that the efficiency of conversion of heat to work in a heat engine has an upper limit.This first formulation in 1824 by Sadi Carnot [1] preceded the proper definition of entropy in 1850 by Rudolf Clausius [2] .Clausius proposed the term Entropy (S) to refer to the unavailable energy when mechanical work is derived from heat energy.This helped formulate a rigorous definition of the second law based on the fact that heat can never flow from a colder to a warmer body without work by special contraptions that produce Entropy and require Information.A popular understanding of the consequences of this law is the heat death of the universe [3] , as heat distributes uniformly in a closed system and all irreversible processes produce heat.If the Universe is a closed system, there will be a moment when no heat gradient is left to produce work and thus the Universe can be considered to be dead.Ludwig Boltzmann [4] used this entropy concept as the measure of the number of possible microscopic arrangements or states of individual atoms and molecules of a system that comply with the macroscopic condition of the system which define its kinetic energy [5] .This energy is expressed as heat, measured in calories defined by the increase in temperature caused when applied to a unit of mass of water.
Confusingly, Shannon used the term Entropy to refer to a different phenomenon.Developments in mathematical statistics by Ronald Fisher led to the development of what we call now Fisher Information (extensively discussed by Frieden [6] ).
Fisher Information presents a way of measuring the amount of information that an observable random variable carries.
Formally, it is the variance of the score, or the expected value of the observed information.Shannon describes the same phenomena but confusingly used the term Information Entropy [7] to describe the probability of a particular event occurring from a random variable.Both definitions overlap but important differences exist.Some bridges to Boltzmann entropy definition and that of Shannon may exist [8] Surprisingly, the concepts of Entropy and Information are still fuzzy and controversial.Many conceptual, theoretical and practical problems regarding information remain unsolved.The use of Information in physics was always problematic.
Maxwell [9] proposed a thought experiment, where a demon controls a door between two chambers containing gas.As individual gas molecules (or atoms) approach the door, the demon quickly opens and closes the door to allow only fastmoving molecules to pass through in one direction, and only slow-moving molecules to pass through in the other.
Because the kinetic temperature of a gas depends on the velocities of its constituent molecules, the demon's actions cause one chamber to warm up and the other to cool down.This would decrease the total entropy of the system, seemingly without applying any work, thereby violating the second law of thermodynamics.This thought experiment used a demon which worked without consuming energy.This, of course, is not possible.Brillouin [10] showed that Maxwell's demon cannot operate because there is a relation between information and entropy [11] .Energy is required to find, use and apply information.That is, information requires energy, but it is not energy.However, Information and Energy are related.Despite the fact that the Second Law of Thermodynamics applies to changes in energy and heat, thus the descriptor Thermodynamics, it is often used when referring to information dynamics, assuming that information and energy may be lost in irreversible processes.This, however, is not true, as discussed in more depth elsewhere [12] .There is a strong relationship between changes in Energy and the Information required by a given system to increase its Free Energy and reduce its thermodynamic Entropy: the more Free Energy needed, the more Information is required; and the more Information is produced the more Free Energy is spent.A review of empirical studies quantifying this relationship has been published recently [13] .In irreversible processes, information is not necessarily consumed whereas thermodynamic entropy is always produced.The Energy -Information relationship underlies all processes where novel structures, forms and systems emerge [14] and is a basic feature of irreversible processes in thermodynamics.
For different disciplines of science, different conceptions of information exist that do not overlap in all their considerations.
For example in quantum mechanics [15] information can suffer entanglement [16] , and for many, but not all [17] , quantum information is indestructible [18][19] .In contrast, biological information can be lost, as exemplified by species and ecosystem extinction [20] .
Thermodynamic entropy has been very successful in explaining complex irreversible processes in chemistry and mechanics and even in biological evolution [21] .Although several authors [22][23][24] [25] have made efforts in clarifying the relation between Entropy and the Second Law of Thermodynamics, and to dispel widespread misleading misconceptions [26][27] [28] .Misconceptions remain, most arising from confusing Thermodynamic Entropy with Information Entropy, and therefore wrongly relating the second law of thermodynamics to Information Entropy.
For example, Thermodynamic Entropy was confused with Shannon's Information Entropy by Brillouin [29].He wrote "Whenever an experiment is performed in the laboratory, it is paid for by an increase of entropy, and a generalized Carnot Principle states that the price paid in increased entropy must always be larger than the amount of information gained".He assumed that thermodynamic entropy and statistical information can be measured with the same units.This rational flaw shared by many researchers has hindered our understanding of the interaction between energy and information.
Here I attempt to clarify some of these issues focusing on differences that are relevant to entropy as the product of the Second Law of Thermodynamics.Thermodynamic Entropy is energy that is not available to produce useful work, and is evidenced as heat.Information Entropy is a different concept: It refers to the probability of a given variable to contain novel or new information.Due to its fuzzy definition, it tends to be referred to as negative Thermodynamic Entropy or Negentropy.However, Thermodynamic Entropy and Information Entropy are two completely different concepts.This confusion also leads to misleading interpretations of the relation between the Second Law with Information.Some fundamental differences between Energy (i.e Thermodynamic Entropy) and Information (i.e Information Entropy) are: Information entropy relates to uncertainty in outcomes, while thermodynamic entropy pertains to energy distribution in physical systems.
Thermodynamic Entropy and Information Entropy are measured differently.The former uses units of energy whereas the later uses units of probability.
Thermodynamic entropy follows the Third Law of Thermodynamics: Entropy = 0 at absolute temperature = 0 0 K, whereas information entropy can be > 0 at 0 0 K Information entropy can have negative values of entropy, grossly contradicting the Third Law.
In thermodynamics, energy in the form of heat flows from warmer to colder parts of the system; whereas information can travel from complex systems with high information content to simpler ones with less information, and the other way round.Belavkin, et al wrote [30] "The second law of thermodynamics forbids in an isolated system, the existence of processes accompanied by an increase of entropy.If there exists an influx of information dI about the system, i.e. if the physical system is isolated only thermally, but not informationally, then the above law should be generalized by substituting inequality dH ≥ 0 with inequality dH + dI ≥ 0. Therefore, if there is an influx of information, then the thermal energy of the system can be converted (without the help of a refrigerator) into mechanical energy.In other words, the existence of perpetual motion of the second kind powered by information becomes possible".This statement assumes that creation and accumulation of information can be achieved with no energetic cost.This of course violates the First Law of Thermodynamics.
Landauer [31] argued that "computing machines inevitably involve devices which perform logical functions that do not have a single-valued inverse.This logical irreversibility is associated with physical irreversibility and requires a minimal heat generation".He however overlooks that the work is done by a machine where the information is stored and is not a direct product of information dynamics.The energy to mold a substrate to engrave information varies depending on the substrate, the machine and the encryption system used: a scripture in stone will require more energy than one written in the sand.But the value of the information does not depend on the substrate where it is encrypted!
The energy required to retrieve, encrypt, transmit, store, create or manipulate information varies according to the specific case analyzed.No general uniform formula exists to predict the costs in energy involved.Only empirical experiments can assess these costs.
Maximum Thermodynamic Entropy refers to total dissipation of energy so that no useful work is produced.Maximum Information Entropy is the least biased distribution that encodes certain given information which maximizes the Information Entropy.
We have to distinguish between Thermodynamic Entropy S and Information Entropy I.In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes.Whereas Thermodynamic Entropy refers to the relationship of the internal energy E that is available or unavailable for transformations in form of heat S and work F. Thermodynamic and Information Entropy refer to very different physical phenomena.Sometimes, negative Thermodynamic Entropy (negentropy) is linked to Information Entropy, a mistake I and many others [32] did many times in the past.Negentropic processes do exist: processes that reduce the production of S to favor F; but the relationship between S and I is much more complex than formerly acknowledged [33] .Here is an attempt to unravel this confusion.

How the Second Law of Thermodynamics affects Information
The material substrate where information is stored suffers the effect of the Second Law: genomes and proteins degrade in time and so does any physical system storing information.But living systems, located far from thermodynamic equilibrium, not only store information but may increase their stored information.An empirical law for irreversible thermodynamics states that the thermodynamic free energy F and the information used to produce the energy required for the work done Φ are related.That is expressed as ΔF ~ ΔΦ [34] , recognizing that increases in useful information and increments in free energy are coupled.This means that far from equilibrium systems suffering irreversible processes dissipate energy but can increase their available useful information or reduce their useless information.Useful here refers to the ability to produce Free Energy F which in turn produces useful Work.
Let's consider Information Complexity as the total amount of information in a system I, and Useful Information Φ the one producing Free Energy F and thus work.Free Energy and Work are thermodynamic concepts so that Helmholtz Free Energy can be represented as F = E -TS, where F is Free Energy, E total energy, T temperature and S thermodynamic entropy.
We might generalize this equation incorporating T in the conceptualization of S, as follows: ΔF= ΣΔEi -ΔS, where Ei represents the different types of energy and S the entropy due to energetic processes, and Δ Φ = ΣΔIi -ΔN, where Φ is useful information or the information that accounts for ΔF, Ii are the different types of information and N is noise, useless information or information that produces entropy.
Using these abstractions we can write ΔF ~ ΔΦ, which seems to be a law for irreversible thermodynamics, as suggested by empirical evidence 2 .The exact relation between F and Φ remains to be untangled but one link is the relation between S and N.In energetic terms S is related to the order or predictability of a system, and so is N.The problem here is that order and complexity are related, and these measures depend on the level of complexity addressed.This introduces distortions when comparing multiple levels or multiple dimensions of energy and information.This relationship means that in order to increase F there needs to be an increase in Φ by increasing I or decreasing N.That is, not any type of information will do.Information may be misleading, false and/or destructive provoking a reduction of F. We call this type of information N or noise.For now, the type of information, Φ or N, can only be assessed empirically through its effect on F.
When increases of F are concomitant to increases in Φ we have a synergistic process.These limitations do not occur with energy, as relations between the different forms of energy are much better understood in physics than those of information.Thus Φ is a proxy of Useful Information.
How is this relationship affected by the Second Law of Thermodynamics?The Second Law states that an open system must dissipate thermodynamic entropy S. Maintenance of information requires energy and thus adds to the dissipation of this entropy.However, information might reduce the production of thermodynamic entropy by increasing free energy F of the system Thus, information has a dual role: its maintenance and creation increases and its working decreases The relation between Information Entropy and Thermodynamic Entropy in this context can be described as follows.
F = E -S Where E is total energy and S the thermodynamic entropy due to energetic processes Φ is useful information or the information that accounts for F is I the total information accounting for its complexity, and N useless information or noise Φ = I -N and F = E -S then Φ = k(F) and F = j(Φ) were k and j are two different functions relating F with Φ We have empirical tools to measure Φ and F quantitatively and experimentally and Total Information I can then be estimated as complexity or I = Φ + N [35] The fact that energy can be present in useful or useless form regarding the power to produce work F = E -TS, and the insight that information can also be classified as useful or useless for the purpose of creating F, ΔF ~ ΔΦ force us to accept that Φ = I -N as not all information serves to produce F.This is fundamental in understanding the working of information in thermodynamics and requires accepting a clear difference between Thermodynamic Entropy (S) and Information Entropy (I).

Discussion
Colloquial language may describe genetic information inside an organism that codes all necessary tools for swimming in the water are useless information if the organism lives in a desert.The usefulness of information is determined by its environment.The same is true for energy.Energy in the form of light is completely useless to a plant that lacks photosynthetic capabilities.The heat produced by its reflection on the leaves might dry them.This same heat from the sun reaching the skin of a reptile might activate its metabolism helping to digest its last meal.Thus, the usefulness of energy and information depends on the ability to produce or capture Free Energy F which is able to generate Work.The structural information in chloroplast, and the metabolic information in endotherm animals allow for the capture of energy that can be used to produce work and is therefore called Free.
No unique way to transform energy into Free Energy exists.Information Φ may arrange existing energy E changing the balance between F and S.An example of how F can create Φ is biological evolution where F is spent in sexual selection, meiosis and reproduction.Selection found that sex by diploid cells are the most efficient in handling, conserving and increasing useful information [36] which seems impossible to achieve without sex [37] .Modern science seems to have achieved something similar with F spend pursuing empirical sciences to produce Φ [38] .The detailed relationship between Φ and F remains to be worked out for each level of complexity.Science cannot predict the structure of information that will produce Free Energy as no general formula for F ~ Φ exists.Engineers, including social engineers [39] , have been successful in finding empirically Φ to increase F for specific situations and are now the most specialized in producing knowledge on a large scale.
This relation F ~ Φ is consilient with many recent theories relating information with the Second Law of Thermodynamics and may serve as a unifying element for different perspectives such as: David Deutsch and Chiara Marietto [40] proposition of the Constructor Theory of Information, describing physical transformations, or tasks, that are possible versus those that are impossible Adrian Bejan and Sylvie Lorente [41] proposition of the Constructal Law describing the design and evolution in nature.
"For a finite-size system to persist in time (to live), it must evolve in such a way that it provides easier access to the imposed (global) currents that flow through it¨.That is, Φ must increase in time to compensate increases in S Adami, C., et al [42] proposition that natural selection forces genomes to behave as a natural "Maxwell Demon" Haken, H. and Portugali.J. [43] proposition of a Synergic Computer to resolve the interplay between Shannon's information and semantic or pragmatic information.
Kolchinsky A., Wolpert D.H. [44] classification of information acquired in a measurement into "accessible" and "inaccessible" Smith, E. [45] representation of information in the biosphere, and the energetic constraints limiting the imposition or maintenance of that information Parrondo, J., Horowitz, J. & Sagawa, T. [46] presentation of a theoretical framework for the thermodynamics of information based on stochastic thermodynamics and fluctuation theorems Varley T, Hoel E. [47] evidence that coarse-gaining can convert information from one 'type' to another and reduction of complexity leads to a loss of synergistic information in scientific models.
Rainer, F., Ebeling, W. [48] evidence that information emerges from structural information in the course of evolution processes.
However, other perspectives are incompatible with the conception of infodynamics presented here.For example, the theoretical mix-up of heat with information, as presented by Vopson [49] with no basis in physics; or equating the cost of transmitting information to that of creating the information [50] , can not be integrated with the concept Φ. Distinguishing between Thermodynamic Entropy and Information Entropy, and disentangling its interactions, is fundamental in advancing our understanding of thermodynamics of irreversible processes.Much work remains to be done.
thermodynamic entropy!When does one effect overcome the other?In living autopoietic systems, information increases and free energy increases.Total consumption of energy generally also increases but efficiency gains in production of free energy might reduce it.Evolution through Natural Selection in systems living in resource poor environments might accumulate information that make them more efficient in resource use, whereas organisms evolving in environments saturated with energetic resources might skip that trend.Proposal Qeios, CC-BY 4.0 • Article, June 18, 2024 Qeios ID: T13JP9.4 • https://doi.org/10.32388/T13JP9.4 5/10