Expected float entropy

Example of an expected float entropy histogram generated by selecting weighted relations uniformly at random. The long left tail shows that certain choices of weighted relations are isolated from other choices in the sense that they give much lower expected float entropy values.

Expected float entropy (efe) stems from information theory but differs from Shannon entropy by including relationships as parameters. It is a measure of the expected amount of information required to specify the state of a system (such as an artificial or biological neural network) beyond what is already known about the system from the relationship parameters. For non-random systems, certain choices of the relationship parameters are isolated from other choices in the sense that they give much lower expected float entropy values and, therefore, the system defines relationships. Expected float entropy is an important definition in a recently developed mathematical theory of consciousness in which, through a variety of learning paradigms, the brain builds a model of the world by building relationships and, in the context of these relationships, a brain state acquires meaning in the form of the relational content of the corresponding experience. The principle article (Quasi-Conscious Multivariate Systems) on this mathematical theory was published in 2015 following a more preliminary publication in 2012.

The nomenclature "float entropy" comes from the notion of floating a choice of relationship parameters over a state of a system, similar to the idiom "to float an idea". Optimisation methods are used in order to obtain the relationship parameters that minimise expected float entropy. A process that performs this minimisation is itself a type of learning paradigm.

Overview

Relationships are ubiquitous among mathematical structures. In particular, weighted relations (also called weighted graphs and weighted networks) are very general mathematical objects and, in the finite case, are often handled in matrix form. They are a generalization of graphs and include all functions since functions are a rather constrained type of graph. It is also the case that consciousness is awash with relationships; for example, red has a stronger relationship to orange than to green, relationships between points in our field of view give rise to geometry, some smells are similar while others are very different, and there's an enormity of other relationships involving many senses, such as between the sound of someone's name, their visual appearance, and the timbre of their voice. Expected float entropy includes weighted relations as parameters and, for non-random systems, certain choices of weighted relations are isolated from other choices in the sense that they give much lower expected float entropy values. Therefore, systems such as the brain define relationships and, according to the theory, in the context of these relationships, a brain state acquires meaning in the form of the relational content of the corresponding experience.

Expected float entropy minimization is very general in scope. For example, the theory has been successfully applied in the context of image processing but also applies to waveform recovery from audio data.

Definitions and connection with Shannon entropy

Definitions

For a nonempty set $S$, a weighted relation on $S$ is a function of the form

$R:S^2\to [0,1]$.

Such a weighted relation is called reflexive if $R(a,a)=1$ for all $a\in S$, and symmetric if $R(a,b)=R(b,a)$ for all $a,b\in S$. The set of all reflexive, symmetric weighted relations on $S$ is denoted ${\mathrm{\Psi }}_S$.

If $S$ is the set of nodes of a system, such as a neural network, then a state of the system $S_i$ is given by the aggregate of the states of the nodes over some range $V:=\{v_1,v_2,\dots ,v_m\}$ of node states. Therefore, each state of the system $S_i$ is determined by a corresponding function $f_i:S\to V$. The set of all possible states of the system is denoted ${\mathrm{\Omega }}_{S,V}$.

Given an element $S_i\in {\mathrm{\Omega }}_{S,V}$, the above definitions give rise to a canonical map from ${\mathrm{\Psi }}_V$ to ${\mathrm{\Psi }}_S$. That is, for $U\in {\mathrm{\Psi }}_V$, the function $R\{U,S_i\}$ defined by

$R\{U,S_i\}(a,b):=U(f_i(a),f_i(b))$, for all $a,b\in S$,

is an element of ${\mathrm{\Psi }}_S$.

For $U\in {\mathrm{\Psi }}_V$ and $R\in {\mathrm{\Psi }}_S$, the float entropy of a state of the system $S_i\in {\mathrm{\Omega }}_{S,V}$, relative to $U$ and $R$, is defined as

$fe(R,U,S_i):={\log }_2(\mathrm{\#}\{S_j\in {\mathrm{\Omega }}_{S,V}:d(R,R\{U,S_j\})\le d(R,R\{U,S_i\})\})$,

where $d$ is a metric given by a matrix norm on the elements of ${\mathrm{\Psi }}_S$ in matrix form. In the article Quasi-Conscious Multivariate Systems the $L_1$ norm is used. The article also includes a more general definition of float entropy called multirelational float entropy and the nodes of the system can be larger structures than individual neurons.

The expected float entropy (efe) of a system, relative to $U\in {\mathrm{\Psi }}_V$ and $R\in {\mathrm{\Psi }}_S$, is defined as

$efe(R,U,P):=\sum _{S_i\in {\mathrm{\Omega }}_{S,V}}P(S_i)fe(R,U,S_i)$,

where $P$ is the probability distribution $P:{\mathrm{\Omega }}_{S,V}\to [0,1]$ determined by the bias of the system due to the long-term effect of the system's inherent learning paradigms in response to external stimulus.

According to the theory, a system (such as the brain and its subregions) defines a particular choice of $U$ and $R$ (up to a certain resolution) under the requirement that the efe is minimized. Therefore, for a given system (i.e., for a fixed $P$), solutions in $U$ and $R$ to the equation

$efe(R,U,P)=\underset{R^{\prime }\in {\mathrm{\Psi }}_S,U^{\prime }\in {\mathrm{\Psi }}_V}{\min }efe(R^{\prime },U^{\prime },P)$

are the weighted relations of interest.

Connection with Shannon entropy

The Shannon entropy $H$ of a system is defined as

$H:=\sum _{S_i\in {\mathrm{\Omega }}_{S,V}}P(S_i){\log }_2\left(\frac{1}{P(S_i)}\right)$.

For $U\in {\mathrm{\Psi }}_V$, $R\in {\mathrm{\Psi }}_S$ and $A_{S_i}:=\{S_j\in {\mathrm{\Omega }}_{S,V}:d(R,R\{U,S_j\})\le d(R,R\{U,S_i\})\}$ the following equalities holds

$\sum _{S_i\in {\mathrm{\Omega }}_{S,V}}P(S_i){\log }_2\left(\frac{1}{P(S_i\mid A_{S_i})}\right)$

          $=\sum _{S_i\in {\mathrm{\Omega }}_{S,V}}P(S_i){\log }_2\left(\frac{\sum _{S_j\in A_{S_i}}P(S_j)}{P(S_i)}\right)=H+\sum _{S_i\in {\mathrm{\Omega }}_{S,V}}P(S_i){\log }_2(\sum _{S_j\in A_{S_i}}P(S_j))$          (1).

The expression on the left is similar in form to the definition of Shannon entropy. The middle expression reveals the value to be similar to that of $efe(R,U,P)$ when the probabilities in the argument of the logarithm are comparable. Indeed, $efe(R,U,P)$ is an approximation of (1). The expression on the right of (1) shows the mathematical connection to Shannon entropy; the first term is the Shannon entropy $H$ of the system and, with consideration of the log function, the second term has a negative value between $-H$ and 0.

Connection to other mathematical theories of consciousness

There are some similarities between the minimization of expected float entropy and the minimization of surprise in Karl J. Friston's Free energy principle. The theory is also somewhat complementary to Giulio Tononi's Integrated information theory (IIT) which was initially developed to quantify consciousness but gave little priority to how systems may define relationships.

See also

• Association rule learning
• Consciousness
• Free energy principle
• Hard problem of consciousness
• Information theory
• Integrated information theory
• Mind–body problem
• Philosophy of mind

References

📰 Article(s) of the same category(ies)

• Exformation
• Channel simulation
• Physical information
• Noogenesis
• Email to the Universe
• Chopra foundation
• Information-theoretic death

---