Fuzzy entropy and similarity based feature selection method

In data analytics and machine learning it is common in the era of big data that there is a wealth of information present as available features for real world problems. However not all the features are relevant for the problem at hand and because of it feature selection, where the purpose is to find the most important features for the problem, is often used. The aim of feature selection is to find the most important features for the analysis task at hand (e.g. statistical classification or regression analysis). Fuzzy entropy and similarity based feature selection method was created in 2011. It is a feature selection method for selecting the most important features in any classification problems. The basic idea, in short, is to use training data to create ideal vectors for each class and then compute similarities of the samples and ideal vectors. Once this has been done, the next step is to use similarity values to calculate fuzzy entropy values for each feature. Lower entropy values indicate structureness, while higher entropy values indicate randomness and hence lower importance. The algorithm, in short, goes as follows.

• Normalize the data into unit interval.$X^d\to [0,1{]}^d$
• Obtain ideal vectors v_i for each class i from the training data set e.g. by using arithmetic mean.

$v_{i,d}=\left(\frac{1}{n_i}\sum _{x\in X_i}{x}_d\right),d=1,\cdots ,D,i=1,\cdots ,N$

where x_d denotes data samples feature d and we restrict the mean calculation to only those samples that belong to class i.

• Obtain similarities $S(x_{j,d},v_{i,d})$ with $x_j\in X_i$ between data sample vector x_j belonging to class i and the ideal vector v_i. For the d^th feature

$S(x_{j,d},v_{i,d})={(1-|{\left(x_{j,d}\right)}^p-{\left(v_{i,d}\right)}^p|)}^{\frac{1}{p}}$ $d=1,\dots ,D,i=1,\dots ,N,j=1,\dots ,n$

• Compute relevance R by computing fuzzy entropy value for each feature

$R_d=\sum _{i=1}^N\sum _{j=1}^{n}H(S(x_{j,d},v_{i,d}))$

where the similarity for feature d of a sample x_j with ideal vector v_i of class i is summed over all samples (j=1,...,n) and classes (i=1,...,N)

• Based on the relevance of the features, remove the most irrelevant feature(s).

For fuzzy entropy H commonly used fuzzy entropy measure is De Luca & Termini's fuzzy entropy measure $H=-\sum _i\mu (x_i)log(\mu (x_i))+(1-\mu (x_i))log(1-\mu (x_i))$ where now elements $x_i$ membership degree $\mu (x_i)$ to fuzzy set is simply replaced by similarity degree $S(x_{j,d},v_{i,d})$.

Also other fuzzy entropy measures can be used e.g. Parkash et al. fuzzy entropy. There fuzzy entropy is calculated as $H=\sum _i{w}_i(sin\frac{\pi \mu (x_i)}{2}+sin\frac{\pi (1-\mu (x_i))}{2}-1)$

Scaling factor to FES was later suggested in. There scaling factor is calculated as

$S{F}_{i,d}=1-\frac{{(\sum _{i\ne j}|v_{i,d}-v_{j,d}{|}^l)}^{\frac{1}{l}}}{N-1}$

The purpose there, in short, is that if the feature in a class takes on average largely different values than in other classes, then this leads to smaller entropy values. Scaled entropy (SE) value is then calculated as

$S{E}_d=\sum _{i=1}^N{H}_{i,d}S{F}_{i,d}$

FES has been implemented in matlab, python and R.

References

This article "Fuzzy entropy and similarity based feature selection method" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Fuzzy entropy and similarity based feature selection method. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

📰 Article(s) of the same category(ies)

• ENTROPY GENERATION MINIMIZATION IN LAMINAR BOUNDARY LAYER FLOW
• Envelope model

---