The feature_weighting module
The feature_weighting module contains the FeatureWeighting class.
This class uses Differentiable Information Imbalance
- class feature_weighting.FeatureWeighting(coordinates=None, distances=None, maxk=None, period=None, verbose=False, n_jobs=2)[source]
- return_backward_greedy_dii_elimination(target_data: Type[Base], initial_weights: ndarray | int | float = None, lambd: float = None, n_epochs: int = 100, learning_rate: float = None, constrain: bool = False, decaying_lr: bool = True)[source]
- Do a stepwise backward elimination of feature weights, always eliminating the lowest weight;
after each elimination the DII is optimized by gradient descent using the remaining features
- Parameters:
target_data – FeatureWeighting object, containing the groundtruth data (D_groundtruth x N array, period (optional)) to be compared to.
initial_weights (np.ndarray or list) – D(input) initial weights for the input features. No zeros allowed here
lambd (float) – softmax scaling. If None (preferred) this chosen automatically with compute_optimal_lambda
n_epochs (int) – number of epochs in each optimization cycle
learning_rate (float) – learning rate. Has to be tuned, especially if constrain=True (otherwise optmization could fail)
constrain (bool) – if True, rescale the weights so the biggest weight = 1
l1_penalty (float) – l1 regularization strength
decaying_lr (bool) – default: True. Apply decaying learning rate = l_rate * 2**(-i_epoch/10) - every 10 epochs the learning rate will be halfed
- Returns:
final_diis – np.ndarray, shape (D). Array of the optmized DII for each of the according weights.
final_weights – np.ndarray, shape (D x D). Array of the optmized weights for each number of non-zero weights.
- History entries added to FeatureWeighting object:
- dii_per_epoch: np.ndarray, shape (D, n_epochs+1, D).
Weights during optimisation for every epoch and every number of non-zero weights. For final weights: weights_list[:,-1,:]
weights_per_epoch: np.ndarray, shape (D, n_epochs+1, ). DII during optimization for every epoch and number of non-zero weights. For final imbalances: diis_list[:,-1]
These history entries can be accessed as follows: objectname.history[‘entry_name’]
- return_dii(target_data: Type[Base], lambd: float = None)[source]
- Computes the DII between two FeatureWeighting objects based
on distances of input data and rank information of groundtruth data.
- Parameters:
target_data – FeatureWeighting object, containing the groundtruth data (D_groundtruth x N array, period (optional)) to be compared to.
lambd (float, optional) – The regularization parameter. Default: 0.1. The higher this value, the more nearest neighbors are included. Can be calculated automatically with ‘return_optimal_lambda’. This sets lambda to a distance smaller than the average distance in the data set but bigger than the minimal distance
- Returns:
dii (float) – The computed DII value. Depends on the softmax scale lambda.
- Raises:
None. –
- return_dii_gradient(target_data: Type[Base], weights: ndarray, lambd: float = None)[source]
- Computes the gradient of the DII between two FeatureWeighting objects
(input object and ground truth object (= target_data)) with respect to the weights of the input features.
- Parameters:
target_data – FeatureWeighting object, containing the groundtruth data (D_groundtruth x N array, period (optional)) to be compared to.
weights (np.ndarray) – The array of weight values for the input values, where D is the dimension of data.
lambd (float, optional) – The regularization parameter. Default: 0.1. The higher this value, the more nearest neighbors are included. Can be calculated automatically with ‘return_optimal_lambda’. This sets lambda to a distance smaller than the average distance in the data set but bigger than the minimal distance
- Returns:
dii_weight_gradient (np.ndarray) – The computed gradient of DII with respect to the weights. Depends on the softmax scale lambda.
- return_lasso_optimization_dii_search(target_data: Type[Base], initial_weights: ndarray | int | float = None, lambd: float = None, n_epochs: int = 100, learning_rate: float = None, l1_penalties: list | float = None, constrain: bool = False, decaying_lr: bool = True, refine: bool = False, plotlasso: bool = True)[source]
Search the number of resulting non-zero weights and the optimized DII for several l1-regularization strengths :param target_data: FeatureWeighting object, containing the groundtruth data
(D_groundtruth x N array, period (optional)) to be compared to.
- Parameters:
initial_weights (np.ndarray or list) – D(input) initial weights for the input features. No zeros allowed. If None (default), the inverse standard deviation of the input features is used
lambd (float or None) – softmax scaling. If None (default), lambd is chosen automatically with compute_optimial_lambda.
n_epochs (int) – number of epochs in each optimization cycle. Default: 100.
learning_rate (float or None) – learning rate. If None (default) is tuned and chosen automatically. Has to be tuned if constrain=True (otherwise optmization could fail).
constrain (bool) – if True, rescale the weights so the biggest weight = 1. Default: False.
l1_penalties (list or None) – l1 regularization strengths to be tested. If None (default), a list of 10 sensible l1-penalties is tested, which are chosen depending on the learning rate.
decaying_lr (bool) – default: True. Apply decaying learning rate = l_rate * 2**(-i_epoch/10) - every 10 epochs the learning rate will be halfed.
refine (bool) – default: False. If True, the l1-penalties are added in between penalties where the number of non-zero weights changes by more than one. This is done to find the optimal l1-penalty for each number of non-zero weights. This option is not suitable for high-dimensional data with more than ~100 features, because the computational time scales with the number of dimensions.
plotlasso (bool) – default: True. If True, a plot is shown, with the optimal DII for each number of non-zero weights, colored by the l1-penalty used. This plot can be used to select select results with reasonably low DII.
- Returns:
num_nonzero_features (np.ndarray) – D-dimensional numbers of non-zero features. Returns nan if no solution was found for a certain number of non-zero weights. In the same order as the according l1-penalties used, final DIIs and final weights.
l1_penalties_opt_per_nfeatures – (np.ndarray): D-dimensional. L1-regularization strengths for each num_nonzero_features, in the same order as the according final DIIs and final weights. If several l1-penalties led to the same number of non-zero weights, the solution with the lowest DII is selected. Returns nan if no solution was found for a certain number of non-zero weights.
dii_opt_per_nfeatures – (np.ndarray): D-dimensional. Final DIIs for each num_nonzero_features, in the same order as the according l1-penalties used and final weights. Returns nan if no solution was found for a certain number of non-zero weights.
weights_opt_per_nfeatures – (np.ndarray): D x D-dimensional. Final weights for each num_nonzero_features, in the same order as the according l1-penalties used and final DIIs used. Returns nan if no solution was found for a certain number of non-zero weights.
- History entries added to FeatureWeighting object:
- l1_penalties (np.ndarray): len(l1_penalties). The l1-regularization strengths tested
(in the order of the returned weights, diis and l1_loss_contributions)
- weights_per_l1_per_epoch (np.ndarray): len(l1_penalties) x n_epochs x D.
All weights for each optimization step for each number of l1-regularization. For final weights: weights_list[:,-1,:]
- dii_per_l1_per_epoch (np.ndarray): len(l1_penalties) x n_epochs.
Imbalance for each optimization step for each number of l1-regularization strength. For final imbalances: diis_list[:,-1]
- l1_term_per_l1_per_epoch (np.ndarray): len(l1_penalties) x n_epochs.
L1 loss contributions for each optimization step for each number of nonzero weights. For final l1_loss_contributions: l1_loss_contributions[:,-1]
These history entries can be accessed as follows: objectname.history[‘entry_name’]
- return_optimal_lambda(fraction: float = 1.0)[source]
Computes the optimal softmax scaling parameter lambda for the DII optimization. This parameter represents a reasonable scale of distances of the data points in the input data set. :param fraction: Zoom in or out from the optimal distance scale.
Default: 1.0. Suggested to keep it at default. Values > 1. show a bigger scale (in the optimization, this means include more neigbors), values < 1 show a smaller scale (in the optimization, this means include less neighbors in the softmax). Values < 1. include on average less neighbors, and very small values only the first neighbor
- return_optimal_learning_rate(target_data: Type[Base], n_epochs: int = 50, n_samples: int = 200, initial_weights: ndarray | int | float = None, lambd: float = None, decaying_lr: bool = True, trial_learning_rates: ndarray = None)[source]
Find the optimal learning rate for the optimization of the DII by testing several on a reduced set :param target_data: FeatureWeighting object, containing the
groundtruth data (D_groundtruth x N array, period (optional)) to be compared to.
- Parameters:
n_epochs (int) – number of epochs in each optimization cycle
n_samples (int) – Number of samples to use for the learning rate screening. Default = 300.
initial_weights (np.ndarray or list) – D(input) initial weights for the input features. No zeros allowed here
lambd (float) – softmax scaling. If None (preferred), this chosen automatically with compute_optimial_lambda
decaying_lr (bool) – default: True. Apply decaying learning rate = l_rate * 2**(-i_epoch/10) - every 10 epochs the learning rate will be halfed
trial_learning_rates (np.ndarray or list or None) – learning rates to try. If None are given, a sensible set of learning rates is tested.
- Returns:
opt_l_rate (float) – Learning rate, which leads to optimal unregularized (no l1-penalty) result in the specified number of epochs.
- History entries added to FeatureWeighting object:
trial_learning_rates: np.ndarray. learning rates which were tested to find optimal one. dii_per_epoch_per_lr: np.ndarray, shape (len(trial_learning_rates), n_epochs+1).
DII for each trial learning rate at each epoch.
- weights_per_epoch_per_lr: np.ndarray, shape (len(trial_learning_rates), n_epochs+1, D).
Weights for each trial learning rate and at each epoch.
These history entries can be accessed as follows: objectname.history[‘entry_name’]
- return_weights_optimize_dii(target_data: Type[Base], n_epochs: int = 100, constrain: bool = False, initial_weights: ndarray | int | float = None, lambd: float = None, learning_rate: float = None, l1_penalty: float = 0.0, decaying_lr: bool = True)[source]
- Optimize the differentiable information imbalance using gradient descent
of the DII between input data object A and groundtruth data object B.
- Parameters:
target_data – FeatureWeighting object, containing the groundtruth data (D_groundtruth x N array, period (optional)) to be compared to.
n_epochs – int, optional The number of epochs in the gradient descent optimization. If None, it is set to 100.
constrain – bool Constrain the sum of the weights to sum up to the number of weights. Default: False
weights (initial) – numpy.ndarray, shape (D,) The array of starting weight values for the input values, where D is the dimension of data. If none, it is initialized to 1/var for each variable This cannot be initialized to 0’s. It can be initialized to all 1 or the inverse of the standard deviation
lambd – float, optional The lambda scaling parameter of the softmax. If None, it is calculated automatically. Default is None.
learning_rate – float, optional The learning rate of the gradient descent. If None, automatically estimated to be fast.
l1_penalty – float, optional The l1-regularization strength, if sparcity is needed. Default: 0 (l1-regularization turned off).
decaying_lr – bool Use exponentially decaying learning rate in gradient descent or not. Default: True.
- Returns:
final_weights – np.ndarray, shape (D). Array of the optmized weights.
- History entries added to FeatureWeighting object:
- weights_per_epoch: np.ndarray, shape (n_epochs+1, D).
List of lists of the weights during optimization.
- dii_per_epoch: np.ndarray, shape (n_epochs+1, ).
List of the differentiable information imbalances during optimization.
- l1_term_per_epoch: np.ndarray, shape (n_epochs+1, ).
List of the l1_penalty terms contributing to the the loss function during optimization.
These history entries can be accessed as follows: objectname.history[‘entry_name’]