Bandit feedback
Bandit feedback is a concept used in Machine Learning, ML, a subset of Artificial Intelligence, AI, particularly in the context of reinforcement learning and recommendation systems. It refers to the feedback or information provided to an algorithm when it makes sequential decisions, typically as a "bandit problem." In bandit feedback,[1]only the reward of the chosen action is taken into account. No information is given about the rewards of the other actions. Bandit feedback is commonly used in multi-armed bandit problems, where an agent must choose between multiple actions with uncertain rewards and explore the different actions to learn which one has the highest reward, while also exploiting the actions with high rewards. There are multiple approaches to learning from bandit feedback, such as off-policy estimators and counterfactual risk minimization. [2] In a bandit problem, an agent is faced with a series of choices, each one associated with an unknown reward. The agent aims to maximize its cumulative reward over time by selecting actions that will likely yield higher rewards. The term "bandit" comes from the idea of a gambler facing multiple slot machines (bandits) with unknown payout probabilities and trying to decide which machine to play to maximize the winnings. There are different types of bandit feedback, such as:
1. Partial Information Feedback: The rewards of some actions are observed. The agent receives feedback only about the reward of the chosen action, not about unchosen actions. This is realistic and challenging as the agent must explore different actions to learn their rewards while still exploiting actions that appear rewarding.
2. Full Information Feedback: The rewards of all actions are observed. The agent receives complete information about the rewards associated with all available actions after each action is taken. This way, the agent directly learns about the most rewarding actions and updates its strategy accordingly.
3. Contextual Bandit Feedback: The agent receives additional context about each decision point. This can help it make better-informed decisions. The context can be thought of as features that describe the current situation or environment.
4. Adversarial Bandit Feedback: There may be adversaries trying to manipulate the agent's decisions by providing misleading feedback. The agent needs to adapt its strategy to take care of potentially biased or adversarial feedback.
Bandit feedback is a crucial aspect of various applications, including online advertising, recommendation systems, clinical trials, and optimization problems. Researchers and practitioners have developed algorithms and strategies to address bandit problems and make effective decisions while balancing exploration (learning) and exploitation (maximizing rewards) based on the available feedback. For instance, multi-armed bandit problems (MABPs) are well-suited to model resource allocation under uncertainty in clinical trials.[3] [4]
Contextual bandit methods are used to select articles for recommendation to a user and continuously modify their strategies to present diverse news recommendations.[5]
A contextual-bandit-based approach that incorporates patient characteristics is used to refine treatment selection in clinical trials.[6]
Bandit solutions provide unified ethical models for randomized clinical trials and comparative effectiveness research.[7]
References
- ↑ https://openreview.net/pdf?id=SJaP_-xAb
- ↑ "ORF523: Optimization with bandit feedback | I'm a bandit". 9 August 2020. Archived from the original on 2020-08-09.
- ↑ Villar, S. S.; Bowden, J.; Wason, J. (2015). "Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges". Statistical Science : A Review Journal of the Institute of Mathematical Statistics. 30 (2): 199–215. doi:10.1214/14-STS504. PMC 4856206. PMID 27158186.
- ↑ Sklar, Michael; Shih, Mei-Chiung; Lavori, Philip (2021). "Bandit Theory". Statistica Sinica. 31: 2289–2307. JSTOR 27089276.
- ↑ Semenov, Alexander; Rysz, Maciej; Pandey, Gaurav; Xu, Guanglin (2022). "Diversity in news recommendations using contextual bandits". Expert Systems with Applications. 195: 116478. doi:10.1016/j.eswa.2021.116478. Unknown parameter
|s2cid=ignored (help) - ↑ Varatharajah, Y.; Berry, B. (2022). "A Contextual-Bandit-Based Approach for Informed Decision-Making in Clinical Trials". Life. 12 (8): 1277. Bibcode:2022Life...12.1277V. doi:10.3390/life12081277. PMC 9410371 Check
|pmc=value (help). PMID 36013456 Check|pmid=value (help). - ↑ https://www.researchgate.net/publication/40696075_Bandit_solutions_provide_unified_ethical_models_for_randomized_clinical_trials_and_comparative_effectiveness_research
This article "Bandit feedback" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Bandit feedback. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
