Maryland/DC Chapter, The Nature Conservancy, Bethesda, Maryland, USA.
Department of Bioscience, Aarhus University, Ronde, Denmark.
Integr Environ Assess Manag. 2020 Nov;16(6):841-852. doi: 10.1002/ieam.4302. Epub 2020 Aug 14.
Nonpoint source water quality management is challenged with allocating uncertain management actions and monitoring their performance in the absence of state-dependent decision making. This adaptive management context can be expressed as a multiarmed bandit problem. Multiarmed bandit strategies attempt to balance the exploitation of actions that appear to maximize performance with the exploration of uncertain, but potentially better, actions. We performed a test of multiarmed bandit strategies to inform adaptive water quality management in Massachusetts, USA. Conservation and restoration practitioners were tasked with allocating household wastewater treatments to minimize N inputs to impaired waters. We obtained time series of N monitoring data from 3 wastewater treatment types and organized them chronologically and randomly. The chronological data set represented nonstationary performance based on recent monitoring data, whereas the random data set represented stationary performance. We tested 2 multiarmed bandit strategies in hypothetical experiments to sample from the treatment data through 20 sequential decisions. A deterministic probability-matching strategy allocated treatments with the highest probability of success regarding their performance at each decision. A randomized probability-matching strategy randomly allocated treatments according to their probability of success at each decision. The strategies were compared with a nonadaptive strategy that equally allocated treatments at each decision. Results indicated that equal allocation is useful for learning in nonstationary situations but tended to overexplore inferior treatments and thus did not maximize performance when compared with the other strategies. Deterministic probability matching maximized performance in many stationary situations, but the strategy did not adequately explore treatments and converged on inferior treatments in nonstationary situations. Randomized probability matching balanced performance and learning in stationary situations, but the strategy could converge on inferior treatments in nonstationary situations. These findings provide evidence that probability-matching strategies are useful for adaptive management. Integr Environ Assess Manag 2020;16:841-852. © 2020 The Authors. Integrated Environmental Assessment and Management published by Wiley Periodicals LLC on behalf of Society of Environmental Toxicology & Chemistry (SETAC).
非点源水质管理面临的挑战是在没有依赖状态的决策制定的情况下，分配不确定的管理措施并监测其性能。这种自适应管理环境可以表示为多臂赌博问题。多臂赌博策略试图在利用似乎可以最大限度提高性能的行动与探索不确定但可能更好的行动之间取得平衡。我们在美国马萨诸塞州进行了多臂赌博策略测试，以告知适应性水质管理。保护和恢复实践人员的任务是分配家庭废水处理，以最大限度地减少对受损水域的 N 输入。我们从 3 种废水处理类型中获得了 N 监测数据的时间序列，并按时间顺序和随机组织了这些数据。时间序列数据集代表基于最近监测数据的非平稳性能，而随机数据集代表平稳性能。我们在假设实验中测试了 2 种多臂赌博策略，通过 20 个连续决策从处理数据中进行采样。确定性概率匹配策略根据每个决策的性能，以最高成功概率分配处理方法。随机概率匹配策略根据每个决策的成功概率随机分配处理方法。这些策略与每个决策均等分配处理方法的非适应性策略进行了比较。结果表明，在非平稳情况下，均等分配对于学习很有用，但与其他策略相比，均等分配往往会过度探索劣等处理方法，从而无法最大限度地提高性能。确定性概率匹配在许多平稳情况下最大化了性能，但该策略没有充分探索处理方法，在非平稳情况下会收敛到劣等处理方法。随机概率匹配在平稳情况下平衡了性能和学习，但在非平稳情况下，该策略可能会收敛到劣等处理方法。这些发现为概率匹配策略在自适应管理中的有效性提供了证据。
