Department of Mathematics and Computing Science, Saint Mary's University, Halifax, NS, Canada.
Canadian Centre on Substance Use and Addiction, Ottawa, ON, Canada.
J Med Internet Res. 2023 Sep 19;25:e43630. doi: 10.2196/43630.
A hallmark of unregulated drug markets is their unpredictability and constant evolution with newly introduced substances. People who use drugs and the public health workforce are often unaware of the appearance of new drugs on the unregulated market and their type, safe dosage, and potential adverse effects. This increases risks to people who use drugs, including the risk of unknown consumption and unintentional drug poisoning. Early warning systems (EWSs) can help monitor the landscape of emerging drugs in a given community by collecting and tracking up-to-date information and determining trends. However, there are currently few ways to systematically monitor the appearance and harms of new drugs on the unregulated market in Canada.
The goal of this work is to examine how artificial intelligence can assist in identifying patterns of drug-related risks and harms, by monitoring the social media activity of public health and law enforcement groups. This information is beneficial in the form of an EWS as it can be used to identify new and emerging drug trends in various communities.
To collect data for this study, 145 relevant Twitter accounts throughout Quebec (n=33), Ontario (n=78), and British Columbia (n=34) were manually identified. Tweets posted between August 23 and December 21, 2021, were collected via the application programming interface developed by Twitter for a total of 40,393 tweets. Next, subject matter experts (1) developed keyword filters that reduced the data set to 3746 tweets and (2) manually identified relevant tweets for monitoring and early warning efforts for a total of 464 tweets. Using this information, a zero-shot classifier was applied to tweets from step 1 with a set of keep (drug arrest, drug discovery, and drug report) and not-keep (drug addiction support, public safety report, and others) labels to see how accurately it could extract the tweets identified in step 2.
When looking at the accuracy in identifying relevant posts, the system extracted a total of 584 tweets and had an overlap of 392 out of 477 (specificity of ~84.5%) with the subject matter experts. Conversely, the system identified a total of 3162 irrelevant tweets and had an overlap of 3090 (sensitivity of ~94.1%) with the subject matter experts.
This study demonstrates the benefits of using artificial intelligence to assist in finding relevant tweets for an EWS. The results showed that it can be quite accurate in filtering out irrelevant information, which greatly reduces the amount of manual work required. Although the accuracy in retaining relevant information was observed to be lower, an analysis showed that the label definitions can impact the results significantly and would therefore be suitable for future work to refine. Nonetheless, the performance is promising and demonstrates the usefulness of artificial intelligence in this domain.
不受监管的毒品市场的一个特点是其不可预测性和不断演变，新引入的物质层出不穷。吸毒者和公共卫生工作人员通常不知道不受监管的市场上出现了哪些新毒品及其类型、安全剂量和潜在的副作用。这增加了吸毒者的风险，包括未知消费和意外药物中毒的风险。早期预警系统 (EWS) 可以通过收集和跟踪最新信息并确定趋势来帮助监测特定社区中新兴毒品的情况。然而，目前加拿大几乎没有系统地监测不受监管市场上新出现的毒品及其危害的方法。
这项工作的目标是通过监测公共卫生和执法机构的社交媒体活动，研究人工智能如何帮助识别与药物相关的风险和危害模式。由于可以将这些信息用作 EWS，因此可以识别不同社区中新出现的毒品趋势，因此这些信息非常有用。
为了进行这项研究，我们手动确定了魁北克省（n=33）、安大略省（n=78）和不列颠哥伦比亚省（n=34）共 145 个相关的 Twitter 账户。2021 年 8 月 23 日至 12 月 21 日期间，通过 Twitter 为这项研究开发的应用程序接口收集了 40,393 条推文。接下来，主题专家 (1) 开发了关键词过滤器，将数据集减少到 3746 条推文，(2) 手动识别了 464 条与监测和早期预警工作相关的推文。使用这些信息，对步骤 1 中的推文应用了一个零镜头分类器，该分类器设置了一组保留（药物逮捕、药物发现和药物报告）和不保留（药物成瘾支持、公共安全报告和其他）标签，以查看它可以多准确地提取步骤 2 中识别的推文。
在观察识别相关帖子的准确性时，该系统共提取了 584 条推文，与主题专家的重叠为 392 条（特异性约为 84.5%）。相反，该系统共识别了 3162 条不相关的推文，与主题专家的重叠为 3090 条（敏感性约为 94.1%）。
这项研究表明，使用人工智能来协助寻找 EWS 的相关推文具有一定的益处。结果表明，它在过滤掉不相关的信息方面非常准确，这大大减少了所需的人工工作量。尽管保留相关信息的准确性观察到较低，但分析表明标签定义会对结果产生重大影响，因此适合未来的工作来进行改进。尽管性能很有前景，但仍表明人工智能在这一领域的有用性。
