Signal Processing and Recognition Group, Universidad Nacional de Colombia, Manizales 170003, Colombia.
Transmedia Research Center, Universidad de Caldas, Manizales 170003, Colombia.
Sensors (Basel). 2023 Jun 14;23(12):5574. doi: 10.3390/s23125574.
Sound synthesis refers to the creation of original acoustic signals with broad applications in artistic innovation, such as music creation for games and videos. Nonetheless, machine learning architectures face numerous challenges when learning musical structures from arbitrary corpora. This issue involves adapting patterns borrowed from other contexts to a concrete composition objective. Using Labeled Correlation Alignment (LCA), we propose an approach to sonify neural responses to affective music-listening data, identifying the brain features that are most congruent with the simultaneously extracted auditory features. For dealing with inter/intra-subject variability, a combination of Phase Locking Value and Gaussian Functional Connectivity is employed. The proposed two-step LCA approach embraces a separate coupling stage of input features to a set of emotion label sets using Centered Kernel Alignment. This step is followed by canonical correlation analysis to select multimodal representations with higher relationships. LCA enables physiological explanation by adding a backward transformation to estimate the matching contribution of each extracted brain neural feature set. Correlation estimates and partition quality represent performance measures. The evaluation uses a Vector Quantized Variational AutoEncoder to create an acoustic envelope from the tested Affective Music-Listening database. Validation results demonstrate the ability of the developed LCA approach to generate low-level music based on neural activity elicited by emotions while maintaining the ability to distinguish between the acoustic outputs.
声音合成是指使用广泛的艺术创新，如为游戏和视频创作音乐，来创建原始声学信号。然而，机器学习架构在从任意语料库学习音乐结构时面临着许多挑战。这个问题涉及到将从其他上下文借用的模式适用于具体的作曲目标。我们使用有标签的相关对齐（LCA），提出了一种将神经反应声化为情感音乐聆听数据的方法，确定与同时提取的听觉特征最一致的大脑特征。为了处理个体内/个体间的可变性，我们使用相位锁定值和高斯功能连接的组合。所提出的两步 LCA 方法采用了单独的耦合阶段，即将输入特征耦合到一组使用中心核对准的情感标签集。然后使用典型相关分析选择具有更高关系的多模态表示。LCA 通过添加向后转换来估计每个提取的大脑神经特征集的匹配贡献，从而实现生理解释。相关估计和分区质量表示性能度量。评估使用矢量量化变分自动编码器从测试的情感音乐聆听数据库中创建声谱包络。验证结果表明，所开发的 LCA 方法能够根据情感引起的神经活动生成低级音乐，同时保持区分声学输出的能力。
