Knowledge extraction from neural networks using the all-permutations fuzzy rule base: the LED display recognition problem.

A major drawback of artificial neural networks (ANNs) is their black-box character. Even when the trained network performs adequately, it is very difficult to understand its operation. In this letter, we use the mathematical equivalence between ANNs and a specific fuzzy rule base to extract the knowledge embedded in the network. We demonstrate this using a benchmark problem: the recognition of digits produced by a light emitting diode (LED) device. The method provides a symbolic and comprehensible description of the knowledge learned by the network during its training.