Implementation of a Framework for Deploying AI Inference Engines in FPGAs

Abstract

The LCLS2 Free Electron Laser (FEL) will generate x-ray pulses to beamline experiments at up to 1 Mhz. These experimentals will require new ultra-high rate (UHR) detectors that can operate at rates above 100 kHz and generate data throughputs upwards of 1 TB/s, a data velocity which requires prohibitively large investments in storage infrastructure. Machine Learning has demonstrated the potential to digest large datasets to extract relevant insights, however current implementations show latencies that are too high for real-time data reduction objectives. SLAC has endeavored on the creation of a software framework which translates MLs structures for deployment on Field Programmable Gate Arrays (FPGAs) deployed at the Edge of the data chain, close to the instrumentation. This framework leverages Xilinx’s HLS framework presenting an API modeled after the open source Keras interface to the TensorFlow library. This SLAC Neural Network Library (SNL) framework is designed with a streaming data approach, optimizing the data flow between layers, while minimizing the buffer data buffering requirements. The goal is to ensure the highest possible framerate while keeping the maximum latency constrained to the needs of the experiment. Our framework is designed to ensure the RTL implementation of the network layers supporting full re-deployment of weights and biases without requiring re-synthesis after training. The ability to reduce the precision of the implemented networks through quantization is necessary to optimize the use of both DSP and memory resources in the FPGA. We currently have a preliminary version of the toolset and are experimenting with both general purpose example networks and networks being designed for specific LCLS2 experiments.