Computer Design Concepts for Machine Learning

Editor's Notes

• #2: Abstract:
• #7: The most fundamental unit of neural network is an artificial neuron. It tries to mimic the actual brain functionality. In real neuron, Dendrites take input signals and process them. Then pass the output to another neuron. In mathematic model, artificial neural network takes the input, calculate the weighted sum and pass it through a nonlinear function and then gives an output. The output then can be the input for next artificial neuron.
• #15: Mitchel’s vs Floating Right shows the outputs of the final fully connected layer from the sample inference. Absolute magnitudes of the scores are reduced because Mitchell’s multiplication has negative error. But the prediction is still accurate because the relative order is preserved. The effect is as if the inference was performed with a lighter image. Same effects are observed with ImageNet, only more difficult because the 1000 outputs were much closer together. Same effects are observed with Mitch-w designs.
• #18: Telum chip design is specifically optimized to run these kinds of mission critical, heavy duty transaction processing and batch processing workloads, while ensuring top-notch security and availability. The goal for this processor and the AI accelerator was to enable embedding AI with super low latency directly into the transactions without needing to send the data off-platform, which brings all sorts of latency inconsistencies and security concerns," said Jacobi. "Sending data over a network, oftentimes personal and private data, requires cryptography and auditing of security standards that creates a lot of complexity in an enterprise environment. We have designed this accelerator to operate directly on the data using virtual address mechanisms and the data protection mechanisms that naturally apply to any other thing on the IBM Z processor.“ Each Telum processor contains eight processor cores and use a deep super-scalar, out-of-order instruction pipeline. Running with more than 5GHz clock frequency, the redesigned cache and chip-interconnection infrastructure provides 32MB cache per core and can scale to 32 Telum chips. The dual-chip module design contains 22 billion transistors and 19 miles of wire on 17 metal layers.
• #19: Four years after the introduction of Loihi, Intel’s first neuromorphic chip, the company is introducing its successor. According to Intel, the second-generation chip will provide faster processing, higher resource density and greater energy efficiency. Intel is also introducing Lava, a software framework for neuromorphic computing. Various processors and pieces of code are often compared to brains, but neuromorphic chips work to much more directly mimic neurological systems through the use of computational “neurons” that communicate with one another. Intel’s first-generation Loihi chip, introduced in 2017, has around 128,000 of those digital neurons. Over the ensuing four years, Loihi has been packed into increasingly large systems, learned to touch and even been taught to smell. However, Loihi 2 is still in its early days. Initially, Intel is offering two Loihi 2-based systems through its neuromorphic research cloud to select members of its Intel Neuromorphic Research Community (INRC). Those systems are Oheo Gulch, which uses a single Loihi 2 chip on an Intel Arria 10 FPGA and is intended for early evaluation, and Kapoho Point, which offers eight Loihi 2 chips in a stackable, “approximately 4×4-inch form factor with an Ethernet interface … ideal for portable projects” The key difference between a traditional ANN and SNN (spiking) is the information propagation approach. SNN tries to more closely mimic a biological neural network. This is why instead of working with continuously changing in time values used in ANN, SNN operates with discrete events that occur at certain points of time. SNN receives a series of spikes as input and produces a series of spikes as the output (a series of spikes is usually referred to as spike trains).
• #20: Intel 7 is rebranded 10 nm Enhanced SuperFin process.
• #21: AMX consists of two components: a set of 2-dimensional registers (tiles) representing sub-arrays from a larger 2-dimensional memory image, and an accelerator able to operate on tiles, the first implementation is called TMUL (tile matrix multiply unit).
• #22: The intel demo uses an internally optimized general matrix multiply gem kernel with and without amx It is recommended that system software initialize AMX state (e.g., by executing TILERELEASE) before disabling amx. This is because maintaining AMX state in a non-initialized state may have negative power and performance implications.
• #23: An Intel architecture host drives the algorithm, the memory blocking, loop indices and pointer arithmetic. Tile loads and stores and accelerator commands are sent to multi-cycle execution units. Status, if required, is reported back. Intel AMX instructions are synchronous in the Intel architecture instruction stream and the memory loaded and stored by the tile instructions is coherent with respect to the host’s memory accesses. There are no restrictions on interleaving of Intel architecture and Intel AMX code or restrictions on the resources the host can use in parallel with Intel AMX (e.g., Intel AVX-512).
• #24: Another form of neural network is convolutional neural network which is mostly used for image recognition, classification and object detection. This network consists of multiple conv-layers and fully-connected layers. Inside the conv-layers there is a form of computation called convolution which passes multiple filters across the input image through sliding window method to extract important features of input image. The convolution accounts for more than 90% of overall time and energy consumption. (CNN is also known as CONV net.)
• #25: In the left picture, you can see 2dimensional convolution. It passes a filter across the input image, performs element-wise matrix multiplications and then sum all the results and then we have an output activation. In next step it moves filter to another part of the image and preforms the same process for entire input image. In right image you can see high-dimensional convolution which applies multiple filters on multiple input image. Input feature map(input Fmap) Output feature map(output Fmap)
• #26: Here we are showing the sliding window process and also, we are emphasizing that this process is nothing only some MAC operations. The input is a 4*4 matrix, and the filter is 3*3. We perform the convolution and obtain the output matrix. Output height = (Input height + padding height top + padding height bottom - kernel height) / (stride height) + 1 (4 + 2*0 -3)/1 + 1 = 2 2 by 2 The example below is taken from the lectures in deeplearning.ai shows that the result is the sum of the element-by-element product (or "element-wise multiplication". The red numbers represent the weights in the filter: (1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4 enter image description here
• #27: There are two phases in neural network processing: training and inference. For training a CNN classifier, we have lots of known and labeled images, feed them to the CNN and train it , after that we have a predicted model which can classify a new input image.
• #28: In the inference phase we are using the trained model. We take the trained CNN, apply a new image (which was not in the training set) to the network and get some output prediction with different probabilities. If the value of P3 is more than other probabilities, our CNN could predict the target correctly. Typically, we train CNNs once on large GPU/FPGA cluster. In contrast, the inference is implemented each time a new data sample has to be classified. Therefore, the literature mostly focuses on accelerating the inference phase.
• #29: In this figure you can see the accuracy of different CNNs and their total MAC operations and parameters. By looking the amount of these operation and parameters we can get the idea about the huge workload that these CNNs are facing. For efficiency of these networks finding ways that accelerate the computation and reduce the energy consumption are important. Top5 means failing selecting correcting in its top 5 guesses
• #30: As you can see in above picture if we want to compare current DNNs hardware, we ‘ll have more efficiency from CPU toward ASIC while we’ll have less flexibility. CPUs have one or multiple cores and they are good for sequential processing. They can fetch a small amount of data quickly. In the other hand, there are lots of huge matrix operations in DNNs (big chunk of data and lots of MAC operation which need parallel processing) so CPUs don’t seem a good choice for DNNs.
• #31: GPUs contain hundreds of cores that can handle many threads simultaneously. Those threads execute the same instruction on different data elements. Basically those threads are executing in the SIMD manner.
• #33: Reducing the operating voltage to 0.75 V, the nominal voltage in 7 nm, would result in 164 W, still way too high. Just to get under 120 W on a single chip, we would have to reduce the voltage down to about 0.67 V, but then that one chip would use up the entire power budget. If we operate at the best energy-efficiency point (0.3 V), each chip will consume only 8.5 W, and six chips will fit in the 120-W budget with room to spare. With six chips the performance would be 2.5 times better than the 118-W single-chip solution, and the energy efficiency is 20 times better than trying to operate at the highest voltage.
• #34: If you want to use FPGA as hardware for DNNs, you must know how program a FPGA in addition to neural network knowledge. But for programming a GPU you need only neural network knowledge. GPUs are good at floating point computation and if we want to use other simpler data types, using GPUs is not efficient because of their high-power consumption. SIMD: Single Instruction and Multiple Data.
• #36: Differences from pipelining: These are individual PEs, they are not stages of instruction execution. (in pipelining each stage executes a piece of instruction, and each stage is not a processing element) Array structure can be nonlinear and multi-dimensional. It can be 2 dimensional or 3 dimensional. The connection between processing elements can be multidimensional. ( if you have a two-dimensional array of PEs, each PE can be connected to all its neighbors.) Each PE can have local memory.
• #45: On the TPU V2 board there are four TPU chips and within a single TPU chip, there are two TPU cores. They use High Bandwidth Memory because the memory bandwidth requirement of matrix multiplication is high. TPU can be used for training and inference. Each TPU core has scalar, vector and matrix units (MXU).
• #47: We discuss the memory access energy is high. Now we will see how we can utilize the architecture with memory hierarchy for energy efficient CNNs accelerators. The actual bottle neck is how we access data. For each multiply and accumulate unit, we have to read three pieces of data from memory and then we need to push the result back to the memory.
• #48: In the previous slide we saw the energy consumption of memory hierarchy and figured out that reading and writing on external memory (data movement toward DRAM) is expensive. So , we are looking to find ways to minimize DRAM access. In other words, every time that a piece of data is moved from an expensive level to a lower cost memory level, we want to reuse it as much as possible. Here, there are three ways to reuse the data in CNN computation.
• #49: For example: if we could take advantage of these data reuse opportunities, the DRAM accesses in AlexNet can be reduced from 2896M to 61M accesses. This will increase energy efficiency.
• #50: Large size of deep neural networks model is an issue because we can not fit them on devices such as cell phones. The second problem of those large models is that the training speed is extremely slow. The third problem of those large models is energy efficiency. Large models basically mean more memory references which result in more energy consumption.
• #51: (2.09, 2.12, 1.92, 1.87 )all the blue square values are in one group and their value is saved in index3. So we need to store the two bits index rather than the 32 bits floating point number. ( We need only two bits to represent each number) We can do both pruning and quantization on the model. It should be mentioned that the quantization can be fixed ( when the same method is used for all data types and layers, filters, and channels in the network) or it can be variable ( different methods of quantization can be used for weights and activations, and different layers, filters, and channels in the networks)
• #53: One way of Pruning is to prune the low-weight connections. All connections with weights below a threshold are removed from the network, converting a dense network into a sparse network. Then we should retrain the network to learn the final weights for the remaining sparse connections. This step is critical. If we use pruned network without retraining, accuracy is significantly impacted. Example: by pruning we could reduce the AlexNet parameters from 60 Million to 6 Million which means ten times less computations.
• #54: For example, the purple line shows the accuracy drops by 4.5 percent when we prune the 80percent of the parameters. But if we retrain the remaining weights the accuracy can be recovered. (which is shown in green line). If we continue training and retraining, we can prune away 90 percent of the parameters and still have the same accuracy.(red line in the figure).
• #61: Application of Resistive Random Access Memory in Hardware Security: A Review Gokulnath Rajendran, Writam Banerjee,* Anupam Chattopadhyay,* and Mohamed M. Sabry Aly