0% found this document useful (0 votes)

61 views66 pages

L03 Perceptron Slides

Uploaded by

mark overmars

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

61 views66 pages

L03 Perceptron Slides

Uploaded by

mark overmars

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 66

STAT 453: Introduction to Deep Learning and Generative Models

Sebastian Raschka
http://stat.wisc.edu/~sraschka/teaching

Lecture 03

The Perceptron
with Applications in Python

An Introduction to Single Layer Neural Networks

Sebastian Raschka STAT 453: Intro to Deep Learning 1

Announcements

• Project groups (by next Thu), 3 members per group -- TA will set
up a document where you can add your team member
preferences

• Project topics (brainstorm with group members)

• HW1 (related to the Perceptron; more about that later)

• Piazza for questions, encouraged to help each other

(but don't share your HW solutions)

Sebastian Raschka STAT 453: Intro to Deep Learning 2

After this lecture, you will be able to
implement your rst neuron model for
making predictions!

Sebastian Raschka STAT 453: Intro to Deep Learning 3

fi
Lecture Overview
1. Brains and neuron models
2. The perceptron learning rule
3. Interlude: "vectorization" in Python
4. Implementing a perceptron in Python
using NumPy and PyTorch
5. Optional: The perceptron convergence theorem
6. Geometric intuition

Sebastian Raschka STAT 453: Intro to Deep Learning 4

Inspired by Biological Brains and Neurons

https://publicdomainpictures.net/en/view-image.php?image=130359&picture=human-brain https://commons.wikimedia.org/wiki/Neuron#/media/File:Mouse_cingulate_cortex_neurons.jpg https://commons.wikimedia.org/wiki/Neuron#/media/File:Pyramidal_hippocampal_neuron_40x.jpg

Do our brains use deep learning?

Sebastian Raschka STAT 453: Intro to Deep Learning 5

Source: https://media.mnn.com/assets/images/2016/10/plane-birds.jpg.1000x0_q80_crop-smart.jpg

Sebastian Raschka STAT 453: Intro to Deep Learning 6

Number of neurons in brains ...

sixteen billion three hundred forty million

On a sidenote:

Source: https://en.wikipedia.org/wiki/
List_of_animals_by_number_of_neurons

Sebastian Raschka STAT 453: Intro to Deep Learning 7

A Biological Neuron

Sebastian Raschka STAT 453: Intro to Deep Learning 8

McCulloch & Pitts Neuron Model

1943

x1
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

w1
<latexit sha1_base64="ozSIzVA/SGXegmac4XRXthOpvw0=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwosjZ8=</latexit>

x2 weighted threshold binary signal

<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

sum
..
.
wm
<latexit sha1_base64="3SltFZgdSbEccduFdJMJ4sVJM+s=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMquUpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2Ucjds=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 9

Logical AND Gate

x1 x2 Out
x1
0 0 0 w1 =1
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

<latexit sha1_base64="ozSIzVA/SGXegmac4XRXthOpvw0=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwosjZ8=</latexit>

0 1 0 weighted
t=1.5
sum
1 0 0 x2 w2 =1
<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit>

1 1 1

Sebastian Raschka STAT 453: Intro to Deep Learning 10

Logical OR Gate

0 1 1 weighted
t=0.5
sum
1 0 1 x2 w2 =1
<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit>

1 1 1

Sebastian Raschka STAT 453: Intro to Deep Learning 11

Logical NOT Gate

x1 Out
0 1
x1 w1 = -1 weighted
t= -0.5
<latexit sha1_base64="ozSIzVA/SGXegmac4XRXthOpvw0=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwosjZ8=</latexit>

sum
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

1 0

Sebastian Raschka STAT 453: Intro to Deep Learning 12

Logical XOR Gate
(Take-home exercise)

x1 x2 Out
x1
0 0 0 w1 =?
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

0 1 1 weighted t=?
sum
1 0 1 x2 w2 =?
<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit>

1 1 0

Sebastian Raschka STAT 453: Intro to Deep Learning 13

Training Single-Layer Neural
Networks

1. Brains and neuron models

2. The perceptron learning rule
3. Interlude: "vectorization" in Python
4. Implementing a perceptron in Python using NumPy and PyTorch
5. Optional: The perceptron convergence theorem
6. Geometric intuition

Sebastian Raschka STAT 453: Intro to Deep Learning 14

Rosenblatt's Perceptron
A learning rule for the computational/mathematical neuron model
Rosenblatt, F. (1957). The perceptron, a perceiving and recognizing
automaton. Project Para. Cornell Aeronautical Laboratory.

Source: http://www.enzyklopaedie-der-wirtschaftsinformatik.de/wi-enzyklopaedie/Members/wilex4/Rosen-2.jpg

Sebastian Raschka STAT 453: Intro to Deep Learning 15

Perceptron Variants

Note that Rosenblatt (and later others) proposed many

variants of the Perceptron model and learning rule.
We discuss a "basic" version;
let's say,

"Perceptron" := "a classic Rosenblatt Perceptron"

Sebastian Raschka STAT 453: Intro to Deep Learning 16

A Computational Model of a Biological Neuron

((
x1
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

00,if z< ✓✓
Threshold ff(z)
(z)==
11,if z > ✓✓
<latexit sha1_base64="HmStRmxc+7HHgB2knpGSNB9+Q5Y=">AAAB+XicbZBNS8NAEIY39avWr6hHL8EieCqJCHosevFYoV/QhrLZTNqlm2zYnRRL6D/x4kERr/4Tb/4bt20O2vrCwsM7M8zsG6SCa3Tdb6u0sbm1vVPereztHxwe2ccnbS0zxaDFpJCqG1ANgifQQo4CuqkCGgcCOsH4fl7vTEBpLpMmTlPwYzpMeMQZRWMNbLuP8IR5c6RAj6QIZwO76tbchZx18AqokkKNgf3VDyXLYkiQCap1z3NT9HOqkDMBs0o/05BSNqZD6BlMaAzazxeXz5wL44ROJJV5CToL9/dETmOtp3FgOmOKI71am5v/1XoZRrd+zpM0Q0jYclGUCQelM4/BCbkChmJqgDLFza0OG1FFGZqwKiYEb/XL69C+qnmGH6+r9bsijjI5I+fkknjkhtTJA2mQFmFkQp7JK3mzcuvFerc+lq0lq5g5JX9kff4ASZeUEQ==</latexit>

X <latexit sha1_base64="Rc8P2/bwG4VpYHBZl3KrPUvxez8=">AAACU3icbVHBahsxENVu0tRx69RNjrkMMQX3YnZLoDm0YJpLjy7EScBrjFaetdVotRtpttRe/I8l0EN+JJccGtneQmP3geDx5s2M9BTnSloKgnvP39l9sfeytl9/9bpx8Kb59vDSZoUR2BeZysx1zC0qqbFPkhRe5wZ5Giu8im/Ol/WrH2iszPQFzXIcpnyiZSIFJyeNmt+T9vw9fIYoxonUpXCj7ALqsEIAEeFPKkEmsACYwycnTJE4RFFlCTcs0QRv/5qWlgj1uJo6araCTrACbJOwIi1WoTdq3kXjTBQpahKKWzsIg5yGJTckhcJFPSos5lzc8AkOHNU8RTssV5ks4J1TxpBkxh1NsFL/7Sh5au0sjZ0z5TS1m7Wl+L/aoKDkbFhKnReEWqwXJYUCymAZMIylQUFq5ggXRrq7gphywwW5b6i7EMLNJ2+Tyw+d0PFvp63ulyqOGjtmJ6zNQvaRddlX1mN9Jtgv9sD+eMz77T36vr+7tvpe1XPEnsFvPAHoia0X</latexit>

<latexit sha1_base64="NZ0OX2X7eZqKxseSRl9vLWZDRqs=">AAACM3icbVDLSgMxFM34dnxVXboJFkFBZEYEBVFEN+JKwarQlJJJ77TBTGZI7gh16D+58UdcCOJCEbf+g2mdha8DgcM59yS5J8qUtBgET97Q8Mjo2PjEpD81PTM7V5lfuLBpbgTURKpScxVxC0pqqKFEBVeZAZ5ECi6j66O+f3kDxspUn2M3g0bC21rGUnB0UrNyEq/ertE9yiJoS10Id5Xt+cE6Zbv0ljIFlGEHkFPG/LBU979pDHSrTDUr1WAjGID+JWFJqqTEabPywFqpyBPQKBS3th4GGTYKblAKBT2f5RYyLq55G+qOap6AbRSDnXt0xSktGqfGHY10oH5PFDyxtptEbjLh2LG/vb74n1fPMd5pFFJnOYIWXw/FuaKY0n6BtCUNCFRdR7gw0v2Vig43XKCr2XclhL9X/ksuNjdCx8+2qgeHZR0TZIksk1USkm1yQI7JKakRQe7II3khr9699+y9ee9fo0NemVkkP+B9fAJspac3</latexit>

x2
<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

w2
1
<latexit sha1_base64="qp47xzdgavbzDAujuQrfWObrWms=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY3KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz96n4y1</latexit>
<latexit

0 <latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit>
<latexit

✓
<latexit sha1_base64="+2a+YdAkK5gXAKPWKNIZ/6djIOk=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCvYD2qVk07SNzSZLMiuUpf/BiwdFvPp/vPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wtr6xuVXcLu3s7u0flA+PmlanhvEG01KbdkQtl0LxBgqUvJ0YTuNI8lY0vp3VW0/cWKHVA04SHsZ0qMRAMIrOanZxxJH2yhW/6s9FViHIoQK56r3yV7evWRpzhUxSazuBn2CYUYOCST4tdVPLE8rGdMg7DhWNuQ2z+bZTcuacPhlo455CMnd/T2Q0tnYSR64zpjiyy7WZ+V+tk+LgOsyESlLkii0+GqSSoCaz00lfGM5QThxQZoTblbARNZShC6jkQgiWT16F5kU1cHx/Wand5HEU4QRO4RwCuIIa3EEdGsDgEZ7hFd487b14797HorXg5TPH8Efe5w+j+Y8o</latexit>
<latexit sha1_base64="+pIqdP8UU8+aMpm9ZD6XcVz9as4=">AAAB7XicbZBNSwMxEIazftb6VfXoJVgET2XXix6LXjxWsB/QLiWbzrax2WRJZoWy9D948aCIV/+PN/+NabsHbX0h8PDODJl5o1QKi77/7a2tb2xubZd2yrt7+weHlaPjltWZ4dDkWmrTiZgFKRQ0UaCETmqAJZGEdjS+ndXbT2Cs0OoBJymECRsqEQvO0FmtHo4AWb9S9Wv+XHQVggKqpFCjX/nqDTTPElDIJbO2G/gphjkzKLiEabmXWUgZH7MhdB0qloAN8/m2U3runAGNtXFPIZ27vydyllg7SSLXmTAc2eXazPyv1s0wvg5zodIMQfHFR3EmKWo6O50OhAGOcuKAcSPcrpSPmGEcXUBlF0KwfPIqtC5rgeN7v1q/KeIokVNyRi5IQK5IndyRBmkSTh7JM3klb572Xrx372PRuuYVMyfkj7zPH6K5jyQ=</latexit>
sha1_base64="9/iGzq3A97j0rqBi7BEikv8Mxvo=">AAAB4nicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6gHUomvdPGZjJDckcoQ/+DGxeK+KPc+W9MHwttPRD4OCch954oU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecZjZWpfqRphmHCR1rGUnByVrtPYyQ+qNX9hr8Q24RgBXVYqTmoffWHqcgT1CQUt7YX+BmFBTckhcJZpZ9bzLiY8BH2HGqeoA2LxbQzduGcIYtT444mtnB/vyh4Yu00idzNhNPYrmdz87+sl1N8ExZSZzmhFsuP4lwxStl8dTaUBgWpqQMujHSzMjHmhgtyBVVcCcH6ypvQvmoEjh98KMMZnMMlBHANt3APTWiBgCd4gTd491Lv1ftY1lXyVr2dwh95nz94VY3Q</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
ŷ
<latexit sha1_base64="Vi95YwknFrFzcB5LyqgiYSoMf0U=">AAAB7nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBfsBbSib7aZdutmE3YkQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3iCRwqDrfjuljc2t7Z3ybmVv/+DwqHp80jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTO/m9e4T10bE6hGzhPsRHSsRCkbRWt3BhGKezYbVmlt3FyLr4BVQg0KtYfVrMIpZGnGFTFJj+p6boJ9TjYJJPqsMUsMTyqZ0zPsWFY248fPFujNyYZ0RCWNtn0KycH9P5DQyJosC2xlRnJjV2tz8r9ZPMbzxc6GSFLliy4/CVBKMyfx2MhKaM5SZBcq0sLsSNqGaMrQJVWwI3urJ69C5qnuWH65rzdsijjKcwTlcggcNaMI9tKANDKbwDK/w5iTOi/PufCxbS04xcwp/5Hz+ALHDj8o=</latexit>

<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit> <latexit sha1_base64="0Hc81E1zKREVtkdUaco0RSs+Ymk=">AAAB63icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCrYW2qVk02wbmmSXZFYoS/+CFw+KePUPefPfmLZ70NYXAg/vzJCZN0qlsOj7315pbX1jc6u8XdnZ3ds/qB4etW2SGcZbLJGJ6UTUcik0b6FAyTup4VRFkj9G49tZ/fGJGysS/YCTlIeKDrWIBaM4s3o2U/1qza/7c5FVCAqoQaFmv/rVGyQsU1wjk9TabuCnGObUoGCSTyu9zPKUsjEd8q5DTRW3YT7fdUrOnDMgcWLc00jm7u+JnCprJypynYriyC7XZuZ/tW6G8XWYC51myDVbfBRnkmBCZoeTgTCcoZw4oMwItythI2ooQxdPxYUQLJ+8Cu2LeuD4/rLWuCniKMMJnMI5BHAFDbiDJrSAwQie4RXePOW9eO/ex6K15BUzx/BH3ucPMe2OUw==</latexit>

Output
.. <latexit sha1_base64="gvQE9cb1Dja5lCiXX5pMwfJapj8=">AAAB9HicbZBNS8NAEIY3ftb6VfXoZbEInkoigh6LXrxZwX5AG8pmO2mXbjZxd1Isob/DiwdFvPpjvPlv3LY5aOsLCw/vzLAzb5BIYdB1v52V1bX1jc3CVnF7Z3dvv3Rw2DBxqjnUeSxj3QqYASkU1FGghFaigUWBhGYwvJnWmyPQRsTqAccJ+BHrKxEKztBafgfhCbO7FJMUJ91S2a24M9Fl8HIok1y1bumr04t5GoFCLpkxbc9N0M+YRsElTIqd1EDC+JD1oW1RsQiMn82WntBT6/RoGGv7FNKZ+3siY5Ex4yiwnRHDgVmsTc3/au0Uwys/E8qeBIrPPwpTSTGm0wRoT2jgKMcWGNfC7kr5gGnG0eZUtCF4iycvQ+O84lm+vyhXr/M4CuSYnJAz4pFLUiW3pEbqhJNH8kxeyZszcl6cd+dj3rri5DNH5I+czx+i4ZKm</latexit>

. Net input
<latexit sha1_base64="vnW9SOTDG2wSeqwpvMYjb0pYOfc=">AAAB+XicbZDLSsNAFIYn9VbrLerSzWARXJVEBF0W3biSCvYCbSiT6Uk7dHJh5qRYQt/EjQtF3Pom7nwbp2kW2vrDwMd/zmHO+f1ECo2O822V1tY3NrfK25Wd3b39A/vwqKXjVHFo8ljGquMzDVJE0ESBEjqJAhb6Etr++HZeb09AaRFHjzhNwAvZMBKB4AyN1bftHsITZveAVERJirO+XXVqTi66Cm4BVVKo0be/eoOYpyFEyCXTuus6CXoZUyi4hFmll2pIGB+zIXQNRiwE7WX55jN6ZpwBDWJlXoQ0d39PZCzUehr6pjNkONLLtbn5X62bYnDtZflJEPHFR0EqKcZ0HgMdCAUc5dQA40qYXSkfMcU4mrAqJgR3+eRVaF3UXMMPl9X6TRFHmZyQU3JOXHJF6uSONEiTcDIhz+SVvFmZ9WK9Wx+L1pJVzByTP7I+fwD13ZPb</latexit>

wm
<latexit sha1_base64="3SltFZgdSbEccduFdJMJ4sVJM+s=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMquUpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2Ucjds=</latexit>

xm m
!
<latexit sha1_base64="UZ/Cq01CQU77ibJgEHsrgiYApIY=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMiuWpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2aijdw=</latexit>

X
f✓ xi w i = ŷ
Inputs
<latexit sha1_base64="kW2ZbIA+FSvwKPaKbZPllX8WYNo=">AAAB9HicbZBNS8NAEIYnftb6VfXoZbEInkoigh6LXvRWwX5AG8pmu2mXbjZxd1Isob/DiwdFvPpjvPlv3LY5aOsLCw/vzLAzb5BIYdB1v52V1bX1jc3CVnF7Z3dvv3Rw2DBxqhmvs1jGuhVQw6VQvI4CJW8lmtMokLwZDG+m9eaIayNi9YDjhPsR7SsRCkbRWn4H+RNmdypJ0Uy6pbJbcWciy+DlUIZctW7pq9OLWRpxhUxSY9qem6CfUY2CST4pdlLDE8qGtM/bFhWNuPGz2dITcmqdHgljbZ9CMnN/T2Q0MmYcBbYzojgwi7Wp+V+tnWJ45WdiehNXbP5RmEqCMZkmQHpCc4ZybIEyLeyuhA2opgxtTkUbgrd48jI0ziue5fuLcvU6j6MAx3ACZ+DBJVThFmpQBwaP8Ayv8OaMnBfn3fmYt644+cwR/JHz+QONXpKY</latexit> <latexit sha1_base64="s2pgoIN7KwQmP75hFuTCkPIYsRc=">AAACH3icbZDLSsNAFIYn9VbrrerSzWAR6qYkIupGkLpxWcFeoKlhMp20Q2eSMHOilpA3ceOruHGhiLjr2zi9LLT1wMDH/5/DmfP7seAabHtk5ZaWV1bX8uuFjc2t7Z3i7l5DR4mirE4jEamWTzQTPGR14CBYK1aMSF+wpj+4HvvNB6Y0j8I7GMasI0kv5AGnBIzkFc8CL3Whz4BkbpX3emVXJ9JL+aWT3Uv85HH86HEXT7xjfIndPoF0mHnFkl2xJ4UXwZlBCc2q5hW/3W5EE8lCoIJo3XbsGDopUcCpYFnBTTSLCR2QHmsbDIlkupNO7svwkVG6OIiUeSHgifp7IiVS66H0Tack0Nfz3lj8z2snEFx0Uh7GCbCQThcFicAQ4XFYuMsVoyCGBghV3PwV0z5RhIKJtGBCcOZPXoTGScUxfHtauqrO4sijA3SIyshB5+gK3aAaqiOKntErekcf1ov1Zn1aX9PWnDWb2Ud/yhr9ACNlol0=</latexit>
i=1

Sebastian Raschka STAT 453: Intro to Deep Learning 17

Terminology
General (logistic regression, multilayer nets, ...):
• Net input = weighted inputs, z
• Activations = activation function(net input); a = σ(z)
• Label output = threshold(activations of last layer); ŷ = f(a)

Special cases:
• In perceptron: activation function = threshold function
• In linear regression: activation = net input = output
(
(
x1
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

00,ifzz<✓✓
Threshold f (z) =
11,ifzz> ✓✓
<latexit sha1_base64="HmStRmxc+7HHgB2knpGSNB9+Q5Y=">AAAB+XicbZBNS8NAEIY39avWr6hHL8EieCqJCHosevFYoV/QhrLZTNqlm2zYnRRL6D/x4kERr/4Tb/4bt20O2vrCwsM7M8zsG6SCa3Tdb6u0sbm1vVPereztHxwe2ccnbS0zxaDFpJCqG1ANgifQQo4CuqkCGgcCOsH4fl7vTEBpLpMmTlPwYzpMeMQZRWMNbLuP8IR5c6RAj6QIZwO76tbchZx18AqokkKNgf3VDyXLYkiQCap1z3NT9HOqkDMBs0o/05BSNqZD6BlMaAzazxeXz5wL44ROJJV5CToL9/dETmOtp3FgOmOKI71am5v/1XoZRrd+zpM0Q0jYclGUCQelM4/BCbkChmJqgDLFza0OG1FFGZqwKiYEb/XL69C+qnmGH6+r9bsijjI5I+fkknjkhtTJA2mQFmFkQp7JK3mzcuvFerc+lq0lq5g5JX9kff4ASZeUEQ==</latexit>

<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit> <latexit sha1_base64="0Hc81E1zKREVtkdUaco0RSs+Ymk=">AAAB63icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCrYW2qVk02wbmmSXZFYoS/+CFw+KePUPefPfmLZ70NYXAg/vzJCZN0qlsOj7315pbX1jc6u8XdnZ3ds/qB4etW2SGcZbLJGJ6UTUcik0b6FAyTup4VRFkj9G49tZ/fGJGysS/YCTlIeKDrWIBaM4s3o2U/1qza/7c5FVCAqoQaFmv/rVGyQsU1wjk9TabuCnGObUoGCSTyu9zPKUsjEd8q5DTRW3YT7fdUrOnDMgcWLc00jm7u+JnCprJypynYriyC7XZuZ/tW6G8XWYC51myDVbfBRnkmBCZoeTgTCcoZw4oMwItythI2ooQxdPxYUQLJ+8Cu2LeuD4/rLWuCniKMMJnMI5BHAFDbiDJrSAwQie4RXePOW9eO/ex6K15BUzx/BH3ucPMe2OUw==</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 18

Perceptron Output

((
0, 0,
z  z✓ < ✓,
ŷ =ŷ = 1, z ✓,
1, z > ✓ <latexit sha1_base64="YQGWhwVmpYvL2KTPGvzXMV6Te2M=">AAACM3icbVDLSgMxFM3UV62vqks3waK4EJkRQRcKRTfiSsGq0Cklk962oZnMkNwRxmH+yY0/4kIQF4q49R9MH+LzQOBwzj1J7gliKQy67qNTGBufmJwqTpdmZufmF8qLSxcmSjSHGo9kpK8CZkAKBTUUKOEq1sDCQMJl0Dvq+5fXoI2I1DmmMTRC1lGiLThDKzXLJ36XYZbm9MAPoCNUxu1lJqfuJl2nN3Sf+tgFZJvU96k31PwOfKmgWp+RZrnibrkD0L/EG5EKGeG0Wb73WxFPQlDIJTOm7rkxNjKmUXAJeclPDMSM91gH6pYqFoJpZIOdc7pmlRZtR9oehXSgfk9kLDQmDQM7GTLsmt9eX/zPqyfY3mtkQsUJguLDh9qJpBjRfoG0JTRwlKkljGth/0p5l2nG0dZcsiV4v1f+Sy62tzzLz3Yq1cNRHUWyQlbJBvHILqmSY3JKaoSTW/JAnsmLc+c8Oa/O23C04Iwyy+QHnPcPuF+n6Q==</latexit>

<latexit sha1_base64="PvzvrBemhNK4vBhn/z8jasoJbkw=">AAACNnicbVBNSwMxFMz67fpV9eglWAQPUnZFUBBF9OJFULAqNKVk09c2mM0uyVuhLv1VXvwd3rx4UMSrP8G03UOtDgSGmTdJ3kSpkhaD4NWbmJyanpmdm/cXFpeWV0qrazc2yYyAqkhUYu4ibkFJDVWUqOAuNcDjSMFtdH/W928fwFiZ6GvsplCPeVvLlhQcndQoXbAOx7zbo0eURdCWOhfuNtvzgx3KDukjZQooww4gp4z5YaEej2gMdLNINUrloBIMQP+SsCBlUuCyUXphzURkMWgUiltbC4MU6zk3KIWCns8yCykX97wNNUc1j8HW88HaPbrllCZtJcYdjXSgjiZyHlvbjSM3GXPs2HGvL/7n1TJsHdRzqdMMQYvhQ61MUUxov0PalAYEqq4jXBjp/kpFhxsu0DXtuxLC8ZX/kpvdSuj41V755LSoY45skE2yTUKyT07IObkkVSLIE3kl7+TDe/bevE/vazg64RWZdfIL3vcPMmOpLg==</latexit>

More convenient to re-arrange:

((
0,0, z z ✓ 
✓<0 0,
ŷŷ==
1,1, z z ✓ >
✓ 0 0, use > 0
<latexit sha1_base64="4bMZ89nB996e4v0iNuLHzdgPd6M=">AAACPnicbVDPSxtBGJ31Z1x/NOrRy4dB6KHKrhRaKIropUcLjQqZEGYnX5LB2dll5lshXfKXefFv8Oaxlx6U4tVjJ8keavTBwOO9783M95JcK0dR9BDMzS8sLi3XVsLVtfWND/XNrQuXFVZiU2Y6s1eJcKiVwSYp0niVWxRpovEyuT4b+5c3aJ3KzE8a5thORd+onpKCvNSpN/lAUDkcwRHwBPvKlNLf5kZh9An4N/gF+8BpgCSAa4QIOA/jWed4qnM03SrdqTeig2gCeEviijRYhfNO/Z53M1mkaEhq4VwrjnJql8KSkhpHIS8c5kJeiz62PDUiRdcuJ+uPYM8rXehl1h9DMFH/T5QidW6YJn4yFTRws95YfM9rFdT72i6VyQtCI6cP9QoNlMG4S+gqi5L00BMhrfJ/BTkQVkjyjYe+hHh25bfk4vAg9vzH58bJaVVHje2wXfaRxewLO2Hf2TlrMslu2W/2yJ6Cu+BP8Dd4no7OBVVmm71C8PIP5KequA==</latexit>

<latexit sha1_base64="UluvzdNqfSb3dnAUicsFbiBwYmA=">AAACO3icbVBNSwMxFMz6bf2qevQSLIoHlV0R9KBQ9OJRxarQXUo2fW1Ds9kleSvUZf+XF/+ENy9ePCji1btpXUGrA4Fh5k2SN2EihUHXfXRGRsfGJyanpkszs3PzC+XFpUsTp5pDjccy1tchMyCFghoKlHCdaGBRKOEq7B73/asb0EbE6gJ7CQQRayvREpyhlRrlc7/DMOvl9NAPoS1Uxu1lJqfuJl2nt3SL+tgBZPSgr/g+9YZ0vw0DB1TzO9ooV9xtdwD6l3gFqZACp43yg9+MeRqBQi6ZMXXPTTDImEbBJeQlPzWQMN5lbahbqlgEJsgGu+d0zSpN2oq1PQrpQP2ZyFhkTC8K7WTEsGOGvb74n1dPsbUfZEIlKYLiXw+1Ukkxpv0iaVNo4Ch7ljCuhf0r5R2mGUdbd8mW4A2v/Jdc7mx7lp/tVqpHRR1TZIWskg3ikT1SJSfklNQIJ3fkibyQV+feeXbenPev0RGnyCyTX3A+PgFiTalz</latexit>

negative threshold
<latexit sha1_base64="n7OD3CvnOqylAInuQLPl7+vDpLc=">AAAB7nicbZDLSgNBEEVr4ivGV9Slm8YguDHMiKDLoBuXEcwDkiH0dGqSJj0PumuEEPIRblwo4tbvceff2ElmoYkXGg63quiqG6RKGnLdb6ewtr6xuVXcLu3s7u0flA+PmibJtMCGSFSi2wE3qGSMDZKksJ1q5FGgsBWM7mb11hNqI5P4kcYp+hEfxDKUgpO1WhddGiLxXrniVt252Cp4OVQgV71X/ur2E5FFGJNQ3JiO56bkT7gmKRROS93MYMrFiA+wYzHmERp/Ml93ys6s02dhou2Lic3d3xMTHhkzjgLbGXEamuXazPyv1skovPEnMk4zwlgsPgozxShhs9tZX2oUpMYWuNDS7srEkGsuyCZUsiF4yyevQvOy6ll+uKrUbvM4inACp3AOHlxDDe6hDg0QMIJneIU3J3VenHfnY9FacPKZY/gj5/MHDjiPXw==</latexit>
✓ = "bias"

Sebastian Raschka STAT 453: Intro to Deep Learning 19

General Notation for Single-Layer Neural Networks

• Common notation (in most modern texts): de ne the bias unit separately
• However, often inconvenient for mathematical notation

b
<latexit sha1_base64="s6L+Z+fhtGywXDdOyCIOKOnOTTA=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjrWYwKFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz/E44zm</latexit>
<latexit

Activation
"separate" bias unit
w1
<latexit sha1_base64="yC0dBIEl9qTv0X7X9wtBoMi5o3k=">AAAB+nicbZBNS8NAEIYnftb6lerRS7AInkoigh6rXjxWsB/QhrLZbtulm03YnVRL7E/x4kERr/4Sb/4bt2kO2vrCwsM7M8zsG8SCa3Tdb2tldW19Y7OwVdze2d3bt0sHDR0lirI6jUSkWgHRTHDJ6shRsFasGAkDwZrB6GZWb46Z0jyS9ziJmR+SgeR9Tgkaq2uXOsgeMb2iyMeZNe3aZbfiZnKWwcuhDLlqXfur04toEjKJVBCt254bo58ShZwKNi12Es1iQkdkwNoGJQmZ9tPs9KlzYpye04+UeRKdzP09kZJQ60kYmM6Q4FAv1mbmf7V2gv1LP+UyTpBJOl/UT4SDkTPLwelxxSiKiQFCFTe3OnRIFKFo0iqaELzFLy9D46ziGb47L1ev8zgKcATHcAoeXEAVbqEGdaDwAM/wCm/Wk/VivVsf89YVK585hD+yPn8ADeOUgA==</latexit>

X !!
<latexit sha1_base64="ozSIzVA/SGXegmac4XRXthOpvw0=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwosjZ8=</latexit>

Xm
x2 w2 ŷ m
X
T xT w + b = ŷ
<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit> <latexit sha1_base64="0Hc81E1zKREVtkdUaco0RSs+Ymk=">AAAB63icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCrYW2qVk02wbmmSXZFYoS/+CFw+KePUPefPfmLZ70NYXAg/vzJCZN0qlsOj7315pbX1jc6u8XdnZ3ds/qB4etW2SGcZbLJGJ6UTUcik0b6FAyTup4VRFkj9G49tZ/fGJGysS/YCTlIeKDrWIBaM4s3o2U/1qza/7c5FVCAqoQaFmv/rVGyQsU1wjk9TabuCnGObUoGCSTyu9zPKUsjEd8q5DTRW3YT7fdUrOnDMgcWLc00jm7u+JnCprJypynYriyC7XZuZ/tW6G8XWYC51myDVbfBRnkmBCZoeTgTCcoZw4oMwItythI2ooQxdPxYUQLJ+8Cu2LeuD4/rLWuCniKMMJnMI5BHAFDbiDJrSAwQie4RXePOW9eO/ex6K15BUzx/BH3ucPMe2OUw==</latexit>

<latexit sha1_base64="dwm2weui/za/rSvUQ73K/Rk4/5k=">AAAB7XicbZDLSgMxFIbP1Futt1GXboJFcFVmRNBl0Y3LCvYC7VAyaaaNzWVIMkIZ+g5uXCji1vdx59uYtrPQ1h8CH/85h5zzxylnxgbBt1daW9/Y3CpvV3Z29/YP/MOjllGZJrRJFFe6E2NDOZO0aZnltJNqikXMaTse387q7SeqDVPywU5SGgk8lCxhBFtntXqGDQXu+9WgFsyFViEsoAqFGn3/qzdQJBNUWsKxMd0wSG2UY20Z4XRa6WWGppiM8ZB2HUosqIny+bZTdOacAUqUdk9aNHd/T+RYGDMRsesU2I7Mcm1m/lfrZja5jnIm08xSSRYfJRlHVqHZ6WjANCWWTxxgopnbFZER1phYF1DFhRAun7wKrYta6Pj+slq/KeIowwmcwjmEcAV1uIMGNIHAIzzDK7x5ynvx3r2PRWvJK2aO4Y+8zx+cYY8j</latexit>
<latexit sha1_base64="Vi95YwknFrFzcB5LyqgiYSoMf0U=">AAAB7nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBfsBbSib7aZdutmE3YkQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3iCRwqDrfjuljc2t7Z3ybmVv/+DwqHp80jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTO/m9e4T10bE6hGzhPsRHSsRCkbRWt3BhGKezYbVmlt3FyLr4BVQg0KtYfVrMIpZGnGFTFJj+p6boJ9TjYJJPqsMUsMTyqZ0zPsWFY248fPFujNyYZ0RCWNtn0KycH9P5DQyJosC2xlRnJjV2tz8r9ZPMbzxc6GSFLliy4/CVBKMyfx2MhKaM5SZBcq0sLsSNqGaMrQJVWwI3urJ69C5qnuWH65rzdsijjKcwTlcggcNaMI9tKANDKbwDK/w5iTOi/PufCxbS04xcwp/5Hz+ALHDj8o=</latexit>

x w
x1 wi1 + + b
i b = = x w + b = ŷ
Output
.. <latexit sha1_base64="gvQE9cb1Dja5lCiXX5pMwfJapj8=">AAAB9HicbZBNS8NAEIY3ftb6VfXoZbEInkoigh6LXrxZwX5AG8pmO2mXbjZxd1Isob/DiwdFvPpjvPlv3LY5aOsLCw/vzLAzb5BIYdB1v52V1bX1jc3CVnF7Z3dvv3Rw2DBxqjnUeSxj3QqYASkU1FGghFaigUWBhGYwvJnWmyPQRsTqAccJ+BHrKxEKztBafgfhCbO7FJMUJ91S2a24M9Fl8HIok1y1bumr04t5GoFCLpkxbc9N0M+YRsElTIqd1EDC+JD1oW1RsQiMn82WntBT6/RoGGv7FNKZ+3siY5Ex4yiwnRHDgVmsTc3/au0Uwys/E8qeBIrPPwpTSTGm0wRoT2jgKMcWGNfC7kr5gGnG0eZUtCF4iycvQ+O84lm+vyhXr/M4CuSYnJAz4pFLUiW3pEbqhJNH8kxeyZszcl6cd+dj3rri5DNH5I+czx+i4ZKm</latexit>

i=1
i=1
. Net input
<latexit sha1_base64="A6o7zW7lnOXrNRdOyoA7bzsujJ4=">AAACUXicbVFNS8QwEJ2t3+vXqkcvg4uwIkgrgl4E0YtHBVeF7VrSbNoNJm1JUnUp/Yse9OT/8OJBMdtdwa+BgZc375HJS5gJro3rvtScicmp6ZnZufr8wuLScmNl9VKnuaKsTVORquuQaCZ4wtqGG8GuM8WIDAW7Cm9PhvOrO6Y0T5MLM8hYV5I44RGnxFgqaPR9zWNJ/GMexy30dS6Dgh965U0hS3wIPLy3vY0hVootPMSRAf2Qxy1fEtMPo+KhvLnAr8N9OTJYQaXvE1MMyqDRdHfcqvAv8MagCeM6CxpPfi+luWSJoYJo3fHczHQLogyngpV1P9csI/SWxKxjYUIk092iSqTETcv0MEqV7cRgxX53FERqPZChVQ631r9nQ/K/WSc30UG34EmWG5bQ0UVRLtCkOIwXe1wxasTAAkIVt7si7RNFqLGfULcheL+f/Bdc7u54Fp/vNY+Ox3HMwjpsQAs82IcjOIUzaAOFR3iFd/ioPdfeHHCckdSpjT1r8KOc+U+YcbFc</latexit>

<latexit sha1_base64="5F4RtKSwN6gLVNG6+drmzi7FRow=">AAACfHicbVFNT9tAEF0bKJDSksKRAyNSpCBUZKOq7aVSVC4cg5SQSHGI1pt1ssqube2OGyLLv6L/jFt/Si8Vm8TiI2Gk1Ty992Z2djZMpTDoeX8dd2Nz6932zm7l/d6Hj/vVTwe3Jsk0422WyER3Q2q4FDFvo0DJu6nmVIWSd8LJ1Vzv/ObaiCRu4SzlfUVHsYgEo2ipQfVPYMRIUQgkj7AOgckUDCAHAT/BhwLuLFY235dsAdMndA4hBFqMxnhm3SuNFMVxGFnnfdmlZfMzO12vH1O0wgyKQbXmXXiLgHXgl6BGymgOqg/BMGGZ4jEySY3p+V6K/ZxqFEzyohJkhqeUTeiI9yyMqeKmny+WV8CpZYYQJdqeGGHBvqzIqTJmpkLrnE9vVrU5+ZbWyzD60c9FnGbIY7a8KMokYALzn4Ch0JyhnFlAmRZ2VmBjqilD+18VuwR/9cnr4Pbywrf45mut8atcxw45IiekTnzynTTINWmSNmHkn3Ps1J0z57/72T13vyytrlPWHJJX4X57BMCwuMI=</latexit>

((
<latexit sha1_base64="vnW9SOTDG2wSeqwpvMYjb0pYOfc=">AAAB+XicbZDLSsNAFIYn9VbrLerSzWARXJVEBF0W3biSCvYCbSiT6Uk7dHJh5qRYQt/EjQtF3Pom7nwbp2kW2vrDwMd/zmHO+f1ECo2O822V1tY3NrfK25Wd3b39A/vwqKXjVHFo8ljGquMzDVJE0ESBEjqJAhb6Etr++HZeb09AaRFHjzhNwAvZMBKB4AyN1bftHsITZveAVERJirO+XXVqTi66Cm4BVVKo0be/eoOYpyFEyCXTuus6CXoZUyi4hFmll2pIGB+zIXQNRiwE7WX55jN6ZpwBDWJlXoQ0d39PZCzUehr6pjNkONLLtbn5X62bYnDtZflJEPHFR0EqKcZ0HgMdCAUc5dQA40qYXSkfMcU4mrAqJgR3+eRVaF3UXMMPl9X6TRFHmZyQU3JOXHJF6uSONEiTcDIhz+SVvFmZ9WK9Wx+L1pJVzByTP7I+fwD13ZPb</latexit>

 00
00,if z <
(z)==
(z)
xm <latexit sha1_base64="UZ/Cq01CQU77ibJgEHsrgiYApIY=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMiuWpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2aijdw=</latexit>
11,if z > 10
<latexit sha1_base64="m1vO6djD/2/n5RhOD3QM1YDZNMw=">AAACTnicbVHPaxNBGJ1NtY1R21SPXj4MQr2UXRHqQSHUi8cUzA/IhjA7+XYzZHZ2nfm2mCz5C3spvflnePFgEZ1N9qCJDwYe730/Zt5EuZKWfP+b1zh48PDwqPmo9fjJ0+OT9umzgc0KI7AvMpWZUcQtKqmxT5IUjnKDPI0UDqPFx8ofXqOxMtOfaZnjJOWJlrEUnJw0bWNoZZLys9Vr+ABhhInUpXDz7BpasIEPIeFXKkHGsAZYwftKCms32HHDBL84sXJD1LN61rTd8c/9DWCfBDXpsBq9afsunGWiSFGTUNzaceDnNCm5ISkUrlthYTHnYsETHDuqeYp2Um7iWMMrp8wgzow7mmCj/t1R8tTaZRq5ypTT3O56lfg/b1xQ/G5SSp0XhFpsF8WFAsqgyhZm0qAgtXSECyPdXUHMueGC3A+0XAjB7pP3yeDNeeD41dtO97KOo8lesJfsjAXsgnXZJ9ZjfSbYDfvOfrJ779b74f3yfm9LG17d85z9g0bzDz4prWY=</latexit>

<latexit sha1_base64="T1uilhZzWsdKA050UzjQrhpqtEw=">AAACLnicbVBdS8MwFE3nd/2a+uhLcAgKIq0ICqKIIvio4JywjJGmt1tYmpYkFbayX+SLf0UfBBXx1Z9htvVBpwcCJ+fce5N7glRwbTzv1SlNTE5Nz8zOufMLi0vL5ZXVW51kikGVJSJRdwHVILiEquFGwF2qgMaBgFrQOR/4tXtQmifyxnRTaMS0JXnEGTVWapYviOatmG71tvExJgG0uMyZnaf7rreDyRHuYSIAe5gQ1y+Ek9GVgAyL2ma54u16Q+C/xC9IBRW4apafSZiwLAZpmKBa130vNY2cKsOZgL5LMg0pZR3agrqlksagG/lw3T7etEqIo0TZIw0eqj87chpr3Y0DWxlT09bj3kD8z6tnJjps5FymmQHJRg9FmcAmwYPscMgVMCO6llCmuP0rZm2qKDM2YdeG4I+v/Jfc7u36ll/vV07Pijhm0TraQFvIRwfoFF2iK1RFDD2gJ/SG3p1H58X5cD5HpSWn6FlDv+B8fQN9YKSI</latexit>

Inputs
<latexit sha1_base64="kW2ZbIA+FSvwKPaKbZPllX8WYNo=">AAAB9HicbZBNS8NAEIYnftb6VfXoZbEInkoigh6LXvRWwX5AG8pmu2mXbjZxd1Isob/DiwdFvPpjvPlv3LY5aOsLCw/vzLAzb5BIYdB1v52V1bX1jc3CVnF7Z3dvv3Rw2DBxqhmvs1jGuhVQw6VQvI4CJW8lmtMokLwZDG+m9eaIayNi9YDjhPsR7SsRCkbRWn4H+RNmdypJ0Uy6pbJbcWciy+DlUIZctW7pq9OLWRpxhUxSY9qem6CfUY2CST4pdlLDE8qGtM/bFhWNuPGz2dITcmqdHgljbZ9CMnN/T2Q0MmYcBbYzojgwi7Wp+V+tnWJ45WdiehNXbP5RmEqCMZkmQHpCc4ZybIEyLeyuhA2opgxtTkUbgrd48jI0ziue5fuLcvU6j6MAx3ACZ+DBJVThFmpQBwaP8Ayv8OaMnBfn3fmYt644+cwR/JHz+QONXpKY</latexit>

b=
<latexit sha1_base64="DCslSBKTzJPCossa3lI7E6Re1xA=">AAAB83icbZBNS8NAEIY39avWr6pHL4tF8GJJRNCLUPTisYK1hSaUzXbSLt1swu5EKKF/w4sHRbz6Z7z5b9y2OWjrCwsP78wws2+YSmHQdb+d0srq2vpGebOytb2zu1fdP3g0SaY5tHgiE90JmQEpFLRQoIROqoHFoYR2OLqd1ttPoI1I1AOOUwhiNlAiEpyhtfyQXtMz6uMQkPWqNbfuzkSXwSugRgo1e9Uvv5/wLAaFXDJjup6bYpAzjYJLmFT8zEDK+IgNoGtRsRhMkM9untAT6/RplGj7FNKZ+3siZ7Ex4zi0nTHDoVmsTc3/at0Mo6sgFyrNEBSfL4oySTGh0wBoX2jgKMcWGNfC3kr5kGnG0cZUsSF4i19ehsfzumf5/qLWuCniKJMjckxOiUcuSYPckSZpEU5S8kxeyZuTOS/Ou/Mxby05xcwh+SPn8wdTHpCQ</latexit>
✓

Sebastian Raschka STAT 453: Intro to Deep Learning 20

fi
General Notation for Single-Layer Neural Networks
• Often more convenient notation: de ne bias unit as w0 and
prepend a 1 to each input vector as an additional "feature" value
• Modifying input vectors is more inconvenient/ine cient
coding-wise, though

1
<latexit sha1_base64="qp47xzdgavbzDAujuQrfWObrWms=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY3KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz96n4y1</latexit>
<latexit

bias unit "included" as w0 !!

m
X
Xm
xixwiw = = x>xw
T = ŷ
w = ŷ
w0 0 0
x1 i=0
<latexit sha1_base64="bcuIWy+A2payM+NYYhRXc5O7Jx8=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+m6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwiojZ4=</latexit>

<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

<latexit sha1_base64="f0bK/7uEdS00YtbqG5UODvQhjXQ=">AAACSnicbVBNS8NAEN3U7/pV9ehlsQj1IokIeimUevGoYG2hqWGy3aSLu0nY3diWkN/nxZM3f4QXD4p4cZtW8Gtg4O17b5jZ5yecKW3bT1Zpbn5hcWl5pby6tr6xWdnavlZxKgltkZjHsuODopxFtKWZ5rSTSArC57Tt355N9PYdlYrF0ZUeJ7QnIIxYwAhoQ3kVcBULBbhNFoY17KpUeBmr2/lNJnI88mw8NF2oB7iOp2bs+iysuQL0wA+yUX5zhb8ewxwXamEegM7GuVep2od2UfgvcGagimZ14VUe3X5MUkEjTTgo1XXsRPcykJoRTvOymyqaALmFkHYNjEBQ1cuKKHK8b5g+DmJpOtK4YL9PZCCUGgvfOCcnq9/ahPxP66Y6OO1lLEpSTSMyXRSkHOsYT3LFfSYp0XxsABDJzK2YDEAC0Sb9sgnB+f3lv+D66NAx+PK42mjO4lhGu2gP1ZCDTlADnaML1EIE3aNn9IrerAfrxXq3PqbWkjWb2UE/qjT/CcjKsZc=</latexit>
i=0
Activation (
w1 (
<latexit sha1_base64="yC0dBIEl9qTv0X7X9wtBoMi5o3k=">AAAB+nicbZBNS8NAEIYnftb6lerRS7AInkoigh6rXjxWsB/QhrLZbtulm03YnVRL7E/x4kERr/4Sb/4bt2kO2vrCwsM7M8zsG8SCa3Tdb2tldW19Y7OwVdze2d3bt0sHDR0lirI6jUSkWgHRTHDJ6shRsFasGAkDwZrB6GZWb46Z0jyS9ziJmR+SgeR9Tgkaq2uXOsgeMb2iyMeZNe3aZbfiZnKWwcuhDLlqXfur04toEjKJVBCt254bo58ShZwKNi12Es1iQkdkwNoGJQmZ9tPs9KlzYpye04+UeRKdzP09kZJQ60kYmM6Q4FAv1mbmf7V2gv1LP+UyTpBJOl/UT4SDkTPLwelxxSiKiQFCFTe3OnRIFKFo0iqaELzFLy9D46ziGb47L1ev8zgKcATHcAoeXEAVbqEGdaDwAM/wCm/Wk/VivVsf89YVK585hD+yPn8ADeOUgA==</latexit>

X (z) =
=
0 ifz z<0 0
0,
x2 ŷ (z) 1 ifz z> 0 1
<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

w2
<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit> <latexit sha1_base64="0Hc81E1zKREVtkdUaco0RSs+Ymk=">AAAB63icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCrYW2qVk02wbmmSXZFYoS/+CFw+KePUPefPfmLZ70NYXAg/vzJCZN0qlsOj7315pbX1jc6u8XdnZ3ds/qB4etW2SGcZbLJGJ6UTUcik0b6FAyTup4VRFkj9G49tZ/fGJGysS/YCTlIeKDrWIBaM4s3o2U/1qza/7c5FVCAqoQaFmv/rVGyQsU1wjk9TabuCnGObUoGCSTyu9zPKUsjEd8q5DTRW3YT7fdUrOnDMgcWLc00jm7u+JnCprJypynYriyC7XZuZ/tW6G8XWYC51myDVbfBRnkmBCZoeTgTCcoZw4oMwItythI2ooQxdPxYUQLJ+8Cu2LeuD4/rLWuCniKMMJnMI5BHAFDbiDJrSAwQie4RXePOW9eO/ex6K15BUzx/BH3ucPMe2OUw==</latexit>
<latexit sha1_base64="dwm2weui/za/rSvUQ73K/Rk4/5k=">AAAB7XicbZDLSgMxFIbP1Futt1GXboJFcFVmRNBl0Y3LCvYC7VAyaaaNzWVIMkIZ+g5uXCji1vdx59uYtrPQ1h8CH/85h5zzxylnxgbBt1daW9/Y3CpvV3Z29/YP/MOjllGZJrRJFFe6E2NDOZO0aZnltJNqikXMaTse387q7SeqDVPywU5SGgk8lCxhBFtntXqGDQXu+9WgFsyFViEsoAqFGn3/qzdQJBNUWsKxMd0wSG2UY20Z4XRa6WWGppiM8ZB2HUosqIny+bZTdOacAUqUdk9aNHd/T+RYGDMRsesU2I7Mcm1m/lfrZja5jnIm08xSSRYfJRlHVqHZ6WjANCWWTxxgopnbFZER1phYF1DFhRAun7wKrYta6Pj+slq/KeIowwmcwjmEcAV1uIMGNIHAIzzDK7x5ynvx3r2PRWvJK2aO4Y+8zx+cYY8j</latexit>
<latexit sha1_base64="Vi95YwknFrFzcB5LyqgiYSoMf0U=">AAAB7nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBfsBbSib7aZdutmE3YkQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3iCRwqDrfjuljc2t7Z3ybmVv/+DwqHp80jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTO/m9e4T10bE6hGzhPsRHSsRCkbRWt3BhGKezYbVmlt3FyLr4BVQg0KtYfVrMIpZGnGFTFJj+p6boJ9TjYJJPqsMUsMTyqZ0zPsWFY248fPFujNyYZ0RCWNtn0KycH9P5DQyJosC2xlRnJjV2tz8r9ZPMbzxc6GSFLliy4/CVBKMyfx2MhKaM5SZBcq0sLsSNqGaMrQJVWwI3urJ69C5qnuWH65rzdsijjKcwTlcggcNaMI9tKANDKbwDK/w5iTOi/PufCxbS04xcwp/5Hz+ALHDj8o=</latexit>

<latexit sha1_base64="m1vO6djD/2/n5RhOD3QM1YDZNMw=">AAACTnicbVHPaxNBGJ1NtY1R21SPXj4MQr2UXRHqQSHUi8cUzA/IhjA7+XYzZHZ2nfm2mCz5C3spvflnePFgEZ1N9qCJDwYe730/Zt5EuZKWfP+b1zh48PDwqPmo9fjJ0+OT9umzgc0KI7AvMpWZUcQtKqmxT5IUjnKDPI0UDqPFx8ofXqOxMtOfaZnjJOWJlrEUnJw0bWNoZZLys9Vr+ABhhInUpXDz7BpasIEPIeFXKkHGsAZYwftKCms32HHDBL84sXJD1LN61rTd8c/9DWCfBDXpsBq9afsunGWiSFGTUNzaceDnNCm5ISkUrlthYTHnYsETHDuqeYp2Um7iWMMrp8wgzow7mmCj/t1R8tTaZRq5ypTT3O56lfg/b1xQ/G5SSp0XhFpsF8WFAsqgyhZm0qAgtXSECyPdXUHMueGC3A+0XAjB7pP3yeDNeeD41dtO97KOo8lesJfsjAXsgnXZJ9ZjfSbYDfvOfrJ779b74f3yfm9LG17d85z9g0bzDz4prWY=</latexit>
1,
Output
<latexit sha1_base64="T1uilhZzWsdKA050UzjQrhpqtEw=">AAACLnicbVBdS8MwFE3nd/2a+uhLcAgKIq0ICqKIIvio4JywjJGmt1tYmpYkFbayX+SLf0UfBBXx1Z9htvVBpwcCJ+fce5N7glRwbTzv1SlNTE5Nz8zOufMLi0vL5ZXVW51kikGVJSJRdwHVILiEquFGwF2qgMaBgFrQOR/4tXtQmifyxnRTaMS0JXnEGTVWapYviOatmG71tvExJgG0uMyZnaf7rreDyRHuYSIAe5gQ1y+Ek9GVgAyL2ma54u16Q+C/xC9IBRW4apafSZiwLAZpmKBa130vNY2cKsOZgL5LMg0pZR3agrqlksagG/lw3T7etEqIo0TZIw0eqj87chpr3Y0DWxlT09bj3kD8z6tnJjps5FymmQHJRg9FmcAmwYPscMgVMCO6llCmuP0rZm2qKDM2YdeG4I+v/Jfc7u36ll/vV07Pijhm0TraQFvIRwfoFF2iK1RFDD2gJ/SG3p1H58X5cD5HpSWn6FlDv+B8fQN9YKSI</latexit>

.. <latexit sha1_base64="gvQE9cb1Dja5lCiXX5pMwfJapj8=">AAAB9HicbZBNS8NAEIY3ftb6VfXoZbEInkoigh6LXrxZwX5AG8pmO2mXbjZxd1Isob/DiwdFvPpjvPlv3LY5aOsLCw/vzLAzb5BIYdB1v52V1bX1jc3CVnF7Z3dvv3Rw2DBxqjnUeSxj3QqYASkU1FGghFaigUWBhGYwvJnWmyPQRsTqAccJ+BHrKxEKztBafgfhCbO7FJMUJ91S2a24M9Fl8HIok1y1bumr04t5GoFCLpkxbc9N0M+YRsElTIqd1EDC+JD1oW1RsQiMn82WntBT6/RoGGv7FNKZ+3siY5Ex4yiwnRHDgVmsTc3/au0Uwys/E8qeBIrPPwpTSTGm0wRoT2jgKMcWGNfC7kr5gGnG0eZUtCF4iycvQ+O84lm+vyhXr/M4CuSYnJAz4pFLUiW3pEbqhJNH8kxeyZszcl6cd+dj3rri5DNH5I+czx+i4ZKm</latexit>

w0 = ✓
wm
<latexit sha1_base64="3SltFZgdSbEccduFdJMJ4sVJM+s=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMquUpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2Ucjds=</latexit>
<latexit sha1_base64="uq85eCvkyXNyYkvb4MzxZsICWH4=">AAAB9XicbZBNSwMxEIaz9avWr6pHL8EieLHsiqAXoejFYwX7Ae1asum0Dc1ml2TWUpb+Dy8eFPHqf/HmvzFt96CtLwQe3plhJm8QS2HQdb+d3Mrq2vpGfrOwtb2zu1fcP6ibKNEcajySkW4GzIAUCmooUEIz1sDCQEIjGN5O640n0EZE6gHHMfgh6yvRE5yhtR5HHZde0zPaxgEg6xRLbtmdiS6Dl0GJZKp2il/tbsSTEBRyyYxpeW6Mfso0Ci5hUmgnBmLGh6wPLYuKhWD8dHb1hJ5Yp0t7kbZPIZ25vydSFhozDgPbGTIcmMXa1Pyv1kqwd+WnQsUJguLzRb1EUozoNALaFRo4yrEFxrWwt1I+YJpxtEEVbAje4peXoX5e9izfX5QqN1kceXJEjskp8cglqZA7UiU1wokmz+SVvDkj58V5dz7mrTknmzkkf+R8/gCeDJFI</latexit>

xm <latexit sha1_base64="UZ/Cq01CQU77ibJgEHsrgiYApIY=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMiuWpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2aijdw=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 21

fi
ffi
General Notation for Single-Layer Neural Networks

Vector dot product

w0 m
m
!
x1
<latexit sha1_base64="bcuIWy+A2payM+NYYhRXc5O7Jx8=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+m6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwiojZ4=</latexit>

X
X
xx0iw
w0i = xT w = ŷ
<latexit sha1_base64="Z7jxfJr8/pbKF9IEHv5u2p28PzU=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwuyjaA=</latexit>

Activation
w1
<latexit sha1_base64="yC0dBIEl9qTv0X7X9wtBoMi5o3k=">AAAB+nicbZBNS8NAEIYnftb6lerRS7AInkoigh6rXjxWsB/QhrLZbtulm03YnVRL7E/x4kERr/4Sb/4bt2kO2vrCwsM7M8zsG8SCa3Tdb2tldW19Y7OwVdze2d3bt0sHDR0lirI6jUSkWgHRTHDJ6shRsFasGAkDwZrB6GZWb46Z0jyS9ziJmR+SgeR9Tgkaq2uXOsgeMb2iyMeZNe3aZbfiZnKWwcuhDLlqXfur04toEjKJVBCt254bo58ShZwKNi12Es1iQkdkwNoGJQmZ9tPs9KlzYpye04+UeRKdzP09kZJQ60kYmM6Q4FAv1mbmf7V2gv1LP+UyTpBJOl/UT4SDkTPLwelxxSiKiQFCFTe3OnRIFKFo0iqaELzFLy9D46ziGb47L1ev8zgKcATHcAoeXEAVbqEGdaDwAM/wCm/Wk/VivVsf89YVK585hD+yPn8ADeOUgA==</latexit>

i=0
i=0
X
<latexit sha1_base64="f0bK/7uEdS00YtbqG5UODvQhjXQ=">AAACSnicbVBNS8NAEN3U7/pV9ehlsQj1IokIeimUevGoYG2hqWGy3aSLu0nY3diWkN/nxZM3f4QXD4p4cZtW8Gtg4O17b5jZ5yecKW3bT1Zpbn5hcWl5pby6tr6xWdnavlZxKgltkZjHsuODopxFtKWZ5rSTSArC57Tt355N9PYdlYrF0ZUeJ7QnIIxYwAhoQ3kVcBULBbhNFoY17KpUeBmr2/lNJnI88mw8NF2oB7iOp2bs+iysuQL0wA+yUX5zhb8ewxwXamEegM7GuVep2od2UfgvcGagimZ14VUe3X5MUkEjTTgo1XXsRPcykJoRTvOymyqaALmFkHYNjEBQ1cuKKHK8b5g+DmJpOtK4YL9PZCCUGgvfOCcnq9/ahPxP66Y6OO1lLEpSTSMyXRSkHOsYT3LFfSYp0XxsABDJzK2YDEAC0Sb9sgnB+f3lv+D66NAx+PK42mjO4lhGu2gP1ZCDTlADnaML1EIE3aNn9IrerAfrxXq3PqbWkjWb2UE/qjT/CcjKsZc=</latexit>

<latexit sha1_base64="6pvg3U5UUGSQra7hL50u8iJ8N5c=">AAACFnicbZDLSgMxFIYz9VbrrerSTbAIdWGZqYJuhKIblxXsBTq1ZNJMG5rMDMkZtQzzFG58FTcuFHEr7nwb08tCWw8kfPz/OSTn9yLBNdj2t5VZWFxaXsmu5tbWNza38ts7dR3GirIaDUWomh7RTPCA1YCDYM1IMSI9wRre4HLkN+6Y0jwMbmAYsbYkvYD7nBIwUid/5Grek8QVzIeiq2PZSfi5nd4mMsUPhlN8P7pdxXt9OOzkC3bJHheeB2cKBTStaif/5XZDGksWABVE65ZjR9BOiAJOBUtzbqxZROiA9FjLYEAk0+1kvFaKD4zSxX6ozAkAj9XfEwmRWg+lZzolgb6e9Ubif14rBv+snfAgioEFdPKQHwsMIR5lhLtcMQpiaIBQxc1fMe0TRSiYJHMmBGd25Xmol0vOcal8fVKoXEzjyKI9tI+KyEGnqIKuUBXVEEWP6Bm9ojfryXqx3q2PSWvGms7soj9lff4ALXOgAA==</latexit>

(
<latexit sha1_base64="ozSIzVA/SGXegmac4XRXthOpvw0=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+l6/XHGr7lxkFbwcKpCr0S9/9QYxSyOukElqTNdzE/QzqlEwyaelXmp4QtmYDnnXoqIRN342X3VKzqwzIGGs7VNI5u7viYxGxkyiwHZGFEdmuTYz/6t1Uwyv/EyoJEWu2OKjMJUEYzK7mwyE5gzlxAJlWthdCRtRTRnadEo2BG/55FVoXVQ9y3eXlfp1HkcRTuAUzsGDGtThFhrQBAZDeIZXeHOk8+K8Ox+L1oKTzxzDHzmfPwosjZ8=</latexit>

x2 ŷ (
<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

w2 <latexit sha1_base64="Vi95YwknFrFzcB5LyqgiYSoMf0U=">AAAB7nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjBfsBbSib7aZdutmE3YkQQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3iCRwqDrfjuljc2t7Z3ybmVv/+DwqHp80jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTO/m9e4T10bE6hGzhPsRHSsRCkbRWt3BhGKezYbVmlt3FyLr4BVQg0KtYfVrMIpZGnGFTFJj+p6boJ9TjYJJPqsMUsMTyqZ0zPsWFY248fPFujNyYZ0RCWNtn0KycH9P5DQyJosC2xlRnJjV2tz8r9ZPMbzxc6GSFLliy4/CVBKMyfx2MhKaM5SZBcq0sLsSNqGaMrQJVWwI3urJ69C5qnuWH65rzdsijjKcwTlcggcNaMI9tKANDKbwDK/w5iTOi/PufCxbS04xcwp/5Hz+ALHDj8o=</latexit>

00,if zz < 0✓ 0
(z)
(z) =
<latexit sha1_base64="dwm2weui/za/rSvUQ73K/Rk4/5k=">AAAB7XicbZDLSgMxFIbP1Futt1GXboJFcFVmRNBl0Y3LCvYC7VAyaaaNzWVIMkIZ+g5uXCji1vdx59uYtrPQ1h8CH/85h5zzxylnxgbBt1daW9/Y3CpvV3Z29/YP/MOjllGZJrRJFFe6E2NDOZO0aZnltJNqikXMaTse387q7SeqDVPywU5SGgk8lCxhBFtntXqGDQXu+9WgFsyFViEsoAqFGn3/qzdQJBNUWsKxMd0wSG2UY20Z4XRa6WWGppiM8ZB2HUosqIny+bZTdOacAUqUdk9aNHd/T+RYGDMRsesU2I7Mcm1m/lfrZja5jnIm08xSSRYfJRlHVqHZ6WjANCWWTxxgopnbFZER1phYF1DFhRAun7wKrYta6Pj+slq/KeIowwmcwjmEcAV1uIMGNIHAIzzDK7x5ynvx3r2PRWvJK2aO4Y+8zx+cYY8j</latexit>

=
<latexit sha1_base64="sAAe226MpFncoK5AcSpzzUnkA9I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlSQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwALsI2g</latexit> <latexit sha1_base64="0Hc81E1zKREVtkdUaco0RSs+Ymk=">AAAB63icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHCrYW2qVk02wbmmSXZFYoS/+CFw+KePUPefPfmLZ70NYXAg/vzJCZN0qlsOj7315pbX1jc6u8XdnZ3ds/qB4etW2SGcZbLJGJ6UTUcik0b6FAyTup4VRFkj9G49tZ/fGJGysS/YCTlIeKDrWIBaM4s3o2U/1qza/7c5FVCAqoQaFmv/rVGyQsU1wjk9TabuCnGObUoGCSTyu9zPKUsjEd8q5DTRW3YT7fdUrOnDMgcWLc00jm7u+JnCprJypynYriyC7XZuZ/tW6G8XWYC51myDVbfBRnkmBCZoeTgTCcoZw4oMwItythI2ooQxdPxYUQLJ+8Cu2LeuD4/rLWuCniKMMJnMI5BHAFDbiDJrSAwQie4RXePOW9eO/ex6K15BUzx/BH3ucPMe2OUw==</latexit>

11,if zz 1✓ >0
. Net input
<latexit sha1_base64="vnW9SOTDG2wSeqwpvMYjb0pYOfc=">AAAB+XicbZDLSsNAFIYn9VbrLerSzWARXJVEBF0W3biSCvYCbSiT6Uk7dHJh5qRYQt/EjQtF3Pom7nwbp2kW2vrDwMd/zmHO+f1ECo2O822V1tY3NrfK25Wd3b39A/vwqKXjVHFo8ljGquMzDVJE0ESBEjqJAhb6Etr++HZeb09AaRFHjzhNwAvZMBKB4AyN1bftHsITZveAVERJirO+XXVqTi66Cm4BVVKo0be/eoOYpyFEyCXTuus6CXoZUyi4hFmll2pIGB+zIXQNRiwE7WX55jN6ZpwBDWJlXoQ0d39PZCzUehr6pjNkONLLtbn5X62bYnDtZflJEPHFR0EqKcZ0HgMdCAUc5dQA40qYXSkfMcU4mrAqJgR3+eRVaF3UXMMPl9X6TRFHmZyQU3JOXHJF6uSONEiTcDIhz+SVvFmZ9WK9Wx+L1pJVzByTP7I+fwD13ZPb</latexit>

<latexit sha1_base64="zyVd33BWAqcAV8UNVlsJ5WQIsDw=">AAACQHicbVDPSxtBGJ31d9eqUY+9fBgECyq7IigURfTSowVjhEwIs5MvyeDs7DLzbSEu+dO89E/oredeeqiUXnvqJNmDRh8MPN773jczL8m1chRFP4K5+YXFpeWVd+Hq+7X1jdrm1q3LCiuxITOd2btEONTKYIMUabzLLYo00dhM7q/GfvMrWqcyc0PDHNup6BvVU1KQlzq1Jneqn4q9h49wBjzBvjKl9PvcKIz2gX+CBzgATgMkAVwjRMB5GM8651Odo+lW6U6tHh1GE8BrElekzipcd2rfeTeTRYqGpBbOteIop3YpLCmpcRTywmEu5L3oY8tTI1J07XJSwAh2vdKFXmb9MQQT9XmiFKlzwzTxk6mggZv1xuJbXqug3mm7VCYvCI2cXtQrNFAG4zahqyxK0kNPhLTKvxXkQFghyXce+hLi2S+/JrdHh7HnX47rF5dVHSvsA9theyxmJ+yCfWbXrMEke2Q/2W/2FHwLfgV/gr/T0bmgymyzFwj+/QeJaqr6</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="aklfhIT305jg37nRsSg2GUsLuE8=">AAACNXicbZBLaxsxFIXvpI+k07R1us3mUlNwoQ0z3bQQEgrddOlA/QDLGI18bYtoNIN0J+AM/mnd9Cd013U2WTSU/oPIj0Vr94DgcI6uHl9WGu05SX5Gew8ePnq8f/Akfnr47PmLxtFh1xeVU9RRhSlcP5OejLbUYc2G+qUjmWeGetnl52XfuyLndWG/8rykYS6nVk+0khyiUaMnvJ7msnX9Bs9QZDTVtlbhPL+Ik7coTvEa36HgGbFEYQgTFCJOt5vzdS7IjjfTo0YzOUlWwl2TbkwTNmqPGj/EuFBVTpaVkd4P0qTkYS0da2VoEYvKUynVpZzSIFgrc/LDegVgga9DMsZJ4cKyjKv074la5t7P8yzszCXP/Ha3DP/XDSqefBzW2pYVk1XriyaVQS5wSRPH2pFiMw9GKqfDW1HNpJOKA/M4QEi3v7xruu9P0uAvEjiAY3gFLUjhA3yCL9CGDij4BjfwC+6i79Ft9HuNay/acHsJ/yj6cw86m6lP</latexit>
sha1_base64="F2iwpCkiwstMnTZXAEiEqXiCgnw=">AAACQHicbVBNSyNBFOxxP9RZd83q0ctjw4LCrsx4UZAV0YtHhY0R0iH0dF6Sxp6eofuNEIf8NC/+hL159uJBEa+e7CRzWKMFDUXVq9fdleRaOYqim2Duw8dPn+cXFsMvS1+/Lde+r5y6rLASGzLTmT1LhEOtDDZIkcaz3KJIE43N5Pxw7Dcv0DqVmb80zLGdir5RPSUFealTa3Kn+qlYv9yAP8AT7CtTSr/PjcLoF/BduITfwGmAJIBrhAg4D+NZZ2+qczTdKt2p1aPNaAJ4S+KK1FmF407tH+9mskjRkNTCuVYc5dQuhSUlNY5CXjjMhTwXfWx5akSKrl1OChjBT690oZdZfwzBRP0/UYrUuWGa+MlU0MDNemPxPa9VUG+nXSqTF4RGTi/qFRoog3Gb0FUWJemhJ0Ja5d8KciCskOQ7D30J8eyX35LTrc3Y85Oovn9Q1bHA1tgPts5its322RE7Zg0m2RW7ZffsIbgO7oLH4Gk6OhdUmVX2CsHzC4gqqvY=</latexit>
<latexit sha1_base64="m1vO6djD/2/n5RhOD3QM1YDZNMw=">AAACTnicbVHPaxNBGJ1NtY1R21SPXj4MQr2UXRHqQSHUi8cUzA/IhjA7+XYzZHZ2nfm2mCz5C3spvflnePFgEZ1N9qCJDwYe730/Zt5EuZKWfP+b1zh48PDwqPmo9fjJ0+OT9umzgc0KI7AvMpWZUcQtKqmxT5IUjnKDPI0UDqPFx8ofXqOxMtOfaZnjJOWJlrEUnJw0bWNoZZLys9Vr+ABhhInUpXDz7BpasIEPIeFXKkHGsAZYwftKCms32HHDBL84sXJD1LN61rTd8c/9DWCfBDXpsBq9afsunGWiSFGTUNzaceDnNCm5ISkUrlthYTHnYsETHDuqeYp2Um7iWMMrp8wgzow7mmCj/t1R8tTaZRq5ypTT3O56lfg/b1xQ/G5SSp0XhFpsF8WFAsqgyhZm0qAgtXSECyPdXUHMueGC3A+0XAjB7pP3yeDNeeD41dtO97KOo8lesJfsjAXsgnXZJ9ZjfSbYDfvOfrJ779b74f3yfm9LG17d85z9g0bzDz4prWY=</latexit>

xm w0 =
<latexit sha1_base64="uq85eCvkyXNyYkvb4MzxZsICWH4=">AAAB9XicbZBNSwMxEIaz9avWr6pHL8EieLHsiqAXoejFYwX7Ae1asum0Dc1ml2TWUpb+Dy8eFPHqf/HmvzFt96CtLwQe3plhJm8QS2HQdb+d3Mrq2vpGfrOwtb2zu1fcP6ibKNEcajySkW4GzIAUCmooUEIz1sDCQEIjGN5O640n0EZE6gHHMfgh6yvRE5yhtR5HHZde0zPaxgEg6xRLbtmdiS6Dl0GJZKp2il/tbsSTEBRyyYxpeW6Mfso0Ci5hUmgnBmLGh6wPLYuKhWD8dHb1hJ5Yp0t7kbZPIZ25vydSFhozDgPbGTIcmMXa1Pyv1kqwd+WnQsUJguLzRb1EUozoNALaFRo4yrEFxrWwt1I+YJpxtEEVbAje4peXoX5e9izfX5QqN1kceXJEjskp8cglqZA7UiU1wokmz+SVvDkj58V5dz7mrTknmzkkf+R8/gCeDJFI</latexit>
✓
<latexit sha1_base64="UZ/Cq01CQU77ibJgEHsrgiYApIY=">AAAB6nicbZBNSwMxEIZn61etX1WPXoJF8FR2RdBj0YvHirYW2qVk07QNTbJLMiuWpT/BiwdFvPqLvPlvTNs9aOsLgYd3ZsjMGyVSWPT9b6+wsrq2vlHcLG1t7+zulfcPmjZODeMNFsvYtCJquRSaN1Cg5K3EcKoiyR+i0fW0/vDIjRWxvsdxwkNFB1r0BaPorLunruqWK37Vn4ksQ5BDBXLVu+WvTi9mqeIamaTWtgM/wTCjBgWTfFLqpJYnlI3ogLcdaqq4DbPZqhNy4pwe6cfGPY1k5v6eyKiydqwi16koDu1ibWr+V2un2L8MM6GTFLlm84/6qSQYk+ndpCcMZyjHDigzwu1K2JAaytClU3IhBIsnL0PzrBo4vj2v1K7yOIpwBMdwCgFcQA1uoA4NYDCAZ3iFN096L9679zFvLXj5zCH8kff5A2aijdw=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 22

Perceptron Learning Rule
Assume binary classi cation task, Perceptron nds decision boundary
if classes are separable

[animated GIF]
x2
<latexit sha1_base64="8ur8Qnjf68veizOKVqkUmBXGiPw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjRfsBbSib7aZdutmE3YlYQn+CFw+KePUXefPfuG1z0NYXFh7emWFn3iCRwqDrfjuFtfWNza3idmlnd2//oHx41DJxqhlvsljGuhNQw6VQvIkCJe8kmtMokLwdjG9m9fYj10bE6gEnCfcjOlQiFIyite6f+rV+ueJW3bnIKng5VCBXo1/+6g1ilkZcIZPUmK7nJuhnVKNgkk9LvdTwhLIxHfKuRUUjbvxsvuqUnFlnQMJY26eQzN3fExmNjJlEge2MKI7Mcm1m/lfrphhe+ZlQSYpcscVHYSoJxmR2NxkIzRnKiQXKtLC7EjaimjK06ZRsCN7yyavQqlU9y3cXlfp1HkcRTuAUzsGDS6jDLTSgCQyG8Ayv8OZI58V5dz4WrQUnnzmGP3I+fwANNo2h</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 23

fi
fi
The Perceptron Learning Algorithm

• If correct: Do nothing if the prediction if output is equal to the target

• If incorrect, scenario a):

If output is 0 and target is 1, add input vector to weight vector

• If incorrect, scenario b):

If output is 1 and target is 0, subtract input vector from weight vector

Guaranteed to converge if a solution exists

(more about that later...)

Sebastian Raschka STAT 453: Intro to Deep Learning 24

The Perceptron Learning Algorithm

Let
D = (hx[1] , y [1] i, hx[2] , y [2] i, ..., hx[n] , y [n] i) 2 (Rm ⇥ {0, 1})n
<latexit sha1_base64="Wal6cJLWU5bzcyF5u646zqqA+4A=">AAADVXiclZLfbtMwFMZP0jFGB6ywS24sKiSQpiqpkOAGaYJd7HIguk1qs8lxndaa7UT2CaKK+lo8yG4m3oQbJE7dSvvTIYGjyJ+/851fEsd5pZXHJPkZxa2NB5sPtx61tx8/ebrTefb82Je1E3IgSl2605x7qZWVA1So5WnlJDe5lif5xadF/eSbdF6V9ivOKpkZPrGqUIIjWWXnB4zAAAeEKQiaNTRwAHNg8IHu11TV5FqY0CzJuU7nUFD2O2XPaB5CChnpPcrMbjkjcDcIe4Hxr8z+GrP/F2YvXP9Ht2t0ew/9TWAoWi/345qWU8+XQDMhg5Qy1OHDqoGE2CmpOTHOwLLt80436SVhsHWRrkQXVuPovHM5GpeiNtKi0Nz7YZpUmDXcoRJaztuj2suKiws+kUOSlhvpsyacijl7Rc6YFaWj2yIL7s2OhhvvZyanpOE49XdrC/O+2rDG4n3WKFvVKK1YPqioNcOSLY4YGysnBeoZCS6condlYsodF0gHsU2bkN795HVx3O+lpD+/7e5/XG3HFryAl/QLUngH+3AIRzAAEV1Gv+IojuOr+Hdro7W5jMbRqmcXbo3Wzh+cKM3Z</latexit>

m
1. Initialize w := 0 <latexit sha1_base64="Ovfw16+rosPXEmOAFXWO3SGmqMw=">AAAB+nicbVDLSsNAFL2pr1pfqS7dDBbBVUlEUASh6MZlBfuANpbJdNIOnUnCzMRSYj/FjQtF3Pol7vwbJ20W2npg4HDOvdwzx485U9pxvq3Cyura+kZxs7S1vbO7Z5f3mypKJKENEvFItn2sKGchbWimOW3HkmLhc9ryRzeZ33qkUrEovNeTmHoCD0IWMIK1kXp2uSuwHvpBOp6iyyvkPIieXXGqzgxombg5qUCOes/+6vYjkggaasKxUh3XibWXYqkZ4XRa6iaKxpiM8IB2DA2xoMpLZ9Gn6NgofRRE0rxQo5n6eyPFQqmJ8M1kFlQtepn4n9dJdHDhpSyME01DMj8UJBzpCGU9oD6TlGg+MQQTyUxWRIZYYqJNWyVTgrv45WXSPK26ht+dVWrXeR1FOIQjOAEXzqEGt1CHBhAYwzO8wpv1ZL1Y79bHfLRg5TsH8AfW5w8qP5NE</latexit>
(assume notation where weight incl. bias)

2. For every training epoch:

A. For every hx[i] , y [i] i 2 D : <latexit sha1_base64="KiRtzBWzB6OcgzuVeAuk4oyYAnU=">AAACJHicbZDLSsNAFIYn9VbrLerSzWARXEhJRFBwU9SFywr2Akksk+mkHTqZhJmJGEIexo2v4saFF1y48VmctFlo64GBj//8hznn92NGpbKsL6OysLi0vFJdra2tb2xumds7HRklApM2jlgkej6ShFFO2ooqRnqxICj0Gen648ui370nQtKI36o0Jl6IhpwGFCOlpb557jLEh4xAN0Rq5AfZQ36XOdTLj2CWlghdUXoon/owYtlV3jfrVsOaFJwHu4Q6KKvVN9/dQYSTkHCFGZLSsa1YeRkSimJG8pqbSBIjPEZD4mjkKCTSyyZH5vBAKwMYREI/ruBE/T2RoVDKNPS1s1hRzvYK8b+ek6jgzMsojxNFOJ5+FCQMqggWicEBFQQrlmpAWFC9K8QjJBBWOteaDsGePXkeOscNW/PNSb15UcZRBXtgHxwCG5yCJrgGLdAGGDyCZ/AK3own48X4MD6n1opRzuyCP2V8/wAUwKWr</latexit>

[i] [i]>
(a) ŷ := x w
[i]
(b) err := (y <latexit sha1_base64="wYimCRAo97FH6nzYS/fz3/CUouY=">AAACEHicbZDLSsNAFIYn3q23qEs3g0XUhZKIoAhC0Y1LBWuFJpbJ9NQOnVyYORFDyCO48VXcuFDErUt3vo3TNgut/jDwzX/OYeb8QSKFRsf5ssbGJyanpmdmK3PzC4tL9vLKlY5TxaHOYxmr64BpkCKCOgqUcJ0oYGEgoRH0Tvv1xh0oLeLoErME/JDdRqIjOENjtexND+Eec1CqoEfHdCu7yZvCL+gO9boM86wY3rdbdtXZdQaif8EtoUpKnbfsT68d8zSECLlkWjddJ0E/ZwoFl1BUvFRDwniP3ULTYMRC0H4+WKigG8Zp006szImQDtyfEzkLtc7CwHSGDLt6tNY3/6s1U+wc+rmIkhQh4sOHOqmkGNN+OrQtFHCUmQHGlTB/pbzLFONoMqyYENzRlf/C1d6ua/hiv1o7KeOYIWtknWwRlxyQGjkj56ROOHkgT+SFvFqP1rP1Zr0PW8escmaV/JL18Q3haZx5</latexit>
ŷ [i] )
[i]
(c) w := w + err ⇥ x
<latexit sha1_base64="ypGRpFBWbrLkFvM0DBOQw+AAwc4=">AAACH3icbVDLSsNAFJ3UV62vqEs3g0UQhJKIqAhC0Y3LCvYBSSyT6aQdOpmEmYlaQv7Ejb/ixoUi4q5/46QtUlsPDJw5517uvcePGZXKsoZGYWFxaXmluFpaW9/Y3DK3dxoySgQmdRyxSLR8JAmjnNQVVYy0YkFQ6DPS9PvXud98IELSiN+pQUy8EHU5DShGSktt89QNker5QfqYwYtLOPU7gkQI6CoaEvmrP2X3qUO9rG2WrYo1Apwn9oSUwQS1tvntdiKchIQrzJCUjm3FykuRUBQzkpXcRJIY4T7qEkdTjvRULx3dl8EDrXRgEAn9uIIjdbojRaGUg9DXlfmectbLxf88J1HBuZdSHieKcDweFCQMqgjmYcEOFQQrNtAEYUH1rhD3kEBY6UhLOgR79uR50jiu2JrfnpSrV5M4imAP7INDYIMzUAU3oAbqAINn8ArewYfxYrwZn8bXuLRgTHp2wR8Ywx9A/KMf</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 25

Ef cient Scienti c Computing:
Vectorization in Python

1. Brains and neuron models

Sebastian Raschka STAT 453: Intro to Deep Learning 26

fi
fi
Running Computations is a Big Part of Deep Learning!

https://www.youtube.com/watch?v=OTOGw0BRqK0

Image source: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRYlMXrK6UlC4lBvLKW4qfHgMUCbQLQ-vnvqA&usqp=C

Sebastian Raschka STAT 453: Intro to Deep Learning 27

Untitled Interlude: "Vectorization" in Python 1/26/19, 10)41 PM

Question for you: What are we computing here?

In [1]:

x0, x1, x2 = 1., 2., 3.

bias, w1, w2 = 0.1, 0.3, 0.5

x = [x0, x1, x2]

w = [bias, w1, w2]

A simple for-loop:
In [2]:

z = 0.
for i in range(len(x)):
z += x[i] * w[i]

print(z)

2.2

In [3]:

z = sum(x_i*w_i for x_i, w_i in zip(x, w))

print(z) Sebastian Raschka STAT 453: Intro to Deep Learning 28
In [1]:

Interlude: "Vectorization" in Python

x0, x1, x2 = 1., 2., 3.
bias, w1, w2 = 0.1, 0.3, 0.5

x = [x0, x1, x2]

w = [bias, w1, w2]
A simple for-loop:

In [2]:

z = 0.
for i in range(len(x)):
z += x[i] * w[i]

print(z)

2.2

A little bit better, list comprehensions:

In [3]:

z = sum(x_i*w_i for x_i, w_i in zip(x, w))

print(z)

2.2

In [4]:

import numpy as np Sebastian Raschka STAT 453: Intro to Deep Learning 29

for i in range(len(x)):
z += x[i] * w[i]

Interlude: "Vectorization" in Python

print(z)

2.2

list comprehensions (still sequential):

In [3]:

z = sum(x_i*w_i for x_i, w_i in zip(x, w))

print(z)

2.2

A vectorized implementation using NumPy:

In [4]:

import numpy as np

x_vec, w_vec = np.array(x), np.array(w)

z = (x_vec.transpose()).dot(w_vec)
print(z)

z = x_vec.dot(w_vec)
print(z)

2.2
2.2

Sebastian Raschka STAT 453: Intro to Deep Learning 30

Interlude: "Vectorization" in Python
vectorization-example-Copy1 slides 1/26/19, 11(17 PM

a)
In [5]: def forloop(x, w):
z = 0.
for i in range(len(x)):
z += x[i] * w[i]
return z

b) def listcomprehension(x, w):

return sum(x_i*w_i for x_i, w_i in zip(x, w))

c) def vectorized(x, w):

return x_vec.dot(w_vec)

x, w = np.random.rand(100000), np.random.rand(100000)
x_vec, w_vec = np.array(x), np.array(w)
Questions for you:
Which
In [6]: %timeit -r 100 -n 10one is the fastest?
forloop(x, w)
How much faster is the fastest one compared to the slowest one?
38.9 ms ± 1.32 ms per loop (mean ± std. dev. of 100 r
uns, 10 loopsSebastian
each)Raschka STAT 453: Intro to Deep Learning 31
def vectorized(x, w):

Interlude: "Vectorization" in Python

return x_vec.dot(w_vec)

x, w = np.random.rand(100000), np.random.rand(100000)
x_vec, w_vec = np.array(x), np.array(w)

In [6]: %timeit -r 100 -n 10 forloop(x, w)

38.9 ms ± 1.32 ms per loop (mean ± std. dev. of 100 r

uns, 10 loops each)

In [7]: %timeit -r 100 -n 10 listcomprehension(x, w)

29.7 ms ± 842 µs per loop (mean ± std. dev. of 100 ru

ns, 10 loops each)

In [8]: %timeit -r 100 -n 10 vectorized(x_vec, w_vec)

46.8 µs ± 8.07 µs per loop (mean ± std. dev. of 100 r

uns, 10 loops each)

file:///Users/sebastian/Desktop/vectorization-example-Copy1.slides.html Page 1 of 1

Sebastian Raschka STAT 453: Intro to Deep Learning 32

Interlude: Connections and Parallel Computation

Image Source: https://timedotcom. les.wordpress.com/

Image Source: https://fossbytes.com/wp-content/uploads/ 2014/05/brain.jpg?w=1100&quality=85
2017/05/nvidia-volta-v100-gpu.jpg

NVIDIA Volta with approx. 2.1 x 1010 transistors Brain with 1.6 x 1010 neurons
approx. only 10 connections per transistor 104 -105 connections per neuron
approx. 1015 connections in total

Sebastian Raschka STAT 453: Intro to Deep Learning 33

fi
Implementing a perceptron in
Python using NumPy and PyTorch

1. Brains and neuron models

Sebastian Raschka STAT 453: Intro to Deep Learning 34

Perceptron Learning Rule
Assume binary classi cation task, Perceptron nds decision boundary
if classes are separable

Sebastian Raschka STAT 453: Intro to Deep Learning 35

fi
fi
The Perceptron Learning Algorithm

• If correct: Do nothing if the prediction if output is equal to the target

• If incorrect, scenario a):

If output is 0 and target is 1, add input vector to weight vector

• If incorrect, scenario b):

If output is 1 and target is 0, subtract input vector from weight vector

Guaranteed to converge if a solution exists

(more about that later...)

Sebastian Raschka STAT 453: Intro to Deep Learning 36

The Perceptron Learning Algorithm

2. For every training epoch:

A. For every hx[i] , y [i] i 2 D : <latexit sha1_base64="KiRtzBWzB6OcgzuVeAuk4oyYAnU=">AAACJHicbZDLSsNAFIYn9VbrLerSzWARXEhJRFBwU9SFywr2Akksk+mkHTqZhJmJGEIexo2v4saFF1y48VmctFlo64GBj//8hznn92NGpbKsL6OysLi0vFJdra2tb2xumds7HRklApM2jlgkej6ShFFO2ooqRnqxICj0Gen648ui370nQtKI36o0Jl6IhpwGFCOlpb557jLEh4xAN0Rq5AfZQ36XOdTLj2CWlghdUXoon/owYtlV3jfrVsOaFJwHu4Q6KKvVN9/dQYSTkHCFGZLSsa1YeRkSimJG8pqbSBIjPEZD4mjkKCTSyyZH5vBAKwMYREI/ruBE/T2RoVDKNPS1s1hRzvYK8b+ek6jgzMsojxNFOJ5+FCQMqggWicEBFQQrlmpAWFC9K8QjJBBWOteaDsGePXkeOscNW/PNSb15UcZRBXtgHxwCG5yCJrgGLdAGGDyCZ/AK3own48X4MD6n1opRzuyCP2V8/wAUwKWr</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 37

Perceptron Code Examples

https://github.com/rasbt/stat453-deep-learning-ss21/blob/
master/L03

Sebastian Raschka STAT 453: Intro to Deep Learning 38

Why Do I "Make" You Understand
NumPy?

Sebastian Raschka STAT 453: Intro to Deep Learning 39

Sebastian Raschka STAT 453: Intro to Deep Learning 40
The perceptron convergence
theorem

1. Brains and neuron models

Sebastian Raschka STAT 453: Intro to Deep Learning 41

Perceptron Convergence Theorem
Let
D = (hx[1] , y [1] i, hx[2] , y [2] i, ..., hx[n] , y [n] i) 2 (Rm ⇥ {0, 1})n
<latexit sha1_base64="Wal6cJLWU5bzcyF5u646zqqA+4A=">AAADVXiclZLfbtMwFMZP0jFGB6ywS24sKiSQpiqpkOAGaYJd7HIguk1qs8lxndaa7UT2CaKK+lo8yG4m3oQbJE7dSvvTIYGjyJ+/851fEsd5pZXHJPkZxa2NB5sPtx61tx8/ebrTefb82Je1E3IgSl2605x7qZWVA1So5WnlJDe5lif5xadF/eSbdF6V9ivOKpkZPrGqUIIjWWXnB4zAAAeEKQiaNTRwAHNg8IHu11TV5FqY0CzJuU7nUFD2O2XPaB5CChnpPcrMbjkjcDcIe4Hxr8z+GrP/F2YvXP9Ht2t0ew/9TWAoWi/345qWU8+XQDMhg5Qy1OHDqoGE2CmpOTHOwLLt80436SVhsHWRrkQXVuPovHM5GpeiNtKi0Nz7YZpUmDXcoRJaztuj2suKiws+kUOSlhvpsyacijl7Rc6YFaWj2yIL7s2OhhvvZyanpOE49XdrC/O+2rDG4n3WKFvVKK1YPqioNcOSLY4YGysnBeoZCS6condlYsodF0gHsU2bkN795HVx3O+lpD+/7e5/XG3HFryAl/QLUngH+3AIRzAAEV1Gv+IojuOr+Hdro7W5jMbRqmcXbo3Wzh+cKM3Z</latexit>

8y [i] 2 D1 : y [i] = 1
and D1 [ D2 = D
<latexit sha1_base64="+NRK6kyZWKvaxCb9nW3Ascrxg9k=">AAACFXicbVDLSsNAFJ34rPUVdelmsAgupCQiKIJQ1IXLCvYBSQyT6bQdOpmEmYkQQn7Cjb/ixoUibgV3/o2TNoK2Hhg4nHMvc+4JYkalsqwvY25+YXFpubJSXV1b39g0t7bbMkoEJi0csUh0AyQJo5y0FFWMdGNBUBgw0glGl4XfuSdC0ojfqjQmXogGnPYpRkpLvnno9iOBGIPpXeZQL4cu5dANkRpixLKr3LfPfqxz2zdrVt0aA84SuyQ1UKLpm59uL8JJSLjCDEnp2FasvAwJRTEjedVNJIkRHqEBcTTlKCTSy8ZX5XBfKz2o4+nHFRyrvzcyFEqZhoGeLPLKaa8Q//OcRPVPvYzyOFGE48lH/YRBFcGiItijgmDFUk0QFlRnhXiIBMJKF1nVJdjTJ8+S9lHd1vzmuNa4KOuogF2wBw6ADU5AA1yDJmgBDB7AE3gBr8aj8Wy8Ge+T0Tmj3NkBf2B8fAOvu558</latexit>

[i] [i]
<latexit sha1_base64="xlHpeIXviTTPyARLsXqq4wNDExU=">AAACR3icdZHNS8MwGMbfzq+5+VH16CU4BE+zFUEvwlAPHie4D9jKSLN0C0vTkqTCKPvvvHj15r/gxYMiHs26HnSbL4T8eJ73JckTP+ZMacd5tQorq2vrG8XNUnlre2fX3ttvqiiRhDZIxCPZ9rGinAna0Exz2o4lxaHPacsf3Uz91iOVikXiQY9j6oV4IFjACNZGiuxT6EIIGDQMgZidQwq3MIEeuICMRyCBOKPlXWfGu/rPR+WeXXGqTlZoEdwcKpBXvWe/dPsRSUIqNOFYqY7rxNpLsdSMcDopdRNFY0xGeEA7BgUOqfLSLIcJOjZKHwWRNEtolKm/J1IcKjUOfdMZYj1U895UXOZ1Eh1ceikTcaKpILODgoQjHaFpqKjPJCWajw1gIpm5KyJDLDHRJvqSCcGdf/IiNM+qruH780rtOo+jCIdwBCfmKy6gBndQh4aJ9wne4AM+rWfr3fqyvmetBSufOYA/VbB+AOZdoDw=</latexit>

<latexit sha1_base64="4Q+84eB8n5BvndVOU3M3SSqYfNc=">AAACFXicbVDLSsNAFJ3UV62vqks3g0VwISUpgiIIRV24rGAfkMQwmU7aoZNJmJkIIeQn3Pgrblwo4lZw5984bSNo64GBwzn3MuceP2ZUKtP8MkoLi0vLK+XVytr6xuZWdXunI6NEYNLGEYtEz0eSMMpJW1HFSC8WBIU+I11/dDn2u/dESBrxW5XGxA3RgNOAYqS05FWPnCASiDGY3mU2dXPoUA6dEKkhRiy7yr3G2Y91bnrVmlk3J4DzxCpIDRRoedVPpx/hJCRcYYaktC0zVm6GhKKYkbziJJLECI/QgNiachQS6WaTq3J4oJU+1PH04wpO1N8bGQqlTENfT47zyllvLP7n2YkKTt2M8jhRhOPpR0HCoIrguCLYp4JgxVJNEBZUZ4V4iATCShdZ0SVYsyfPk06jbml+c1xrXhR1lMEe2AeHwAInoAmuQQu0AQYP4Am8gFfj0Xg23oz36WjJKHZ2wR8YH9+vxp58</latexit>
8y 2 D2 : y =0
Assume the input vectors come from two linearly
separable classes such that a feasible weight vector w *
exists.
The perceptron learning algorithm is guaranteed to
converge to a weight vector in the feasible region in a
nite number of iterations such that
[i] > [i]
8x 2 D1 : w x >0
[i] > [i]
8x 2 D2 : w x 0
Sebastian Raschka STAT 453: Intro to Deep Learning 42
fi
Perceptron Convergence Theorem -- Proof

Let us slightly rewrite the update rule (upon misclassi cation)

for convenience when we construct the proof:

w[i+1] = w[i] + x[i] if (w[i] )> x[i]  0 , x[i] 2 D1

[i+1] [i] [i] [i] > [i] [i]

w =w x if (w ) x > 0 ,x 2 D2

Here [i + 1] refers to the weight vector of the next

training example (that is, the weight after updating)

Sebastian Raschka STAT 453: Intro to Deep Learning 43

fi
Perceptron Convergence Theorem -- Proof
From the previous slide:
[i+1] [i] [i] [i] > [i] [i]
w =w +x if (w ) x  0 ,x 2 D1

We can rewrite this as follows:

[i+1] [0] [1] [i]

w
<latexit sha1_base64="Z1mdSMoxFT1+HKPi48NHgAmxJ1A=">AAACOnicbZDLSsNAFIYn9VbrLerSzWARhEJIRNCNUHTjsgV7gTSWyXTSDp1cmJmoJeS53PgU7ly4caGIWx/ASRrEth4Y+Pn+c5hzfjdiVEjTfNFKS8srq2vl9crG5tb2jr671xZhzDFp4ZCFvOsiQRgNSEtSyUg34gT5LiMdd3yV+Z07wgUNgxs5iYjjo2FAPYqRVKivN3s+kiPXS+7T28SmNctJ4QWcgaZCtV/0kCErR4ZhzBvUSft61TTMvOCisApRBUU1+vpzbxDi2CeBxAwJYVtmJJ0EcUkxI2mlFwsSITxGQ2IrGSCfCCfJT0/hkSID6IVcvUDCnP6dSJAvxMR3VWe2p5j3MvifZ8fSO3cSGkSxJAGefuTFDMoQZjnCAeUESzZRAmFO1a4QjxBHWKq0KyoEa/7kRdE+MSylm6fV+mURRxkcgENwDCxwBurgGjRAC2DwCF7BO/jQnrQ37VP7mraWtGJmH8yU9v0D6IGs2g==</latexit>
=w +x + ... + x

Also, we can drop this term if we

initialize the weight vector as 0m <latexit sha1_base64="mZaBmXIv9kAQNbyEPbV3Hk650Jw=">AAAB6nicbZBNSwMxEIZn/az1q+rRS7AInsquCHosevFY0X5Au5ZsOtuGJtklyQql9Cd48aCIV3+RN/+NabsHbX0h8PDODJl5o1RwY33/21tZXVvf2CxsFbd3dvf2SweHDZNkmmGdJSLRrYgaFFxh3XIrsJVqpDIS2IyGN9N68wm14Yl6sKMUQ0n7isecUeuse/9Rdktlv+LPRJYhyKEMuWrd0lenl7BMorJMUGPagZ/acEy15UzgpNjJDKaUDWkf2w4VlWjC8WzVCTl1To/EiXZPWTJzf0+MqTRmJCPXKakdmMXa1Pyv1s5sfBWOuUozi4rNP4ozQWxCpneTHtfIrBg5oExztythA6opsy6dogshWDx5GRrnlcDx3UW5ep3HUYBjOIEzCOASqnALNagDgz48wyu8ecJ78d69j3nripfPHMEfeZ8/916Nkw==</latexit>

[i+1] [1] [i]

<latexit sha1_base64="ivs2DyrG6dK+k+1lgM+dHFRqra8=">AAACJ3icbZBdS8MwFIbT+TXnV9VLb4JDEAalFUFvlKE3Xk5wH7DVkWbpFpamJUnVUfpvvPGveCOoiF76T0y3Cm7zQMLL8+aQc14vYlQq2/4yCguLS8srxdXS2vrG5pa5vdOQYSwwqeOQhaLlIUkY5aSuqGKkFQmCAo+Rpje8zPzmHRGShvxGjSLiBqjPqU8xUhp1zfNOgNTA85P79DZp04rjpvAM/sKHDGaoAi3L0veUQd20a5Ztyx4XnBdOLsogr1rXfO30QhwHhCvMkJRtx46UmyChKGYkLXViSSKEh6hP2lpyFBDpJuM9U3igSQ/6odCHKzimfzsSFEg5Cjz9MptTznoZ/M9rx8o/dRPKo1gRjicf+TGDKoRZaLBHBcGKjbRAWFA9K8QDJBBWOtqSDsGZXXleNI4sR+vr43L1Io+jCPbAPjgEDjgBVXAFaqAOMHgEz+ANvBtPxovxYXxOnhaMvGcXTJXx/QO6EqU2</latexit>
w =x + ... + x

Sebastian Raschka STAT 453: Intro to Deep Learning 44

Perceptron Convergence Theorem -- Proof
From the previous slide, the update rule:
[i+1] [1] [i]
<latexit sha1_base64="ivs2DyrG6dK+k+1lgM+dHFRqra8=">AAACJ3icbZBdS8MwFIbT+TXnV9VLb4JDEAalFUFvlKE3Xk5wH7DVkWbpFpamJUnVUfpvvPGveCOoiF76T0y3Cm7zQMLL8+aQc14vYlQq2/4yCguLS8srxdXS2vrG5pa5vdOQYSwwqeOQhaLlIUkY5aSuqGKkFQmCAo+Rpje8zPzmHRGShvxGjSLiBqjPqU8xUhp1zfNOgNTA85P79DZp04rjpvAM/sKHDGaoAi3L0veUQd20a5Ztyx4XnBdOLsogr1rXfO30QhwHhCvMkJRtx46UmyChKGYkLXViSSKEh6hP2lpyFBDpJuM9U3igSQ/6odCHKzimfzsSFEg5Cjz9MptTznoZ/M9rx8o/dRPKo1gRjicf+TGDKoRZaLBHBcGKjbRAWFA9K8QDJBBWOtqSDsGZXXleNI4sR+vr43L1Io+jCPbAPjgEDjgBVXAFaqAOMHgEz+ANvBtPxovxYXxOnhaMvGcXTJXx/QO6EqU2</latexit>
w =x + ... + x
Let's multiply both sides byw⇤ : <latexit sha1_base64="0F8Uwu542wH44M1cD9DXEpHQmR0=">AAAB83icbVDLSgMxFL1TX7W+qi7dBIsgLsqMCLosunFZwT6gM5ZMmmlDM5mQZJQy9DfcuFDErT/jzr8x085CWw8EDufcyz05oeRMG9f9dkorq2vrG+XNytb2zu5edf+grZNUEdoiCU9UN8SaciZoyzDDaVcqiuOQ0044vsn9ziNVmiXi3kwkDWI8FCxiBBsr+X6MzSiMsqfpw1m/WnPr7gxomXgFqUGBZr/65Q8SksZUGMKx1j3PlSbIsDKMcDqt+KmmEpMxHtKepQLHVAfZLPMUnVhlgKJE2ScMmqm/NzIcaz2JQzuZZ9SLXi7+5/VSE10FGRMyNVSQ+aEo5cgkKC8ADZiixPCJJZgoZrMiMsIKE2NrqtgSvMUvL5P2ed2z/O6i1rgu6ijDERzDKXhwCQ24hSa0gICEZ3iFNyd1Xpx352M+WnKKnUP4A+fzBxy6kbc=</latexit>

⇤ T [i+1] ⇤ T [1] ⇤ T [i]

(w ) w
<latexit sha1_base64="2bagsj4kiE4Bp4QcNqvpbc577yY=">AAACX3icfZHNS8MwGMbT+rG5zVn1JF6CQ5gOSiuCXoShF48T9gVbN9Is3cLSD5J0Okr/SW+CF/8T022HfYgvJDw87+8lyRM3YlRIy/rS9L39g8Nc/qhQLB2XT4zTs7YIY45JC4cs5F0XCcJoQFqSSka6ESfIdxnpuNOXrN+ZES5oGDTlPCKOj8YB9ShGUllDY1bt+0hOXC95TwfJbXozaMJ1p0drtpPCJ/gP95FxGVWDpmmqfZfdQKmTDo2KZVqLgrvCXokKWFVjaHz2RyGOfRJIzJAQPduKpJMgLilmJC30Y0EihKdoTHpKBsgnwkkW+aTwWjkj6IVcrUDChbs+kSBfiLnvKjK7p9juZeZfvV4svUcnoUEUSxLg5UFezKAMYRY2HFFOsGRzJRDmVN0V4gniCEv1JQUVgr395F3RvjNtpd/uK/XnVRx5cAmuQBXY4AHUwStogBbA4FvTtaJW0n70nF7WjSWqa6uZc7BR+sUvLo607Q==</latexit>
= (w ) x + ... + (w ) x

All these terms are > 0, because remember that we have

w[i+1] = w[i] + x[i] if (w[i] )T x[i]  0 , x[i] 2 D1
<latexit sha1_base64="VqXIKUaMsmh1IsyUEjebniqF/eE=">AAACh3icbZHbThsxEIa9Cy0kPQV62RuLqFLaVHQXECBVSOFw0UsqkYOU3UReZxYsvN6VPQtEq30VHqp3vA3OAQQhI1n6/c388ngmyqQw6HkPjruy+u792nql+uHjp89fahubHZPmmkObpzLVvYgZkEJBGwVK6GUaWBJJ6EbXp5N89wa0Eam6wHEGYcIulYgFZ2jRsHYfJAyvori4LQdFXzT9sKRH9DW0qPmM7p5Q8IcGCHdYUBHT6ZU2Fn0/BhdLjBKoZ+t/LUkJNYOcyeKsHPrDWt3b9qZB3wp/LupkHufD2v9glPI8AYVcMmP6vpdhWDCNgksoq0FuIGP8ml1C30rFEjBhMZ1jSb9bMqJxqu1RSKf0paNgiTHjJLKVkybNYm4Cl+X6OcaHYSFUliMoPnsoziXFlE6WQkdCA0c5toJxLWyvlF8xzTja1VXtEPzFL78VnZ1t3+p/e/XWyXwc6+Qb2SIN4pMD0iJ/yTlpE+6sOj+dXWfPrbi/3X33cFbqOnPPV/Iq3ONHr7LDjQ==</latexit>

so the updates are all to make the net inputs more positive

⇤ T [j]
Now, let ↵ = min (w ) x , j = 1, ..., i
[j] <latexit sha1_base64="BFdhenkfmyvzJR7u9fgzL6/qK8M=">AAACQnicbVDLSgMxFM34tr6qLt0Ei6BShhkRFEQounFZwdpKZ1oyacamZjJDklFLmG9z4xe48wPcuFDErQszbQVfBwKHc87l3pwgYVQqx3m0xsYnJqemZ2YLc/MLi0vF5ZVzGacCkxqOWSwaAZKEUU5qiipGGokgKAoYqQdXx7lfvyZC0pifqX5C/AhdchpSjJSR2sULD7Gki+Ah9FLeMUGi9G1LN3t+lmkvohxm0DuAm16EVDcI9U3W0tvZVusMfim32TBeznO9Q7cMbdsuQ9oulhzbGQD+Je6IlMAI1XbxwevEOI0IV5ghKZuukyhfI6EoZiQreKkkCcJX6JI0DeUoItLXgwoyuGGUDgxjYR5XcKB+n9AokrIfBSaZ3y1/e7n4n9dMVbjva8qTVBGOh4vClEEVw7xP2KGCYMX6hiAsqLkV4i4SCCtTZsGU4P7+8l9yvmO7hp/ulipHozpmwBpYB5vABXugAk5AFdQABnfgCbyAV+veerberPdhdMwazayCH7A+PgFOza89</latexit>
x

then (w⇤ )T w[i+1]

<latexit sha1_base64="kBTXL4xEo91UMh4mpttWC3dVfLU=">AAACG3icbZDLSsNAFIYnXmu9RV26GSxCVShJEXRZdOOyQm/QpGUynbRDJ5MwM1FKyHu48VXcuFDEleDCt3HSZlFbfxj4+M85zDm/FzEqlWX9GCura+sbm4Wt4vbO7t6+eXDYkmEsMGnikIWi4yFJGOWkqahipBMJggKPkbY3vs3q7QciJA15Q00i4gZoyKlPMVLa6pvVshMgNfL85DHtJefpWa8B550uvbDdFDpDAh3EohGCtG+WrIo1FVwGO4cSyFXvm1/OIMRxQLjCDEnZta1IuQkSimJG0qITSxIhPEZD0tXIUUCkm0xvS+GpdgbQD4V+XMGpOz+RoEDKSeDpzmxtuVjLzP9q3Vj5125CeRQrwvHsIz9mUIUwCwoOqCBYsYkGhAXVu0I8QgJhpeMs6hDsxZOXoVWt2JrvL0u1mzyOAjgGJ6AMbHAFauAO1EETYPAEXsAbeDeejVfjw/icta4Y+cwR+CPj+xc9d6Di</latexit>
↵i
Sebastian Raschka STAT 453: Intro to Deep Learning 45
Perceptron Convergence Theorem -- Proof
From the previous slide, we had the inequality:

⇤ T [i+1]
(w ) w
<latexit sha1_base64="kBTXL4xEo91UMh4mpttWC3dVfLU=">AAACG3icbZDLSsNAFIYnXmu9RV26GSxCVShJEXRZdOOyQm/QpGUynbRDJ5MwM1FKyHu48VXcuFDEleDCt3HSZlFbfxj4+M85zDm/FzEqlWX9GCura+sbm4Wt4vbO7t6+eXDYkmEsMGnikIWi4yFJGOWkqahipBMJggKPkbY3vs3q7QciJA15Q00i4gZoyKlPMVLa6pvVshMgNfL85DHtJefpWa8B550uvbDdFDpDAh3EohGCtG+WrIo1FVwGO4cSyFXvm1/OIMRxQLjCDEnZta1IuQkSimJG0qITSxIhPEZD0tXIUUCkm0xvS+GpdgbQD4V+XMGpOz+RoEDKSeDpzmxtuVjLzP9q3Vj5125CeRQrwvHsIz9mUIUwCwoOqCBYsYkGhAXVu0I8QgJhpeMs6hDsxZOXoVWt2JrvL0u1mzyOAjgGJ6AMbHAFauAO1EETYPAEXsAbeDeejVfjw/icta4Y+cwR+CPj+xc9d6Di</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="EQP/6SMwMgPcCD0u9uAOVI1JrVY=">AAACEHicbZBLSwMxFIXv+Ky16ujWTbAIVaHMdKNLwY3LCn3BzLRk0kwbmnmQZJQyzP9w419x40IRV+78N2baLmrrgcDHOQm59/gJZ1JZ1o+xsbm1vbNb2ivvVw4Oj8zjSkfGqSC0TWIei56PJeUsom3FFKe9RFAc+px2/cldkXcfqZAsjlpqmlAvxKOIBYxgpa2B2ai5IVZjP8ie8n52mV/0W2jZcdiV7eXIHVHkYp6MMWIDs2rVrZnQOtgLqMJCzYH57Q5jkoY0UoRjKR3bSpSXYaEY4TQvu6mkCSYTPKKOxgiHVHrZbLccnWtniIJY6BMpNHOXX2Q4lHIa+vpmMbZczQrzv8xJVXDjZSxKUkUjMv8oSDlSMSqKQkMmKFF8qgETwfSsiIyxwETpOsu6BHt15XXoNOq25gcLSnAKZ1ADG67hFu6hCW0g8Ayv8A4fxovxZnzO69owFr2dwB8ZX79XCp9c</latexit>
sha1_base64="YwT+534rIr3qtNvZpveMSBhmVQ8=">AAACG3icbZDLSsNAFIYn9VbrLerSzWARqkJJutFl0Y3LCr1Bk5bJdNIOnUzCzEQpIe/hxldx40IRV4IL38ZJm0Vt/WHg4z/nMOf8XsSoVJb1YxTW1jc2t4rbpZ3dvf0D8/CoLcNYYNLCIQtF10OSMMpJS1HFSDcSBAUeIx1vcpvVOw9ESBrypppGxA3QiFOfYqS0NTBrFSdAauz5yWPaTy7S834TLjo9emm7KXRGBDqIRWME6cAsW1VrJrgKdg5lkKsxML+cYYjjgHCFGZKyZ1uRchMkFMWMpCUnliRCeIJGpKeRo4BIN5ndlsIz7QyhHwr9uIIzd3EiQYGU08DTndnacrmWmf/VerHyr92E8ihWhOP5R37MoAphFhQcUkGwYlMNCAuqd4V4jATCSsdZ0iHYyyevQrtWtTXfW+X6TR5HEZyAU1ABNrgCdXAHGqAFMHgCL+ANvBvPxqvxYXzOWwtGPnMM/sj4/gU8N6De</latexit>
↵i
(

Using the Cauchy-Schwarz inequality, we can then say

⇤ 2 [i+1] 2 ⇤ T [i+1] 2
||w || · ||w <latexit sha1_base64="YSgoXbrhxve+hE4EKORsHzsMzEU=">AAACVXicbVFNS8MwGE7rnHN+rOrRS3AIm8Joh6DHoRePE/YFazfSNO3C0g+SVBnd/uQu4j/xIph1O7jNFwIPzwd588RNGBXSNL80/aBwWDwqHZdPTs/OK8bFZU/EKceki2MW84GLBGE0Il1JJSODhBMUuoz03enLSu+/Ey5oHHXkLCFOiIKI+hQjqaixweZzO0Ry4vrZx2KU3S3m81ET2tiLJdyWhvTectYytAMCbZcGtdp2uD7qwP1Mbq2r3Niomg0zH7gPrA2ogs20x8bS9mKchiSSmCEhhpaZSCdDXFLMyKJsp4IkCE9RQIYKRigkwsnyVhbwVjEe9GOuTiRhzv5NZCgUYha6yrlaWexqK/I/bZhK/8nJaJSkkkR4fZGfMihjuKoYepQTLNlMAYQ5VbtCPEEcYak+oqxKsHafvA96zYal8NtDtfW8qaMErsENqAELPIIWeAVt0AUYLMG3pmm69qn96AW9uLbq2iZzBbZGr/wConCzsA==</latexit>
|| (w ) w
as well as

||w⇤ ||2 · ||w[i+1] ||2

<latexit sha1_base64="hMjsgpvB8oZtDsRUn7dAH4HagUc=">AAACLnicbZDLSsNAFIYnXmu9RV26GSxCVShJEXRZFMFlBXuBJC2T6aQdOpmEmYlS0j6RG19FF4KKuPUxnLZZ2NYDAx//fw5zzu/HjEplWe/G0vLK6tp6biO/ubW9s2vu7ddllAhMajhikWj6SBJGOakpqhhpxoKg0Gek4fevx37jgQhJI36vBjHxQtTlNKAYKS21zZvh0A2R6vlB+jhqpaej4bBVhi7uRArOWg49s72pDd0ugUUXsbiHID1pldtmwSpZk4KLYGdQAFlV2+ar24lwEhKuMENSOrYVKy9FQlHMyCjvJpLECPdRlzgaOQqJ9NLJuSN4rJUODCKhH1dwov6dSFEo5SD0ded4fznvjcX/PCdRwaWXUh4ninA8/ShIGFQRHGcHO1QQrNhAA8KC6l0h7iGBsNIJ53UI9vzJi1Avl2zNd+eFylUWRw4cgiNQBDa4ABVwC6qgBjB4Ai/gA3waz8ab8WV8T1uXjGzmAMyU8fMLpcaoxA==</latexit>
(↵i)2

So, we can nally de ne the lower bound of the size of the weights
↵ 2 i2
||w[i+1] ||2
<latexit sha1_base64="Ksl5iqQyUNnihiWifLlV5Wm7I/s=">AAACMHicbVDLSsNAFJ34rPUVdelmsAiiUJIi6LLoQpcV7AOatEymk3bo5MHMRClJPsmNn6IbBUXc+hVO0ixs64GBwznnMvceJ2RUSMN415aWV1bX1ksb5c2t7Z1dfW+/JYKIY9LEAQt4x0GCMOqTpqSSkU7ICfIcRtrO+Drz2w+ECxr493ISEttDQ5+6FCOppL5+kySWh+TIcePHtBd36Zlpp0nSq0FoDQm0XI5wbCEWjpDSaK+WxrMTp3k67esVo2rkgIvELEgFFGj09RdrEODII77EDAnRNY1Q2jHikmJG0rIVCRIiPEZD0lXURx4RdpwfnMJjpQygG3D1fAlz9e9EjDwhJp6jktmqYt7LxP+8biTdSzumfhhJ4uPpR27EoAxg1h4cUE6wZBNFEOZU7QrxCKmOpOq4rEow509eJK1a1VT87rxSvyrqKIFDcAROgAkuQB3cggZoAgyewCv4AJ/as/amfWnf0+iSVswcgBloP7/qHKq5</latexit>
||w⇤ ||2

Sebastian Raschka STAT 453: Intro to Deep Learning 46

fi
fi
Perceptron Convergence Theorem -- Proof
Now that we de ned the lower bound of the size of the weights,
let us get the upper bound.

For that, let's go back to the update rule

[i+1] [i] [i] [i] T [i] [i]

w
<latexit sha1_base64="VqXIKUaMsmh1IsyUEjebniqF/eE=">AAACh3icbZHbThsxEIa9Cy0kPQV62RuLqFLaVHQXECBVSOFw0UsqkYOU3UReZxYsvN6VPQtEq30VHqp3vA3OAQQhI1n6/c388ngmyqQw6HkPjruy+u792nql+uHjp89fahubHZPmmkObpzLVvYgZkEJBGwVK6GUaWBJJ6EbXp5N89wa0Eam6wHEGYcIulYgFZ2jRsHYfJAyvori4LQdFXzT9sKRH9DW0qPmM7p5Q8IcGCHdYUBHT6ZU2Fn0/BhdLjBKoZ+t/LUkJNYOcyeKsHPrDWt3b9qZB3wp/LupkHufD2v9glPI8AYVcMmP6vpdhWDCNgksoq0FuIGP8ml1C30rFEjBhMZ1jSb9bMqJxqu1RSKf0paNgiTHjJLKVkybNYm4Cl+X6OcaHYSFUliMoPnsoziXFlE6WQkdCA0c5toJxLWyvlF8xzTja1VXtEPzFL78VnZ1t3+p/e/XWyXwc6+Qb2SIN4pMD0iJ/yTlpE+6sOj+dXWfPrbi/3X33cFbqOnPPV/Iq3ONHr7LDjQ==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="q+kI6dZtWxcZyJcuwi9H/sBseUI=">AAACfHicbZHbSsQwEIbTetpdT6u33gRF8IS2KiqIIOiFlwquCtu6pNnpGkzTkkzVpfRVfCjvfBuzB0XXHQj8+f4ZMpmJMikMet6n405MTk3PVKq12bn5hcX60tydSXPNocFTmeqHiBmQQkEDBUp4yDSwJJJwHz1f9Pz7F9BGpOoWuxmECesoEQvO0KJW/T1IGD5FcfFaPhZNse2HJT2jf6FF2z/o7RsFpzRAeMOCipj2r3RjtG7z8XZMoQTq2fydMZZQA8iZLC7Llt+qr3m7Xj/of+EPxRoZxnWr/hG0U54noJBLZkzT9zIMC6ZRcAllLcgNZIw/sw40rVQsARMW/TmWdN2SNo1TbY9C2qe/KwqWGNNNIpvZa9KMej04zmvmGJ+EhVBZjqD44KE4lxRT2lsKbQsNHGXXCsa1sL1S/sQ042hXV7ND8Ee//F/c7e/6Vt94pEJWyCrZID45JufkilyTBuHOpLPlHDiHbtXdc48G43Kd4dyWyZ9wT74AK3jCmg==</latexit>
sha1_base64="QmDFYUNY6J1FfiHSnLcgP1REaN4=">AAACh3icbZHbahsxEIa1W7eN3ZPTXuZG1BTcuji7bWgDpeAeLnqZQnwA79po5VlbWKtdpNnEZtlX6UPlLm8T+dBSOx4Q/PpmfjSaiTIpDHrereM+qDx89PioWnvy9NnzF/Xjlz2T5ppDl6cy1YOIGZBCQRcFShhkGlgSSehH8x+rfP8KtBGpusRlBmHCpkrEgjO0aFz/EyQMZ1FcXJejYihafljSr3QXWtT6hxZ/UfCFBggLLKiI6fpKm/u+t6PLA0YJ1LP17w+khNpAzmTxsxz743rDa3vroPeFvxUNso2Lcf0mmKQ8T0Ahl8yYoe9lGBZMo+ASylqQG8gYn7MpDK1ULAETFus5lvSNJRMap9oehXRN/3cULDFmmUS2ctWk2c+t4KHcMMf4PCyEynIExTcPxbmkmNLVUuhEaOAol1YwroXtlfIZ04yjXV3NDsHf//J90fvQ9q3+7TU637fjOCIn5DVpEp98Jh3yi1yQLuFOxXnnfHTO3Kp76n5yzzelrrP1vCI74X67A65yw4k=</latexit>
=w +x if (w ) x  0 ,x 2 D1

and apply the squared L2 norm on both sides

[i+1] 2 [i] [i] 2
||w
<latexit sha1_base64="mh6nOSvUgtWXLeXFDyjggPq2Oe4=">AAACLXicbVDLSsNAFJ34rPUVdelmsAhCoSRF0I1Q1IXLCvYBaVom00k7dPJgZqKWJD/kxl8RwUVF3PobTtoI2npg4HDOucy9xwkZFdIwJtrS8srq2npho7i5tb2zq+/tN0UQcUwaOGABbztIEEZ90pBUMtIOOUGew0jLGV1lfuuecEED/06OQ2J7aOBTl2IkldTTr5Ok4yE5dNz4Ie3GFi2bdpok3Sq8gPOWncIy/JEecynL9vSSUTGmgIvEzEkJ5Kj39NdOP8CRR3yJGRLCMo1Q2jHikmJG0mInEiREeIQGxFLURx4Rdjy9NoXHSulDN+Dq+RJO1d8TMfKEGHuOSma7inkvE//zrEi653ZM/TCSxMezj9yIQRnArDrYp5xgycaKIMyp2hXiIeIIS1VwUZVgzp+8SJrViqn47WmpdpnXUQCH4AicABOcgRq4AXXQABg8gRcwAe/as/amfWifs+iSls8cgD/Qvr4B2/2ptA==</latexit>
|| = ||w + x ||
= ||w[i] ||2 + 2(x[i] )T w[i] + ||x[i] ||2
<latexit sha1_base64="WrQgutLGBRW3hS2PwnaeJJGEzrA=">AAACRXicbVBJSwMxGM241rpVPXoJFqFSKDNF0ItQ9OKxQjeYmZZMmmlDMwtJRlum/XNevHvzH3jxoIhXzbQj1NYHgcd735bnhIwKqesv2srq2vrGZmYru72zu7efOzhsiCDimNRxwALecpAgjPqkLqlkpBVygjyHkaYzuEn85j3hggZ+TY5CYnuo51OXYiSV1MlZV3A8tjwk+44bP0zasUntyXjcLsMiLMPCrzNMnbN2DcLF8iKcmzGcm9HJ5fWSPgVcJkZK8iBFtZN7troBjjziS8yQEKahh9KOEZcUMzLJWpEgIcID1COmoj7yiLDjaQoTeKqULnQDrp4v4VSd74iRJ8TIc1RlcqtY9BLxP8+MpHtpx9QPI0l8PFvkRgzKACaRwi7lBEs2UgRhTtWtEPcRR1iq4LMqBGPxy8ukUS4Zit+d5yvXaRwZcAxOQAEY4AJUwC2ogjrA4BG8gnfwoT1pb9qn9jUrXdHSniPwB9r3D2+Tst0=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 47

fi
Perceptron Convergence Theorem -- Proof
Now that we de ned the lower bound of the size of the weights,
let us get the upper bound.

For that, let's go back to the update rule

[i+1] [i] [i] [i] T [i] [i]
w
<latexit sha1_base64="VqXIKUaMsmh1IsyUEjebniqF/eE=">AAACh3icbZHbThsxEIa9Cy0kPQV62RuLqFLaVHQXECBVSOFw0UsqkYOU3UReZxYsvN6VPQtEq30VHqp3vA3OAQQhI1n6/c388ngmyqQw6HkPjruy+u792nql+uHjp89fahubHZPmmkObpzLVvYgZkEJBGwVK6GUaWBJJ6EbXp5N89wa0Eam6wHEGYcIulYgFZ2jRsHYfJAyvori4LQdFXzT9sKRH9DW0qPmM7p5Q8IcGCHdYUBHT6ZU2Fn0/BhdLjBKoZ+t/LUkJNYOcyeKsHPrDWt3b9qZB3wp/LupkHufD2v9glPI8AYVcMmP6vpdhWDCNgksoq0FuIGP8ml1C30rFEjBhMZ1jSb9bMqJxqu1RSKf0paNgiTHjJLKVkybNYm4Cl+X6OcaHYSFUliMoPnsoziXFlE6WQkdCA0c5toJxLWyvlF8xzTja1VXtEPzFL78VnZ1t3+p/e/XWyXwc6+Qb2SIN4pMD0iJ/yTlpE+6sOj+dXWfPrbi/3X33cFbqOnPPV/Iq3ONHr7LDjQ==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="q+kI6dZtWxcZyJcuwi9H/sBseUI=">AAACfHicbZHbSsQwEIbTetpdT6u33gRF8IS2KiqIIOiFlwquCtu6pNnpGkzTkkzVpfRVfCjvfBuzB0XXHQj8+f4ZMpmJMikMet6n405MTk3PVKq12bn5hcX60tydSXPNocFTmeqHiBmQQkEDBUp4yDSwJJJwHz1f9Pz7F9BGpOoWuxmECesoEQvO0KJW/T1IGD5FcfFaPhZNse2HJT2jf6FF2z/o7RsFpzRAeMOCipj2r3RjtG7z8XZMoQTq2fydMZZQA8iZLC7Llt+qr3m7Xj/of+EPxRoZxnWr/hG0U54noJBLZkzT9zIMC6ZRcAllLcgNZIw/sw40rVQsARMW/TmWdN2SNo1TbY9C2qe/KwqWGNNNIpvZa9KMej04zmvmGJ+EhVBZjqD44KE4lxRT2lsKbQsNHGXXCsa1sL1S/sQ042hXV7ND8Ee//F/c7e/6Vt94pEJWyCrZID45JufkilyTBuHOpLPlHDiHbtXdc48G43Kd4dyWyZ9wT74AK3jCmg==</latexit>
sha1_base64="QmDFYUNY6J1FfiHSnLcgP1REaN4=">AAACh3icbZHbahsxEIa1W7eN3ZPTXuZG1BTcuji7bWgDpeAeLnqZQnwA79po5VlbWKtdpNnEZtlX6UPlLm8T+dBSOx4Q/PpmfjSaiTIpDHrereM+qDx89PioWnvy9NnzF/Xjlz2T5ppDl6cy1YOIGZBCQRcFShhkGlgSSehH8x+rfP8KtBGpusRlBmHCpkrEgjO0aFz/EyQMZ1FcXJejYihafljSr3QXWtT6hxZ/UfCFBggLLKiI6fpKm/u+t6PLA0YJ1LP17w+khNpAzmTxsxz743rDa3vroPeFvxUNso2Lcf0mmKQ8T0Ahl8yYoe9lGBZMo+ASylqQG8gYn7MpDK1ULAETFus5lvSNJRMap9oehXRN/3cULDFmmUS2ctWk2c+t4KHcMMf4PCyEynIExTcPxbmkmNLVUuhEaOAol1YwroXtlfIZ04yjXV3NDsHf//J90fvQ9q3+7TU637fjOCIn5DVpEp98Jh3yi1yQLuFOxXnnfHTO3Kp76n5yzzelrrP1vCI74X67A65yw4k=</latexit>
=w +x if (w ) x  0 ,x 2 D1

and apply the squared L2 norm on both sides

[i+1] 2 [i] [i] 2
||w <latexit sha1_base64="mh6nOSvUgtWXLeXFDyjggPq2Oe4=">AAACLXicbVDLSsNAFJ34rPUVdelmsAhCoSRF0I1Q1IXLCvYBaVom00k7dPJgZqKWJD/kxl8RwUVF3PobTtoI2npg4HDOucy9xwkZFdIwJtrS8srq2npho7i5tb2zq+/tN0UQcUwaOGABbztIEEZ90pBUMtIOOUGew0jLGV1lfuuecEED/06OQ2J7aOBTl2IkldTTr5Ok4yE5dNz4Ie3GFi2bdpok3Sq8gPOWncIy/JEecynL9vSSUTGmgIvEzEkJ5Kj39NdOP8CRR3yJGRLCMo1Q2jHikmJG0mInEiREeIQGxFLURx4Rdjy9NoXHSulDN+Dq+RJO1d8TMfKEGHuOSma7inkvE//zrEi653ZM/TCSxMezj9yIQRnArDrYp5xgycaKIMyp2hXiIeIIS1VwUZVgzp+8SJrViqn47WmpdpnXUQCH4AicABOcgRq4AXXQABg8gRcwAe/as/amfWifs+iSls8cgD/Qvr4B2/2ptA==</latexit>
|| = ||w + x ||
= ||w[i] ||2 + 2(x[i] )T w[i] + ||x[i] ||2
<latexit sha1_base64="WrQgutLGBRW3hS2PwnaeJJGEzrA=">AAACRXicbVBJSwMxGM241rpVPXoJFqFSKDNF0ItQ9OKxQjeYmZZMmmlDMwtJRlum/XNevHvzH3jxoIhXzbQj1NYHgcd735bnhIwKqesv2srq2vrGZmYru72zu7efOzhsiCDimNRxwALecpAgjPqkLqlkpBVygjyHkaYzuEn85j3hggZ+TY5CYnuo51OXYiSV1MlZV3A8tjwk+44bP0zasUntyXjcLsMiLMPCrzNMnbN2DcLF8iKcmzGcm9HJ5fWSPgVcJkZK8iBFtZN7troBjjziS8yQEKahh9KOEZcUMzLJWpEgIcID1COmoj7yiLDjaQoTeKqULnQDrp4v4VSd74iRJ8TIc1RlcqtY9BLxP8+MpHtpx9QPI0l8PFvkRgzKACaRwi7lBEs2UgRhTtWtEPcRR1iq4LMqBGPxy8ukUS4Zit+d5yvXaRwZcAxOQAEY4AJUwC2ogjrA4BG8gnfwoT1pb9qn9jUrXdHSniPwB9r3D2+Tst0=</latexit>

Leads to
[i+1] 2 [i] 2 [i] 2
||w
<latexit sha1_base64="cRZ2ZhxSmnvkG8rX3SFwDNCiryU=">AAACNXicbZDLSsNAFIYn9VbrLerSzWARhEJJiqDLohsXLirYCzRpmUwn7dDJhZmJWpK8lBvfw5UuXCji1ldw0mZRWw8M/PzfOcw5vxMyKqRhvGmFldW19Y3iZmlre2d3T98/aIkg4pg0ccAC3nGQIIz6pCmpZKQTcoI8h5G2M77KePuecEED/05OQmJ7aOhTl2IkldXXb5LE8pAcOW78kPbiLq2YdpokvRq0GIGLMEeVOfA4B/p62aga04LLwsxFGeTV6Osv1iDAkUd8iRkSomsaobRjxCXFjKQlKxIkRHiMhqSrpI88Iux4enUKT5QzgG7A1fMlnLrzEzHyhJh4jurMdhWLLDP/Y91Iuhd2TP0wksTHs4/ciEEZwCxCOKCcYMkmSiDMqdoV4hHiCEsVdEmFYC6evCxataqp9O1ZuX6Zx1EER+AYnAITnIM6uAYN0AQYPIFX8AE+tWftXfvSvmetBS2fOQR/Svv5BShTrXQ=</latexit>
||  ||w || + ||x ||

Sebastian Raschka STAT 453: Intro to Deep Learning 48

fi
Perceptron Convergence Theorem -- Proof
Now that we de ned the lower bound of the size of the weights,
let us get the upper bound.

For that, let's go back to the update rule

[i+1] [i] [i] [i] T [i] [i]
w =w +x if (w ) x  0 ,x 2 D1

(
<latexit sha1_base64="VqXIKUaMsmh1IsyUEjebniqF/eE=">AAACh3icbZHbThsxEIa9Cy0kPQV62RuLqFLaVHQXECBVSOFw0UsqkYOU3UReZxYsvN6VPQtEq30VHqp3vA3OAQQhI1n6/c388ngmyqQw6HkPjruy+u792nql+uHjp89fahubHZPmmkObpzLVvYgZkEJBGwVK6GUaWBJJ6EbXp5N89wa0Eam6wHEGYcIulYgFZ2jRsHYfJAyvori4LQdFXzT9sKRH9DW0qPmM7p5Q8IcGCHdYUBHT6ZU2Fn0/BhdLjBKoZ+t/LUkJNYOcyeKsHPrDWt3b9qZB3wp/LupkHufD2v9glPI8AYVcMmP6vpdhWDCNgksoq0FuIGP8ml1C30rFEjBhMZ1jSb9bMqJxqu1RSKf0paNgiTHjJLKVkybNYm4Cl+X6OcaHYSFUliMoPnsoziXFlE6WQkdCA0c5toJxLWyvlF8xzTja1VXtEPzFL78VnZ1t3+p/e/XWyXwc6+Qb2SIN4pMD0iJ/yTlpE+6sOj+dXWfPrbi/3X33cFbqOnPPV/Iq3ONHr7LDjQ==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="q+kI6dZtWxcZyJcuwi9H/sBseUI=">AAACfHicbZHbSsQwEIbTetpdT6u33gRF8IS2KiqIIOiFlwquCtu6pNnpGkzTkkzVpfRVfCjvfBuzB0XXHQj8+f4ZMpmJMikMet6n405MTk3PVKq12bn5hcX60tydSXPNocFTmeqHiBmQQkEDBUp4yDSwJJJwHz1f9Pz7F9BGpOoWuxmECesoEQvO0KJW/T1IGD5FcfFaPhZNse2HJT2jf6FF2z/o7RsFpzRAeMOCipj2r3RjtG7z8XZMoQTq2fydMZZQA8iZLC7Llt+qr3m7Xj/of+EPxRoZxnWr/hG0U54noJBLZkzT9zIMC6ZRcAllLcgNZIw/sw40rVQsARMW/TmWdN2SNo1TbY9C2qe/KwqWGNNNIpvZa9KMej04zmvmGJ+EhVBZjqD44KE4lxRT2lsKbQsNHGXXCsa1sL1S/sQ042hXV7ND8Ee//F/c7e/6Vt94pEJWyCrZID45JufkilyTBuHOpLPlHDiHbtXdc48G43Kd4dyWyZ9wT74AK3jCmg==</latexit>
sha1_base64="QmDFYUNY6J1FfiHSnLcgP1REaN4=">AAACh3icbZHbahsxEIa1W7eN3ZPTXuZG1BTcuji7bWgDpeAeLnqZQnwA79po5VlbWKtdpNnEZtlX6UPlLm8T+dBSOx4Q/PpmfjSaiTIpDHrereM+qDx89PioWnvy9NnzF/Xjlz2T5ppDl6cy1YOIGZBCQRcFShhkGlgSSehH8x+rfP8KtBGpusRlBmHCpkrEgjO0aFz/EyQMZ1FcXJejYihafljSr3QXWtT6hxZ/UfCFBggLLKiI6fpKm/u+t6PLA0YJ1LP17w+khNpAzmTxsxz743rDa3vroPeFvxUNso2Lcf0mmKQ8T0Ahl8yYoe9lGBZMo+ASylqQG8gYn7MpDK1ULAETFus5lvSNJRMap9oehXRN/3cULDFmmUS2ctWk2c+t4KHcMMf4PCyEynIExTcPxbmkmNLVUuhEaOAol1YwroXtlfIZ04yjXV3NDsHf//J90fvQ9q3+7TU637fjOCIn5DVpEp98Jh3yi1yQLuFOxXnnfHTO3Kp76n5yzzelrrP1vCI74X67A65yw4k=</latexit>

implies

<latexit sha1_base64="XF1Gr8h4738t5rWCuUTTLtNRQb4=">AAAB73icbZBNS8NAEIYnftb6VfXoZbEInkoigh5LvXisYD+gCWWznbRLN5u4uxFK6J/w4kERr/4db/4bt20O2vrCwsM7M+zMG6aCa+O6387a+sbm1nZpp7y7t39wWDk6buskUwxbLBGJ6oZUo+ASW4Ybgd1UIY1DgZ1wfDurd55QaZ7IBzNJMYjpUPKIM2qs1fUbfDgkft6vVN2aOxdZBa+AKhRq9itf/iBhWYzSMEG17nluaoKcKsOZwGnZzzSmlI3pEHsWJY1RB/l83yk5t86ARImyTxoyd39P5DTWehKHtjOmZqSXazPzv1ovM9FNkHOZZgYlW3wUZYKYhMyOJwOukBkxsUCZ4nZXwkZUUWZsRGUbgrd88iq0L2ue5furar1RxFGCUziDC/DgGupwB01oAQMBz/AKb86j8+K8Ox+L1jWnmDmBP3I+fwB5xo+W</latexit>
and apply the squared L2 norm on both sides
[i+1] 2 [i] [i] 2
||w
<latexit sha1_base64="mh6nOSvUgtWXLeXFDyjggPq2Oe4=">AAACLXicbVDLSsNAFJ34rPUVdelmsAhCoSRF0I1Q1IXLCvYBaVom00k7dPJgZqKWJD/kxl8RwUVF3PobTtoI2npg4HDOucy9xwkZFdIwJtrS8srq2npho7i5tb2zq+/tN0UQcUwaOGABbztIEEZ90pBUMtIOOUGew0jLGV1lfuuecEED/06OQ2J7aOBTl2IkldTTr5Ok4yE5dNz4Ie3GFi2bdpok3Sq8gPOWncIy/JEecynL9vSSUTGmgIvEzEkJ5Kj39NdOP8CRR3yJGRLCMo1Q2jHikmJG0mInEiREeIQGxFLURx4Rdjy9NoXHSulDN+Dq+RJO1d8TMfKEGHuOSma7inkvE//zrEi653ZM/TCSxMezj9yIQRnArDrYp5xgycaKIMyp2hXiIeIIS1VwUZVgzp+8SJrViqn47WmpdpnXUQCH4AicABOcgRq4AXXQABg8gRcwAe/as/amfWifs+iSls8cgD/Qvr4B2/2ptA==</latexit>
|| = ||w + x ||
= ||w[i] ||2 + 2(x[i] )T w[i] + ||x[i] ||2
<latexit sha1_base64="WrQgutLGBRW3hS2PwnaeJJGEzrA=">AAACRXicbVBJSwMxGM241rpVPXoJFqFSKDNF0ItQ9OKxQjeYmZZMmmlDMwtJRlum/XNevHvzH3jxoIhXzbQj1NYHgcd735bnhIwKqesv2srq2vrGZmYru72zu7efOzhsiCDimNRxwALecpAgjPqkLqlkpBVygjyHkaYzuEn85j3hggZ+TY5CYnuo51OXYiSV1MlZV3A8tjwk+44bP0zasUntyXjcLsMiLMPCrzNMnbN2DcLF8iKcmzGcm9HJ5fWSPgVcJkZK8iBFtZN7troBjjziS8yQEKahh9KOEZcUMzLJWpEgIcID1COmoj7yiLDjaQoTeKqULnQDrp4v4VSd74iRJ8TIc1RlcqtY9BLxP8+MpHtpx9QPI0l8PFvkRgzKACaRwi7lBEs2UgRhTtWtEPcRR1iq4LMqBGPxy8ukUS4Zit+d5yvXaRwZcAxOQAEY4AJUwC2ogjrA4BG8gnfwoT1pb9qn9jUrXdHSniPwB9r3D2+Tst0=</latexit>
(

0
Thus
<latexit sha1_base64="KxR+33HRklC9cMM/IPSJxMjyggQ=">AAACGHicbZDLSgMxFIYzXmu9jbp0EyyCC6mZUtu6K7pxWcFeYGYomTRtQzMXkoxQhnkMN76KGxeKuO3OtzGdzkJbDwQ+/v+c5OT3Is6kQujbWFvf2NzaLuwUd/f2Dw7No+OODGNBaJuEPBQ9D0vKWUDbiilOe5Gg2Pc47XqTu7nffaJCsjB4VNOIuj4eBWzICFZa6ptXiZNdYouR5yaojG5QtVq/ROVao2HVrjUgq1KpoxQ6nEKU9s2SlrKCq2DlUAJ5tfrmzBmEJPZpoAjHUtoWipSbYKEY4TQtOrGkESYTPKK2xgD7VLpJtlMKz7UygMNQ6BMomKm/JxLsSzn1Pd3pYzWWy95c/M+zYzVsuAkLoljRgCweGsYcqhDOU4IDJihRfKoBE8H0rpCMscBE6SyLOgRr+cur0KmULc0P1VLzNo+jAE7BGbgAFqiDJrgHLdAGBDyDV/AOPowX4834NL4WrWtGPnMC/pQx+wFw6puU</latexit>

[i+1] 2 [i] 2 [i] 2

||w
<latexit sha1_base64="sF3qpTo/9pWtoWknABTm8osIbzw=">AAACXnicdVFNSwMxEM2uWmv9aNWL4CVYBEEpWbG2vRW9eKxgrbC7lmyabYPZD5KsWrb7J72JF3+K2X5ALToQeLw3M29m4sWcSYXQp2GurW8UNotbpe2d3b1yZf/gUUaJILRLIh6JJw9LyllIu4opTp9iQXHgcdrzXm5zvfdKhWRR+KDGMXUDPAyZzwhWmupXktSZNrHF0HNTVENN1LLQBapdW/V6o6EBshCqt7LJxAmwGnl++pY9pzY7t1zNPV9Ch1O4Ks6l8yXhfUnI+pXqwgourODCClo1NI0qmEenX/lwBhFJAhoqwrGUtoVi5aZYKEY4zUpOImmMyQseUlvDEAdUuul0swyeamYA/UjoFyo4ZZcrUhxIOQ48nZkPK1e1nPxLsxPlN92UhXGiaEhmRn7CoYpgfms4YIISxccaYCKYnhWSERaYKP0jJX2Exabwf/B4WbM0vr+qtm/m5yiCY3ACzoAFGqAN7kAHdAEBX4ZhlIxt49ssmLtmeZZqGvOaQ/ArzKMfXSSzxQ==</latexit>
||  ||w || + ||x ||

Sebastian Raschka STAT 453: Intro to Deep Learning 49

fi
Perceptron Convergence Theorem -- Proof
Now, we simply expand:

[i+1] 2 [i] 2 [i] 2

||w <latexit sha1_base64="VlJexkld9GzMR5JEXg9pp90wJzM=">AAACOHicbZDLSsNAFIYn9VbrLerSzWARhEJJiqDLoht3VrAXaNIymU7aoZMLMxO1JHksNz6GO3HjQhG3PoGTNovaemDg5//OYc75nZBRIQ3jVSusrK6tbxQ3S1vbO7t7+v5BSwQRx6SJAxbwjoMEYdQnTUklI52QE+Q5jLSd8VXG2/eECxr4d3ISEttDQ5+6FCOprL5+kySWh+TIceOHtBd3acW00yTp1SC0GIEQLvKcVubA4xzo62WjakwLLgszF2WQV6Ovv1iDAEce8SVmSIiuaYTSjhGXFDOSlqxIkBDhMRqSrpI+8oiw4+nhKTxRzgC6AVfPl3Dqzk/EyBNi4jmqM9tVLLLM/I91I+le2DH1w0gSH88+ciMGZQCzFOGAcoIlmyiBMKdqV4hHiCMsVdYlFYK5ePKyaNWqptK3Z+X6ZR5HERyBY3AKTHAO6uAaNEATYPAE3sAH+NSetXftS/uetRa0fOYQ/Cnt5xdc3a3y</latexit>
||  ||w || + ||x ||
[i+1] 2 [i 1] 2 [i 1] 2 [i] 2
||w <latexit sha1_base64="06kvBhTMP+V71uZaOrpdxX3r0zw=">AAACVnicbZFdS8MwFIbTzrk5v6peehMcgjAc7RD0cuiNlxPcB6zdSLN0C0s/SFJ1tP2TeqM/xRsx3SbMbQcCL+9zDjl540aMCmmaX5pe2Cnulsp7lf2Dw6Nj4+S0I8KYY9LGIQt5z0WCMBqQtqSSkV7ECfJdRrru9CHn3RfCBQ2DZzmLiOOjcUA9ipFU1tDw09T2kZy4XvKaDZI+rVlOlqaDBoQ2IxDCdX79x2sr6O0/2sIWZGhUzbo5L7gprKWogmW1hsa7PQpx7JNAYoaE6FtmJJ0EcUkxI1nFjgWJEJ6iMekrGSCfCCeZx5LBS+WMoBdydQIJ5+7qRIJ8IWa+qzrzXcU6y81trB9L785JaBDFkgR4cZEXMyhDmGcMR5QTLNlMCYQ5VbtCPEEcYal+oqJCsNafvCk6jbql9NNNtXm/jKMMzsEFuAIWuAVN8AhaoA0w+ADfmq4VtE/tRy/qpUWrri1nzsC/0o1fFHW0nw==</latexit>
||  ||w || + ||x || + ||x ||
[i+1] 2 [i 2] 2 [i 2] 2 [i 1] 2 [i] 2
||w <latexit sha1_base64="UqpF0KC7wKhja2wB6bQDw1ScT8I=">AAACcnicdZHLSgMxFIYz463W26i4UdBoEYRqmSmCLotuXFawtjCdlkyaaUMzF5KMWqZ9AF/PnU/hxgcw086iNw8Efv4vh3Pyx40YFdI0vzV9ZXVtfSO3md/a3tndM/YPXkUYc0xqOGQhb7hIEEYDUpNUMtKIOEG+y0jd7T+mvP5GuKBh8CIHEXF81A2oRzGSymobn8Nh00ey53rJ+6iV2LRoOaPhsFWGsMkIhHCe35QzXpxCH7NoGbP+ZxPSNgpmyRwXXBRWJgogq2rb+Gp2Qhz7JJCYISFsy4ykkyAuKWZklG/GgkQI91GX2EoGyCfCScaRjeClcjrQC7k6gYRjd7ojQb4QA99VN9NdxTxLzWXMjqV37yQ0iGJJAjwZ5MUMyhCm+cMO5QRLNlACYU7VrhD3EEdYql/KqxCs+ScvitdyyVL6+bZQecjiyIETcAGugAXuQAU8gSqoAQx+tCPtVDvTfvVj/VzPstO1rOcQzJR+/QcuIb3Z</latexit>
||  ||w || + ||x || + ||x || + ||x ||
...
<latexit sha1_base64="6zhPYCd7LoWJJ/0EStBY4k8k58I=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8BQSEfRY9OKxov2ANpTNdtMu3WzC7kQooT/BiwdFvPqLvPlv3LY5aOsLCw/vzLAzb5hKYdDzvp3S2vrG5lZ5u7Kzu7d/UD08apkk04w3WSIT3Qmp4VIo3kSBkndSzWkcSt4Ox7ezevuJayMS9YiTlAcxHSoRCUbRWg+u6/arNc/15iKr4BdQg0KNfvWrN0hYFnOFTFJjur6XYpBTjYJJPq30MsNTysZ0yLsWFY25CfL5qlNyZp0BiRJtn0Iyd39P5DQ2ZhKHtjOmODLLtZn5X62bYXQd5EKlGXLFFh9FmSSYkNndZCA0ZygnFijTwu5K2IhqytCmU7Eh+Msnr0LrwvUt31/W6jdFHGU4gVM4Bx+uoA530IAmMBjCM7zCmyOdF+fd+Vi0lpxi5hj+yPn8AUvmjSI=</latexit>

i
X
[i+1] 2 [1] 2 [j] 2
||w ||  ||w || + ||x ||
<latexit sha1_base64="ht1LYz0tIsHy5AckfbXeKzg+kRk=">AAACRnicbVBNS8NAFHyp3/Wr6tHLYhEEoSQi6EUoevGoYFshiWWz3ei2mw92N2pJ8+u8ePbmT/DiQRGvbtqI2jqwMMzM470dL+ZMKtN8NkpT0zOzc/ML5cWl5ZXVytp6U0aJILRBIh6JSw9LyllIG4opTi9jQXHgcdryeie537qlQrIovFD9mLoBvg6ZzwhWWmpX3MHACbC68fz0LrtKbbZrudlgcLWHkMMpQmjM/3Z3kSOToJ12jywts+wndp/HuqNYu1I1a+YQaJJYBalCgbN25cnpRCQJaKgIx1LalhkrN8VCMcJpVnYSSWNMevia2pqGOKDSTYc1ZGhbKx3kR0K/UKGh+nsixYGU/cDTyfxWOe7l4n+enSj/0E1ZGCeKhmS0yE84UhHKO0UdJihRvK8JJoLpWxG5wQITpZsv6xKs8S9PkuZezdL8fL9aPy7qmIdN2IIdsOAA6nAKZ9AAAg/wAm/wbjwar8aH8TmKloxiZgP+oARfWXmy5w==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="zcxLwyt3iYRHJjeUzUypTj8Zjw4=">AAACO3icbZBLSwMxFIXv+Ky1anXrJlgEoVBmutGNILhxWcE+oDMtmTTTps08SDJqmc6vc+PanT/BjQtF3JppK2rrhcDhnBNy87kRZ1KZ5rOxsrq2vrGZ28pvF3Z294r7hYYMY0FonYQ8FC0XS8pZQOuKKU5bkaDYdzltuqPLLG/eUiFZGNyocUQdH/cD5jGClba6RWcysX2sBq6X3KWdpM3KlpNOJp0qQjanCKGF/DstI1vGfjcZnlvaZulP7T6rDWe1brFkVszpoGVhzUUJ5lPrFp/sXkhinwaKcCxl2zIj5SRYKEY4TfN2LGmEyQj3aVvLAPtUOskUQ4qOtdNDXij0CRSaur9vJNiXcuy7upntKhezzPwva8fKO3MSFkSxogGZPeTFHKkQZUxRjwlKFB9rgYlgeldEBlhgojT5vIZgLX55WTSqFUvraxNycAhHcAIWnMIFXEEN6kDgAV7gDd6NR+PV+JjhWjHm3A7gzxifX6qJsjU=</latexit>
sha1_base64="P/wjO6RPCOEOYUodaP4qtxBYN9c=">AAACRnicbVDLSsNAFL2pr1pfVZduBosgFEriRjdC0Y3LCvYBSVom04kdnTyYmaglzde5ce3OT3DjQhG3TtqKWj0wcDjnXO6d48WcSWWaT0Zhbn5hcam4XFpZXVvfKG9utWSUCEKbJOKR6HhYUs5C2lRMcdqJBcWBx2nbuz7N/fYNFZJF4YUaxtQN8GXIfEaw0lKv7I5GToDVwPPT26yb2qxqudlo1D1AyOEUITTjf7lV5Mgk6KVXx5aWWfYdu8tjV5NYr1wxa+YY6C+xpqQCUzR65UenH5EkoKEiHEtpW2as3BQLxQinWclJJI0xucaX1NY0xAGVbjquIUN7WukjPxL6hQqN1Z8TKQ6kHAaeTua3ylkvF//z7ET5R27KwjhRNCSTRX7CkYpQ3inqM0GJ4kNNMBFM34rIAAtMlG6+pEuwZr/8l7QOapbm52alfjKtowg7sAv7YMEh1OEMGtAEAvfwDK/wZjwYL8a78TGJFozpzDb8QgE+AVg5suM=</latexit>
j=1
i
X
[i+1] 2 [j] 2
||w ||  ||x ||
<latexit sha1_base64="gEkXb4DkVNJ4FXmGQst6Z7Fr/OE=">AAACLnicbVDLSsNAFJ3UV62vqks3g0UQhJIUQTdCUQSXFewDkrRMppN22smDmYlaknyRG39FF4KKuPUznLQFtfXAwOGcc5l7jxMyKqSuv2q5hcWl5ZX8amFtfWNzq7i90xBBxDGp44AFvOUgQRj1SV1SyUgr5AR5DiNNZ3iR+c1bwgUN/Bs5ContoZ5PXYqRVFKneJkklodk33Hju7Qdm/TIsNMkaVcgtBiBCpaIvE48ODOUTdOf+H0WH0zCnWJJL+tjwHliTEkJTFHrFJ+tboAjj/gSMySEaeihtGPEJcWMpAUrEiREeIh6xFTURx4Rdjw+N4UHSulCN+Dq+RKO1d8TMfKEGHmOSma7ilkvE//zzEi6p3ZM/TCSxMeTj9yIQRnArDvYpZxgyUaKIMyp2hXiPuIIS9VwQZVgzJ48TxqVsqH49XGpej6tIw/2wD44BAY4AVVwBWqgDjB4AE/gDbxrj9qL9qF9TqI5bTqzC/5A+/oG5pWprw==</latexit>
j=1
Sebastian Raschka STAT 453: Intro to Deep Learning 50
Perceptron Convergence Theorem -- Proof
i
X
[i+1] 2
From ||w ||  ||x[j] ||2 we can nally get the
<latexit sha1_base64="gEkXb4DkVNJ4FXmGQst6Z7Fr/OE=">AAACLnicbVDLSsNAFJ3UV62vqks3g0UQhJIUQTdCUQSXFewDkrRMppN22smDmYlaknyRG39FF4KKuPUznLQFtfXAwOGcc5l7jxMyKqSuv2q5hcWl5ZX8amFtfWNzq7i90xBBxDGp44AFvOUgQRj1SV1SyUgr5AR5DiNNZ3iR+c1bwgUN/Bs5ContoZ5PXYqRVFKneJkklodk33Hju7Qdm/TIsNMkaVcgtBiBCpaIvE48ODOUTdOf+H0WH0zCnWJJL+tjwHliTEkJTFHrFJ+tboAjj/gSMySEaeihtGPEJcWMpAUrEiREeIh6xFTURx4Rdjw+N4UHSulCN+Dq+RKO1d8TMfKEGHmOSma7ilkvE//zzEi6p3ZM/TCSxMeTj9yIQRnArDvYpZxgyUaKIMyp2hXiPuIIS9VwQZVgzJ48TxqVsqH49XGpej6tIw/2wD44BAY4AVVwBWqgDjB4AE/gDbxrj9qL9qF9TqI5bTqzC/5A+/oG5pWprw==</latexit>
j=1
upper bound.

[j] 2
Let <latexit sha1_base64="Z8dz9v4AKYnLI+UN2SvNch0hBNE=">AAACDHicbVC7TsMwFHV4lvIqMLJYVEhMVVKQYEGqYGEsEn1ISVo5rtOaOk5kO6hVkg9g4VdYGECIlQ9g429w2g7QciXLR+ecq3vv8SJGpTLNb2NpeWV1bb2wUdzc2t7ZLe3tN2UYC0waOGShaHtIEkY5aSiqGGlHgqDAY6TlDa9zvfVAhKQhv1PjiLgB6nPqU4yUprqlsuMRheAldAI0gmmqPzXw/GSUdRL73s3StFPVLrNiTgouAmsGymBW9W7py+mFOA4IV5ghKW3LjJSbIKEoZiQrOrEkEcJD1Ce2hhwFRLrJ5JgMHmumB/1Q6McVnLC/OxIUSDkOPO3Md5XzWk7+p9mx8i/chPIoVoTj6SA/ZlCFME8G9qggWLGxBggLqneFeIAEwkrnV9QhWPMnL4JmtWKdVszbs3LtahZHARyCI3ACLHAOauAG1EEDYPAInsEreDOejBfj3fiYWpeMWc8B+FPG5w/K3pt4</latexit>
= max ||x ||

[i+1] 2
then ||w <latexit sha1_base64="8c7SIE0kmnWOIHqOB4ixuDLewdk=">AAACD3icbVDLSgMxFM34rPU16tJNsCiCUGaKoMuiG5cV7APaacmkd9rQTGZIMkqZ9g/c+CtuXCji1q07/8ZM24W2Hkg4nHMv997jx5wp7Tjf1tLyyuraem4jv7m1vbNr7+3XVJRIClUa8Ug2fKKAMwFVzTSHRiyBhD6Huj+4zvz6PUjFInGnhzF4IekJFjBKtJE69slo1AqJ7vtB+jBup0125nrj0ahdwrjFwXw+aIJZxy44RWcCvEjcGSmgGSod+6vVjWgSgtCUE6WarhNrLyVSM8phnG8lCmJCB6QHTUMFCUF56eSeMT42ShcHkTRPaDxRf3ekJFRqGPqmMttdzXuZ+J/XTHRw6aVMxIkGQaeDgoRjHeEsHNxlEqjmQ0MIlczsimmfSEK1iTBvQnDnT14ktVLRNfz2vFC+msWRQ4foCJ0iF12gMrpBFVRFFD2iZ/SK3qwn68V6tz6mpUvWrOcA/YH1+QPsGpvl</latexit>
||  i

Sebastian Raschka STAT 453: Intro to Deep Learning 51

fi
Perceptron Convergence Theorem -- Proof

lower bound upper bound

[i+1] 2 ↵ 2 i2
||w || ||w[i+1] ||2  i
<latexit sha1_base64="Pa0/JLqOqpgcRB0lytlyotxOGCk=">AAACLXicbVDLSsNAFJ34rPVVdelmsAiiUJIi6LKoC5cV7AOStEymk3boZBJmJkpJ8kNu/BURXFTErb/hNO2ith4YOJxzLnPv8SJGpTLNsbGyura+sVnYKm7v7O7tlw4OmzKMBSYNHLJQtD0kCaOcNBRVjLQjQVDgMdLyhrcTv/VEhKQhf1SjiLgB6nPqU4yUlrqluzR1AqQGnp88Z53EpheWm6VppwqdPoGOLxBOHMSiAdIS7VSzZG6gc55Hs26pbFbMHHCZWDNSBjPUu6V3pxfiOCBcYYaktC0zUm6ChKKYkazoxJJECA9Rn9iachQQ6Sb5tRk81UoP+qHQjyuYq/MTCQqkHAWeTk4WlYveRPzPs2PlX7sJ5VGsCMfTj/yYQRXCSXWwRwXBio00QVhQvSvEA6QbUrrgoi7BWjx5mTSrFUvzh8ty7WZWRwEcgxNwBixwBWrgHtRBA2DwAt7AGHwar8aH8WV8T6MrxmzmCPyB8fMLi0Gpgw==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="N7on5w8OFVe4wZzx8M3r8t7Sr4Q=">AAACInicbZDLSsNAFIZPvNZaNbp1M1gEUShJN7oUdOGygr1Ak5bJdNIOnVyYmSglzQu58VVEcFERn8VJ2kVt/WHg5z/nMOd8XsyZVJY1MzY2t7Z3dkt75f3KweGReVxpySgRhDZJxCPR8bCknIW0qZjitBMLigOP07Y3vsvr7WcqJIvCJzWJqRvgYch8RrDSUd+8n06dAKuR56cvWS/tsivbzabTXh05Q4ocX2CSOpjHI6wj1qtn6dJA77Jozfpm1apZhdC6sRemCgs1+uaHM4hIEtBQEY6l7NpWrNwUC8UIp1nZSSSNMRnjIe1qG+KASjctrs3QuU4GyI+EfqFCRbo8keJAykng6c58Ublay8P/at1E+TduysI4UTQk84/8hCMVoRwdGjBBieITbTARTO+KyAhrQkoDLmsI9urJ66ZVr9naP1pQglM4gwuw4Rpu4QEa0AQCr/AOM/gy3oxP43uOa8NYcDuBPzJ+fgFJJafr</latexit>
sha1_base64="Fa2nh52SXsH+ubH7GEfTph3Hopk=">AAACLXicbVDLSgMxFM3UV62vUZdugkUQhTLTjS6LunBZwT6gM1MyaaYNzTxIMkrJzA+58VdEcFERt/6G6WNRWw8EDuecS+49fsKokJY1Ngpr6xubW8Xt0s7u3v6BeXjUFHHKMWngmMW87SNBGI1IQ1LJSDvhBIU+Iy1/eDvxW0+ECxpHj3KUEDdE/YgGFCOppa55l2VOiOTAD9Rz7qkOvbTdPMu8KnT6BDoBR1g5iCUDpCXqVXO1MOBdTKN51yxbFWsKuErsOSmDOepd893pxTgNSSQxQ0J0bCuRrkJcUsxIXnJSQRKEh6hPOppGKCTCVdNrc3imlR4MYq5fJOFUXZxQKBRiFPo6OVlULHsT8T+vk8rg2lU0SlJJIjz7KEgZlDGcVAd7lBMs2UgThDnVu0I8QLohqQsu6RLs5ZNXSbNasTV/sMq1m3kdRXACTsE5sMEVqIF7UAcNgMELeANj8Gm8Gh/Gl/E9ixaM+cwx+APj5xeKAal/</latexit>
||w⇤ ||2 <latexit sha1_base64="8c7SIE0kmnWOIHqOB4ixuDLewdk=">AAACD3icbVDLSgMxFM34rPU16tJNsCiCUGaKoMuiG5cV7APaacmkd9rQTGZIMkqZ9g/c+CtuXCji1q07/8ZM24W2Hkg4nHMv997jx5wp7Tjf1tLyyuraem4jv7m1vbNr7+3XVJRIClUa8Ug2fKKAMwFVzTSHRiyBhD6Huj+4zvz6PUjFInGnhzF4IekJFjBKtJE69slo1AqJ7vtB+jBup0125nrj0ahdwrjFwXw+aIJZxy44RWcCvEjcGSmgGSod+6vVjWgSgtCUE6WarhNrLyVSM8phnG8lCmJCB6QHTUMFCUF56eSeMT42ShcHkTRPaDxRf3ekJFRqGPqmMttdzXuZ+J/XTHRw6aVMxIkGQaeDgoRjHeEsHNxlEqjmQ0MIlczsimmfSEK1iTBvQnDnT14ktVLRNfz2vFC+msWRQ4foCJ0iF12gMrpBFVRFFD2iZ/SK3qwn68V6tz6mpUvWrOcA/YH1+QPsGpvl</latexit>

combined Since the number of iterations i has an upper

[i+1] 2 ↵ 2 i2 bound,
i ||w || we can conclude that the weights only
<latexit sha1_base64="VuVVRvVCuODYxIME5ryKgy9oEPM=">AAACOXicbVDLSsNAFJ3Ud31VXboZLIIoSFIEXYpuXFawWmjScjO9aYdOHsxMlJLmt9z4F+4ENy4UcesPOE278HVg4HDOucy9x08EV9q2n6zSzOzc/MLiUnl5ZXVtvbKxea3iVDJssFjEsumDQsEjbGiuBTYTiRD6Am/8wfnYv7lFqXgcXelhgl4IvYgHnIE2UqdSd33U0OHU7SEdjdwQdN8Psru8nbX4gePlo1G7VphuIIFlLoikD0bi7VqefRto7xfRvFOp2od2AfqXOFNSJVPUO5VHtxuzNMRIMwFKtRw70V4GUnMmMC+7qcIE2AB62DI0ghCVlxWX53TXKF0axNK8SNNC/T6RQajUMPRNcryo+u2Nxf+8VqqDEy/jUZJqjNjkoyAVVMd0XCPtcolMi6EhwCQ3u1LWB9OQNmWXTQnO75P/kuvaoWP45VH19GxaxyLZJjtkjzjkmJySC1InDcLIPXkmr+TNerBerHfrYxItWdOZLfID1ucXzqKuIw==</latexit>
||w⇤ ||2 change a nite number of times and will
converge if the classes are linearly
||w⇤ ||2 separable.
i
↵2
⇤
<latexit sha1_base64="8XyUQ/hpevXoCRMKz7TPeHseaKs=">AAACHXicbZA9SwNBEIb3/IzxK2ppsxgEsQh3Imgp2lhGMCrkLmFuM5cs2ftwd08Jl/sjNv4VGwtFLGzEf+MmXqGJLyw8vDPDzrx+IrjStv1lzczOzS8slpbKyyura+uVjc0rFaeSYYPFIpY3PigUPMKG5lrgTSIRQl/gtd8/G9Wv71AqHkeXepCgF0I34gFnoI3Vrhxy6gq8pdQNJLDM9VEDHQ7dEHTPD7L7vJXt58Nh6yCnmQsi6YHBdqVq1+yx6DQ4BVRJoXq78uF2YpaGGGkmQKmmYyfay0BqzgTmZTdVmADrQxebBiMIUXnZ+Lqc7hqnQ4NYmhdpOnZ/T2QQKjUIfdM52lpN1kbmf7VmqoNjL+NRkmqM2M9HQSqojukoKtrhEpkWAwPAJDe7UtYDE5M2gZZNCM7kydNwdVBzDF8cVk9OizhKZJvskD3ikCNyQs5JnTQIIw/kibyQV+vRerberPef1hmrmNkif2R9fgPdk6Jp</latexit>

<latexit sha1_base64="xL2b+0uuneWLA4OhRB/n7H1MhF0=">AAAB7nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxgmkLbSib7aRdutnE3Y1QQn+EFw+KePX3ePPfuG1z0NYXFh7emWFn3jAVXBvX/XZKa+sbm1vl7crO7t7+QfXwqKWTTDH0WSIS1QmpRsEl+oYbgZ1UIY1Dge1wfDurt59QaZ7IBzNJMYjpUPKIM2qs1e7px4wq7Fdrbt2di6yCV0ANCjX71a/eIGFZjNIwQbXuem5qgpwqw5nAaaWXaUwpG9Mhdi1KGqMO8vm6U3JmnQGJEmWfNGTu/p7Iaaz1JA5tZ0zNSC/XZuZ/tW5mousg5zLNDEq2+CjKBDEJmd1OBlwhM2JigTLF7a6EjaiizNiEKjYEb/nkVWhd1D3L95e1xk0RRxlO4BTOwYMraMAdNMEHBmN4hld4c1LnxXl3PhatJaeYOYY/cj5/AIYOj60=</latexit>
<latexit

Sebastian Raschka STAT 453: Intro to Deep Learning 52

fi
Perceptron Convergence Theorem -- Proof

[i+1] 2 2 2
<latexit sha1_base64="2x+Hoc8bYNnyWENuByAIyqXOm/k=">AAACIHicbVDLSgMxFM34rPVVdekmWARBkJki6FJ047KCbYWZabmT3mmDmQdJRinT+RQ3/oobF4roTr/GdOzC14HAyTn3kpwTpIIrbdvv1szs3PzCYmWpuryyurZe29hsqySTDFssEYm8CkCh4DG2NNcCr1KJEAUCO8H12cTv3KBUPIkv9ShFP4JBzEPOQBupVzvyAtRAOfUGSMdjLwI9DML8tujmLt93/GI87jZKM/dApEMwN95tFL1a3T6wS9C/xJmSOpmi2au9ef2EZRHGmglQynXsVPs5SM2ZwKLqZQpTYNcwQNfQGCJUfl4GLOiuUfo0TKQ5saal+n0jh0ipURSYyUkA9dubiP95bqbDYz/ncZppjNnXQ2EmqE7opC3a5xKZFiNDgElu/krZECQwbTqtmhKc35H/knbjwDH84rB+cjqto0K2yQ7ZIw45IifknDRJizByRx7IE3m27q1H68V6/RqdsaY7W+QHrI9PyT+itQ==</latexit>
i ||w || ↵ i

||w[i+1] ||2
<latexit sha1_base64="JyIP1lInlSlDfjGKg3+t2j+RGNM=">AAACAnicbZDLSsNAFIYnXmu9RV2Jm8EiCEJJiqDLohuXFewF0rRMppN26GQSZiZKSYIbX8WNC0Xc+hTufBsnbRba+sPAx3/OYc75vYhRqSzr21haXlldWy9tlDe3tnd2zb39lgxjgUkThywUHQ9JwignTUUVI51IEBR4jLS98XVeb98TIWnI79QkIm6Ahpz6FCOlrb55mKbdAKmR5ycPWS9x6JntZmnaq8G+WbGq1lRwEewCKqBQo29+dQchjgPCFWZISse2IuUmSCiKGcnK3ViSCOExGhJHI0cBkW4yPSGDJ9oZQD8U+nEFp+7viQQFUk4CT3fm68r5Wm7+V3Ni5V+6CeVRrAjHs4/8mEEVwjwPOKCCYMUmGhAWVO8K8QgJhJVOraxDsOdPXoRWrWprvj2v1K+KOErgCByDU2CDC1AHN6ABmgCDR/AMXsGb8WS8GO/Gx6x1yShmDsAfGZ8/TUqXVQ==</latexit>
sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit>
sha1_base64="Xq1iO/ZsvTmfKCSKJ7BQTlg6QEM=">AAAB93icbZBPS8MwGMbf+nfOqdWbeAkOQRBGu4seBS8eJ7g/0HUjzdItLE1LkiqjHV78Kl48KOI38ea3Md120M0HAj+eJyHv+wQJZ0o7zre1tr6xubVd2invVvb2D+zDSkvFqSS0SWIey06AFeVM0KZmmtNOIimOAk7bwfimyNsPVCoWi3s9Sagf4aFgISNYG6tvH+d5N8J6FITZ47SXeezC9ad53qujvl11as5MaBXcBVRhoUbf/uoOYpJGVGjCsVKe6yTaz7DUjHA6LXdTRRNMxnhIPYMCR1T52WyFKTozzgCFsTRHaDRzf7/IcKTUJArMzWJctZwV5n+Zl+rwys+YSFJNBZl/FKYc6RgVfaABk5RoPjGAiWRmVkRGWGKiTWtlU4K7vPIqtOo11/CdAyU4gVM4Bxcu4RpuoQFNIPAEL/AG79az9Wp9zOtasxa9HcEfWZ8/y26V6A==</latexit>
sha1_base64="tA9bfXNMdBpLVtt2UUb3oXQ52Do=">AAACAnicbZDLSsNAFIYn9VbrLepK3AwWQRBK0o0ui25cVrAXSNMymU7aoZNJmJkoJQlufBU3LhRx61O4822ctFlo6w8DH/85hznn9yJGpbKsb6O0srq2vlHerGxt7+zumfsHbRnGApMWDlkouh6ShFFOWooqRrqRICjwGOl4k+u83rknQtKQ36lpRNwAjTj1KUZKWwPzKE17AVJjz08esn7i0HPbzdK0X4cDs2rVrJngMtgFVEGh5sD86g1DHAeEK8yQlI5tRcpNkFAUM5JVerEkEcITNCKORo4CIt1kdkIGT7UzhH4o9OMKztzfEwkKpJwGnu7M15WLtdz8r+bEyr90E8qjWBGO5x/5MYMqhHkecEgFwYpNNSAsqN4V4jESCCudWkWHYC+evAztes3WfGtVG1dFHGVwDE7AGbDBBWiAG9AELYDBI3gGr+DNeDJejHfjY95aMoqZQ/BHxucPTAqXUQ==</latexit>
↵ <latexit sha1_base64="C9EVecZrAqUOi6KF3c6v2jUjVDg=">AAAB73icbZBNS8NAEIYn9avWr6pHL4tF8FSSIuix6MVjBfsBbSyT7aZdutnE3Y1QQv+EFw+KePXvePPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVFHWpLGIVSdAzQSXrGm4EayTKIZRIFg7GN/M6u0npjSP5b2ZJMyPcCh5yCkaa3V6KJIRPtT65Ypbdeciq+DlUIFcjX75qzeIaRoxaahArbuemxg/Q2U4FWxa6qWaJUjHOGRdixIjpv1svu+UnFlnQMJY2ScNmbu/JzKMtJ5Ege2M0Iz0cm1m/lfrpia88jMuk9QwSRcfhakgJiaz48mAK0aNmFhAqrjdldARKqTGRlSyIXjLJ69Cq1b1LN9dVOrXeRxFOIFTOAcPLqEOt9CAJlAQ8Ayv8OY8Oi/Ou/OxaC04+cwx/JHz+QO00o+8</latexit>
2

<latexit sha1_base64="g+IRNevPc1g3Nk0ZK/Q3r3J8OUI=">AAAB7HicbZBNS8NAEIYn9avWr6pHL8EieCqJCHosevFYwdRCG8pmO2mXbjZhdyKU0t/gxYMiXv1B3vw3btsctPWFhYd3ZtiZN8qkMOR5305pbX1jc6u8XdnZ3ds/qB4etUyaa44BT2Wq2xEzKIXCgARJbGcaWRJJfIxGt7P64xNqI1L1QOMMw4QNlIgFZ2StoBshsV615tW9udxV8AuoQaFmr/rV7ac8T1ARl8yYju9lFE6YJsElTivd3GDG+IgNsGNRsQRNOJkvO3XPrNN341Tbp8idu78nJiwxZpxEtjNhNDTLtZn5X62TU3wdToTKckLFFx/FuXQpdWeXu32hkZMcW2BcC7ury4dMM042n4oNwV8+eRVaF3Xf8v1lrXFTxFGGEziFc/DhChpwB00IgIOAZ3iFN0c5L86787FoLTnFzDH8kfP5A8PYjqQ=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 53

Perceptron Convergence Theorem -- Proof
⇤
In the convergence theorem, we can assume that||w <latexit sha1_base64="BwWi+7pv7xs5NwtLFVpGkGvOoKU=">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEcVESEXQjFN24rGAf0MYymU7aoZMHMxOlJsVfceNCEbf+hzv/xkmbhbYeGDiccy/3zHEjzqSyrG+jsLC4tLxSXC2trW9sbpnbOw0ZxoLQOgl5KFoulpSzgNYVU5y2IkGx73LadIdXmd+8p0KyMLhVo4g6Pu4HzGMEKy11zb007fhYDVwveRjfHacpukB21yxbFWsCNE/snJQhR61rfnV6IYl9GijCsZRt24qUk2ChGOF0XOrEkkaYDHGftjUNsE+lk0zSj9GhVnrIC4V+gUIT9fdGgn0pR76rJ7OkctbLxP+8dqy8cydhQRQrGpDpIS/mSIUoqwL1mKBE8ZEmmAimsyIywAITpQsr6RLs2S/Pk8ZJxdb85rRcvczrKMI+HMAR2HAGVbiGGtSBwCM8wyu8GU/Gi/FufExHC0a+swt/YHz+AOi1lNY=</latexit>
|| = 1
(so you may remove it from all equations)

[i+1] 2 ↵ 2 i2 [i+1] 2 2 2
i ||w || , i ||w || ↵ i
||w⇤ ||2
<latexit sha1_base64="9DwdRFQNxq71HNR6poETRxcSq9k=">AAAB+HicbZA9SwNBEIb3/IzxI6eWNotBsAp3ImgZtLGwiGA+IAlhbzOXLNnbO3bnlHjkl9hYKGLrT7Hz37hJrtDEFxYe3plhZt8gkcKg5307K6tr6xubha3i9s7uXsndP2iYONUc6jyWsW4FzIAUCuooUEIr0cCiQEIzGF1P680H0EbE6h7HCXQjNlAiFJyhtXpuqXMLIWoxGCLTOn7suWWv4s1El8HPoUxy1XruV6cf8zQChVwyY9q+l2A3YxoFlzApdlIDCeMjNoC2RcUiMN1sdviEnlinT8NY26eQztzfExmLjBlHge2MGA7NYm1q/ldrpxhedjOhkhRB8fmiMJUUYzpNgfaFBo5ybIFxLeytlA+ZZhxtVkUbgr/45WVonFV8y3fn5epVHkeBHJFjckp8ckGq5IbUSJ1wkpJn8krenCfnxXl3PuatK04+c0j+yPn8ATg4k3E=</latexit>
<latexit

<latexit sha1_base64="O35eBYnwQQpV4dauRwyF7NiOuLI=">AAACIHicbVDLSgMxFM34rPVVdekmWARBkJki6FJ047KCbYXOdLiT3mmDmQdJRinT+RQ3/oobF4roTr/GtM7C14HAyTn3kpwTpIIrbdvv1szs3PzCYmWpuryyurZe29hsqySTDFssEYm8CkCh4DG2NNcCr1KJEAUCO8H12cTv3KBUPIkv9ShFL4JBzEPOQBvJrx25AWrwOXUHSMdjNwI9DML8tujlXb7veMV43GtMzdwFkQ7B3HivUfi1un1gT0H/EqckdVKi6dfe3H7CsghjzQQo1XXsVHs5SM2ZwKLqZgpTYNcwwK6hMUSovHwasKC7RunTMJHmxJpO1e8bOURKjaLATE4CqN/eRPzP62Y6PPZyHqeZxph9PRRmguqETtqifS6RaTEyBJjk5q+UDUEC06bTqinB+R35L2k3DhzDLw7rJ6dlHRWyTXbIHnHIETkh56RJWoSRO/JAnsizdW89Wi/W69fojFXubJEfsD4+ATTZovQ=</latexit>

<latexit sha1_base64="VuVVRvVCuODYxIME5ryKgy9oEPM=">AAACOXicbVDLSsNAFJ3Ud31VXboZLIIoSFIEXYpuXFawWmjScjO9aYdOHsxMlJLmt9z4F+4ENy4UcesPOE278HVg4HDOucy9x08EV9q2n6zSzOzc/MLiUnl5ZXVtvbKxea3iVDJssFjEsumDQsEjbGiuBTYTiRD6Am/8wfnYv7lFqXgcXelhgl4IvYgHnIE2UqdSd33U0OHU7SEdjdwQdN8Psru8nbX4gePlo1G7VphuIIFlLoikD0bi7VqefRto7xfRvFOp2od2AfqXOFNSJVPUO5VHtxuzNMRIMwFKtRw70V4GUnMmMC+7qcIE2AB62DI0ghCVlxWX53TXKF0axNK8SNNC/T6RQajUMPRNcryo+u2Nxf+8VqqDEy/jUZJqjNjkoyAVVMd0XCPtcolMi6EhwCQ3u1LWB9OQNmWXTQnO75P/kuvaoWP45VH19GxaxyLZJjtkjzjkmJySC1InDcLIPXkmr+TNerBerHfrYxItWdOZLfID1ucXzqKuIw==</latexit>

||w⇤ ||2
i , i
↵2 ↵2
<latexit sha1_base64="9DwdRFQNxq71HNR6poETRxcSq9k=">AAAB+HicbZA9SwNBEIb3/IzxI6eWNotBsAp3ImgZtLGwiGA+IAlhbzOXLNnbO3bnlHjkl9hYKGLrT7Hz37hJrtDEFxYe3plhZt8gkcKg5307K6tr6xubha3i9s7uXsndP2iYONUc6jyWsW4FzIAUCuooUEIr0cCiQEIzGF1P680H0EbE6h7HCXQjNlAiFJyhtXpuqXMLIWoxGCLTOn7suWWv4s1El8HPoUxy1XruV6cf8zQChVwyY9q+l2A3YxoFlzApdlIDCeMjNoC2RcUiMN1sdviEnlinT8NY26eQztzfExmLjBlHge2MGA7NYm1q/ldrpxhedjOhkhRB8fmiMJUUYzpNgfaFBo5ybIFxLeytlA+ZZhxtVkUbgr/45WVonFV8y3fn5epVHkeBHJFjckp8ckGq5IbUSJ1wkpJn8krenCfnxXl3PuatK04+c0j+yPn8ATg4k3E=</latexit>
<latexit

<latexit sha1_base64="8XyUQ/hpevXoCRMKz7TPeHseaKs=">AAACHXicbZA9SwNBEIb3/IzxK2ppsxgEsQh3Imgp2lhGMCrkLmFuM5cs2ftwd08Jl/sjNv4VGwtFLGzEf+MmXqGJLyw8vDPDzrx+IrjStv1lzczOzS8slpbKyyura+uVjc0rFaeSYYPFIpY3PigUPMKG5lrgTSIRQl/gtd8/G9Wv71AqHkeXepCgF0I34gFnoI3Vrhxy6gq8pdQNJLDM9VEDHQ7dEHTPD7L7vJXt58Nh6yCnmQsi6YHBdqVq1+yx6DQ4BVRJoXq78uF2YpaGGGkmQKmmYyfay0BqzgTmZTdVmADrQxebBiMIUXnZ+Lqc7hqnQ4NYmhdpOnZ/T2QQKjUIfdM52lpN1kbmf7VmqoNjL+NRkmqM2M9HQSqojukoKtrhEpkWAwPAJDe7UtYDE5M2gZZNCM7kydNwdVBzDF8cVk9OizhKZJvskD3ikCNyQs5JnTQIIw/kibyQV+vRerberPef1hmrmNkif2R9fgPdk6Jp</latexit>

<latexit sha1_base64="xkFPDBVxdEKehgToBlt49xbEO6Q=">AAACB3icbZC7SgNBFIZnvcZ4W7UUZDAIVmE3CFoGbSwjmAtkYzg7OZsMmb04MyuEJZ2Nr2JjoYitr2Dn2zhJttDEHwY+/nMOZ87vJ4Ir7Tjf1tLyyuraemGjuLm1vbNr7+03VJxKhnUWi1i2fFAoeIR1zbXAViIRQl9g0x9eTerNB5SKx9GtHiXYCaEf8YAz0Mbq2kecegLvKfUCCSzzfNQwzjwQyQDuKuOuXXLKzlR0EdwcSiRXrWt/eb2YpSFGmglQqu06ie5kIDVnAsdFL1WYABtCH9sGIwhRdbLpHWN6YpweDWJpXqTp1P09kUGo1Cj0TWcIeqDmaxPzv1o71cFFJ+NRkmqM2GxRkAqqYzoJhfa4RKbFyAAwyc1fKRuACUSb6IomBHf+5EVoVMqu4ZuzUvUyj6NADskxOSUuOSdVck1qpE4YeSTP5JW8WU/Wi/Vufcxal6x85oD8kfX5A3plmRA=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 54

Geometric Intuition

1. Brains and neuron models

Sebastian Raschka STAT 453: Intro to Deep Learning 55

The Perceptron Learning Algorithm

• If correct: Do nothing if the prediction if output is equal to the target

• If incorrect, scenario a):

If output is 0 and target is 1, add input vector to weight vector

• If incorrect, scenario b):

If output is 1 and target is 0, subtract input vector from weight vector

Guaranteed to converge if a solution exists

(more about that later...)

Sebastian Raschka STAT 453: Intro to Deep Learning 56

The Perceptron Learning Algorithm

2. For every training epoch:

A. For every hx[i] , y [i] i 2 D : <latexit sha1_base64="KiRtzBWzB6OcgzuVeAuk4oyYAnU=">AAACJHicbZDLSsNAFIYn9VbrLerSzWARXEhJRFBwU9SFywr2Akksk+mkHTqZhJmJGEIexo2v4saFF1y48VmctFlo64GBj//8hznn92NGpbKsL6OysLi0vFJdra2tb2xumds7HRklApM2jlgkej6ShFFO2ooqRnqxICj0Gen648ui370nQtKI36o0Jl6IhpwGFCOlpb557jLEh4xAN0Rq5AfZQ36XOdTLj2CWlghdUXoon/owYtlV3jfrVsOaFJwHu4Q6KKvVN9/dQYSTkHCFGZLSsa1YeRkSimJG8pqbSBIjPEZD4mjkKCTSyyZH5vBAKwMYREI/ruBE/T2RoVDKNPS1s1hRzvYK8b+ek6jgzMsojxNFOJ5+FCQMqggWicEBFQQrlmpAWFC9K8QjJBBWOteaDsGePXkeOscNW/PNSb15UcZRBXtgHxwCG5yCJrgGLdAGGDyCZ/AK3own48X4MD6n1opRzuyCP2V8/wAUwKWr</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 57

Geometric Intuition

Decision boundary
Weight vector is perpendicular
to the boundary. Why?
1
1 1
0 1 1
0 0 0
0
0

Sebastian Raschka STAT 453: Intro to Deep Learning 58

Geometric Intuition
Decision boundary
Weight vector is perpendicular
to the boundary. Why?
1
1 1
0 1 1
0 0 0 Remember,
0 (
0 0, wT x  0
ŷ =
1, wT x > 0
<latexit sha1_base64="dKrDqbr36yVAapgs2DTA7WQPTxQ=">AAACWHicfVFNSwMxEM2u9aPrV9Wjl2ARPIjsiqAgiujFY4VWC00t2XS2DWazS5JVy7J/UvCgf8WLabuIVnEg8ObNm8nkJUwF18b33xx3rjK/sLhU9ZZXVtfWaxubtzrJFIMWS0Si2iHVILiEluFGQDtVQONQwF34cDWu3z2C0jyRTTNKoRvTgeQRZ9RYqldLyJCafFTgM0xCGHCZMztNF56/j8kpJjE1wzDKn4r75lfyXGAiAPuYEC/4V3Y+FRGQ/XJur1b3D/xJ4N8gKEEdldHo1V5IP2FZDNIwQbXuBH5qujlVhjMBhUcyDSllD3QAHQsljUF384kxBd61TB9HibJHGjxhv3fkNNZ6FIdWOV5az9bG5F+1Tmaik27OZZoZkGx6UZQJbBI8dhn3uQJmxMgCyhS3u2I2pIoyY//CsyYEs0/+DW4PDwKLb47qF5elHUtoG+2gPRSgY3SBrlEDtRBDr+jDqTjzzruL3EW3OpW6TtmzhX6Eu/kJ57ixqg==</latexit>

T
w x = ||w|| · ||x|| · cos(✓)
{
<latexit sha1_base64="n4Z9qoOdfiH+D4+P0e2A2JjgSUY=">AAACOXicbVDLSgMxFM3UV62vqks3wSLUjcyIoBuh6MZlhb6gU0smzdjQzGRI7mjLtL/lxr9wJ7hxoYhbf8BMW1GrBwIn59zLvfd4keAabPvRyszNLywuZZdzK6tr6xv5za2alrGirEqlkKrhEc0ED1kVOAjWiBQjgSdY3eudp379hinNZViBQcRaAbkOuc8pASO182U3IND1/OR2dFXBX5/+CJ/i4fDbGw6xSzsSfoj9b9GlUhdd6DIg++18wT6wx8B/iTMlBTRFuZ1/cDuSxgELgQqiddOxI2glRAGngo1ybqxZRGiPXLOmoSEJmG4l48tHeM8oHexLZV4IeKz+7EhIoPUg8Exlurae9VLxP68Zg3/SSngYxcBCOhnkxwKDxGmMuMMVoyAGhhCquNkV0y5RhIIJO2dCcGZP/ktqhweO4ZdHhdLZNI4s2kG7qIgcdIxK6AKVURVRdIee0At6te6tZ+vNep+UZqxpzzb6BevjEyMXruA=</latexit>
<latexit sha1_base64="tFROaEGhmnnMpUlcEPFjR2z/BNI=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9dDL+27Vq3lzkVXwC6hCoUbf/eoNEpbFKA0TVOuu76UmyKkynAmcVnqZxpSyMR1i16KkMeogn286JWfWGZAoUfZJQ+bu74mcxlpP4tB2xtSM9HJtZv5X62Ymug5yLtPMoGSLj6JMEJOQ2dlkwBUyIyYWKFPc7krYiCrKjA2nYkPwl09ehdZFzbd8f1mt3xRxlOEETuEcfLiCOtxBA5rAIIJneIU3Z+y8OO/Ox6K15BQzx/BHzucPm4aNZQ==</latexit>

So this needs to be 0 at the boundary,

and it is zero at 90 <latexit sha1_base64="TH4rbGhP6Fn1kar1I9QiqEcvskc=">AAAB73icbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRb0YvHCvYD2lg220m7dLOJuxuhhP4JLx4U8erf8ea/cdvmoK0vLDy8M8POvEEiuDau++0UVlbX1jeKm6Wt7Z3dvfL+QVPHqWLYYLGIVTugGgWX2DDcCGwnCmkUCGwFo5tpvfWESvNY3ptxgn5EB5KHnFFjrfaV+9BlXLFeueJW3ZnIMng5VCBXvVf+6vZjlkYoDRNU647nJsbPqDKcCZyUuqnGhLIRHWDHoqQRaj+b7TshJ9bpkzBW9klDZu7viYxGWo+jwHZG1Az1Ym1q/lfrpCa89DMuk9SgZPOPwlQQE5Pp8aTPFTIjxhYoU9zuStiQKsqMjahkQ/AWT16G5lnVs3x3Xqld53EU4QiO4RQ8uIAa3EIdGsBAwDO8wpvz6Lw4787HvLXg5DOH8EfO5w9sfo+O</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 59

Geometric Intuition
Every input vector on this side
What else does this mean? will have an angle with the weight
vector that is < 90
<latexit sha1_base64="TH4rbGhP6Fn1kar1I9QiqEcvskc=">AAAB73icbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRb0YvHCvYD2lg220m7dLOJuxuhhP4JLx4U8erf8ea/cdvmoK0vLDy8M8POvEEiuDau++0UVlbX1jeKm6Wt7Z3dvfL+QVPHqWLYYLGIVTugGgWX2DDcCGwnCmkUCGwFo5tpvfWESvNY3ptxgn5EB5KHnFFjrfaV+9BlXLFeueJW3ZnIMng5VCBXvVf+6vZjlkYoDRNU647nJsbPqDKcCZyUuqnGhLIRHWDHoqQRaj+b7TshJ9bpkzBW9klDZu7viYxGWo+jwHZG1Az1Ym1q/lfrpCa89DMuk9SgZPOPwlQQE5Pp8aTPFTIjxhYoU9zuStiQKsqMjahkQ/AWT16G5lnVs3x3Xqld53EU4QiO4RQ8uIAa3EIdGsBAwDO8wpvz6Lw4787HvLXg5DOH8EfO5w9sfo+O</latexit>

Decision boundary

1 Assume origin (0, 0)

1 1
0 and no bias
1 1
0 0 0
0
0

So, we could scale the weights and/or inputs by an arbitrary factor and
still get the same classi cation results
(but large inputs will take much longer to converge if you check the bounds
we de ned previously ...)

Sebastian Raschka STAT 453: Intro to Deep Learning 60

fi
fi
Geometric Intuition

input vector for an example with label 1

weight vector must be somewhere such that the angle
CORRECT SIDE
is < 90 degrees to make a correct prediction
WRONG SIDE
The dot product will then be positive, i.e., > 0, since
wT x = ||w|| · ||x|| · cos(✓)
<latexit sha1_base64="n4Z9qoOdfiH+D4+P0e2A2JjgSUY=">AAACOXicbVDLSgMxFM3UV62vqks3wSLUjcyIoBuh6MZlhb6gU0smzdjQzGRI7mjLtL/lxr9wJ7hxoYhbf8BMW1GrBwIn59zLvfd4keAabPvRyszNLywuZZdzK6tr6xv5za2alrGirEqlkKrhEc0ED1kVOAjWiBQjgSdY3eudp379hinNZViBQcRaAbkOuc8pASO182U3IND1/OR2dFXBX5/+CJ/i4fDbGw6xSzsSfoj9b9GlUhdd6DIg++18wT6wx8B/iTMlBTRFuZ1/cDuSxgELgQqiddOxI2glRAGngo1ybqxZRGiPXLOmoSEJmG4l48tHeM8oHexLZV4IeKz+7EhIoPUg8Exlurae9VLxP68Zg3/SSngYxcBCOhnkxwKDxGmMuMMVoyAGhhCquNkV0y5RhIIJO2dCcGZP/ktqhweO4ZdHhdLZNI4s2kG7qIgcdIxK6AKVURVRdIee0At6te6tZ+vNep+UZqxpzzb6BevjEyMXruA=</latexit>

Sebastian Raschka STAT 453: Intro to Deep Learning 61

Geometric Intuition

input vector for an example with label 1

This is the new weight vector

CORRECT SIDE

WRONG SIDE

For this weight vector, we make a wrong prediction;

hence, we update

Sebastian Raschka STAT 453: Intro to Deep Learning 62

The Perceptron Learning Algorithm

• If correct: Do nothing if the prediction if output is equal to the target

• If incorrect, scenario a):

If output is 0 and target is 1, add input vector to weight vector

• If incorrect, scenario b):

If output is 1 and target is 0, subtract input vector from weight vector

Guaranteed to converge if a solution exists

(more about that later...)

Sebastian Raschka STAT 453: Intro to Deep Learning 63

Perceptron Conclusions

The (classic) Perceptron has many problems

(as discussed in the previous lecture)

• Linear classi er, no non-linear boundaries possible

• Binary classi er
• Does not converge if classes are not linearly separable
• Many "optimal" solutions in terms of 0/1 loss on the
training data, most will not be optimal in terms of
generalization performance

Sebastian Raschka STAT 453: Intro to Deep Learning 64

fi
fi
https://qph.fs.quoracdn.net/main-qimg-305eb8136c4a20f348bb7ab465bc2e10 http://theconversation.com/want-to-beat-climate-change-protect-our-natural-forests-121491

Sebastian Raschka STAT 453: Intro to Deep Learning 65

Perceptron Fun Fact

[...] Where a perceptron had been trained to distinguish between - this was for
military purposes - it was looking at a scene of a forest in which there were
camou aged tanks in one picture and no camou aged tanks in the other. And the
perceptron - after a little training - made a 100% correct distinction between
these two di erent sets of photographs. Then they were embarrassed a few hours
later to discover that the two rolls of lm had been developed di erently. And so
these pictures were just a little darker than all of these pictures and the
perceptron was just measuring the total amount of light in the scene. But it was
very clever of the perceptron to nd some way of making the distinction.

-- Marvin Minsky, AI researcher & author of the "Perceptrons" book

Source: https://www.webofstories.com/play/marvin.minsky/122

Sebastian Raschka STAT 453: Intro to Deep Learning 66

fl
ff
fi
fi
fl
ff

Logical Question Paper
No ratings yet
Logical Question Paper
2 pages
CS3491 - Aiml - Unit V Neural Networks
No ratings yet
CS3491 - Aiml - Unit V Neural Networks
42 pages
Worksheet 1 Ch5
No ratings yet
Worksheet 1 Ch5
4 pages
Year 11 Mock Revision Pack
No ratings yet
Year 11 Mock Revision Pack
385 pages
Lecture 30 Histology of Cns
No ratings yet
Lecture 30 Histology of Cns
56 pages
Final ML
No ratings yet
Final ML
49 pages
Learning Vision From Models Rivals Learning Vision From Data (Google, MIT 2023 ) SynCLR
No ratings yet
Learning Vision From Models Rivals Learning Vision From Data (Google, MIT 2023 ) SynCLR
21 pages
Wavefit: An Iterative and Non-Autoregressive Neural Vocoder Based On Fixed-Point Iteration
No ratings yet
Wavefit: An Iterative and Non-Autoregressive Neural Vocoder Based On Fixed-Point Iteration
8 pages
Fragile Networks Identifying Vulnerabilities and Synergies in An Uncertain World 1st Edition Anna Nagurney Instant Download
No ratings yet
Fragile Networks Identifying Vulnerabilities and Synergies in An Uncertain World 1st Edition Anna Nagurney Instant Download
58 pages
Poster Genome Sco Chromosome 2
No ratings yet
Poster Genome Sco Chromosome 2
1 page
BMC Adv Partitions
No ratings yet
BMC Adv Partitions
32 pages
COMP7015 Assignment 4 Sample Solutions
No ratings yet
COMP7015 Assignment 4 Sample Solutions
4 pages
Statistics
No ratings yet
Statistics
3,280 pages
Conditional Formatting
No ratings yet
Conditional Formatting
1,926 pages
Pay BKN
No ratings yet
Pay BKN
3,788 pages
Phy Sin Marking
No ratings yet
Phy Sin Marking
40 pages
TestReport 4030106100
No ratings yet
TestReport 4030106100
1 page
SPC Format
No ratings yet
SPC Format
61 pages
Dsa Sheet Aarti Gupta
No ratings yet
Dsa Sheet Aarti Gupta
7,023 pages
Summary of Data Base Clue and Cue Problem List Initial Diagnosis Planning Diagnosis Therapy Monitoring Education
No ratings yet
Summary of Data Base Clue and Cue Problem List Initial Diagnosis Planning Diagnosis Therapy Monitoring Education
2 pages
Feb Attendance
No ratings yet
Feb Attendance
5,147 pages
Worksheet 2 Ch5
No ratings yet
Worksheet 2 Ch5
4 pages
Crime Statistics 2021 2022 Web
No ratings yet
Crime Statistics 2021 2022 Web
16 pages
Week 6 Recitation Assignment
No ratings yet
Week 6 Recitation Assignment
32,767 pages
Domino-Portraits by Robert Bosch
No ratings yet
Domino-Portraits by Robert Bosch
4 pages
Excel 1
No ratings yet
Excel 1
27 pages
Computer Graphics Lab Manual: Sachdeva Institute of Technology
No ratings yet
Computer Graphics Lab Manual: Sachdeva Institute of Technology
5 pages
Biological Molecules PDF Dna Carbohydrates
No ratings yet
Biological Molecules PDF Dna Carbohydrates
1 page
Excel Obstacle Course Advanced XTIMER
No ratings yet
Excel Obstacle Course Advanced XTIMER
2,067 pages
MCAT Prep Questions
No ratings yet
MCAT Prep Questions
11 pages
INH1
No ratings yet
INH1
3 pages
April To Sep
No ratings yet
April To Sep
1,646 pages
3F3 3 Fast Fourier Transform
No ratings yet
3F3 3 Fast Fourier Transform
50 pages
CS 106: Artificial Intelligence: Informed Search
No ratings yet
CS 106: Artificial Intelligence: Informed Search
48 pages
Slip Regression Classification
No ratings yet
Slip Regression Classification
2 pages
Prealgebra and Introductory Algebra Fourth Edition (Margaret L. Lial, Diana L. Hestwood Etc.) (Z-Library)
No ratings yet
Prealgebra and Introductory Algebra Fourth Edition (Margaret L. Lial, Diana L. Hestwood Etc.) (Z-Library)
1,338 pages
12th Bps Salary Arrears Calculator. Clerk
No ratings yet
12th Bps Salary Arrears Calculator. Clerk
1,639 pages
Microsoft Excel Workbook 2.3 - Introducing Spreadsheets
No ratings yet
Microsoft Excel Workbook 2.3 - Introducing Spreadsheets
2,341 pages
World War One
No ratings yet
World War One
1 page
Microsoft Excel Workbook 2.2 - Introducing Spreadsheets
No ratings yet
Microsoft Excel Workbook 2.2 - Introducing Spreadsheets
2,341 pages
Quiz (8603) For Assignment 03213363218
No ratings yet
Quiz (8603) For Assignment 03213363218
51 pages
M1 SuppEx 1
No ratings yet
M1 SuppEx 1
7 pages
Loretta PDF
No ratings yet
Loretta PDF
3 pages
Ics 2306 - Computer Networks
No ratings yet
Ics 2306 - Computer Networks
2 pages
Time Table Feb 2025 Eee
No ratings yet
Time Table Feb 2025 Eee
33 pages
Microsoft Excel Workbook 2.4 - Introducing Spreadsheets
No ratings yet
Microsoft Excel Workbook 2.4 - Introducing Spreadsheets
2,341 pages
AAI Fire Service Paper PDF 2
No ratings yet
AAI Fire Service Paper PDF 2
34 pages
Sheet 1
No ratings yet
Sheet 1
1,356 pages
Tables
No ratings yet
Tables
5 pages
Week 8 - Variational Inference - AC
No ratings yet
Week 8 - Variational Inference - AC
15 pages
General Mathematics Mod 5
No ratings yet
General Mathematics Mod 5
25 pages
Wyk - 2122 - M1 - Nicole Lam
No ratings yet
Wyk - 2122 - M1 - Nicole Lam
22 pages
Slip
No ratings yet
Slip
2 pages
2D Exercises
No ratings yet
2D Exercises
27 pages
Evolution and Human Behaviour Darwinian Perspectives On The Human Condition by John Cartwright
No ratings yet
Evolution and Human Behaviour Darwinian Perspectives On The Human Condition by John Cartwright
508 pages
Additional Questions 2
No ratings yet
Additional Questions 2
27 pages