Sn G3 Uplate Jan sama TB 1. Rydap Lacansd Gh phy Bye Laree gy} po 9 shah tg dete Zs Images —» Hin-mox. Normalization PCA —».ieeStanded zation Hear remalizalion® Outlier —» Robust Sealing Nean AE a iin Ba a. pay BN ed Guthes J) waddle 9) Robust Scobing: aa rae 7 es crs Q,—S, 50'f--— 25+, 2 3 = a 4 GM Ser fa ss eee FS 7s : Gradient descont has Sehey Gnaitatons: use &D. be Clleulate ors oe LR mod d LR Simple model optimizer Calewlake grockient AL ae NN Woe ww AL 4! & do,Fj Ee Epon 9)" a : ‘- ania - of ag i Gaia - ye = Zz we Gecoaiuct Ae scent et, Seon. Lireitotion, Single gett d 2 : ey = _ Agile Vee call. wh oy Cost Punction “ty oe one. palate, ‘De pa an all * te Calewdote Cost abso 2 asing all data. _ 4 3 6 pi Me a A\\ data = Batch So Called GD "Botd GD” fe iterates —+- Epoch Epoch = US ~All dle ne = iterationTr Batch GD -»..use all data in one ie tntich2. Fo One eee A iteration. eBotch GD problems : —use all date, sin chator. — ins ae Cause. Preble Ln. eae amount. of data. aderclidad ores ek x - : x ai my Be | unit he be perFormad Tr beta Bion = = B wed chodld Ke memory. 7 "Dota Calculation every ~ikelin Hein —Cantt Rit in. pene bang tie Since. Deb ee fe ig: Data _ BD_ Gantt be efficient 5 Jn Longe data. saddle. =—poik—s 35 feet KA. Tem “atl Spe —decrense. lea to solve this problem. Sto chasti a es use. ee To eewiabiawee to update ss41% 9. pOArsioss 2 EO aide 5 ISM A EE ea = i, oO eae 4g eal / oe Bn. xe capicl? Ttevakion Use %, a ts Kea. ell dala a haut | ar i updates? ot “peat We. inate Batch» me updates. : eas AR rheretions 3 ane: sua Hee ei im iterations ae al Be a Pupckt ma aan ti icaht Wrench. am nd me pointTe .Grodiet check sahter Pinish Epoch. E posh. ane detonate the tte. nde cact Batch: Gul-pass th ree dataset into NM. eh oh once. ,So we clivide the cotaset into number of batches opi as é Lterokions EL we. Seat abate Les = one batch Size+ 20 Epoch = lee = Se. iterattons Ori hose = i & « eS 5a points Se eh ils ae withed e v4 ge —_— Cradient_ ae SS ae Cost Prcainieiaee on. walt dataset ge CSy pContonc! plot. — ae’. stochastic: a One oh dis advan. af Stochaskic GD itis hate ae- ie we make... Chie eAeY. ib Not: ins itevation because neisy ebatizaic wiye misleading 5: as drsulbobosdioMot toni ia dwt * Buk- noisy 5 ah lit cgae hos. Adbvonkag= in saddle ~ poink z s 1 i akss ee as as. 5 Saddle pink. Nese er ar ae Oh p 4° Bo monile ent apd. —E uf hewmen nee al liheBioge oi. _looo % Se =. Fa,000 iterations. aE —epdake Al aba _ SE mee oe of Le TO ee pHing oath) GD Gs oa 5 ete ale practiod Smaltiply ohise Pah aoe Ee 2. 7 Oy -=2 SC a. ae CAs 98h of | 2. Cost ee dae pe)ee eee eee eee eee eee eee eee eee eC ECE ECECUeCECCUeCCUweCUwhUCUCUCUwrmUwmwhUCUwvUCUCUMUEUCUMUCUEUCUwUwhUULUwCUL Lt Date wisi Sapalatsy Divide dt to Shades Each, Batch. hs Be ib eee TE bene St te Batch oil Epoch ji. oh 2 Bee <5. looe Epoch we have Sooo “iteration te Pass om oll dea toh. ‘ L bokal no oki teration =5000) one iteration 2 1 betel Ae TP T pas ob all wns batt d epoch fo ge a . Batch GD om . Ste chadGer GR - Hinw Botok CD A Eack batch has its gradient: —» the Ai f erence between gradient: jn ead batch isu hig. Since_ we do feature Seoking { . ad So grostect Yor vaithin Certain. Scale. Pind, Denit make Check until T Pinish epoch Tn stochastic, Winibateh_» tt is important te onake Ranclon shulhle datoAfter. CobeeQate. Parameters > the. Most : i mportewt point. Js mocled. performance. SubEle. tie arrangement ob. observation +d Practica pall paths mash usable is mine bate) i BOO Q4E™ sa on. PCD. De Bee ietht me MLe iP the dato isnt igh fe gad oy - DL oy Mini B, Bateh most ‘subDic tents Cokie ManoBon ina De ish. & Conver, So_the osicoblakion is. success ir becall minimum and Sackdlle point to avoid the. imodiel Stackin ch _DL- =>. Advanced opt. Pars. DL . ~ it Aw bY ae AS ited of bye ~ Ba Meth « en tke Punctron + a a | Ay Bet. b pasthndend rep 4st | W. teed ze i ee Parameters of ara leperSe, thousand of Parameters in MAL... * Training eee , iE Assume @ Ba Qe gee soe cest toi ty he AT Aw — all W's Ab —all bi” ret: 4 s Apply Chain. Yo less to ge ae Back. /pPrepagation Pa gees Arts = pf 2swe ss ——> ee = ae Eo dw Back pro Bois ws odelitional st tw C a 3 pos to oe : «problems in ones \.2. ‘ont Gradient. Aisgadete. - gradient very small. Ro; Adtde naka puck. ta soley vanishi Wir a radiert We must make surco befoKe we Ne thot the qraclient is? teed ort long. ime+ Homentum based CD What is. the reason. of. —Nanishing.— or. e—Eniplad ing — Gradient inDL.2 Tr Back prop ee js Caleulateat OL. amultipli icakion. Chain rule. of othac terms, if ae is Jess thon A, When. eulltialy rhe NaQue of ° graciemt KO Oe = the volt oF upolate a Update squetion, Accum. | te Bl he hd i momentum: berm 6 <1 "hyper parander Ad the bapininge— a sy gt 8, = So EQ md LA BA ye a" cgi I eS es ace Age yoke Sa eg OE etn ae ea ave aaa Tn hg oSoe @ = oh TG ll BOR ad bar jtermatien Aap de { bud 4 =o 6. ipo i is <2 ATG J+ “Lace gy Fhe a Berge LAV. G Fos te Ue 3 Cag 9 + ELBA» BLasg] Sat Ck pO + BA 6, HBKTE+RTQ egg aas eed i YALEG fe LG i ae Expenenki mo D cocighted Doing fink as = iRenectionVU = BU h-T 8, Sia ee ee Srp ee cai: le Geta xy Buzree Crp aniee td) Ah oSS jig db LEG TT 4g kind ii, a vabae = Chasen 0 On eielag eacliven e on gp Valo. d Pers Sd spoon We Manse Ey Lachaise andy iP T howe. Vanishing gradient ya dtlatle tna bi Aisavantage.s y ied Lis Tn CD ye ee Q = 6 /. E+) t ae ee TR mine at Ze bess be : rod. x0 "Soil jk continue bo 5 Py oe ene ior Aa ee will happen because gras Be One Buti Deu. WWE mo Suctiann i it. Caw. Ganka ind i becouse st Paha dey beised On, Vee iste CM. ga 5 Oe, eee ‘ ; ect RRS $d i NL) dovnlias leas =. ESsu ss. noble wcunepdacil bens Grodient: descent: OS. Cil\ates in and.out. of the. mipimaiy jee maki _U.= turns Near the minima G haben: a coenge “aster Vanilla a GD) ae ipa is « the besti- F spabdk ill Pia. on SX perience... ae«e and bey. Yeo. estpee Solve. OSCi llekbion problem —> Nes trov..Accelerated Cractienk NAG. —— pe ON. & is s—appasite 7 ag = Si + Ave ye oe a momentum since Ee a henck Step. ce Z—nalagead ope “ay BEA ee 6 ad vam oR Base Sb Se AS od ote a> sohe aspille. Bans proslern PEDaly Cee ee Noi ee + practical LAB 4. Se i ’ Problem 1. el mig “ / —Diaglewariable LR vs a a —%| 7 NBGA 5 “uid Aon es _abjective = 3 = Poy a2 procimatly Vane —— cata ies a ie She bine GS Cx)s- 4-6 pe eee be Pind &_, é suck thet a o ca js naan Emre st ee van chion HSE = 509,G)-1 Echo f Dns ob aot | Find Hedel_parameters Ceo) NSE 2g ae St a aa : an’ Pisdandiditin® stalls G0, 6 aa gy 3 fiedichie bce te G4 a Ce SemOR GON yee me) Ses al fark 2,om g39458 Dates Nes +E OS ¥eT tret*® Jd 2a. peas IM ay Es co gu) Je ~GEe —— \ ellZ. ie ge 4? Yeedec = a \flel™ ~ = eet J PTBeat ae se ge be. a OS ee eee Donk Assign new Voriobles in preg ramminy justo y eh the trite oli as OR? my iterdibe. aie ate 8k ee 4-258, = 0 4 A=000|__ Fortin —Tange Co. } leoe).: ae