\a\alas, , OPTIMA ATION ‘Mety 5 NaNRee He FE QRUMZ SON METHODS XK the gitlovent ASP as of! _ ORM aAen -bechntgues ose! 0S Following t- | —7 Gradient mescen’. | —? Slochoshe Gradient descent — conbugate.. Gradtont = momonrven — ryorrhod : — Hdapve fast osder motiod —7 Borch — nusmalidavion, ; — polyats Avesaging | | £ 2 aie ; MES MO, Cy-gi). [oss -faneon WD a7 micint ze - — Loss 4uncBon. (We . ‘ : east “unc 2 maxtmze — cose function O 0 o * pilaster —> opbrrigation $5 0, method-techntgue of speedily a mink og an Obsective funetos HED Powomeresiacd,, bY x. meee | Ex “the objective Juneaton may be mean spose \ ‘ ees08 Bon, j 7 NSF 's an objective uncigon' wlfeh Ps | Seeded 40 be aficihaed. Ne | * Cost RO : | wre mais a ‘seBoo hat measuses “the Peslemmance af of mmrdule fos ang qe data.ee KUL Quanittes “the exvow he bude en predete Vowes 0d eeu) actual Value.) hte mating oO hyyothest s votih total Paxametess p-the cost tine Won 25 catcudated , > * | ¥ehe cosk dunction 85 apreetaod by Atblorent optima ation aigustthnS ttth . moatfing. the vatues of antval Ppavometers, ‘ a ' Uypothests ° ho (eda @prxc4Oix: parsarne teeS » Oe 6, \ ‘ cost function 1 To, @)- a Che(o3- poy Groal. « meat seep) Qn far : 10s ; MiINAimMige Co ,0) wdhewe TM -sepwe Seni s the Mumbes of : ak Samples, 4 actuol output b C7 , bh predfeteot output. i ; BD cost" tunciton Moy consist ot mony. rrksimdn potars®. aie Gb cates’ ony one QwboL minimum oxtsk and mnviny focat msienun | CU ospreesent., vt fe, | the” optimigation moy converge fo dtlferent Potts! i 2 : % a canker dierent Showing puint Si, loosting sateFS ace |: xX Gradient © te b. rreasuseS < how, much -the output of — A gradien! ? fe anges vorth seSronse —0 e av’ dunction ch changes! th input | value. > > HE also Measuses —the chonge fy off weights worth segord ~to -Ahe change in orsoy > Righes the gradient, tho Stoopes the slope and ake fagtes a model eon fear. sut *P Srope Ss yOsO Ah© modeh Stops heorses > Fai mathematteak -tevms oO grodkene Ts O porto esate wstth respecte fast i Spect +o vt 1OPUbES sich tone SECEP zed 2S feck. | X Gradient = Des see NT ARR —> is fs an “eprint gation algostthm bo ong Tackioe Geowing models AS well OS reutal reroosk Hhodels ky means of minteshat eee beboeen actual, and &y, TIS, Gn Rewated optimigatfon Ardiogy he refer Yatue Et wos | oxtoinait % ts aide wy F PromEA CAUCHY fm 1844, i ct ° On AS Steere st “descent. > It is O Geer) conver —+ We fy Detion | a Best way) 0 fed Qocad mismo 2 a function wing gf soction Pected mesyjts Algesttho fo oa ction, Socatis mo sinum 4 v S . descents htawog. ) iA . Fah we mee words Oo negative qsadtent Os away tom the gradient of’ tre inction| Ot the cursient point, Pt wis give‘the focal Bees er mmm& 4 xt f We OVE toyoorsd 5 Ov posite qvaday CF Noyutds -the gradient: of he hic Gn Ol -he cuir’ pint «woe Wilh gebi the bows. wax 'ansen or thes caves. funcdon . Non- conver conter 7 Compute. the. grad rent CSio pe) the. Presse oxdey esta ve of ~the fincttn ots that potot. Ss make. Sree 0 “the, Oppostte__ Atsectioy to He » Gsadtentioprostt® Atcectton of stope-fncsco™ e PB Nom, l tage : Boo. weight: vidatan! sui < tS given Megtoompy rhe [i £9070 “ge i j= 95- * oe 7660, ©), K Steps ‘nvolved 0, Qradont descent oe.[row bhe. Curent point + hy Olpho. Hines “tKe | Jae gnariiente™ at thal pine. ; ‘ | none How leasing ato Lioce -the pestoimence of” \% “tearing sake \- OP Lim’ grtion wwe wr . : | rr. 13 -the Shge of the steps -thot ave taken Ap seach “the * nicky alice. ‘smo value ¢ | PANS {9 Jyeteanty ©. j tod based op “te —> at 8 evaluated Ond | unde ' hy behaviow of the cost efunction. (eadrg rate must) be chosen ‘casetully- . —*toasping sate Ps Op tinal « modes. convesges t to the iMMUrD,, > af the. Joownig vate fs Elis more Ure burt i wore “efi pi ah * / fora js 6 Smo tcontog “The. optimal £ “100 Jersge ore seguises Searching a tor belie “feats is r6ORR h pe) "t. a roe _ Swhfey -seacheS causes ie fr erbeSrours pois’ the" chas Be Shy ' soir) ‘poink ie : " Updates Case — tev ah eanotsg ate 75, to° gine ea Ree Smolt SE AakeS toove Lime j abot a5, e > es, optimal. 7 Foxe z rs at leoisnta are “then “Phe OP miaation process Converge es mdatinum’ vatue | utkKin molevant Hime Pested... =D “He aighewy ¥ cage. 31. TH ‘Jeatoleg wate T= too high, tt pox ) oe Overstrors ~ Dra dhesge>, moves awoy : Lop the Midieourd po. mascot valte, “the oS oO seat the pexty seNANCa argestthm AecseaseSssp x Types of) Gradtent, Reseent : Hl veh nnn ma a -> Based on “the, one to Vasious «at ing. Aah, eeyee grote “Jesrorst Loosing gost CArhe Aivided, into : ‘ . * Batch Gesadient _bescent » sii ay % StochaSWe | Gradfonk | Op9qenk . | -Kowinrt batch Grecian’ pescent , geod rity Y 1 * Seach Grodtent’ descent Caan) 79 Used *0,, Bea ~the estos tos eO0h. point 8p: athe vatoing rend | Update, “the. model odter evaluating on eg 4 “examples. | Ne. Proceduse fs) ov0n 9 the. tsaingng. efoch. \ | | , m0 “ Sheng voondls 2 isa greedy arprroach cohese We hove +o, Suny eves a eam Pes fos each Update. Moat, Lages: ot b yk oy [A> atertore sche ceatnt dese euk produces, hess ocise’ 4p, Corrosion Pd 1 1 Othes gradient descont becouse of cpdatio ithe “parameters bit. computing :the avesage oF ON tho fesoieing SampleS -sorho sthon he value “ola Stogle., Sampe, eg Produces Skable | gradient dlescent (comes? He eg! eoinputertiona tly ef kent aa atl esos? Ome Used tox at aig, Samples,} sometimes Stable extor gysadtent con ead to ocak” mirfrna. cod unfike Stochasive ' grodiett descent, nd A6isy SferS Osd “these Lo help to get aut of the Jocat fninima He Here Anove menoty 1S ~veguised -te process * a -the ‘esofning ata,gt, . fy ey > verending on j computer, sesounees TE (a9 JAKE too Song fos processing al raining Samples as O batch. : , > Fochasve Gradient pest Ga) * SGe ‘Sc 0 typeof gradient descent that SUN OMe t¥AMng example poy iteration, vo ro thes Word, iy processes a Axairing © poch 4 Cath rarely uftK a dataset. and Updates each trotrfing OxAMPES PasAMPless One Oct a Kime ' : s ¥0s Le ~sogutses only on@ -beataing. example at O time phone FETS ebakes to, atove. fh a\iocatd ho mmernuyy Fibs ¥ ; ' . \ —rnseneyes ihe Kat %5 Castes -te Rey fq the memory duc too Sing'e. Axcketg exame being processed ty the corkuask ; Feeds Compuraford ly dast os oriy one Sore iS pxdcessed at a tre . ¥ Fox darges datasets, Tt can Cconvesge Aostor _ OS FF ‘couses Updn es to the Updotes wore sequently r 4 : ¥ Due to Frequent Updates , the SterS taker, . foward s thé ‘nila of the 055 function have OSdillatins that can help tv, gek at of te Yoral mninimbng of He 9095 unesen, ns 5 4 1—> DiSadvantages | reat ¥ Pxequent Updates the Stor a-laken athe iinfina’ are very Norsyy , thts cay often goan the gradient clexren\ folo “the Otter, Aveo tfons, AlSo due! to. noisy avers FL may, Ladee, fonges to achieve .convevgence vo! if kK Regutenl updates axe computationally) oy exPensive, | ays , in} i | bh Conk DeSconk + I. a Rats. Gooden AAS sof ? wish batch Gsadtant pescent ts the combnation \ | | ot “both yatch gpoidfent descent and’ Stochasthe sgpodient descent : ae ditdes. tthe, training datasets into Sas paveh “Shae thes nperteenS che, Undales on.) , “those, batches 1 Se Pewarely . ae % | YS Hence we can e&ehetye achfeve & , Specto I _ sitet ae descent with Mig eogtiaty Hane +0) ae on! Allocated rechony. : x Computetionally eR chant E #F ‘produces stove gractiont dastent coesgence , eae acon Bee descen C66) tadike a tail LER iteration ic 4! Bryn SYR DE sseguies, ‘ deoxsring xate Pye Ricl 1 Beguise » Totttol Patan oles. | ‘ Compare: gradient $o¥ moe goby evil! ef) ay Update» ga end wWhike, =f OSPR.ailigha.- wow % . x Sinchast ¢ Geoadtenk with — rementum 7 — wwe wr > The Cemesits of sto chas He Gpadient escent argosth anes ep XK Gersoy arpa Ge focal ste miniendlm, ‘ ©. y ba, Nex sreo ting And missing the 70 bine * ON Oe coset. c0v-lastes: ups olepe Gradient descent. Wd Yo athe meted a pomentinn -& gestgred to OC elevate feasnt ng ate eStecfa\y an othe Prosisof nigh ne, ‘ 5! Cuno e pee an ot ey deret decayny moving aves pus t> Moe Sp thsi “pate oa qunduced a yasase- we 4S athe Connu * the mg ergortthm -bhot | ploys are. vele dikec#oon and sped ot fees: se the» pasamtee> move akyough the pOLsdmo te Space - 34 gansiaa. > Arse we vebct wOnew = wpa OSE geese -@ Wee tod MYE —p 8 -eEedt ao ge 4 bro SN)— ee Ces eer eer tens” Fa KR §—> & YO. if LL tc, 0), Ye O -~ ©: - & 9. Stochastic Gradient pesce ent), LOTT, Moment, Meee EQ. ei 44 F001 Se ery, Oe o4av— eg. J > Qratlont esVimate , Nock evelo city .Ulndare yon, : L Kpemekes ot - Sloehagie gpadiont descen anette. Gy 1a i = Grerting “Kvopped. fn Locak mfohouro . an Over Shooting. ond reo “the. ‘global Mmrgma. > —> Ah byter Perrainé lire 2 -Co,; 4) eteurines Tow Qufekiy 485 Gyrree? wadlans at shee 4 “ psevious gradtonts Storer decay? aglelas- - aa teow 1x Reet dtont SEL with Nest si me TRomentuny— LALO Dep fntesinn Upolarte : Oe oo COmpure gradient Lge = nVo EtCedilo) if) © Mpute Velo ctty update -V¥ env : AMY Uelater gy iby Dy a end wile . 7th Sao WHA nesteroY momentum fhe gsadtont &> evatiatel otter the CUSEHt veld eS aappliod. eb Urdlarting -the. woetpht. i 2 Bas Oexe: ant> Based on updated Value of O | the Joss: B asctsh FS evaluated « challerges of Gradient pescent ‘ i tocal Weima and Saddle pofnt | Bi Vorkesin and Exphding qradioe: o exbotld2 - Sepianed 2 a engostibm + ' . c ae _sAceumuite sjuoved cpadtont sires aa Y uw 2 oar SoS ae RE tot i i Bes. ~sacthev iets adaptive . . | athe teoneirg? yoke 12 “chosen “actoatliag “fo she behavics' at OSS = ncHon . SIKS “Quysitiim % sy! ?odtoluaity od a the ‘Leasning, scutes’ 6f ail porrametess Medulle by. Scat oe irvexsa pete | Ap the sguarse x00 the sum dle | KNsesteak squmed ea ‘_s The Paxametess with the He best or faxgest |, postial, deshatte" of she Loss ey a. cossepindy Napid, decease 2 they Qearsnfng sate , LHe pasametess wtth Small pastas Chevivartive> hove a ~ 3 teonpng wate . a 8S QA vesutt ,AdaGrad might rRematnny Es betome -too Siow,\Cind if, woth eyerelaiy Stop Faprtng PrOGHRS SY » ’ : *¥ athe RMS yroH allgertthros- 1B Chumilate, ‘Squared Gradient. : Yo Sie ty _ Rosermmsnice pistes Cons pe) — a a i AMS peop aalgatittng wend es, ope to AD Pershstiny poktoy’ fo: “the NON Conves setting by Charger the: gradient Lactumulatien \ fnto On Bo eacpone nia It podighted imeving avesage . Cail | Algosithm e—stfted—wy) USS | an | i Sz ronen ia Glecoyfng ‘Overoge- 4D hoses? th i “history Poko! “the rae past, so hot felean Oorvesge Sonal} pier ©. Koding a Cone, bowl. essen eigen th ecient coment : 5 cer Siyf) pt estrrotic A BRIO) shlopine Manet EARLES C BEND) update based Ast ppament 6S ti mate S= 3,8 G-8g “ — Update biased Second moment eS timote + Bee Pvt Az 1)904 A _e"The rame -ADAm clegues fon ghsose “adaptive monents”. TF Moment + RNSpoop ; _bleaore ntiiny helps &9-B Stroetioing the pP&geSs of: oplimiaation and ems prop, helps in eda pth | Deost ng yore. |= By focexposttliig the. Rims prop Algoith™, it uses the, adaptive Beaxning WOK, es concer > aHefmomentun is evaluated by &>! of “the. adient. a “45 Incuuded 10 Eeinating the_ fest asdes Moment —? The 4 Conce pt of RmspsoP the - algosrthm . “hy estimating the second” osdes, npmenr of the groeelient ae ROU ee . > Te yprovides the fofeymation about the Qvemge Step Fae ef Hesent porometers Cenetght and bo) ot 0 reusal etwo%k LoheN Chuving the tyajectow or path towards advieving minimum OY maximim points. : — eee ae consat of asexdging OU HE points ° Tajectoy throu pasametes Space Vidited. by an optimigation olgosithen rf + tesations ot gradient: Uescont visits points COU -.- ECL) then the ourtpre oF _ pier nee FS Be ES ee ee ae