Download as PDF, PPTX
Uploaded byHuahai Yang
Generative AI: Past, Present, and Future – A Practitioner's Perspective
The document provides an overview of generative AI, detailing its historical development, technical foundations, and applications, particularly highlighting its ability to create diverse content. It discusses the limitations of current generative models and assesses their cognitive capabilities, emphasizing challenges such as hallucination and the lack of agency and expressiveness. The author suggests integrating generative AI with traditional AI approaches to improve functionality and addresses the impact of generative AI on job markets and productivity.
In this document
Powered by AI
Overview of presentation by Huahai Yang on generative AI, its past, present, and future.
Explores how the internet transformed generative AI, shaping its potential and applications.
Agenda includes the rise of generative AI, historical progress, psychological assessments, and future paths.
Generative AI creates content using machine learning and deep learning, generating various data types.
Generative AI achieved 100 million monthly active users in two months, illustrating rapid growth in applications.
Questions whether generative AI signals the beginning of artificial general intelligence.
Discusses the historical context and significant contributors to the field of artificial intelligence.
Tracks the important historical milestones from the 1940s to present in the development of generative AI.
Outlines McCulloch and Pitts' early theories about neural action potentials representing logical propositions.
Rosenblatt introduced the perceptron model as a probabilistic method for neural organization and learning.
Minsky and Papert formalized perceptrons and highlighted their limitations, ushering in an AI winter.
Rumelhart and colleagues advanced learning algorithms to include backpropagation for multi-layer perceptrons.
Hinton introduced algorithms to better train deep networks, solving overfitting and convergence issues.
Krizhevsky's work on ImageNet advanced deep learning with practical applications and reduced classification error.
Explains the significance of attention mechanisms in transformers for managing long-range dependencies in data.
Discusses advanced generative language models like GPT-3, which has evolved with large parameter configurations.
Overview of generative AI achievements and their impact on various domains.
Highlights PARRY's interaction with psychiatrists to illustrate challenges in current AI assessments.
Contrasts behaviorism principles with generative AI functionalities and learning methods.
Chomsky's criticisms of behaviorism and responses regarding generative AI's capabilities and creativity.
Investigates generative AI's cognitive capabilities and what tasks it excels or struggles in.
Outlines generative AI's ability in perception and language but challenges in organization and reasoning.
Details the core functions of generative AI in mapping data for various tasks.
Describes how reasoning problems can often be approached as perceptual tasks in training models.
Evaluates what generative AI can effectively execute in both perceptual and non-perceptual tasks.
Explores difficulties generative AI faces in organizing knowledge and aligning with societal values.
Details the phenomenon of hallucination where AI blurs the line between fact and fiction.
Discusses how minor perturbations in input can lead to significant misinterpretations in AI learning.
Highlights vulnerabilities in AI where systematic attacks can mislead model behavior.
Evaluates the high training cost and resource demands associated with large generative AI models.
Outlines the constraints of geometric representations in AI compared to human interpretability.
States the difficulties in adapting AI models and communicating with them effectively.
Investigates the lack of agency in generative AI models and their reactive nature.
Describes the deficient sensory understanding and emotional awareness present in generative AI systems.
Assesses which job categories could be displaced and the overall workplace impact of generative AI.
Explores paths forward for generative AI, focusing on integration and human-AI collaboration.
Discusses the need for cognitive systems that combine and contrast bottom-up and top-down processing.
Links generative AI advancements back to Good Old AI, proposing an integration for better functionality.
Describes two approaches for integrating generative AI within existing AI frameworks.
Lists essential components needed for deploying generative AI in enterprise settings.
Explains Juji's approach to creating responsive AI featuring individualized conversation strategies.
Describes how Juji combines symbolic and data-driven elements to enhance generative AI functionalities.
Outlines Juji's method for managing conversation topics effectively within its AI systems.
Discusses the future vision for more efficient and capable generative AI systems.
Explores avenues for enhancing human-AI interaction and alignment in generative AI developments.
Summarizes the transformative potential of generative AI in reshaping various sectors.
End of the presentation by Huahai Yang, providing contact information for further inquiries.
More Related Content
Similar to Generative AI: Past, Present, and Future – A Practitioner's Perspective
Generative AI: Past, Present, and Future – A Practitioner's Perspective
- 1. GENERATIVE AI: PAST, PRESENT,AND FUTURE – A PRACTITIONER'S PERSPECTIVE Huahai Yang Co-founder & CTO, Juji, Inc. August 14, 2023
- 2. INTERNET CHANGED THEWORLD What would Generative AI do? 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 2
- 3. AGENDA Rise of GenAI Howwe got here Assessment from psychology Paths forward 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 3
- 4. GENERATIVE AI Generative AIis a subset of artificial intelligence that focuses on creating new content. It is often based on the frameworks of machine learning and deep learning. The systems learn patterns, features, and correlations from massive amounts of data, and they can generate output such as images, music, voice, text, code, or other types of content that mirrors the learned data. 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 4
- 5. GENERATIVE AI APPLICATIONS FastestGrowing App in History Re ac h e d 1 0 0 m illion m ont hly ac t ive use r s in t wo m ont hs Open Models L ar g e L ang uag e Mode ls wit h a g r owin g e c osyst e m Fast Growing Text to Image App 1 5 m illion use r s Open Models T e x t t o Im ag e Mode ls 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 5
- 6. INFLATED EXPECTATION? Beginning ofartificial general intelligence (AGI)? 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 6
- 7. ORIGIN OF AI Goals Thestudy is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. Some Participants • John McCarthy • Marvin Minsky • Oliver Selfridge • Claude Shannon • John Nash 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 7 1956 Dartmouth Summer Research Project on Artificial Intelligence • Herbert Simon • Allan Newell • John H. Holland • William Ross Ashby • Warren S. McCulloch
- 8. LANDMARKS OF GENERATIVEAI 1940s 1950s-1970s 1980s-2006 2006-present McCulloch, Warren S., and Walter Pitts. "A logical calculus of the ideas immanent in nervous activity." The bulletin of mathematical biophysics 5 (1943). • Rosenblatt, F. "The perceptron: a probabilistic model for information storage and organization in the brain." Psychological review 65.6 (1958). • Minsky, M. L. and Papert, S. A. Perceptrons: an Introduction to Computational Geometry. MIT Press (1969). • Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." nature 323.6088 (1986). • Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18.7 (2006). • Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "ImageNet classification with deep convolutional neural networks." Advances in neural information processing systems 25 (2012). • Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017). 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 8
- 9. MCCULLOCH, WARREN S.,AND WALTER PITTS. (1943) A LOGICAL CALCULUS OF THE IDEAS IMMANENT IN NERVOUS ACTIVITY. • All-or-nothing nature of neural action potentials (threshold) • “Response of any neuron is equivalent to a logic proposition which proposed its adequate stimulus.” • Physiological relations of nervous activities correspond to relations among the propositions. • Neuron is a logic expression of disjunction, conjunction, negation, i.e. a Boolean function • Learning (facilitation or inhabitation) as changing the function expression • Prove some neural activities are realizable by this calculus • Equivalent to Turing machine (1936), hence an universal computing machine 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 9
- 10. ROSENBLATT F. (1958)THE PERCEPTRON: A PROBABILISTIC MODEL FOR INFORMATION STORAGE AND ORGANIZATION IN THE BRAIN. • Propositional logic expression -> vector dot product • Probabilistic interpretation of stimuli contribution p(xi)p(d|xi) • Permit geometrical interpretation of tasks • Proposed a learning algorithm • Given a series of stimuli samples and corresponding outcome • Calculate predicted values (1) • Update weights by prediction error (2) • Iterate • Converge when sample features are linearly separable • All these become basic ingredients of GenAI today • “Father of Deep Learning” – C.C. Tappert, 2019 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 10 (1) (2) Sum f y x1 x2 x3 w1 w3 w2
- 11. MINSKY, M. L.AND PAPERT, S. A. (1969, 1988) PERCEPTRONS: AN INTRODUCTION TO COMPUTATIONAL GEOMETRY. • Formal treatment of Rosenblatt’s perceptrons • Deal with simple (single layer, no loops) perceptron only • Prove some theorems regarding the ability of simple perceptrons to recognize some global patterns • Connectedness (figure-ground) • Parity (odd, even) • Notably, simple perceptrons with limited number of A- unites that has local connections only, cannot handle XOR patterns • People (incl. authors) extrapolated this limit to all perceptrons • First AI winter • “This is quite a humorous turn of events. The psychologists offer an implemented information- processing model, which the computer scientists reject in favor of a creative psychological theory!” - J.B. Pollack, 1988 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 11 XOR
- 12. • RUMELHART, DAVIDE., GEOFFREY E. HINTON, AND RONALD J. WILLIAMS. (1986) LEARNING REPRESENTATIONS BY BACK- PROPAGATING ERRORS. • Improve Rosenblatt’s learning algorithm: backpropagation • Learning non-linearly separable features needs multiple layer perceptrons • Difficult to converge in multiple layers • Introduce these changes • Linear threshold function f is replaced by non-linear function, e.g. sigmod (3) • Error function is replaced by Mean Squared Error (4) • Pass back partial derivatives of error (5) • Update weights only after going through all samples (6) 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 12 (3) (4) (5) (6)
- 13. • HINTON, GEOFFREYE., SIMON OSINDERO, AND YEE-WHYE TEH. (2006) A FAST LEARNING ALGORITHM FOR DEEP BELIEF NETS. • Some problems in training of deep network • Vanishing/exploding gradients • Overfitting • Explaining away • Solution • Contrastive diverge learning for Restricted Boltzmann Machines (RBM) • Single layer -> two layers, one up, one down • Gibbs sampling back and forth until convergence • Measure KL divergence between input data and generated data to update weights • Stacked RBMs build up abstractions layer by layer • Able to see the generated images at each layer 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 13
- 14. • KRIZHEVSKY, ALEX,ILYA SUTSKEVER, AND GEOFFREY E. HINTON. (2012) IMAGENET CLASSIFICATION WITH DEEP CONVOLUTIONAL NEURAL NETWORKS. • Breakout moment of deep learning (DL) • Won ImageNet Large-Scale Visual Recognition Challenge • Reduce top-5 error from 26% to 15.3% • Machine learning is dominated by DL since • Techniques • GPU enable training of large DL network • Rectified Linear Unit (ReLU) non-linear function • Dropout to reduce overfitting 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 14
- 15. • VASWANI, ASHISH,ET AL. (2017) ATTENTION IS ALL YOU NEED. • Transformer for predicting next token in a sequence • self-attention mechanisms handle long range dependencies • process the entire sequence at once, highly parallelizable • less gradient vanishing/exploding, flexible 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 15
- 16. INSTRUCTGPT, CHATGPT, GPT-4,GPT-5… • Larger models, GTP-3 has 175 billion parameters • Align GPT to user tasks and requirements • Supervised fine-tuning (SFT) • Reinforcement learning from human feedback (RLHF) • Mixture of experts (MoE) 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 16
- 17. ASSESSMENT What has GenAI achieved? 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 17
- 18. TURING TEST ISINADEQUATE • PARRY has passed Turing Test in 1972 • PARRY simulated a person with paranoid schizophrenia • One group of experienced psychiatrists interacted with either PARRY or real patients • Another group of 33 psychiatrists were shown transcripts • They were asked to tell which were human, which were computer programs • They could tell correctly 48% of the time 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 18
- 19. GENERATIVE AI ISBEHAVIORISM Behaviorism Objective observable behaviors Environment determines behaviors Adapt via learning only All organisms learn in similar ways Behaviors can be shaped via reinforcement No such thing as mind * Gen AI End to end Data driven Model change is via training only Learn all tasks in similar ways Behaviors can be shaped via reinforcement Gen AI has mind already; No such thing as mind 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 19 * B.F. Skinner, Can Psychology Be a Science of Mind? American Psychologist, November 1990, Vol. 45, No. 11, 1206-1210
- 20. CHOMSKY’S ATTACK ONBEHAVIORISM DOES NOT WORK ON GEN AI Criticisms Lack of innate abilities Absence of creativity Reject internal mental states Response Pretrained models considered innate Gen AI is shown to be creative Gen AI does have internal mental states 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 20
- 21. ASSESSMENT IN COGNITIVECAPABILITIES 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 21
- 22. WHAT HAS GENERATIVEAI ACHIEVED? Able to do Perception Memory Language Unable to do Organization of knowledge Mental images and propositions Attention and consciousness 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 22 Depends Problem solving and creativity Decision making and reasoning
- 23. CORE COMPETENCE: LEARNMAPPING IN VECTOR SPACES • Image classification: pixels => labels • Image generation: labels => pixels • Regression: raw data => numbers • Embedding: raw data => vectors • Prompting: text sequence => text sequence • Perception: forward mapping • Generation: backward mapping 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 23
- 24. PERCEPTION SOLVING REASONINGPROBLEMS • Reasoning problems -> perceptual problems • After seeing enough cases • Chunking • Deep learning builds abstraction layer by layer • Training with enough data • Also chunking • Learned knowledge representation may not be the same as human’s 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 24 • Reporter: How many moves do you see ahead while playing chess? • Capablanca: Only one, but it’s always the right one
- 25. WHAT GENERATIVE AICAN DO: PERCEPTUAL TASKS PERCEPTUAL Perception Memory Language NON-PERCEPTUAL Organization of knowledge Mental images and propositions Attention and consciousness 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 25 DEPENDS Problem solving and creativity Decision making and reasoning
- 26. ORGANIZATION OF KNOWLEDGE AtOdds with Human Hallucination Accidental properties Against societal values High cost Poverty of Representation Expressiveness Mental images Propositions Flexibility and velocity of updates 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 26
- 27. HALLUCINATION IS UNAVOIDABLE •No distinction between fact and fiction • Generation is based on probabilistic sampling 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 27
- 28. ACCIDENTAL PROPERTIES MAYBE LEARNED • Minor perturbation in input space, may result in big change in concept space • Learned features may not match human conception 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 28
- 29. ADVERSARIAL ATTACKS AGAINSTINSTRUCTIONS • Universal, transferable adversarial attacks can be systematically trained 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 29
- 30. HUGE COST • Powerinefficient • GPT-3: 1024 GPUs, 34 days to train, 1287 MWh of power • Information inefficient • GPT-3: 175B parameters, 800GB to store • Wikipedia: 22GB to store • Exponential growth of model size • Labor intensive • SFT, RFHL take huge human effort • OpenAI paid Keyan less than $2 per hour , had to read and label between 150 and 250 passages of text per 9-hour shift • Only huge commercial entities can afford to train large models • Monopolization hampers innovation • Lack of openness and accountability • Exacerbate digital divide 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 30
- 31. LACK OF EXPRESSIVENESS •Geometric representation is the only game in town • Elegant and powerful, but does not cover everything • Godel’s incomplete theorem 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 31 • In principle, these can be represented in vector space, but only inefficiently • Propositions • Graphs • World models • Mental images • Knowledge of GenAI is not human interpretable
- 32. CHANGE AND COMMUNICATIONARE DIFFICULT • The only way to change a Gen AI model is to train with raw data • Risk of new data breaks existing behaviors 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 32 • The only way to communicate with a Gen AI model is via the same type of data as training data • Prompt ”engineering” is actual an art • Model output are uncertain • Stochastic generation • Setting temperature to 0 does not guarantee same results • No real time active learning from experience • Models are frozen, there is only nature, no nurture
- 33. ATTENTION AND CONSCIOUSNESS Lackof Agency Top-down processing Proactive action Individual differences Lack of Embedment Knowledge of physicality Empathy 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 33
- 34. LACK OF AGENCY •Reactive, not proactive • No explicit goals, other than please users • Only responsive to external changes, not internal • Will not ask user questions, only answer them 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 34 • There are only data-driven, bottom-up processes • Top down attention mechanism would be nice • No individual differences • Models are essentially the average of training data • May be told to play certain role, no true identity • Cannot detect user’s individual differences
- 35. LACK OF EMBEDMENT •Lack sensory input and output • Sight, sound, touch, taste, smell • Spatial relations, movement, temperature • Understand physical limitations 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 35 • Lack emotion understanding • Fatigue, fear, nervous, happy • Social norm, body language • Low empathy • Limited persuasion and influence
- 36. WOULD GEN AITAKE MY JOB? Most Jobs are Safe Increased personal productivity New job categories will appear Enterprise use is challenging Low Level Creative Jobs Loss Copy writers Illustrators Translators … 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 36 Impact Similar to Search Engine
- 37. PATHS FORWARD Practice: IntegratedAI Common Ingredients Juji ways Science Beyond Current State of Gen AI Human AI collaboration 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 37
- 38. COGNITION REQUIRES TWOWAY PROCESSING Bottom-up Top-down Data-driven Goal-driven Sub-symbolic Symbolic Machine knowledge Human knowledge Generative Curated Frozen Fluid 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 38
- 39. GOAI WAS DEAD,LONG LIVES THE GOAI • Good Old AI (GOAI), • Expert Systems, knowledge graph, semantic Web, etc. • Victim of 2nd AI winter • Failure due to the weak perceptual foundation • Gen AI now provides a solid perceptual foundation. • The same forces leading to rise of Gen AI, apply to GOAI • Powerful GPU => better graph search • Abundant realistic data => better knowledge base • Better software tools and practices • Need to integrate Gen AI with GOAI. 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 39
- 40. TWO ROADS TOINTEGRATION GOAI as Basis Engineer in nature Pragmatic Gen AI as Basis Reductionist in nature Impractical 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 40
- 41. INGREDIENTS FOR ENTERPRISEGEN AI DEPLOYMENT Agent Framework Perceptors and Actuators Central Control Unit Memory Customized Plugins No-Code Platform Friendly Graphical Interface Test and Evaluation Tools Performance Reporting Live Human Integration Infrastructure Scalable Platform DevOps Security Compliance 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 41
- 42. JUJI: RESPONSIBLE EMPATHETICPERSONA (REP) 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 42 Agency • Each conversation backed by own REP • Stateful, keep context • Has its own event loop • Not just reactive, also proactive • Has agenda • Can loop back to agenda • Interruptible • React properly to interruptions • Resume after interruption Individual Difference • Proprietary psychometric models • Measure in serendipity • Good validity • Better reliability than paper- pencil instruments • Individualized experience • Messaging tailored to users • Conversation path customized to users
- 43. JUJI: SYMBOLIC ASBONES, DATA-DRIVEN AS FLESH 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 43 Production Rules • Rule engine as the backbone • Match -> Action • Parallel processing • Best match fires • Match & action can be anything • Patterns • Function calls Gen AI Functions • Gen AI is a function, use it as such in rules • Measure similarity to input • Approve a response • Choose from multiple matches • Choose from multiple responses • Extract entities • Test if a question is asked • Verify a question is answered • Query user data • …
- 44. JUJI: AUTOMATIC DIALOGMANAGEMENT 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 44 Topic Abstraction • Topic as basic dialog unit • Meaningful • Flexible • Composable • Agenda include topics • Tracked to ensure completion • Allow out of order topics • Topics can have own agenda Society of Minds • Topics all have chances • Tried in parallel • Compete for taking effect • All can also contribute • Domain specific language • Topic generation • Topic manipulation
- 45. BEYOND CURRENT STATEOF GEN AI GEN AI Usability Smaller Faster More robust Run in more devices Theory Efficient algorithms Continuous learning Geometric information theory? Integration with GOAI Top-down attention Knowledge sharing New AI Paradigms Causal AI Evolutionary AI Embedded AI 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 45
- 46. RESEARCH ON HUMANAI COLLABORATION EVALUATION Model performance System usability Domain use of AI DESIGN Interaction methods Human in the loop Humanist AI OPENNESS Explainability Accessibility Education ALIGNMENT Safety Privacy Trust 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 46
- 47. SUMMARY Generative AI, oncea whisper, now a song, Overcoming perception, proving skeptics wrong. Behaviorist at heart, in a system hybrid aligned, With old AI, a novel dance designed. Shining bright, productivity's new dawn, A dream realized, a promise drawn. 2023 Generative AI: Past, Present, and Future – A Practitioner's Perspective 47
- 48. THANK YOU Huahai Yang [email protected] Juji.io 2023Generative AI: Past, Present, and Future – A Practitioner's Perspective 48