Generative AI for the rest of us

Generative AI for the rest of us

• 1.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Massimo Re Ferrè Senior Principal Technologist, AWS Generative AI for the rest of us
• 2.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 2 Mainframes Zooming out a bit Technology wave #1 Data center
• 3.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 3 Mainframes Zooming out a bit Personal Computers Technology wave #2 Technology wave #1 Data center
• 4.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 4 Mainframes Zooming out a bit Phyisical Servers Virtual Machines Personal Computers Technology wave #2 Technology wave #1 Data center
• 5.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 5 Mainframes Zooming out a bit Phyisical Servers Virtual Machines Personal Computers Technology wave #2 Technology wave #1 T e c h n o l o g y d e l i v e r y m o d e l Data center Cloud
• 6.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 6 Mainframes Zooming out a bit Phyisical Servers Virtual Machines Personal Computers Containers Functions Technology wave #2 Technology wave #1 T e c h n o l o g y d e l i v e r y m o d e l Data center Cloud
• 7.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 7 Mainframes Zooming out a bit Phyisical Servers Virtual Machines Personal Computers Containers Functions Generative AI Technology wave #3 Technology wave #2 Technology wave #1 T e c h n o l o g y d e l i v e r y m o d e l Data center Cloud
• 8.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. What is Generative AI? 8
• 9.
  © 2023, AmazonWeb Services, Inc. or its affiliates. What is Generative AI (in simple terms) 9 - Traditional AI/ML: “Is this a picture of Rome or Florence?” - [ Discriminative ] - Gen AI: “Compare Rome Vs. Florence for someone interested in history” - [ Generative ]
• 10.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Gen AI “prompt” 10 A T I T S V E R Y C O R E ( T H E L L M - L A R G E L A N G U A G E M O D E L ) , G E N A I I S A F A K E . B U T A U S E F U L O N E submit
• 11.
  © 2023, AmazonWeb Services, Inc. or its affiliates. This is how I like to think about an LLM 11 * or any profession that has nothing to do with a job in IT for that matter q An LLM is akin to a … windsurfer professional* qVery proficient in English qAnd that had memorized all Wikipedia and all IT forums out there (and a lot more) q They know Stack Overflow inside out! But don’t have a window to check the weather (or a watch to check the time, etc) q On their own, they have no relation to reality (beyond what they read) q But they are great at generating free form content based on what they know “have seen"
• 12.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Why is Gen AI useful? [ the builder view ] Source of unstructured knowledge How can I use this knowledge and reason about it to create a new asset? An asset being a piece of code, a whole program, a blog, an architecture, a troubleshooting workflow, a db query and more outside of the IT realm (a poem, a picture, a receipt …)
• 13.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Why is Gen AI useful? [ the builder view ] Read and memorize it all (LOL – yeah sure) (1) Source of unstructured knowledge You
• 14.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Why is Gen AI useful? [ the builder view ] You Read and memorize it all (LOL – yeah sure) Search engines (possibly not relevant and still hard - you are the integrator and generator of a new asset – text or code) (1) (2) Source of unstructured knowledge asset
• 15.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Why is Gen AI useful? [ the builder view ] 15 You Read and memorize it all (LOL – yeah sure) LLM Train on it (doable) (1) (2) (3a) Natural language conversation (3b) Source of unstructured knowledge asset asset Search engines (possibly not relevant and still hard - you are the integrator and generator of a new asset – text or code)
• 16.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. My first Gen AI application 16
• 17.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Real life use case 17 B A C K G R O U N D : I H A T E W H A T S A P P V O C A L M E S S A G E S !!!!
• 18.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Real life use case – the ClickOps version 18 B A C K G R O U N D : I H A T E W H A T S A P P V O C A L M E S S A G E S Audio file Text file LLM Audio to text translation Text summarization Text file
• 19.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Real life use case – the ClickOps version 19 B A C K G R O U N D : I H A T E W H A T S A P P V O C A L M E S S A G E S Prompt Output (generated asset)
• 20.
  © 2023, AmazonWeb Services, Inc. or its affiliates. Real life use case – the application version 20 M Y F I R S T ( N O N T U T O R I A L - B A S E D H E L L O - W O R L D ) G E N E R A T I V E A I A P P L I C A T I O N
• 21.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 21 Real life use case – the application version L A M B D A C A L L S A N E X T E R N A L L L M S E R V I C E https://it20.info/2023/08/building-a-generative-ai-application-using-aws-step-functions/
• 22.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Making LLMs useful 22
• 23.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 23 Why are people talking about things like Agents, Tools, RAG.. q The LLM is just one (fundamental) component of Generative AI q The LLM could hallucinate, don’t have knowledge of recent / private / live information, can’t do advanced math, may have limited reasoning capabilities, etc. q You need something to complement its capabilities and guide/help it q Especially for “real” business use cases that go beyond “toying around”
• 24.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 24 Why are people talking about things like Agents, Tools, RAG.. T H E R E ’ R E T W O D I M E N S I O N S T H E L L M O P E R A T E S I N ( L E V E L O F A B S T R A C T I O N A N D D O M A I N S ) Developing code Debugging code Living life Deploying code Domains Writing a novel Organizing travels
• 25.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 25 Why are people talking about things like Agents, Tools, RAG.. T H E R E ’ R E T W O D I M E N S I O N S T H E L L M O P E R A T E S I N ( L E V E L O F A B S T R A C T I O N A N D D O M A I N S ) Developing code Debugging code Deploying code Domains Autocomplete a function method Build a new ERP from scratch Resolve an error message Rearchitect the app to avoid this error at scale Suggest what I could do today Organize my whole life for the next 10 years Level of abstraction Simple task Complex task Writing a novel Living life Organizing travels Tell me how long it takes driving from Florence to Rome Plan in details all my 1-year long sabbatical
• 26.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 26 Why are people talking about things like Agents, Tools, RAG.. T H E R E ’ R E T W O D I M E N S I O N S T H E L L M O P E R A T E S I N ( L E V E L O F A B S T R A C T I O N A N D D O M A I N S ) Domains Level of abstraction Simple task Complex task Progressive complexity Completion Chat Reasoning Acting à à à coverage Domain A function of the corpus data
• 27.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 27 Why are people talking about things like Agents, Tools, RAG.. T H E R E ’ R E T W O D I M E N S I O N S T H E L L M O P E R A T E S I N ( L E V E L O F A B S T R A C T I O N A N D D O M A I N S ) Domains Level of abstraction Simple task Complex task Large Language model Smaller purpose built/tuned model Models may need to be helped / guided to achieve goals where task complexity is too high or simply for missing domain knowledge
• 28.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 28 Why are people talking about things like Agents, Tools, RAG.. A N E X A M P L E O F C O T ( C H A I N O F T H O U G H T S ) https://arxiv.org/abs/2201.11903 But sometimes in-prompt Chain of Thoughts (CoT) isn’t enough for the LLM to reason properly W elcom e to the m agic world of “prom pt engineering”
• 29.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 29 Why are people talking about things like Agents, Tools, RAG.. A N E X A M P L E O F T H E F A C T C H E C K I N G W I T H P R O M P T C H A I N I N G P R O C E S S https://it20.info/2023/6/the-dark-zone-between-the-magic-genai-experience-and-the-large-language-model/ Q: What is the biggest clock in the world?
• 30.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 30 Why are people talking about things like Agents, Tools, RAG.. LLM A N E X A M P L E O F T O O L S You “what’s the weather like today in Rome?” Math function code Web search code “Calculate <very complex formula>” (1a) (2a) (1b) (2b)
• 31.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 31 Why are people talking about things like Agents, Tools, RAG.. A N E X A M P L E O F R E A C T ( R E A S O N I N G A N D A C T I N G ) https://arxiv.org/abs/2210.03629
• 32.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 32 Why are people talking about things like Agents, Tools, RAG.. LLM A N E X A M P L E O F R E A C T ( R E A S O N I N G A N D A C T I N G ) You Iterating reasoning code “Write the solution for <very complex task>” (1) (2) https://arxiv.org/abs/2210.03629 Tool
• 33.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 33 Why are people talking about things like Agents, Tools, RAG.. LLM Vector DB A N E X A M P L E O F R A G ( R E T R I E V A L - A U G M E N T E D G E N E R A T I O N ) You Private corpus of data embedding “Write a draft email on <specific company secret topic>” (1) (2)
• 34.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Prompt context Vs. RAG Vs. fine-tuning 34
• 35.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 35 Prompt context Vs. RAG Vs. fine-tuning: I am lost q Fair. There are three ways to increase an LLM answer precision and correctness 1. Provide context in the prompt 2. Augment the LLM with an external source of vectorized data at inference time (RAG) 3. Fine tune the LLM with additional data q There isn’t a global right or wrong approach. As often happens, it depends q Also they are not mutually exclusive q They could (and often should) be used together to achieve optimal results
• 36.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 36 q Rate of the change of the data source q Limits, cost, latency, speed of prompt context tokens q Cost of fine tuning q including the work required to “prepare the data” q Cost of creating and maintaining the vector store Prompt context Vs. RAG Vs. fine-tuning: when to use what?
• 37.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 37 q Complexity of the architecture q fine tuning may make the architecture easier (with an upfront fine-tuning investment) q Shape and location of the data source q Precision of the outcome q no absolute rules exist, testing may be required q Personal experience of the team building the solution q “I have always used RAG and that’s what I am comfortable with” Prompt context Vs. RAG Vs. fine-tuning: when to use what?
• 38.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Who’s Gen AI for? 38
• 39.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 39 Who’s Gen AI for? q For the developer that is writing code q e.g. code assistants e.g. AWS CodeWhisperer q For the developer that wants to use English as a programming language q e.g. the example of the WhatsApp vocal messages q For the ops person that does not want to write a SQL query to extract data q e.g. https://www.honeycomb.io/blog/introducing-query-assistant
• 40.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 40 Who’s Gen AI for? q For the business analyst that wants to create a report off of a spreadsheet q For the journalist that wants to draft an article on a specific topic q Etc. etc. q Come see me later to chat about the story of my plumber impressed by “chat … chat …. chat something” (true story)
• 41.
  © 2023, AmazonWeb Services, Inc. or its affiliates. 41 Conclusions q Get ready for this new wave. It’s coming and (I think) it’s staying. q LLMs have moved the needle of the art of possible q But LLMs alone are not enough. You need to … make LLMs useful. q Gen AI is for everyone, not just for “builders”. It’s for “consumers” of tech too. q Go explore! Go build!
• 42.
  © 2023, AmazonWeb Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Massimo Re Ferrè Senior Principal Technologist, AWS Twitter: @mreferre E-mail: [email protected] Thanks!