How GenAI Copilot Simplified My Data Modeling Tasks

What would it mean for your business if you freed half of your AI budget next year? Developing AI-based solutions is often frustrating for the data scientist. Up to 70% of almost every project is just scraping together data, cleaning and tying it all together. To make things even worse, this is done per-project basis, with very little reusable assets being formed while doing it. I’ve been watching this inefficiency and outright waste of money for years while a better alternative exists right under our noses; with proper pipes and thinking of data as a product, you can start to create a portfolio of reusable, high-quality, and governed data assets that deliver data-intensive applications up to 90% faster, with lower costs to boot. And no, a data lake does not solve the problem. It can be a part of the solution, but by itself it doesn’t bring any tangible business outcomes. By creating a catalog of data products with a clear contract on what it serves, who owns it, and how to access it, you create use for it, and accidentally solve the data governance & ownership problems too. I’ll happily discuss this more if you’re interested. Meshly Oy provides open-source based data control solutions to serve up-to-date, curated data for your AI & BI needs, regardless of where your data lives.

My Weekly Finds 📊 Data & Visualization Tools 1. Observable - A collaborative data canvas that lets cross-functional teams explore, visualize, and refine analysis together in real-time using AI-powered tools, SQL queries, or point-and-click interactions—eliminating the need to hop between separate tools during the data discovery process. https://observablehq.com/ 🚀 Infrastructure & Performance 1. Vectroid - Serverless vector database that eliminates the traditional tradeoffs between speed, accuracy, and cost by dynamically managing HNSW memory footprints and using lazy loading with vector compression to deliver high-performance vector search without infrastructure overhead. https://lnkd.in/gTQPBaKM 🤖 AI-Powered Development 1. Claude Code Subagents for Parallel Development - Transform sequential development workflows into concurrent, assembly-line processes where different agents simultaneously handle distinct aspects of complex tasks, dramatically reducing development time while maintaining quality through dedicated context isolation. https://lnkd.in/gpNzm5ZT 2. Claude Code Router - Configurable proxy layer that enables dynamic routing between multiple AI models (Claude, GPT, etc.) through custom rules and transformers, allowing developers to optimize model selection for specific tasks while maintaining Claude Code's interface and tooling ecosystem. https://lnkd.in/gDjAjv3i 🎥 Meeting & Collaboration Tools 1. Vexa - Open-source, self-hosted API that programmatically deploys transcription bots into Google Meet calls, enabling developers to build privacy-first meeting assistants with real-time multilingual transcription under their own infrastructure control. https://lnkd.in/gvmb84Hs 📈 Industry Analysis 1. Amazon S3 Vectors Impact Analysis - Explores how Amazon S3 Vectors won't kill vector databases but will instead mature the ecosystem into a tiered storage model where different solutions serve different performance and cost needs—with S3 providing ultra-low-cost storage for cold data while specialized vector databases handle high-performance, real-time applications. https://lnkd.in/gq9wSsRw

Hot take: You don’t need to replace your BI stack. You need to elevate it. AI isn’t here to wipe out dashboards or analytics teams. It’s here to stack on top of what you’ve already built. Here’s the modern hierarchy: 🔽 Data Infrastructure Your warehouses, pipelines, and integrations. 🔽 Data Analytics Your dashboards, reports, and human insights. 🔼 AI (Advanced ML + LLMs) The final layer: helping you predict, automate, and personalize at scale. If your BI stops at reporting, you're missing out. Modern BI isn't just "what happened?" — it's "what’s next?" 📍 Book a free strategy session: https://lnkd.in/gPsVvhAx Let’s future-proof your BI stack, without starting from scratch. ♻ Repost this if you’ve ever been told AI makes BI obsolete. (It doesn’t.) P.S. What layer are you strongest in right now: Infra, Analytics, or AI? 👇

🚀 Top 10 AI in Data Analytics This Week! AI is transforming how we analyze, visualize & act on data. Here’s the freshest roundup of tools & trends you shouldn’t miss ⬇️ 1️⃣ Databricks 🤝 OpenAI Enterprise teams can now build AI agents directly on their data with Agent Bricks. Supercharging analytics with GPT-powered insights! 2️⃣ Google Cloud’s New AI Agents 🌐 Six powerful AI agents launched for data engineering, feature building & conversational analytics — making pipelines smarter & faster. 3️⃣ Snowflake Cortex AI ❄️ Natural language to SQL at enterprise scale. Security-first AI analytics for organizations that care about governance. 4️⃣ Microsoft Fabric + Copilot 💡 Copilot is now built into Fabric — write queries, automate BI, and document workflows with simple prompts. 5️⃣ Google BigQuery + Duet AI 📊 AI-assisted queries & dashboards in BigQuery Studio. Save time, cut costs, and optimize performance. 6️⃣ Tableau Pulse with Einstein Copilot ⚡ From dashboards to conversational insights — Tableau makes analytics feel more human & interactive. 7️⃣ ThoughtSpot Sage 🔍 Search-driven analytics with GPT-4. Ask a question in plain English, get instant insights from your enterprise data. 8️⃣ Powerdrill Bloom 🌸 Upload data → get guided insights, visualizations, and anomalies detected automatically. AI-first analytics at your fingertips. 9️⃣ AutoML & Augmented Analytics 🤖 Feature engineering, model selection & tuning now automated. Democratizing ML for analysts of every skill level. 🔟 Conversational Analytics 💬 Plain English to SQL, DAX & dashboards. Analytics is becoming as easy as asking a question. ✨ The future of data analytics isn’t code-heavy — it’s conversation-driven.

An uncomfortable truth from Airbyte CEO Michel Tricot 👇 You can implement the most sophisticated LLM, fine-tune it perfectly, build beautiful interfaces - none of it matters if your data is trapped in silos. Before your business users can ask "What's driving churn in enterprise accounts?" your AI needs unified access to CRM data, product usage logs, support ticket history, billing systems, customer success platforms, etc. Not just snapshots. Real-time, comprehensive data from every source. This is why we're seeing enterprises hit a wall with AI initiatives. They're trying to build intelligence on top of fragmented data. It's like asking someone to solve a puzzle when half the pieces are locked in different rooms. The companies succeeding with self-service analytics? They solved data movement first. Read Michel's full analysis on building AI-ready data architectures in The New Stack: https://lnkd.in/gyaibDHA

Old systems focused on preparing data for people. New systems prepare data for AI agents — and that requires speed, feedback, and constant learning. via Tomasz Tunguz - AI breaks the data stack. Most enterprises spent the past decade building sophisticated data stacks. ETL pipelines move data into warehouses. Transformation layers clean data for analytics. BI tools surface insights to users. This architecture worked for traditional analytics. But AI demands something different. It needs continuous feedback loops. It requires real-time embeddings & context retrieval. Consider a customer at an ATM withdrawing pocket money. The AI agent on their mobile app needs to know about that $40 transaction within seconds. Data accuracy & speed aren’t optional. Netflix rebuilt their entire recommendation infrastructure to support real-time model updates1. Stripe created unified pipelines where payment data flows into fraud models within milliseconds2. The modern AI stack requires a fundamentally different architecture. Data flows from diverse systems into vector databases, where embeddings & high-dimensional data live alongside traditional structured data. Context databases store the institutional knowledge that informs AI decisions. AI systems consume this data, then enter experimentation loops. GEPA & DSPy enable evolutionary optimization across multiple quality dimensions. Evaluations measure performance. Reinforcement learning trains agents to navigate complex enterprise environments. Underpinning everything is an observability layer. The entire system needs accurate data & fast. That’s why data observability will also fuse with AI observability to provide data engineers & AI engineers end-to-end understanding of the health of their pipelines. Data & AI infrastructure aren’t converging. They’ve already fused.

I test. You learn. This week: Semantic metadata for metric views. You can now add semantic metadata to metric views in Databricks. 🌈 The promise: - Make #Genie smarter - Make AI/BI dashboards cleaner and more consistent 👉 Three types of metadata you can add to metric views: - Synonyms: Should help Genie understand what you mean. Users say "Customer tier," it will more easily capture that you're referring to "Customer segment." - Display names: Replace technical column names automatically with labels humans actually understand in visualization tools. No more explaining what "cust_acq_cost_mtd" means. - Format specifications: Control how values should be displayed, ensuring consistency. ⚠️What I found in testing: The semantic metadata doesn't seem to be implemented on the consumption side yet (or it's not rolled out to my workspace): - I would for example expect when used in AI/BI dashboards, it would use the display name as label of the axis, or automatically use the format. - Genie didn't pick up my synonym tests, however I used very unrelated aliases, so maybe I made it too hard? ❗ If you want to try this out, don't forget to switch your SQL warehouse channel to preview.

🤖 From RAG to Agentic Systems: How We’re Powering Next-Gen Analytics at DataZymes In the evolving world of Generative AI, one of the biggest challenges is not just retrieving knowledge but orchestrating how different AI agents collaborate to solve complex queries. At DataZymes, I’ve been working on Retrieval-Augmented Generation (RAG) and experimenting with different Agentic frameworks — each designed with unique strengths and trade-offs. Here’s a quick overview of the frameworks I’ve explored and their applications: 🔹 LangChain – One of the earliest frameworks for building LLM-powered applications. Use case: Great for rapid prototyping, chaining tools, and experimenting with different retrieval strategies. 🔹 LlamaIndex – Focused on structured + unstructured data ingestion with powerful indexing options. Use case: Best when working with large, complex knowledge bases that need efficient retrieval. 🔹 LangGraph – Purpose-built for agentic workflows, allowing for stateful, multi-step reasoning. Use case: Ideal when multiple agents/tools must collaborate (e.g., SQL + RAG endpoints) to solve a single query. 🔹 DSPy – A declarative framework that auto-optimizes prompts, retrievers, and reasoning steps. Use case: Eliminates manual prompt engineering by using optimizers to iteratively improve accuracy on downstream tasks. 🔹 Haystack – An open-source framework for building production-ready RAG pipelines. Use case: Strong for search + retrieval across enterprise knowledge bases with flexible LLM + retriever integrations. 🔹 CrewAI / AutoGen – Multi-agent collaboration frameworks with predefined agent roles. Use case: Best for scenarios where multiple specialized agents (e.g., Data Analyst, Retriever, Orchestrator) must work together. ✨ Each framework approaches the agentic challenge differently — from indexing and prompt optimization to multi-agent orchestration. ⚖️ Takeaway: In domains like pharma and healthcare, the right agentic workflow must not only deliver accurate retrieval but also ensure compliance, explainability, and scalability.

The task was to create a ✨ 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗲𝗱, 𝗿𝗲𝘂𝘀𝗮𝗯𝗹𝗲 𝘁𝗼𝗼𝗹 ✨ that automates one of the most tedious parts of data science: creating a mapping schema for raw categorical data. The new command takes any dataset, finds all unique values in a field, and uses an LLM to generate a clean, structured category map—saving me countless hours of manual work. 🤖 This wasn't just a simple script. We gave the AI a high-level "vibe" and some architectural context, and it executed a full engineering sequence: 1️⃣ Scaffolding a new database migration. 2️⃣ Updating the Eloquent model. 3️⃣ Creating the new, complex Artisan command from scratch. 4️⃣ Modifying the database seeder to use the new logic. ...all while flawlessly (almost) adhering to our project's specific patterns. ✔️ But a cool tool is only half the story. 🎯 The real value is what it unlocks. To test it, we immediately pointed this new tool at a dataset I care about personally: my hometown of 𝗘𝘃𝗲𝗿𝗲𝘁𝘁'𝘀 police dispatch logs. 📍 The insights it helped uncover for 2025 were eye-opening: 📉 𝗗𝗲𝘁𝗮𝗶𝗹 / 𝗣𝗮𝘁𝗿𝗼𝗹 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆: Down a staggering 𝟰𝟱.𝟲𝟴% 📈 𝗪𝗮𝗿𝗿𝗮𝗻𝘁 𝗦𝗲𝗿𝘃𝗶𝗰𝗲: Up an incredible 𝟭𝟱𝟳.𝟯𝟱% This is the real power of this workflow. We went from a complex feature idea to a statistically significant insight about my own community in a fraction of the time. ⚡ The attached paper is our deep dive into formalizing this process. 🔬 It breaks down the workflow, the math behind the AI's "attention mechanism," and showcases the raw data output. It's a testament to how this new paradigm isn't just about developer productivity—it's about accelerating the path to discovery. Check out the full paper attached below with an Appendix detailing trends in Everett crime over the last 5 years. 👇 🤔 Beyond just writing code faster, what’s the most valuable, non-obvious outcome you’ve achieved using AI in your development workflow? #AI #SoftwareDevelopment #LLM #DeveloperTools #DataAnalytics #CaseStudy #Laravel #GenAI #FutureOfWork #AIinDev #FormalMethods

🚀 Are AI Agents Moving from Experiments to Production — Faster Than We Think? I attended the AI Show & Tell at Microsoft’s R&D lab in NYC last night, hosted by Cedric Vidal and Cassidy Fein. Three talks showed what production-ready AI agents actually look like: Nimbleway AI – Solving the Live Data Problem 🍕 Roee M. posed a deceptively simple question: “Can AI really tell you the best pizza in NYC?” LLMs give confident answers that can be outdated, biased, or just plain wrong because they lack access to current information. Roee compared various approaches: Traditional APIs → accurate but rigid Manual browsing → reliable but doesn’t scale Nimble’s browser agents → grounding multi-agent systems in live, structured web data The same infrastructure that helps you find the best pizza is already powering enterprise pricing intelligence and market analysis at scale. 📊 TextQL – 100,000 Tables, Zero Configuration Ethan Ding showed how TextQL’s analytics agents query petabytes of data with natural language and zero setup. What used to take days of waiting for SQL queries now happens in seconds of conversation. The data team bottleneck → automated away. 🧠 Arc Intelligence Agents That Compound Intelligence Jarrod Barnes presented perhaps the most intriguing idea: agents that actually learn from experience. Arc is an open-source continual-learning framework that uses online prompt optimization and reward modeling in production. Their demo: an SRE Agent resolving Kubernetes incidents — and getting smarter with each attempt. Most agents perform identically on task 1 and task 100. Arc’s don’t. They self improve. 🔮 What’s Next? If today’s agents can fetch live data, automate analytics, and learn on the fly… 👉 how far will they go in reshaping our daily work and decisions? Will they stay as assistants — or become teammates that replace entire workflows? #AIAgents #ContinualLearning