Michail Christidis

Data engineers. You haven't partitioned data like this before. Spatial data doesn’t just grow, it sprawls. And if it’s not spatially partitioned, every query becomes a full table scan disguised as “analytics.” Spatially partitioned GeoParquet fixes that. By arranging data by geography, you can run spatial queries that only touch what matters saving compute, cutting costs, and making geospatial actually fast. Here’s a practical example on how to inspect and visualize spatial partitions using Overture Maps data by Kyle Barron: https://lnkd.in/ez3kXr7b If your pipelines touch location data, this isn’t a GIS thing, it’s a data engineering performance problem. 🌎 I'm Matt and I talk about modern GIS, earth observation, AI, and how geospatial is changing. 📬 Want more like this? Join 8k+ others learning from my newsletter → forrest.nyc

195

9 Comments

🚀Great news for anyone interested in deep learning! 🎉 📘 🐍 The 3𝐫𝐝 𝐞𝐝𝐢𝐭𝐢𝐨𝐧 𝐨𝐟 𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐏𝐲𝐭𝐡𝐨𝐧 by François Chollet and Matthew Watson is coming soon — and for the first time, a 𝐅𝐑𝐄𝐄 online version will be available. This new edition brings updated coverage of Keras, TensorFlow, generative AI models, PyTorch, and JAX, with hands-on Python examples. The book is available to pre-order: 📘 Publisher (September 22, 2025): https://lnkd.in/dHaA4jBy 📘 Amazon (September 30, 2025): https://lnkd.in/d9Jb2uzm A 𝐅𝐑𝐄𝐄 online version of the book is available here: https://lnkd.in/dq5bQ3-h Thank you to the authors for providing a 𝐅𝐑𝐄𝐄 online version of the book! 🙏 #books #newbooks #freebooks #python #deeplearning #ai #datascience

127

3 Comments

Overfitting is a critical issue in machine learning and data modeling. It occurs when a model learns the training data too well, capturing noise and minor fluctuations rather than the underlying pattern. This results in a model that performs exceptionally on training data but poorly on new, unseen data. Overfitting can lead to: ✅ Poor generalization, meaning the model fails to perform well on new data. ✅ High variance, where the model's predictions vary greatly for different data sets. ✅ Increased error rates on unseen data, making the model unreliable for real-world applications. ✅ Misleading insights, as the model's decisions are based on noise rather than true patterns. ✅ Inefficient models, which are overly complex and harder to interpret. A visualization from a Wikipedia article illustrates this issue (link: https://lnkd.in/eP5gS4U6). The green line represents an overfitted model and the black line represents a regularized model. While the green line best follows the training data, it is too dependent on that data and is likely to have a higher error rate on new unseen data, as illustrated by black-outlined dots, compared to the black line. For regular tips on data science, statistics, Python, and R programming, check out my free email newsletter. More info: http://eepurl.com/gH6myT #datascienceeducation #businessanalyst #database #datasciencecourse

138

12 Comments

🛑 The Model-Centric Trap: Why Your Best Architecture Still Fails at Forecasting We’ve already questioned biased loss functions. We’ve challenged the myth of stationarity. Now it’s time to confront the biggest systemic failure in time series forecasting: 👉 The Misplaced Focus on the Model In today’s AI world, the default response to poor forecast accuracy is predictable: “Let’s try a more complex model.” ARIMA → Prophet → LSTM → Transformer → Foundation Model... This mindset is a trap. Especially in forecasting. 💥 Why Time Series Is Different Unlike vision or NLP, time series data is not i.i.d. A single data quality issue at time t can corrupt the model’s understanding of the entire future trajectory: t+1, t+2, ..., t+h. This makes time series uniquely vulnerable to the “Garbage In, Garbage Out” problem. 🔥 The Claim We need to shift from Model-Centric to Data-Centric Forecasting. Spending 80% of your time fine-tuning a 100M-parameter Transformer and only 20% on data cleansing is fundamentally backward. ✅ A simple Exponential Smoothing model on clean, context-rich data will often outperform a state-of-the-art DL model trained on noisy, poorly engineered inputs. 💡 Prototypes for a Data-Centric Philosophy Systematic Data Labeling (or Un-Labeling) Identify and flag “Forecast-Killing Events” (e.g., IT outages, one-off promos, supply shocks). This isn’t outlier detection—it’s contextual noise detection. Contextual Feature Engineering is King Lagged Y_t is not enough. The real power lies in exogenous variables—but only if they’re clean and perfectly aligned. Known-Future Variables Matter If your covariate at time t+h is forecasted, its error compounds the target error. Holidays? Yes. Predicted demand? Handle with care. Data Versioning & Governance If data quality matters more than model architecture, we need Git-like version control for datasets. Know exactly which data snapshot produced which forecast. Let’s stop tweaking the 15th layer of our deep learning model. Let’s start building a culture of Data Quality Engineering in time series. 🔍 Your Turn What’s the single most persistent data quality issue that has ruined your best time series model? 👇 Drop it in the comments. Stephan Kolassa Srinivas ChillaraIvan Svetunkov Nikos Kafritsas Tim Januschowski #Forecasting #DataCentricAI #TimeSeries #MLOps #DataQuality #ModelCentricTrap #FoundationModels

45

12 Comments

Ever wondered which exact columns flow from your raw sources all the way to downstream models in dbt? I just published a new video where I demo column-level lineage in dbt using dbt-colibri, created by Bas Nederkoorn, that takes lineage beyond models into columns in dbt core. This is a game-changer for understanding how data flows in your dbt project on a column level. 🎥 Watch here: https://lnkd.in/d7eJgasC Big shoutout to Bas Nederkoorn for creating this amazing project for the dbt community! 🙌 #dbt #analyticsengineering

67

5 Comments

Here is why Data science workflows are the perfect use case for AI agents. And at the same time… they’re the worst. How? Because no workflow is more iterative. You explore → check results → adjust → try again. (That’s exactly what agents are designed to do) The problem is: our work isn’t just iteration. It’s context, judgment, and knowing when the data doesn’t make sense. I've built my own Agents and gave them messy, open-ended question: 1. The first pass was completely wrong 2. Instead of stopping, it retraced its steps, refined the query, and improved each loop 3. By the third iteration, it was surprisingly close to useful After a while, you realize: Agents don’t replace data scientists. They just show us which parts of the loop can be automated, and which parts still depend on human judgment. 👉 Would you let an agent handle the trial-and-error part of your analysis? Or is that still too risky?

56

30 Comments

𝗔𝗜 𝗙𝗼𝗿 𝗘𝘃𝗲𝗿𝘆𝗼𝗻𝗲 Great post by: Dirk Zee Original post below ⬇️⬇️⬇️ 𝗔𝗜 𝗙𝗼𝗿 𝗘𝘃𝗲𝗿𝘆𝗼𝗻𝗲 Basic Concepts Explained: • 𝗗𝗮𝘁𝗮 𝗠𝗶𝗻𝗶𝗻𝗴: Extracting insights from large datasets efficiently. • 𝗖𝗵𝗮𝘁𝗯𝗼𝘁𝘀: Automated conversational agents for diverse applications. • 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Complex pattern recognition through layered neural networks. • 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: Simulating human intelligence in machines. • 𝗦𝗽𝗲𝗲𝗰𝗵 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻: Transcribing spoken language into text accurately. • 𝗔𝗻𝗼𝗺𝗮𝗹𝘆 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻: Identifying unusual patterns in data automatically. • 𝗥𝗲𝗶𝗻𝗳𝗼𝗿𝗰𝗲𝗺𝗲𝗻𝘁 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Learning optimal actions through trial and error. • 𝗨𝗻𝘀𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗲𝗱 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Discovering patterns without labeled data. • 𝗦𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗲𝗱 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Training models using labeled datasets for prediction. • 𝗡𝗮𝘁𝘂𝗿𝗮𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Understanding and generating human language. • 𝗖𝗼𝗺𝗽𝘂𝘁𝗲𝗿 𝗩𝗶𝘀𝗶𝗼𝗻: Extracting insights from visual data. • 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Algorithms learning from data to make predictions. ______________ Follow for more Useful Infographics

157

12 Comments