TimescaleDB 2.24 Enhances Direct Compress for Faster Ingest

Fast APIs aren’t magic, they’re the result of smart design choices made early ⚡🚀 ➡️Top 5 common ways to improve API performance. 👉Result Pagination: This method is used to optimize large result sets by streaming them back to the client, enhancing service responsiveness and user experience. 👉Asynchronous Logging: This approach involves sending logs to a lock-free buffer and returning immediately, rather than dealing with the disk on every call. Logs are periodically flushed to the disk, significantly reducing I/O overhead. 👉Data Caching: Frequently accessed data can be stored in a cache to speed up retrieval. Clients check the cache before querying the database, with data storage solutions like Redis offering faster access due to in-memory storage. 👉Payload Compression: To reduce data transmission time, requests and responses can be compressed (e.g., using gzip), making the upload and download processes quicker. 👉Connection Pooling: This technique involves using a pool of open connections to manage database interaction, which reduces the overhead associated with opening and closing connections each time data needs to be loaded. The pool manages the lifecycle of connections for efficient resource use. Over to you: What other ways do you use to improve API performance? #api #systemdesign #backend #performance #softwareengineering

2025: 📦 166 unique releases, 1,162 total changes, avg. new release every 2.4 days 📊 10x cost reduction for customers, hours old insights into sub-second latency ⚡ 70-node Spark clusters replaced with single digit Feldera instances 2026: Make incremental compute inevitable Full story: https://lnkd.in/gsEwqMBs

Heavy downside at CloudWatch: Managing logs at scale usually sucks. With Unified Data Management, this may be solved! Previously: writing custom functions just to dump everything into S3 so you can query it with other services like Athena (or paying a 💩 load of money for ingestion of data you don't even need) 💸 Features that stand out: 1. 𝗦𝟯 𝗧𝗮𝗯𝗹𝗲𝘀 𝘀𝘂𝗽𝗽𝗼𝗿𝘁: You can query logs directly with Athena/Redshift via Iceberg. No more custom scripts!!! 🥹 2. 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀: You can filter and transform logs before they go into the storage. Finally! 💪 3. 𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗲𝘄: See logs across ALL accounts! Seems like CloudWatch grew up into a proper data backend 💾 Seems like a "boring" infra update, but it's not! P.S. If you want to dive deeper into CloudWatch, check out our book's sample chapter at cloudwatchbook.com or grab some free tips at https://lnkd.in/e3QT5s7Q 🕵️♂️

✨ TimescaleDB 2.24 feature teardown: Continuous Aggregates now speak UUIDv7 UUIDv7 is a new time-based identifier format (RFC 9562): globally unique, but with an embedded timestamp that preserves ordering. That detail matters architecturally: the ID itself can serve as a stable partitioning key, so append-mostly workloads naturally write into the most recent partition. Unlike randomized UUIDs, UUIDv7 preserves locality, which makes it especially well suited for TimescaleDB hypertables. We’re already seeing this pattern emerge in event-heavy schemas. In 2.24, we added UUIDv7 support to Continuous Aggregates, TimescaleDB’s incremental materialized views. Hypertables partitioned by UUIDv7 now work seamlessly through an enhanced time_bucket that accepts UUIDv7 values and returns precise, timezone-aware timestamps. Continuous aggregates can therefore compute rollups directly over UUIDv7-partitioned tables, enabling efficient time-series analytics even when the primary key is UUIDv7. All towards building the best Postgres, now with more use cases. 🐯🚀 Tiger Data (creators of TimescaleDB)

The latest update for #Last9 includes "How to Handle #CloudMonitoring Overload?" and "How to Track Down the Real Cause of Sudden Latency Spikes". #Monitoring #Observability #TimeSeries #SRE #DevOps https://lnkd.in/eV2ztwiV

S3-native is the new default… but raw object storage alone will slow your “real-time” system to a crawl. Every read is a network hop + a paid request. 100–300 ms per S3 access stacked inside streaming queries = latency spikes and a painful bill. So how do you keep S3 as your durable, cheap base layer and still hit real-time SLAs? Meet Foyer – a hybrid cache for S3-native systems! Built in Rust, Foyer unifies: ✅ Memory for the hottest data ✅ Local disk for warm data ✅ S3/object storage as the durable source of truth All behind a single cache API. In practice, that means: ➡️ Most reads are served from RAM or local SSD, not S3 ➡️ Far fewer S3 requests → lower object-store fees ➡️ Predictable latency for continuous queries and streaming joins In RisingWave, Foyer sits at the heart of a three-tier storage stack: 1️⃣ Memory → sub-ms access for hot state and MVs 2️⃣ Local disk (Foyer) → extended cache for working sets 3️⃣ S3 → elastic, durable storage for all state and recovery RisingWave can lean fully on S3 for durability and scale, while Foyer keeps real-time analytics responsive, even when queries continuously scan logs, materialized views, and Iceberg-backed tables. Foyer is not tied to RisingWave alone. Any S3-native or object-store-based system (think OLTP-as-a-service, streaming engines, log stores) can plug it in to turn “cheap but slow” storage into a low-latency engine. Already in production at ➡️ RisingWave – real-time SQL stream processing ➡️ Chroma – vector DB for LLM apps ➡️ And now, recently, SlateDB – an embedded object-storage engine Want to see what hybrid caching can do for S3-native stream processing and Iceberg workloads? Join the community Slack: https://lnkd.in/dCNX8VQf

Shipped v1.4 of the nowcasting project. A few major upgrades since v1.0.0... 1. Market data integration (yfinance) – daily financial indicators paired with slow-moving macro data. 2. Experiment tracking with #MLflow – run identity, metrics, and run-scoped artifacts for reproducibility. 3. S3-backed storage (#Terraform-managed) – storage for datasets, models, metrics, and predictions; pipeline is now fully stateless and cloud-ready. This completes the V1 storage + tracking layer. Up next: FastAPI inference service deployed on ECS/Fargate. Repo: https://lnkd.in/eKDvGXVH Stay tuned 👀 #MachineLearningEngineering #MLOps #DataEngineering #AWS

The latest update for #Last9 includes "How to Track Down the Real Cause of Sudden Latency Spikes" and "Which #Observability Tool Helps with Visibility Without Overspend". #Monitoring #TimeSeries #SRE #DevOps https://lnkd.in/eV2ztwiV

Reposting this because the closing lesson is pure gold, and applies far beyond engineering: Sometimes the most elegant solutions come not from adding complexity, new tools, or bigger infrastructure, but from truly understanding the underlying system and why it behaves the way it does. WHOOP’s team didn’t fix a scaling problem with more architecture, more spend, or more layers. They solved it by deeply understanding S3’s partitioning behavior, and making one small, thoughtful change that delivered an exponential jump in reliability. There’s a powerful business parallel here: In Customer Success, GTM, operations, or product strategy, we often default to “bigger, faster, more.” More processes, more tech, more headcount, more dashboards. But the real breakthroughs usually come from slowing down, studying the root cause, and designing something simple, elegant, and durable. Solve the why before you scale the how. Kudos to WHOOP for a brilliant demonstration of engineering craftsmanship — and a reminder for leaders everywhere that clarity beats complexity, and understanding beats over-engineering.

🚀 Scaling the Transactional Outbox to 2+ BILLION messages/day (without breaking reliability) Most teams implement the outbox pattern… and then hit a wall when throughput grows. I went through a great technical walkthrough that shows how to scale an outbox pipeline from “works in prod” → ~27,000 msgs/sec by focusing on a few brutally practical optimizations: ✅ 1) Increase batch size Stop sending 1-by-1. Bigger batches = fewer DB + broker round trips. ✅ 2) Speed up DB reads (covered index) A proper index turns the “find pending messages” query into a fast, predictable operation. ✅ 3) Bulk DB writes in ONE round trip Instead of updating rows individually, group updates into a single command. ✅ 4) Parallel processors (without collisions) Use FOR UPDATE SKIP LOCKED so multiple workers can safely pick unique batches without stepping on each other. ✅ 5) Broker-side batch publishing (MassTransit) Let the broker do what it’s good at: batching + publishing efficiently. 🔍 Big takeaway: Outbox scaling is rarely “add more servers.” It’s database shape + lock strategy + batch economics + publish mechanics. Curious: what’s been your biggest bottleneck with the outbox pattern? DB locks? Slow reads? Publish latency? Retries? #dotnet #csharp #sql #microservices #distributedSystems #messaging #MassTransit #OutboxPattern #PerformanceEngineering #SystemDesign