Jeppe Hallgren

Two LLM giants just moved the goalposts for open-weight-based results on SWE-bench: > DeepSeek-V3.1 ~66%  > Qwen-3 Coder ~70% While OpenAI / Anthropic closed-model top submissions (GPT-5 / Opus 4.1) are at ~75%. A few insights: 1> Closed source foundation labs submissions are gradually plateauing on SWE-bench, while I've heard their leaders stating that they will reach 90%+ by the end of 2025 2> Open-weight models are closing the gap 3> Tool usage, algorithms, and systems built with and around models are now pushing the upper bound of the submissions More👇 --- Reminder: SWE-bench = brutal benchmark. Real GitHub issues → model must write a patch → patch must pass the repo’s tests. Until ~2024, no model cleared 30%. When Devin burst into the world, they reported 13% on the full benchmark! (equivalent roughly to ~25% on the Verified subset) During 2024, OpenAI reduced the Full benchmark to a Verified subset of 500 samples: https://lnkd.in/dsFdEH76 But even then, they have been reporting over n=477 only (!) https://lnkd.in/d-yzQZ-V At Qodo, we noticed that it is possible that even in the Verified subset, many of the supposed "solutions" are not well defined by the "problem" (estimating ~20%) - that is, sometimes their tests check for a solution that is broader than the issue, sometimes narrower So the verified isn't that verified. What could be an alternative benchmark for heavy-duty software development? More on that later this month. --- Now on SWE-bench: Qwen-3 Coder: 69.6–70%  DeepSeek-V3.1: 66% Claude 4.1 Opus: ~72–74.5% GPT-5: ~72-74.9% Gemini 2.5 Pro: 63.8% So:  two open models are within single-digit points of the very best closed models. And Qwen-3 Coder actually beats Gemini ?!@# --- DeepSeek-V3.1’s recipe: • 671B param Mixture-of-Experts (~37B active) • Trained on ~15T tokens • 128K context • Hybrid reasoning (“thinking” vs “fast”) • Post-training for agent loops + tool use Qwen-3 Coder’s recipe: • 480B param MoE (~35B active) • 1M token context (!!) • Specifically tuned for agentic coding • Released open weights, ready to use Both models weren’t just trained to autocomplete code. They’re built to act like agents: write → run → debug → iterate. That’s why they are improving on SWE-bench I think. --- Closed-source progress, meanwhile, feels plateaued. GPT-3.5 → GPT-4 = leap. GPT-4 → GPT-5 = smaller, incremental. Claude: same story. Meanwhile, open models: Nearing closed-source performance. Outperforming GPT-4 on SWE-bench 1/X inference cost? Is the moat eroding? A researcher from Alibaba's Qwen mentioned that they keep training specifically for SWE-like tasks. -- Takeaway: Open models (DeepSeek-V3.1, Qwen-3 Coder) are becoming first-class contenders now. Not side projects. Not toys. GLM-4.5 is also a strong one

145

12 Kommentarer

Before ChatGPT was cool, we were already doing “AI-like” magic with code. Amplication was the OG code generator - back when LLMs weren’t even in the conversation. 😅 When I started Amplication a few years ago, my mission was clear: help developers move faster by generating code they could trust. Along the way, I learned some powerful lessons: Developers love speed - but never at the expense of control. Standardization beats shortcuts - teams want consistency, not just productivity hacks. Building for individuals is not enough - true impact comes when you empower the organization as a whole. What worked: we proved that code generation could save teams countless hours and create maintainable services. What didn’t: scaling this to the organizational level was still a huge challenge. Every team has its own tools, workflows, and boundaries — and that fragmentation slows everyone down. These lessons are exactly why I’ve started working on something new: Overcut It builds on what I learned at Amplication, but takes the next step — bringing AI-powered workflows to the entire SDLC, at the team and organization level. I can honestly say I’ve never been this excited about a product. Seeing the first workflows already running with early users is surreal.🤩 Overcut already live in a controlled launch, powering real workflows with early users. Excited to share the first use cases and impact soon. 🚀 meanwhile you can take a peek at http://overcut.ai.

94

26 Kommentarer

LLMs weren't built to think in APIs, and APIs weren't built for LLMs to use. This mismatch is why 90% of "AI agents" are still just chatbots. Here's the problem: When you give an LLM a datetime parameter, what's the cardinality? Infinite. When you ask it to call Salesforce's API directly, "get deals" doesn't exist—you need 4+ API calls just to get what a human thinks of as one action. The solution? Abstraction layers that reduce cardinality. Instead of infinite datetime options, give the LLM "tomorrow at 3pm PST." Instead of 4 Salesforce API calls, give it one "get_deals" function. At Arcade.dev, we've spent years building these abstractions. Our tools work because we give LLMs what they need: lower cardinality, clearer intent mapping, and error handling designed for probabilistic systems. Stop forcing square pegs into round holes. Build for how LLMs actually work.

61

8 Kommentarer

Google's new LLM can run on just 0.5 GB RAM. (let's fine-tune it 100% locally) Google released Gemma 3 270M, a new model for hyper-efficient local AI! We'll fine-tune this model and make it very smart at playing chess and predict the next move. Tech stack: - Unsloth AI for efficient fine-tuning. - Hugging Face transformers to run it locally. Here's how you can do it in 5 easy steps: 1️⃣ Load the model ↳ We start by loading the Gemma 3 270M and its tokenizer using Unsloth. 2️⃣ Define LoRA config ↳ We'll use LoRA for efficient fine-tuning. ↳ To do this, we use Unsloth's PEFT and specify: - The model - LoRA low-rank (r) - Layers for fine-tuning (target_modules) 3️⃣ Load dataset ↳ We'll fine-tune Gemma 3 to make it extremly smart at playing chess. ↳ Given a set of previous move (one move missing) & the final result it has to predict the missing move. ↳ In order to do this we're using the ChessInstruct dataset from HuggingFace. 4️⃣ Prepare dataset ↳ Next, we use a conversation style dataset to fine-tune Gemma 3. ↳ The standardize_data_formats method converts the dataset to the correct format for finetuning purposes! 5️⃣ Define Trainer and train the model ↳ We create a Trainer object by specifying the training config, like learning rate, model, tokenizer, and more, and initiate training. Done! This gives us the fine-tuned Gemma 3 that can find the exact missing chess move instead of randomly generating some moves. Find the complete fine-tuning guide by Unsloth in the comments. _____ Share this with your network if you found this insightful ♻️ Follow me (Akshay Pachaar) for more insights and tutorials on AI and Machine Learning!

1.489

67 Kommentarer

"With only 27 million parameters, HRM achieves exceptional performance on complex reasoning tasks using only 1000 training samples. The model operates without pre-training or CoT data, yet achieves nearly perfect performance on challenging tasks including complex Sudoku puzzles and optimal path finding in large mazes. Furthermore, HRM outperforms much larger models with significantly longer context windows on the Abstraction and Reasoning Corpus (ARC)" https://lnkd.in/gZU-AkKp https://lnkd.in/g9dqj_WA

173

19 Kommentarer

GPT-5 is the best coding model in the world and is now the default model in Cursor! Cursor: “It’s the most intelligent coding model our team has tested.” There is an amazing story about how this model became so much better at coding, featuring some innovative core research + behind-the-scenes efforts by the startups solutions team. In building GPT-5 the research team invested heavily in coding improvements and trained a model that showed large gains on the leading coding and SWE benchmarks. TL;DR: it was SOTA in coding. However, as we tested early alpha snapshots of the model GPT-5 with several of our most sophisticated coding customers, we realized there were still some gaps in performance and personality that only showed up when you used the models in products like Cursor. Making the experience great is about more than just the raw intelligence, which was solid - it’s also about how the model actually behaves in the product that matters for creating magical end-user experiences. For example, the first model snapshot would silently make many sequential tool calls without summarizing to the user what it’s doing or why, which created a less helpful developer experience. Or, it would produce front-end designs that weren’t as beautiful and clean, even though they were functional and produced with fewer errors than prior models. Seeing these gaps, a small group of startups solutions team members and engineers sprung into action. They decided to try to tackle this problem and bring the customer voice more squarely into our research process, working backwards from the actual tasks that startups needed the model to perform in their product. The key realization is that it’s not enough to saturate the public benchmarks; we had to make sure the models work and feel right in products like Cursor. A small team sprinted to work backward from these customer use cases and create representative evals, graders, and rubrics for scoring the model’s improvements on the key dimensions that matter - some of which were subjective by nature, like front-end design. In close collaboration with the research team, these inputs were incorporated into the training of subsequent model snapshots. The cross-functional team ran rapid cycles of development/testing/iteration to steer the behavior and performance of the model. The result is an *exceptional* coding model. It not only excels at core coding tasks - producing high-quality code, fixing bugs, editing code, and completing repo-level refactors - but also has a collaborative personality and an ability to generate strikingly clean & beautiful front-end designs. It’s been an incredibly exciting time at OpenAI. We are loving GPT-5 and I’m excited for developers everywhere to experience this model! https://openai.com/gpt-5/

284

15 Kommentarer

I love deep-diving on the technical stuff, but finally I wrote a more 'high-level' overview on how AI is impacting engineering industries: https://lnkd.in/e8pN7CtE The case is clear for LLMs (chatbots, agents, etc.), but here I focus on 'everything else'. Video: GE Aerospace engine testing for flying through ‘ice clouds’ in their specialized test facility. Credit to GE Aerospace LinkedIn page as the video source. #ai #chatgpt #engineering #automotive #aerospace

465

12 Kommentarer