No-label, prompt-driven vision is here, but errors, closed models, and shifting roles for specialists show democratization still comes with fine print.
Nick Byrne wasn’t expecting this when he logged onto Hacker News that Tuesday morning. Scrolling through familiar headlines, he landed on a deceptively simple question: What if you could build a computer vision model without ever labeling a single image? What if you could just…prompt?
“Traditional computer vision has always required significant domain expertise,” Byrne explains. “You need someone highly specialized to build models, label images, and train the system. That’s always been a bottleneck.”
Computer vision, a branch of artificial intelligence, teaches machines to interpret the visual world, identifying objects in images and videos much like human eyes and brains. Historically, these tasks required thousands of annotated images and algorithmic precision crafted by experts.
But a blog post by Simon Edwardsson, the one Byrne had stumbled upon, challenged that foundation.
In his blog, Edwardsson ran a small experiment. He tested Google’s Gemini 2.5 Pro, the cutting-edge multimodal large language model (LLM), on MS-COCO, a popular benchmark dataset used to test object detection. He analyzed Gemini’s results against YOLO v3’s, a classic computer vision model released in 2018 that had been rigorously trained on thousands of labeled images.
Gemini, using nothing but natural language prompts, achieved a mean Average Precision (mAP) score of 0.34—just above YOLO v3’s 0.33.
To the untrained eye, that may seem like a minor win. But to researchers and developers, it means more. Gemini had not been explicitly trained on this dataset. It was simply given an image, a list of valid object classes, and a clear prompt, summarized as: Tell me what you see.
Edwardsson hadn’t written a single line of training code or manually labeled any bounding boxes. He just asked the model, and it answered.
Instead of months spent collecting training data and tuning models, developers can now type: Find all the dogs in this image.
While AI models aren’t guaranteed to “Find all the dogs”, there’s steady progress in their improvements.
The most important takeaway is that the barrier to entry for working with computer vision models has plummeted. Developers without specialized computer vision experience can now prototype powerful applications using plain English.
It’s a vision of the future that’s both thrilling and deeply democratic.
But not so fast. Byrne points out the limitations. “These models are generalists,” he says. “They won’t replace specialists anytime soon.”
Gemini made plenty of errors: missing objects, drawing imprecise bounding boxes, and even, in some cases, generating outputs so chaotic they were unusable. More complex outputs like segmentation masks caused it to break entirely.
Jen Bishop, Byrne’s colleague who previously collaborated on real-world vision projects, reinforces these practical constraints, “In projects where precise logo detection mattered, we still had to meticulously train models on specific logos and contexts. General-purpose prompts aren’t yet sufficient for highly specialized commercial use cases.”
There’s another hitch: access. Gemini 2.5 Pro is proprietary. It’s powerful but closed. You pay to access the API. You cannot modify it. You don’t know what’s inside.
For Byrne and others in the open source community, this raises concerns.
Transparency matters. Not just for understanding performance, but for ensuring trust, flexibility, and cost-effectiveness.
As Travis Oliphant, founder of OpenTeams said,
“If we rely solely on closed models, we limit who can innovate.”
Byrne has begun replicating Edwardsson’s test using open-weight models—publicly available alternatives that allow full inspection, modification, and experimentation. The results so far aren’t quite as good as Gemini’s, but they’re improving fast.
This highlights a central tension in modern AI: open vs. proprietary. Who controls the tools? Who owns the data? And what happens when foundational capabilities are locked behind paywalls?
Computer vision has long straddled these two worlds. On one side: the open source community, building foundational libraries like OpenCV, PyTorch, and TensorFlow. On the other: tech giants offering sleek, integrated services like AWS Rekognition or Google Cloud Vision.
Each has its strengths. Open source is infinitely flexible, but demands technical expertise and time. Proprietary tools offer speed and support, but can be expensive, opaque, and inflexible.
Increasingly, the two are blending. Many commercial AI platforms are built atop open source cores. The result is a “hybrid” model: developers get the transparency and control of open systems, with the reliability and polish of commercial platforms.
OpenTeams, Byrne’s employer, wants to bridge that gap—providing the flexibility of open source tools for computer vision along with deep expertise.
Despite the growing power of prompts, domain experts aren’t disappearing. Their role is shifting.
“Before,” Byrne explains, “experts had to spend weeks building and retraining models for every new task. Now, we can let general-purpose models handle the easy stuff, and let people focus on hard problems.”
In practice, that means faster prototyping, broader experimentation, and better use of human talent. “We won’t replace experts,” Byrne says, “We’ll free them to work on things that require their level of skill.”
The more “general-purpose” our machines become, the less tedious human work needs to be.
Experts are increasingly free from grating routines. Democratized AI enables them to tackle specialized cases, adapt to new contexts, and find creative solutions to problems.
Tedious work can be outsourced to a model. The strategic part still belongs to us.
That’s why Byrne’s team at OpenTeams is betting on something deceptively radical: keep the transparency, keep the flexibility, and layer on the kind of enterprise-grade support that big organizations actually trust.
In this telling, the “open vs. proprietary” debate is less about ideology and more about speed, control, and trust. If you own your AI, you can adapt faster than the people who don’t. That’s the real advantage. And one of the many benefits open source provides in computer vision.
The promise of multimodal AI in computer vision isn’t that it makes experts obsolete. It’s that it makes them matter more.
Building a JupyterLab plugin usually starts with small experiments – you test your ideas, change a few lines, reload, and repeat. Learn how we integrated AI into the Playground AI
Share Your Feedback
