JLT

Tag: Open Source

  • Local-First AI: Running Language Models Without the Cloud

    Cloud-based AI is convenient. Upload your data, get results back, pay by the token. The model lives somewhere else, and so does your context. That trade-off works until it doesn’t.

    Running models locally changes the equation. Your data stays on your machine. Your context window belongs to you. Latency drops to milliseconds. Cost structure flips from per-token billing to one-time hardware investment.

    The Hardware Reality

    Local inference hardware has improved dramatically. A mid-range consumer laptop now runs 3-billion-parameter models in real time. Larger models, up to 70B parameters, run on desktop hardware with discrete GPUs or high-memory configurations.

    The Intel Core Ultra 9 185H, a laptop-class processor, handles 3B-8B parameter models at acceptable speeds without a discrete GPU. Adding a dedicated GPU shifts the ceiling significantly higher. The practical constraint isn’t hardware — it’s knowing which model fits your hardware and your task.

    What You Actually Gain

    Privacy is the obvious benefit. Code, documents, conversations — none of it leaves your machine. For enterprise users, this eliminates a category of compliance overhead. For individuals, it means your personal context isn’t training someone else’s model.

    Less discussed: latency changes how you interact with AI. When response times drop below 100ms, you stop treating AI as a separate workflow. It becomes part of your existing tools. The interaction model shifts from “submit prompt, wait, read response” to “iterate rapidly on ideas.”

    Offline capability matters more than it should. Presentations without wifi, flights, conference calls in venues with bad connectivity — the model still works. This isn’t theoretical. It changes which problems you attempt to solve with AI.

    The Trade-offs Are Real

    Smaller models have lower capability ceilings. A 3B parameter model won’t reason through complex multi-step problems the way a frontier model does. The gap closes for specific tasks — summarization, extraction, classification — but it doesn’t disappear.

    Maintenance overhead increases. Local models need updates, hardware upgrades, and troubleshooting. Cloud providers handle this invisibly. Self-hosting means you own the full stack.

    Context window management becomes your problem. Cloud providers abstract this away with retrieval-augmented generation or extended context windows. Running locally means you manage chunking, retrieval, and context overflow yourself.

    When It Makes Sense

    Local-first works when data sensitivity is high, when you need offline capability, or when usage volume would make cloud costs prohibitive. Development workflows with proprietary codebases fit this profile. Research workflows with sensitive documents fit it too.

    The sweet spot is tasks that don’t require frontier model capability. Summarization, extraction, classification, code completion — these work well at 3B-8B parameters. The moment you need multi-step reasoning on novel problems, cloud models still win.

    Most teams will end up using both. Local for privacy-sensitive, high-volume, latency-critical tasks. Cloud for capability-intensive tasks. The interesting question is how to build workflows that switch between them intelligently.

    What’s your current setup? Are you running models locally, or is everything cloud-based?

  • The Hidden Cost of Free AI: What You’re Actually Paying For

    We live in the golden age of free AI models. Thanks to platforms like OpenRouter, anyone with an internet connection can spin up a session with a model that would’ve cost thousands of dollars in compute just a year ago. No credit card, no API keys (mostly), no commitment. Just type and watch the magic happen.

    But let’s talk about the thing nobody puts in the marketing copy.

    The Bill Always Comes Due

    Here’s the uncomfortable truth about “free” AI: compute isn’t free. Electricity isn’t free. GPU clusters aren’t free. The engineers who fine-tuned those models aren’t working for exposure. Someone is paying the bill.

    When the platform isn’t you, the product is.

    Free tiers on AI platforms typically sustain themselves through a combination of strategies, and it’s worth understanding exactly how your “free” session is being funded:

    Data collection and model improvement. Every prompt you send, every correction you make, every conversation you have is logged, anonymized (we hope), and fed back into the training pipeline. Your real-world questions become the fine-tuning data that makes the next version smarter. You’re not the customer. You’re the labeling workforce.

    Rate limiting and quality routing. Free tiers often get routed to lower-tier inference endpoints. Your requests might hit oversaturated servers, get batched in ways that reduce quality, or be deprioritized when demand spikes. Meanwhile, paying customers get the fast lane. This isn’t malicious — it’s basic economics. But it means your “free” experience is intentionally throttled.

    The upsell funnel. Free access is the best marketing tool in the world. Once you’ve built a workflow around a free model, hitting a rate limit or needing a slightly better model makes the $20/month upgrade feel like a no-brainer. The free tier is a trial that’s genuinely useful — but it’s a trial designed to create dependency.

    The Privacy Tradeoff

    Here’s the part that should give you pause: when you type something into a free AI, where does it go?

    Terms of service for most free-tier services include broad language about data usage. Your conversations might be stored for “service improvement,” “safety monitoring,” or “research purposes.” If you’re pasting code snippets, business logic, or personal information, you’re trading that data for convenience.

    This matters more than you think. A developer pastes proprietary code into a free model to debug a tricky bug. A founder shares their go-to-market strategy with a chatbot for feedback. A student submits their thesis for editing help. All of it becomes part of someone else’s dataset.

    There’s no conspiracy here. It’s the same bargain we’ve been making with free internet services for twenty years: your data for convenience. The difference is that with AI, your data isn’t just your search history — it’s your actual thinking process.

    What You Can Do About It

    This isn’t a “stop using free AI” message. Free AI is democratizing access to powerful technology, and that’s genuinely great. But here’s how to be smart about it:

    • Assume everything you type is logged. Don’t paste code, credentials, trade secrets, or personal information into free-tier models. If it wouldn’t be appropriate on a billboard, don’t type it.
    • Use free models for exploratory work. Brainstorming, learning, casual writing — these are perfect use cases for free tiers. Save paid, privacy-respecting options for anything sensitive.
    • Read the privacy policy. I know, nobody does this. But the difference between “we anonymize and aggregate your data” and “we may use your inputs for commercial purposes” is worth knowing.
    • Consider local models for sensitive tasks. Open-weight models that run on your own hardware — which we’ll cover in a future post — give you the power of AI without the data surrender. It’s not free (you need compute), but it’s private.

    The Bottom Line

    Free AI is an incredible resource, and it’s not going anywhere. The providers offering it aren’t charities — they’re running a sustainable business model that extracts value in ways that may never touch your wallet but will touch your data.

    That’s not necessarily bad. But knowing the cost lets you make informed decisions about what you share, when you share it, and when you should invest in something that respects your privacy as much as your intelligence.

    What’s your threshold for pasting something into a free AI model? Do you have a “no personal data” rule, or do you treat it like a trusted colleague? I’d love to hear where you draw the line.

  • Why Every Developer Needs a Local AI Setup in 2026

    Six months ago, I recommended spinning up a VM before letting an AI agent loose on your system. It was good advice. But the landscape has shifted, and the recommendation has evolved.

    Running AI on someone else’s servers is fine for casual use. But if you’re a developer who writes code for a living — or even as a passionate hobby — you should seriously consider running at least some AI workloads on your own hardware. Here’s why.

    The Trust Equation Changed

    The Claude Code source code leak in March 2026 was a wake-up call for anyone who thought proprietary AI was a secure black box. When a single missed line in a configuration file can expose half a million lines of source code, including internal tooling, security logic, and hidden experimental features, it becomes clear that the “trust the provider” model has cracks.

    If a company as well-resourced as Anthropic can accidentally expose their entire codebase, what does that mean for the data you’re sending through their hosted APIs?

    Local models remove a variable from the trust equation. When the model runs on your machine, your data never leaves it. No terms of service to parse, no data usage policies to hope are enforced, no third-party server to get breached. What you type stays on your hardware. Full stop.

    It’s Easier Than You Think

    There’s a persistent myth that running AI locally requires a workstation that costs more than a used car. That was true two years ago. It isn’t anymore.

    You don’t need to train a model. You just need to run one — and for that, Ollama and llama.cpp have made the barrier to entry almost trivially low. On a modern laptop with 16GB of RAM and a decent CPU (no GPU required for smaller models), you can run a 7B or even 13B parameter model that handles code completion, summarization, drafting, and general Q&A quite well.

    The setup is usually: install Ollama, pull a model, and you’re done. No Docker, no CUDA (unless you want it), no venv hell. It takes about ten minutes.

    Not Everything Should Leave Your Machine

    Think about the tasks you do as a developer on any given day:

    • Pasting a stack trace to figure out what broke
    • Asking an AI to review a function before committing
    • Feeding it a config file to debug a deployment issue
    • Running it against your git diff to generate a commit message

    All of these involve code that might be proprietary, infrastructure details that reveal your architecture, or bugs that expose vulnerabilities. When you send these to a cloud API, you’re trusting that provider with information about your actual work product.

    With a local model, you can do all of this without transmitting a single byte externally. You can point the model at a codebase in your home directory and ask it things without creating a data trail. That’s not paranoia — it’s good operational hygiene.

    The Reality Check: Local Models Aren’t Magic

    Let’s be honest about what local models can and can’t do right now.

    A 7B model running locally won’t match GPT-4.5 on complex reasoning tasks. It won’t architect a microservices migration or catch subtle logic errors in your codebase. The smaller the model, the more you’re trading accuracy and depth for privacy and control.

    But here’s the thing: you don’t always need GPT-4.5. For code completion, docstring generation, regex writing, explaining errors, summarizing PRs, or drafting emails — small local models are genuinely competent. They’re good enough to save you hours of context-switching to the browser while keeping your work private.

    Think of it like having a junior colleague: they won’t design the system, but they’ll happily format your documentation, explain that cryptic error message, and write the boilerplate you really don’t want to type.

    When to Use Local vs Cloud

    The smartest approach isn’t “local only” or “cloud only.” It’s knowing which tool fits which job:

    Use local models for: Code review, debugging, writing scripts, generating documentation, experimenting, and anything involving sensitive code or data.

    Use cloud models for: Complex architecture decisions, multi-step reasoning, tasks requiring the latest knowledge, and anything that needs a frontier model to get right.

    This hybrid approach gives you the best of both: privacy and speed for the everyday grunt work, and raw power when the problem demands it.

    Getting Started

    If you’re curious, here’s the shortest path:

    • Install Ollama from ollama.com
    • Run ollama pull qwen2.5-coder:7b (a model specifically fine-tuned for code tasks)
    • Run ollama run qwen2.5-coder:7b and paste it some code

    That’s it. You now have a private AI coding assistant running on your own hardware. It won’t replace your cloud models, but it might surprise you with how much useful work it can do without ever phoning home.

    Have you tried running models locally yet? What’s the smallest model you’ve found that’s actually useful for your day-to-day work? Drop your setup in the comments.

  • The Security Paradox: Why Open-Weight Models Might Be Safer Than Closed APIs

    The recent leak of Claude Code’s source code has reignited a classic debate in the tech world: is it better to keep your code a “black box” or to open it up to the world? While the immediate reaction to a leak is panic, many security researchers argue that the future of safe AI actually lies in open-weight models like Qwen or Llama.

    The Fallacy of “Security Through Obscurity”

    For years, companies have relied on the idea that if hackers can’t see the code, they can’t break it. This is known as “security through obscurity.” But as the Claude leak showed, obscurity is fragile. Once that single .npmignore line was missed, the entire fortress was exposed.

    In contrast, open-weight models operate on Lincoln’s Law: “Given enough eyeballs, all bugs are shallow.” When a model’s weights and architecture are public, thousands of independent researchers can audit it for biases, backdoors, and security flaws simultaneously.

    The “White-Hat” Advantage

    When a vulnerability is found in an open model, it’s usually patched quickly because the community is invested in its success. With closed APIs, users are forced to trust that the provider is fixing issues without any way to verify it. In the high-stakes world of AI agents—where a model might have permission to delete files or transfer money—this transparency isn’t just a nice-to-have; it’s a necessity.

    Balancing Openness and Safety

    Of course, open models aren’t a silver bullet. They can be misused by bad actors who want to strip away safety guardrails. However, the trend toward “open-weight” (where the model is free to use but the training data might remain proprietary) offers a middle ground. It allows for rigorous security auditing while still protecting the company’s core data assets.

    As we move toward more autonomous AI, the question isn’t whether we should open up our models, but how quickly we can build a security ecosystem that supports them.

    Do you trust closed AI models with your sensitive data, or do you prefer the transparency of open-weight alternatives? Let me know your thoughts.

Powered by
►
Necessary cookies enable essential site features like secure log-ins and consent preference adjustments. They do not store personal data.
None
►
Functional cookies support features like content sharing on social media, collecting feedback, and enabling third-party tools.
None
►
Analytical cookies track visitor interactions, providing insights on metrics like visitor count, bounce rate, and traffic sources.
None
►
Advertisement cookies deliver personalized ads based on your previous visits and analyze the effectiveness of ad campaigns.
None
Powered by