The first time I tried to run a LLM locally, I wasn’t really impressed. Granted that I was running it on a poor MacBook Air with 8GB of RAM that was holding on for dear life, the gap between the local and cloud LLM world was massive.
It’s a very different story now, though. Local models have gotten dramatically better, and so have the tools built around them. While I’m not here to claim that local LLMs beat cloud models outright, I’ve realized they don’t really need to.
It took me far too long to realize that a lot of the AI features I use regularly don’t require a frontier model in the first place. This is exactly why I’ve stopped paying for the following features when my laptop can already handle them locally.
Interacting with PDFs
I don’t need to pay to chat with my PDFs
If there’s one AI tool I can talk about endlessly, it’s Google’s NotebookLM (now Gemini Notebook). The biggest reason why I say that is because the idea it’s built to solve is incredibly useful for consumers. While the tool’s execution of the idea is excellent in most ways, there are certain documents you should keep out of cloud LLMs. As you might’ve figured, NotebookLM is very much powered by a cloud LLM under the hood. That’s where running something locally starts to make a lot more sense.
If all I want to do is summarize a PDF, ask questions about it, pull out specific details, or turn a long document into something easier to digest, I don’t necessarily need to upload it to someone else’s servers first. A local model can handle a surprising amount of that on my own machine, and for more sensitive files, that’s a trade-off I’m increasingly willing to make.
This is where Open Notebook comes in. It’s positioned as an open-source, self-hosted NotebookLM alternative. This means it’s also built around the idea of giving an AI a specific collection of sources and then asking questions based on those sources, rather than treating the model like a general-purpose chatbot. The big difference is that you get much more control over where everything runs.
Open Notebook supports local models through tools like Ollama, so you can keep the documents, embeddings, and the model processing them on your own hardware instead of sending the files to a cloud AI provider! Just as I do with NotebookLM, I can upload the PDFs I’d like it to work with and then ask questions grounded specifically in those documents.
That means I can use it for the same kind of jobs I usually rely on NotebookLM for, like summarizing long reports, pulling out specific details, etc. However, I get a lot more control and don’t need to worry about whether the documents I’m working with should be uploaded to a cloud service in the first place.
Generating podcasts
My laptop has its own podcast studio now
Being able to interact with PDFs without worrying about hallucinations is great, but it’s far from NotebookLM’s coolest feature. The tool’s best feature has, and probably always will be, Audio Overviews. If you aren’t all that familiar with it, the feature lets you generate incredibly realistic and engaging podcasts that turn whatever sources you’ve uploaded into a conversation between AI hosts.
Instead of simply reading out a summary, the hosts actually bounce ideas off one another, explain difficult concepts, and make connections between different parts of your sources. While I’ve tried countless NotebookLM competitors that offer Audio Overview-like features, none of them were particularly impressive. Well, except Open Notebook.
What makes it so good is how much control it gives you over what comes out. You can choose how many speakers you want, give each one a personality and voice, control the tone of the episode, and even choose which model is responsible for writing the outline and script.
The really interesting part for me now, though, is that none of this necessarily has to happen in the cloud. Open Notebook can use a local LLM to write the episode and a local text-to-speech model to voice it. So something I would’ve once assumed required a service like NotebookLM is another AI feature my laptop can handle on its own!
Transcribing audio and meetings
I’ll happily let Whisper listen to it
One task AI is excellent for is transcription. No one, and I mean no one, enjoys listening to the same few seconds of an audio recording over and over again just to figure out what someone said, typing it out, rewinding, and then doing the exact same thing for the next sentence.
This is especially frustrating when you’re dealing with longer recordings like interviews, lectures, meetings, and in my case, press briefings. While there are now countless AI tools (and devices like the Plaud Note and the Flowtica Scribe) that will happily do the boring bit for you, the same privacy issue as above creeps in. You’re often uploading entire conversations to a company’s servers just to get a transcript back.
That’s not always a deal-breaker, but it can be a pretty uncomfortable trade-off when the recording contains confidential work discussions, unreleased product details, or anything else you’d rather keep to yourself. Fortunately, Whisper, OpenAI’s open-source speech recognition model, can handle the entire transcription process locally.
It’s available in several sizes, and you can install it directly on your computer. The models also support multilingual transcription, and using them is as easy as dropping in an audio file and letting Whisper turn it into text. Since it’s a local model, you don’t need to worry about running into usage limits or needing to pay a separate subscription just to transcribe another recording. Once it’s on your machine, you can essentially throw as much audio at it as your hardware can handle.
If you don’t want to deal with the command line, apps like MacWhisper make the whole process much more approachable. You can import a recording, choose the Whisper model you want to use, and get a transcript back without the audio ever leaving your Mac.
Local models are great for all the basic stuff
Claude doesn’t need to fix every typo
Like everyone, I lean on AI chatbots for a lot of basic tasks. Cleaning up a paragraph, summarizing something, turning messy notes into bullet points, brainstorming a few headline ideas, or extracting specific information from a chunk of text are all things I use them for constantly.
None of these tasks need a frontier model. With how annoying AI limits are and how often I turn to chatbots for tiny, throwaway tasks throughout the day, using a local model for the basic stuff just makes more sense! On my Mac, I run local models via Ollama or LM Studio, and they do the basic job well. Given I use chatbots on my phone for tasks like these too, I also run models locally on my iPhone through Google’s AI Edge Gallery.
I’d rather use cloud AI where it actually matters
Just reiterating what I said at the very beginning, I’m in no way saying that local LLMs can do everything cloud LLMs can. It would simply be unfair to expect that from a model running on your own hardware versus one running across enormous data centers with access to far more compute.
For anything complicated, I still reach for cloud models without thinking twice. However, for all the smaller AI features and everyday tasks I use constantly, local models have reached a point where they’re more than good enough. And if my laptop can already do the job, I’d rather let it.