asciimoo/hister

My coding agent uses the hister cli, i.e. a local index. That often requires me to seed it manually as a downside. The upside is that it caches website contents via browser plugin, which is a nice workaround for bot blocking. Thanks asciimoo for https://github.com/asciimoo/hister

Thank you so much for building this! It's a really awesome piece of work and I'm grateful for your work. I'd love to sponsor you on Github in the near future. The only thing that I think would be interesting to see is native support for crawling via a sitemap.xml instead of recursively. I worked around this by implementing a basic scraper that fetched pages exclusively from the sitemap.xml to add into Hister. I think you're already aware of this, but I also experienced some data loss during the import because I was running a concurrent reindex. I clocked it pretty quickly so I didn't think too much of it. [1] [1] "TODO store new documents in both indexes while running reindex to guarantee…

I'm probably in the minority but I can't stand having more than 10 tabs and I start have the urge to finish the task that I'm focused on to just close as many tabs as possible. So I really read the people talking about 100s of tabs and when I see a colleague having them, I feel that my cognitive ability to deal with that is not there. I understand that people would like to keep things and that bookmarks sucks, but I would suggest for those to explore Hister [1] and Karakeep [2]. It is much useful that I don't worry about losing stuff now. [1] https://github.com/asciimoo/hister/ [2] https://github.com/karakeep-app/karakeep

On a large scale, not that I'm aware of (I'd be interested if that exists!). But on the small local scale, that's what Hister is, it caches and indexes web pages as you browse the web in your browser. Going via the browser avoids all the problems of bot detection etc, since you are a human visitor. That gives your own search engine of the web that you've already visited, at least: https://github.com/asciimoo/hister

> Nowadays I need to keep my own copies... https://github.com/asciimoo/hister > Hister is a private search engine for the pages you visit and the files you keep. It indexes their full contents so you can find information again from the web interface, terminal, or an AI assistant connected through MCP.

You can self-host Marginalia [1] or Hister [2], for example. Takes up some space, but it's totally doable. Your biggest problem (assuming you have disk space) will be crawling. [1] : https://github.com/MarginaliaSearch/MarginaliaSearch [2] : https://github.com/asciimoo/hister