Craig Stadler <
cstadler18@hotmail.com> writes:
I spent the last year gathering and normalizing as many resources for
Usenet messages and built a search engine over public Usenet
discussions going back to 1981. Posting here because this seemed like
the one group that would actually care.
https://www.usenet-rewind.com/
Since Google Groups stopped indexing new Usenet content a
while back, and search over its historical archive has always been
rough- I wanted something that treats this as a real research
archive -- full-text search, date-range/filtering,etc and most
importantly as much Usenet data as possible going back as far as I could find.
The index currently holds (to date) roughly 980 million messages and is
still growing. Sources are archival backups, data donations, and ongoing crawls of several thousand still-active public NNTP servers. Binary and yEnc-encoded content is stripped where possible to keep the index
focused on text.
The index was built using Apache Solr 10, MariaDB 12 (rocksdb), Ubuntu 22.04.5 LTS & custom Python scripts, all on nvme disk.
This is public archival content, same legal basis as DejaNews and
Google Groups before it. There's a removal process for anyone who
wants their own posts taken down, and author contact info is masked
by default.
Is there any way you can create a gopher equivalent to your site?
Any chance that you'll adopt a public API? I would consider writing a
feature to my finger service + a door on my bbs to interface with it.
--
Daniel
sysop | air & wave bbs
finger |
info@bbs.airandwave.net
--- PyGate Linux v1.5.19
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)