Candlekeep Forum
Candlekeep Forum
Home | Profile | Register | Active Topics | Active Polls | Members | Private Messages | Search | FAQ
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 Forgotten Realms Products
 Forgotten Realms Software
 New Realmslore search tool: Ask Valhaeria
 New Topic  New Poll New Poll
 Reply to Topic
 Printer Friendly
Author  Topic Next Topic  

Italian Archmage Karsus
Acolyte

28 Posts

Posted - 12 Nov 2022 :  00:02:53  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote  Delete Topic
Hey, guys, I've been working on and off for a year now on a tool to find Realmslore quickly. We've been using a prototype at the wiki, it's been invaluable. If anyone wants to take a look, it's here at

https://askvalhaeria.com

Basically, you input a subject and it tries to find matches in existing sources. The database on Ask Valhaeria so far only includes web sources that were put out for free by Wizards or TSR, so it shouldn't step on any legal toes. I might add search data for other books to that database later; the database doesn't actually store the books, it only stores searches it has already made inside the books, so it shouldn't be anywhere near infringement.

Right now it's ugly as sin, I've been putting off a migration and redesign. But I could really use some feedback, if anyone has two cents to share about it. Next I gotta change the way it shows links so it's a little more user friendly, and then make all of the webpage way, way prettier.

It has other functions, too: if you click "ZOOM" on a source, it allows you to restrict your searches to that source alone. That allows us at the wiki to find where in a book a specific word or sequence shows up without having to sift through the whole book by hand. It allows for multiple queries separated by bullet points, so that we can copypaste straight off the wiki. The last function it has is a plagiarism search. It finds matches for longer sequences (minimum 5 or so words), allowing us to catch plagiarism in the wiki in a blink (as it exposes us to legal action, it's central we don't let it exist for long!).

I've been wondering what else I could add to Valhaeria to make it more useful, but in the meantime, I got told you peeps would enjoy it. So I figured I'd show ya.

Karthak
Acolyte

7 Posts

Posted - 12 Nov 2022 :  12:42:52  Show Profile Send Karthak a Private Message  Reply with Quote
Seems very useful, only thing I could think of adding is an index to show users what terms are currently searchable.
Go to Top of Page

Athreeren
Seeker

59 Posts

Posted - 12 Nov 2022 :  15:33:51  Show Profile Send Athreeren a Private Message  Reply with Quote
Thanks, this seems very useful! However, I have tried lots of terms that give me results on the wiki but not on this platform. How does the platform search for terms? Do you have a database that was filled manually, or did you scrape the wiki?
Go to Top of Page

Italian Archmage Karsus
Acolyte

28 Posts

Posted - 12 Nov 2022 :  22:26:44  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Thanks for your feedback!

Karthak, there's 11 million terms in the database that it has indexes for. It can't be displayed- as it stores not just individual words, but 2-word and even 3-word sequences. The list would be endless and not easy to find; it's easiest to just ask for a term yourself and see if it's there.

Athreeren, it's not wiki scraped, it's only indexing a specific subset of the sources. Later versions may include more sources; the database scrapes sources, not wiki pages. We use Valhaeria to help us research subjects and write wiki pages, not the other way around.
Go to Top of Page

Karthak
Acolyte

7 Posts

Posted - 13 Nov 2022 :  02:09:00  Show Profile Send Karthak a Private Message  Reply with Quote
Ah right, I was figuring it was a short list of maybe 10,000 terms, no point in having an index even if it was possible when most search terms will return some sources.

Is there a difference between Database A and B?
Go to Top of Page

Italian Archmage Karsus
Acolyte

28 Posts

Posted - 13 Nov 2022 :  12:31:47  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Karthak, there is no difference between the two databases. They are a holdover from the prototype, I should later remove them.

Edited by - Italian Archmage Karsus on 13 Nov 2022 12:35:40
Go to Top of Page

sleyvas
Great Reader

USA
11313 Posts

Posted - 14 Nov 2022 :  19:32:34  Show Profile Send sleyvas a Private Message  Reply with Quote
searched mingari - no results
searched savyels (for the entry of a bounty hunter in powers and pantheons) - no results
searched Lauzoril - lot of results, but not to anything I would have expected (i.e. nothing to Dreams of the Red Wizards, spellbound, etc...)... but don't see how you can read the results, it just tells you where to look.

Alavairthae, may your skill prevail

Phillip aka Sleyvas
Go to Top of Page

Italian Archmage Karsus
Acolyte

28 Posts

Posted - 14 Nov 2022 :  22:12:39  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
That's right, sleyvas. It just tells you where to look, but the idea is that it never misses anything- so it helps you sift that much faster.

Valhaeria right now runs on web-only stuff- and I'm working on adding links to every web-available source. I'm working on a prototype with books indexed in it, but it's not quite ready for primetime- we just use it internally in the wiki. Even then, it still only tells us where to look, it just happens to do so in a comfortable blink.

Edited by - Italian Archmage Karsus on 14 Nov 2022 22:14:56
Go to Top of Page

BadCatMan
Senior Scribe

Australia
397 Posts

Posted - 15 Nov 2022 :  02:35:07  Show Profile Send BadCatMan a Private Message  Reply with Quote
Consider it the shareware demo version.

But I've been using the full prototype version, with sourcebooks, novels, and magazines, as well as the free articles you get in Ask Valhaeria, and it has been an absolute godsend to Realmslore research for the Forgotten Realms Wiki. Rather than spending ages searching books manually, even searching PDFs en masse, we can have a list of every source in under a second. That can be way too much for a common topic or a regular word, of course, and long phrases run the risk of false positives. But for less common stuff, it's a very quick way to zero in on the relevant sources, down to within a page or so. All we need to do is find the book and open to the page.

With it, we've been able to write articles that cover every source to mention a topic, and can include some obscure lore and forgotten sources, without having to come back later when we discover or obtain something new. We can also more easily find references for unsourced pages, so we're steadily cleaning them up on the wiki.

It's a D&D-dedicated search engine and the quickest and most comprehensive resource for lore. If you're looking for some Realmslore, do consider trying out Ask Valhaeria.

BadCatMan, B.Sc. (Hons), M.Sc.
Scientific technical editor
Head DM of the Realms of Adventure play-by-post community
Administrator of the Forgotten Realms Wiki
Go to Top of Page

sleyvas
Great Reader

USA
11313 Posts

Posted - 15 Nov 2022 :  15:19:28  Show Profile Send sleyvas a Private Message  Reply with Quote
I must say, I like the concept. Lord knows I'm constantly using the DOS findstr and powershell's select-string to find things for work. Never tried searching PDF's though without opening them and doing a search within the book because like many things you don't get useful outputs. This will definitely help me in my realms research, as I often end up at the wiki to look up things.

Alavairthae, may your skill prevail

Phillip aka Sleyvas
Go to Top of Page

Italian Archmage Karsus
Acolyte

28 Posts

Posted - 25 Nov 2022 :  22:37:53  Show Profile Send Italian Archmage Karsus a Private Message  Reply with Quote
Thanks for the vote of confidence, BCM!
I've been banging my head against the code for a bit, looking to upgrade the database. In my next delivery I'll see about throwing in archive links for every web resource, the novels with Google Books, Google Reader, Internet Archive and Amazon links (all of those let you have a taste for free, I figure they'd be useful for a thrifty lore scholar), and look at getting a presentable version of the magazines somehow.
Go to Top of Page
   Topic Next Topic  
 New Topic  New Poll New Poll
 Reply to Topic
 Printer Friendly
Jump To:
Candlekeep Forum © 1999-2022 Candlekeep.com Go To Top Of Page
Snitz Forums 2000