Alicornutopia

Is there an index of people's index's?
I could make a post of that too, if there isn't one.

It would be great to import them if it wasn't too much work!

Random question- Is there a simple way to find the word-count glowfics? The number of comments work for judging the rough length, but the content of comments varies too much to be super helpful.

I have never written a crawler so I have no idea how much work it'd be! Intuitively it seems that the hardest part would be actually linking a given post on dreamwidth to a given character.

A possibly easier workaround would be linking dreamwidth usernames to glowfic accounts and then the crawler wouldn't necessarily know that a given post is by that glowficcer's one specific character, but it would at least know it was by that glowficcer.

IIRC the "slurp up this glowfic" part already works (Marri imported a bunch of Effulgence and then un-imported it because too much data), and the only trouble would be grabbing up all the links and feeding them to the glowfic-slurping routines.

Oh! Well that's pretty rad then.

Is the "too much data" a money problem?

Kinda, yeah.

How much of a money problem?

More database space is $9 a month. This is neither unafforable nor something I am unwilling to pay; I just don't see any reason to start paying for it now before the rest of the site is done :)

Ohh I see, gotcha.

(Btw I added you on hangouts to discuss any ways I might have of contributing to the project!)

(Although I'm at work rn so not terribly responsive.)

You know, guys, I've got a perfectly good crawler that then takes all the comments, users and icons from dreamwidth pages and puts it into epub format. :P

I can just add a feature that outputs it into some other format if you want. It shouldn't be too difficult. Adding it to the glowfic site would be a bit more difficult, since I'd need some way of putting it into the database (so I'd need to know the structure and find out how to do that), but I could always output it into some sort of text document or something, to be later imported to the site.

If there are other pages that I should be able to scrape data from, that's also fine. The current limit is knowing where to gather the data from; I just need a list of either the threads to get all the comments from, or a list of indexes of the threads, in which case I'll have to make sure I can get all the links off the page properly (without going an accidentally adding all the non-thread links), but that shouldn't be too hard. I'm currently getting all the links (and then posts and comments) from: Alicorn's sandbox index, the Effulgence index, the Incandescence index, the Pixiethreads index, Marri's index, the Radon Absinthe index, and the Glowfic community (I go through each of the months). If someone makes an index of old threads somewhere, or an index of indexes of old threads, or something, I can get all of them too.

I have been waiting for you to get it into a state you're happy with, but then I'm totally down to convert it into a Rails-happy version I can use on my site.