Glowfic Ebooks

[Throne3d]

I have generated EPUBs for Silmaril and for Lintamande's sandboxes (hardcoded list of the ones on dreamwidth, I think it's complete, and then it scrapes the Constellation account; has the sections in a weird order because of how I've done it but I'll probably fix that at some point).

• Lintamande's Sandboxes
• Silmaril

The "leave of absence" page crashes my EPUB reader, on Calibre (edit: it appears to sort-of work on this online reader! Not that that's much use), presumably due to its size (it's 3.1MB, and Calibre warns for anything over 260KB, so… yeah). I'm not sure if it'll do that to everyone's, and it's still useful as a backup, but it might not, y'know, actually be useful if you plan on reading Elentári on an EPUB reader (oops), or actually any threads with a huge number of replies in one go – that thread had ~4500, I think. This is because I no longer split on large files (and haven't for a while), though I might re-implement that at some point.

I also have HTML copies of these things that could in theory be placed onto the site or something! (Seeing as there is an old HTML copy available.) It doesn't have a TOC yet but it wouldn't be hard to have it automatically generate one, and it displays like this currently – you might notice the resemblance to the EPUB, as opposed to this which is the current HTML copy. The styling can, however, easily be changed, such as making the icons full-size (compare this).

[DanielH]

On my computer at least, Calibre warns about large chapters but doesn’t actually have problems with them; I’ll check Leave of Absence and see what it does to Calibre for me.

[DanielH]

And I have scrolled through that entire chapter, from its ToC entry to the link saying it continues with Halls of Stone, without crashing Calibre.

[Throne3d]

Hm, not sure why it was crashing on me then – it worked to load it in the 'edit book' menu but not in the actual book reader itself. Might be something to do with my configuration.

[Throne3d]

I still can't get it to function properly so I'm probably going to end up splitting the HTML files again, but do it based on number of replies instead of file size this time (so it's more consistent and so I can still hopefully internalize links properly, which is another thing I need to implement). Both Gitden and Calibre continue to crash on me when I try to load either Silmaril or the collection of Lintamande's sandboxes if I open up 'leave of absence', though they work okay on shorter chapters, and 'edit book' in Calibre still works.

I fixed an issue with how it imported the subthreads from the Effulgence index, like the "repealing" one, because it was doing it wrong and only imported two sub-threads for that one ("Rêverie" and "very similar seven-year-olds", oops). Next time I update the EPUB this will be fixed.

In other news, I have processed all the current 'groups' (things that I scrape, "Incandescence" is a group, "Constellation" is a group, "Silmaril" is a group, etc.) that I have available after making a couple of changes to how it processes things (see the aforementioned index thing, also now it does default icons for Constellation characters properly). I have bundled up the files (into a .7z file, which may not be supported by your system – try getting 7-zip) and uploaded a backup here, so there is a nice JSON hash of all the data from all the EPUBs I have generated as of earlier today, which is hopefully good for if anything horrible should happen. (This might contain more data than the EPUBs, though they can as a last resort be re-scraped for data if necessary.)

I have also made it so I can output stuff to my locally-hosted development copy of the Constellation, limited slightly by the Constellation's code's ability to handle threads, which results in this nice thing and here's more sample (yes, it also imports characters and icons and where applicable default icons, although only the ones that are used in the threads it imports, but again, this is currently only on my locally-hosted development copy, not the actual thing), and it should work for anything my scraper can scrape, so it should be able to do Incandescence and also copy over all the current Constellation threads and such. Unfortunately, to run it on the actual Constellation would probably take a while and be rather slow and also probably impact how it functions during that time, though it's possible that I can import it into my local copy, dump the database, hand it to Marri, and then she copies the data onto the current Constellation and maps things where appropriate (so whatever ID it has for "Pedro" on my database gets changed to whatever the ID is for "Pedro" on the actual thing), and then also consolidate duplicate characters. Multi-threaded posts are not supported in the Constellation's display, and posts that branch halfway down a chain are not supported by the database system, so these will get 'imported' weirdly currently.

[Throne3d]

I have updated the Silmaril EPUB with some updated code which should do splits on large chapters to allow it to function on a wider range of ebook readers (because Calibre kept crashing on me with the other one, whereas this one it doesn't crash and also seems to function properly). I also updated the non-split version (but using the new code and just telling it to split after 20k replies, so it might have done something weirdly, do let me know).

I think the version with splits should function mostly the same, just with some pages ending slightly shorter than you would expect and the chapter continuing on the next page fine, but there might be issues with some readers? It's what I used to do before I rewrote the code to use Ruby, but I'm not sure – I think someone mentioned it acting funny.

Silmaril (with splits)
Silmaril (without splits)

Something about my scraper seems to be running slowly, now – I'm not sure what I actually did to make it run more slowly, and I can't find anything while profiling it to see where it spends most of its time, so that's kinda confusing. Also I probably won't be using this to scrape stuff onto the Constellation anytime soon, because it takes more memory (RAM) than I anticipated and I'm not sure if the Constellation 'dynos' would be able to cope with that. I can fix it to do them in smaller chunks (and therefore use less memory), but it'd require quite large changes to the code.

[pedromvilar]

i think this thread ought to be renamed to like "glowfic ebooks" or something along those lines

[Throne3d]

Updated EPUBs as of some point in the past few hours! (These ones have splits in, so if anyone is in fact frustrated by splits, let me know and I can probably generate ones without splits or try to work out some way for the splits to be less frustrating.)

• Peterverse
• Throne3d's Sandboxes
• Zodiac
• Opalescence
• Lintamande's Sandboxes
• Silmaril
• Lighthouse

Processing a Constellation EPUB now (with the continuities that have individual EPUBs excluded, to make it smaller), so that should be up soon.

[Throne3d]

• Constellation

Still comes out at 115MB (which is slightly less than the last one, I think), but it contains a lot of icons and things so that's kinda understandable.

[DanielH]

I am continuously frustrated by splits; I have old versions of Effulgence and Incandescence from before you removed them. Could you please generate unsplit versions?