Alicornutopia

Posted: **Tue Dec 22, 2015 5:44 pm**

Some bug reports for your code, as of the most recent Effulgence you linked:

Sometimes (in Calibre) the headers of some comments are much taller than they should be
It might just be my monitor, but the Alicorn green is much too blue.
Not exactly a bug, and I don’t think there’s anything to be done about it, but (at least on the readers I use) I much prefer to not split the large chapters like Calibre recommends.
You already know you don’t have symbella support.

Other than that, your ebooks seems good. I will look into my version and yours and see if there’s anything that I have that can be moved to your version. If so, I’ll send you a pull request. I really don’t like having two versions floating around (forks make things harder for everybody else), but you update more consistently than I do so yours should be the main version.

Speaking of updates, here are the Effulgence versions with my code in case anybody else prefers them.

Posted: **Tue Dec 22, 2015 5:51 pm**

DanielH wrote:Some bug reports for your code, as of the most recent Effulgence you linked:

Sometimes (in Calibre) the headers of some comments are much taller than they should be

It might just be my monitor, but the Alicorn green is much too blue.

Not exactly a bug, and I don’t think there’s anything to be done about it, but (at least on the readers I use) I much prefer to not split the large chapters like Calibre recommends.

You already know you don’t have symbella support.
Other than that, your ebooks seems good. I will look into my version and yours and see if there’s anything that I have that can be moved to your version. If so, I’ll send you a pull request. I really don’t like having two versions floating around (forks make things harder for everybody else), but you update more consistently than I do so yours should be the main version.

Speaking of updates, here are the Effulgence versions with my code in case anybody else prefers them.

Yep! I already noticed the thing in Calibre, and it seems to be because I don't want it to split the username box over two pages (so I used the "page-break-inside: avoid;" CSS attribute). Unfortunately, when I do that, Calibre likes to generate it over two pages, then go "oh wait, shouldn't do this", and shove the huge one on the next page. It also ignores some of the time when I tell it not to split. I wasn't really sure how to fix it (other than putting up clear divs before each post, which I think would work), and it mostly renders fine, but that's an ongoing issue.

I'm not sure about the alicorn green - perhaps? I've just modified it up in the "lum" value, but I did notice that it made some of them look weird, not just lighter. I could try editing that later.

The splitting is only a small thing, as far as I can tell, but it does severely impact the performance on iBooks if you don't have it split up (for some reason, scrolling over pages (about 3 posts long) suffers by taking a few seconds if you have large chapters). I'm not actually using iBooks anymore (I've moved over to Android), but I'm not sure who's using what, so... I'm not sure what to do about that.

And that's a good point! I should definitely go back and look at adding symbella support. Thanks!

Currently I'm looking into an issue of it not properly doing moiety on the new version (weirdly, it does pythbox, but not, for example, "subtly_artistic"?). It's probably just a stupid thing I did when I ported over, so it shouldn't take long, but I'll upload a change when I'm done.

Yeah, I agree about the two-version thing. Sorry about that. :\

Posted: **Tue Dec 22, 2015 6:06 pm**

My favorite Android reader is Gitden, which is sometimes slow. I think that more has to do with the size of the book as a whole instead of the size of the chapter, though. If you say it’s a thing in iBooks I’ll believe that.

My first guess based on those two examples is that it’s an underscore vs. hyphen issue in screennames, but if you mean it does pythbox but none of the others then it seems like you aren’t correctly checking community membership.

Most of my glowfic-related coding effort will go into consolidating the two versions and then into trying to understand Marri’s platform. With any luck both of our versions will be obsolete by the end of 2016 (number invented out of thin air).

Posted: **Tue Dec 22, 2015 6:09 pm**

DanielH wrote:My favorite Android reader is Gitden, which is sometimes slow. I think that more has to do with the size of the book as a whole instead of the size of the chapter, though. If you say it’s a thing in iBooks I’ll believe that.

My first guess based on those two examples is that it’s an underscore vs. hyphen issue in screennames, but if you mean it does pythbox but none of the others then it seems like you aren’t correctly checking community membership.

Most of my glowfic-related coding effort will go into consolidating the two versions and then into trying to understand Marri’s platform. With any luck both of our versions will be obsolete by the end of 2016 (number invented out of thin air).

Nope, nope. It was underscores vs hyphens. I realised that just a few minutes before you replied. Ugh. :\

Edit: Welp, that should be fixed. I've pushed a change, and I'm testing it now (and a debug that gave a message when it failed to find a moiety is now being silent (which I added after finding the moiety problem, woo >.>), so hopefully that fixed it).

Posted: **Wed Dec 23, 2015 10:46 am**

Updated sandbox (as of this post's creation): https://www.dropbox.com/s/24cvmgp418sch ... .epub?dl=0
Pixiethreads!: https://www.dropbox.com/s/6i7s7xq7trqla ... .epub?dl=0
Reprocessed Incandescence: https://www.dropbox.com/s/qhhcik7l66r4a ... .epub?dl=0

(Generated using the new code - seems fine, looking over it quickly, but there could be issues.)

Well, based on the descriptions of the code updates I've made ("commit messages"), it seems that since my last post I've:

added (hard-coded) "disregards-the-lines" to ErinFlight's collection (from the sandbox "Loki and Raiden", which says it's "(with ErinFlight)")
fixed the discarding of previously-downloaded pages in the "flats" section (e.g. "flats_sandbox" used to delete the already-downloaded stuff if you'd since executed "tocs_sandbox")
added pixiethreads support (hopefully; currently untested, but I'll upload the epub when I've tested it)
added symbol(a) support for Effulgence (currently testing)
fixed how it scrapes Effulgence and Incandescence ToCs (it was duplicating all the chapters <.<)
changed what it outputs when it's generating stuff
hopefully fixed the things I did wrong at first (e.g. symbola)

Edit 2015-12-23 23:45 GMT: Oh, and I just made it actually save the epub with the proper name (not just sandbox.epub >.>).

Edit 2015-12-23 23:51 GMT: So it appears that the calibre reader, at least, doesn't go to fallback fonts if it can't find the symbol in the default font. At least, with the rule:

Code: Select all

font-family: sans-serif, "Symbola"

It displays "✴ ☼ Ω ⍋ ✝ ♪ ᗜ Ø ⚜ ҂ ✧ ⚚ ∀ Ψ Ɣ ఒ" like this:

Whereas with:

Code: Select all

font-family: "Symbola"

It displays like this:

(They're both missing the ᗜ (Aegis?) character, but it seems to not fall-back to Symbola from the default sans-serif for some of the others (e.g. ⚜), at least in calibre?)
It's weird. Also, I forgot to tell it to use these fonts in the ToC, so that fails at the moment.

Edit 2015-12-24 01:23 GMT: Added the other fonts used in the previous effulgence2epub thing. Not sure why I didn't add those at first. Fixed something I broke with the incandescence ToC scraper. Fixed the title and authors (Aestrix and Kappa, right?) of the pixiethreads generation. Added a link to the pixiethreads epub download at the top of this post.

Edit 2015-12-24 01:48 GMT: Add a re-processed "Incandescence" (so other people can tell me if it's doing stuff weirdly, hopefully). Also, I've perhaps got a workaround (involving manual effort) to allow the "discretion advised" chapters to be loaded! I'll test it and see what the results are.

Posted: **Wed Dec 23, 2015 8:35 pm**

Sorry in advance for the third post in a row.

Updated sandbox (as of 2015-12-24 03:20 GMT!): https://www.dropbox.com/s/24cvmgp418sch ... .epub?dl=0
Pixiethreads!: https://www.dropbox.com/s/6i7s7xq7trqla ... .epub?dl=0
Reprocessed Incandescence: https://www.dropbox.com/s/qhhcik7l66r4a ... .epub?dl=0
Reprocessed Effulgence: https://www.dropbox.com/s/h693wn2fvdl69 ... .epub?dl=0

I realised a bit of a flaw with the whole "this should work on Windows now!" thing. I was saving files with question marks in their names on Linux. Windows doesn't like any of the characters \/:*?"<>| in file-names. That's fixed now (because they're all replaced to ugly, longer versions of themselves - if you see a 1640.html~×QMARK×~style=site&view=flat somewhere, that's due to ugliness).

The workaround for the "discretion advised" message seems to be working, actually. I also have confirmation that images embedded in posts will be downloaded into the epub (and displayed there), though links (to both internal and external websites) still won't work (they'll just take the reader to that external website) unless generated by the script (so basically only the table of contents will work).

Good news! The symbella support in the Effulgence epub seems to be working nicely. At least on the table of contents. In places. The Aegis shield appears fine (see evidence and more evidence) in some scenarios (see when not (should be "✴ ☼ Ω ⍋ ✝ ♪ ᗜ Ø ⚜ ҂ ✧ ⚚ ∀ Ψ Ɣ ఒ" (this) - where the hell did the Aegis shield after the Angela note go?)). I'm a bit confused. Maybe it's just that Calibre's renderer doesn't like it sometimes? e_e

Oh well.

TL;DR: Everything seems to be working now. You can find updated copies of stuff at the top of this post, and the code is still on its BitBucket page (or an image of what it's like now is available here; I recommend the BitBucket page). Now I'm gonna go to sleep.

PS: Marri, if you see this and want me to scrape the data and just shove it in a specific format for you somewhere so you don't need to fix your scraper to work with threads, I should be perfectly able to save all the data you want in something like a JSON format. Message me on Google Hangouts some time? Same for (probably) anyone else.

Posted: **Wed Jan 06, 2016 6:01 pm**

So, I've been regularly updating both the code and the generated epubs (mainly the sandbox). Not much has changed, visually, but in case anyone didn't realise I was releasing updated versions... I am. They're on Dropbox, still, and can be found in this folder (the sandbox, specifically, is here). I'm uploading an updated copy of the sandbox now, so hopefully that'll contain (most of) the new content from yesterday and before.

Posted: **Thu Jan 07, 2016 12:41 am**

I think at some point there came to be multiple versions by different people. This confuses me and I'm not sure what to upload.

Posted: **Thu Jan 07, 2016 1:18 am**

TL;DR: There are a very few things that my version still gets better, but in general you should go with Throne3d’s.

In particular, Throne3d has a better table of contents, better chapter heading support, and (I believe) better cross-device compatibility. There were some unavoidable tradeoffs involved in the compatibility which I don’t like, but for most people those tradeoffs are worth it. We also handle Symbella support differently; I know mine fails to work in some cases so there’s a chance he gets it better instead of just different. Finally, his version of Effulgence also matches his versions of the others better than mine, giving a more consistent look-and-feel given that you use his other epubs elsewhere.

The other differences are mostly stylistic:

Which parts of the comment header information to include; he includes the icon summary and I don’t; I include the original comment timestamp and he doesn’t. I would prefer to have both, but then the headers get more crowded. Mine makes it easier to look back at the original Dreamwidth comment or to compare the real-world times of two different comments; his is probably better for just plain reading, especially if you sometimes have trouble with the icons.
He puts a border around post icons while I don’t; this is most noticeable for Jarvis’s default icon.
We use slightly different colors for various things
His branching point messages are more clearly visible
Probably a bunch of other minor things

The first of two major non-stylistic issue which my version does better is when the authors accidentally reply to the wrong comments. Often, one author replies to the second-most-recent comment instead of the most recent. When viewing on Dreamwidth, this rarely makes much of a difference, especially in flat view. In the ebooks there are alerts for thread branches (like how the Bellparties and some sandboxes branch), and I deliberately suppress these when it looks like a mistake (the first of two branches has exactly one comment). I hope to, maybe this weekend, submit a pull request to his code which contains my code for handling that special case.

The second of the two issues mine gets better is that I have .mobi (Kindle format) support. This can be created using Amazon’s free KindleGen program for any ebook with the command

Code: Select all

kindlegen -dont_append_source -c2 effulgence.epub

executed at the command prompt/terminal in the folder where the epub is. In theory, this could be done by anybody who needed it; in practice, it is better for us creators to do it.

Posted: **Thu Jan 07, 2016 1:26 am**

OK, Throne3d's versions it is. Maybe at some point I will muster the can to put "alternate version" links if I can remember where-all these are linked to.

Alicornutopia

Glowfic Ebooks

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence

Re: EPUB and MOBI of Effulgence