| 1 person marked this as a favorite. |
Was searching through Paizo PDFs for the word "Janderhoff" recently when I noticed that certain fonts used throughout numerous Paizo publications turn the double effs into typographic ligatures so that search functions can't recognize them. I checked it against the hard copy, and found that it exists there as well, and wasn't just a problem with my PDF reader or the PDF file displaying fonts incorrectly.
I then did a messageboard search and found THIS DISCUSSION. It seems that although folks were aware of it back then, it's continued (see "Curse of the Crimson Throne hardcover" page 413).
Don't know if this is still on people's radar, but I thought I'd mention it as it impacts the usefulness of the PDFs.
Chief Technical Officer
|
| 2 people marked this as a favorite. |
This is a problem inherent to the PDF format. Essentially, during the PDF creation process, the individual characters in the ligature are replaced with the ligature itself—that is, the letters "f" and "f" are being replaced with the glyph "ff"—and the PDF format doesn't actually store what the original characters were. In addition to compromising searchability, when you copy text containing a ligature, your PDF reader essentially has to guess what those characters originally were, and sometimes it will guess wrong, especially when it comes to the spacing around them.
You'll find a similar issue with text that's presented in an all-capitals font. For example, a PDF may contain the instruction that it's supposed to display the small cap "A" glyph, but it doesn't actually know whether that glyph is standing in for an uppercase "A" or a lowercase "a"—that bit of information is forever lost during the PDF creation process. So when you go to select text out if it, the PDF viewer you're using has to try to reverse-engineer which characters each glyph actually represents, and some applications are better at that than others. So copying the small-caps word "ADOBE" in different PDF viewers could result in you getting "Adobe", "ADOBE", "adobe", or even something like "aDoBe".
The reason the PDF format doesn't store that data is that the Adobe created the PDF standard with archiving and final document delivery as their primary goal; they didn't intend for people to get data back *out* of it. And even though its clear that many, many people use it for that (in spite of its deficiencies), Adobe apparently has minimal interest in modifying the format to make it work better for that.
(The problem Sean was talking about in the post you linked was slightly different—it involved a font that didn't actually *contain* the ligature glyphs, so nothing displayed at all.)
| 2 people marked this as a favorite. |
This is a problem inherent to the PDF format. Essentially, during the PDF creation process, the individual characters in the ligature are replaced with the ligature itself—that is, the letters "f" and "f" are being replaced with the glyph "ff"—and the PDF format doesn't actually store what the original characters were. In addition to compromising searchability, when you copy text containing a ligature, your PDF reader essentially has to guess what those characters originally were, and sometimes it will guess wrong, especially when it comes to the spacing around them.
You'll find a similar issue with text that's presented in an all-capitals font. For example, a PDF may contain the instruction that it's supposed to display the small cap "A" glyph, but it doesn't actually know whether that glyph is standing in for an uppercase "A" or a lowercase "a"—that bit of information is forever lost during the PDF creation process. So when you go to select text out if it, the PDF viewer you're using has to try to reverse-engineer which characters each glyph actually represents, and some applications are better at that than others. So copying the small-caps word "ADOBE" in different PDF viewers could result in you getting "Adobe", "ADOBE", "adobe", or even something like "aDoBe".
The reason the PDF format doesn't store that data is that the Adobe created the PDF standard with archiving and final document delivery as their primary goal; they didn't intend for people to get data back *out* of it. And even though its clear that many, many people use it for that (in spite of its deficiencies), Adobe apparently has minimal interest in modifying the format to make it work better for that.
(The problem Sean was talking about in the post you linked was slightly different—it involved a font that didn't actually *contain* the ligature glyphs, so nothing displayed at all.)
It is with the deepest respect that I tilt my head sharply to the right, Spock an eyebrow and quietly intone "ummmmmm?"
Vic, I openly grant that I am not a professional typesetter or layout person, but what you're saying doesn't match what experience I do have. All of the following should be prefaced with "in my experience..." PDF in fact stores precisely what it's given from the originating program. In your case I believe it's InDesign. Both the ligature creation and uppercase/lowercase modifications are done at that stage of publication and the output from InDesign obeys what it is told.
Let me explain. An author produces a mostly unformatted manuscript containing only the essentials such as bold and italic. The person doing layout "places" that document into their InDesign layout as an object, just like a picture or box, and proceeds to lay it out by shaping the text box where they want it on pages, and flowing it around other objects. They also apply formatting by highlighting sections of text and applying saved styles. For instance, the original document might have said "adobe", but the layout person highlighted it and selected their "header 4" style, which changes font, font size, left/right alignment, kerning, yada-yada, and applies a "to upper" transform. The result is text that looks like "ADOBE" but is not.
If the layout person prints that page from InDesign, the result will be "ADOBE", on paper. But if they highlight "ADOBE" and copy it and paste it into say... Notepad - which cannot take the formatting metadata - the text will show as "adobe", because that's what it actually IS. Same as if you rotated the text 180 degrees... copy & paste won't make it upside-down in Notepad because that orientation is a transform being applied to the actual text.
Anyway, the PDF format follows the same, blending both results. The appearance in the PDF will be "ADOBE", just as the font shown will mimic what was seen in InDesign because it gets embedded in the file (if necessary). Thing is, the actual text is "adobe", and that's exactly what you're going to get if you copy/paste from the PDF.
In the case of ligatures, that (I believe) alters the actual text during layout. It's an optional layout choice made while placing/laying out the text. In this case, "fi" gets merged into the single-character ligature in InDesign not just cosmetically but actually. Exporting to PDF produces a single character and you can't find "find" because you've actually got a 3-letter-word in the PDF. Again, that's because the edit was done in InDesign before PDF export.
I absolutely, positively can produce a PDF that contains a non-ligatured double-eff.
The point I'm making is - unless I'm very mistaken - this is avoidable, by Paizo not using ligatures when doing layout. It's not about PDF at all... the alteration is done before that stage.
This is sort of like the inverted-background colour thing that keeps cropping up when copy/pasting images from your PDFs. Yes, there's something goofy in the alpha channels of those images. No, it's not our readers that is doing it... it's something being done as standard-operating-procedure during the creation and editing of those images by Paizo. Yes, some readers don't exhibit the issue, but it's inherently not them causing it.
Anyway, I just felt like I've got some technical knowledge that might be of some value to correct a misunderstanding as to where the OP's issue is originating. Or I'm way, way off-base. Either way, deepest respect and thanks for engaging with your customers and discussing the topic at all.
| 1 person marked this as a favorite. |
I apologize for perhaps not understanding the issue completely, but as I mentioned in the OP, the "ff" glyph appears in the print documents as well, not just the pdfs. Is this because the final layouts are sent in PDF format to the printers with the "ff" glyph? Pardon me if I am misconstruing this.
Thanks for addressing this issue, Vic!
This is definitely an issue with the text being output. Either the program is composing the character, or the original document uses it. The PDF reader does not try to "read" the letter shapes when the meta data is included, which in my experience, it always is with paizo documents. For a document to be copy-and-pastable requires a actual text, not just a contour of the letters.
|
Chief Technical Officer
|
| 3 people marked this as a favorite. |
I apologize for perhaps not understanding the issue completely, but as I mentioned in the OP, the "ff" glyph appears in the print documents as well, not just the pdfs. Is this because the final layouts are sent in PDF format to the printers with the "ff" glyph? Pardon me if I am misconstruing this.
The ligatures do appear in the print documents, but it's not because they're sent to the printer as PDFs (though they are)—it's because we want them there, as they make printed text more attractive and easier to read.
Anguish, you are correct that the ligatures are created within InDesign, but I didn't bother talking about that because it's not relevant to my point, which is that it's the PDF conversion process which is throwing away the knowledge of what the pre-ligature characters were. InDesign's file format stores the text as the individual letters "f" and "f" as well as the knowledge that they should be displayed and printed using the "ff" glyph. The PDF file format stores only the latter information, and that's the fault here. You are correct that we could work around this fault by not using ligatures in our layout design, but that makes for a less attractive and less readable printed product.
So, why not use ligatures for the print edition and then disable them for the PDF edition? It's because disabling them invariably makes the text wider, and every now and then, it would cause a word to move to the next line, and tiny changes like that can sometimes have a ripple effect causing major layout issues. (This is a bigger issue for Paizo than most publishers, as we don't generally leave much free space in our layouts—our editing team does a thorough copyfit pass after layout, adding and removing text as needed to ensure that paragraphs, sections, columns, and pages break exactly where we want them to.) With very few exceptions, our PDFs are designed to replicate our printed products as precisely as possible.
Anyway, the PDF format follows the same, blending both results. The appearance in the PDF will be "ADOBE", just as the font shown will mimic what was seen in InDesign because it gets embedded in the file (if necessary). Thing is, the actual text is "adobe", and that's exactly what you're going to get if you copy/paste from the PDF.
I can guarantee that that's not the case. I just grabbed a random PDF that was sitting on my desktop (it happens to be Chapter 3 of the Advanced Player's Guide, First Printing). I opened it in Adobe Reader, copied the first all-caps header I spotted in it, TYPES OF FEATS, and here's what I get when I paste it: Types of Feats. Next, I opened the same PDF in Apple Preview, and copied the same text. Pasting, I get this: Types of feaTs.
I don't have access to the original InDesign file at the moment, but I would expect the original text to be "Types of Feats". Which would mean that Reader got it right, but not because it's stored that way (it isn't); it got it right because Reader is just smarter at reverse-engineering the glyph-to-character mapping than Preview is. (And there is a non-zero chance that the original text is "TYPES OF FEATS", in which case they both got it wrong.)
| 1 person marked this as a favorite. |
Thanks for explaining everything, Vic. It now all makes sense to me.
|
Chief Technical Officer
|
| 1 person marked this as a favorite. |
Let me give you another example that directly ties to the problem in the original post. I'm looking at the PDF of the Pathfinder Chronicles Campaign Setting (yeah, I have a bunch of really old PDFs on my desktop), and on page 133, there's a section about Flotsam & Jetsam, Inc. In that section header, "Fl" is a ligature, and when I copy and paste the text from Adobe Reader, I get this: Fl otsam & Jetsam, Inc.. When I search for the word "flotsam", this does not appear in the results. But if I search instead for "fl otsam", I *do* get the result. The problem is not that the ligature exists—it's that Adobe Reader is interpreting the ligature incorrectly, determining that the kerning following the ligature is a space when it's not. This is a bug in the app. (My argument, though, is that if Adobe would just store what the text is in addition to what the glyphs are, there would be no opportunity for interpretation, so this bug could not exist.)
Apple's Preview app interprets this one in a far worse state: fLoTsA m & JeTsA m , inC.. Interestingly, it means that if I search for "flot" in Preview, I get this result, even though I wouldn't get that result for the same search in Adobe Reader. (The fact that different apps can give different search results just goes to underline the fact that there are bugs in each app's interpretation... and that the real problem is that the PDF format needs this kind of interpretation at all.)
I also want to make it clear that this really isn't about ligatures. Even if we stopped using ligatures, these interpreters have similar problems with kerned text, with drop caps, with all-caps fonts, and probably a bunch of other things I haven't noticed yet, all of which would go away if the PDF format just stored the actual text properly.
But a takeaway is that if you search using a different PDF reader, you will probably get different results. And maybe a better takeaway is that you might get even better search results by searching for parts of the word that avoid ligatures, and (though I didn't go into detail on the drop cap thing) maybe even avoid initial letters. So instead of searching for "Janderhoff", try searching for "anderho".
Some interesting search results on this topic using my Mac:
Search term: Janderhoff
Results:
Finder: 63 documents
Adobe Acrobat Reader: 66 documents
Search term: Janderho (without the ligature)
Results:
Finder: 64 documents
Adobe Acrobat Reader: 66 documents
Search term: anderho (no "J" or ligature)
Results:
Finder: 0 (finder can't search partials
Adoble Acrobat Reader: 68
I can guarantee that that's not the case. I just grabbed a random PDF that was sitting on my desktop (it happens to be Chapter 3 of the Advanced Player's Guide, First Printing). I opened it in Adobe Reader, copied the first all-caps header I spotted in it, TYPES OF FEATS, and here's what I get when I paste it: Types of Feats. Next, I opened the same PDF in Apple Preview, and copied the same text. Pasting, I get this: Types of feaTs.
Adobe might be inclined to lie about itself. What happens when you paste it into Wordpad, or Microsoft office?
I can certainly believe InDesign could, but does not, embed regular text while expressing the ligature in the appearance of the text. I have seen this behavior in some PDFs, and also with the really bad OCR of some of my older PDF purchases, like the Rules Cyclopedia. Standard or embedded fonts should present no challenges for text copy-and-paste.
Some of the PDFs I buy are beautiful about having clean text, and some are just a mess, barely copy and pastable at all.
|
Chief Technical Officer
|
| 1 person marked this as a favorite. |
I'm curious about whether Paizo outputs tagged PDFs, which should provide alternative text for ligatures and hyphens that accessibility devices like screen readers (and, coincidentally, indexing for PDF search) can better handle. Not all PDF readers will take advantage of tagged content, but it should work around issues like this at least in Adobe Acrobat/Reader.
It seems pretty clear that we were not doing that, at least with the older PDFs I was looking at; I'll find out if we have changed that, and if not, if there's a good reason behind it.
I'd also be curious about whether something's adding special characters near or between ligature characters, such as non-joiners or fractional spaces, in the input format or InDesign.
I doubt it. The guy who wrote the article you linked to describes the whole problem really well: "The result is that Acrobat (or Reader, or whatever PDF viewer you use) is simply placing letters or words or lines on a page, without any understanding of how they fit together."
In the case of those ligatures, I'm pretty sure I know why they're getting it wrong:
Kerning is the adjustment of spacing around characters. When two characters are kerned too tightly, they run together, and when they are kerned too loosely, the spacing looks awkward—worst case, the gap between letters can be mistaken for a space. In either case, the text is less attractive and harder to read. Each individual font includes information about how much space should be placed in between every possible different pairing of glyphs in that font. (This can be overridden within the page layout program, but if the font is designed well, the kerning information it contains will allow the page layout program to get it right automatically the vast majority of the time.)
Oversimplifying a bit, when the page is turned into a PDF, the word "Flotsam" is created by placing the "Fl" ligature at a certain spot, then at another spot (driven by the kerning information), the "o" character should appear, and so on. Remember—the PDF is not storing the word Flotsam, it's storing the locations where the glyphs should appear, and that's it. So when you copy the text out of the PDF, the PDF reader's interpreter is trying to figure out what that word is, and in the example I gave above, it appears that Reader is assuming that the kerning following the "Fl" is big enough to be a space. Apple's reader is using a different algorithm, though, and it decided that gap isn't big enough to be a space... but then it made the opposite decision between the "a" and the "m", so neither interpreter got the word right. It's not about the insertion of special characters—it's just about reverse-engineering the kerning and not getting the answer right.
Here's the line I draw: if you can copy-paste the text into a word processor and get the right answer, then it's working as it should. If you can't, it's bugged.
It's not really a bug as much as a missing feature. As Vic said earlier, the PDF format was never really designed for this use case.
And/But there's no alternative to the PDF format, so we are stuck with it.
(At least none that I know of, or that is at least somewhat common.)
It's not really a bug as much as a missing feature. As Vic said earlier, the PDF format was never really designed for this use case.
Well, the Internet wasn't invented to swap cat pictures, either. :)
Also, I'll just note that I've been able to use free design software and mostly avoid this issue. So either Adobe is not keeping up with that standards set by others, or they are operating on a hidden interest.
|
Chief Technical Officer
|
I'm curious about whether Paizo outputs tagged PDFs, which should provide alternative text for ligatures and hyphens that accessibility devices like screen readers (and, coincidentally, indexing for PDF search) can better handle. Not all PDF readers will take advantage of tagged content, but it should work around issues like this at least in Adobe Acrobat/Reader.It seems pretty clear that we were not doing that, at least with the older PDFs I was looking at; I'll find out if we have changed that, and if not, if there's a good reason behind it.
Chris Lambertz, who has created most of our PDFs since about 2011, says "our PDF export settings have had 'tagged' checked for as far back as I can recall." So it sounds like that's not the miracle workaround we're looking for.
Chris Lambertz, who has created most of our PDFs since about 2011, says "our PDF export settings have had 'tagged' checked for as far back as I can recall." So it sounds like that's not the miracle workaround we're looking for.I'm curious about whether Paizo outputs tagged PDFs, which should provide alternative text for ligatures and hyphens that accessibility devices like screen readers (and, coincidentally, indexing for PDF search) can better handle. Not all PDF readers will take advantage of tagged content, but it should work around issues like this at least in Adobe Acrobat/Reader.It seems pretty clear that we were not doing that, at least with the older PDFs I was looking at; I'll find out if we have changed that, and if not, if there's a good reason behind it.Huh. Short of decompiling the PDFs or seeing the InDesign files (my rates are reasonable!) I couldn't say what the issue is, then. I tried to reproduce it during the weekend using InDesign files from similar non-Paizo books that I've worked on and couldn't.
The only outlier I can think of is the Paizo store watermarking process. I'm not sure if or how it would affect tagged text.
I remember seeing a bug report for InDesign recently that was related to this, and here it is
but can't seem to find it anymore. Something about using InDesign's "export to PDF" function versus "printing" to PDF.Probably unrelated, but figured I'd throw it out there.
Definitely unrelated. We only generate PDFs via InDesign's export functions.
Could it maybe be related to the version of InDesign? In other words, later versions maybe output more tagging data than earlier ones, and this may account for the difference?
|
For what it's worth (which is admittedly not much), I often have the same problem with academic articles generated using LaTeX, for which there's no InDesign or other Adobe software anywhere in the equation.
Paizo PDFs are only the 3rd or 4th circle of PDF hell. Your typical LaTeX PDF file definitely counts as 9th circle. As someone who has worked on search engines that try to extract text from PDFs, I would not want to deal with Paizo PDFs, but you can if you need to. I will specifically exclude most LaTeX files from indexing though--there's usually just no hope, no light at the end of that tunnel.
| 1 person marked this as a favorite. |
I am finding this thread both useful and interesting.
(My life has clearly taken an unanticipated turn somewhere along its path.)
Could it maybe be related to the version of InDesign? In other words, later versions maybe output more tagging data than earlier ones, and this may account for the difference?We keep up-to-date on our Adobe apps these days.
Sure, but I'm guessing those older PDFs you were looking at weren't recently re-exported with a new version of InDesign, hence why the internet advice seems to not match with expected results.
For example, I opened up the Villain Codex PDF, and on page 7 there is the word "difficult." I can copy and paste this word (and search for it, indeed searching just for "ff" also produces results where the ff is a ligature) just fine, despite when zooming in I can see that the "ff" is a ligature. Furthermore, when I copy/paste some all-caps headings, some remain all-caps (such as the word "INTRODUCTION" in the table of contents) and some are Title case (such as the word "Introduction" on page 4). I'm guessing they were cased this way in the actual InDesign file as well.