PDF is an entirely different beast than images, it even has its own scripting elements for interactive stuff. I get that you only want a basic viewer that wouldn't support any of that stuff, but I'm not sure even that will fit within the memory and processing power constraints of the Prizm. I'm having a hard time finding any libraries, even commercial ones, tailored for embedded systems so we could have an idea of the minimum memory requirements.
Your best would be
http://mupdf.com/ , that's what was used in a viewer for the Nspire series. However, the Nspires have over ten times the RAM we have available on the Prizm, and generally their CPUs are faster, too. You would probably also end up with a giant add-in that takes up a quarter of the Prizm's storage memory. The font rendering libraries alone will be heavy, but if I remember correctly, ProgrammerNerd already ported one, so that part would not be the hardest one.
The only way I could see this being done would be by developing a reader from scratch, with the intricacies of the Prizm's memory model in mind, but usually when you reduce memory usage, you trade it for more computing time (see also: paging is slow/unfeasible, etc.).
Personally I'm willing to write a PDF viewer off as technically impossible - but then again, I'm not too knowledgeable of the format in question, so I may be in the wrong here. Of course, tricks like rendering on a computer and transferring the result over the Prizm would defeat the point and be pretty much useless