Show HN: I am building a new Python library to read/write PDF files (opens in new tab)

In the README for that repository it mentions "schizophrenic files". What is a schizophrenic file, out of interest?

"The ALAN Parsers Project"

Bravo! Best wordplay I've reaed today.

The PDF format is diverse enough for such a new project to still have plenty of incompatibilities. If you wanted to find many of them quickly, you might want to have a look at the documents used as test cases in other projects, such as pdf.js:

https://github.com/mozilla/pdf.js/tree/master/test/pdfs

This is tangential to your submission, but PDF is the file format I use for exercising any library that claims to be a declarative file format (ala https://github.com/kaitai-io/kaitai_struct_formats#readme )

J is for complex numbers. While math.sqrt(-1) raises an exception, cmath.sqrt(-1) returns 1j.

There is not distinction between values and references in python, everything is a reference. In fact, primitive like numbers are big struct objects in cpython, you cannot just manipulate the raw numbers.

The difference will rather wether you can modify an object or not. You cannot modify numbers, as they are immutable. Any increment will produce a new object. But you can modify a list. This gives the feeling numbers are passed as values and list are passed as references.

  Commiting a change (from Jun 5 20:57) that I don't understand any more 

     // From real life, lightly modified. Note the "/companyName, LLC" as key!

With absolutely no slight toward the author, that matches my mental model of dealing with PDFs: `git commit -mwtf`

QPDF is a good C++ library for "content preserving" PDF transformations, and is used by the Python PikePDF library.

I had the same experience. Thanks for the summary. Need to read that the next time.

Merely for your consideration, if those were actual issues on that repo, (a) it would allow adding labels to them (as in https://github.com/pdf-association/pdf-issues/issues?q=is%3A... ) (b) folks could comment, acting as a low-rent stackoverflow, and (c) it would allow anyone to contribute new ones versus the "PR against README.md" situation right now

That also more closely matches the mental model of those items: bugs against the specification, whether the official PDF Association agrees that they are or not

Latex seems way better than either to me (but then, I know Latex). Certainly Latex makes it much easier to get consistency and precision. For your use-case, generate Latex code from a template using Python or whatever, substituting in what you need, then compile the Latex into a PDF. If you need graphical elements or precise layout control, use the Latex package TikZ.

If what you need is very simple (e.g. no word wrapping, same number of variable strings in the same positions), even manipulating the code of a template PDF directly is not too hard. This library would help with that.

> the most reliable way of generating PDFs

See pandoc: https://pandoc.org/

And a variety of intermediate or input text formats, where you can pick your preferred poison whether for book publishing, research papers, math papers, technical documentation, slides, etc.

Including the author's own djot: https://github.com/jgm/djot

EDIT:

Sibling reply suggests latex. OK, but then you're also learning latex.

> find a version of iText RUPS application from somewhere on the internet

You mean this, right? https://github.com/itext/i7j-rups#readme

I once had to do this and turned all the docx into one document, used Word to export as PDF and then used a PDF splitter to get separate documents.

You could script Libre Office to do this.

>No open source tool does a proper conversion //

Presumably the key word here is "proper" because LibreOffice, etc., read docx and write pdf. For example, `libreoffice --headless --convert-to pdf myfile.docx`.

I have used Perl and Win32::OLE for this kind of job.

Converting to PDF is actually quite easy. Before Office 2010, you had to print to Postscript and then convert to PDF using Ghost. Nowadays Word gives you the option of saving to PDF.

Yeah, this would be horrible, but on flip side relatively easy to do with Word to hand.

it's such a complex messy format that i'm really not surprised.

ilovepdf.com is free but isn't open source.

A PDF generator library that only generated guarenteed PDF/A compliant PDFs would actually be really good selling point for a new PDF library.

As I'm currently fighting with css3/paged media[pm] - I've recently tried to figure it out.

Theres a rather comprehensive list at: https://www.print-css.rocks/tools

As far as Foss tools go, I've only found paged.js (a polyfyll) in combination with a browser print-to-pdf (eg wkhtmltoodf (webkit) or puppeteer (chrome)) that has any semblance of css support.

There's also ghostscript - but AFAIK it doesn't support much/any css3 for print.

[pm] https://www.w3.org/TR/css-page-3/

"list" is probably harsh, but I've had good luck trawling through the GitHub topics to find such things: https://github.com/search?p=1&q=topic%3Apdf&type=Repositorie...

Makes me miss freshmeat.net which would have been my answer a few years ago (freshcode.club just isn't the same, although bless them for trying)

The best generator is Prince XML but it can be expensive.

In python I would choose WeasyPrint most of the time these days.

Dompdf is a good html to pdf library for PHP.

It's actually not so bad: it's mostly ASCII, even though some parts of it really need to be treated as binary. If you open up a simple/old PDF in your favorite text editor, you can begin to grok the basic structures quite easy.

One trick for getting started: PDFs are read from the bottom. The first thing that is read is actually an offset pointing back to the xref table, at the end of the file. Then, the xref table itself points to the latest version of all of the objects.

The part you're most likely interested in is the content streams, which contain postscript-like drawing commands. To get a feel for it, following the official spec when reading a simple-looking document can help.

edit: I didn't link any actually useful resources, in part because I actually just have a corpus of files in various file formats that I keep handy as a reference for some weird reason. However, Googling for simple PDF files yielded this, which I feel is very readable in a text editor. https://www.africau.edu/images/default/sample.pdf

The PDF spec is quite readable. Use the “Adobe equivalent” of PDF 1.7 here: https://www.pdfa.org/resource/pdf-specification-index/

ReportLab can render gradients, but it's poorly documented. See eg https://stackoverflow.com/questions/452074/creating-a-gradie...

I use the method with `canvas.clipPath(path, stroke=False, fill=True)` on a path I've parsed manually from SVG then `canvas.linearGradient`.

This issue?

https://github.com/Kozea/WeasyPrint/issues/1719

I recently tried pdfplumber [1] to extract tables from (relatively) difficult formatted tables in PDF, and it was a great experience. I can recommend it. Before I ended up using pdfplumber, I tried at least three other PDF packages and they did not work as easily or as expected.

[1]: https://github.com/jsvine/pdfplumber

Fun project story... During the first covid school shutdown my son's day care wanted parents to print a daily screening symptom checklist, take a photo of it and email it to them every morning. This was a tedious process that I automated with PyPDF2 + PDFtk + pypdftk. It's easy to generate your own PDF's but it's harder to take an existing, outdated, non-editable PDF and automatically fill it out.

Eventually I turned it into a website, added AWS API Gateway + Lambda and put the whole thing up for other daycare parents to use. Two weeks later the daycare switched to google forms and my project was not useful anymore.

Table elements not being in consecutive display order in any direction. Like elements 0-5 from column a, followed by elements 10-12 from column b, elements 6,8.19 from column a, then elements 19-4 in reverse order from column d, all of column c....

This is a real thing I dealt with.

I asked the vendor if they had a 3d model viewer, so we could inspect a part they were making. He sent me a PDF. Full pan, tilt, zoom, and hideable pieces. I don't know what kind of witchcraft was involved, but I suspect OP will come out of this cursed by its unholy nature.

My understanding is that this is largely because you're fighting an adversarial format provider in Adobe, I've read a few papers and journal entries on file format polyglotting, with some focus on PDF and approaches are constantly shifting in nature due to Adobe mooting pathways to success, I think it's partly for security and also IMO partly for obscurity as PDF is a horrific format in all reality except for human visual interpretation. Many organisations and industries ONLY produce public data via PDF and it makes the parsing of that information a far more difficult task (again, I suspect by design). After trying a few parsing options for PDF, I've come to hate it as a format. Luckily though, some options from a few cloud providers seem to be really hammering the problems complexity down, but the cost of the solutions can be very steep.

I think you might mean PyMuPDF (https://github.com/pymupdf/PyMuPDF), a Python library built on top of the MuPDF C library (https://mupdf.com/).

PyMuPDF and MuPDF are both available under dual open source AGPL and commercial licenses. They have been around for many years and are under continual development.

[Disclaimer, i work for Artifex, who wrote MuPDF and recently acquired PyMuPDF.]

Curious to hear what you hate about it? Any specific point? If it’s just the code age then keep in mind the main author is Robin is celebrating his 75th birthday today. Like most early Python pioneers he comes from lisp background and that’s pretty much the same style of code you see by Peter Norvig and other of his generation.

There is a command line utility (pdf2text) that will also parse the pdf to an XML tree and you can query with XPaths. I found it works well.

https://pdfminersix.readthedocs.io/en/latest/reference/comma...

This is great, thank you for posting it.

Despite the thousands of pages of ISO 32000, the reality is that the format is not defined. Acrobat tolerates unfathomably malformed PDF files generated by old software that predates the opening-up of the standard when people were reverse-engineering it. There’s always some utterly insane file that Acrobat opens just fine and now you get to play the game of figuring out how Acrobat repaired it.

Plus all the fun of the fact that you can embed the following formats inside a PDF:

PNG, JPEG (including CMYK), JPEG 2000 (dead), JBIG2 (dead), CCIT G4 (dead, fax machines), PostScript Type1 fonts (dead), PostScript Type3 fonts (dead), PostScript CIDFonts (pre-Unicode, dead), CFF fonts (the inside of an OTF), TrueType fonts, ICC Profiles, PostScript functions defining Color spaces, XML forms (the worst), LZ compressed data, Run-length compressed data, Deflate-compressed data.

All of which Acrobat will allow to be malformed in various non-standard ways so you need to write your own parsers.

Note the lack of OpenType fonts, also lack of proper Unicode!

Others have given some good answers already, but I'll add one more: PDF is all about creating a final layout, but given a final layout, there are an infinite number of inputs that could have produced it, but if you are parsing the PDF, most of the time you are trying to get back something higher level than e.g. a command to draw a single character at some X,Y location. But many PDFs in the wild were not generated with that type of semantic extraction in mind, so you have to sort of fudge things to get what you want, and that's where it becomes complex and crazy.

For example, I once had to try to parse PDF invoices generated by some legacy system, and at the bottom was a line that read something like, "Total: $32.56". But in the PDF there was an instruction to write out the string "Total:" and a separate one to write out the amount string, but there was nothing in the PDF itself that correlated the two in any way at all (they didn't appear anywhere close to either other in the page's hierarchy, they weren't at a fixed set of coordinates, etc, etc.).

1. PDF is mostly designed as a write-only (or render-only) format. PDF’s original purpose was as a device-independent page output language for printers, because PostScript documents at the time were specific to the targeted printer model. Interpreting a PDF for any other purpose than rendering resembles the task of disassembling machine code back into intelligible source code.

2. Many PDF documents do not conform to the PDF specification in a multitude of ways, yet Adobe Acrobat Reader still accepts them, and so PDF parsers have to implement a lot of kludgy logic in an attempt to replicate Adobe’s behavior.

3. The format has grown to be quite complex, with a lot of features added over the years. Implementing a parser even for spec-compliant PDFs is a decidedly nontrivial effort.

So PDF is a reasonably good output format for fixed-layout pages for display and especially for print, but a really bad input format.

My current company uses ML to parse PDF invoices and identify fraud. I have no idea how the devs manage this black magic wizardry because they also spend time contributing to infra code before they hired more people like me on board. If anyone wants a great startup idea, look to solving a problem involving parsing PDFs en masse. Maybe something in legal tech. That market is absolutely ripe for disruption.

PDF has always seemed to be a janky Adobe product.

Should a modern, open version of PDF be created knowing that how it evolved from the original concept in 1991? Shouldn't we at some point say, we need to start over and created PDF2?

It's not that bad, it's just that the problem is big enough in scope to get right that the state of the art is provided by private industry, and to a lesser extent some open source tools, and you're probably way better off joining them rather than trying to beat them unless you want to grind your brain into the dust for a page layout spec.

The PDF format is itself a weird hybrid of text and binary.

(I have written a PDF parser myself.)