KPhotoAlbum has lots of built-in features, but in practice the more convenient¹ way to interface with it from simple unix tools is to just operate on the index.xml where all of the metadata is stored.²

kpa-grep

I started kpa-grep back in 2011, around when I hit 90k pictures (I'm over 200k now.) The originally documented use case was kpa-grep --since "last week" --tags "office" which was probably for sorting work pictures out from personal ones. (The fuzzy timedateparser use was there from day one; since then, I'm not sure I've used anything other than "last week" or "last month", especially since I never implemented date ranges.) I've worked on it in bursts; usually there's feedback between trying to do something with a sub-gallery, trying to script it, and then enhancing kpa-grep to handle it. The most recent burst added two features, primarily inspired by the tooling around my Ice Cream Blog -

• A sqlite-based cache of the XML file. Back in the day it took 6-8 seconds to parse the file, on a modern laptop with SSD and All The RAM it's more like 2.5 seconds - the sqlite processing takes a little longer than that but subsequent queries are near-instant, which makes it sensible to loop over kpa-grep output and do more kpa-grep processing on it. A typical "pictures are ready, create a dummy review post for the last ice cream shop with all pictures and some metadata" operation was over a minute without the cache, and is now typically 5-10 seconds even with a stale cache.
• Better tag support - mostly fleshing out unimplemented combinations of options, but in particular allowing --tag and --since to filter --dump-tags, which let me pick out the most recent Locations which are tagged ice cream, filter out city names, and have a short list of ice cream shops to work with. (Coming soon: adding some explicit checks of them against which shops I've actually reviewed already.)

As far as I know I don't have any users, but nonetheless it is on github, so I've put some effort into keeping it clean³; recently that's also included coming up with a low-effort workflow for doing releases and release artifacts. This is currently a shell script involving debspawn build, dpkg-parsechangelog, and gh release upload which feels like an acceptable amount of effort for a single program with a man page.

pojkar

pojkar is a collection of Flickr upload tools that work off of KPhotoAlbum.⁴ The currently active tools are sync-to-flickr and auto-cropr.

sync-to-flickr

sync-to-flickr is the engine behind a simple workflow: when I'm reviewing photos in KPhotoAlbum, I choose particular images for posting by adding the Keyword tag flickr to the image. Once I've completed a set and quit out of KPhotoAlbum, I run sync-to-flickr sync which looks for everything tagged flickr, uploads it to Flickr with a title, description, and rotation (and possibly map coordinates, except there are none of those in the current gallery.) There's also a retry mechanism (both flickr's network and mine have improved in the last decade so this rarely triggers.) Once a picture has been uploaded, a flickd tag is added to it, so future runs know to skip it.

After all of that, the app collects up the tags for the posted set of pictures; since social media posting⁵ has length limits (and since humans are reading the list) we favor longer names and names that appear more often in the set; then we drop tags that are substrings of other tags (dropping Concord in favor of Concord Conservation Land since the latter implies the former well enough.) Finally we truncate the list to fit in a post.

auto-cropr

Flickr has an obscure⁶ feature where you could select a rectangle on a picture (in the web interface) and add a "note" to that region. auto-cropr used the API to look for recent instances of that which contained a magic string - then picked up the geometry of the rectangle and cropped just that area, posting it as a new flickr picture - and then cross linking them, replacing the original comment with a link to the new image. Basically this let you draw the viewer's attention to a particular area and then let them click to zoom in on it and get more commentary as well as a "closeup".

Note that these "views" are only on Flickr, I don't download or back them up at all (I should fix that.)

fix-kpa-missing/kpa-insert

As part of the Nokia 6630 image fixing project there ended up being a couple of different cleanups which I needed to review carefully, so I wanted the tools to produce diffable changes, which lxml doesn't really guarantee⁷. Currently, the XML written out by KPhotoAlbum is pretty structured - in particular, any image with no tags is a one-line <image ... /> and I was particularly looking to make corrections to things that were fundamentally untagged⁸/untaggable (for fix-kpa-missing) or insert lines that were already one-line-per-picture, I just had to get them in the right place.

When I started the image recovery, I ended up just adding a bunch of images with their original datestamps (from 2005), but KPhotoAlbum just added them to the end of the index (since they were "new" and I don't use the sorting features.) So I had the correct lines for each image (which checksums and dimensions), I could just chop them out of the file. Then kpa-insert takes these lines, and walks through the main index as well. For basically any line that doesn't begin with <image it just copies it through unchanged to the new index; when it finds an image line, it grabs the attributes (startDate, md5sum, and pathname specifically) and then checks them against the current head of the insertion list⁹. Basically, if the head of the index was newer than the head of the insertions, copy insertions over until that's no longer true. If they match exactly - the original version just bailed so I could look at them, then once I figured out that they really were duplicates, I changed it to output rm commands for the redundant files (and only kept the "more original" line from the original index.)

The output was a diffable replacement index that I could review, and check that the "neighbor" <image> entries made sense, and that nothing was getting added from elsewhere in the tree, and other basic "eyeball" checks. Since I had to do this review anyway to make sure I hadn't made any mistakes of intent, it made sense to write the code in a "direct but brittle" style - anything weird, just bail with a good traceback; I wouldn't even look at the diffs until the code "didn't find anything weird." That also meant that I'd done the least amount of work¹⁰ necessary to get the right result - basically a degenerate case of Test Driven Development, where there's one input (my existing index) and one test (does the new index look right.)

I also didn't have any of my usual user interface concerns - noone (not even me) was ever going to run this code after making this one change. I did keep things relatively clean with small helper functions because I expected to mine it for snippets for later problems in the same space - which I did, almost immediately.

For fix-kpa-missing, I'd noticed some "dead space" in the main KPhotoAlbum thumbnail view, and figured that it was mostly the result of an old trailcam⁸ project. I was nervous about "losing" metadata that might point me at images I should instead be trying to recover, but here was a subset that I knew really were (improperly but correctly) discarded images - wouldn't it be nice to determine that they were the only missing images and clean it up once and for all?

So, the same "only look at <image lines" code from kpa-insert, extract the pathname from the attributes, and just check if the file exists; I could look for substrings of the pathname to determine that it was a trailcam pic and was "OK", plus I could continue with the "direct but brittle" approach and check that each stanza I was removing didn't have any tags/options - but just blow up if it found them. Since it found none, I knew that

• I had definitely not (mis-)tagged any of the discarded pictures
• I didn't have to write the options-handling code at all. (I suspect I will eventually need this, but the tools that are likely to need it will have other architectural differences, so it makes sense to hold off for now.)

There were a couple of additional scripts cobbled up out of these bits:

• fix-kpa-PAlbTN which looked for Photo Album ThumbNails from the Nokia project and make sure they didn't exist anywhere else in the tree since I was discarding the ones that I had real pictures for and wanted to be sure I'd really finished up all of the related work while I still had Psion 5 code in my head...
• find-mbm which used magic.from_file to identify all of the Psion Series 5 multi-bitmap image files (expensively, until the second or third pass when I realized that I had all the evidence I needed that they only existed in _PAlbTN subdirectories, and could just edit the script to do a cheap path test first - effectively running file on a couple of hundred files instead of two hundred thousand.) This was just to generate filenames for the conversion script, it didn't do any of the work directly.

Conclusion

I now have three entirely different sets of tooling to handle index.xml that take very different approaches:

• kpa-grep uses SQL queries on a sqlite cache of the entire index (read-only, and generates it by LXML-parsing the whole file if it's out of date)
• pojkar does directly LXML parsing and rewriting (since it's used for uploads that used to be expensive, it does one parse up front and then operates on an internal tree, writing that out every time an upload succeeds for consistency/checkpointing)
• kpa-insert &c. treat the index.xml as a very structured text file - and operate efficiently but not very safely, relying on my reading the diffs to confirm that the ad-hoc tools worked correctly regardless of not being proper.

Fortunately I've done all of the data-cleaning I intend to do for now, and the kpa-grep issue list is short and mostly releng, not features. I do eventually want a full suite of "manipulate images and tags" CLI tools, and I want them to be faster than 2.5s per operation¹¹ - but I don't have a driving project that needs them yet - my photoblogging tools are already Fast Enough™.