Creating a relationship between a buffer and a post (part 1)

In order to support multiple blogging back-ends, it is necessary that we work at some level of abstraction. One piece of blog software’s notion of tags isn’t necessarily going to line up with another’s, etc. So we introduce the notion of a post:

A post is an alist consisting of the fields:

:blog (#+POST_BLOG)
A string naming an entry in org-blog-alist
:category (#+POST_CATEGORY)
A list of strings naming categories to which the post belongs
:content (body after export)
A string containing HTML-formatted content
:date (#+DATE)
A date and time for the post
:excerpt (#+DESCRIPTION)
A string containing an optional excerpt of the post
:id (#+POST_ID)
A string containing a unique ID (generally numeric) for the post
:link (#+POST_LINK)
A string containing a link to the permanent location of the post
:name (#+POST_NAME)
A string containing the canonical name for the post
:parent (#+POST_PARENT)
A string containing a unique ID (generally numeric) for the parent of the post
:status (#+POST_STATUS)
A string denoting the status (`draft’, `published’) of the post
:tags (#+KEYWORDS)
A list of strings representing the names of tags
:title (#+TITLE)
A string containing the title of the post
:type (#+POST_TYPE)
A string containing an optional format for the post

It’s not absolutely essential that every field be present; parent and excerpt, for instance are pretty thoroughly optional. Some fields are really intended to be filled in by the blogging software, like id and link. One thing I did do was, whenever it seemed to make sense, I used a standard org-mode property name—so :date is derived from #+DATE, for instance. Whenever I “make up” a property name, I keep it in the #+POST_ namespace, to try and avoid collisions.

So, given a buffer, how do we get to a post? The answer is: the org-mode exporter.

Now the code I’m presenting here works with org-mode < 8.0. I’m hoping, once I’ve gotten this initial round of development all worked out, that I’ll be able to convert over to using that interface, which, based on my light reading, should be somewhat nicer to work with. We’ll probably end up with our own org-blog-post export format that will work in a fairly standard fashion. But that’s for later. For now:

(defun org-blog-buffer-extract-post ()
  "Transform a buffer into a post.

We do as little processing as possible on individual items, to
retain the maximum flexibility for further transformation."
  (save-excursion
    (save-restriction
      (let ((org-export-inbuffer-options-extra '(("POST_BLOG" :blog)
                                                 ("POST_CATEGORY" :category)
                                                 ("POST_ID" :id)
                                                 ("POST_LINK" :link)
                                                 ("POST_NAME" :name)
                                                 ("POST_PARENT" :parent)
                                                 ("POST_STATUS" :status)
                                                 ("POST_TYPE" :type)))
            (org-export-date-timestamp-format "%Y%m%dT%T%z")
            (org-export-with-preserve-breaks nil)
            (org-export-with-priority nil)
            (org-export-with-section-numbers nil)
            (org-export-with-sub-superscripts nil)
            (org-export-with-tags nil)
            (org-export-with-toc nil)
            (org-export-with-todo-keywords nil))
        (sort
         (list (cons :blog (property-trim :blog))
               (cons :category (property-split :category))
               (cons :date (let ((timestamp (property-trim :date)))
                             (when timestamp
                               (list (date-to-time timestamp)))))
               (cons :excerpt (property-trim :description))
               (cons :id (property-trim :id))
               (cons :link (property-trim :link))
               (cons :name (property-trim :name))
               (cons :parent (property-trim :parent))
               (cons :status (property-trim :status))
               (cons :tags (property-split :keywords))
               (cons :title (property-trim :title))
               (cons :type (property-trim :type))
               (cons :content (org-no-properties (condition-case nil
                                                     (org-export-as-html nil nil nil 'string t nil)
                                                   (wrong-number-of-arguments
                                                    (org-export-as-html nil nil 'string t nil))))))
         '(lambda (a b)
            (string< (car a) (car b))))))))

org-blog-buffer-extract-post starts off with what may actually be a bit of superfluous code—I know that org-export-as-html calls save-excursion, so it might not actually be necessary for us to do it. But I’d rather be safe. The same is true for the save-restriction.

We then make sure that the exporter will pick up our custom properties by adding them to org-export-inbuffer-options-extra, and we set a number of items that describe things about what the export will end up including and/or how particular items will look. In fact, these should all be override-able for an individual post by using the #+OPTIONS property—these are just the defaults that I think are sane.

Then the magic happens.

If you’re not used to a very functional style of programming, this code may be a little confusing—all the action is really happening down at the bottom of the function, where org-export-as-html is being called. In fact, if I’m truthful, I’m vaguely amazed it works at all.

See, when org-export-as-html gets run, in addition to returning the document transformed into HTML, it places a bunch of meta-data in the org-infile-property-plist. Our function property-trim is a wrapper for pulling values out of that list and removing any leading spaces:

(defun property-trim (k)
  "Get a property value trimmed of leading spaces."
  (let ((v (plist-get (org-infile-export-plist) k)))
    (when v
      (replace-regexp-in-string "^[[:space:]]+" "" v))))

We run that across most of the property items to get a good value. We also have a variant, property-split, that will split a value on commas, returning a list:

(defun property-split (k)
  "Get a property value trimmed of leading spaces and split on commas."
  (let ((v (property-trim k)))
    (when v
      (split-string v "\\( *, *\\)" t))))

This is used in possibly multi-valued fields, as for tags or categories.

If you look closely, you can see org-export-as-html getting run in order to provide the value for the :content field. But looking at the code again—and this is some of the first code I wrote—I don’t know how that is guaranteed to happen before everything else starts looking at the property list items.

Perhaps it will all become clearer (and less side-effect-y) with the new exporter.

Anyway, time to write a test or two. We’ll begin by extracting a post structure from an empty buffer:

(ert-deftest ob-test-extract-from-empty ()
  "Try extracting a post from an empty buffer."
  (with-temp-buffer
    (should (equal (org-blog-buffer-extract-post) '((:blog)
                                                    (:category)
                                                    (:content . "\n")
                                                    (:date)
                                                    (:excerpt)
                                                    (:id)
                                                    (:link)
                                                    (:name)
                                                    (:parent)
                                                    (:status)
                                                    (:tags)
                                                    (:title)
                                                    (:type))))))

As we would expect, we end up with an alist that is basically devoid of values, except for the content, which is pretty darn bare. In fact, down the road, we will probably do some more massaging of content that will change even that, but we test against what we have now.

Then we build a test that actually extracts some content. Including

(ert-deftest ob-test-extract-from-empty ()
  "Try extracting a post from an empty buffer."
  (with-temp-buffer
    (insert "\
 #+POST_BLOG: t1b
 #+POST_CATEGORY: t1c1, t1c2
 #+DATE: [2013-01-25 Fri 00:00]
 #+DESCRIPTION: t1e
 #+POST_ID: 1
 #+POST_LINK: http://example.com/
 #+POST_NAME: t1n
 #+KEYWORDS: t1k1, t1k2, t1k3
 #+TITLE: Test 1 Title
 #+POST_TYPE: post

Just a little bit of content.")
    (should (equal (org-blog-buffer-extract-post) '((:blog . "t1b")
                                                    (:category "t1c1" "t1c2")
                                                    (:content . "\n\n<p>Just a little bit of content\n</p>")
                                                    (:date (20738 4432))
                                                    (:excerpt . "t1e")
                                                    (:id . "1")
                                                    (:link . "http://example.com/")
                                                    (:name . "t1n")
                                                    (:parent)
                                                    (:status)
                                                    (:tags "t1k1" "t1k2" "t1k3")
                                                    (:title . "Test 1 Title")
                                                    (:type . "post"))))))

And that’s it for now. Next time we’ll look at the process of merging a post structure back into a buffer. Once we have our two-way transformation capability, the world is our mollusk. Well, once we have that and a little XML-RPC code.

Creating a new post

Picking up from where we left off yesterday, let’s think about what we want our work-flow to be.

In terms of how the user uses org-blog I want it to get out of the way as much as possible, so I’m going to try and keep the majority of the user’s interaction focused on two actions: starting a new post and saving it.

I have a couple of different blogs I post to, so I also want to make sure that org-blog seamlessly supports managing content for more than one blog.

If we’re not going to have to type everything that describes a blog in each time we create or save a post, we have to have that configured somewhere. So right off the bat, we need to create a place to put our config info:

(defcustom org-blog-alist nil
  "An alist for specifying blog information.

There are a number of parameters.  Some day I will enumerate
them.")

OK, the docstring is a little bit of a cop-out, but we don’t yet know what parameters will be pertinent. With a place to list the blogs we work with regularly, let’s look at creating a new post.

The first step will be to figure out what blog the user wants to write the post for. If there are no blogs configured, we can accept any name the user wants to give us. If there’s only one blog configured, we can reasonably assume that’s it. If there’s more than one, we should prompt with the available choices.

(defun org-blog-get-name (&optional post)
  "Get a name of a blog, perhaps working from a post.

If we're given a post structure, we will extract the blog name from it.
Otherwise, if there's only one entry in the `org-blog-alist', we
will use that entry by default, but will accept anything, as long
as the user confirms it, and if they don't enter anything at all,
we default to unknown."
  (or (cdr (assoc :blog post))
      (and (equal (length org-blog-alist) 1)
           (caar org-blog-alist))
      (empty-string-is-nil (completing-read
                            "Blog to post to: "
                            (mapcar 'car org-blog-alist) nil 'confirm))
      "unknown"))

You might be confused about the optional post parameter. My crystal ball tells me that we will also want this function to be able to tell us what blog is associated with an already-existing post, so we have the option of passing in a post structure that will be consulted for a :blog entry, which we will prefer to anything else. Other than that the code is pretty much what I laid out above.

There is a reference to a small supplementary function that I was a little surprised I needed—it turns out that completing-read will return the empty string if the user just hits enter. This doesn’t get us any useful information, so I wrote this short function to coerce the empty string to nil, so the or will fall through in that event:

(defun empty-string-is-nil (string)
  "Return any string except the empty string, which is coerced to nil."
  (unless (= 0 (length string))
    string))

Now to write some tests. We want to test getting the blog name in all the ways that are available. For the last two tests, where we’re testing code paths that depend on the output of completing-read, we take advantage of the el-mock library to sub in a version that returns a constant that represents what we want to hear.

(ert-deftest ob-test-get-name-from-blog ()
  "Test getting the blog name from a blog spec"
  (should (string= (org-blog-get-name '((:blog . "foo"))) "foo")))
(ert-deftest ob-test-get-name-from-alist ()
  "Test getting the blog name from the alist"
  (let ((org-blog-alist '(("bar"))))
    (should (string= (org-blog-get-name) "bar"))))
(ert-deftest ob-test-get-name-from-completing-read ()
  "Test getting the blog name from completing-read"
  (with-mock
   (stub completing-read => "baz")
   (should (string= (org-blog-get-name) "baz"))))
(ert-deftest ob-test-get-name-from-default ()
  "Test getting the blog name from default"
  (with-mock
   (stub completing-read => "")
   (should (string= (org-blog-get-name) "unknown"))))

The design and implementation of org-blog

So, after some quantity of hacking, I have produced a first, basic, working version of org-blog (currently available on the dirty branch on github).

However, its git history reflects all the twists and turns and dead-ends and unfortunate omnibus-commits I did along the way. I really don’t want to have that be the basis for future development, etc., so I’m going to start filtering things into the master branch on github in logical, digestible, clean chunks.

Having made that choice, I figured, “What the hell?”, I might as well get some blog content out of it as well.

So, for the next however long it takes, I’ll be refactoring the code as it stands into logical sets of changes, and posting articles discussing them, as well all the other issues surrounding the implementation.

My ambition, of course, is to get a post written each day, but don’t hold me to that.

So, to start us out, here’s the beginning of our mode:

(provide 'org-blog)

(define-minor-mode org-blog-mode
  "Toggle org-blog mode.

With no argument, the mode is toggled on/off.
Non-nil argument turns mode on.
Nil argument turns mode off.

Commands:

\\{org-blog-mode-map}"
  :init-value nil
  :lighter " org-blog")

This is about as minimalist a minor mode as you can get—it does nothing at all, and even uses the define-minor-mode macro to avoid doing most of the work itself. But it’s a start.

One other thing I really believe in, though, is testing. So we’re going to try and keep everything reasonably tested as we go along. So here’s the test to make sure that enabling org-blog-mode goes as expected:

(ert-deftest ob-test-enable-org-blog-mode ()
    "Test turning on the org-blog minor mode"
    (with-temp-buffer
      (org-blog-mode)
      (eq org-blog-mode 1)))

That’s it until next time.