Proposal: Markdown, plugged in

[Rad Geek]

Markdown should allow plugins for link references.

See: <http://www.radgeek.com/markdown/>

Why? Because it's potentially very useful, and because it can be  
implemented in Markdown without any real cost for those who don't use it.  
It allows us to be lazy by writing convenient plugins that automate the  
translation from human-readable references to URIs (for example, by  
running over a list of commonly-used 'bookmarks', or by consulting an  
InterWiki map and generating the link programmatically from the name of  
the reference). It enhances something that Markdown does; and it does so  
without requiring the internals of Markdown itself to be kludged. It lets  
us be more lazy about irrelevant computer details, and it lets Markdown be  
Markdown. And that's a good thing.

I've implemented a simple proof of concept, based on PHP Markdown 1.0b5.  
I'm more comfortable with PHP than I am with Perl, so I did it in PHP; but  
the implementation is very simple and should be easy to roll into  
Markdown.pl (or into any other implementation of Markdown in a language  
that supports callbacks).

Here's what it does: it allows anyone using Markdown to define a callback  
which plugs into Markdown's link reference processing. If the callback is  
defined, Markdown calls it whenever it encounters a reference-style link  
whose reference is not explicitly defined within the Markdown blurb. The  
callback either returns a link (in which case Markdown runs the link into  
the HTML) or NULL (in which case Markdown treats it as an undefined link:  
the Markdown cruft is left in-place). The plugin function can generate the  
link however it pleases; Markdown doesn't need to care. (My implementation  
demonstrates the use of a simple bookmark file, a crude function for  
extracting the source of an article that is referenced within a post, and  
the use of InterMap to generate links into Wikis.)

See <http://www.radgeek.com/markdown/> for a dingus demonstrating the  
implementation. The technical details of the plugin interface are  
explained in the PHP source itself, a text/plain serving of which is  
linked from the Dingus.

Let me know what you think. It would be nice to see something *like* this  
in future versions of Markdown--although of course the exact details of  
the implementation are all up for grabs, and of course any decision is up  
to those who actually do the work of maintaining those future versions.  
For those who enjoy such things, I've included a much more longwinded  
explanation of why I think this is a good idea, why it's a better idea  
than the alternatives, and why it's natural for being incorporated into  
Markdown (this is otherwise known as: a lot of talk over a very simple  
thing). That follows below.

-RG

Markdown should allow plugins for link references. It should be pretty  
clear why this would be useful; but it might also not be clear why it's  
better than the alternatives. So let's stop for a moment and go over some  
background points.

Point 1.
--------
Markdown is for "posting": it exists in order to make writing blobs of  
content for the web *quick* and *natural*. It makes our lives easier  
because it eliminates a lot of the repetitive work involved in  
hand-writing blobs of raw (X)HTML. Computers are better than humans for  
doing monotonous, repetitive work; Markdown shifts more of that work from  
the human to the computer. Markdown does the grunt-work for us and lets us  
keep our mind on writing. That's why Markdown is a good thing.

Point 2a.
---------
I write on a weblog; I expect that most of you do, too. A lot of my posts  
link back to a source that I link to on a regular basis (other weblogs,  
Salon, the New York Times, Slashdot...). In practice this means that a  
*significant* percentage of the posts I write repeat at least one or more  
of the following references, more or less verbatim, at the bottom of the  
post:

     [Alas, A Blog]: http://amptoons.poliblog.com/blog/
     [Dive Into Mark]: http://www.diveintomark.org/
     [New York Times]: http://www.nytimes.com/
     [Salon]: http://www.salon.com/
     [Slashdot]: http://www.slashdot.com/ "/.: News for nerds, stuff that  
matters"

... and so on.

None of this is hard work. But it *is* work; I need to break out of what  
I'm doing and either punch out a memorized URI or else spend a couple  
seconds looking up the URI in a fresh tab. (If I don't break out at, say,  
the end of a paragraph, I'm liable to forget to put the reference in at  
the end at all.) I shouldn't have to do this. Why? Because my computer  
could easily do it for me. Mapping from human-readable titles to URIs is  
cognitive grunt-work that every web browser in the world does for you with  
bookmarks. Markdown is good because it lets me be lazy about irrelevant  
computer grunt-work. URIs are irrelevant computer grunt-work. So Markdown  
should let me be lazy about URIs too.

Of course, we shouldn't expect Markdown to be all things to all people;  
Markdown should be Markdown, and should do a limited number of things  
well. But one of the things that Markdown does is offer links based on  
human-readable references, rather than URIs; this just makes the reference  
system easier to use.

Point 2b.
---------
Besides weblog posts, one of the obvious potential applications for  
Markdown syntax is to drop it into a Wiki package for use as  
WikiFormatting. But any WikiFormatting worth its salt needs to be able to  
link to other entries in the Wiki without requiring the user to type any  
URI cruft. It's pretty clear how you could map this behavior into Markdown  
syntax without too much pain: that is, by treating references that aren't  
explicitly defined elsewhere within the Markdown blob as intra-Wiki links  
(or, if they're qualified by a namespace, as inter-Wiki links). Wiki  
packages allow you to programmatically generate the URI of a given entry  
within the Wiki; but Markdown has no way of knowing about that.

Point 3.
--------
Of course, the problem with (2a), (2b), and any related application is  
that they're tricky to generalize. Markdown.pl (or markdown.php) are  
single files that do a simple thing very well. They shouldn't have to  
undergo the kludging needed to make Markdown aware of where other files  
are or how to generate Wiki URIs. One thing you might be inclined to  
suggest is just to write a second filter to run right after Markdown: if a  
link has an undefined reference, Markdown just leaves it in the Markdown  
format, just as it was. It wouldn't be hard to write a second text filter  
to scrape the Markdown output and do whatever should be done to the links  
that remain.

But I don't think that's a good solution, for a couple of reasons. One of  
the reasons is purely ideological, and the other is purely practical, but  
they do have a certain conceptual connexion between them. The ideological  
reason for concern is this: the "references" on links are conceptual  
elements *of Markdown*; `[this][]` and `[that][this]` are only links to  
the resource referenced as "this" from within Markdown's syntax and  
semantics. So if a text filter is processing the references on Markdown  
links, it should be (in some sense) from *within* Markdown, not as a  
filter run over Markdown's output (which is in (X)HTML, not in Markdown).

The practical reason is this: Markdown's syntax might *change*. In  
particular, the link syntax might change. (We have, for example, discussed  
the merits of `[[this]]` or `[this]` as a synonym for `[this][]`.) If  
Markdown's link syntax changes then the post-processing filter would  
*also* have to change to keep consistent with Markdown. That's because the  
post-processing solution doesn't operate from *within* Markdown. If it  
operated from within Markdown, then *Markdown* could tell it what marks up  
a reference, rather than making the author keep up with those details at  
each new release.

Of course, swapping out one regex for another in your post-processor is  
*not* hard work at all. But it *is* work; it's monotonous and repetitive  
work that Markdown could do for you. Indeed, it's work that Markdown  
*should* do for you. The question is how you do that without kludging  
Markdown to include a bunch of extra stuff that doesn't generalize well  
 from one setting to the other.

A Solution.
-----------
A solution to the difficulty may be blindingly obvious. The software that  
should know how to generate links into a Wiki is the Wiki software; the  
software that should know what links you commonly use on your weblog is  
your weblog software. So why not create an interface that lets Markdown  
identify the references, and then passes them on to the software that it's  
dropped into for figuring out what to do with them?

That means a plug-in for the plug-in. In practice, all we have to do is  
define a callback function that is set before we call the Markdown()  
filter.

Of course, Markdown includes a syntax for *explicit* references. We  
shouldn't interfere with Markdown syntax; we should be able to extend  
Markdown references but not to break them. (In any case, if a user  
*explicitly* defines the URI for a reference within the Markdown blob, who  
is Markdown, or the weblog/Wiki/etc. software, to say otherwise?)  
Soexplicit references should take precedence; so what we'll do is change  
the anchor-parsing subroutines within Markdown to *fall back* to the  
plugged-in function if, and only if, Markdown can't find the reference on  
its own. The plugged-in function returns a link, in which case Markdown  
runs the link into the HTML output; or else it returns failure, in which  
case Markdown falls back to its own default: it leaves the Markdown cruft  
in place, without any modification.

If no plug-in function is defined, Markdown just falls straight into its  
own defaults, and goes on its merry way.

If we do it this way, Markdown can easily be dropped in to many more web  
writing environments than it can now. We can do less work in dealing with  
irrelevant computer stuff; we can concentrate more work on writing. With a  
defined interface, people can easily swap popular and/or useful modules  
for extending Markdown's references, and new versions of Markdown won't  
break older modules being used with them, even if the syntax changes  
substantially, as long as the internal interface stays the same. Markdown  
doesn't have to be kludged to deal with non-generalizable stuff; we don't  
need to generalize, because plug-in functions specify what needs to be  
specified for a given environment. Any hacking that needs to be done is  
limited to simple plugin functions, and hacking simple plugin functions is  
better than hacking Markdown itself.

The reference system is extended to do an even better job at what,  
conceptually, it's supposed to do. Markdown stays Markdown; we all get to  
be lazier. And that's a good thing.

At least, so it seems to me.

Comments?

-- 
Charles Johnson <technophilia at radgeek.com>
AIM: AiPuch
WWW: http://www.radgeek.com/

Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/