[ANN] MDTest 1.0

[Michel Fortin]

I'm announcing today a new testsuite application for Markdown called  
MDTest. I've been using it as a replacement for John's Markdown Test  
for some time now I think it's ready for release.

First, I want to say Markdown Test has been very useful in developing  
PHP Markdown. The testsuite covers many cases, however I still found  
the need to add other tests for PHP Markdown, to cover bugs specific  
to my implementation, or simply to covers areas for which Markdown  
Test had no test case. The tests I added cover practically everything  
that was changed or fixed in PHP Markdown in the last couple of  
years; I expect that to be useful to others.

I've also made a new testsuite dedicated to PHP Markdown Extra, with  
tests for all the features. It has been very practical, and worked  
very well with Markdown Test, except for one thing.

HTML normalization is the weak point of Markdown Test. Because  
different outputs can have the same HTML meaning, Markdown Test has  
an option to compare outputs filtered through Tidy. Personally, I  
have to say that I find Tidy to be more trouble than it is worth:  
because it fixes too much the HTML garbage you send to it, it often  
fixes the strange markup a bug in Markdown can generate.

I've made some time ago a modification allowing Markdown Test to  
print a diff of the expected output whenever a test failed, and I've  
been using it since instead of Tidy because I find it more reliable  
to check the differences myself. The only problem with this approach  
is that having made a couple of architectural changes to PHP  
Markdown, especially in the HTML block parser and in the way it  
generates hashes, it often generate whitespace differently around  
tags; having changed back the list processing so that it no longer  
produce strangely-indented list items and sublists (a workaround no  
longer necessary for PHP Markdown), it forces me to check many things  
manually every time.

So I decided recently that I'd build a better tool. It's called  
MDTest, and it's written in PHP. The idea is that by using PHP I can  
leverage PHP 5's HTML parser and normalize whitespace directly from  
the DOM tree. This is what it does, and I've found it to be a much  
better alternative than Tidy (although not perfect either). MDTest  
can also show a short diff for every failed test, which may prove  
useful to double check things with normalization off. And because  
it's written in PHP, it can call the Markdown function by itself  
without the need for an intermediary script. It can still call  
Markdown.pl or any other script, like Markdown Test does.

The benchmarking info is a lot more detailed too. It shows the time  
it takes to process each test, and compiles a nice summary at the  
end. The total parse time excludes the time MDTest is running --  
reading files, creating a diff or normalizing the output. MDTest also  
point out the average, minimum, first-quartile, median, third- 
quartile, and maximum time for running tests, and it provides these  
times as a difference with the minimum time too. I think this will  
allow some cross-implementation benchmarking by mostly eliminating  
overhead time for starting a process when calling a script.

For instance, here are the benchmarks for PHP Markdown on my iBook:

                        Total   Avg.   Min.    Q1.   Med.    Q3.   Max.
     Parse Time (ms):    1356     29      2      4      7     22    554
     Diff. Min. (ms):    1237     26      0      1      4     20    551

And with Markdown.pl 1.0.2b8:

                        Total   Avg.   Min.    Q1.   Med.    Q3.   Max.
     Parse Time (ms):   11143    242    177    182    196    225   1696
     Diff. Min. (ms):    2986     64      0      5     18     48   1519

This benchmark includes time for processing three testsuites: one is  
John Gruber's Markdown testsuite, the one comming with Markdown Test,  
the two others are PHP Markdown's and PHP Markdown Extra's own  
testsuites.

MDTest works with multiple testsuites (which are simply test folders)  
and by default use all the folders it can find in its directory with  
the .mdtest extension. Having a separate testsuite for PHP Markdown  
is useful in the sense that I don't have to mix my tests with John's  
and merge the changes whenever he updates his; it's useful also  
because I don't have to wait for an update to John's testsuite to add  
my own tests, and because I can put in some extra tests if needed.  
Eventually, both could be merged I suppose. PHP Markdown Extra's  
testsuite simply covers the features of PHP Markdown Extra.

There is still a little strangeness about some test cases right now:  
PHP Markdown fails some tests from John's base Markdown testsuite,  
this because John's testsuite is expecting empty title attributes on  
images and/or links when no title has been specified (a bug I'd say).  
There's an opposite test in the PHP Markdown testsuite checking that  
the titles are indeed not present when not specified. I think both  
should be harmonised. Also PHP Markdown Extra will fail PHP  
Markdown's extensive Emphasis test because Extra doesn't accept  
underscore-emphasis in the middle of a word while regular Markdown  
does. There's a special test in the Extra testsuite for this emphasis  
deviation.

So I'm publishing today MDTest, bundled with John's Markdown Test  
testsuite (unchanged) and the two others I have made for PHP Markdown  
and PHP Markdown Extra.

In the MDTest folder, you'll find the mdtest.php script, which you  
can run on the command line. It works pretty much like Markdown Test,  
except it takes one-letter arguments (because of an implementation  
detail). The normalization feature requires PHP 5, so Mac OS X's  
default PHP install won't work for that, unfortunately. Here's the  
help page:

     MDTest Usage
     ============

     ./mdtest.php [-dnvh] [-l library_path [-f function]] [-t test_dir]*
     ./mdtest.php [-dnvh] [-s script_path] [-t test_dir]*

      Options      | Description
      -------      | -----------
      -n           | normalize HTML output before compare
      -d           | show a diff of output vs. expected output
      -l library   | PHP library to load (like markdown.php)
      -f function  | PHP function to call (Markdown)
      -s script    | script to execute (like Markdown.pl)
      -t test_dir  | testsuite directory to use
      -v           | display MDTest version
      -h           | show this help

There's also a handy index.php file which provides a simple web  
interface to the script. What's particularly nice is that it takes  
all the files in the Implementation subfolder and creates a menu to  
select what script, or PHP library, to test.

You can download it here (167 Kb):
<http://www.michelf.com/docs/projets/mdtest-1.0.zip>

(MDTest is available under the GNU General Public License.)



Michel Fortin
michel.fortin at michelf.com
http://www.michelf.com/