About the Author

Chris Shiflett

Hi, I’m Chris: entrepreneur, community leader, husband, and father. I live and work in Boulder, CO.


JApacheCon

I'm at JApacheCon (ApacheCon that has been infested with Java) this week in sunny San Diego. On Sunday, Geoff Young and I gave our tutorial, Power PHP Testing, which went really well. Most of the attendees had PHP experience and no testing experience, so it was a perfect fit.

We covered testing theory as well as some practical examples using phpt, Simple-Test, PHPUnit, and (of course) Apache-Test. We have tarballs for each framework that provide everything you need (I'll link to these in another post), including a Makefile so that make test runs your test suite.

The challenge with being the authors of one of the testing frameworks is that we are obviously biased. (This is stated in the abstract.) Our approach was to try to make every testing framework shine as much as possible, because the purpose of the talk is to encourage testing in the PHP community, regardless of which tool is used. I'll probably elaborate later, but here are some of the things we did for each framework:

  • Simple-Test - Because the default reporter (reporter.php) is an HTML reporter, automated testing is difficult. (You'd have to write your own HTTP client to aggregate the results.) Luckily, Mike Lively has created a TAP reporter (tap-reporter.php), so we used this in our demonstration (and tarball).
  • PHPUnit - PHPUnit has a number of dependencies (which is less of a problem if you already have a fairly complete PEAR installation), so we bundled everything you need. We also wrote our own phpunit tool to run the tests (this is executed when you type make test), and it modifies include_path to resolve the inconsistent ways that dependencies are included.
  • phpt - Executing phpt tests is actually very straightforward, so all we did was add the test target to the Makefile to execute the proper command. We did add an assertNoUnwantedPuke target to our Makefile to clean up the mess that is created whenever tests fail.
  • Apache-Test - Because Apache-Test already supports automated testing with make test, no changes were necessary. However, we did create a tarball to demonstrate using Simple-Test (instead of test-more.php) within the Apache-Test framework.

The biggest difference between the various frameworks is how failures are handled, and this is where Apache-Test shines. With some enhancements to the TAP reporter for Simple-Test, however, I think it can be just as good. I'll probably blog about this separately after speaking with the authors of the other frameworks, because I want to make sure that we're highlighting the advantages and disadvantages of each framework fairly.

The conference has been great for the hallway conversations and the opportunity to meet and work with other speakers and attendees, but the interesting sessions are few and far between. Andrei Zmievski's recent comment, "2:00 is the next real session," sums this up nicely.

About this post

JApacheCon was posted on Tue, 13 Dec 2005. If you liked it, follow me on Twitter or share:

16 comments

1.Mike Lively said:

It was spooky reading my name on Planet PHP :P. If you had some specific suggestions for enhancements to the TAP-Reporter let me know. I may have already taken care of them. I have a more TAP compliant version then the one posted on my website. I just need to dig it up. Would have done it a while ago but I didn't know anyone cared :P.

Tue, 13 Dec 2005 at 23:42:08 GMT Link


2.Chris Shiflett said:

Hi Mike,

I can't find your contact information on your web site, but there is one feature that tops the list of desired enhancements for the TAP reporter.

Currently, the output looks something like this:

ok 1 - True assertion got True at line [x]

or:

not ok 1 - True assertion got False at line [x]

I think it would help people debug failed tests if the description of each test was based on the name of the method. Here's an example method from our test suite:

public function testBlankCredentials()

{

$return = create_user('', '');

$this->assertFalse($return);

}

As far as I can tell, the only way you can describe a test with Simple-Test is with the name of the method, so it would be nice if this test would output the following upon failure:

not ok 1 - blank credentials [line x]

This would let developers use descriptive method names to make their reports more useful.

That's the major enhancement I thought of.

Wed, 14 Dec 2005 at 18:47:09 GMT Link


3.Andrei Zmievski said:

Man, I thought it was JAvacheCon..

Wed, 14 Dec 2005 at 22:32:11 GMT Link


4.Mike Lively said:

Yah, I hadn't noticed that till now...

m@ is my e-mail address.

Wed, 14 Dec 2005 at 23:17:59 GMT Link


5.Alex said:

Hey Chris,

I've tried to find the Apache-Test project but there is only a place-holder site. Can you give us a link for a download?

I'm one of the core-devs of the Joomla! project and we are actually creating a Joomla! testing framework based on one of these basic frameworks.

At the moment we use Mikes simple-test and it works quite well. Looking to other ideas is always a good idea that's why I would like to test the Apache-Test.

Thx

Alex

Thu, 15 Dec 2005 at 10:26:58 GMT Link


6.Geoffrey Young said:

simple-test's tap reporter has lots of potential, but I think to make it truly awesome it ought to mirror the output of Test::More just like test-more.php does.

really, nobody would be using the tap reporter outside of Test::Harness, so going the extra mile to match that output format (odd spacing and all) would have lots of extra benefits, especially in Apache-Test - users could seamlessly switch between Test::More in t/foo.t and test-more.php or tap reporter in t/foo.php or t/response/TestBar/foo.php and the output would be identical! really, that would be a huge win (besides the fact that Test::More and clones have far better diagnostic output than anything else).

but maybe such verbose output, like the line number of the error, and expected and received values, aren't possible because you're limited by what simple-test provides to the reporter? hmm...

Thu, 15 Dec 2005 at 15:24:21 GMT Link


7.Chris Shiflett said:

Hi Alex,

Yeah, the project just changed hands (from the httpd project to the mod_perl project), so the site needs some content.

I offer a demo that might help a bit more as well as a skeleton that's easier to start with when using it for your own project (these are old):

http://shiflett.org/apache-test-demo.tar.gz

http://shiflett.org/apache-test-skeleton.tar.gz

Geoff has the tarballs of our talk up:

http://people.apache.org/~geoff/power-php-testing/

I think we've made a few tweaks since posting those, but they should give you the basic idea behind each of the projects.

Thu, 15 Dec 2005 at 15:24:38 GMT Link


8.Marcus baker said:

Hi.

Just to correct one minor point, The HtmlReporter in SimpleTest is not the default. This is because there is no default :). It's deliberately always decided at the top level test script exactly so that you can replace it with whatever you need.

Such as a TAP reporter :).

Also, I'd like to know what you wrote about testing practices? This is a pretty fast moving topic right now. We've had mock objects (essential, but you've never mentioned them) and tools like JBehave following on from that work. And then there are tools like FIT and Selenium, which can (with some difficulty) be made to work with PHP code.

I don't want to sound critical. Hell, I am in favour of any talk that gets PHP code tested :). It's just that your feature comparison seems extremely barren.

We should get an e-mail correspondence going...

yours, Marcus

Thu, 15 Dec 2005 at 15:25:40 GMT Link


9.Chris Shiflett said:

Thanks for the correction. I guess I should have said that the HTML reporter was the only one I could find in the distribution. :-)

The feature comparison is barren, because I want to do that in another post (after speaking with people like you and Sebastian).

Thu, 15 Dec 2005 at 15:34:45 GMT Link


10.Marcus Baker said:

Hi.

Just scanned the rest of the postings and had a few alarm bells ring. I just don't need more bug reports right now ;).

1) Regarding reporting the test name, this is only possible in PHP5 unless you use underscores to separate words. This is a limitation of PHP4 in that it reports the method names in lowercase, munging the words together. Unpicking the words within the reporter is pretty trivial.

2) The error messages in SimpleTest include the failed test line number, the expected value and the actual value. It does this in pretty gruesome detail. You can also write custom expectations and substitute your own. The expectations nest in places, although not yet to arbitrary complexity.

Mike, I care. I know I have been lax getting back to you on occasion and accept full blame for any missing TapReporter features. It's mainly babies (I have another on the way) and an increased writing load. Mail me please, and hopefully I can give you an update on the 1.0.1alpha3 version, which has a much simplified MVC structure.

yours, Marcus

Thu, 15 Dec 2005 at 15:41:21 GMT Link


11.Marcus Baker said:

Hello again!

There is a TextReporter (in reporter.php) for the command line and an XmlReporter (in xml.php). This is for remote stuff and talking to other test tools. There is even an Eclipse plug-in.

Anyways, I'll check out your slides later today :).

yours, Marcus

Thu, 15 Dec 2005 at 15:45:00 GMT Link


12.mike lively said:

I kind of feel bad for not updating this back in may, but I had written a newer and more complete version of tap-reporter.php. I just never got around to posting it to my site. I have went ahead and updated everything and created a new tar-ball. The post describing the changes can be found here:

http://www.digitalsandwich.com/arch...pache-Test.html

A direct link to the file here:

http://www.digitalsandwich.com/uplo...che-test.tar.gz

Fri, 16 Dec 2005 at 01:58:07 GMT Link


13.Grumpy said:

Always learn a lot from you. Thanks Chris!

Grumpy

http://grumpy.blogs.com/grumpy/

Sun, 22 Jan 2006 at 06:33:41 GMT Link


14.Andre Matos said:

Hi,

I can't find the file apache-test-demo.tar.gz to download. The link "http://shiflett.org/apache-test-demo.tar.gz" is broken.

Could you please let me know the new link or if you can send to my email?

Thanks.

Andre

andrelmmatos@gmail.com

Thu, 07 Jun 2007 at 16:47:45 GMT Link


15.Ashton Honnecke said:

Ditto Andre.

$ wget http://shiflett.org/apache-test-demo.tar.gz

--11:35:30-- http://shiflett.org/apache-test-demo.tar.gz

=> `apache-test-demo.tar.gz'

Resolving shiflett.org... 64.151.81.46

Connecting to shiflett.org|64.151.81.46|:80... connected.

HTTP request sent, awaiting response... 404 Not Found

11:35:30 ERROR 404: Not Found.

Thu, 15 Nov 2007 at 18:36:24 GMT Link


16.Chris Shiflett said:

Thanks for letting me know. It should be fixed:

http://shiflett.org/apache-test-demo.tar.gz

Thu, 15 Nov 2007 at 18:43:46 GMT Link


Hello! What’s your name?

Want to comment? Please connect with Twitter to join the discussion.