About the Author

Chris Shiflett

Hi, I’m Chris: web craftsman, community leader, husband, father, and partner at Fictive Kin.

Testing PHP

I guess this is my belated ApacheCon blog. I had a lot of fun as usual, and I got a chance to meet a few new people and hang out with old friends. The talk that Geoff and I gave went really well, and it ended up being as funny as we had hoped. More exciting than the talk is the project behind it. I think we've created a really nice testing framework for PHP applications, and I'm going to try to describe it for those who missed the talk.

There are a couple of resources available for download:

What's so great? Apache-Test - a testing framework for Apache that adheres closely to strict testing ideologies. Geoff and I have added features to Apache-Test that let you use it to test PHP applications. In addition, we have provided a PHP implementation of Test::More - a popular CPAN module that has some very simple functions that make writing tests easy.

The idea for this project began at OSCON, where Geoff and I were discussing testing tools and methodologies. The Perl community has embraced testing as a key component of a developer's skillset, whereas the PHP community still relies heavily upon echo and manual testing with a browser. Since Geoff is a Perl guy, I'm a PHP guy, and we happen to be good friends, we decided to create a way to let PHP developers benefit from Perl's mature testing tools and methodologies. Since Geoff is really a mod_perl guy and interested in all things Apache, he is a big fan of the Apache-Test framework. This seemed like a great tool to provide to PHP developers, so we proposed a talk about it. Once the talk was accepted, we had to deliver.

What makes Apache-Test so great for PHP developers? It provides several key features not available (to my knowledge) with any other testing framework:

  • You don't have to write any Perl. This certainly isn't a unique feature, but one problem with our attempt at being funny in the naming of the talk is that people tend to think that using this framework requires you to know Perl, and that's just not true.
  • You don't have to write any tests in your application. One thing that bugs me about several of the testing approaches I have seen is the necessity of including tests within the application itself. Thus, your tests can affect the behavior of your application, which totally ruins the concept of testing. You see, testing an application should be an example of a control experiment, where there is only one thing that changes - your application. If all tests pass, you make changes to your application, then some tests fail, you want to be assured that the cause of the problem is the changes you made to your application, not tests that you have written. In fact, a test shouldn't even affect any of the other tests, much less the actual application you're testing. I think this flaw in existing approaches has stymied our adoption of testing as a necessary skill.
  • Writing tests is easy. I think this is a key feature. If writing tests is a hassle, no one will want to do it. Let's be honest - developers are lazy. In fact, we take pride in being lazy. The PHP implementation of Test::More that is now included in Apache-Test provides simple functions for writing tests. You can use these with any programming paradigm, which means you don't have to use an object-oriented design, but you can if you want.
  • The tests are repeatable. This means that you can repeat your entire test suite as many times as you want, and running the tests doesn't affect the environment (or future iterations of the same tests) in any way.
  • The tests are automatic. You don't have to manually browse your application or execute tests. You just type make test, and Apache-Test takes care of everything else.
  • You get a self-contained and pristine Apache environment. Not including tests in your code isn't enough - you want to have as much consistency in your environment as possible. Apache-Test gives you a separate Apache environment with its own configuration (including a pristine httpd.conf and php.ini) using your build. You augment the default configuration in a separate file, so that every change must be deliberate (it is also a good practice to keep your changes to the real Apache and PHP in a separate file, so that you can be sure that it has the exact same configuration as your testing environment). When you type make test, Apache-Test starts the server, runs your test suite, generates a report, and stops the server.
  • Your application is executed with the real PHP. You can now also choose to run tests with the command line client.
  • It's very mature and stable. Major companies and open source projects have been using Apache-Test for years. It's a proven tool.

These are the major features (that I can think of) that I find particularly appealing. I'm sure there are others. Perhaps a nice side-effect is the potential for cross-pollination between the Perl and PHP communities.

If you're interested in learning more, please download the demo and the slides from our talk. To run the demo, you'll need Apache-Test 1.16 or greater. You can find the most recent version from search.cpan.org.

This testing framework is only available for Apache.

About this post

Testing PHP was posted on Wed, 01 Dec 2004. If you liked it, follow me on Twitter or share:


1.Lukas said:

I dont quite follow your arguments:

1) using any of the php unit testers you can test from the shell or from the web

2) you dont have to mix in your test code with your application

3) having a separate apache for testing is also not ideal since it can produce different outout than on your normal environment

4) you can create repeatable tests with any of the php test tools I know

Since I dont know your tools I cant comment on what is easier to use. Its not clear from your arguments but I assume your test tools can handle crash situations that a web based unit tester will not be able to handle.

Thu, 02 Dec 2004 at 10:30:39 GMT Link

2.Chris Shiflett said:

Hi Lukas,

Perhaps you can clarify some things for me. Here are my questions (numbered according to your points):

1. I haven't seen any unit testers for PHP that test your application from the Web without requiring you to manually test with a browser. I might be wrong in thinking that none exist, but I haven't seen this demonstrated. Also, you can do more than just unit testing with this framework. Unit testing is the easiest, but this is because PHP lacks some other tools that I plan to help create.

2. This might be true, but can you be specific about which one(s) allow this? I have definitely seen tests within application code in some examples and tutorials I have seen.

3. It's your Apache. This argument actually makes no sense, because each request Apache receives spawns a separate child process. Apache-Test just makes sure that the one you're using for testing isn't affected by outside factors at all. This is definitely ideal - the point of testing is to provide consistency in everything except that which is being tested.

4. Sure, not every feature in my list is unique, but the list as a whole is (as far as I know).

> I assume your test tools can handle crash situations

Yes, there is way to handle this. It cannot execute tests that happen after the crash, of course.

Thu, 02 Dec 2004 at 18:10:00 GMT Link

3.Geoffrey Young said:

"3) having a separate apache for testing is also not ideal since it can produce different outout than on your normal environment"

I'll disagree with that part in particular and leave the others to Chris :)

first of all, it's not a different Apache, per se - the binary (and shared objects) are identical. what is different is that your Apache _environment_ starts out in a clean state, at first consisting of only that which you need to get Apache and PHP running initially.

so, sure, you will likely get different results when running your application within the test environment and your "normal" production environment. in fact, that's the point - by starting off with a clean slate you know exactly what variables are important to your application.

let's try using a simple but concrete example. you try your code in the testing environment and find that it fails. further investigation finds that it is because register_globals is off. why is it off? because the testing framework starts with a clean slate and does not inherit from your local php.ini.

what do you do with this newfound knowledge? you start by overriding (not replacing) the php.ini given by the testing environment using standard httpd.conf directives, such as "php_flag register_globals On".

the point of the exercise is that with this failure you have just learned an important piece of information about your application that you may not have known before (since you probably didn't write the code you are forced to maintain :) that is, you (or your coworkers) never need to again consider whether that configuration option matters - you know it does, and it is immediately obvious to anyone in the future because removing it from your test environment will cause tests to fail.

there is (much) more that I could say about the benefits of all of this, but I'm out of time :)

Thu, 02 Dec 2004 at 18:11:23 GMT Link

4.Chris Shiflett said:

Geoff's comment reminds me that a FAQ for this project might be good. I think a lot of people unfamiliar with testing will have a hard time "getting it" without some of these types of questions answered very thoroughly.

Fri, 03 Dec 2004 at 04:08:16 GMT Link

5.Carolyn Smith said:

I am impressed by your lingo concerning software. It reminds me of some very important material. Is software your only field of expertise? Do you know anything about history?

Sun, 27 Mar 2005 at 20:36:26 GMT Link

6.Marcus Baker said:

Hi Chris/Geoff.

You actually convinced me of the argument for 3 until you explained it :). We run tests as part of a rollout for example. Having the tests run in a different apache environment just invites trouble. We could have a successfult test run, but a broken buid. There is no way you can rely on the actual php.ini and the apache test one ever being kept in sync manually.

Is 3 just an option or are you forced to do it that way. If you are forced then that would invalidate it for me.

Lukas' other points were essentially correct, even last year.

On a more affrmative note regarding using apache-test as an uber test framework, are you still interested in integrating SimpleTest with it?

yours, Marcus

Fri, 25 Nov 2005 at 02:43:41 GMT Link

7.Chris Shiflett said:

Hi Marcus,

> You actually convinced me of the argument

> for 3 until you explained it

Heh. :-)

I think I just failed to explain it very well. Apache-Test doesn't use a different build. It's still your Apache, your libraries, etc. Apache-Test just takes care of starting Apache, running your tests, and stopping Apache. No outside factors can affect your test suite. This consistency is an advantage.

Geoff's point is even more important. By augmenting the default httpd.conf and php.ini within the Apache-Test environment, your configuration must be deliberate and well-documented. This helps you better understand the application your testing, its dependencies, etc.

Someone already beat us to Apache-Test + SimpleTest. :-) See the last link in the bulleted list here:


I haven't tried it yet, but I plan to do so very soon.

Fri, 25 Nov 2005 at 16:31:35 GMT Link

8.Marcus Baker said:


OK, so if I don't specify anything it just runs the normal apache set-up, right? We tend to use cfengine to configure live servers and once you go the cfengine route it's a bit all or nothing. Specifying in both cfengine *and* apache-test would be a real accident prone pain.

Regarding using apache-test as an uber runner, obviously you get Perl integration for free, what about...say...Ruby?

Regarding the TapReporter for SimpleTest, I think it stalled a little which was mainly my fault. Some additional features were needed from SimpleTest for comlete compliance which at the time I didn't want to add, because of more pressing features. It is likely that these can now be added, but I lost contact with the project. if you are reinvestigating this area then hopefully interest will ripple down and things will restart.

As web developers we work with a huge range of environments and languages. Java folks can just use JUnit, whereas we need a multi language framework. I am interested to see where this one goes.

yours, Marcus

Sun, 27 Nov 2005 at 01:01:19 GMT Link

9.Millar said:

Thanks for this, its helpful. I used it on my server.

Sun, 10 Sep 2006 at 16:19:30 GMT Link

10.Standard Warfare said:

Only way you can really test PHP, unlike HTML, where you don't even need a server, but I suppose you gotta do it..

Sun, 08 Oct 2006 at 09:55:51 GMT Link

Hello! What’s your name?

Want to comment? Please connect with Twitter to join the discussion.