PHP TestFest Success Metrics

Cal asked on Twitter[1]:

What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
   through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
   how to write and run them

For the group of folks in #3, I hope that they'll become frequent core
contributors, once they know the environment and how to write and run
phpt tests.

#1 and #2 are difficult to measure, but with #3, it may be possible to
measure the number of people who have never contributed to php-src who
have submitted new phpt tests during PHP TestFest.

For all 3 items, I'd like to record the number of groups who
participated and held at least one PHP TestFest event, the number of
total PHP TestFest events, and the number of people who attended each
event. It would also be good information to know the number of phpt
tests contributed by each group (i.e., the aggregate of tests
submitted by members of each group).

So, that's a quick brain dump.

Thoughts?

Cheers,
Ben

[1] https://twitter.com/CalEvans/status/887380464855658498

Cal asked on Twitter[1]:

> What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
   through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
   how to write and run them

For the group of folks in #3, I hope that they'll become frequent core
contributors, once they know the environment and how to write and run
phpt tests.

#1 and #2 are difficult to measure, but with #3, it may be possible to
measure the number of people who have never contributed to php-src who
have submitted new phpt tests during PHP TestFest.

For all 3 items, I'd like to record the number of groups who
participated and held at least one PHP TestFest event, the number of
total PHP TestFest events, and the number of people who attended each
event. It would also be good information to know the number of phpt
tests contributed by each group (i.e., the aggregate of tests
submitted by members of each group).

So, that's a quick brain dump.

Thoughts?

Technically would it be a possibility to use the GitHub-clone of the
official PHP-Repo to create PRs against? That way we would (relatively)
easy see who never contributed to the core before. If participating
usergroups would create a fork that would then be forked from the
participants that should work as well and we could get all the
contributions from a user group in one batch and we would be getting the
number of participating usergroups en passant…

I'd also back the idea that it's more important to get people to testing
and contribute to tests over the sheer number of tests. After all every
participant is a win for the PHP-Project! And if people only get to setup
their system and don't contribute that's fine. The only way to measure that
is by getting the number of participants from the different usergroups and
that needs to happen via a separate "survey"…

We can however try to find out how the test-suite improved by registering
the number of contributed and fixed tests. But I'd only see that as an
overall figure to see how much the testfest helped to improve the tests.
I'D not break that down to a "per-usergroup" or "per-user" ratio (you can
do that but it doesn't say anything…)

My 0.02€

Cheers

Andreas

···

Am Dienstag, 18. Juli 2017 21:09:39 UTC+2 schrieb Ben Ramsey:

Honestly, I was hoping for a hard metric.

% code Coverage
##k Unit tests

Something that we can put on a thermometer and see the progress.

Cheers! :slight_smile:
=C=

···

On Tuesday, July 18, 2017 at 3:09:39 PM UTC-4, Ben Ramsey wrote:

Cal asked on Twitter[1]:

> What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
   through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
   how to write and run them

For the group of folks in #3, I hope that they'll become frequent core
contributors, once they know the environment and how to write and run
phpt tests.

#1 and #2 are difficult to measure, but with #3, it may be possible to
measure the number of people who have never contributed to php-src who
have submitted new phpt tests during PHP TestFest.

For all 3 items, I'd like to record the number of groups who
participated and held at least one PHP TestFest event, the number of
total PHP TestFest events, and the number of people who attended each
event. It would also be good information to know the number of phpt
tests contributed by each group (i.e., the aggregate of tests
submitted by members of each group).

So, that's a quick brain dump.

Thoughts?

Cheers,
Ben

[1] https://twitter.com/CalEvans/status/887380464855658498

The overall progress or the best groups?

For the overall progress we can use the number of overall added and fixed
tests as well as the improvement in CodeCoverage (we're not disputing the
value of that metric here) - even broken up to different extensions.

That will show the impact of the testfest for the PHP-Project.

But breaking that down to a group-level is IMHO dangerous as there will be
groups with more and less contributors so the absolute numbers don't say
anything. And every participating group is a "winner".

But - as I said: Just my 0.02€ :wink:

Cheers

Andreas

···

Am Dienstag, 18. Juli 2017 21:39:27 UTC+2 schrieb c...@calevans.com:

Honestly, I was hoping for a hard metric.

% code Coverage
##k Unit tests

Something that we can put on a thermometer and see the progress.

Overall progress would be an interesting metric, if it can be done, if not
I don't think it's a big deal.

On another point, I have to agree that I don't think it wise to break it
down to a group level for a myriad of reasons, including how groups vary
wildly in makeup.

I don't think this is a contest anyway, at least the way I see it.

Mike

···

On Tuesday, July 18, 2017 at 3:45:22 PM UTC-4, and...@heigl.org wrote:

Am Dienstag, 18. Juli 2017 21:39:27 UTC+2 schrieb c...@calevans.com:

Honestly, I was hoping for a hard metric.

% code Coverage
##k Unit tests

Something that we can put on a thermometer and see the progress.

The overall progress or the best groups?

For the overall progress we can use the number of overall added and fixed
tests as well as the improvement in CodeCoverage (we're not disputing the
value of that metric here) - even broken up to different extensions.

That will show the impact of the testfest for the PHP-Project.

But breaking that down to a group-level is IMHO dangerous as there will be
groups with more and less contributors so the absolute numbers don't say
anything. And every participating group is a "winner".

But - as I said: Just my 0.02€ :wink:

Cheers

Andreas

Hi,

Perhaps [also] a tweetometer to inform the twitterverse each time a
testfest PR has been created?

I know it's not a success metric as such but I think it would aid
publicise what's happening :slight_smile:

k.

···

On 18 July 2017 at 20:39, <cal@calevans.com> wrote:

Honestly, I was hoping for a hard metric.

% code Coverage
##k Unit tests

Something that we can put on a thermometer and see the progress.

Cheers! :slight_smile:
=C=

On Tuesday, July 18, 2017 at 3:09:39 PM UTC-4, Ben Ramsey wrote:

Cal asked on Twitter[1]:

> What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
   through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
   how to write and run them

For the group of folks in #3, I hope that they'll become frequent core
contributors, once they know the environment and how to write and run
phpt tests.

#1 and #2 are difficult to measure, but with #3, it may be possible to
measure the number of people who have never contributed to php-src who
have submitted new phpt tests during PHP TestFest.

For all 3 items, I'd like to record the number of groups who
participated and held at least one PHP TestFest event, the number of
total PHP TestFest events, and the number of people who attended each
event. It would also be good information to know the number of phpt
tests contributed by each group (i.e., the aggregate of tests
submitted by members of each group).

So, that's a quick brain dump.

Thoughts?

Cheers,
Ben

[1] https://twitter.com/CalEvans/status/887380464855658498

--
You received this message because you are subscribed to the Google Groups
"PHP TestFest" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to testfest+unsubscribe@phpcommunity.org.
To post to this group, send email to testfest@phpcommunity.org.
Visit this group at
https://groups.google.com/a/phpcommunity.org/group/testfest/.
To view this discussion on the web visit
https://groups.google.com/a/phpcommunity.org/d/msgid/testfest/02954b8c-a328-4b90-98ef-971cfa167c4f%40phpcommunity.org.

--
http://about.me/kenguest/

I’ve updated the PHP TestFest website with copy that makes my goals
clearer. Let me know what you think.

https://phptestfest.org/
https://phptestfest.org/start/

Specifically…

“Growing the PHP core one test at a time”

“Become a PHP core contributor! Join a PHP user group in person or
online, and PHP TestFest will show you how.”

“Have you ever wanted to contribute to PHP but have been afraid that
your C skills aren’t up for the challenge? Well, have no fear! If you
know PHP, you can contribute by writing tests. Through your local user
group, PHP TestFest will show you how.”

“PHP TestFest seeks to create more PHP core contributors by introducing
PHP programmers to the PHP language test suite, teaching them how to
write phpt tests. We do this by using our relationships with PHP user
groups to host PHP TestFest events around the world.”

Let me know what you think.

Cheers,
Ben

···

On Jul 18, 2017, at 2:09 PM, Ben Ramsey <ben@benramsey.com> wrote:

Cal asked on Twitter[1]:

What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
  through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
  how to write and run them