RFC: Limiting grant duration
I am thinking about adding this to the grants operation rules:
If the grant does not finish within three times of the proposed duration or within two years, the grant will be considered a failure.
First of all, it should be noted that it won't affect most grants (see the reason 2).
Reason 1: Budget structure & allocation to new grants
TPF does not allow over-allocation of the budget. If we have 3 x $3,000 grants running and our budget is $9,000, no more grants can be funded.
If a grant worth $3,000 is running for five years, this $3,000 is stuck in the TPF safe. Even if we get good proposal, we cannot fund it using this money.
The aim is to increase liquidity of the fund.
Reason 2: If a grant runs more than a year, chance of failure increases
During 2009-2013, we managed 17 grants. Here is the breakdown:
| Grant | Result | Proposed | Actual |
|---|---|---|---|
| A | Success | 4 | 4 |
| B | Success | 6 | 7 |
| C | Success | 4 | 12 |
| D | Success | 3 | 5 |
| E | Success | 6 | 9 |
| F | Success | 2 | 5 |
| G | Success | 3 | 3 |
| H | Success | 4 | 2 |
| I | Success | 1 | 11 |
| J | Success | 3 | 8 |
| K | Success | 2 | 5 |
| L | Success | 4 | 9 |
| M | Failure | 3 | 12 |
| N | Failure | - | 13 |
| O | Failure | 7 | 41 |
| P | Failure | 3 | 26 |
| Q | Failure | 3 | 32 |
(Proposed/Actual: Duration in months. The grant N did not estimate duration)
Statistically, all the grant failed if they ran for more than a year. And except for the grants C and I, all of them failed after running more than 2.5 times of the proposed duration.
Would like to have your feedback.
Secretary, The Perl Foundation.
This sounds very reasonable. You might also consider limiting the proposed duration to something like 2 months per $1,000 dollars of grant money.
Although the proposal is not a bad idea to help mitigate the effects of this problem, it seems to me we need a better understanding of why this is happening in the first place. Because if grants fail like this it makes it harder to convince companies to support the program. So we really should have some sort of post mortem... Is it that we are dreadfully underestimating the amount of work in the grants, or is it that the grants themselves are so small that people working on them have a hard time prioritizing the effort? Or both or something else? I would say 5 out of 17 fails and only 3 out of 17 finishing in the time frame originally envisioned should ring some warning bells.
I would think some sort of short questionnaire at the minimum would assist.
Just thinking about it, I kinda feel 'three times the proposed timeframe' is way too generous. I would think if a grant has produced nothing within the originally scoped timeframe that there needs to be some sort of vote or review to decide if the work is worth continuing. I would say there is some sort of onus on the developer to shout of if there is a problem, like if she or he had some life crisis that is interfering with the work, or after spending half the allotted time one realizes that the scope of work is way harder than originally imagined.
My gut feeling here is that the grant amounts are quite small and as a result we are treating grantees like volunteers mostly (and cutting them a ton of slack as a result) Perhaps that is the real issue.
None of the failed grants ran less than a year, but all of the successful grants did. A one-year limit makes the most sense on that basis.
Some overoptimistic estimates were seriously missed, but the grants succeeded anyway and I’d feel loathe to sacrifice those.
On this basis, I would suggest something like “if the grant runs more than a year and twice its estimate, it will be considered a failure”. That is, any grant estimated at shorter than 4 months has some extra slack to finish late, but grants estimated at 4 months or longer may not only run up to twice their estimate.
I note that this rule would have left the failed 3–month grants running 3 months longer than the committee-proposed rule would have (because they fall below the 4-months inflection point) – but it would also have cut the failed 7-month grant short 7 months earlier than the committee-proposed (because it is well beyond the 4-month inflection point). I also note that it would have permitted all of the successful grants to succeed, whereas the committee-proposed rule would have stopped two of them short.
Purely from the perspective of the grants, this makes the most sense to me.
Now —
That said, the upshot of the rule I propose would be that any chunk of the budget will be tied up for up to one year. Based on the budget, this sets a limit on the number of grants that can be funded in parallel. The question is, does TPF consider that number acceptable? If not, then the rule will have to be adjusted accordingly.
This consideration applies equally to whatever rule is under consideration.
I disagree very strongly.
Strict control is important only if the value of the project is marginal; I would hope that the grants here yield results worth much more than the grant money itself. (And indeed, relative to market rates, these grants are pittance.)
This is not a commercial operation; it is an attempt to create common good for the community. As such, it should not be run with project management practices premised on the needs of enterprises.
Still,
I agree with this despite what I said above. It’s just that I think the question is whether the developer is finding time to devote to the project at all. As far as I remember from the failed grants, a shared feature of them all were long stretches of inactivity (despite occasional bursts of progress in some of them). So if the estimated time has passed, it’s time to ask whether it has any forward motion. If yes, then I think plenty of slack is justifiable. If not, better to release the resources than keep hoping.
(A post-mortem like you proposed in the other comment also sounds like a good idea – for every grant actually, not just the failed ones.)
I agree this is not commercial... And when I set goals and timelines for the Catalyst project I do reserve the right to assert my personal life and career needs over tentative timeframes. But also it would be good if we could find a way to better estimate. If that means doing a smaller job for a grant, then do that. Do the smallest useful amount of work and apply for a second grant to improve the project. I still think it would be worth understanding how this happens such that we can improve the project. Maybe we need sanity checkers on the proposals to make a better decision about if a timeline seems fair. I know we always tend to want to promise more and underestimate.
Right. It's in our agenda to eliminate or decrease failures. Grant failure happens, but we should make every effort to prevent that. Fortunately we don't have recent failure examples but we will analyze carefully as they happen.
Yes, I feel something similar. Open source activities are meant to be fun and I would like to keep it fun. While we need to put some restriction on the grants as we use the community money from the donors, I'd like to seek a reasonable balance.
So the question is where we can put a bar to mitigate risk of running a grant for long period without becoming too restrictive. I still think 3 times or 2 years is a reasonable bar. I will continue the discussion with the Grants Committee and send the updates to the community.