This entire post is under the assumption that the PS3 platform is not going to be an island unto itself. I'm assuming that the PS3 is being designed to do distributed computing, and that machines will collaborate with each other to compute sub problems individually and share answers after the fact.
1) The cost of communicating the solution must be less than the cost of computing the solution.
Corollary: The cost of scheduduling, distributing the problem and receiving the results back must be less than the time to compute the answer.
There's no point in asking someone else what 1+1 is when it takes less time to figure it out than to ask the question. Are there any kind of problems that are easy to describe, require lots of computation and cost little to send the results around?
2) What happens when an ISP hiccups and a block of machines become unreachable, the content of their computation becomes unreachable?
3) What about latency between machines? For a real time game, Machine X is going to have to get data from Machine Y at least 30 times a second.
4) Am I in control of what code is being run on my machine? If I'm playing Game Y will my machine be calculating results for Game Z? Can I donate my down time to a particular game?
5) Dynamically adding and losing processors that are not localized for real-time computation and gaming, has anyone done this before? Are there any working examples, ones where the games noticeably improve with the addition of client machines?