I have tried it only once (on my drifting car!) before. The following reasons are at least what I can think of that would cause the problem you mentioned:
1. Differential too stiff or you are using the original gear differential.
2. using too many shims on either or both sides of the rims that stops them from running smoothly.
3. You are using a "one-way".
4. You are using metallic wheels which are heavier and that add extra load onto the U-joints.