A little test animation of a hypotrochoid gear system with a 3-sided inner gear, a 5-sided outer gear and eight simple radial gears in between.
The inner gear has 24 teeth and the outer gear has 40 teeth leading to a 3:5 teeth ratio. This defines the relation of the shapes of the inner and outer gears. The shape of the inner gear can be expressed by these formulas with a being the angle: x = r * sin(a) y = r * cos(a) r = 1 + cos(3 * a) / 9
The shape of the outer gear is calculated using an iterative fitting process.
Even though this is just an animation, I have a physical version here printed in 3D by shapeways.