This paper presents a gouge-free tool path generation method for direct ball-end cutter CNC milling of cloud of point. In this method, a base surface with simple geometry, which can approximately reflect the features and the changes of surface shape, is first constructed, and the guide curves with different patterns of the cutter motion are then planned on it. After the cloud of point is offset by a distance of the ball-end cutter radius, the guide curves are deformed onto the offset of cloud of point by point projection operation along the computed direction instead of the traditional z-axis direction, thereby forming the desired tool paths directly on the cloud of point. In projecting guide curves, the computation of the projecting direction and construction of the weight function are discussed in detail to eliminate the projection error as much as possible. The technique of adjusting the path points is also presented to eliminate the unfavorable undercuts and overcuts. Since the cloud of point is offset a distance of the cutter radius, the gouge between the cutter and the point set surface can be automatically avoided. Also, various patterns of the cutter motion can be conveniently planned on the simple base surface, such that the cloud of point can be milled along the generated tool paths with the most suitable topology. Finally, several examples are given to validate the proposed method.