I can't see any harm in leaving them attached. The only thing that will happen over time is that the bearings will become magnetized a little.