Usually magnetic materials are also electrically conducting. Static electric charges could be induced in the magnets we are testing. Then opposite charges will attract each other towards themselves. The gravitational force of attraction also attracts objects.
If two objects repel like north pole - north pole then we know for sure that the repelling force is due to magnetism only. If two objects attract, that could be other attractions, then we cannot say it is due to magnetism only.
