Do I have to worry about temperature with neodymium magnets?

Yes.  Neodymium-Iron-Boron magnets are sensitive to heat. 

If a magnet heated above its maximum operating temperature (176°F (80°C) for standard N grades) the magnet will permanently lose a fraction of its magnetic strength.

If they are heated above their Curie temperature (590°F (310°C) for standard N grades), they will lose all of their magnetic properties. 

Different grades of neodymium different maximum operating and Curie temperatures.