Changing money in India is not too difficult and can be done safely at the right place.
There are some money changers in Navadvipa. I've read that their rates are a little less than Kolkata. You can also change at the Kolkata or Delhi Airports. You will need to change some money at the State Bank of India counter (24 hours) located just behind the customs "Green Channel". They will give you a computerized exchange encashment receipt. Keep the encashment receipt so you can buy a train ticket or change your excess rupees back when you leave.
DELHI - VRINDAVAN
You can change your money at the airport at Thomas Cook or the bank just outside the customns in the arrival hall. There are places that change foreign currency in Paharganj, and you can change US dollars and UK Pounds notes or Traveller's cheques in Vrindavan, (if you want to buy a lot of stuff in Loi Bazaar before you come back, it would be more convenient to change the money there) but if you have Euros, Shekels, or anything else, change it here at the airport.
NOTE: Try to get some small rupee notes. The bank will want to load you up with 1000 and 500 Rupee notes (less counting for them!) but those are hard to use unless you are making a big purchase. Hundred Rupee notes are better, and some fifty rupee notes mixed in will be helpful. This request sometimes falls on deaf ears as several times I have had them give me all big notes anyway. Take as many notes as you want in hundreds and give it back to them and ask firmly "Can I have that in HUNDREDS PLEASE".
You can get a little better rate for Traveller's cheques. Be sure to keep the receipts separate from the Traveller's cheques. See the section above on Traveller's cheques before buying any.
It is not too difficult to get change at a shop for a hundred or fifty rupee note. Also be careful not to accept any torn notes as change or notes which are repaired with a cellotape. Politely hand it back and ask for another note. No one will want them when you try to spend them. The State Bank of India will take a torn note and exchange it, but there is always a line and it's not worth the hassle. There is a man near Holi Gate in Mathura who will charge you a few rupees to change a bad note for a good note.
You may discover that the notes are stapled together into bundles of a hundred notes with huge brass staples. You can break open the bundle by flipping down about half way down the stack, (making a "V", with the stapled part at the bottom) then pulling the stacks apart hard with both hands. Alternatively, this is also one of the many uses of the Swiss Army Knife. Work the knife blade or screwdriver under the top of the staple and twist it to pry the staple out.
The hotel or DTP prepaid taxi booth (more about that later) can easily deal with a Rs. 500 note. The taxi to Mathura/Vrindavan may have trouble making change. If you get stuck, there is also a Thomas Cook office at the airport, but sometimes they are not there, and their rates are not as good.
Take a moment to figure out what you're going to do next. There's nobody breathing down your neck or ogling your wad of notes right now. (Be very careful about this in other places!)
I would suggest keeping most of your money in reach, but out of sight, in your money belt or waistpack in the compartment nearest to your body. If that's not practical due to bulk, keep it in a secure compartment in your hand luggage. The same goes for your wallet, plane ticket, and passport. Put the money you know you'll be needing soon in the outermost waistpack pocket or in your shirt pocket. What you want to avoid at all times is flashing a lot of cash. Generally, I keep at most Rs 100 in my waistpack outer pocket (including small bills and coins). That way when you go to pay for something and everybody looks to see how much you've got (I guarantee they will), you won't attract attention.