This could have been one of the major city's of the region because of its huge natural harbor, yet it became a quiet town - for Asian norms that is - off the beaten track. War sometimes beats the track though, so check with the locals if it's safe to go there.
Things that you might remember most: the strong smell of curry, filling whole streets. Wild monkeys on the road south, following the sea. Some of the best beaches of the country, in Nilevali. Authentic little fishing villages to the north that have never thought of the possibility that a passing tourist might think they're authentic little fishing villages. In Trincomalee itself some temples. A weird kind of deer living in a 'Natural Reserve', right in the city center.