Can someone find out what sites I visit knowing my ip address?

A simple example. I send an email. The receiver found my IP address from the headers. Is it possible for him to find out the sites I visited using my IP address or only an ISP can do that?