Should I be worried if a couple of sites that I don’t want to be associated with are linking to me?

Matt's answer:
Today’s question comes from Jay in Spain. Jay asks, “Recently I found two porn websites linking to my site. I disavow those links and wrote to the admins asking them to remove those links. But what can I do if someone– like my competition– is trying to harm me with bad backlinks?” So you’ve done exactly the right thing. You got in touch with the site owners and you said, “Look, please don’t link to me. I don’t want to have anything to do with your site.” And then if those folks aren’t receptive, then just go ahead and disavow those links. As long as you’ve taken those steps, you should be in good shape. But if there’s any site that you don’t want to be associated with that’s linking to you, and you want to say, “Hey, I got nothing to do with this site,” you can just do a disavow. And you can even do it at a domain level. So you could say, disavow domain:example.com and that will ignore all links from example.com. At that point, you should be in good shape and I wouldn’t worry about it after that.
by Matt Cutts - Google's Head of Search Quality Team