Here's what works for me:
Go to the Shell cloth, and up the rock incline behind it.
Jump out to the farthest of the two small "islets" that stick out from the main island.
Swim hard ("running" swim) straight at the horizon, and let the current carry you sideways.
You'll need to turn your avatar to the right every once in a while to stay lined up with the horizon. The goal is to stay perpendicular to the current.
The tower should then slide into view on your right after a minute or so, hopefully close enough for you to be able to stand up and walk around.
I know a couple of people with older systems have noted that this solution doesn't quite cut it for them, in which case you can try a couple of other things:
1) The quab you're after on the tower should migrate to the main island once all of his buddies have been kicked off and you've linked out. If he's still on the tower, you can also try to:
2) Invite someone to Ahnonay with you, and see if they can swim out to rid you of the pesky quab.