As you can see the Quarry is within walking distance to downtown.  However you still may want to consider the golf cart rental to view more of the island.