Walk away from Silver Bay through the forest and head inland, go past the farm buildings of Bryn y Bar, currently being renovated , follow the road be careful not to follow the coastal path! You then join the road you may have driven in on - so watch out for cars.
Follow the instructions careful, now as the cottage of Tŷ Werl is easily missed, remember you should be heading for the church. Hopefully this street view image will help; you are looking straight at the cottage and the footpath is to RHS. Heading inland your route takes you inland back to Rhoscolyn, and depending upon where you parked, you could easily cut this walk short, or stop for a pub lunch at the White Eagle.