I'm not a vertigo sufferer so it's difficult to know what triggers the 'panic' response, but the Agia Irini descent is one of the more popular walks and I've seen it done by people who, to put it politely, appear from their choice of footwear not to be regular walkers!
On the assumption that you'll be intending to start from the top and walk down, there's only one section which could be described as at all airy, and that's a stepped descent which has a wooden barrier rail for protection. The steps are wide, so one could walk on the inside to avoid the view.
If you walk up, which plenty of people do, unless you actively choose to look down I can't think of anything which could cause concern.
Good luck and I hope you enjoy the walk.