Yeah they're all good. In my case I created the AG listener from SSMS. Assuming that app knows what its doing (I know, probably a bad assumption) So, I would expect it to set things up the way they need to be.
Ahhhh, well there's your problem. 
Actually it's very likely that didn't quite do it. Can you drop the listener and then following the instructions in this part:
It's annoying but that's an unfortunate feature of AG listeners in Azure, they don't seem to work correctly if you use SSMS. The cluster manager instructions should do the trick.
I outlined the listener in black in this picture (DEV-LIS), you'd right click on that and choose Properties, the PTRRecord check box would appear there:
Well that almost worked. I'm not using ILB so I had to skip some steps. At the end of the process though, SSMS does not see a listener
So, I recreated the listener from SSMS. Now Failover Cluster manager sees it. I can connect to it from the Primary replica but not anywhere else
OK, you HAVE to create a load balancer, it's not optional, at least not with Classic. Unless you find instructions for Resource Manager deployments that say "Do not create a load balancer" you'll need to set one up, either external or internal. Make sure that's set up first and then re-create the listener.
Not using classic. RM all the way
and these instructions:
say nothing about it
and this article:
https://azure.microsoft.com/en-us/documentation/articles/load-balancer-get-started-ilb-arm-ps/
says nothing about Availability groups.
At least for RM, it looks like they're mutually exclusive.
Crap. Sorry. Give Microsoft a shout.
More on this. I was shown this blog post: https://blogs.msdn.microsoft.com/igorpag/2016/01/25/configure-an-ilb-listener-for-sql-server-alwayson-availability-groups-in-azure-arm/#comment-2655
which confirms your idea, Robert! So kudos to you on that!! I'm working through it and have hit a couple of bugs with the author is fixing in real time.