Ensure that Azure App Services restrict access to the SCM site that's used by your app.
In addition to being able to control access to your app, you can restrict access to the SCM site that's used by your app.
Add access restriction rules under network settings for the scm site used by your app