SSRS lose security after service is idle for several minutes on Azure VM

I have several instances of SSRS running without any problems on different servers. However, our production instance is running on an Azure VM and it seems to do something weird after being idle for several minutes. Once server is idle for a several minutes no user, regardless of permission level can upload a new report or edit an existing report. See the error generated in log file below. The only way to fix this is to restart the SSRS Server service. This is only a problem on the Azure SSRS instance. Right now I created a scheduled task to restart the service periodically but his is a hacky temporary solution. Any help with this would be greatly appreciated.

library!ReportServer_0-1!2f48!11/07/2017-10:45:09:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: , Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error log for more details. ---> System.TypeInitializationException: The type initializer for 'Microsoft.ReportingServices.Library.Global' threw an exception. ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.ReportingServices.Storage, Version=13.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. Access is denied.
at Microsoft.ReportingServices.Library.Global..cctor()
--- End of inner exception stack trace ---
at Microsoft.ReportingServices.Library.Global.EnsureModeFromCatalog()
at Microsoft.ReportingServices.WebServer.RsSoapBase.Init(String itemPath, Boolean checkAccessToSharePoint)
--- End of inner exception stack trace ---;