We’ve been working in a user acceptance testing (UAT) environment which, as compared to development, is dead slow.
It’s a complicated environment using FBA, SQL 2008, SSRS and extended web applications accessible over the internet using https, so it’s been hard to track down the issue.
For a previous client, we used FBA with an LDAP role provider (and membership provider). One of my colleagues, far more clever than I, determined that the "out of the box" LDAP role provider, when used in that environment, wasn’t scaling well. To solve this problem for that client, he implemented a nice caching scheme in a custom role provider.
This situation seemed similar, so we looked into replicating that solution to the today’s client. As I was debugging that, I noticed that this message would frequently appear in the System log (from Event Viewer):
A worker process with process id of ‘XXX’ serving application pool ‘Home – 80’ has requested a recycle because it reached its virtual memory limit.
I took this to mean that the app pool was recycling far, far too often and that would explain a performance problem.
I looked at the app pool’s properties and its "Recycling" page showed that the property "Maximum virtual memory (in megabytes)" had been set to true and had been set to 5000. That seems like enough, but I decided to unset the value and that had an immediate positive effect. No more app pool recycling. No more mysterious slow-downs and pauses.
I don’t really understand the underlying "stuff" that’s going on there, but clearly some kind of cause/effect thing is happening and for now, the UAT environment is usable.