So I am working on a project at a client which is a server application and a component of other server applications. Its a complex data service which can be deployed as a component of other applications and can be hosted as a WCF service and called by remote clients. The application has a lot of configuration settings which we control in the app.config file. We have service locator settings, middle-ware settings, database settings, logging settings, etc etc.
This is a re-engineering of an old system which used to take all these settings on the command line, or when running as a windows service, through registry entries. In order to smooth the transition to this new system, we have enabled the software to continue to take these settings through the command line/registry settings in that it used to use. The way we do this is to use a base application configuration file, open it as an XML document and use XPaths to swap the overrides into the XML. We then write that XML out to a seperate file and repoint our App Domain to use that config file. The base application config file can even be passed as a command line argument.
This has been working for use since about last summer when we implemented it. We had one simple stumbling block along the way when somebody added a Log4Net call in application before the configuration swap has been completed. This is a problem as when log4net initialises it reads the configuration file. The problem with this is that .NET caches the application config file on first read. However, with the help of a great little tool called Reflector I found this little static cache and therefore was able to use reflection to wipe it out and reset the framework to think it hasn’t loaded it yet. With this little change in place, all was right with the world. Until this week.
This week another change was implemented thanks to some unknown obscure functionality required some a couple of people located in one small corner of one office in the world. We had to read one of our configuration overrides from a database, otherwise we weren’t going to get signed off for release. Therefore, without a choice, we had to use the config/override settings that we had for the database connection to connect the database and read this setting. We put this change in and tested it, it all seemed good. We then promoted our change to the QA environment and it was tested when used as a component of another application. That is when it started to throw an Exception. It was actually WCF which threw the Exception saying that there was no endpoint in the config file for the contract we wanted.
This exception puzzled us a lot, we could see that our config file was being generated properly, we could see that our application was able to read it properly, otherwise it wouldn’t have known to call a WCF service. It was even more confusing when the error didn’t happen on one of the machines. After a whole day of debugging a head scratching, we figured out that the .NET framework has a SECOND cache for configuration data. Not that actual configuration file, as when we asked for our sections from the System.Configuration.ConfigurationManager, we got the values we expected. The reason the single machine that did work worked, was because the Sybase database driver wasn’t installed properly and therefore the connection didn’t open.
The only solution I have come up with so far this week is the run the database connection code in a separate AppDomain. This obviously has prevented the caching on configuration in the primary AppDomain as static variables are per AppDomain. Once I have found this extra little cache I will let you all know where it is. Until then, if anyone else knows or has the same issue, let me know.