llong
Posts:83
| | 09/25/2007 6:10 PM |
| John, I have turned to you because you are likely to have a couple ideas and, well, I bought your module so I at least have some basis. My installation of DNN is restarting every 45mins to an hour's and I don't know how to solve why. I have page blaster installed and I notice a fairly regular "file not found" error in the server logs… Exception information:
Exception type: FileNotFoundException
Exception message: Could not find file 'fc1c75d9482da9177a4e31f1c030737d.pb.webinfo'. I have had issues with a module looking for non-existent .resx files constantly and spiking the CPU to 100% be the developer couldn't figure it out. I watch Process Explorer and captured a restart on Process Monitor but nothing stands out to me. I recently updated to 4.5.5 but this didn't happen right away. Where should I start? | | | |
|
|
|
John Mitchell
Posts:2779
| |
llong
Posts:83
| | 09/26/2007 10:59 AM |
| Hi John.
I have version 2.2.4 installed. Is there an update I should have?
The memory settings are at the defaults: 500MB virtual and 192MB used. I have played with the app pool settings a bit include using the web garden. We'll see if those change anything. | | | |
|
llong
Posts:83
| | 09/26/2007 11:07 AM |
| | hmmm, web gardens keep sessions in seperate processes so that could be a problem. Not sure how IIS would divvy up request between the processes so safer to put back to one I figure. If I could confirm the sessions stay in the same process for the same visitor then perhaps that could work. That assumption is based on a very low understanding of DNN code behaviour. | | | |
|
John Mitchell
Posts:2779
| | 09/26/2007 11:30 AM |
| 2.2.4 is the lates released version.
If you are using a web garden then you should set it back to one process. That may be the problem with the file not found errors. DNN has several problems in a web garden scenario.
If you also have control of the server then you should increase the memory limit, or not limit it at all. 192mb for a DNN site is the bare minimum, and if you add in PageBlaster caching then you will easily go over that and cause an app recycle. | | | |
|
llong
Posts:83
| | 09/26/2007 11:49 AM |
| Yes, the web gardens didn't work so back to one. It was only a garden for about 5 minutes.
I have increased the "used" memory to 400 mb (virtual is still 500). I take it more will stay in the RAM now?
| | | |
|
John Mitchell
Posts:2779
| | 09/26/2007 11:57 AM |
| | If the app pool process uses enough memory to reach the limit then the application will recycle, so yes the application will allow more RAM to be used without recycling. You can also set a memory limit in the Snapsis.PageBlaster.config file if you want to keep the website from using too much memory. | | | |
|
llong
Posts:83
| | 09/26/2007 12:01 PM |
| Thank you.
I will monitor for a bit more for the missing file errors. | | | |
|
llong
Posts:83
| | 09/26/2007 12:23 PM |
| Snapsis is now maxing out the CPU. I have watched for over 10 minutes as Process Monitor reports the following type on events:
127735 10:21:21.7552415 AM w3wp.exe 5420 CreateFile D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\971 Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened SUCCESS
127747 10:21:21.7617162 AM w3wp.exe 5420 QueryOpen D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\1110\aea4c9569a4ae6ef9f1f12519fd4c43d.pb.webinfo CreationTime: 9/25/2007 3:14:45 PM, LastAccessTime: 9/26/2007 12:23:56 AM, LastWriteTime: 9/25/2007 3:14:45 PM, ChangeTime: 9/25/2007 3:14:45 PM, AllocationSize: 8,192, EndOfFile: 6,833, FileAttributes: A SUCCESS
127748 10:21:21.7625413 AM w3wp.exe 5420 QueryDirectory D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\971\* Filter: *, 1: . SUCCESS Read Metadata
127749 10:21:21.7629711 AM w3wp.exe 5420 QueryDirectory D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\971 0: .., 1: 25d63c2606fb68f41e4d88961c6f611c.pb.webinfo, 2: 2ca9886ab962950501c70ae6aa22a5a8.pb.webinfo, 3: 6331848f389aa52c358a25828d4e8122.pb.webinfo, 4: 857e597b67a4592f00731140cce4b09a.pb.webinfo, 5: 9755b8bb3077e59a639937ad700c2d36.pb.webinfo, 6: d0517f8bce6ae68e3b1e3355485d2805.pb.webinfo, 7: d5b745aa70a46dce28d39d2570f528aa.pb.webinfo SUCCESS Read Metadata
127752 10:21:21.7639460 AM w3wp.exe 5420 QueryDirectory D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\971 NO MORE FILES Read Metadata
127753 10:21:21.7641876 AM w3wp.exe 5420 CloseFile D:\virtuals\legal-design_com\dotnetnuke\Web\DesktopModules\Snapsis\PageBlaster\Cache\tabid\971 SUCCESS
| | | |
|
John Mitchell
Posts:2779
| | 09/26/2007 12:28 PM |
| That looks normal to me. PageBlaster will read those files from disk and into memory, and it will also write to disk as a backup to the memory cache, and delete the files from disk when they are expired. Are you sure that PageBlaster is maxing out the CPU? | | | |
|
llong
Posts:83
| | 09/26/2007 12:35 PM |
| Those do look like normal requests. Thing is I watched as it created 40,000+ entries in 10 minutes - not too much else was being reported. I have a PML file of the time frame if you would like to see it.
BTW, I restarted the app pool running DNN. Pageblaster made similar entries for about a minute and now things look back to normal.
| | | |
|
John Mitchell
Posts:2779
| | 09/26/2007 12:42 PM |
| That is normal for the cache to build up. Those entries can really add up on a process monitor because it logs every access as you can see above.
Opened SUCCESS , A SUCCESS , SUCCESS Read Metadata, SUCCESS Read Metadata, NO MORE FILES Read Metadata, etc.
All of those (and probably more ) were most likey from just one file operation.
If you have a lot of traffic then there could be a lot of files being accessed from a cold start.
| | | |
|
llong
Posts:83
| | 09/26/2007 12:51 PM |
| | I figured there would be some overhead to building the cache - I have over 1000 active pages/tabs and 4000+ visitors a day. What would trigger it to rebuild everything like that? Is there a way I can get this to happen in the middle of the night so my site doesn't slow to a crawl? It defeats the purpose of the cache if it takes up all the resources and people can't see the pages. | | | |
|
John Mitchell
Posts:2779
| | 09/26/2007 1:22 PM |
| My guess is that you had several things starting out cold. Is your database on the same server? I wouldn't expect your CPU to be at 100% for 10 minutes everytime the application starts, but that is not the worst thing that can happen. If you throttled it down to 50% then it would take much longer to get the job done, so it's better to let the processor work as hard as it needs to in my opinion.
DNN itself is pretty hard on resources at startup, but the PageBlaster disk cache files really help to limit that since the database doesn't have to be called for every page request if there is a page cached to disk.
As a test, while everything is running smoothly, try recycling the App Pool and see if it causes the same CPU spike.
| | | |
|
llong
Posts:83
| | 09/27/2007 1:16 PM |
| John,
I'm saddened to report the server had severe trouble last night. From about 9:30pm to 2:45 am the CPU stayed at near 100%. Not good :(
Clearly this was not the case of rebuilding the cache. I set the App Pool to recycle automatically at 2:50 so my guess is that was the only thing that stopped it. It appears the larger memory simply meant it could go longer at 100%.
Worse is I only have the information from the application event viewer. It simply shows me when the CPU is over 99% so I don't have any details about the underlying causes. All I can assume is it would look similar to the 10 minutes I mentioned above. That incidence also stopped when I restarted the w3wp process.
Recycling the app pool results in some heavy activity but it barely hits 100% and lasts about 2 minutes - what I would except from normal operation.
Can you offer any suggestions? | | | |
|
John Mitchell
Posts:2779
| | 09/27/2007 1:27 PM |
| That may be much harder to track down.
I don't think it is related to how much memory is being used. If it is a run-away process that also consumes memory then it will run out of memory in a short amount of time.
I recently helped someone track down a simular issue that turned out to be Search Indexing running from a scheduled process.
I would recommend disabling Search Indexing and any other scheduled process to see if the problem goes away.
You can disable them by logging in as host and going to the Host > Scheduler page. | | | |
|
llong
Posts:83
| | 09/27/2007 1:57 PM |
| The search log is littered with failures but I can do without searched so I'm going to try, run on app start, run every 24 hours. This should mean it will run every day at 2:45am when I have set for the app pool to recycle. No?
Alternative, I could buy a different search provider. Do you have any recommendations?
On another topic. While I was reviewing the event log and everything was work fine, this appeared:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 9/27/2007 11:14:45 AM
Event time (UTC): 9/27/2007 6:14:45 PM
Event ID: 718ef519281946af913355cdabd92c75
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/w3svc/35/Root-1-128353904590301206
Trust level: Full
Application Virtual Path: /
Application Path: D:\my_path\dotnetnuke\Web\
Machine name: SPRY
Process information:
Process ID: 4460
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
Exception type: FileNotFoundException
Exception message: Could not find file 'c5249016632dd8bca4defa71c5cea4e8.pb.webinfo'.
Request information:
Request URL: http://www.mydomain.com/mypage.aspx
Request path: /mypage.aspx
User host address: 124.120.130.87
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE
Thread information:
Thread ID: 9
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileInfo.get_Length()
at Snapsis.HttpModules.PageBlaster.Utils.CleanFolders(String path, TimeSpan cleanupInterval, Boolean recurse)
at Snapsis.HttpModules.PageBlaster.Utils.CleanFolders(String path, TimeSpan cleanupInterval, Boolean recurse)
at Snapsis.HttpModules.PageBlaster.Utils.CleanFolders(String path, TimeSpan cleanupInterval, Boolean recurse)
at Snapsis.HttpModules.PageBlaster.Config.Settings.GetConfig(HttpApplication app)
at Snapsis.HttpModules.PageBlaster.PageBlasterModule.Init(HttpApplication app)
at System.Web.HttpApplication.InitModulesCommon()
at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)
at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)
at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)
at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
Server is still fine but is this normal or not? | | | |
|
John Mitchell
Posts:2779
| | 09/27/2007 2:06 PM |
| Yes, setting your search indexing schedule up like that may help, but I would turn it off completely until everything runs normal for a day, and then turn it back on to see if everything still runs normal.
That error in the event log indicates you may have more than one process running the same website, which would also give you the other problems you are experiencing. We already confirmed you are not running a web garden. Do you have more than one IIS website pointing at the same DNN instance? | | | |
|
llong
Posts:83
| | 09/27/2007 2:21 PM |
| I have one IIS server and one app pool per DNN instance. I have a back up instance of the instance in question but it doesn't get traffic. How can I ensure it isn't crossing paths?
I just confirmed it has:
seperate files,
seperate IIS website,
seperate machine keys,
seperate DB, and
seperate app pool.
I have other instances of DNN running but they are old and don't have PageBlaster installed. | | | |
|
John Mitchell
Posts:2779
| | 09/27/2007 3:30 PM |
| It's possible that you are getting enough traffic to kick off multiple http applications (internal to the website). You can use the attached newer version that will not write those errors to your event log if you like. All you need to do to upgrade is put the attached DLL in your /bin folder. |
 Attachment: Snapsis.HttpModules.PageBlaster.zip
| | |
|