Large File Uploading in ASP.NET102
Pages:
1
|
LiquidLayer private msg quote post Address this user | |
| Large File Uploading in ASP.NET Old post I found but we are looking for best case config seetings for ASP.net with Windows Server Enterprise 2008 , Comments welcome on the following "old school" post. Published 11 May 04 10:40 AM | mhawley I posted this to the ASP.NET forums back in November of 2002, and, too much of my dismay, still is an issue and people are still having problems figuring this out. I don't really know how many times I've helped people with this problem, and just recently I had to do the same. So, for my own personal reference (mainly), and to share the information, I decided to re-post it in my blog... First the good... To upload large files, and not receive the DNS Error or the page stopping while uploading, we found several settings between the machine.config and the web.config files that you need to modify. There are specifically 3 places, 2 of which can be overwritten in your web.config file. In your web.config, add a line under your system.web <httpRuntime executionTimeout="54000" maxRequestLength="512000" /> where execution timeout is in seconds, and maxRequestLength is in KB. executionTimeout, is basically the amount of time a thread will continue to run, and accept data by IIS/.NET. maxRequestLength, is the total amount of data that can be sent through HTTP Post to the server. The default is 4MB (4096)...and is generally set low so that your server will not be overwhelmed by possible DoS attacks. In your machine.config, modify responseDeadlockInterval to equal the same amount of time for executionTimeout. responseDeadlockInterval, is basically the amount of time that the Client's browser and Server will continue to communicate. Every several minutes or so, the server polls the client, asking if they have any more information to send, if they do not receive anything back after several times, then the server stops the current thread and all communication is stopped. This is the cause of the DNS Error you may see sometimes. These 3 changes will allow you to successfully upload large files. Now...the bad... Memory deallocation is a major issue, that has no current solution. Whenever you upload files via HTTP Post, the file is stored in the memory of the aspnet_wp.exe process, and never deallocates completely (if your lucky a few MB gets released). One of the config settings for .NET processes, allows them to utilize 60% of physical memory on the server, at which point the process is recycled and all execution is stopped. Whenever a new upload is started, though, some memory is de-allocated, but not enough compared to memory that was used in prior uploads. Microsoft is aware of this problem, and has assured us that it will be fixed with some upcoming releases of the .NET framework. Some solutions that they suggested to us, was to use Classic ASP, Third Party Components, or Custom Built ISAPI filters. Because of our solution we were using this in, we could do none of the three, so we topped the server out at 2GB of RAM. This has provided us with some breathing room if several people start uploading huge files, and gives us enough time to restart IIS if we start nearing 1.3GB of RAM being used by aspnet_wp.exe. |
||
| Post 1 • IP flag post | ||
|
|
LiquidLayer private msg quote post Address this user | |
| Asp.Net Configuration option in Visual Studio. A full list of settings and comments can be found in machine.config.comments usually located in \Windows\Microsoft.Net\Framework\v2.x\Config |
||
| Post 2 • IP flag post | ||
|
|
LiquidLayer private msg quote post Address this user | |
| Source: http://www.motobit.com/help/scptutl/pa31.htm Upload - Files and forms with large/huge size. Article Member of ScriptUtils.ASPForm | Changes | Purchase | Download Upload - Files and forms with large/huge size. Brief overview of work with very large files. ASPForm configuration and IIS setting for huge uploads or urlencoded forms with big fields. Main idea for big uploads Upload components (including ASPNet HTMLInputFile) usually use memory to store temporary upload data. It is a problem, if you are uploading very large files. Huge-ASP upload works with unique hi-performance technology, which lets you read upload source data in small data blocks (Request.BinaryRead) and store the data on temporary disk folder - this is automatic Huge-ASP upload work, of course. You can specify size of a source data block and maximum amount of memory to store data (to store small uploads in memory). For example, HTMLInputFile from ASPNet takes around 5MB of memory to upload 1MB file. You will need at least 500 MB of a free memory if you have 10 clients uploading one 10MB file with ASPNet HTMLInputFile upload control. Huge-ASP upload takes less than 1MB (one megabyte) of memory to do the same action (and 100MB disk size to store temporary data, of course). Important points for huge uploads with Huge-ASP upload 1. Server.ScriptTimeout and ScriptUtils.ASPForm.ReadTimeout property. Big uploads take much time to complete. Next table contains approximate time values to upload files: File size Line speed [kb/s] Upload time Upload time [s] 1MB 33 00:04:33 273 10MB 33 00:45:27 2727 100MB 33 07:34:33 27273 1MB 64 00:02:05 125 10MB 64 00:20:50 1250 100MB 64 03:28:20 12500 1 000MB 64 34:43:20 125000 1MB 256 00:00:35 35 10MB 256 00:05:52 352 100MB 256 00:58:36 3516 1 000MB 256 09:45:56 35156 1MB 8000 00:00:01 1 10MB 8000 00:00:11 11 100MB 8000 00:01:52 113 1 000MB 8000 00:18:45 1125 2 000MB 80000 00:03:45 225 Huge-ASP upload has a great performance - it lets you really accept 2GB files in 200second on 100Mb/s Ethernet line. But you have to set ScriptTimeout value to have enough time to transfer data and some time more to process uploaded files on server-side. If you want to accept up to 1GB uploads from min. 256k leased lines, set at least: Server.ScriptTimeout = 40000 There is no problem to set a big value for ScriptTimeout - Huge-ASP upload checks client connection during upload and server-side upload process is stopped when client interrupts upload. You can also change this value by some dynamic algorithm with parameter of source form size. For example, on 10Mb/s Ethernet line (real 8000kb/s, 1MB/s): Dim ScriptTimeout, TotalBytes TotalBytes = Request.TotalBytes 'TotalBytes/1000000 - time to transfer data 'TotalBytes/5000000 - time to save data on server-side ScriptTimeout = 100 + TotalBytes/1000000 + _ TotalBytes/5000000 Server.ScriptTimeout = ScriptTimeout 2. ScriptUtils.ASPForm.SizeLimit Size limit Second important value is a ScriptUtils.ASPForm.SizeLimit property. Huge-ASP upload has no limit set by default. It means you can upload and accept any files up to 2GB of size (2^31-1B; Huge-ASP upload can accept up to 4GB, but you will probably not have a client which let's you upload files bigger than 2GB ...). Internet Explorer (from v 5.0) and Netscape Navigator let's you upload files with size up to 2GB also. Be careful to have this value without limit - upload scripts are possible point for DOS attacks. If you have public upload script, set lower value rather. You can also create some dynamic algorithm to limit upload size based on logged on users, local IP addresses, etc. Form.SizeLimit = GetLimitFromDB(LOGON_USER) Please see ScriptUtils.ASPForm.Upload - Monitor and handle upload state/result article to handle "upload size exceeds the limit" state. 3. ScriptUtils.ASPForm.TempPath Temporary path Temporary upload data are stored in predefined folder. Please, set a folder with adequate free size. For example, if you have up to 10 clients uploading up to 1GB file, you need to have 10GB temporary folder to store temporary form data (and second 10GB free space on destination disk/store). 4. IIS 6 and AspMaxRequestEntityAllowed The AspMaxRequestEntityAllowed property specifies the maximum number of bytes allowed in the entity body of an ASP request. The default value for the AspMaxRequestEntityAllowed property is 204800 bytes (200kB). You can get the current value using adsutil.vbs: cscript adsutil.vbs get w3svc/ASPMaxRequestEntityAllowed Please increase this value if you are uploading large files in IIS6 You can increase this value using set command (AppID 1, limit 200MB): cscript adsutil.vbs set w3svc/1/ASPMaxRequestEntityAllowed 204800000 See also: You may receive a 403 error when you use an ASP request to upload a large file AspMaxRequestEntityAllowed on MS site 5. IIS 7 and maxAllowedContentLength maxAllowedContentLength value specifies the maximum length of content in a request, in bytes. The default value is 30 000 000 (30MB). Please increase this value using web.config for IIS7. See also: maxAllowedContentLength on MS site 6. URLScan and other security utilities URLScan has its own limit for POST data. The value name is MaxAllowedContentLength and it is located in WINDOWS\system32\inetsrv\urlscan\urlscan.ini, default value is 30000000 (30MB). You shlould increase the value to process large uploads if you have URLScan on your site. See also MaxAllowedContentLength urlscan. 7. Progress bar Use progress bar every time you want to accept big uploads. Client browsers have no progress indicator by default. And clients-people know probably nothing about this problem. You will have many people, which will try to upload same files again and again (they will think nothing changed after "upload button" click) - without progress bar. Huge-ASP progress bar is open immediately after Form submit button and client can see that something happened - and can see progress of upload. maxRequestLength - Maximum request length exceeded - IIS6 Symptoms When you try to use the ASPForm object to upload a large file with ASP and IIS6, the file may not be uploaded. This problem occurs because the default value for the maxRequestLength parameter in the <httpRuntime> section of the Machine.config file is 4096 (4 megabytes). As a result, files that are larger than this value are not uploaded by default. Resolution To resolve this problem, use one of the following methods: In the Machine.config file, change the maxRequestLength attribute of the <httpRuntime> configuration section to a larger value. This change affects the whole computer. In the Web.config file, override the value of maxRequestLength for the application. For example, the following entry in Web.config allows files that are less than or equal to 8 megabytes (MB) to be uploaded: <httpRuntime maxRequestLength="8192" /> Web resources http://support.microsoft.com/default.aspx?scid=kb;EN-US;295626 http://www.google.com/search?q=maxRequestLength Other references ScriptUtils.ASPForm.MaxMemoryStorage ScriptUtils.ASPForm.ChunkReadSize ScriptUtils.ASPForm.Upload - Monitor and handle upload state/result |
||
| Post 3 • IP flag post | ||
|
|
LiquidLayer private msg quote post Address this user | |
| Source: http://www.websupergoo.com/helpupload50/source/2-tech_notes/3-web.config.htm Web.Config A number of configuration settings become very relevant when performing uploads. These can be changed by adding sections or values into the Web.Config file. A typical configuration section is given below. <system.web> <httpModules> <add name="Progress" type="WebSupergoo.ABCUpload5.ProgressModule, ABCUpload5, Version=5.3.0.0, Culture=neutral, PublicKeyToken=1f89539196ce5fbf"/> </httpModules> <compilation> <assemblies> <add assembly="ABCUpload5, Version=5.3.0.0, Culture=neutral, PublicKeyToken=1f89539196ce5fbf" /> </assemblies> </compilation> <httpRuntime maxRequestLength="1048576" executionTimeout="3600" /> <sessionState timeout="60" /> ... httpModules sub-tag The Progress Module is required for the Pure HTML Progress Bar, for GigUpload and for Corruption Autofix functionality. The Progress Module is a .NET HTTP Module designed to intercept page requests and preprocess them before passing them on to ASP.NET. To integrate the Progress Module into your ASP.NET application you need to add it using a line in the httpModules subsection of the web.config file. This is shown in the example web.config file above. compilation sub-tag To integrate and use ABCUpload objects into your ASP.NET application you need to add a reference to the assembly (stored in the GAC) using a line in the compilation subsection of the web.config file. This is shown in the example web.config file above httpRuntime maxRequestLength This attribute is used to limit the size of uploads by rejecting any which exceed a certain limit. The limit refers to the total size of the HTTP upload in KB (approximately equal to the sum of all the files being upload). You should set a sensible limit here to stop malicious visitors using up your bandwidth by uploading excessively large files. If the size of an upload is too great the server will refuse to accept it. Because the server is refusing the request the uploading browser will report that the submission page is not available. This is a client side error message rather than a server side error message and it means that you cannot normally provide a sensible error message to users if they submit files which are too large. However using ABCUpload you can report back a sensible error message via a progress window. If you believe that visitors may attempt to perform uploads greater than the maximum you should use the progress bar and the note on the progress window will inform your visitor why their upload has been rejected. In the example web.config file above the maxRequestLength is set to 1 GB. httpRuntime executionTimeout The execution time-out refers to the number of seconds an ASP.NET page is given before the operation is assumed to have failed and the page terminated. If you are uploading a large file the code that is receiving the transfer may time out before the file has been completely uploaded. In the example web.config file above the executionTimeout is set to one hour. sessionState timeout The session time-out refers to the number of minutes before the user session is aborted. If a large file is being uploaded it is desirable to maintain the session state. The session time-out should always be longer than the amount of time you expect uploads to take. Note that this value is only relevant if you have session state enabled. In the example web.config file above the sessionState is set to one hour. processModel responseDeadlockInterval The responseDeadlockInterval is specified in the machine.config file and defaults to three minutes. It specifies the time interval after which the process will be restarted if no responses have been made and requests are still queued. Under ASP.NET requests that take longer than the deadlock interval can cause problems under some very specific circumstances. Sometimes a client may make multiple page requests which ASP.NET may queue one after the other. If the first request in the queue takes longer than the deadlock interval then ASP.NET will mistakenly assume the whole process is deadlocked and restart it. These types of situation are very unusual and are more commonly encountered in test environments than real world ones. For this situation to arise a client must make multiple uploads simultaneously. At least two uploads must take longer than the timeout. The client must make other page requests at the same time which IIS must decide to queue (whether it decides to queue them or not depends on whether IIS thinks they form part of the same session). These queued requests must not time out or be dismissed on the client side. Neither the client nor any other visitors must make other aspx page requests during this time. |
||
| Post 4 • IP flag post | ||
|
|
LiquidLayer private msg quote post Address this user | |
| Source: http://weblogs.asp.net/jgalloway/archive/2008/01/08/large-file-uploads-in-asp-net.aspx Large file uploads in ASP.NET Uploading files via the FileUpload control gets tricky with big files. The default maximum filesize is 4MB - this is done to prevent denial of service attacks in which an attacker submitted one or more huge files which overwhelmed server resources. If a user uploads a file larger than 4MB, they'll get an error message: "Maximum request length exceeded." Increasing the Maximum Upload Size The 4MB default is set in machine.config, but you can override it in you web.config. For instance, to expand the upload limit to 20MB, you'd do this: <system.web> <httpRuntime executionTimeout="240" maxRequestLength="20480" /> </system.web> Since the maximum request size limit is there to protect your site, it's best to expand the file-size limit for specific directories rather than your entire application. That's possible since the web.config allows for cascading overrides. You can add a web.config file to your folder which just contains the above, or you can use the <location> tag in your main web.config to achieve the same effect: <location path="Upload"> <system.web> <httpRuntime executionTimeout="110" maxRequestLength="20000" /> </system.web> </location> What Happens When I Upload A File That's Too Big? While expanding the upload restriction is a start, it's not a full solution for large file uploads. Milan explains one of the biggest problems with large file uploads in The Dark Side Of File Uploads: It gets really interesting if someone uploads a file that is too large. Regardless of what your maxRequestLength setting mandates, IIS has to guzzle it, and then ASP.NET checks its size against your size limit. At this point it throws an exception. As Milan explains, you can trap the exception, but it's trickier than you'd expect. He talks about overriding Page.OnError and checking for HTTP error code 400 when the error is HttpException, which as he says is less than ideal. At Least Give Me A Warning If we've got a set limit on file upload sizes, we should at least tell our users what it is. Since this is a configurable value which we may change later, the best is to make our file size warning read directly from web.config setting. The best way to do this is to pull back the httpRuntime section as a HttpRuntimeSection object, which isn't too hard given: System.Configuration.Configuration config = WebConfigurationManager.OpenWebConfiguration("~"  ;HttpRuntimeSection section = config.GetSection("system.web/httpRuntime" as HttpRuntimeSection;double maxFileSize = Math.Round(section.MaxRequestLength / 1024.0, 1); FileSizeLimit.Text = string.Format("Make sure your file is under {0:0.#} MB.", maxFileSize); A Real Solution: an HttpModule to Handle File Uploads There are better solutions to handling large file uploads in ASP.NET. A custom HttpHandler can provide a better user experience by displaying upload progress and allowing you to handle a file size problem in a more controlled fashion. Here's a summary from a cursory search: FileUploader.NET (MediaChase, $310 and up) RadUpload (Telerik, $249) NeatUpload (Free, LGPL license) Which others did I miss? One of the longest running threads on the ASP.NET Forums (going back 5 years): HttpHandler or HttpModule for file upload, large files, progress indicator? A Better Solution: a RIA upload component The project which caused me to look into this, Video.Show, was based on ASP.NET and Silverlight 1.0, so we weren't able to take advantage of RIA platforms which would support more advance upload handling. In most cases, though, I'd recommend replacing the FileUpload component with a Silverlight or Flash based file upload control. In addition to a better upload experience, these controls generally look better than the the generic button displayed for the <input type="file"> element which is rendered by the FileUpload control. The input / file element doesn't allow for CSS formatting, although smart CSS hackers always seem to find a way around these things. Although there doesn't seem to be a project / component to support Silverlight uploads yet, Liquid Boy has a nice sample of a Silverlight 1.1 (oops, 2.0) based upload control. I've heard good things about SWFUpload, a Flash and JavaScript based upload system. Developers are responsible for handling a few JavaScript events as well as accepting the file on the server. That can be done pretty easily, as you can see from this ASP.NET sample implementation. Here's a screenshot from one of the SWFUpload online demos: **** IIS 7 NOTE **** What About IIS7? Oh, and there's more thing to worry about. IIS7 has a built-in request scanning which imposes an upload file cap which defaults to 30MB. Again, this is a good feature, but it gets in the way if you're looking to upload files larger than 30MB. Steve Schofield posted about how to change this from the comandline: appcmd set config "My Site/MyApp" -section:requestFiltering -requestLimits.maxAllowedContentLength:104857600 -commitpath:apphost Why is this such a pain? Browsers, and HTML in general, were never designed to handle large uploads gracefully. Jeff Atwood and I discussed this back in September, and he summed up the issue pretty well in his post asking Why Are Web Uploads So Painful? It's disappointing that browser standards have failed us to the point that we need to use browser extension technologies like Flash and Silverlight as a band-aid here. Browsers should support uploads over both HTTP and FTP (it is the file transfer protocol, after all), and should manage the upload in a side or toolbar that allows us to continue browsing without breaking the upload. The Firefox Universal Upload add-on is an example of how this should work out of the box. Historical trivia: In ASP.NET 1.0 and 1.1, the entire file was loaded into memory before being written to disk. There were improvements in ASP.NET 2.0 to stream the file to disk during the upload process. Published Tuesday, January 08, 2008 1:13 AM by Jon Galloway Filed under: ASP.NET, Browsers / Web Development |
||
| Post 5 • IP flag post | ||
Pages:
1This topic is archived. Start new topic?
