Excluding node_modules folder from ASP.NET compilation

We started to make use of using Github URLs as npm dependencies (make sure to reference specific commits to ensure repeatable builds) to replace some JavaScript files that we had manually added to the project. Unfortunately one of the repositories we added contains an invalid razor view file (.cshtml), which causes razor view compilation to fail with the following error.

/temp/node_modules/ace-builds/demo/kitchen-sink/docs/razor.cshtml(6): error ASPPARSE: Encountered end tag "a" with no matching start tag.  Are your start/end tags properly balanced?

Razor view compilation failure caused by the node_modules folder is a bit of a recurring issue I have had. In the past I had been able to solve the problem by upgrading npm, but that was not going to work this time. What I needed to do was to get the ASP.NET compiler to ignore the node_modules folder.

The view compilation was configured in the .csproj file using the AspNetCompiler task e.g.

This results in a command like the following being executed during the build process.

C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_compiler.exe -v temp -p D:\repos\onemodel\webapp

If you run aspnet_compiler.exe with the help toggle you will see that there is a flag (-x) that can be used to exclude directories. Frustratingly there is no way to provide values to this flag using the provided AspNetCompiler task. Instead you need to use the Exec task to call aspnet_compiler.exe so you can pass in a value for node_modules. Replacing the AspNetCompiler task with Exec ends up like the following example.

The downside of this solution is that it is reliant on the aspnet_compiler executable always being in the same location, however that could probably be dynamically determined with some more MSBuild configuration.

Advertisements
Excluding node_modules folder from ASP.NET compilation

ASP.NET UI Testing With Selenium in 2017: 3. Page Objects and using WebDriver

This is the third post in a series on creating automated UI tests for an ASP.NET app using C# and Selenium. My goal is to highlight and explain the key pieces required starting with Selenium WebDriver in 2017.

Last post I took a bit of a tangent from WebDriver and instead looked at starting the application automatically when tests are run. This time I have added some more realistic UI tests on a mock login page in the web app in the SeleniumExamples repo. To reduce code duplication and make the tests less brittle I have used the page object pattern, and to wait for the login request/response I used the WebDriverWait class.

Up until now I have not required the Selenium.Support nuget, but this week I have added it in as it provides classes for waiting and creating page objects.

Page Objects

Page objects are not specific to Selenium, but are a pattern for decoupling the logic for interacting with a UI element from your tests. This reduces the brittleness of UI tests, making updating your tests for UI changes quicker.

There are plenty of good resources available explaining page objects. Martin Fowler’s bliki is a must read, and the Selenium wiki has a good explanation and example.

In the PageObjects namespace in Selenium.Support there are classes to help simplify page object creation. Unfortunately there does not seem to be a lot information on them (the page in the Selenium wiki is not relevant for .NET). I found the best place to learn and get some examples were the test classes in the Selenium source code, but hopefully the page object class for the login page is a better example.

Waiting

When UI testing with Selenium there will be times when you need to wait for things such as responses to requests or animations to complete. A basic, but inefficient, approach would be to use Task.Delay (or an equivalent) to cause the test to wait. WebDriver provides a better approach for waiting where you set a max timeout and it polls the condition until it is reached, or the time expires.

For .NET there are 2 types of waits available in WebDriver – implicit and explicit. You may see another type, fluent waits, mentioned around the internet, but this is not available in .NET. As far as I can tell fluent waits behave the same as explicit waits, but are declared in a fluent manner.

The Selenium docs provide some details on implicit and explicit waits, but I found this answer on StackOverflow a better source of information. To summarise implicit waits are a one size fits all solution that only applies to finding elements, while explicit waits provide more flexibility and allow waiting for any condition to be met. As the official documentation mentions, implicit and explicit waits should not be mixed as they can cause unreliable wait times.

I used explicit waiting in the login page object to wait for the simulated login request/response to complete. I haven’t tried, but using implicit waits with the page object pattern seems like it would be difficult and I think having the explicit waits in the code to document where there is expected to be a delay is beneficial.

Wrap up

That’s it for this week. From here the plan is to go and create some tests and then look at libraries like Seleno to see how they make it easier.

ASP.NET UI Testing With Selenium in 2017: 3. Page Objects and using WebDriver

ASP.NET UI Testing With Selenium in 2017: 2. Running your application

This is the second post in a series on creating automated UI tests for an ASP.NET app using C# and Selenium. My goal is to highlight and explain the key pieces required starting with Selenium WebDriver in 2017.

In the last post I introduced WebDriver and created a bare minimum example for getting WebDriver running and using it to open the Google homepage in Chrome. The Selenium docs provides examples of the WebDriver API including fetching pages, locating elements and getting text values. These are worth a look to get an understanding of some of the functionality available.

I was planning on providing some WebDriver examples this week, but there is no point in me rehashing the ones provided in the docs. There are a few things worth highlighting, but to do this I will need an example web app and this week I am going to explain how to run the application before executing the tests.

Testing your application with Selenium (and xUnit)

You could start your web application manually before executing tests, but it does become tedious, so it makes sense to setup the tests to execute the application automatically.

A lot of the code I added to start the application comes from Michael Whelan’s post “Black-Box Testing ASP.Net: IIS Express and Selenium WebDriver”, which in turn comes from Seleno (which Michael contributes to). Take a look at Seleno as it will simplify your UI testing. I have not introduced it into my examples, because I wanted to build up an understanding of how to UI test without a particular framework, however in a real UI test solution I would use Seleno.

This week I added a test (Run_Application_And_Navigate_To_Home_Page in _2_Running_A_Web_App) that opens the homepage of the example web app and checks for the element containing “Hi!”. I’ve kept the test simple and haven’t added any asserts or waits, which would there would normally be, but the test will still fail as an exception is thrown if the element is not found.

 

 

The code for starting the web app is located in the WebAppFixture class. It uses the Process object to start an IIS Express process running the example web application. As mentioned before it is based on code from Michael Whelan and Seleno.

 

 

To run this code before the tests, I have setup WebAppFixture as a test collection fixture. This is xUnit specific, but other test frameworks will have similar ways of configuring code to run before tests. This requires an intermediate class (WebAppFixtureTestCollection) to configure the test collection.

 

 

 

Wrap up

That’s all for this post. I had started some more specific examples with WebDriver, but realised I needed an example web app for the tests. That led to automating the startup of it, which became the focus for this week.
Thanks again to Michael Whelan and the other Seleno contributors. Next post I intend to add some more specific examples now that I have an example web application for the tests.

ASP.NET UI Testing With Selenium in 2017: 2. Running your application

ASP.NET UI Testing With Selenium in 2017: 1. Introduction to WebDriver

In this blog series I am going to work through creating automated UI tests for an ASP.NET app using C# and Selenium. I was motivated to write this because I struggled to find relevant and accurate information online, which is I think is just a result of how long Selenium has been around and improved over time. My goal is to highlight and explain the key pieces required starting with Selenium WebDriver in 2017.

What is WebDriver? (WebDriver vs. RemoteControl)

WebDriver is the tool we are going to use to automate interacting with the UI. It is pretty powerful and can be used for any browser automation task, not just UI testing. It relies on each browser’s native support for automation and requires a driver for each browser. The official WebDriver documentation is actually pretty good, even if it seems like it might be out of date (ignore the disclaimer at the top).

RemoteControl (RC) was the original version of Selenium, but was replaced by WebDriver (the wikipedia article provides some history). RC injected custom JavaScript to perform the automation, required running Selenium Server and did not support dynamic pages as well as WebDriver does. That should be the extent of what you need to know about RC and you are safe to ignore any info or guides that refer to it.

Which nugets to use?

This was a point of confusion for me initially that nowhere explained. There are multiple official nuget packages, what do they all do and which ones do I need?

The ones you will require are Selenium.WebDriver and Selenium.Support. WebDriver is the WebDriver API (pretty straightforward) and Support provides better ways of interacting with the API for C#.

You can ignore the Selenium.RC nuget and also Selenium.WebDriverBackedSelenium as it provides WebDriver functionality but with the RC API. It exists to transition code using RC to WebDriver, without it to be rewritten to work with the WebDriver API.

Using WebDriver

To go along with this blog series I have created a SeleniumExamples repo on GitHub which will have code that I refer to. At the time of this post there is a very basic example of using WebDriver with the driver for Chrome to go to the Google homepage in the Open_Google_Chrome method of the _1_Bare_Minimum_Web_Driver_Example class.

To setup the project I added xUnit and Selenium.WebDriver through nuget.

xUnit is my personal preference to run tests and any other test framework, e.g. NUnit, would also work. If you are using xunit, don’t forget the Visual Studio runner.

Selenium.WebDriver gives us access to the WebDriver API. The project is not doing anything advanced at this stage, so I didn’t need Selenium.Support.

Since I decided to use Chrome, I needed to add the Chrome driver. There is a 3rd party nuget package that adds it, but I decided to download it and add it into a Dependencies directory in the project. I wanted to do this a bit out of habit (at One Model we follow this pattern for other required executables) and to make sure I could control where the driver gets added so that I can make them all live in the same folder (when I add others).

I also decided to set the project to copy the driver into the output (bin) directory on compile. I did this in case I want to deploy the test project elsewhere in the future to run tests, and this way all that is required is copying the entire bin directory. This might end up being unnecessary, but did not hurt to do now.

To create an instance of ChromeDriver requires that chromedriver.exe is somewhere in the computer’s PATH, or that you pass the full directory of where it is located as an argument. I prefer that the repo can be cloned and run without any system configuration, so I took the approach of providing the directory as an argument.

From there the code to open the browser and navigate to Chrome is pretty straight forward.

driver.Navigate().GoToUrl("https://www.google.com.au");

Wrap Up

That’s it for this post. In the next post I add an example web application for testing on and have it start automatically before the UI tests run.

ASP.NET UI Testing With Selenium in 2017: 1. Introduction to WebDriver

Redirecting HTTP requests to use HTTPS in IIS behind an Elastic Load Balancer

There are plenty of examples of using the URL Rewrite module to redirect HTTP requests to use HTTPS instead in IIS, however they all seem to rely on checking the {HTTPS} IIS server variable to determine if the request used HTTPS or not. If your IIS server is behind an Elastic Load Balancer in AWS HTTPS requests are converted to HTTP at the load balancer and the previous examples do not work.

Instead of checking the {HTTPS} variable you can instead look for the custom header X-Forwarded-Proto that Elastic Load Balancer adds with the protocol used between the client and the load balancer. You can configure this behaviour by adding the following rule into the <system.webServer> <rewrite> <rules> section of your web.config.

 

Redirecting HTTP requests to use HTTPS in IIS behind an Elastic Load Balancer

Revisiting ASP.NET compiler error caused by node_modules and Visual Studio

The first time I saw this issue was prior to npm version 3 being installed with Node.js. I wrote about it then and it continues to be one of my most popular postsThe latest version of Node.js includes version 3 of npm (and has for a while now), which stores modules in a flat structure to fix the Windows file name limit. I thought this would be the end of this

ASPRUNTIME: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

ASP.NET compiler error, but it has occurred since so it is time to revisit it.

The latest appearances have been on developer’s machines, not the build server. Our build server uses npm version 3 and has not had any issues since I updated it to that. Investigating the errors revealed that once again the node_modules were nested, but the developer’s had the latest version of Node.js installed and running npm proved that it was version 3.

The problem was that Visual Studio had installed the node_modules using its own version (older) of npm. The solution in these cases was to delete the node_modules folder and issue the “npm install” command manually through a command prompt.

It’s frustrating that Visual Studio is still using an older version of npm, especially since the flattening of the node_modules folder was done to benefit Windows users. If you wish to still install modules through Visual Studio you may want to investigate http://jameschambers.com/2015/09/upgrading-npm-in-visual-studio-2015/ and http://scottaddie.com/2015/08/16/npm-vs-windows-max_path-limitation/.

Revisiting ASP.NET compiler error caused by node_modules and Visual Studio

ASP.NET compiler error caused by node_modules

Update: I have revisited this issue, because Visual Studio still uses an old version of npm and installs the modules in a nested structure. You can edit the version of npm Visual Studio uses or run the npm install commands yourself to prevent this issue.

Problem

Recently our build server (AppVeyor) encountered an error while compiling our application.

ASPRUNTIME: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

Initially I was confused, because the build had succeeded locally, so I began researching to determine what had happened. First I googled the error to see if any of the top search results were similar circumstances. Luckily for me John Reilly had blogged about the issue and that also made me remember previously reading about node module nesting causing issues. We have some node modules installed and it seemed a possible culprit. I updated my local modules and when I rebuilt the project I encountered the error too, confirming the node modules as the cause.

Attempted Solution

My first thoughts on how to resolve the error was to configure the ASP.NET compiler to ignore the node_modules directory. The node_modules directory is not required as part of the ASP.NET compile process (for us at least), so it seemed sensible that the compiler should ignore it. Unfortunately I did some research and I could discover a possible way of doing this.

Solution

After trying to have the compiler ignore the node_modules directory, I decided to upgrade to npm version 3. I knew that version 3 featured a flatter dependencies structure, which would fix the problem of long file paths being caused by nested dependencies. Be aware that upgrading npm on Windows involves some manual configuration, see here. After I upgraded and reinstalled the modules the build was succeeding again locally.

Configuring the build server to use npm 3 was fairly straight forward. The build server was already scripted to call “npm install”, I simply had to update the script to install npm 3 beforehand. The install script ended up becoming as follows.

 

Solution Analysis

Upgrading npm resolved the build error, but it didn’t really resolve the problem. Even though we specify the versions of the modules we depend on those modules can have their own dependencies and may not specify the exact version they depend on. This potential variance in builds is undesirable. It may result in errors such as this scenario initially occurring on the build server or worse, the build could be successful with some subtle variance in the results produced.

I happened to discus this scenario with Joshua Wulf and he mentioned that he’d heard a talk that recommended that all node modules be committed into your project’s source control. I haven’t done any further research on this yet, but I think this is something we will have to take up. The build process be repeatable (i.e. the same output should be produced every time the same commit is built) and using npm to fetch the node modules breaks this.

Additional Research

I actually did some additional investigation, in between discovering the node modules were causing the build error and resolving the error. I wanted to know why the ASP.NET compiler was even looking at the node_modules directories. First I enabled diagnostic logging of MSBuild as shown in the following screenshot.

diagnostic msbuild

With the additional diagnostic data I could see that the error occurred in the MvcBuildViews target. The diagnostic data also gave me the aspnet_compiler.exe command that was being run. I took this command and ran it manually adding the “-errorstack” option to it as well. This gave me the following stack trace.

error ASPRUNTIME: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

[PathTooLongException]: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
at System.IO.PathHelper.GetFullPathName()
at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
at System.IO.Path.GetFullPathInternal(String path)
at System.Web.Hosting.MapPathBasedVirtualPathEnumerator..ctor(VirtualPath virtualPath, RequestedEntryType requestedEntryType)
at System.Web.Hosting.MapPathBasedVirtualPathCollection.System.Collections.IEnumerable.GetEnumerator()
at System.Web.Compilation.BuildManager.PrecompileWebDirectoriesRecursive(VirtualDirectory vdir, Boolean topLevel)

at System.Web.Compilation.BuildManager.PrecompileWebDirectoriesRecursive(VirtualDirectory vdir, Boolean topLevel)
at System.Web.Compilation.BuildManager.PrecompileAppInternal(VirtualPath startingVirtualDir, IEnumerable`1 excludedVirtualPaths)
at System.Web.Compilation.BuildManager.PrecompileApp(VirtualPath startingVirtualDir, IEnumerable`1 excludedVirtualPaths)
at System.Web.Compilation.BuildManager.PrecompileApp(ClientBuildManagerCallback callback, IEnumerable`1 excludedVirtualPaths)
at System.Web.Compilation.BuildManagerHost.PrecompileApp(ClientBuildManagerCallback callback, List`1 excludedVirtualPaths)
at System.Web.Compilation.BuildManagerHost.PrecompileApp(ClientBuildManagerCallback callback, List`1 excludedVirtualPaths)
at System.Web.Compilation.ClientBuildManager.PrecompileApplication(ClientBuildManagerCallback callback, Boolean forceCleanBuild)
at System.Web.Compilation.ClientBuildManager.PrecompileApplication(ClientBuildManagerCallback callback)
at System.Web.Compilation.Precompiler.Main(String[] args)

I’ve cut out a several levels of recursive calls to the BuildManager PrecompileWebDirectoriesRecursive method. From the stack trace I started to look at the source code for some of the classes and methods on referencesource, but I decided I was getting sidetracked by my curiosity and start working on solving the error before I could understand why the ASP.NET compiler looks at the node_modules directory. If anyone knows what the compile was doing, I’d be interested to know.

ASP.NET compiler error caused by node_modules