Making updates to citibank-statement-to-sheets

For months now I have been using my app to import Citibank statements into Google sheets without any issues. This weekend, however, I imported my latest statement and opened the sheet to discover something had went wrong and the import had put values into the wrong cells.  “This should be pretty straightforward to fix” I thought. “I just need to add a test for this case.” Of course it wasn’t quite as simple as I hoped.

Updating to TypeScript version 2

I opened Visual Studio Code and the first thing I saw was the message

Version mismatch! global tsc (1.8.10) != VS Code's language service (2.2.1). Inconsistent compile errors might occur.

While only a warning, this did prompt me to update the version of TypeScript used in the project and also to question whether I needed to have it installed globally.

Since the application is compiled with webpack and I don’t run the tsc command directly anymore I didn’t see a reason to have it installed globally, however the official docs always seem to provide the install command with the global flag, so I was unsure if it would work. I did uninstall it globally and had no issues from doing so.

With TypeScript only installed locally I then updated it by uninstalling it locally, increasing the version in my package.json file to the latest version and then running npm install again. As part of the upgrade in my tsconfig.json I changed the target from “es6” to “es2015”. It did appear to work as “es6”, but according the documentation that is not a supported value.

Migrating from typings to @types

Now that I was using TypeScript version 2, I could add definition files directly from npm. I uninstalled the typings package and removed it from my package.json. I also deleted the typings.json file and all the typings I had imported with typings. Then I installed the definition files through npm like so “npm install @types/pdf --save-dev”.

After doing this I was getting TypeScript compile errors when building the app where it was complaining that it could not find types that were in the newly imported definition files e.g.

ERROR in ./src/PdfScraping/PdfScraper.ts
(34,38): error TS2304: Cannot find name 'PDFDocumentProxy'.

A hunch told me that I probably needed to update ts-loader and a google search found this StackOverflow question that confirmed it. Updating fixed the error and the application was compiling once again.

I did find that VS Code works better with the latest TypeScript version and @types. Before it wouldn’t always find definitions and would then incorrectly display errors in the UI, but I haven’t had that happen since updating. I did need to close and reopen VS Code though to get it to pick up new definition files as I added them.

Sidetracked by a Chrome behaviour change

After getting the application compiling again, I ran the tests to check that everything still behaved correctly. They failed!

This stumped me for a long time. The output I was getting from karma contained

Chrome 56.0.2924 (Windows 10 0.0.0): Executed 0 of 0 ERROR (0.002 secs / 0 secs)

which showed that it wasn’t picking up the tests. Because I had not run the tests before making any changes I thought that it was something I had changed that was causing this. It was a while before I noticed this message in the console in Chrome when running the tests

Refused to execute script from 'http://localhost:3334/base/src/Statements/Parsing/StatementParserTests.ts' because its MIME type ('video/mp2t') is not executable.

This appears to be a change in Chrome’s behaviour and jtson provided a fix in this Github issue. Adding

mime: {
 "text/x-typescript": ["ts"]
},

into karma.conf.js fixed the issue. This was not the last frustrating problem I encountered.

Could not test locally

I also had issues setting up localhost as an authorised JavaScript origin in the Google Developer Console for the application. This was annoying as it prevented me from running the application locally to test. I attempted to get it to work for a while, but in the end gave up and decided to release it and test it live, hoping that the updates didn’t cause any issues.

Unfortunately those plans never work out and the application is currently broken with a script error. I will have to work out how to get localhost working again so that I can test and fix the issue locally.

Update: A day later

Well, it turns out I simply didn’t wait long enough for the new authorised origin to be accepted. I tried setting it again, waited a bit longer and then tried out testing locally and it worked. I then also tried the application out again and it worked too. Following that I tested out the deployed version and it also worked!

I’m not sure what happened, maybe it was tiredness (it was late at night) causing me to not pay enough attention. Oh well, the upside is it is working and I don’t need to make any further changes. I will have to keep an eye out and see if it plays up again.

Making updates to citibank-statement-to-sheets

Explaining webpack loader setup using ts-loader

I am working on a personal project that uses PDF.js and the easiest way to get this setup was to use the webpack distribution. I’m also using TypeScript and wanted to set up webpack to compile my .ts files so that I only had to run webpack to build my application. This post is to address a few things I found, as a complete beginner to webpack, weren’t explained well. If you’re looking for a complete guide with all the commands to run I recommend this post by James Brantly.

The documentation for ts-loader provides this example for webpack.config.js

module.exports = {
 entry: './app.ts',
 output: {
   filename: 'bundle.js'
 },
 resolve: {
   // Add `.ts` and `.tsx` as a resolvable extension.
   extensions: ['', '.webpack.js', '.web.js', '.ts', '.tsx', '.js']
 },
 module: {
   loaders: [
     // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
     { test: /\.tsx?$/, loader: 'ts-loader' }
   ]
 }
}

The key parts are the resolve and module sections.

resolve.extensions

The resolve.extensions property is required so webpack can find any TypeScript files referenced that need compiling. For example the following import in app.ts

   import PdfScraper from "./PdfScraping/PdfScraper";

might be compiled to something like the following JavaScript (in this case using AMD modules)

   const PdfScraper_1 = require("./PdfScraping/PdfScraper");

When webpack resolves this require (or another type of import) it uses the extensions property to determine the type of file that resolve it. Using the earlier example config webpack would be able to resolve any of PdfScraping/PdfScraper, PdfScraping/PdfScraper.webpack.js, PdfScraping/PdfScraper.web.js, PdfScraping/PdfScraper.ts, PdfScraping/PdfScraper.tsx or PdfScraping/PdfScraper.js.

Without .ts in the extensions webpack won’t resolve TypeScript files from imports and you will end up with errors like

ERROR in ./src/index.ts
Module not found: Error: Cannot resolve 'file' or 'directory' ./PdfScraping/PdfScraper in D:\repos\matthewrwilton\csv-citibank-statement\src
@ ./src/index.ts 4:21-56

module.loaders

This section configures loaders to handle different file types e.g. the example is configuring ts-loader to handle any .ts or .tsx files. The loader only applies if the condition in the test property matches the file name.

The webpack documentation also mentions that there are exclude and include properties that can be configured, but fails to explain them well. These exist to provide additional conditions for whether a loader should apply. For example the ts-loader could be configured for all .ts files, but then you could setup an exclude property to ignore .ts files in a certain directory because you want to use a different loader for those. I am not sure what happens if both the include and exclude conditions are true and I couldn’t find any examples using and explaining the include configuration for a loader. This is certainly part of webpack that could be better defined.

Explaining webpack loader setup using ts-loader

TypeScript to ES2015 (ES6) to ES5 via Babel 6

Note: ES2015 is the official name for ES6, but to provide a better distinction between it and ES5 (the previous version) I’ll refer to it as ES6 from here on.

 

I upgraded One Model from Babel 5 to Babel 6 recently and wanted to share the experience as there were a few issues.

Why use TypeScript and Babel?

While some think that TypeScript and Babel are exclusive choices, we think there is room for both. TypeScript’s strong typing helps reduce errors by detecting some at compile time and it provides an improved development experience in Visual Studio.

However compiling to ES5 with TypeScript is not as complete as it is with Babel. Some modern language features, such as Array.prototype.find, cannot be compiled to ES5 with TypeScript.

Babel 6 is also designed as a platform for JavaScript tooling (because of its ability to understand source code as an abstract syntax tree), not just as a transpiler from ES6 to ES5. Having Babel 6 in our process allows us to leverage these tools.

Understanding the key changes from Babel 6 to Babel 5

Babel 5 was just a tool, you gave it some ES6 and it gave you some ES5 back. We used it in a gulp pipeline and it only required a single npm package gulp-babel. Babel 6 is a platform and extremely configurable, it won’t compile ES6 to ES5 unless you configure it to.

Configuring Babel 6 involves selecting the plugins and/or presets you want to apply. Plugins apply granular changes to your code e.g. transpiling for of loops to ES5. Presets are a grouping of plugins that you would likely want to apply together e.g. the es2015 plugin groups all of the ES6 language features and will transpile ES6 code to ES5.

What are these other stage-0, stage-1 etc. presets?

The time between ES5 and ES6 was too long (6 years), so ECMAScript updates are becoming more agile. It is now basically a release train process, where every year there will be a language update. Features move through stages until they get to stage 4, at which point they are ready and should be included in the next release. For more information about this see Dr. Axel Rauschmayer’s write up on the TC39 process.

The stage presets allow you to target the latest language features with stage-0 including the least mature features (those most likely to change). There are (at the time of writing) no stage 4 features, so the stage 3 preset contains the most mature of the proposed new language features.

We currently have our TypeScript compile target set to ES6 and are not using features beyond that and I am not sure how well TypeScript handles compiling features newer than ES6. For the moment we are just using the es2015 preset. This GitHub issue looks to be the best place to start investigating handling newer features in TypeScript.

Compiling ES6 modules to other module types with Babel 6

In Babel 5 there was a module option that you specified to convert ES6 modules to other types. This option has been removed and you now choose the module type with a plugin. In our case we want AMD modules and we use the es2015-modules-amd plugin.

Issues with module imports

The first problem I ran into was

$ is not a function

We were importing jQuery into our files using a wildcard like so

   import * as $ from “jquery”;

If you look at the JavaScript generated by Babel 6, all dependencies go through a function to wrap them as required by their import type. This was a change from what was generated by Babel 5. For wildcard imports they go through a function called _interopRequireWildcard, which if the import is not an ES6 module creates a new object with all the same properties of the import except that the import becomes assigned to the .default property of the new object. For jQuery this means that calls such as $.each work, but $(“some-selector”) will not.

The solution was to update our jQuery imports to

   import $ from “jquery”;

but this also required changing the export in the jQuery TypeScript definition to

   declare module “jquery” {

       export {$ as default};

   }

Refer to DefinitelyTyped issue 6315 and TypeScript issue 4337 for further information. We also had to do this for Numeral.js, toastr, and bloodhound in typeahead.js.

Update: as of TypeScript 1.8 the changed export produces the following error.

Cannot re-export name that is not defined in the module.

The recommendation is to use a local variable declaration to capture the global name before exporting it. Changing the jQuery export to the following resolves the error.

   declare module "jquery" {

       const $: JQueryStatic;

       export {$ as default};

   }

Issues with dependencies/imports in script that doesn’t go through Babel

Babel 6 also exports modules in a differently to Babel 5. This is not an issue in any code that is generated by Babel, it generates the correct code to handle it, but becomes a problem if you have code that does not get generated by Babel.

We have small sections of JavaScript at the bottom of views to import (using require.js) and construct the root module for the view. These all had to be updated to refer to the .default property of the dependency.

Getting Array.prototype.find to work

This function was probably one of the ES6 features we wanted to use the most! Unfortunately it doesn’t work straight out of the box with Babel, but fortunately it is not difficult to get working. All that is required is to add the babel-polyfill script to the web pages (before any other Babel generated code).

We did this by installing the babel-polyfill module, copying the file from the dist folder into our application and then including it as a script in our layout view.

That’s it

Those were the issues I had upgrading to Babel 6. Finally I have a snippet of the gulp task that compiles our JavaScript.

TypeScript to ES2015 (ES6) to ES5 via Babel 6

Optional interfaces in TypeScript

Does the following TypeScript look like it will compile to JavaScript without errors?

Spoiler: it does. Because the option1 and option2 properties of IOptions are optional, it is valid to assign a variable to a IOptions type that does not provide those properties.

I realised this recently when working with the DefinitelyTyped highcharts definition file and wondered why I could assign a HighchartsAxisOptions[] variable to a property of type HighchartsAxisOptions. All of the properties in the HighchartsAxisOptions type are optional and so it was valid.

In these cases we aren’t benefitting from any compiler checks, because there aren’t any that can be done. The benefits we do get are a specification of what properties are may be provided and the types of those properties, which can be used as suggestions or auto-complete options in an IDE. .

Optional interfaces in TypeScript