Making updates to citibank-statement-to-sheets

For months now I have been using my app to import Citibank statements into Google sheets without any issues. This weekend, however, I imported my latest statement and opened the sheet to discover something had went wrong and the import had put values into the wrong cells.  “This should be pretty straightforward to fix” I thought. “I just need to add a test for this case.” Of course it wasn’t quite as simple as I hoped.

Updating to TypeScript version 2

I opened Visual Studio Code and the first thing I saw was the message

Version mismatch! global tsc (1.8.10) != VS Code's language service (2.2.1). Inconsistent compile errors might occur.

While only a warning, this did prompt me to update the version of TypeScript used in the project and also to question whether I needed to have it installed globally.

Since the application is compiled with webpack and I don’t run the tsc command directly anymore I didn’t see a reason to have it installed globally, however the official docs always seem to provide the install command with the global flag, so I was unsure if it would work. I did uninstall it globally and had no issues from doing so.

With TypeScript only installed locally I then updated it by uninstalling it locally, increasing the version in my package.json file to the latest version and then running npm install again. As part of the upgrade in my tsconfig.json I changed the target from “es6” to “es2015”. It did appear to work as “es6”, but according the documentation that is not a supported value.

Migrating from typings to @types

Now that I was using TypeScript version 2, I could add definition files directly from npm. I uninstalled the typings package and removed it from my package.json. I also deleted the typings.json file and all the typings I had imported with typings. Then I installed the definition files through npm like so “npm install @types/pdf --save-dev”.

After doing this I was getting TypeScript compile errors when building the app where it was complaining that it could not find types that were in the newly imported definition files e.g.

ERROR in ./src/PdfScraping/PdfScraper.ts
(34,38): error TS2304: Cannot find name 'PDFDocumentProxy'.

A hunch told me that I probably needed to update ts-loader and a google search found this StackOverflow question that confirmed it. Updating fixed the error and the application was compiling once again.

I did find that VS Code works better with the latest TypeScript version and @types. Before it wouldn’t always find definitions and would then incorrectly display errors in the UI, but I haven’t had that happen since updating. I did need to close and reopen VS Code though to get it to pick up new definition files as I added them.

Sidetracked by a Chrome behaviour change

After getting the application compiling again, I ran the tests to check that everything still behaved correctly. They failed!

This stumped me for a long time. The output I was getting from karma contained

Chrome 56.0.2924 (Windows 10 0.0.0): Executed 0 of 0 ERROR (0.002 secs / 0 secs)

which showed that it wasn’t picking up the tests. Because I had not run the tests before making any changes I thought that it was something I had changed that was causing this. It was a while before I noticed this message in the console in Chrome when running the tests

Refused to execute script from 'http://localhost:3334/base/src/Statements/Parsing/StatementParserTests.ts' because its MIME type ('video/mp2t') is not executable.

This appears to be a change in Chrome’s behaviour and jtson provided a fix in this Github issue. Adding

mime: {
 "text/x-typescript": ["ts"]
},

into karma.conf.js fixed the issue. This was not the last frustrating problem I encountered.

Could not test locally

I also had issues setting up localhost as an authorised JavaScript origin in the Google Developer Console for the application. This was annoying as it prevented me from running the application locally to test. I attempted to get it to work for a while, but in the end gave up and decided to release it and test it live, hoping that the updates didn’t cause any issues.

Unfortunately those plans never work out and the application is currently broken with a script error. I will have to work out how to get localhost working again so that I can test and fix the issue locally.

Update: A day later

Well, it turns out I simply didn’t wait long enough for the new authorised origin to be accepted. I tried setting it again, waited a bit longer and then tried out testing locally and it worked. I then also tried the application out again and it worked too. Following that I tested out the deployed version and it also worked!

I’m not sure what happened, maybe it was tiredness (it was late at night) causing me to not pay enough attention. Oh well, the upside is it is working and I don’t need to make any further changes. I will have to keep an eye out and see if it plays up again.

Advertisements
Making updates to citibank-statement-to-sheets

Reflection on citibank-statement-to-sheets

A little while ago I finished the next version of my statement parser to push the data into Google Sheets. The following is my reflection on the project.

Ditching the jQuery habit

When I started building this application I took the approach of starting simple and only adding things as they were required, but I didn’t apply this line of thought to jQuery. For whatever reason, despite seeing web development without jQuery being promoted around the web, I didn’t think to start without it. The changes to this version of the application added some complexity so I decided to use Knockout to abstract all the UI interaction. In the process of changing the application I realised jQuery could be removed completely.

Maybe the use of jQuery was too ingrained, because using Knockout and a view model is definitely simpler and I should have taken that approach from the start. Next time I will. I also should have used templates and created multiple view models for the different application states to break it up and remove all the state flag properties from it.

Disappointment with Google’s APIs

At work we are pulling data from APIs and I get to see a lot of poorly designed ones. The worst I have had to experience so far has been ADP’s where their online documentation is almost completely out of date and they distribute the updated information by emailing a PDF! Coming from that I had high hopes for Google’s API for working with Drive and Sheets, but unfortunately reality did not live up to expectation.

I found the API difficult to work with and in the end regretted trying to use the JavaScript version. I think it would have been easier to just have sent AJAX requests directly. The documentation for the API is not clear on the expected structure of the parameters for many requests and there were several points where I had to repeatedly call a function and look at the error returned to determine how to build the required parameter. This was an underwhelming experience and having to wrestle with this was a point of demotivation in the project.

If I was using the a strongly typed language this wouldn’t have been a problem, but for JavaScript it was. I believe all the different libraries and the online documentation is generated from a single source, but there needs to be more online documentation (at least for JavaScript). They could also provide TypeScript definitions. I did start to create some, but decided that I wasn’t going to be able to commit to finishing them.

Reflection on citibank-statement-to-sheets

Reflection on citibank-statement-to-csv project

Every month when I get my credit card statement I like to go through it and categorise the transactions (a bit like in you can in Netbank), so that I can see where and how I am spending my money. The statements come as PDFs and getting the data into spreadsheets involved copying and pasting data from the PDF into Notepad++, running a few regular expressions (setup as a macro) to format the data and then copying and pasting that into the spreadsheet. It wasn’t a difficult process, but I thought it would be fun to try and automate it, even if it that turned out like xkcd 1319.

The application was completed last week and is available here with source code here. The following is my short reflection on the project.

Architecture and language

An entirely client side JavaScript application that runs in the browser is definitely a winner. What can’t you do in a browser now. It also meant I could make the app easily available through github.io.

I’ve mentioned that I am a fan of TypeScript, but usually in regards to having strong typing. I also like the improved tooling support. I could write JavaScript without intellisense etc. if required, but why when better things are available? Computers exist to make our lives easier and free us to do better things so why not let IDEs support us to develop software. I did try out Sublime for developing this app, but I found it a bit lacking. I probably didn’t have it setup properly, but configuring an IDE was not something I wanted to waste time on.

Time taken

The 4 months from first commit (Feb 11th) to last (Jun 24th) is more than I would have expected at the start. I think the biggest factor in this was that I found learning new libraries and tools hindered my momentum early on. From the start I decided to use this as an opportunity to use new tools and libraries, which was good, but it was frustrating especially when it seemed to take 2 hours of research to make 5 minutes actual progress.

The main offenders here were webpack and pdf.js, I found the documentation of these pretty unhelpful at times. For webpack I had look for the way other people configured it and dig into what it was actually doing and for pdf.js I read through all the provided examples and built a prototype to check it could do what I wanted. Using TypeScript also meant that setting up webpack, jasmine and karma were all a little bit trickier. I found this boilerplate helpful.

Webpack

Once I had webpack configured it worked well. The continuous test runner was very quick, by the time I had changed to the console after saving a change the tests had completed. This fast feedback is something I now miss in my normal C# and Visual Studio development environment.

TDD

I took the inside out approach to code the StatementParser and CsvConverter classes, but didn’t create any tests for PdfScraper. I probably should have created some dummy PDF files and approached it from a TDD perspective as well.

Styling

Definitely not a strong point of mine. Making something look like a design is easy, but coming up with an appealing design is challenging. It looks alright now, but that was several hours on its own (yes, even for how simple it is). I decided to use material design concepts and referenced Materialize for CSS.

Browser quirks

I rhetorically asked before what can’t you do in a browser now, but there are still a few quirks. Converting the csv data into a file and downloading it was two lines of code (thanks data URIs), but giving the file a meaningful name required 8 and is a bit of a browser hack that may break with future updates.

Where to now?

When I started the project I was using Excel, but I have since moved to Google Sheets. Google Sheets has a nice API for making changes so the next steps are to fork this project and create a version that adds the data directly into the sheet.

Reflection on citibank-statement-to-csv project

Explaining webpack loader setup using ts-loader

I am working on a personal project that uses PDF.js and the easiest way to get this setup was to use the webpack distribution. I’m also using TypeScript and wanted to set up webpack to compile my .ts files so that I only had to run webpack to build my application. This post is to address a few things I found, as a complete beginner to webpack, weren’t explained well. If you’re looking for a complete guide with all the commands to run I recommend this post by James Brantly.

The documentation for ts-loader provides this example for webpack.config.js

module.exports = {

entry: './app.ts',

output: {

filename: 'bundle.js'

},

resolve: {

// Add `.ts` and `.tsx` as a resolvable extension.

extensions: ['', '.webpack.js', '.web.js', '.ts', '.tsx', '.js']

},

module: {

loaders: [

// all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`

{ test: /\.tsx?$/, loader: 'ts-loader' }

]

}

}

The key parts are the resolve and module sections.

resolve.extensions

The resolve.extensions property is required so webpack can find any TypeScript files referenced that need compiling. For example the following import in app.ts

   import PdfScraper from "./PdfScraping/PdfScraper";

might be compiled to something like the following JavaScript (in this case using AMD modules)

   const PdfScraper_1 = require("./PdfScraping/PdfScraper");

When webpack resolves this require (or another type of import) it uses the extensions property to determine the type of file that resolve it. Using the earlier example config webpack would be able to resolve any of PdfScraping/PdfScraper, PdfScraping/PdfScraper.webpack.js, PdfScraping/PdfScraper.web.js, PdfScraping/PdfScraper.ts, PdfScraping/PdfScraper.tsx or PdfScraping/PdfScraper.js.

Without .ts in the extensions webpack won’t resolve TypeScript files from imports and you will end up with errors like

ERROR in ./src/index.ts
Module not found: Error: Cannot resolve 'file' or 'directory' ./PdfScraping/PdfScraper in D:\repos\matthewrwilton\csv-citibank-statement\src
@ ./src/index.ts 4:21-56

module.loaders

This section configures loaders to handle different file types e.g. the example is configuring ts-loader to handle any .ts or .tsx files. The loader only applies if the condition in the test property matches the file name.

The webpack documentation also mentions that there are exclude and include properties that can be configured, but fails to explain them well. These exist to provide additional conditions for whether a loader should apply. For example the ts-loader could be configured for all .ts files, but then you could setup an exclude property to ignore .ts files in a certain directory because you want to use a different loader for those. I am not sure what happens if both the include and exclude conditions are true and I couldn’t find any examples using and explaining the include configuration for a loader. This is certainly part of webpack that could be better defined.

Explaining webpack loader setup using ts-loader

TypeScript to ES2015 (ES6) to ES5 via Babel 6

Note: ES2015 is the official name for ES6, but to provide a better distinction between it and ES5 (the previous version) I’ll refer to it as ES6 from here on.

 

I upgraded One Model from Babel 5 to Babel 6 recently and wanted to share the experience as there were a few issues.

Why use TypeScript and Babel?

While some think that TypeScript and Babel are exclusive choices, we think there is room for both. TypeScript’s strong typing helps reduce errors by detecting some at compile time and it provides an improved development experience in Visual Studio.

However compiling to ES5 with TypeScript is not as complete as it is with Babel. Some modern language features, such as Array.prototype.find, cannot be compiled to ES5 with TypeScript.

Babel 6 is also designed as a platform for JavaScript tooling (because of its ability to understand source code as an abstract syntax tree), not just as a transpiler from ES6 to ES5. Having Babel 6 in our process allows us to leverage these tools.

Understanding the key changes from Babel 6 to Babel 5

Babel 5 was just a tool, you gave it some ES6 and it gave you some ES5 back. We used it in a gulp pipeline and it only required a single npm package gulp-babel. Babel 6 is a platform and extremely configurable, it won’t compile ES6 to ES5 unless you configure it to.

Configuring Babel 6 involves selecting the plugins and/or presets you want to apply. Plugins apply granular changes to your code e.g. transpiling for of loops to ES5. Presets are a grouping of plugins that you would likely want to apply together e.g. the es2015 plugin groups all of the ES6 language features and will transpile ES6 code to ES5.

What are these other stage-0, stage-1 etc. presets?

The time between ES5 and ES6 was too long (6 years), so ECMAScript updates are becoming more agile. It is now basically a release train process, where every year there will be a language update. Features move through stages until they get to stage 4, at which point they are ready and should be included in the next release. For more information about this see Dr. Axel Rauschmayer’s write up on the TC39 process.

The stage presets allow you to target the latest language features with stage-0 including the least mature features (those most likely to change). There are (at the time of writing) no stage 4 features, so the stage 3 preset contains the most mature of the proposed new language features.

We currently have our TypeScript compile target set to ES6 and are not using features beyond that and I am not sure how well TypeScript handles compiling features newer than ES6. For the moment we are just using the es2015 preset. This GitHub issue looks to be the best place to start investigating handling newer features in TypeScript.

Compiling ES6 modules to other module types with Babel 6

In Babel 5 there was a module option that you specified to convert ES6 modules to other types. This option has been removed and you now choose the module type with a plugin. In our case we want AMD modules and we use the es2015-modules-amd plugin.

Issues with module imports

The first problem I ran into was

$ is not a function

We were importing jQuery into our files using a wildcard like so

   import * as $ from “jquery”;

If you look at the JavaScript generated by Babel 6, all dependencies go through a function to wrap them as required by their import type. This was a change from what was generated by Babel 5. For wildcard imports they go through a function called _interopRequireWildcard, which if the import is not an ES6 module creates a new object with all the same properties of the import except that the import becomes assigned to the .default property of the new object. For jQuery this means that calls such as $.each work, but $(“some-selector”) will not.

The solution was to update our jQuery imports to

   import $ from “jquery”;

but this also required changing the export in the jQuery TypeScript definition to

   declare module “jquery” {

       export {$ as default};

   }

Refer to DefinitelyTyped issue 6315 and TypeScript issue 4337 for further information. We also had to do this for Numeral.js, toastr, and bloodhound in typeahead.js.

Update: as of TypeScript 1.8 the changed export produces the following error.

Cannot re-export name that is not defined in the module.

The recommendation is to use a local variable declaration to capture the global name before exporting it. Changing the jQuery export to the following resolves the error.

   declare module "jquery" {

       const $: JQueryStatic;

       export {$ as default};

   }

Issues with dependencies/imports in script that doesn’t go through Babel

Babel 6 also exports modules in a differently to Babel 5. This is not an issue in any code that is generated by Babel, it generates the correct code to handle it, but becomes a problem if you have code that does not get generated by Babel.

We have small sections of JavaScript at the bottom of views to import (using require.js) and construct the root module for the view. These all had to be updated to refer to the .default property of the dependency.

Getting Array.prototype.find to work

This function was probably one of the ES6 features we wanted to use the most! Unfortunately it doesn’t work straight out of the box with Babel, but fortunately it is not difficult to get working. All that is required is to add the babel-polyfill script to the web pages (before any other Babel generated code).

We did this by installing the babel-polyfill module, copying the file from the dist folder into our application and then including it as a script in our layout view.

That’s it

Those were the issues I had upgrading to Babel 6. Finally I have a snippet of the gulp task that compiles our JavaScript.

 

TypeScript to ES2015 (ES6) to ES5 via Babel 6

Optional interfaces in TypeScript

Does the following TypeScript look like it will compile to JavaScript without errors?

 

Spoiler: it does. Because the option1 and option2 properties of IOptions are optional, it is valid to assign a variable to a IOptions type that does not provide those properties.

I realised this recently when working with the DefinitelyTyped highcharts definition file and wondered why I could assign a HighchartsAxisOptions[] variable to a property of type HighchartsAxisOptions. All of the properties in the HighchartsAxisOptions type are optional and so it was valid.

In these cases we aren’t benefitting from any compiler checks, because there aren’t any that can be done. The benefits we do get are a specification of what properties are may be provided and the types of those properties, which can be used as suggestions or auto-complete options in an IDE. .

Optional interfaces in TypeScript