ADHD Dev Journal: Module Federation Thoughts

8:52 AM

Using Task management software consistently for more than a couple weeks is hard.

9:02 AM

https://stackoverflow.com/questions/23844761/upstream-sent-too-big-header-while-reading-response-header-from-upstream

https://gist.github.com/magnetikonline/11312172#determine-fastcgi-response-sizes

https://tech.setepontos.com/2018/03/25/control-synology-dsm-services-via-terminal-ssh/

9:55 AM

Webpack Module Federation creates a really interesting opportunity to evaluate code ownership and maintenance in a web application. If we break up any application into small pieces then there has to be tools and processes in place that reliably reassemble those pieces together back into the running application that the user interacts with. The reasons for breaking the code apart into different modules in the first place is important. There’s several reasons:

  1. Understanding. It’s easier to understand and comprehend the scope of a function when it’s been isolated and designed with a specific interface.

  2. Engineer ownership. The actual people that do the work may not want to be slowed down by understanding and running tests for a larger code-base, they may want to write, test and deploy a piece of code that is smaller in scope so they can move faster.

  3. Isolation from side-effects. Depending on the technology and interface definition, the two different code-bases may be less impacted by bugs in each other when they are isolated.

Most of the other reasons I can think of depend on the method of isolation/separation. For instance, in module federation, JS components are loaded from an http endpoint and injected into the host application in the browser at runtime. This means the developer of the host application isn’t able to see and predict the effects of every change that happens in a federated module they are loading. Whereas if the module was loaded a build time, using webpack on the CI/CD server, then the JS code that is loaded will always be knowable and testable by developers before it is run by the user’s browser.

The big drawback of build-time integration is that fixes and updates to the module must be pulled, tested, and approved by a developer before they are deployed to any users. So, runtime integration through webpack module federation gives us back some of the freedom we had on the web before we all started using bundlers and NPM for our dependencies. Back in the day we could choose which dependencies were loaded as always the latest version, or which ones were pinned to a specific version or “fingerprint”.

<html>
    <head>
        <!-- This component will be cached by the browser, but can be updated by whoever can push changes to the CDN, no change to this html file is needed. -->
        <script type="text/javascript" src="cdn.mydomain.com/js/always-latest-component.js"></script>
        <!-- This component has a fingerprint which means if a new version is pushed, then this HTML file must be updated too -->
        <script type="text/javascript" src="cdn.mydomain.com/js/pinned-component.6fe4afd32a.js"></script>
    </head>
</html>

That’s a simplified example of what we’re talking about. It wasn’t an issue when the html file and JS file were maintained and owned by the same team. But, as organizations grow and ownership is divided further and further, the repo and build process responsible for the html file may not be the same as the JS file. This can cause terrible slowdowns when a component developer can’t see their changes in production until the upstream team adopts them and deploys them.

The adoption of webpack and NPM set the default for modern applications to always have every dependency pinned and fingerprinted at build time. This type of bundling and compilation step made the end result more predictable for web developers by simplifying the mental model of where your code comes from.

Changing that model to include runtime dependencies through webpack module federation, modern ES Modules in the browser, or a script tag with a src that wasn’t compiled in webpack will complicate the maintenance and debugging story for an application, while offering certain advantages in letting different developers update and maintain different parts of an application independently.

ADHD Dev Journal: After a long weekend

Getting away with my wife for some relaxation and eating out was truly special. We had some close friends offer to watch the kids for a couple days. It’s been over a year since we’ve gotten to completely unplug and relax like that for more than a couple hours. Our favorite things to do were eating out, and shopping for little decorations to go around the house.


Some articles I’ve been looking at this morning.

https://www.slowboring.com/p/china-speech

wikimedia threats?

hyperkitty

Keybase… belongs to zoom?

handshake

keys.pub

Tasmota

https://github.com/ct-Open-Source/tuya-convert

https://github.com/xoseperez/espurna

https://platformio.org/

Home Ventilation Project

10:44 AM

Time to write up some recommendations for authentication steps.

9:43 PM

I spent a little more time on the self-hosted budgeting app Firefly-iii. Setting it up was not quite as straight-forward as I had hoped. Basically the author is aggressively moving forward with minimum system requirements and forcing PHP 8 for the latest version, which isn’t available on my 8 year old NAS chip architecture. Had to figure out how to roll back, since PHP will still attempt to execute the new code but will fail on newer syntax. Also there are always little gotchas if you don’t keep each element of the application up to date, so since I was running a version of MariaDB that was old, it didn’t install correctly at one point. That troubleshooting process took a while, and then for some reason the CSV-Importer application that is a sort of side-car to the main application is erroring out without showing me the message, and that makes it very difficult to troubleshoot. Without CSV importing I’m not sure it’s worth setting up the budget app at all. I’m certainly not going to enter transactions manually. I’m not even certain I’m willing to import them on a monthly basis based on my past patterns with these types of finance apps.

ADHD Dev Journal: Middle of the Hackathon

The amazing thing about PHP and MySQL web apps is that I can troubleshoot and fix them even though I know so very little about PHP. It’s a really robust ecosystem. I don’t want to manage my own application servers for everything but the PHP applications are the simplest to manage. And searching stackoverflow returns just about everything you could imagine.

It’s good because I staked a part of my business on a self-hosted paid PHP application 5 years ago: Pancake and it’s still going strong and relatively easy for me to fix even when it’s running on an 8 year old Synology DiskStation.

This looks like an interesting budgeting app to try out. Firefly-iii

ADHD Dev Journal: Starting a Hackathon

I need to remember to get up and move and exercise sometime today when I’m starting to lose focus. I woke up late, stayed up late working last night, but it was fun. I’m looking forward to expaining what I did to some of my coworkers.

Kip peering around computer monitor

ADHD Dev Journal: Serverless Next.js

Doing a little catching up on the weekend, trying to get this demo site working for a hackathon. My preferred stack is nextjs + prisma. I’ve always deployed it on Vercel before, but at my current company we use AWS directly and extensively. I am finally trying out the serverless tool for deploying apps, and it has not been the easy straightforwrad experience I hoped for.

First, I needed to sort out the authentication thing, and of course a big corporate IT organization manages their AWS access in a very particular way, which was a little confounding to me coming from the startup - quick and dirty way of doing things.

Second was the fact that I was using a newer configuration thing called “serverless components“ and specifically the one for NextJS which is supposed to take all the config for a nextjs app and abstract it into a single line with good defaults.

Unfortunately I ended up running into an edge case because the client binaries for Prisma are not copied into the deployment package that is uploaded to AWS Lambda. There is an example workaround in this repo config which almost worked. But for some reason I had to copy the client files directly into the root of my deployment directory, otherwise Prisma couldnt find it and I would get errors int the logs like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
ERROR   PrismaClientInitializationError3 [PrismaClientInitializationError]: Query engine binary for current platform "rhel-openssl-1.0.x" could not be found.
This probably happens, because you built Prisma Client on a different platform.
(Prisma Client looked in "/var/task/chunks/query-engine-rhel-openssl-1.0.x")

Searched Locations:

/.prisma/client
/Users/tim/granular/csf-portal-next/node_modules/@prisma/client
/var/task
/var/task/chunks
/var/task/chunks
/tmp/prisma-engines
/var/task/chunks

You already added the platforms "darwin", "rhel-openssl-1.0.x" to the "generator" block
in the "schema.prisma" file as described in https://pris.ly/d/client-generator,
but something went wrong. That's suboptimal.

Please create an issue at https://github.com/prisma/prisma/issues/new
at cb (/var/task/chunks/930.js:36253:17)

[truncated]

Anyway, you see this is a big pain and it requires customizing the serverless config with custom shell scripts. And if there’s one thing I’ve learned it’s that writing shell scripts in .yml files is sign that things are going off the rails in a bad way.

Here’s my serverless config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
myNextApp:
component: "@sls-next/serverless-component@3.3.0"
inputs:
build:
env:
DATABASE_URL: ${env.DATABASE_URL}
postBuildCommands:
- PDIR=node_modules/.prisma/client/;
LDIR=.serverless_nextjs/api-lambda/;
if [ "$(ls -A $LDIR)" ]; then
cp "$PDIR"query-engine-rhel-* $LDIR;
cp "$PDIR"schema.prisma $LDIR/chunks/;
fi;
- PDIR=node_modules/.prisma/client/;
LDIR=.serverless_nextjs/default-lambda/;
if [ "$(ls -A $LDIR)" ]; then
cp "$PDIR"query-engine-rhel-* $LDIR;
cp "$PDIR"schema.prisma $LDIR/chunks/;
fi;

This puts the files in the place that prisma is looking for them on the lambda.

Of course you still need to make sure your project can actually access your database and is in the same VPC with the lambdas. There is no end to the configuration power you have with AWS, but, I heartily recommend just using Vercel and some publicly accessible hosted Postgres Database.

Or maybe I should have just used a serverlss configuraiton that put a Create-React-App build into an S3 bucket and a seperate Express API on a lambda instead of trying to put them together into a single project. My current approach could be an example of too much magic again.

AWS security roles are such a pain to deal with. It’s after midnight now but I feel like I really accomplished something. A NextJS application deployed on lambdas and cloudfront, where the API handlers connect to a postgres database using Prisma DB client. Was that so much to ask for? Apparently it is. Tomorrow I’ll probably just scrap the whole thing and put it on a plain EC2 instance. But this CI deployment script should be super simple compared to what it would take to deploy a whole next app to a EC2 instance. npm ci && npx serverless

ADHD Dev Journal: API Keys

9:24 AM

  • Learned that for Rest endpoints, the Pancacke App API key is passed in through the HTTP Header X-API-KEY

10:43 AM

Absorbing more information about carbon and climate change.

Brighten the Clouds? - don’t forget that it reflects energy back onto the earth that would have been radiated out…

Sequestration of plant material may not actually work if you just bury it.

Olivine seems fine… but it’s impurities are kinda toxic I guess?

Parasols in spaaaaace!

Recyling… not our finest idea

What a mess. Global Analysis Paralysis has set in… and for good reason.

10:56 AM

Discussion of a hackathon project for next week. I’m good at starting new things from scratch.

2:29 PM

Been setting up a new application environment in AWS for the hackathon. Trying to make lots of decisions.

decisions whiteboard

4:32 PM

Always double-check the region in the AWS console. And RDS with Postgres and RDS Aurora with Postgres are two totally different things.

ADHD Dev Journal: Dentistification

Going to the dentist today to fill a cavity.

bear brushing teeth

Spent time trying to write a node CLI tool script to simulate web calls to my time tracking app, and failed to get authorization cookies to stick around. So annoying. I don’t want to build a whole fake authentication lifecycle, but that’s probably the best option since there’s no real API. Well, their API “documentation” doesn’t have any details on auth tokens.

I thought I could add a custom menubar app to my mac that allowed me to track my time and stay on focus better. It’s a classic way of spending time on a productivity tool instead of being productive. It’s amazing that after a few weeks of not doing much coding I have to go back to documentation for things that I’m sure I had memorized before.

Later…

Now I’m reading about magnetic-confinement for fusion. Fusion energy, man! Such an exciting mad-scientist type of thing to think about.

the sun

I want to lay out all the areas of work in my big project on the whiteboard and think about a cascade of issues that need to be solved. I enjoy zooming out on the problems more than writing code some days. If I do that for too long then I swing back and then just want to write code. It’s hard to do any one thing for too long.

Wow that was a really unfocused day. I had such a hard time getting anything useful done. We had a sick kid that messed up our sleep last night. Maybe that was the problem. It’s hard have days like this where my mind wandered so much. I need to find ways to apply my energy towards useful things.

ADHD Dev Journal: Off & On

Great 3-day weekend. Played with my kids, picnicked with neighbors. It’s amazing how much easier it is to enjoy the weekend with this new job compared to working for an early stage startup.

Lots to catch up on today. Feeling a little overwhelmed again.

Over the weekend I read some more about ADD and considered a few different things to try.

  1. Large daily doses of a multi-vitamin like EMP+

  2. A desk bike like the Flexispot, as reviewed on ars.

What about a standing desk? Hmm might need to do some business expenses at the end of the year. Wait… is my LG 5K actually VESA mount compatible? Is the built-in stand tall enough for standing desk configuration?

9:03 AM

I’m very interested in Watermelon DB. I actually tried using Nozbe (they own this github repo) as my task manager like 10 years ago. It was very powerful.

watermelon db logo

So it’s a database for making react-native apps fast that would be slowed down by loading and unloading large persisted caches into Redux stores. Multi-threading, native threads, indexing. What is it using for it’s DB on native? On the web it can use Loki.js.

Is the app I’m building really a offline-first app?

Watermelon does not really take care of syncing. Wait they have flow types? Not typescript? Wat?

So if we want to build the app with strong offline support we need to consider if it needs a full offline database? Doe the use-cases support that? Does data get created and updated on the mobile client that will then be synced to the back-end eventually? And I know we have sync endpoints for reading data but do we have endpoints for bulk updating data that was changed locally?

Checked into PouchDB years ago. I don’t think Couch DB on the server is a viable option for this app. It’s not a great, or feature-full production app database.

11:21 AM

Having trouble caring about work at the moment.

2:12 PM

Looking up information about OAuth and react-native.

2:52 PM

Thinking about digital nomad life in Costa Rica. Apparently cost of living isn’t actually that low any more. New digital nomad visa doesn’t really apply since I would be spending less than 6 months anyway.

3:12 PM

react-native-app-auth

Microsoft Authentication Library (MSAL)

React Native MSAL

4:06 PM

SolidJS - I like that people are still perfecting the front-end functional reactive programming paradigm.

4:19 PM

Oh wow. “Moonfall”. This new Roland Emmerich film looks like exactly the kind of escapist nonsense I want to see.

Dev Journal: Maps in Apps

Yeah I’m posting more gifs. Sorry, not sorry.

friday

Read some more of that book last night. Still not sure what to do about everything.

8:54 AM

Calling the doctor now just to see if my old prescription helps out at all again. One of the problems with medication is that I don’t know if it really works if I skip so many days. I often skip taking any prescription about 1/3 of the days of the week. Consistent habits are not easy for me.

9:05 AM

OK, only on hold for 5 minutes that time. Got things sorted out.

Time to catch up on slack…

9:43 AM

Mushroom growing from start to finish.

10:01 AM

The US Army operated A Small Nuclear Power Plan in Greenland in the 1960s

10:38 AM

Mapbox in react isn’t rendering the base-map tiles. It seems like maybe I’m getting some sort of authentication error using the mapbox api. I’m gonna drop this customized map component and just use the base library components.

11:21 AM

Success! Got the react native app to render a map with the same large GeoJSON object in a webview and compare it with a map rendered a native map component.

video demo

I would say, that on my iPhone SE2 from 2020, the interactions is roughly equally good. The line drawing isn’t as clear in web-view as in the native component.

comparison

I should try to compare raster tile layer performance too.

12:57 PM

Doing some planning for a hackathon in 2 weeks. Gotta figure out the scope of what we will tackle. I’ve done a lot of miro planning for this developer portal already, but don’t want to overwhelm people with everything in my mind. I tend to run on ahead and then I expect people to get annoyed if I’ve thought too far ahead of where we are at in a project — this might be one of those strengths of ADD that I need to learn to capture and not be ashamed of.

1:01 PM

My wife brought me a salad for lunch. Thank you dear.

salad

2:21 PM

Great conversations about frontend architecture. On the surface we were talking about shared state management and offline syncing functionality. One of the classic hard problems is cache invalidation. But more deeply the question came up, how do I help other frontend developers communicate and listen to the other devs that rely on their expertise. What is the wedge that’s drving these developers apart. How do we improve our listening and empathy as an organization…

Dev Journal: React native and webviews

Let’s get this going again.

baby yoda dance

I have some meetings today. I have a call with a client about design work at the end of the day. I still need to call my doctor. I still need to test out a webview in a react native app. Writing these posts is like journaling during the day.

Last night I checked out a book from the libarary about ADHD. I feel like I need to take this seriously and start really paying attention to this side of myself — not just be embarassed and ashamed because of the stigma around having a “disorder”. I could find coping mechanisms and look for opportunities to use my strengths, but I have a deep-seated belief that ADD makes me a bad worker, and it will get me fired.

9:54 AM

Great meeting about our big project, it kept me engaged with problem solving. Now, I think I’m going to try to load the web view in the mobile app like I started yesterday.

10:02 AM

My iPhone doesn’t have VPN access to the necessary dev servers. Need to run the web application container locally so I can do a real tests.

10:49 AM

Distracted with an article about mid-century cryptography machines. How amazing would it be to build a purely mechanical or maybe an electromechanical crypto device.

11:11 AM

Trying to make some progress. Discovering lots of little issues with the container react application. Like some bizarre browser detection feature.

unsupported browser?

I mean this works… technically. So can I get a mapbox map to show up here? If I can get the same map and data to render in both ways then this test will be done.

3:25 PM

I had another good meeting. I like filling up my day with meetings like that, I feel productive despite myself when I can help make decisions and discuss problems. Now it’s time for a little design meeting with my longest-running freelance client. We manage to do some good design work, and it’s almost all on the phone while I work in figma and he watches me. Some designers would probably lose their minds. I feel really proud of the work we did to re-design hundreds of screens of functionality over the last couple of years.

5:24 PM

Did some good work today. Did not call my doctor. Took a short nap over lunch. I never got past that stupid unsupported browser warning. So I’m going to have to yank all of the features out of this web app and just make a really minimal reproduction app that doesn’t have any of the login stuff in it tomorrow. And then I need to figure out some good answers about offline sync options.

Dev Blogging: An Attempt to Maintain Focus

I want to put my effort and focus into my job. I’m finding it difficult lately. For various reasons I’m not feeling very productive, seeing a decreased amount of focus during the day. Yesterday I was able to focus for a few meetings, but the rest of the day I was distracted constantly. Then about 4:30 I felt the energy drain out and my mind felt fuzzy and I was suddenly able to focus. 45 minutes of productivity per day is not enough.

The honesty is difficult. This blog has not been about things like this in a long time. I don’t like the idea of publishing these thoughts forever, they are pretty cringe. At the same time I’ve been watching youtube channels where people do fun projects and also just show the real side of themselves. I don’t think I can do video right now, but I can write and publish a blog. And it might help me when I feel like I need to just get some of the thoughts out of my head. ADHD is difficult to manage and having a place to pour some thoughts out might help.

So that’s this post: a log of my thoughts and accomplishments from one day. An account of what I’m doing to deal with various challenges and tasks.

Today I plan to do the following:

  1. Have several short meetings about current projects.
  2. Edit and Post this file at the end of the day.
  3. Call and set up a Doctor’s appointment.
  4. Create a react-native test application with an embedded webview that can load with injected authentication tokens.

This file will be updated as I proceed.

9:16 AM

Just wrote the above. Listening to Pandora station “Tiësto Radio”.

Installed https://github.com/react-native-webview/react-native-webview in my test project. It’s an expo app. Something weird happened to my project. The start script changed form expo start to react-native start. When I run react-native run-ios it starts up a very slow xcode build process.

I have about 1 hour until my first meeting of the day.

9:29 AM

I did somehow get the simulator to open my project.

expo start --dev-client then click the link in the web UI “Run on iOS Simulator” whick opens a custom URL scheme link on the device.

It shows the app with custom native code inside it. Now can I get it to run on my actual device?

simulator

I want to skip the EAS build service for now. Can I just use the built in Apple tools? Plug in my iphone with USB. Launch XCode. Apple Docs. Add my work apple id to the accounts preferences in XCode. Change the project settings the associate with my work team for apple developer program membership.

Receive error in XCode:

1
2
Failed to register bundle identifier
The app identifier "com.timbly.donewithit" cannot be registered to your development team because it is not available. Change your bundle identifier to a unique string to try again.

Wat? Go make some more coffee. Just opened a new bag of “Late Night” from Methodical Coffee. Dark roast provided by bottomless.

9:54 AM

Worrying about tornadoes.

9:58 AM

If the proof of concept app doesn’t run on an actual device it doesn’t count as working in my mind. It’s important to carry test projects through the lenght of the development cycle. We call them tracer bullets sometimes. There’s not a lot of meat on the bones, but the project needs to traverse the full length of the process to find any gotchas.

Changing the bundle identifier to something else fixed the error in xcode, because I created the project and gave it a bundle identifier before I added my new apple ID that’s associated with a developer account.

10:05 AM

Build errors relating to expo. errors

Attempting to close xcode and open the .xcworkspace file instead

It’s building for longer - so this feels promising.

No, it failed in the node scripts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
+ node /Users/tim/Sites/expo/DoneWithIt/node_modules/react-native/cli.js bundle --entry-file index.js --platform ios --dev true --reset-cache --bundle-output /Users/tim/Library/Developer/Xcode/DerivedData/DoneWithIt-hicxjkqmmrjkcrhavcnkptgegaxi/Build/Products/Debug-iphoneos/donewithit.app/main.jsbundle --assets-dest /Users/tim/Library/Developer/Xcode/DerivedData/DoneWithIt-hicxjkqmmrjkcrhavcnkptgegaxi/Build/Products/Debug-iphoneos/donewithit.app
warning: the transform cache was reset.
Welcome to React Native!
Learn once, write anywhere


events.js:167
throw er; // Unhandled 'error' event
^

Error: EMFILE: too many open files, watch
at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:123:28)
Emitted 'error' event at:
at NodeWatcher.checkedEmitError (/Users/tim/Sites/expo/DoneWithIt/node_modules/sane/src/node_watcher.js:143:12)
at FSWatcher.emit (events.js:182:13)
at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:129:12)
+ [[ true != true ]]
Welcome to React Native!
Learn once, write anywhere


node:events:371
throw er; // Unhandled 'error' event
^

Error: EMFILE: too many open files, watch
at FSEvent.FSWatcher._handle.onchange (node:internal/fs/watchers:204:21)
Emitted 'error' event on NodeWatcher instance at:
at NodeWatcher.checkedEmitError (/Users/tim/Sites/expo/DoneWithIt/node_modules/sane/src/node_watcher.js:143:12)
at FSWatcher.emit (node:events:394:28)
at FSEvent.FSWatcher._handle.onchange (node:internal/fs/watchers:210:12) {
errno: -24,
syscall: 'watch',
code: 'EMFILE',
filename: null
}
Command PhaseScriptExecution failed with a nonzero exit code

Googling

Attempting: brew reinstall watchman

Requires XCode command line tools. Attempting xcode-select --install

too long

11:26 AM

Two meetings. Need a break. Time to make some lunch and then watch some youtube.

12:00 PM

I had a confusing meeting that was full of information that I don’t know what to do with it. I have enough to keep me busy this is a confusing thing to try to figure out.

12:21 PM

I feel so lazy. I can’t do anything.

YouTube is stupid. There’s nothing on there. I don’t want to play games.

Time to call that doctor.

Reinstalling watchman again while on hold.

It’s weird that macOS is version 11 now. It’s been OS X v 10.x for almost 20 years.

3:38 PM

Got sick while on hold. Really sick to my stomach. Had to lay down. Took a long nap. I feel like a failure. Back up and trying again. OK, let’s see if the stupid project will build again.

3:43 PM

OK the development app started on my iPhone but I can’t get it to connect to my launcher.

3:50 PM

OK, got it figured out. It seems to be just my laptop’s IP address and port 8081. And it reloads when I change my JS. Now I can start testing it for real on a real device. Success!

4:05 PM

Love this channel: https://www.youtube.com/watch?v=fNxcOON1VNM


Outcomes

Meetings: partial success, had to skip 2 while sick

Posting this file to my blog… waiting until the next day.

Call and set up a Doctor’s appointment: fail.

Create a react-native test application: Partial Success. Still need to test the webview and the auth tokens.

icon

Speed up react and typescript testing with baretest and es-build

I have to admit that frontend testing is not my favorite thing to spend time on. Frontend tests tend to be fragile and tightly-coupled to non-functional aspects of the code like text labels and classnames. I prefer to write tests for pure functions that calculate or transform data in somewhat complex ways, and most of this happens on the backend. I fully expect developers to write tests for their backend API code, even to the level of testing that ORM models and SQL scripts work as expected.

But still I can’t leave the testing story where I did in the previous post about developing with estrella and es-build. Basically I just left mocha and ts-node as the solution in my sample project. But that’s not a great answer. I have seen that it is a fairly slow and heavy combination of tools. The popular alternative of Jest, Babel and Enzyme may not be fast enough for everyone either.

So I took my philosophy to creating a custom dev and production build script and applied it to the test script. The first step was to identify the minimal testing solution for my needs. Research revealed a new JS testing tool released last year called Baretest.

Minimalism (or when and why to reach for something weird)

Sometimes complex tools are complex for good reasons. Even if I don’t agree with all of the design decisions, I think the React team has built a VDOM library that is going to be the standard for building web applications for a long time to come. React is not minimal like preact – or a dozen other minimal alternatives – but, it creates an abstraction layer that is widely learned and used by many developers. Conceptually very similar to JQuery back in the day, it’s just so widely known that it’s useful as a target for teams to standardize on.

Other times a tool is just the standard because of popularity and there’s room to build something tiny to replace it that meets just the requirements you have. For example React is good, but Create React App installs half of NPM when you start it up. Every dependency and line of code in the dependency is something that theoretically affects the code you have to ship and maintain, including your build-tools. When someone new joins the team, they will have to learn the tools you’ve chosen, so the tools should be small and easy to understand, with a minimum of “magic”.

Baretest is about 50 lines of code with one dependency. It will execute faster and with less overhead than Jest. Instead of babel and tsc we can use es-build, it is faster that babel to compile typescript and JSX into compatible standard JS. Source transpilation and test runner overhead are the two long poles I wanted to address in test startup time for my sample project. Once I’ve seen how small and simple a testing script can be, I will be better informed about reaching for a big library in the future.

Step 1: Write a test with baretest

Here’s my setup helper setup.ts file for creating a dom environment in Node.

1
2
3
4
5
6
7
8
9
import "jsdom-global/register";
import raf from "raf";
export function setup(): HTMLElement | null {
raf.polyfill();
const appDiv = window.document.createElement("div");
appDiv.id = "app";
window.document.body.appendChild(appDiv);
return window.document.getElementById("app");
}

Here are the basic requirements to import

1
2
3
4
5
import assert from "assert";
import { Baretest } from "../src/typings/baretest";
import { render } from "react-dom";
import utils from "react-dom/test-utils";
import { setup } from "./helpers/setup";

Here’s the actual test function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
export default async (test: Baretest): Promise<void> => {
test("should return hello world", () => {
const rootContainer = setup();

render(
AppComponent({ fancyName: "hello world", isVisible: true }),
rootContainer
);

assert.ok(rootContainer);
const header = rootContainer.querySelector("header");
assert.ok(header);
assert.strictEqual(header.textContent, "hello world");
});
};

And finally a test.ts file that imports and runs all the tests we are going to create.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import baretest from "baretest";
import { Baretest } from "../src/typings/baretest";

const test: Baretest = baretest("Render App");

// A big ol' list of tests to set up and run
// import new tests and call them here
import configureAppTest from "./app.spec";
configureAppTest(test);

!(async function () {
await test.run();
process.exit();
})();

Step 2: Compile tests and source code

To run the tests I create another file in the scripts/ directory that will get built once by es-build and then executed whenever I need to run my tests. This script is about 41 lines of code and 2 external dependencies (+2 transitive deps) to replace Jest (76 deps) and Babel (>15 deps [how to even count everything?]).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#!/usr/bin/env node
import { BuildConfig } from "estrella";
import { build, glob, cliopts, scandir, file, watch, basename } from "estrella";
import { spawnSync, spawn, exec, execSync } from "child_process";
import Rimraf from "rimraf";
const testOutputDir = "build/_tests/";

/*
* Builds all src and test files then executes the tests whenever it finishes.
* Watch mode is supported with -watch.
*/
!(async function () {
await Rimraf.sync(testOutputDir + "*");
const files = glob("src/**/*.ts*").concat(glob("tests/**/*.ts"));

// the es-build options we will apply to your typescript
const buildOpts: BuildConfig = {
entry: files,
outdir: testOutputDir,
outbase: "./",
format: "cjs",
bundle: false,
debug: true,
sourcemap: true,
minify: false,
silent: true,
tslint: "off",
onEnd: startTests,
};
await build(buildOpts);
})();

/**
* Spawns a node process to run the tests
*/
async function startTests() {
console.log("🎸 Built Source Code");
const time = new Date().getTime();
const nodeTest = spawn(`${process.execPath}`, [`build/_tests/tests/test.js`]);

nodeTest.stdout.on("data", (data) => {
console.log(`[TEST]: ${data}`);
});

nodeTest.stderr.on("data", (data) => {
console.error(`[TEST ERROR]: ${data}`);
});

nodeTest.on("close", (code) => {
console.log(`🎸 Test run finished in ${new Date().getTime() - time}ms`);
if (!cliopts.watch) {
process.exit();
}
});
}

Step 2: Run the tests

1
2
3
4
5
6
7
...
"test": "npm run build:scripts && node build/_scripts/test.js",
"test:quick": "node build/_scripts/test.js",
"test:watch": "npm run build:scripts && node build/_scripts/test.js -watch",
"build:scripts": "esbuild scripts/*.ts --platform=node --format=cjs --outdir=build/_scripts/"
},
...

I added a couple different scripts to my package.json to make it easy, but I discovered that npm, pnpm, and yarn all add about .4 seconds of overhead to any script execution. So if you really want it to be fast then add executable permissions to your built file (chmod +x) and then run it directly in the terminal ./build/_scripts/test.js.

Here’s a typical run completing in less than 1 second, for example.

Step 3: Add live-watching.

1
./build/_script/test.js -watch

Sorry that was a trick, estrella gives us file watching and incremental builds as part of it’s built-in options – so any code changes are picked up and cause a rerun of tests within about 1 second. This is very powerful! Getting instant feedback on your changes is super helpful for speeding up development time and catching errors. It’s just as powerful as linting for improving code cleanliness. Your IDE probably takes almost a second to update the red squigglies on large JS file, so in theory we get feedback about the behavior and functionality of some of our code as quickly as possible! Now of course if your tests are integration tests they are going to be slower while they deal with more layers of the stack, but for simple unit tests this approach should be effective.

View the Source Code for this sample project.