Debugging with a debugger and breakpoints is recommended rather than using console logs. Chrome provides a built-in debugger for JavaScript-based apps.

This post covers configuring and running a debugger for various Node.js apps in Chrome DevTools.

Prerequisites

• Google Chrome installed

• Node.js app bootstrapped

Setup

Open chrome://inspect, click Open dedicated DevTools for Node and open the Connection tab. You should see a list of network endpoints for debugging.

Run node --inspect-brk app.js to start the debugger, it will log the debugger network. Choose the network in Chrome to open the debugger in Chrome DevTools.

Debugging basics

The variables tab shows local and global variables during debugging.

The step over next function call option goes to the following statement in the codebase, while the step into next function call option goes deeper into the current statement.

Add logs in the debug console via logpoints by selecting the specific part of the codebase so it logs when the selected codebase executes.

Related reading

• Debugging Node.js apps with Visual Studio Code debugger

Boilerplate

Here is the link to the boilerplate I use for the development.

To send e-mails in a production environment, use services like Sendgrid.

Verify the e-mail address on the Sender Management page and create the SendGrid API key on the API keys page.

import nodemailer from 'nodemailer';
(async () => {
  const emailConfiguration = {
    auth: {
      user: process.env.EMAIL_USERNAME, // 'apikey'
      pass: process.env.EMAIL_PASSWORD
    },
    host: process.env.EMAIL_HOST, // 'smtp.sendgrid.net'
    port: process.env.EMAIL_PORT, // 465
    secure: process.env.EMAIL_SECURE, // true
  };
  const transport = nodemailer.createTransport(emailConfiguration);
  const info = await transport.sendMail({
    from: '"Sender" <sender@example.com>',
    to: 'recipient1@example.com, recipient2@example.com',
    subject: 'Subject',
    text: 'Text',
    html: '<b>Text</b>'
  });
  console.log('Message sent: %s', info.messageId);
})();

Demo

The demo with the mentioned example is available here.

Related reading

• Sending e-mails with Mailtrap

Web scraping means extracting data from websites. This post covers extracting data from the page's HTML tags.

Prerequisites

• cheerio package is installed

• HTML page is retrieved via an HTTP client

Usage

• create a scraper object with load method by passing HTML content as an argument

  • set decodeEntities option to false to preserve encoded characters (like &) in their original form

  const $ = load('<div><!-- HTML content --></div>', { decodeEntities: false });

• find DOM elements by using CSS-like selectors

  const items = $('.item');

• iterate through found elements using each method

  items.each((index, element) => {
    // ...
  });

• access element content using specific methods

  • text - $(element).text()

  • HTML - $(element).html()

  • attributes

    • all - $(element).attr()
    • specific one - $(element).attr('href')

  • child elements

    • first - $(element).first()
    • last - $(element).last()
    • all - $(element).children()
    • specific one - $(element).find('a')

  • siblings

    • previous - $(element).prev()
    • next - $(element).next()

Disclaimer

Please check the website's terms of service before scraping it. Some websites may have terms of service that prohibit such activity.

Related reading

• Web scraping with jsdom

Demo

The demo with the mentioned examples is available here.

GitHub provides GraphQL API to create integrations, retrieve data, and automate workflows.

Prerequisites

• GitHub token (Settings → Developer Settings → Personal access tokens)

Integration

Below is an example of retrieving sponsorable users by location.

export async function getUsersBy(location) {
  return fetch('https://api.github.com/graphql', {
    method: 'POST',
    body: JSON.stringify({
      query: `query {
        search(type: USER, query: "location:${location} is:sponsorable", first: 100) {
          edges {
            node {
              ... on User {
                bio
                login
                viewerCanSponsor
              }
            }
          }
          userCount
        }
      }`
    }),
    headers: {
      ContentType: 'application/json',
      Authorization: `Bearer ${process.env.GITHUB_TOKEN}`
    }
  })
    .then((response) => response.json())
    .then((response) => response.data?.search?.edges || []);
}

Demo

The demo with the mentioned example is available here.

Web scraping means extracting data from websites. This post covers extracting data from the page's HTML when data is stored in JavaScript variable or stringified JSON.

The scraping prerequisite is retrieving an HTML page via an HTTP client.

Examples

The example below moves data into a global variable, executes the page scripts and accesses the data from the global variable.

import jsdom from 'jsdom';
fetch(URL)
  .then((res) => res.text())
  .then((response) => {
    const dataVariable = 'someVariable.someField';
    const html = response.replace(dataVariable, `var data=${dataVariable}`);
    const dom = new jsdom.JSDOM(html, {
      runScripts: 'dangerously',
      virtualConsole: new jsdom.VirtualConsole()
    });
    console.log('data', dom?.window?.data);
  });

The example below runs the page scripts, and access stringified JSON data.

import jsdom from 'jsdom';
fetch(URL)
  .then((res) => res.text())
  .then((response) => {
    const dom = new jsdom.JSDOM(response, {
      runScripts: 'dangerously',
      virtualConsole: new jsdom.VirtualConsole()
    });
    const data = dom?.window?.document?.getElementById('someId')?.value;
    console.log('data', JSON.parse(data));
  });

Disclaimer

Please check the website's terms of service before scraping it. Some websites may have terms of service that prohibit such activity.

Related reading

• Web scraping with cheerio

Gumroad allows verifying license keys via API calls to limit the usage of the keys. It can be helpful to prevent the redistribution of products like desktop apps.

Allow generating a unique license key per sale in product settings, and the product ID will be shown there. Below is the code snippet for verification.

try {
  const requestBody = new URLSearchParams();
  requestBody.append('product_id', process.env.PRODUCT_ID);
  requestBody.append('license_key', process.env.LICENSE_KEY);
  requestBody.append('increment_uses_count', true);
  const response = await fetch('https://api.gumroad.com/v2/licenses/verify', {
    method: 'POST',
    body: requestBody
  });
  const data = await response.json();
  if (data.purchase?.test) {
    console.log('Skipping verification for test purchase');
    return;
  }
  const verificationLimit = Number(process.env.VERIFICATION_LIMIT);
  if (data.uses >= verificationLimit + 1) {
    throw new Error('Verification limit exceeded');
  }
  if (!data.success) {
    throw new Error(data.message);
  }
} catch (error) {
  if (error?.response?.status === 404) {
    console.log("License key doesn't exist");
    return;
  }
  console.log('Verifying license key failed', error);
}

Demo

The demo with the mentioned example is available here.

Gotenberg is a Docker-based stateless API for PDF generation from HTML and Markdown files.

To get started, configure Docker compose and run the docker-compose up command.

version: '3.8'
services:
  gotenberg:
    image: gotenberg/gotenberg:7
    ports:
      - 3000:3000

API is available on http://localhost:3000 address.

Run the following commands to generate PDFs

• from the given URL

  curl \
    --request POST 'http://localhost:3000/forms/chromium/convert/url' \
    --form 'url="https://sparksuite.github.io/simple-html-invoice-template/"' \
    --form 'pdfFormat="PDF/A-1a"' \
    -o curl-url-response.pdf

• from the given HTML file

  curl \
    --request POST 'http://localhost:3000/forms/chromium/convert/html' \
    --form 'files=@"./index.html"' \
    --form 'pdfFormat="PDF/A-1a"' \
    -o curl-html-response.pdf

PDF/A-1a format is used for the long-term preservation of electronic documents, ensuring that documents can be accessed and read even as technology changes.

Demo

The demo with the mentioned examples is available here. It also contains examples using Fetch API and Axios package.

Handlebars is a template engine that can create server-side views, e-mail templates, and invoice templates by injecting JSON data into HTML.

Resolving all variables in a Handlebars template is essential to maintain the accuracy of the displayed information and to prevent incomplete content or layout problems.

The following snippet checks for missing variables by overriding the default nameLookup function. It logs a warning for unresolved variables and sets the default value, empty string, in this case.

// ...
const originalNameLookup = Handlebars.JavaScriptCompiler.prototype.nameLookup;
Handlebars.JavaScriptCompiler.prototype.nameLookup = function(
  parent,
  name,
  type
) {
  if (type === 'context') {
    const messageLog = JSON.stringify({
      message: `Variable is not resolved in the template: ${name}`,
      level: WARNING_LEVEL
      // ...
    });
    return `${parent} && ${parent}.${name} ? ${parent}.${name} : (console.log(${messageLog}), ''`;
  }
  return originalNameLookup.call(this, parent, name, type);
};
// ...
const result = Handlebars.compile(template)(data);

Extending package declarations locally is one of the options for outdated package typings.

Create a declaration file .d.ts (e.g., handlebars.d.ts), and put it inside the src directories.

Find the exact name of the package namespace inside the node_modules types file (e.g. handlebars/types/index.d.ts).

Extend the found namespace with your needed properties, like classes, functions, etc.

// handlebars.d.ts
declare namespace Handlebars {
  export class JavaScriptCompiler {
    public nameLookup(parent: string, name: string, type: string): string | string[];
  }
  export function doSomething(name: string): void;
  // ...
}

Notion is a versatile workspace tool combining note-taking, task management, databases, and collaboration features into a single platform.

It also supports integration with Notion content, facilitating tasks such as creating pages, retrieving a block, and filtering database entries via API.

Prerequisites

• Notion account
• Generated Integration token (Settings & Members → Connections → Develop or manage integrations → New integration)
• Notion database ID (open database as full page, extract database ID from the URL (https://notion.so/<USERNAME>/<DATABASE_ID>?v=v))
• Added Notion connection (three dots (...) menu → Add Connections → choose created integration)
• @notionhq/client package installed

Integration

Below is an example of interacting with Notion API to create the page (within the chosen database) with icon, cover, properties, and child blocks.

const { Client } = require('@notionhq/client');
const notion = new Client({ auth: process.env.NOTION_INTEGRATION_TOKEN });
const response = await notion.pages.create({
  parent: {
    type: 'database_id',
    database_id: process.env.NOTION_DATABASE_ID
  },
  icon: {
    type: 'emoji',
    emoji: '🆗'
  },
  cover: {
    type: 'external',
    external: {
      url: 'https://cover.com'
    }
  },
  properties: {
    Name: {
      title: [
        {
          type: 'text',
          text: {
            content: 'Some name'
          }
        }
      ]
    },
    Score: {
      number: 42
    },
    Tags: {
      multi_select: [
        {
          name: 'A'
        },
        {
          name: 'B'
        }
      ]
    },
    Generation: {
      select: {
        name: 'I'
      }
    }
    // other properties
  },
  children: [
    {
      object: 'block',
      type: 'bookmark',
      bookmark: {
        url: 'https://bookmark.com'
      }
    }
  ]
});

Check out Sparxno for more Notion-related content.

Puppeteer is a headless browser for automating browser tasks. Here's the list of some of the features:

• Turn off headless mode

  const browser = await puppeteer.launch({
    headless: false
    // ...
  });

• Resize the viewport to the window size

  const browser = await puppeteer.launch({
    // ...
    defaultViewport: null
  });

• Emulate screen how it's shown to the user via the emulateMediaType method

  await page.emulateMediaType('screen');

• Save the page as a PDF file with a specified path, format, scale factor, and page range

  await page.pdf({
    path: 'path.pdf',
    format: 'A3',
    scale: 1,
    pageRanges: '1-2',
    printBackground: true
  });

• Use preexisting user's credentials to skip logging in to some websites. The user data directory is a parent of the Profile Path value from the chrome://version page.

  const browser = await puppeteer.launch({
    userDataDir: 'C:\\Users\\<USERNAME>\\AppData\\Local\\Google\\Chrome\\User Data',
    args: [],
  });

• Use Chrome instance instead of Chromium by utilizing the Executable Path from the chrome://version URL. Close Chrome browser before running the script

  const browser = await puppeteer.launch({
    executablePath: puppeteer.executablePath('chrome'),
    // ...
  });

• Get value based on evaluation in the browser page

  const shouldPaginate = await page.evaluate(() => {
    // ...
  });

• Get HTML content from the specific element

  const html = await page.evaluate(
    () => document.querySelector('.field--text').outerHTML,
  );

• Wait for a specific selector to be loaded. You can also provide a timeout in milliseconds

  await page.waitForSelector('.success', { timeout: 5000 });

• Manipulate with a specific element and click on some of the elements

  await page.$eval('#header', async (headerElement) => {
    // ...
    headerElement
      .querySelectorAll('svg')
      .item(13)
      .parentNode.click();
  });

• Extend execution of the $eval method

  const browser = await puppeteer.launch({
    // ...
    protocolTimeout: 0,
  });

• Manipulate with multiple elements

  await page.$$eval('.some-class', async (elements) => {
    // ...
  });

• Wait for navigation (e.g., form submitting) to be done

  await page.waitForNavigation({ waitUntil: 'networkidle0', timeout: 0 });

• Trigger hover event on some of the elements

  await page.$eval('#header', async (headerElement) => {
    const hoverEvent = new MouseEvent('mouseover', {
      view: window,
      bubbles: true,
      cancelable: true
    });
    headerElement.dispatchEvent(hoverEvent);
  });

• Expose a function in the browser and use it in $eval and $$eval callbacks (e.g., simulate typing using the window.type function)

  await page.exposeFunction('type', async (selector, text, options) => {
    await page.type(selector, text, options);
  });
  await page.$$eval('.some-class', async (elements) => {
    // ...
    window.type(selector, text, { delay: 0 });
  });

• Press the Enter button after typing the input field value

  await page.type(selector, `${text}${String.fromCharCode(13)}`, options);

• Remove the value from the input field before typing the new one

  await page.click(selector, { clickCount: 3 });
  await page.type(selector, text, options);

• Expose a variable in the browser by passing it as the third argument for $eval and $$eval methods and use it in $eval and $$eval callbacks

  await page.$eval(
    '#element',
    async (element, customVariable) => {
      // ...
    },
    customVariable
  );

• Mock response for the specific request

  await page.setRequestInterception(true);
  page.on('request', async function (request) {
    const url = request.url();
    if (url !== REDIRECTION_URL) {
      return request.continue();
    }
    await request.respond({
      contentType: 'text/html',
      status: 304,
      body: '<body></body>',
    });
  });

• Intercept page redirections (via interceptor) and open them in new tabs rather than following them in the same tab

  await page.setRequestInterception(true);
  page.on('request', async function (request) {
    const url = request.url();
    if (url !== REDIRECTION_URL) {
      return request.continue();
    }
    await request.respond({
      contentType: 'text/html',
      status: 304,
      body: '<body></body>',
    });
    const newPage = await browser.newPage();
    await newPage.goto(url, { waitUntil: 'domcontentloaded', timeout: 0 });
    // ...
    await newPage.close();
  });

• Intercept page response

  page.on('response', async (response) => {
    if (response.url() === RESPONSE_URL) {
      if (response.status() === 200) {
        // ...
      }
      // ...
    }
  });

Boilerplate

Here is the link to the boilerplate I use for the development.

Image upscaling can be done using Real-ESRGAN, a super-resolution algorithm. Super-resolution is the process of increasing the resolution of the image.

Real-ESRGAN provides Linux, Windows and MacOS executable files and models for Intel/AMD/Nvidia GPUs.

The snippet below demonstrates bulk image upscaling with scale factor 4 and using the realesrgan-x4plus-anime model.

(async () => {
  const inputDirectory = path.resolve(path.join(__dirname, 'pictures'));
  const outputDirectory = path.resolve(
    path.join(__dirname, 'pictures_upscaled')
  );
  const modelsPath = path.resolve(path.join(__dirname, 'resources', 'models'));
  const execName = 'realesrgan-ncnn-vulkan';
  const execPath = path.resolve(
    path.join(__dirname, 'resources', getPlatform(), 'bin', execName)
  );
  const scaleFactor = 4;
  const modelName = 'realesrgan-x4plus-anime';
  if (!fs.existsSync(outputDirectory)) {
    await fs.promises.mkdir(outputDirectory, { recursive: true });
  }
  const commands = [
    '-i',
    inputDirectory,
    '-o',
    outputDirectory,
    '-s',
    scaleFactor,
    '-m',
    modelsPath,
    '-n',
    modelName
  ];
  const upscaler = spawn(execPath, commands, {
    cwd: undefined,
    detached: false
  });
  upscaler.stderr.on('data', (data) => {
    console.log(data.toString());
  });
  await timers.setTimeout(600 * 1000);
})();

Demo

The demo with the mentioned example is available here.

Boilerplate

Here is the link to the boilerplate I use for the development.

App

Here is the link to the app I use for image upscaling.

Releasing Electron desktop apps can be automated with Electron Forge and GitHub Actions. This post covers the main steps for automation.

Prerequisites

• bootstrapped Electron app
• GitHub personal access token (with repo and write:packages permissions) as a GitHub Action secret (GH_TOKEN)

Setup

Run the following commands to configure Electron Forge for the app release.

npm i @electron-forge/cli @electron-forge/publisher-github -D
npm i electron-squirrel-startup
npx electron-forge import

The last command should install the necessary dependencies and add a configuration file.

Update the forge.config.js file with the bin field containing the app name and ensure the GitHub publisher points to the right repository.

Put Windows and MacOS icons paths in the packagerConfig.icon field, Windows supports ico files with 256x256 resolution, and MacOS supports icns icons with 512x512 resolution (1024x1024 for Retina displays). Linux supports png icons with 512x512 resolution, also include its path in the BrowserWindow constructor config within the icon field.

// forge.config.js
const path = require('path');
module.exports = {
  packagerConfig: {
    asar: true,
    icon: path.join(process.cwd(), 'main', 'build', 'icon'),
  },
  rebuildConfig: {},
  makers: [
    {
      name: '@electron-forge/maker-squirrel',
      config: {
        bin: 'Electron Starter'
      }
    },
    {
      name: '@electron-forge/maker-dmg',
      config: {
        bin: 'Electron Starter'
      },
    },
    {
      name: '@electron-forge/maker-deb',
      config: {
        bin: 'Electron Starter',
        options: {
          icon: path.join(process.cwd(), 'main', 'build', 'icon.png'),
        },
      }
    },
    {
      name: '@electron-forge/maker-rpm',
      config: {
        bin: 'Electron Starter',
        icon: path.join(process.cwd(), 'main', 'build', 'icon.png'),
      }
    }
  ],
  plugins: [
    {
      name: '@electron-forge/plugin-auto-unpack-natives',
      config: {}
    }
  ],
  publishers: [
    {
      name: '@electron-forge/publisher-github',
      config: {
        repository: {
          owner: 'delimitertech',
          name: 'electron-starter'
        },
        prerelease: true
      }
    }
  ]
};

Upgrade the package version before releasing the app. The npm script for publishing should use publish command. Set productName field to the app name.

// package.json
{
  // ...
  "version": "1.0.1",
  "scripts": {
    // ...
    "publish": "electron-forge publish"
  },
  "productName": "Electron Starter"
}

GitHub Action workflow for manually releasing the app for Linux, Windows, and MacOS should contain the below configuration.

# .github/workflows/release.yml
name: Release app
on:
  workflow_dispatch:
jobs:
  build:
    strategy:
      matrix:
        os:
          [
            { name: 'linux', image: 'ubuntu-latest' },
            { name: 'windows', image: 'windows-latest' },
            { name: 'macos', image: 'macos-latest' },
          ]
    runs-on: ${{ matrix.os.image }}
    steps:
      - name: Github checkout
        uses: actions/checkout@v4
      - name: Use Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - name: Publish app
        env:
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
        run: npm run publish

Windows startup events

Add the following code in the main process to prevent Squirrel.Windows launches your app multiple times during the installation/updating/uninstallation.

// main/index.js
if (require('electron-squirrel-startup') === true) app.quit();

Boilerplate

Here is the link to the boilerplate I use for the development. It contains the examples mentioned above with more details.

Formatting helps to stay consistent with code style throughout the whole codebase. Include format script in pre-hooks (pre-commit or pre-push). This post covers Prettier setup with JavaScript and TypeScript code.

Start by installing the prettier package as a dev dependency.

npm i prettier -D

Specify rules inside the .prettierrc config file.

{
  "singleQuote": true,
  "trailingComma": "all"
}

Add format script in the package.json file.

{
  "scripts": {
    // ...
    "format": "prettier --write \"{src,test}/**/*.{js,ts}\""
  }
}

Notes

If you use Eslint, install the eslint-config-prettier package as a dev dependency and update the Eslint configuration to use the Prettier config.

{
  // ...
  "extends": [
    // ...
    "prettier"
  ]
}

Using Visual Studio Code, you can install a prettier-vscode extension and activate formatting when file changes are saved.

Boilerplate

Here is the link to the boilerplate I use for the development.

Regarding microservices observability, tracing is important to catch bottlenecks of the services like slow requests and database queries.

OpenTelemetry is a set of monitoring tools that support integration with distributed tracing platforms like Jaeger, Zipkin, and NewRelic, to name a few. This post covers Jaeger's tracing setup for Node.js projects.

Start by setting up the Docker compose via the docker-compose up command. Jaeger UI will be available at http://localhost:16686.

version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.46
    environment:
      - COLLECTOR_ZIPKIN_HTTP_PORT=:9411
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - 6831:6831/udp
      - 6832:6832/udp
      - 5778:5778
      - 16685:16685
      - 16686:16686
      - 14268:14268
      - 14269:14269
      - 14250:14250
      - 9411:9411
      - 4317:4317
      - 4318:4318

The code below shows setting up the tracing via Jaeger. Jaeger doesn't require a separate exporter package since OpenTelemetry supports it natively. Others need to use an exporter package. Filter traces within Jaeger UI by service name or trace ID stored within the logs.

Use resources and semantic resource attributes to set new fields for the trace, like service name or service version. Auto instrumentation identifies frameworks like Express, protocols like HTTP, databases like Postgres, and loggers like Winston used within the project.

Process spans (units of work in distributed systems) in batches to optimize tracing performance. Also, terminate the tracing during the graceful shutdown.

import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
const traceExporter = new OTLPTraceExporter({
  url: 'http://localhost:4318/v1/traces',
});
const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: `<service-name>-${process.env.NODE_ENV}`,
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.npm_package_version ?? '0.0.0',
    env: process.env.NODE_ENV || '',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  spanProcessor: new BatchSpanProcessor(traceExporter),
});
sdk.start();
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.error('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Import tracing config as the first thing inside the entry file.

import './tracing';
// ...

Search → Service menu should show the service name in Jaeger UI. Happy tracing!

Demo

The demo with the mentioned examples is available here.

Boilerplate

Here is the link to the boilerplate I use for the development.

Streaming is useful when dealing with big files in web apps. Instead of loading the entire file into memory before sending it to the client, streaming allows you to send it in small chunks, improving memory efficiency and reducing response time.

The code snippet below shows streaming the binary CSV and base64-encoded PDF files with NestJS. Use the same approach for other types of files, like JSON files.

Set content type and filename headers so files are streamed and downloaded correctly. Base64 file is converted to a buffer and streamed afterward. Read files from a file system or by API calls.

import { Controller, Get, Param, Res } from '@nestjs/common';
import { Response } from 'express';
import { createReadStream } from 'fs';
import { readFile } from 'fs/promises';
import { join } from 'path';
import { Readable } from 'stream';
@Controller('templates')
export class TemplatesController {
  @Get('csv')
  getCsvTemplate(@Res() res: Response): void {
    const file = createReadStream(join(process.cwd(), 'template.csv'));
    res.set({
      'Content-Type': 'text/csv',
      'Content-Disposition': 'attachment; filename="template.csv"'
    });
    file.pipe(res);
  }
  @Get('pdf/:id')
  async getPdfTemplate(
    @Param('id') id: string,
    @Res() res: Response
  ): Promise<void> {
    const fileBase64 = await readFile(
      join(process.cwd(), 'template.pdf'),
      'base64'
    );
    // const fileBase64 = await apiCall();
    const fileBuffer = Buffer.from(fileBase64, 'base64');
    const fileStream = Readable.from(fileBuffer);
    res.set({
      'Content-Type': 'application/pdf',
      'Content-Disposition': `attachment; filename="template_${id}.pdf"`
    });
    fileStream.pipe(res);
  }
}

Boilerplate

Here is the link to the boilerplate I use for the development.

Node.js version 20 brings a stable test runner so you can run tests inside *.test.js files with node --test command. This post covers the primary usage of it regarding spies and mocking for the unit tests.

Spies are functions that let you spy on the behavior of functions called indirectly by some other code while mocking injects test values into the code during the tests.

mock.method can create spies and mock async, rejected async, sync, chained methods, and external and built-in modules.

• Async function

import assert from 'node:assert/strict';
import { describe, it, mock } from 'node:test';
const calculationService = {
  calculate: () => // implementation
};
describe('mocking resolved value', () => {
  it('should resolve mocked value', async () => {
    const value = 2;
    mock.method(calculationService, 'calculate', async () => value);
    const result = await calculationService.calculate();
    assert.equal(result, value);
  });
});

• Rejected async function

const error = new Error('some error message');
mock.method(calculationService, 'calculate', async () => Promise.reject(error));
await assert.rejects(async () => calculateSomething(calculationService), error);

• Sync function

mock.method(calculationService, 'calculate', () => value);

• Chained methods

mock.method(calculationService, 'get', () => calculationService);
mock.method(calculationService, 'calculate', async () => value);
const result = await calculationService.get().calculate();

• External modules

import axios from 'axios';
mock.method(axios, 'get', async () => ({ data: value }));

• Built-in modules

import fs from 'fs/promises';
mock.method(fs, 'readFile', async () => fileContent);

• Async and sync functions called multiple times can be mocked with different values using context.mock.fn and mockedFunction.mock.mockImplementationOnce.

describe('mocking same method multiple times with different values', () => {
  it('should resolve mocked values', async (context) => {
    const firstValue = 2;
    const secondValue = 3;
    const calculateMock = context.mock.fn(calculationService.calculate);
    calculateMock.mock.mockImplementationOnce(async () => firstValue, 0);
    calculateMock.mock.mockImplementationOnce(async () => secondValue, 1);
    const firstResult = await calculateMock();
    const secondResult = await calculateMock();
    assert.equal(firstResult, firstValue);
    assert.equal(secondResult, secondValue);
  });
});

• To assert called arguments for a spy, use mockedFunction.mock.calls[0] value.

mock.method(calculationService, 'calculate');
await calculateSomething(calculationService, firstValue, secondValue);
const call = calculationService.calculate.mock.calls[0];
assert.deepEqual(call.arguments, [firstValue, secondValue]);

• To assert skipped call for a spy, use mockedFunction.mock.calls.length value.

mock.method(calculationService, 'calculate');
assert.equal(calculationService.calculate.mock.calls.length, 0);

• To assert how many times mocked function is called, use mockedFunction.mock.calls.length value.

mock.method(calculationService, 'calculate');
calculationService.calculate(3);
calculationService.calculate(2);
assert.equal(calculationService.calculate.mock.calls.length, 2);

• To assert called arguments for the exact call when a mocked function is called multiple times, an assertion can be done using mockedFunction.mock.calls[index] and call.arguments values.

const calculateMock = context.mock.fn(calculationService.calculate);
calculateMock.mock.mockImplementationOnce((a) => a + 2, 0);
calculateMock.mock.mockImplementationOnce((a) => a + 3, 1);
calculateMock(firstValue);
calculateMock(secondValue);
[firstValue, secondValue].forEach((argument, index) => {
  const call = calculateMock.mock.calls[index];
  assert.deepEqual(call.arguments, [argument]);
});

Running TypeScript tests

Add a new test script

{
  "type": "module",
  "scripts": {
    "test": "node --test",
    "test:ts": "glob -c \"node --loader tsx --no-warnings --test\" \"./src/**/*.{spec,test}.ts\""
  },
  "devDependencies": {
    // ...
    "glob": "^10.3.1",
    "tsx": "^3.12.7"
  }
}

Demo

The demo with the mentioned examples is available here.

Boilerplate

Here is the link to the boilerplate I use for the development.

Async API documentation is used for documenting events in event-driven systems, like Kafka events. All of the event DTOs are stored in one place. It supports YAML and JSON formats.

It contains information about channels and components. Channels and components are defined with their messages and DTO schemas, respectively.

{
  "asyncapi": "2.6.0",
  "info": {
    "title": "Events docs",
    "version": "1.0.0"
  },
  "channels": {
    "topic_name": {
      "publish": {
        "message": {
          "schemaFormat": "application/vnd.oai.openapi;version=3.0.0",
          "payload": {
            "type": "object",
            "properties": {
              "counter": {
                "type": "number"
              }
            },
            "required": ["counter"]
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "EventDto": {
        "type": "object",
        "properties": {
          "counter": {
            "type": "number"
          }
        },
        "required": ["counter"]
      }
    }
  }
}

Autogeneration

Async API docs can be autogenerated by following multiple steps:

• define DTOs and their required and optional fields with ApiProperty and ApiPropertyOptional decorators (from the @nestjs/swagger package), respectively
• generate OpenAPI docs from the defined DTOs
• parse and reuse component schemas from generated OpenAPI documentation to build channel messages and component schemas for Async API docs

Validation

Use AsyncAPI Studio to validate the written specification.

Preview

There are multiple options

• AsyncAPI Studio

• VSCode extension asyncapi-preview, open the command palette, and run the Preview AsyncAPI command.

UI generation

• Install @asyncapi/cli and corresponding template package (e.g., @asyncapi/html-template, @asyncapi/markdown-template)
• Update package.json with scripts

{
  "scripts": {
    // ...
    "generate-docs:html": "asyncapi generate fromTemplate ./asyncapi/asyncapi.json @asyncapi/html-template --output ./docs/html",
    "generate-docs:markdown": "asyncapi generate fromTemplate ./asyncapi/asyncapi.json @asyncapi/markdown-template --output ./docs/markdown"
  }
}

Linting represents static code analysis based on specified rules. Please include it in the CI pipeline.

Setup

Run the following commands to generate the linter configuration using the eslint package.

npm init -y
npm init @eslint/config

Below is an example of the configuration. Some rules can be ignored or suppressed as warnings.

// .eslintrc.js
module.exports = {
  env: {
    commonjs: true,
    es2021: true,
    node: true,
    jest: true,
  },
  extends: 'airbnb-base',
  overrides: [
  ],
  parserOptions: {
    ecmaVersion: 'latest',
  },
  rules: {
    'import/no-extraneous-dependencies': 'warn',
    'import/prefer-default-export': 'off',
  },
};

Ignore the files with the .eslintignore file.

dist

Linting

Configure and run the script with the npm run lint command. Some errors can be fixed automatically with the --fix option.

// package.json
{
  "scripts": {
    // ...
    "lint": "eslint src",
    "lint:fix": "npm run lint -- --fix"
  }
}

Boilerplate

Here is the link to the boilerplate I use for the development.

I recently migrated the Node.js app from Heroku to Fly.io, mainly due to reduced costs.

This blog post will cover the necessary steps in the migration process.

Prerequisites

• Heroku app running

• Use the exact versions for dependencies and dev dependencies in package.json so installation and build steps can pass successfully

• Use the same Node.js version in Dockerfile, package.json, and GitHub Actions workflow

• Use API gateway or custom domain for the service so web apps and mobile apps don't get affected by changing the URL of the service

Migration steps

• Migrate environment variables and secrets

• Migrate the Postgres database with the following commands (the ssl field in database configuration options is not needed)

fly secrets set HEROKU_DATABASE_URL=$(heroku config:get DATABASE_URL)
fly ssh console
apt update && apt install postgresql-client
pg_dump -Fc --no-acl --no-owner -d $HEROKU_DATABASE_URL | pg_restore --verbose --clean --no-acl --no-owner -d $DATABASE_URL
exit
fly secrets unset HEROKU_DATABASE_URL

• Migrate the Redis database if it's used

• Include the deployment step in the GitHub Actions workflow

References

• Deploying Node.js apps to Heroku

• Deploying Node.js apps to Fly.io

ChatGPT is a large language model (LLM) that understands and processes human prompts to produce helpful responses. OpenAI provides an API to interact with the ChatGPT model (gpt-3.5-turbo).

Prerequisites

• OpenAI account
• Generated API key
• Enabled billing

Integration

Below is an example of interacting with ChatGPT API based on a given prompt.

const handlePrompt = async (prompt) => {
  const response = await axios.post(
    'https://api.openai.com/v1/chat/completions',
    {
      model: 'gpt-3.5-turbo',
      messages: [
        {
          role: 'user',
          content: prompt
        }
      ]
    },
    {
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`
      }
    }
  );
  return response?.data?.choices?.[0]?.message?.content;
};

Node.js provides asynchronous methods for fs, dns, stream, and timers modules that return Promises.

const {
  createWriteStream,
  promises: { readFile }
} = require('fs');
const dns = require('dns/promises');
const stream = require('stream/promises');
const timers = require('timers/promises');
const sleep = timers.setTimeout;
const SLEEP_TIMEOUT_MS = 2000;
(async () => {
  const fileName = 'test-file';
  const writeStream = createWriteStream(fileName, {
    autoClose: true,
    flags: 'w'
  });
  await stream.pipeline('some text', writeStream);
  await sleep(SLEEP_TIMEOUT_MS);
  const readFileResult = await readFile(fileName);
  console.log(readFileResult.toString());
  const lookupResult = await dns.lookup('google.com');
  console.log(lookupResult);
})();

Use the promisify function to convert other callback-based functions to Promise-based.

const crypto = require('crypto');
const { promisify } = require('util');
const randomBytes = promisify(crypto.randomBytes);
const RANDOM_BYTES_LENGTH = 20;
(async () => {
  const randomBytesResult = await randomBytes(RANDOM_BYTES_LENGTH);
  console.log(randomBytesResult);
})();

Error tracking and alerting are crucial in the production environment, proactively fixing the errors leads to a better user experience. Sentry is one of the error tracking services, and it provides alerting for unhandled exceptions. You should receive an email when something wrong happens.

Sentry issues show the error stack trace, device, operating system, and browser information. The project dashboard shows an unhandled exception once it's thrown. This post covers the integration of several technologies with Sentry.

Node.js

• Create a Node.js project on Sentry

• Install the package

npm i @sentry/node

• Run the following script

const Sentry = require('@sentry/node');
Sentry.init({
  dsn: SENTRY_DSN
});
test();

Next.js

• Create a Next.js project on Sentry (version 13 is not yet supported)

• Run the following commands for the setup

npm i @sentry/nextjs
npx @sentry/wizard -i nextjs

Gatsby

• Create a Gatsby project on Sentry

• Install the package

npm i @sentry/gatsby

• Add plugin in Gatsby config

module.exports = {
  plugins: [
    // ...
    {
      resolve: '@sentry/gatsby',
      options: {
        dsn: SENTRY_DSN
      }
    }
  ]
};

React Native

• Create a React Native project on Sentry

• Run the following commands for the setup

npm i @sentry/react-native
npx @sentry/wizard -i reactNative -p android

This post covers some logging practices for the back-end (Node.js) apps.

• Avoid putting unique identifiers (e.g., user id) within the message. A unique id will produce a lot of different messages with the same context. Use it as a message parameter.

• Use the appropriate log level for the message. There are multiple log levels

  • info - app behavior, don't log every single step
  • error - app processing failure, something that needs to be fixed
  • debug - additional logs needed for troubleshooting
  • warning - something unexpected happened (e.g., third-party API fails)
  • fatal - app crash, needs to be fixed as soon as possible

Don't use the debug logs on production. Put log level as an environment variable.

• Stream logs to the standard output in JSON format so logging aggregators (Graylog, e.g.) can collect and adequately parse them

• Avoid logging any credentials, like passwords, auth tokens, etc.

• Put correlation ID as a message parameter for tracing related logs.

• Use a configurable logger like pino

const pino = require('pino');
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  redact: {
    paths: ['token'],
    remove: true,
  },
});
logger.info({ someId: 'id' }, 'Started the app...');
const correlationId = request.headers['correlation-id'] || uuid.v4();
logger.debug({ data: 'some data useful for debugging', correlationId }, 'Sending the request...');