One of them is the epoll API.

The epoll API

epoll() is a Linux kernel system call that provides an efficient way to monitor multiple file descriptors to check if I/O is possible on any of them. It is a new mechanism and introduced in Linux 2.6. It is similar to other system calls, such as select() and poll(), but it is more performant. epoll() scales much better when monitoring large numbers of file descriptors.

Due to its performance and flexibility, epoll() is commonly used in modern network and server applications to manage a large number of concurrent connections with minimal latency.

An epoll instance maintains two lists:

• Interest list. Contains a list of file descriptors to be monitored.

• Ready list. Contains a list of file descriptors that are ready for I/O.

With this design, epoll is able to scale to thousand of file descriptors without any performance penalties.

Notification Modes

epoll() supports two types of notification modes:

• Level-Triggered (LT). This is the default mode in which as long as a file descriptor is ready for I/O, subsequent calls to epoll_wait() will continue to report it.

• Edge-Triggered(ET). In this mode, epoll_wait() only reports an event when a change occurs. The subsequent calls to epoll_wait() will block although there’s still data available on a file descriptor.

How It Works

This API consists of three system calls:

• epoll_create(). Create a new epoll instance and returns a file descriptor referring to the instance.

• epoll_ctl(). Add, remove, or modify a file descriptor into the interest list associated with an epoll instance.

• epoll_wait(). Wait until I/O is ready for any of the file descriptors in the interest list.

To make it clear, we are going to implement a simple example of using the epoll API.

#include <sys/epoll.h>
#include <fcntl.h>

extern "C"
{
#include "lib/error_functions.h"
#include "lib/tlpi_hdr.h"
}

#define MAX_BUF 1000 // Maximum bytes fetched by a single read()
#define MAX_EVENTS 5 // Maximum number of events to be returned from a single epoll_wait call

int main(int argc, char *argv[])
{
    printf("Run %d\n", argc);
    int epfd, ready, fd, s, j, numOpenFds;
    struct epoll_event ev;
    struct epoll_event evlist[MAX_EVENTS];
    char buf[MAX_BUF];

    epfd = epoll_create(argc - 1);
    if (epfd == -1)
    {
        errExit("epoll_create");
    }

    // Open each file and add it into the "interest list"
    for (j = 1; j < argc; j++)
    {
        printf("Opening file descriptor for \"%s\"\n", argv[j]);
        fd = open(argv[j], O_RDONLY);
        if (fd == -1)
        {
            errExit("open");
        }
        printf("Opened \"%s\" on fd %d\n", argv[j], fd);

        ev.events = EPOLLIN;
        ev.data.fd = fd;
        if (epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) == -1)
        {
            errExit("epoll_ctl");
        }
    }

    numOpenFds = argc - 1;

    while (numOpenFds > 0)
    {
        // Fetch up to MAX_EVENTS items for the ready list

        printf("About to epoll_wait()\n");
        ready = epoll_wait(epfd, evlist, MAX_EVENTS, -1);
        if (ready == -1)
        {
            if (errno == EINTR)
            {
                continue;
            }
            else
            {
                errExit("epoll_wait\n");
            }
        }
        printf("Ready %d\n", ready);

        // Deal with returned list of events
        for (j = 0; j < ready; j++)
        {
            printf(" fd=%d; events: %s%s%s\n", evlist[j].data.fd,
                   (evlist[j].events & EPOLLIN) ? "EPOLLIN " : "",
                   (evlist[j].events & EPOLLHUP) ? "EPOLLHUP " : "",
                   (evlist[j].events & EPOLLERR) ? "EPOLLERR " : "");

            if (evlist[j].events & EPOLLIN)
            {

                s = read(evlist[j].data.fd, buf, MAX_BUF);
                if (s == -1)
                {
                    errExit("read");
                }
                printf("  read %d bytes: %.*s\n", s, s, buf);
            }
            else if (evlist[j].events & (EPOLLHUP | EPOLLERR))
            {
                printf("Closing fd %d\n", evlist[j].data.fd);
                if (close(evlist[j].data.fd) == -1)
                {
                    errExit("close");
                }
                numOpenFds--;
            }
        }
    }

    printf("All file descriptors closed; bye\n");
    exit(EXIT_SUCCESS);
}

In this article, we will explore the UNIX Domain Socket which is used for inter-process communication on the same host system. We will focus on stream sockets.

UNIX Domain Socket

The socket address structure for UNIX Domain Socket is as follow:

struct sockaddr_un {
\tsa_family_t sun_family;    // Always AF_UNIX
\tchar sun_path[108];        // NULL-terminated socket pathname
};

The operation flow will be as follows.

11111

The operation flow from the client side is as follows:

11# Simple Server-Client Implementation

Server

#include <sys/un.h>
#include <sys/socket.h>
#include <ctype.h>

...

#define BUF_SIZE 512
#define SV_SOCK_PATH "/tmp/unix_socket"

int main()
{
    std::cout << "Hello, Linux IPC and Socket Domain!\n";

    struct sockaddr_un addr;
    int sfd, cfd;
    ssize_t numRead;
    char buf[BUF_SIZE];

    sfd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (sfd == -1)
    {
        // Exit error
    }

    if (remove(SV_SOCK_PATH) == -1 && errno != ENOENT)
    {
        // Exit error
    }

    memset(&addr, 0, sizeof(struct sockaddr_un));
    addr.sun_family = AF_UNIX; // UNIX Domain address
    strncpy(addr.sun_path, SV_SOCK_PATH, sizeof(addr.sun_path) - 1);

    if (bind(sfd, (struct sockaddr *)&addr, sizeof(struct sockaddr_un)) == -1)
    {
        // Exit error
    }
    if (listen(sfd, BACKLOG) == -1)
    {
        // Exit error
    }

    // Handle client connections iteratively
    for (;;)
    {
        std::cout << "Waiting for a connection...\n";
        cfd = accept(sfd, NULL, NULL);
        if (cfd == -1)
        {
            // Exit error;
        }

        // Transfer data
        while ((numRead = read(cfd, buf, BUF_SIZE)) > 0)
        {
            if (write(STDOUT_FILENO, buf, numRead) != numRead)
            {
                // Exit error
            }
        }

        if (numRead == -1)
        {
            // Exit error
        }
        close(cfd);
    }

    std::cout << "Client disconnected.\n";

    return 0;
}

Client

#include <sys/un.h>
#include <sys/socket.h>
#include <ctype.h>

...

#define BUF_SIZE 512
#define SV_SOCK_PATH "/tmp/unix_socket"

int main(int argc, char *argv[])
{
    struct sockaddr_un addr;
    int sfd;
    ssize_t numRead;
    char buf[BUF_SIZE];

    sfd = socket(AF_UNIX, SOCK_STREAM | SOCK_NONBLOCK, 0); /* Create client socket */
    if (sfd == -1) {
        // Error exit 
    }

    /* Construct server address, and make the connection */
    memset(&addr, 0, sizeof(struct sockaddr_un));
    addr.sun_family = AF_UNIX;
    strncpy(addr.sun_path, SV_SOCK_PATH, sizeof(addr.sun_path) - 1);

    if (connect(sfd, (struct sockaddr *)&addr,
                sizeof(struct sockaddr_un)) == -1) {
        // Error connect exit
    }

    /* Copy stdin to socket */
    while ((numRead = read(STDIN_FILENO, buf, BUF_SIZE)) > 0) {
        if (write(sfd, buf, numRead) != numRead) {
\t\t        // error write
        }

        ssize_t numBytes = read(sfd, buf, BUF_SIZE-1);
        printf("Read %zd bytes from server: ", numBytes);
        if (numBytes == -1)
            continue;
        buf[numBytes] = '\0'; /* Null-terminate the string */
        printf("Received from server: %s\n", buf);
    }
    if (numRead == -1)
        // exit error read
    exit(EXIT_SUCCESS); /* Closes our socket; server sees EOF */
}

Concurrent Server Implementation

The previous example is only a simple server-client implementation. It supports only one client connection at one time. Of course, this is usable in the real-world. In practice, we expect the server to be able to accept multiple connections at the same time. This also applies on communication between processes on the same host. So, how can we achieve this?

There are multiple implementation variants to build a concurrent server.

111You can find the implementation examples below.

Pre-created Threads in Server Pool

// main.cpp
#include <iostream>

#include <sys/un.h>
#include <sys/socket.h>
#include <ctype.h>

#include <atomic>
#include <thread>
#include <vector>
#include <mutex>
#include <queue>
#include <condition_variable>
#include <functional>

extern "C"
{
#include "lib/error_functions.h"
#include "lib/tlpi_hdr.h"
}

#define BUF_SIZE 10
#define SV_SOCK_PATH "/tmp/ud_ucase"
#define BACKLOG 5
#define MAX_THREADS 5

class ServerPool {
public:
    ServerPool(size_t num_threads = MAX_THREADS)
        : running(true), active_threads(0) 
    {
        threads.reserve(num_threads);
        for (int i = 0; i < MAX_THREADS; ++i) {
            threads.emplace_back(&ServerPool::worker_loop, this);
        }
    }

    ~ServerPool() {
        running = false;
        condition.notify_all();
        for (auto &thread: threads) {
            if (thread.joinable()) {
                thread.join();
            }
        }
    }

    void add_task(std::function<void()> task) {
        {
            std::cout << "Adding task to the queue. Current active threads: " << active_threads.load() << std::endl;
            std::unique_lock<std::mutex> lock(threads_mutex);
            tasks.push(std::move(task));

            std::cout << "Task added. Total tasks: " << tasks.size() << std::endl;  
        }
        condition.notify_one();
    }

    int get_active_threads() const {
        return active_threads.load();
    }

    void stop() {
        running = false;
        condition.notify_all();
    }

    void wait_for_completion() {
        std::unique_lock<std::mutex> lock(threads_mutex);
        condition.wait(lock, [this] { return tasks.empty() && active_threads.load() == 0; });
    }

    void reset() {
        std::unique_lock<std::mutex> lock(threads_mutex);
        while (!tasks.empty()) {
            tasks.pop();
        }
        active_threads = 0;
    }

private:
    void worker_loop() {
        while(true) {
            std::function<void()> task;
            {
                std::unique_lock<std::mutex> lock(threads_mutex);
                condition.wait(lock, [this] {
                    return !tasks.empty() || !running;
                });

                std::cout << "Thread " << std::this_thread::get_id() << " checking for tasks. Active threads: " << active_threads.load() << std::endl;

                if (!running && tasks.empty()) {
                    return; // Exit if no tasks and not running
                }

                task = std::move(tasks.front());
                tasks.pop();
                active_threads++;
            }

            try {
                std::cout << "Thread " << std::this_thread::get_id() << " executing task. Active threads: " << active_threads.load() << std::endl;
                task();
            } catch (const std::exception& e) {
                std::cerr << "Task execution error: " << e.what() << std::endl;
            }

            active_threads--;
        }
    }

    std::vector<std::thread> threads;
    std::queue<std::function<void()>> tasks;
    std::condition_variable condition;
    std::mutex threads_mutex;
    std::atomic<bool> running{true};
    std::atomic<int> active_threads{0};
};



int main()
{
    std::cout << "Hello, Linux IPC and Socket Domain!\n";

    struct sockaddr_un addr;
    int sfd, cfd;
    ssize_t numRead;
    char buf[BUF_SIZE];

    sfd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (sfd == -1)
    {
        errExit("socket");
    }

    if (remove(SV_SOCK_PATH) == -1 && errno != ENOENT)
    {
        errExit("remove-%s", SV_SOCK_PATH);
    }

    memset(&addr, 0, sizeof(struct sockaddr_un));
    addr.sun_family = AF_UNIX; // UNIX Domain address
    strncpy(addr.sun_path, SV_SOCK_PATH, sizeof(addr.sun_path) - 1);

    if (bind(sfd, (struct sockaddr *)&addr, sizeof(struct sockaddr_un)) == -1)
    {
        errExit("bind");
    }
    if (listen(sfd, BACKLOG) == -1)
    {
        errExit("listen");
    }

    ServerPool server_pool;

    // Handle client connections iteratively
    for (;;)
    {
        std::cout << "Waiting for a connection...\n";
        cfd = accept(sfd, NULL, NULL);
        if (cfd == -1)
        {
            errExit("accept");
        }

        std::cout << "Client connected.\n";
        server_pool.add_task([cfd]() {
            ssize_t numRead;
            char buf[BUF_SIZE];

            // Read data from the client
            while ((numRead = read(cfd, buf, BUF_SIZE)) > 0)
            {
                std::cout << "Received " << numRead << " bytes: ";
                for (ssize_t i = 0; i < numRead; ++i)
                {
                    std::cout << static_cast<char>(toupper(buf[i]));
                }
                std::cout << std::endl;

                // Echo back to the client
                if (write(cfd, buf, numRead) != numRead)
                {
                    errMsg("partial/failed write");
                }
            }

            if (numRead == -1)
            {
                errExit("read");
            }

            close(cfd); // Close the client socket
        });
        std::cout << "Task added to server pool. Active threads: " << server_pool.get_active_threads() << std::endl;
    }

    std::cout << "Client disconnected.\n";

    return 0;
}

Single Process Concurrent Server using epoll()

// main.cpp
#include <iostream>

#include <sys/un.h>
#include <sys/socket.h>
#include <ctype.h>

#include <atomic>
#include <thread>
#include <vector>
#include <mutex>
#include <queue>
#include <condition_variable>
#include <functional>

extern "C"
{
#include "lib/error_functions.h"
#include "lib/tlpi_hdr.h"
}

#include <sys/epoll.h>

#define BUF_SIZE 512
#define SV_SOCK_PATH "/tmp/ud_ucase"
#define BACKLOG 5
#define MAX_THREADS 5
#define MAX_BUF 512

int main()
{
    std::cout << "Hello, Linux IPC and Socket Domain!\n";

    struct sockaddr_un addr;
    int sfd, cfd;
    ssize_t numRead;
    char buf[BUF_SIZE];

    sfd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (sfd == -1)
    {
        errExit("socket");
    }

    if (remove(SV_SOCK_PATH) == -1 && errno != ENOENT)
    {
        errExit("remove-%s", SV_SOCK_PATH);
    }

    memset(&addr, 0, sizeof(struct sockaddr_un));
    addr.sun_family = AF_UNIX; // UNIX Domain address
    strncpy(addr.sun_path, SV_SOCK_PATH, sizeof(addr.sun_path) - 1);

    if (bind(sfd, (struct sockaddr *)&addr, sizeof(struct sockaddr_un)) == -1)
    {
        errExit("bind");
    }
    if (listen(sfd, BACKLOG) == -1)
    {
        errExit("listen");
    }

    int connected_clients = 0;
    // Create an epoll
    int epfd = epoll_create(1);
    if (epfd == -1)
    {
        errExit("epoll_create");
    }

    int epfd2 = epoll_create(5);
    if (epfd2 == -1) {
        errExit("epoll_create2");
    }

    struct epoll_event ev;
    ev.events = EPOLLIN; // Interested in read events
    ev.data.fd = sfd; // Monitor the server socket

    if (epoll_ctl(epfd, EPOLL_CTL_ADD, sfd, &ev) == -1)
    {
        errExit("epoll_ctl");
    }
    std::cout << "Server is listening on " << SV_SOCK_PATH << std::endl;

    #define MAX_EVENTS 10
    struct epoll_event evlist[MAX_EVENTS];

    // Handle client connections iteratively
    for (;;)
    {
        std::cout << "Waiting for a connection...\n";
        int ready = epoll_wait(epfd, evlist, MAX_EVENTS, -1);
        if (ready == -1)
        {
            if (errno == EINTR)
            {
                continue; // Interrupted by a signal, retry
            }
            errExit("epoll_wait");
        }

        printf("Ready %d events\n", ready);

        for (int j=0; j<ready; j++) {
            std::cout << "Event detected on fd: " << evlist[j].data.fd << " " << evlist[j].events << "\n";
            if (evlist[j].events & EPOLLIN) // Check if the event is for reading
            {
                if (evlist[j].data.fd != sfd) {
                    int numRead = read(evlist[j].data.fd, buf, MAX_BUF);
                    if (numRead == -1)
                      errExit("read");

                    std::cout << "Received " << numRead << " bytes: ";
                    for (ssize_t i = 0; i < numRead; ++i)
                    {
                        buf[i] = toupper(buf[i]);
                    }

                    // Echo back to the client
                    if (write(evlist[j].data.fd, buf, numRead) != numRead)
                    {
                        errMsg("partial/failed write");
                    }
                } else {
                    std::cout << "New connection detected.\n";
                    int cfd = accept(sfd, NULL, NULL);
                    if (cfd == -1)
                    {
                        errExit("accept");
                    }

                    // Add the new client socket to the epoll instance
                    struct epoll_event clientEv;
                    clientEv.events = EPOLLIN; // Interested in read events 
                    clientEv.data.fd = cfd;
                    if (epoll_ctl(epfd, EPOLL_CTL_ADD, cfd, &clientEv) == -1)
                    {
                        errExit("epoll_ctl");
                    }
                    connected_clients++;
                }
            }

            if (evlist[j].events & (EPOLLHUP | EPOLLERR)) // Check for hangup or error
            {
                std::cout << "Closing fd " << evlist[j].data.fd << "\n";

                // Remove the client socket from the epoll instance
                if (epoll_ctl(epfd, EPOLL_CTL_DEL, evlist[j].data.fd, NULL) == -1)
                {
                    errExit("epoll_ctl-del");
                }
                if (close(evlist[j].data.fd) == -1)
                {
                    errExit("close");
                }
                connected_clients--;
            }
            else
            {
                std::cerr << "Unexpected event detected.\n";
                continue; // Skip unexpected events
            }
        }

        printf("Connected Device %d\n", connected_clients);
    }

    std::cout << "Client disconnected.\n";

    return 0;
}

#

Actually this is not my first time I encounter this. In my previous project, the company built their own Remote Procedure Call (RPC) framework. The framework supports both synchronous and asynchronous inter-process communication between processes. It supports not only the request-reply pattern but also notification (subscribe and publish) pattern.

I know how to use the framework to facilitate IPC for our applications. After using it for a while, it made me wonder how it works and how it was built. Admittedly I did not spend time to fully understand how the framework was built. However, after attending the meetup, it becomes clear to me how the puzzle pieces are connected. Now, with the fresh insights from the meetup, I would like to document what I learned in this and upcoming articles.

Linux Inter-Process Communication

Linux provides facilities to enable processes and threads to exchange data with one another and to synchronize their actions. In [2], the author describes the communication facilities available on Linux: Pipe, FIFO, Message Queue, stream socket, shared memory, etc.

I won’t discuss all available facilities in this article and focus only on FIFO, Message Queue, and Stream Socket. I think these are the options commonly used in real world applications. These are also the ones presented in the meetup.

FIFO

FIFO (First-In First-Out) is a mechanism to allow a process to communicate with another process. It operates as a un-directional data channel. A process writes data into the FIFO and another process reads the data from it. This mechanism is suitable in a simple producer-consumer scenario. In order to allow bidirectional communications, two FIFOs will be needed.

A FIFO can be created by calling mkfifo given the path name. Upon created, the FIFO can be found in the file system.

#include <sys/stat.h>

int mkfifo(const char *pathname, mode_t mode);

Message Queue

Message Queue provides a more structured form of communication between two processes. It allows messages which consist of a numeric type plus a body containing data, to be exchanged. Hence, the message boundaries can be preserved between calls.

Compared to FIFO, message queue supports asynchronous messaging. The messages sent by the producer can be queued, and the receiver can retrieve the messages later.

The Message Queue APIs are as follows.

#include <fcntl.h> /* Defines O_* constants */
#include <sys/stat.h> /* Defines mode constants */
#include <mqueue.h>

mqd_t mq_open(const char *name, int oflag, ...
 /* mode_t mode, struct mq_attr *attr */);

int mq_send(mqd_t mqdes, const char *msg_ptr, size_t msg_len,
 unsigned int msg_prio);

ssize_t mq_receive(mqd_t mqdes, char *msg_ptr, size_t msg_len,
 unsigned int *msg_prio);

int mq_close(mqd_t mqdes);

int mq_unlink(const char *name);

Socket

Socket is a method of IPC that allow data to be exchanged bidirectionally either on the same host (UNIX Domain Socket) or on different hosts connected by a network (IPv4 or IPv6). Every sockets implementation provides at least two types: stream and datagram sockets. In the Internet domain, the stream socket uses TCP and the datagram socket uses UDP. The first one is reliable while the second one is not.

The Linux Programming Interface book by Michael Kerrisk shows the overview of system calls used in both types of sockets.

The main advantage of sockets is it accepts multiple client connections on a single server socket. This is useful in server environments where scalability and simultaneous connections are important.

As you can guess, sockets are commonly used in most real-world applications. They provide a robust solution when you need both the reliability of structured communication and the flexibility of managing multiple concurrent connections.

Conclusion

In this article, I have shown some of the commonly used Linux mechanisms for inter-process communication. In the upcoming articles, we will see how to implement IPC using UNIX domain socket and how to use epoll system call to allow asynchronous communication calls.

References

[1] Exploring Linux API slide by Henrique Marks.

[2] The Linux Programming Interface by Michael Kerrisk.

BLE and GATT

BLE is a wireless technology designed for short-range communication between devices while keeping power consumption to a minimum. Hence, it is commonly used in the Internet of Things (IoT) landscape. It does not only manages how data is organized and exchanged between devices, but also provides a standardized and power efficient way for BLE devices to communicate.

GATT, short for Generic Attribute Profile, is the protocol that defines the structure and behavior for communication between BLE-enabled devices. It extends the Attribute Protocol (ATT) by defining several key concepts:

• Characteristics - Individual data elements. This is where the actual data is stored.

• Services - Collections of related characteristics that group data into logical entities.

• Profile - High level definition of how a device should behave to enable a specific application. It contains one or more services.

A profile sits on top of the hierarchy. It contains a pre-defined collection of services. Profiles can be either pre-defined by the Bluetooth SIG ( Profiles Overview) or by the peripheral designers. One example is the Heart Rate Profile, which combines the mandatory Heart Rate Service and the optional Device Information Service.

Services are used to break data up into logical entities (characteristics). A service can be identified by either a 16-bit UUID (officially adopted BLE services) or a 128-bit UUID (for custom services). If we look at the official Heart Rate Service, we can see that the service has one mandatory characteristic (i.e., Heart Rate Measurement) and two optional ones (i.e., Body Sensor Location and Heart Rate Control Point). Each characteristics then contains a single data point.

GATT Operations

As mentioned before, data is stored within characteristics. Once the data has been structured, clients interact with the data through several operations:

• Read .

The client requests to read the value of a Characteristic from the server.

• Write .

The client can also send data to the server to update the value of a characteristic. There are two variants. The first one is a write that requires an acknowledgment from the server and the second one is a write command without a response/ acknowledgement.

• Notify .

The server can be configured to automatically send updates of a characteristic’s value to the client whenever the value changes. To configure the server, the client needs to send a subscription request.

• Indicate .

This is similar to notify; the difference is that the client must send an acknowledgement back to the server to confirm it received the data.

Example using NimBLE

NimBLE is a lightweight, open-source Bluetooth Low Energy (BLE) stack designed especially for resource-constrained devices. It has been integrated into ESP-IDF framework for ESP32 devices. The main advantages of NimBLE are its modular architecture and small memory footprint. This makes NimBLE an excellent choice for building BLE applications.

Write and Read Operations

In NimBLE, all GATT services and characteristics can be defined in the gatt_svr_svcs_service table. For each characteristics, we can specific where it supports read, write, notify, and indicate operations. We also need to assign callback in the access_cb variable and the characteristic handle in the val_handle field. Below is an example showing a Heart Rate service definition:

/* Heart rate service UUID */
static const ble_uuid16_t heart_rate_svc_uuid = BLE_UUID16_INIT(0x180D);

/* Characteristics Handle */
static uint16_t heart_rate_chr_val_handle;
static uint16_t body_sensor_loc_chr_val_handle;

/* Callback function */
static int gatt_svr_chr_access_heart_rate(uint16_t conn_handle, uint16_t attr_handle,
                                          struct ble_gatt_access_ctxt *ctxt, void *arg);

/* GATT services table */
static const struct ble_gatt_svc_def gatt_svr_svcs[] = {
    { /* Service: Heart-rate */
     .type = BLE_GATT_SVC_TYPE_PRIMARY,
     .uuid = &heart_rate_svc_uuid.u,
     .characteristics = (struct ble_gatt_chr_def[]){
         {
             /* Characteristic: Heart-rate measurement */
             .uuid = BLE_UUID16_DECLARE(GATT_HRS_MEASUREMENT_UUID),
             .access_cb = gatt_svr_chr_access_heart_rate,
             .val_handle = &heart_rate_chr_val_handle,
             .flags = BLE_GATT_CHR_F_READ | BLE_GATT_CHR_F_NOTIFY,
         },
         {
             /* Characteristic: Body sensor location */
             .uuid = BLE_UUID16_DECLARE(GATT_HRS_BODY_SENSOR_LOC_UUID),
             .access_cb = gatt_svr_chr_access_heart_rate,
             .val_handle = &body_sensor_loc_chr_val_handle,
             .flags = BLE_GATT_CHR_F_READ,
         },
         {
             0, /* No more characteristics in this service */
         },
     }
    },
    {
     0, /* No more services */
      },
};

The accompanying callback function handles events. In this example, it handles only read operations because the characteristic only supports read operations.

static int gatt_svr_chr_access_heart_rate(uint16_t conn_handle, uint16_t attr_handle,
                                  struct ble_gatt_access_ctxt *ctxt, void *arg)
{
    /* Local variables */
    int rc;

    /* Handle access events */
    /* Note: LED characteristic is write only */
    switch (ctxt->op)
    {

    /* Read characteristic event */
    case BLE_GATT_ACCESS_OP_READ_CHR:
        /* Verify connection handle */
        if (conn_handle != BLE_HS_CONN_HANDLE_NONE) {
            ESP_LOGI(TAG, "characteristic read; conn_handle=%d attr_handle=%d",
                    conn_handle, attr_handle);
        } else {
            ESP_LOGI(TAG, "characteristic read by nimble stack; attr_handle=%d",
                    attr_handle);
        }

        /* Verify attribute handle */
        if (attr_handle == heart_rate_chr_val_handle) {
            /* Update access buffer value */
            heart_rate_chr_val[1] = get_heart_rate();
            rc = os_mbuf_append(ctxt->om, &heart_rate_chr_val,
                                sizeof(heart_rate_chr_val));
            return rc == 0 ? 0 : BLE_ATT_ERR_INSUFFICIENT_RES;
        }
        goto error;

    /* Write characteristic event */
    case BLE_GATT_ACCESS_OP_WRITE_CHR:
    /* Unknown event */
    default:
        goto error;
    }

error:
    ESP_LOGE(tag,
             "unexpected access operation to led characteristic, opcode: %d",
             ctxt->op);
    return BLE_ATT_ERR_UNLIKELY;
}

Notification and Indication

The NimBLE library provides APIs to handle notification and indication operations.

int ble_gatts_notify_custom(uint16_t conn_handle, uint16_t chr_val_handle,
                            struct os_mbuf *txom);

int ble_gatts_indicate_custom(uint16_t conn_handle, uint16_t chr_val_handle,
                              struct os_mbuf *txom);

These operations will be performed only if the client specifies that it would like to subscribe to any changes. The subscription event will be handled at the GAP layer. This can be done by handling the BLE_GAP_EVENT_SUBSCRIBE event.

static int gap_event_handler(struct ble_gap_event *event, void *arg) {
    ...

    /* Subscribe event */
    case BLE_GAP_EVENT_SUBSCRIBE:
        /* Print subscription info to log */
        ESP_LOGI(TAG,
                "subscribe event; conn_handle=%d attr_handle=%d "
                "reason=%d prevn=%d curn=%d previ=%d curi=%d",
                event->subscribe.conn_handle, event->subscribe.attr_handle,
                event->subscribe.reason, event->subscribe.prev_notify,
                event->subscribe.cur_notify, event->subscribe.prev_indicate,
                event->subscribe.cur_indicate);

        /* GATT subscribe event callback */
        /* Check connection handle */
            if (event->subscribe.conn_handle != BLE_HS_CONN_HANDLE_NONE) {
                ESP_LOGI(TAG, "subscribe event; conn_handle=%d attr_handle=%d",
                        event->subscribe.conn_handle, event->subscribe.attr_handle);
            } else {
                ESP_LOGI(TAG, "subscribe by nimble stack; attr_handle=%d",
                        event->subscribe.attr_handle);
            }

            /* Check attribute handle */
            if (event->subscribe.attr_handle == heart_rate_chr_val_handle) {
                /* Update heart rate subscription status */
                heart_rate_chr_conn_handle = event->subscribe.conn_handle;
                heart_rate_chr_conn_handle_inited = true;
                heart_rate_ind_status = event->subscribe.cur_indicate;
            }
}

This handle logs subscription events and updates internal state, ensuring the server sends notifications or indications only when appropriate.

Final Thoughts

BLE GATT’s structured approach—organizing data into characteristics, services, and profiles—is foundational for efficient and interoperable BLE communication in IoT devices. Whether you’re building wearable health monitors, smart home gadgets, or industrial sensors, understanding and leveraging GATT operations can greatly enhance your application’s performance and reliability.

NimBLE, with its lightweight design and efficient API set, stands out as a powerful tool for implementing these concepts on resource-constrained devices like the ESP32. Its modular architecture ensures that you can easily scale, customize, and optimize your BLE applications as your requirements evolve.

References

[1] ESP-IDF BLE Data Exchange

[2] Adafruit GATT

Bluetooth Low Energy (BLE) is a low-power variant of Bluetooth Classic standard. Originally developed by Nokia under the name WIbree in 2006, this solution tried to address the scenarios that contemporary wireless technologies did not address at that time. Some targeted applications or scenarios are in the healthcare, fitness, smart home, and home entertainment industries. These applications generally have low cost and low power requirements where the devices are able to operate for months or years on a button cell.

As the smartphones, tablets, mobile computing, and Internet-of-Thing (IoT) grow rapidly, the BLE has also grown fast and has been widely adopted.

One of the strong advantage of BLE or Bluetooth is its interoperability. It means that it is easy to connect devices and smartphones from different vendors together. It is becoming so common in the Internet of Things (IoT) field to connect embedded devices, such as ESP32, with a smartphone. This opens a unique opportunity for new innovative solutions that might not possible before.

That is why being able to develop a BLE-powered application might be an important skill to have for (embedded) software engineers. In this article, I will cover the basic concepts and architecture of BLE.

Architecture

The Bluetooth Low Energy protocol stack is divided into three layers: Application, Host, and Controller.

Application Layer

This is the highest layer where applications are built using BLE as the underlying communication protocol. It contains the logic and user interface related to the actual use-case of the application. The application layer relies on the API interfaces provided by the Host Layer.

Host Layer

The Host Layer defines the BLE protocol itself. It consists of Generic Access Profile (GAP), Generic Attribute Profile (GATT), Attribute Protocol (ATT), Security Manager (SM), and Logical Link Control and Adaptation Protocol (L2CAP). GAP and GATT are the backbone of BLE data transfer protocol and it is imperative for application developers to understand these modules to successfully build a BLE-powered application. Both of them will be covered in more detail in the next section.

Host Controller Interface (HCI)

HCI is a standard protocol that acts as an interface between the host and controller layer. The connection generally takes place via a UART or USB interface. If the host and controller layer are implemented on the same chip (a system-on-chip or SoC), HCI is referred to as Virtual Host Controller Interface (VHCI).

Controller Layer

The Controller Layer consists of the Physical Layer (PHY) and Link Layer (LL). The Physical Layer interacts with the actual hardware (analog communication circuitry) for signal modulation and demodulation.

The Link Layer is implemented as a combination of custom hardware and software. The hardware side implements functionalities that are computationally expensive on separate hardware to avoid overloading the CPU running the software stack.

The software side manages the connection state between devices and device roles. A BLE device can have the following roles: Advertiser, Scanner, Master, and Slave.

User applications are not able to use the controller layer directly.

Protocol Basics

This sections describes the basics of BLE protocol within the Host Layer.

Generic Access Profile (GAP)

The Generic Access Profile (GAP) defines how devices interact with each other at a lower level. It defines the roles (i.e., Advertiser, Scanner, and Initiator) and the connection states (i.e., Idle, Device Discovery, and Connection). Note that a BLE device can have more than one role.

In Idle mode, the device is in a state without any role.

In device discovery mode, a device can become an advertiser indicating its presence to other devices. It can also adopt the scanner role. As a scanner, it continuously scans the environment and detects the presence of connectable advertisers. If the scanner wants to establish a connection with a advertiser, it switches its role to Initiator.

Once a device is connected to another device, it can adopt a peripheral role (slave) or a central role (master).

Generic Attribute Profile (GATT)

GATT defines the hierarchical data structure and data exchange mechanism between connected devices. It is built on top of Attribute Protocol (ATT), which defines the basic data structure called Attribute and data access method based on a server/ client architecture.

An attribute data structure typically consists of:

• Handle. A unique 16-bit identifier for each attribute.
• Type. An UUID.
• Value.
• Permissions. A metadata specifying whether the attribute is readable or writable or both.

GATT extends ATT by defining hierarchical data structure which consists of three concepts:

• Characteristics
• Service
• Profile

Profile sits on the highest level in the hierarchy. It consists of a predefined set of service. The service itself contains a set of characteristics. Both service and characteristic are based on attributes (i.e., handle, type, value, and permissions). A characteristics itself contains the actual user data that the client can read from and write to.

This marks the end of the article. GATT is a big subject and I will cover this in more detail in the next article. I hope that you can learn something new about BLE from this article.

References

[Book] Getting Started with Bluetooth Low Energy by Kevin Townsend et. al.
[ESP-IDF] BLE-Introduction

Prerequisites

• YouTube Data API credentials

• AI model API key

• Notion API integration

• n8n installation

• Telegram API key

• Heroku.

Setup

Here's how the workflow functions: When a user sends a YouTube video URL to a Telegram Bot, it triggers the workflow. First, an Information extractor LLM parses the video ID from the chat message. Using this ID, we retrieve the video's transcript by sending an HTTP request to a server that is responsible for retrieving the transcript.. Then, an AI model analyzes and summarizes the transcript content. The summary is saved to a Notion database, and finally, we send the results back to the user through the Telegram API.

Here is the complete picture of the workflow.

Setting up Telegram Bot

Setting up the telegram bot is a straightforward process though Telegram’s bot father. n8n has native support for the Telegram node, so we only need to provide the API key to start using it.

Setting up Notion

n8n also natively supports the Notion node. We need to create a new Notion database to store the video summaries. Don’t forget to active the API connection to the database so that n8n can access the database through API.

Setting up AI Prompts

For this setup, I am using Gemini AI model. However, it is not limited to Gemini AI. It is easy to change the model to any AI model.

I created two AI nodes in my workflow. The first node is a simple LLM to extract a YouTube video ID from the chat message. The second one is an AI Agent node. This node allows us to specify the model, memory, and tools. We use Gemini AI model as well to summarize the video using the transcript content passed from the previous. The node also connects to the YouTube API tool so that it is able to retrieve the video’s description and title, enhancing the result.

Here is the prompt I used for the first node.

You are an expert extraction algorithm.
Extract only the YouTube video ID from the provided text.
If you do not know the value of an attribute asked to extract, you may omit the attribute's value.

Here is the second prompt.

Role: You are an expert AI assistant specialized in analyzing video transcripts and generating concise, informative summaries.

Goal: Your primary goal is to create a clear and accurate summary of the provided video transcript, presented as a bulleted list. The summary should capture the main topics and key points of the video content, including references to important timestamps.

Context: You will receive a block of text representing the full transcript or captions of a YouTube video. This transcript may contain timestamps (e.g., [00:01:23], 0:05:10.234 --> 0:08:567) and the spoken words.

Instructions:

Get information regarding the Youtube Video. Use the Youtube API tool to retrieve the video title and description. 

Identify Main Topics: Read through the entire transcript to understand the core subjects discussed.

Extract Key Points: Pinpoint the most important statements, arguments, findings, or conclusions presented in the video.

Identify Key Timestamps: Note the approximate starting timestamps associated with the key points or main topic shifts you identify. Use the timestamps provided in the transcript.

Synthesize Information: Combine the main topics and key points into a coherent summary.

Include Timestamps: When mentioning a key point or topic in the summary, include its corresponding approximate start timestamp in parentheses (e.g., (0:02:15)).

Maintain Neutrality: Summarize the content objectively, without adding personal opinions or interpretations unless explicitly asked.

Focus on Content: Ignore conversational filler words (e.g., "um," "uh," "like," "you know") and repetitive phrases where possible, unless they are crucial to the meaning or tone. Focus on the substance of what is being said.

Desired Length (Optional - Adapt): Adjust the number of bullet points based on the desired level of detail (e.g., 3-5 points for a standard summary, more for detailed).

Output Format: Starts from the video Title followed by summarized video description. Then present the summary as a bulleted list. Each bullet point should represent a key topic or finding. Ensure timestamps are included parenthetically where relevant points are mentioned. Start directly with the bulleted list.

Input Data:

The video transcript will be provided below, enclosed in triple backticks or following a specific marker (e.g., "TRANSCRIPT START").

{{ $json.transcript_data }}

(Adjust the placeholder {{ $json.transcript_data }} based on how the transcript data is passed from the previous node in your n8n workflow. It might be $json.text, $input.item.json.transcript, etc.)

Example Interaction:

Input: (A full video transcript about making pasta, with timestamps)

Output (Bulleted List Summary):

The video provides a step-by-step guide on making fresh pasta from scratch (0:00:15).

It details the necessary ingredients, primarily flour and eggs (0:01:05).

The process involves mixing, kneading (0:02:30), and resting the dough (0:05:10).

Key techniques like using the right flour and sufficient kneading are emphasized for texture (0:04:00).

Demonstrates how to roll and cut the pasta into desired shapes, such as fettuccine (0:06:45).

Concludes by showing the final cooked pasta served with sauce (0:08:20).

Now, analyze the following transcript and generate the summary based on these instructions.

Retrieving YouTube Video Transcript

Retrieving YouTube Video transcript is quite challenging. YouTube Data API does not allow public users to retrieve video transcript unless you are the creator of the video. I have come across some outdated solutions which do not work anymore. So, this means I need to build my own custom solution.

One approach is to use Gemini AI directly by giving the video URL and asking the model to summarize it. This works in chat mode but not via an API call. For some reasons, the Gemini models gives the summary of a wrong video, and I don’t quite understand why this happens. So, I have to find another solution.

Fortunately, there are libraries that can retrieve transcripts or captions given a video ID. There is one library written in JavaScript and another library in Python . Since these are external libraries, it is not easy to use them within the n8n environment using the custom code node. It is not possible in Python because n8n uses Pyodide to provide Python support. This limits the available Python packages that can be used. It is possible though in JavaScript but it only works on a self-hosted n8n instance. This entails updating the environment variables and a Dockerfile, which is very technical.

So, I’ve come up with an idea to host a server on Heroku and install the python library. This approach does not require any changes on the n8n instance and it also works on both self-hosted and cloud-hosted n8n instances.

Another challenge is that YouTube bans IPs if there’s too many requests from those IPs. Fortunately, we can use a Proxy server as a workaround. One Proxy server provider that is recommended by the library’s author is WebShare. So I decide to try it and it works quite well.

If you want to setup this workflow yourself, you should try WebShare. I appreciate if you can use my referral link .

If you want to know the implementation detail, you can check my repository here .

Conclusion

The AI-powered YouTube Video Summarizer workflow using n8n offers a practical solution for individuals who want to efficiently learn from YouTube videos without watching them in full. By integrating various tools such as Telegram, Notion, and AI models, this workflow automates the process of extracting, summarizing, and storing video content. This not only saves time but also can be useful for future references. The flexibility of the system allows for further customization and extension, opening up possibilities for further creative applications.

What’s Next

We now have a automation workflow that lets you get a video summary and store it on Notion’s database. There are lots of possibilities for what you can do next. You can of course consume the information yourself, or feed the information to another AI agent or n8n workflow to perform other creative tasks, like recommending next topics for your social media posts or generating new articles. The sky is the limit here.

Let me know if you have some other ideas in the comments section below. If you have suggestions for new automation workflows that could be useful, I’d love to hear them.

Workflow automation tools are software platforms that allow users to create automated sequences of actions across various services and applications with a set of predefined rules. By using workflow automation, the teams can focus on the most important things they need to get done, and not on repetitive administrative tasks. With the right tool and setup, you can reduce errors, speed up processes, and increase productivity.

There are lots of workflow automation tools out there. In this article, I am going to describe one of the most popular and the most promising platform, n8n .

About n8n

n8n is an open-source workflow automation platform that was launched in 2019 by Jan Oberhauser. Similar to other automation platforms, it offers a drag-and-drop interface, third-party applications or services (including AI) integration, and custom code extension.

Advantages over other platforms

n8n has some advantages and unique features compared to other platforms.

• Visual workflow builder. It offers an drag-and-drop interface to design workflows.

• Self-Hosting. It is also possible to self-host n8n on your own server for free. With self-hosting, it is possible to install custom community nodes.

• Cost Efficiency. Compared to other platforms, n8n cloud costs around $20 per month for 2500 executions with unlimited steps.

• Extensibility. n8n supports custom code that can be written in either JavaScript or Python. It is also possible to connect to any API. Hence, n8n can support virtually unlimited integration.

• AI Integration. Easy to build a AI-powered automation workflow.

• Strong community support with 600+ pre-built workflow templates.

Learn More

If you are thinking about automating your workflow, it might be worthwhile to try n8n. I myself have built a workflow using n8n to automate my blogging workflow using Hashnode and Notion. You can read more in this article . If cost is your main concern, don’t worry because it is possible to self-host n8n. If you want to a quick start, you can try n8n using n8n directly and you will get a 14-day trial period.

Summary

Modern workflow automation tools have evolved from simple script-based automation to sophisticated platforms with visual interfaces and advanced features. Whether you're a small business owner, developer, or enterprise solution architect, workflow automation tools can significantly improve your operational efficiency and help you focus on more strategic tasks rather than routine operations.

These tools now incorporate artificial intelligence and machine learning capabilities, making them more intelligent and adaptive to business needs.

However, there’s lots of room for improvement. If we look at the structure of the workflow that has been built using Make, it looks quite complex and not so intuitive. That makes things if there is anything I can do to improve it. When I discovered n8n, I immediately could see that I can build a better automated blogging workflow using this platform. And I will describe the setup in this article.

Automate Blogging Workflow

Imagine managing blogs with numerous posts. As a content creator, you want to focus on creating posts for your blogs. The process of writing and updating previously uploaded blog posts involves tedious navigation through the blogging platform, like Hashnode. Wouldn't it be more efficient to manage all your posts within Notion and having them automatically uploaded to the blogging platform? This automation workflow aims to boost your productivity by streamlining the process, allowing you to focus more on content creation and less on the administrative tasks of blogging.

About n8n

n8n is an open-source workflow automation platform that allows users to connect various applications and automate tasks with a drag-and-drop interface. It offers an intuitive interface to easily build a workflow automation with the possibility to insert custom code. This gives the flexibility of code with the speed of no-code.

Compared to Zapier, IFTTT, and Make, n8n offer several unique advantages:

• Intuitive interface. It is very easy to specify trigger options and to connect different applications and tools using a drag-and-drop interface.

• Custom-code. It supports custom codes written in JavaScript and Python. This enables technical users to build powerful automatic workflows, giving more flexibility.

• Self-hosted option. Hence, it is possible to run n8n totally free.

Building the Workflow

Building a workflow on n8n is much easier. It is pretty intuitive to connect the apps, tools, and supporting logic nodes (such as If-else, filter, merge, and so on). It is similar to a flow-chart.

The components needed for this workflow are as follows.

• Notion, where the contents of the blog post are stored.

• Hashnode.

• AWS S3, to store images.

• Code node for custom codes.

Here is the complete setup of the workflow. As you can see, the workflow looks simpler compared to the one I built on Make. It is also easier to understand, in my opinion. There is no weird construct in the workflow.

The workflow consists of two parts. The first part is the trigger. The workflow is triggered when there’s some updates (i.e., a new page has been created or updated) in the table.

The second one is the main part for the workflow. It retrieves Notion pages and for each page, it will parse the full contents, upload images to AWS S3, convert the contents to Markdown format, and publish the contents to Hashnode using GraphQL.

The conversion from Notion blocks to Markdown has to be manually implemented in a code node. This is how it looks like.

# Loop over input items and add a new field called 'myNewField' to the JSON of each one
output = {}
output["markdown"] = ""
output["items"] = []

def check_annotate(item, type):
  parsed_text = ""
  rich_texts = item.json[type]["text"]
  for rich_text in rich_texts:
    text = rich_text["plain_text"].strip().rstrip()
    if rich_text["annotations"]["bold"]:
      text = "*" + text + "*"
    if rich_text["annotations"]["italic"]:
      text = "**" + text + "**"
    if rich_text["href"]:
      text = "[" + text + "](" + rich_text["href"] + ")"

    parsed_text += text + " "

  parsed_text = parsed_text.replace('    ', '\t').replace('"', '\"').replace('
', '\n')
  parsed_text += "\n\n"
  return parsed_text

for item in _input.all():
  parsed_text = ""
  if (item.json["type"] == "paragraph"):
    parsed_text = check_annotate(item, item.json["type"])
    output["items"].append(parsed_text)
  elif (item.json["type"] == "image"):
    parsed_text = "![](" + item.json["image"]["file"]["url"] + ")\n\n"
    output["items"].append(parsed_text)
  elif (item.json["type"] == "divider"):
    output["items"].append("---

")
  elif ("heading_" in item.json["type"]):
    parsed_text = check_annotate(item, item.json["type"])
    if item.json["type"] == "heading_1":
      parsed_text = "# " + parsed_text
    elif item.json["type"] == "heading_2":
      parsed_text = "## " + parsed_text
    elif item.json["type"] == "heading_3":
      parsed_text = "### " + parsed_text
    output["items"].append(parsed_text)
  elif (item.json["type"] == "bulleted_list_item"):
    parsed_text = check_annotate(item, item.json["type"])
    parsed_text = "- " + parsed_text + "\n"
    output["items"].append(parsed_text)
  elif (item.json["type"] == "numbered_list_item"):
    parsed_text = "1"
    output["items"].append(parsed_text)
  elif (item.json["type"] == "code"):
    parsed_text = check_annotate(item, item.json["type"])
    parsed_text = "```\n" + parsed_text + "```\n\n"
    output["items"].append(parsed_text)

  output["markdown"] += parsed_text
return output

Pricing Models

Compared to Make’s pricing model which is based on the number of operations performed, n8n uses a different pricing model. The price depends solely on your hosting solution and not the number of operations. If you self-host n8n, you can run it for free. n8n also offers a n8n cloud-host option, it will cost you around $20 per month. Hence, it is much cheaper than Make.

Summary

n8n offers a very intuitive, easy-to-use interface to build any automation workflow. Compared to Make, it also offers a better pricing model. By using Hashnode, Notion, and n8n, we are able to build a automatic blogging workflow that can significantly enhance your productivity and streamline your blog management process. Having this automation allows content creators to focus on content creation rather than mundane administrative tasks. I hope this article can be helpful!. Happy automating.

Before we dive into the specifics, let's take a moment to understand what Make is and how it compares to other popular automation tools like Zapier and IFTTT.

Make, formerly known as Integromat, is a versatile automation platform that allows users to design, build, and automate workflows called "scenarios." These scenarios can connect various apps and services, enabling seamless data transfer and task execution. Make stands out for its visual approach to automation, representing workflows as a series of interconnected modules.

Compared to Zapier and IFTTT, Make offer several unique advantages:

• Visual Interface. Make provides a flowchart-like UI interface to easily visualize and create a complex workflow at glance.

• Flexibility. It allows for branching paths, loops, and conditional logic, giving users more control over their automations.

• Better pricing model. Make’s pricing model is based on the number of operations performed. This is more cost-effective for users with complex but infrequently run automations.

In this article, we'll harness the power of Make to create an automated workflow that retrieves your blog posts from Notion and publishes them directly to Hashnode. This integration will save you valuable time and ensure a smooth, consistent publishing process.

Building Hashnode Custom App

The official Hashnode module is unfortunately not available on Make yet. The good news is that it is possible to create our own custom app on Make. This custom app relies on Hashnode GraphQL API to perform query or mutation actions.

For this use case, I have built the custom Hashnode app which supports modules to create a new post and update an existing post. In the future, I might add other modules as I see fit for my use cases.

If you want to try it, here is the link to the custom app → Hashnode Make Module .

Building the Scenario

Now it is time to create the scenario. Make makes it possible to easily create a scenario with a very easy-to-use UI. It might be awkward to use at first, but I think the learning curve is not steep.

Integrating Notion with Hashnode is not as easy as I thought. Hashnode accepts content in Markdown format, while Notion uses a different kind of data format. It uses the concept of blocks, and each block represents a different content type. For example, a block can be a paragraph, an image, a list, or a heading. Hence, before we are able to publish the content from Notion to Hashnode, we have to convert Notion’s blocks to Markdown. That’s where the complexity of the scenario comes from.

Here is the complete setup of my scenario. Currently, it only supports blocks with heading, paragraph, image, list items, and link types. Of course, I will update the scenario to support different types of blocks.

Notes

Make’s pricing model is based on the number of operations performed. Right now, the scenario is not very efficient in terms of the number of operations. Each block of Notion is going to consume around 8 operations (and can be more if it is a paragraph that has lots of annotations) in this scenario. If you have a blog post with many blocks, the number of operations consumed can be very high. That’s why it is important to combine several paragraphs or blocks into one block on Notion to reduce the number of operations. (Always use SHIFT+ENTER when writing a new paragraph on Notion).

How to Use

Simply create a Database and connect it to Make. Here is the template of the database → Link .

What’s Next

I will keep updating the scenario to support different types of blocks while trying to optimize the number of operations. I also plan to extend the scenario to support other tasks such as automatically sharing new posts on social media.

Thanks to Make, automating my blogging workflow has never been easier. Using a low-code approach and drag-and-drop operations, I was able to streamline my blogging workflow by automatically publishing my blog posts on Hashnode from Notion. Having this robust automation allows you to focus on what matters most - creating great content - while the technical aspects of publishing are handled seamlessly in the background. I hope that this article can be helpful!

With this design pattern, we can achieve:

1. Separation of concerns, e.g. separate UI from the business logic. A button UI can be reused for other use cases.
2. Decoupled code.
3. Highly extensible, e.g. add new commands.
4. Testable and maintainable code.

   Structure

It consists of 5 components:

• Command . An interface for executing an operation

• ConcreteCommand . Connects receiver with an action and implements Execute

• Client . Creates and schedules ConcreteCommands .

• Invoker. Runs the commands, typically using callback. For example when a button is pressed.

• Receiver . Performs the operation requested by the command.

Implementation

Classic Example

class Command {
public:
    virtual void Execute() const = 0;
    virtual ~Command() = default;
};

class BookVenueCommand: public Command {
    Venue* _venue;
    int _remainingSeats, _numberOfSeatstToBook;
    ...
    public:
        BookVenueCommand(Venue* venue) : _venue(venue) {}

        virtual void Execute() override {
                _remainingSeats = _venue->BookSeats(_numberOfSeatsToBook);
        }

        BookVenueCommand(Venue* venue, int numberSeatsToBook, TicketType ticketType) {
            ...
        }

        int GetNumberOfRemainingSeats() { return _remainingSeats; }
}

class ReverseTicketCommand: publicCommand {
    PriceHandler* _priceHandler;
    VneueType _venueType;
    ...
    public:
        virtual void Execute() override {
            ...
            double ticketPrice = _priceHandler->handlePrice(ticket);
            _tickets->push_back(ticket);

            std::cout << fmt::format("You reserved a ticket for {0} ...", ticket.getNumberOfSeats();
        }
}

int main() {
    BookVenueCommand* bookHugeTheather = new BookVenueCommand(_hugeTheatre, 10, ticketType);
    bookHugeTheather->Execute();

}

Macro Command

class MacroCommand : public Command {

    std::list<Command*>* _commands;

    public:
        MacroCommand() { _commands = new std::list<Command*>(); }

        virtual void Add(Command* command) {
            _commands->push_back(command);
        }

        virtual void Remove(Command* command) {
            _commands->remove(command);
        }

        virtual void Execute() {
            std::list<Command&>::iterator iterator;

            for(iterator=_commands->begin(); iterator != _commands->end(); ++iterator)
            {
                (*iterator)->Execute();
            }
        }    
}

int main()
{
    MacroCommand* bookingMacroCommand = new MacroCommand;

    BookVenueCommand* command1 = new BookVenueCommand(...);
    bookingMacroCommand->Add(command1);
    BookVenueCommand* command2 = new BookVenueCommand(...);
    bookingMacroCommand->Add(command2);

    bookingMacroCommand->Execute();

}

Undo Command

// Simply adding a new method in the command class
virtual void Undo() = 0;

// Save the commands into history.

// Can be combined with Memento design pattern and/or Prototype to handle history

Notes

Chain Responsibility passes a request sequentially along a dynamic chain of potential receivers until one of them handles it.

Handlers in Chain Responsibility can be implemented as Commands. Execute different operations over the same context object/ request.

Requests in Chain Responsibility can be implemented as Commands. Execute the same operation in a series of different contexts linked into a chain.

Command establishes unidirectional connection between senders and receivers.

Mediator eliminates direct connections between senders and receivers

Observers lets receivers to dynamically subscribe to and unsubscribe from receiving requests.

Credits

Command in C++ / Design Patterns (refactoring.guru)

Strategy design pattern uses composition approach.

It separates the business logic of a class from the implementation details of algorithms (SRP).

Adding a new algorithm won’t affect the client code (Open/Close Principle).

Structure

Implementation

Classic Example (Dynamic)

// Define the interface common to all supported versions of some algorithms
class Strategy {
public:
    virtual ~Strategy() = default;
    virtual std::string doAlgorithm(std::string_view data) const = 0;
}

class ConcreteStrategyA : public Strategy {
public:
    std::string doAlgorithm(std::string_view data) const override
    {
        std::string result(data);
        std::sort(std::begin(result), std::end(result));
        return result;
    }
}

class ConcreteStrategyB : public Strategy {
public:
    std::string doAlgorithm(std::string_view data) const override
    {
        std::string result(data);
        std::sort(std::begin(result), std::end(result), std::greater<>());
        return result;
    }
}

class Context
{
private:
    std::unique_ptr<Strategy> m_strategy;

public:
    explicit Context(std::unique_ptr<Strategy>&& strategy = {}): strategy_(std::move(strategy)) {}

    // ALternatively use factory pattern
    void set_strategy(std::unique_ptr<Strategy>&& strategy) {
        m_strategy = std::move(strategy);
    }

    void doSomething() const {
        if (m_strategy) {
            std::string result = m_strategy->doAlgorithm("abcde");
        } else {
            std::cout << "Context Strategy is not set
";
        }
    }
}

int main() {
    Context context(std::make_unique<ConcreteStrategyA>());
    context.doSomething();
}

Static Strategy (Template)

If you want to avoid using vtable.

template<typename LS>
struct TextProcessor {
    void append_list(const vector<string> &items) {
        m_list_strategy.start(m_oss);
        for (auto & item: items)
            m_list_strategy.add_list_item(m_oss, item);
        m_list_strategy.end(m_oss);
    }

    string str() const { return m_oss.str(); }
private:
    ostringstream       m_oss;
    LS                  m_list_strategy;
};

int main() {
    // markdown
    TextProcessor<MarkdownListStrategy> tp1;
    tp1.append_list({ "foo", "bar", "baz" });
    cout << tp1.str() << endl;

    // html
    TextProcessor<HtmlListStrategy> tp2;
    tp2.append_list({ "foo", "bar", "baz" });
    cout << tp2.str() << endl;

    return EXIT_SUCCESS;
}

Functional approach using Lambda

The strategy can be implemented as an anonymous function instead. With this approach, we can avoid bloating the code with extra classes and interfaces

Credits

https://refactoring.guru/design-patterns/strategy

https://vishalchovatiya.com/posts//strategy-design-pattern-in-modern-cpp/

It achieves loose coupling between the sender of a request and its receiver.

It enforces separation of concerns between the sender/ client and the handlers.

Use Cases

This pattern is suitable when there is a need to process different kinds of requests in a various way.

You can implement this design patterns in the following use-cases.

• Filtering requests as in an authentication process.

In this case, the request will be passed to the receiver if the validation process in each handler passes.

• Finding a suitable handler as used in the customer service workflow.

In this case, the correct handler will pick up, process, and drop the request.

Structure

Implementation

Classic Example

class Handler
{
    Handler* _nextHandler;
public:
    virtual Handler* SetNext(Handler *handler) = 0;
    virtual std::string Handle(std::string request) = 0;
};

class DefaultAbstractHandler : public Handler {
    std::string Handle(std::string request) override {
        if (_nextHandler != nullptr) {
                return _nextHandler->Handle(request);
        }

        return {};
    }

    Handler* SetNext(Handler* handler) override {
        _nextHandler = handler;
        return handler; // so you can link the next handler in a convenient way.
    }
}

class SmallHandler: public DefaultAbstractHandler {
    std::string Handle(std::string request) override {
        if (request == "small") {
            return "Small handler\n";
        } else {
            return DefaultAbstractHandler::Handle(request);
        }    
    }
}

...

int main() {
    SmallHandler* smallHandler = new SmallHandler;
    MediumHandler* mediumHandler = new mediumHandler;
    smallHandler->SetNext(mediumHandler);

    std::vector<std::string> requests {"small", "medium", "large"};
    for (auto& inp: requests) {
        smallHandler->Handle(inp);
    }
}

Improving Classic Example using std::vector

Improve further decoupling → Handlers are not aware of each other anymore

class PriceHandler {
    std::vector<PriceReceiver*> _receivers;

public:
    virtual PriceHandler* SetNext(PriceReceiver* handler) {
        _receivers.push_back(handler);
        return this;
    }

    virtual double HandlePrice(Ticket ticket) {
        for (auto receiver: _receivers) {
            double res = ...;

            if (res) {
                return res;
            }
        }

        return 0;
    }

    virtual double HandleTotalPrice(std::vector<Ticket> tickets) {
        for (Ticket ticket: tickets) {
            totalPrice += HandlePrice(ticket);
        }

        return totalPrice;
    }
}

Using Boost Event

Mentioned here .

This implementation, however, does not allow you to drop a request if it has been handled.

Image Credits

Chain of Responsibility in C++ / Design Patterns (refactoring.guru)

Note that design patterns are not ready-to-use code solutions. They describe or suggest how the solution should look like.

Benefits of Design Patterns

• Teach how to solve all sorts of problems using principles of Object-Oriented Design

• Adhere to SOLID principles

• Achieve highly cohesive modules with minimal coupling (Extensibility and reusability).

• Communication tools between designers and developers

Types of Design Patterns

Creational Design Patterns

• Factory/ Abstract Factory

• Builder

• Prototype

• Singleton

Structural Design Patterns

• Adapter

• Bridge

• Composite

• Decorator

• Facade

• Flyweight

• Proxy

Behavioral Design Patterns

• Chain of Responsibility

• Command

• Interpreter

• Iterator

• Mediator

• Memento

• Observer

• State

• Strategy

• Template Method

• Visitor

One of the features offered in Outerbase is the ability to create and install plugins. Plugins enable us to perceive and interpret our data in custom visually engaging experiences. In this article, I will describe how I created a plugin for Outerbase to visualize GPS information.

About the Plugin

I would like to create a plugin that allows users to visualize the addresses in the database table on a map view (Google Maps in this case). Instead of just looking at GPS coordinates in a number form, why not visualize them directly on a map? It is more appealing and useful.

For the demo, I am using a dummy database generated using UIBakery (https://uibakery.io/sql-playground). They can generate a Car dealer database which contains a table that stores the location of all cars they have. The plugin uses the longitude and latitude information to show the location on a map.

It is a simple plugin but I imagine this plugin can also be used in different scenarios or domains, for example, Asset Management where you can show the location of your assets visually on a map.

Creating the Plugin

In this section, we will see how the configuration model is defined. After the configuration model is ready, we can start building the configuration view and the table plugin view.

Configuration Model

Here we define a class model that captures the types of information for the plugin to render. For the Map View plugin, we want the users to specify which column(s) to reference the following fields:

1. Longitude and Latitude.

2. Image.

3. Title.

4. Subtitle.

5. Description.

The longitude and latitude are used to create the markers on the map; the remaining items are used to render the "detail view" when the user clicks on the marker.

Two extra fields are added: longitudeKey and latitudeKey.

class OuterbasePluginConfig_$PLUGIN_ID {
    // Inputs from Outerbase for us to retain
    tableValue = undefined
    count = 0
    page = 1
    offset = 50
    theme = "light"

    // Inputs from the configuration screen
    imageKey = undefined
    optionalImagePrefix = undefined
    titleKey = undefined
    descriptionKey = undefined
    subtitleKey = undefined
    longitudeKey = undefined
    latitudeKey = undefined

    // Variables for us to hold state of user actions
    deletedRows = []

    constructor(object) {
        this.imageKey = object?.imageKey
        this.optionalImagePrefix = object?.optionalImagePrefix
        this.titleKey = object?.titleKey
        this.descriptionKey = object?.descriptionKey
        this.subtitleKey = object?.subtitleKey
        this.latitudeKey = object?.latitudeKey
        this.longitudeKey = object?.longitudeKey
    }

    toJSON() {
        return {
            "imageKey": this.imageKey,
            "imagePrefix": this.optionalImagePrefix,
            "titleKey": this.titleKey,
            "descriptionKey": this.descriptionKey,
            "subtitleKey": this.subtitleKey,
            "latitudeKey": this.latitudeKey,
            "longitudeKey": this.longitudeKey
        }
    }
}

Configuration View

After defining the configuration model, we can start defining how we present the user interface to the users so that they can fill in the configuration model. For this plugin, I would like to ask the users to specify which column each field should link to. In addition, the detail view will be previewed when the user starts filling in the fields.

Thankfully Outerbase team has done an awesome job providing an example plugin code. I used the example code and made some small modifications to build the configuration view for my plugin.

What I did was add two extra fields for the longitude and latitude. The preview is still displayed on the right side of the view.

class OuterbasePluginConfiguration_$PLUGIN_ID extends HTMLElement {

    // ...

    render() {
        let sample = this.config.tableValue.length ? this.config.tableValue[0] : {}
        let keys = Object.keys(sample)

        if (!keys || keys.length === 0 || !this.shadow.querySelector('#configuration-container')) return

        this.shadow.querySelector('#configuration-container').innerHTML = `
        <div style="flex: 1;">

            // ... Truncated ...

            <p class="field-title">Subtitle Key</p>
            <select id="subtitleKeySelect">
                ` + keys.map((key) => `<option value="${key}" ${key === this.config.subtitleKey ? 'selected' : ''}>${key}</option>`).join("") + `
            </select>

            <p class="field-title">Longitude Key</p>
            <select id="longitudeKeySelect">
                ` + keys.map((key) => `<option value="${key}" ${key === this.config.longitudeKey ? 'selected' : ''}>${key}</option>`).join("") + `
            </select>

            <p class="field-title">Latitude Key</p>
            <select id="latitudeKeySelect">
                ` + keys.map((key) => `<option value="${key}" ${key === this.config.latitudeKey ? 'selected' : ''}>${key}</option>`).join("") + `
            </select>

            <div style="margin-top: 8px;">
                <button id="saveButton">Save View</button>
            </div>
        </div>

        <div style="position: relative;">
            <div class="preview-card">
                <img src="${sample[this.config.imageKey]}" width="100" height="100">

                <div>
                    <p style="margin-bottom: 8px; font-weight: bold; font-size: 16px; line-height: 24px; font-family: 'Inter', sans-serif;">${sample[this.config.titleKey]}</p>
                    <p style="margin-bottom: 8px; font-size: 14px; line-height: 21px; font-weight: 400; font-family: 'Inter', sans-serif;">${sample[this.config.descriptionKey]}</p>
                    <p style="margin-top: 12px; font-size: 12px; line-height: 16px; font-family: 'Inter', sans-serif; color: gray; font-weight: 300;">${sample[this.config.subtitleKey]}</p>
                </div>
            </div>
        </div>
        `

        // ....
        // Whenever the latitudeKey and longitudeKey are changed, re-render the view.

        var latitudeKeySelect = this.shadow.getElementById("latitudeKeySelect");
        latitudeKeySelect.addEventListener("change", () => {
            this.config.latitudeKey = latitudeKeySelect.value
            this.render()
        });

        var longitudeKeySelect = this.shadow.getElementById("longitudeKeySelect");
        longitudeKeySelect.addEventListener("change", () => {
            this.config.longitudeKey = longitudeKeySelect.value
            this.render()
        });
    }
}

Here is the final result.

Table Plugin View

This is the most important part of the plugin. The way it works is as follows: It receives all of the rows from the table and displays the results to the users. Using this plugin, we want to display the results to the users on a Google Maps view.

Given the longitude and latitude information, we will create a marker for each entry in the table. When the user clicks on one for the marker, they will see more detailed information about the marker.

The first challenge in using embedded Google Maps API in this plugin is importing the Google Maps API. Google provides a guide on how to use Google Maps API in a Web component. The document can be found here. It is still in preview mode, so I guess things might change in the future.

To import Google Maps API, I created a new DOM element for script. Here we can specify the src of the script. After that, we need to append this element to the shadowDOM. The script will be evaluated when it is being appended.

// ...
var script = document.createElement('script')
script.type = 'text/javascript'
script.async = true
script.src = "//maps.googleapis.com/maps/api/js?key=<YOUR_API_KEY>&libraries=maps,marker&v=beta&callback=initMap";

// ...

class OuterbasePluginTable_$PLUGIN_ID extends HTMLElement {
    // ...

    constructor() {
        super()

        this.shadow = this.attachShadow({ mode: "open" })    
        this.shadow.appendChild(script)
        this.shadow.appendChild(templateTable.content.cloneNode(true))
    }

After the Google Maps Javascript API has been included, we can start using the map by using the <gmp-map> custom element in the render function of the plugin table class. To add a marker, we can nest an <gmp-advanced-marker> element inside <gmp-map>. So in the code snippet, you can see that the <gmp-advanced-marker> element is added for every table row in the database (this.config.tableValue).

In addition, I would like to show a small info window showing more detailed information when a marker is being clicked on. To achieve this, we need to add an event listener to every <gmp-advanced-marker> element and the callback whenever the event occurs. The info window is just an HTML element. Since every map marker is unique, we need to pass some information from the TableValue. This can be easily done by adding data-<attribute> elements into the <gmp-advanced-marker> element. We can access this information in the callback by accessing the dataset variable.

Here is what the complete render() function looks like.

render() {

        console.log(this.config.page)

        this.shadow.querySelector("#container").innerHTML = `
        <div class="grid-container">
            <h1>Welcome to the Outerbase Car Dealership!</h1>
            <div class="grid-item">
                <gmp-map id="marker-click-event-example" center="43.4142989,-124.2301242" zoom="4" map-id="DEMO_MAP_ID">
                    ${this.config?.tableValue?.length && this.config?.tableValue?.map((row) => `
                        <gmp-advanced-marker position="${row[this.config.latitudeKey]},${row[this.config.longitudeKey]}" title=${row[this.config.titleKey]} 
                            data=title='${row[this.config.titleKey]}'
                            data-image='${row[this.config.imageKey]}'
                            data-description='${row[this.config.descriptionKey]}'
                            data-subtitle='${row[this.config.subtitleKey]}'
                        >
                        </gmp-advanced-marker>
                    `).join("")}
                </gmp-map>
            </div>

            <div style="display: flex; flex-direction: column; gap: 12px;">
                <h1>What Next?</h1>
                <button id="previousPageButton">Previous Page</button>
                <button id="nextPageButton"}>Next Page</button>
            </div>
        </div>
        `

        const advancedMarkers = this.shadow.querySelectorAll("#marker-click-event-example gmp-advanced-marker");

        for (const advancedMarker of advancedMarkers) {
            customElements.whenDefined(advancedMarker.localName).then(async () => {
                advancedMarker.addEventListener('gmp-click', async () => {

                    if (this.infoWindow) {
                        this.infoWindow.close()
                    }

                    const {InfoWindow} = await google.maps.importLibrary("maps");

                    const content = document.createElement('div');
                    content.classList.add("property")
                    content.innerHTML = `
                        <style>
                            #theme-container {
                                height: 100%;
                            }

                            #container {
                                display: flex;
                                flex-direction: column;
                                height: 100%;
                                overflow-y: hidden;
                                width: 450px;
                            }

                            .grid-container {
                                flex: 1;
                                display: grid;
                                // grid-template-columns: repeat(2, minmax(0, 1fr));
                                gap: 12px;
                                padding: 12px;
                            }

                            .grid-item {
                                position: relative;
                                display: flex;
                                flex-direction: column;
                                background-color: transparent;
                                border: 1px solid rgb(238, 238, 238);
                                border-radius: 4px;
                                box-shadow: 0 4px 4px 0 rgba(0, 0, 0, 0.05);
                                overflow: clip;
                            }

                            img {
                                vertical-align: top;
                                height: 300px;
                                object-fit: cover;
                            }

                            .contents {
                                padding: 12px;
                            }

                            .title {
                                font-weight: bold;
                                font-size: 16px;
                                line-height: 24px;
                                font-family: "Inter", sans-serif;
                                line-clamp: 2;
                                margin-bottom: 8px;
                            }

                            .description {
                                flex: 1;
                                overflow: hidden;
                                text-overflow: ellipsis;
                                font-size: 14px;
                                line-height: 20px;
                                font-family: "Inter", sans-serif;

                                display: -webkit-box;
                                -webkit-line-clamp: 3;
                                -webkit-box-orient: vertical;  
                                overflow: hidden;
                            }

                            .subtitle {
                                font-size: 12px;
                                line-height: 16px;
                                font-family: "Inter", sans-serif;
                                color: gray;
                                font-weight: 300;
                                margin-top: 8px;
                            }

                            p {
                                margin: 0;
                            }

                            .dark {
                                #container {
                                    background-color: black;
                                    color: white;
                                }
                            }
                        </style>

                        <div id="theme-container">
                            <div id="container">
                                 <div class="grid-item">
                                    ${ advancedMarker.dataset.image ? `<img src="${advancedMarker.dataset.image}">` : `` }

                                    <div class="contents">
                                        ${ advancedMarker.dataset.title ? `<p class="title">${advancedMarker.dataset.title}</p>` : `` }
                                        ${ advancedMarker.dataset.subtitle ? `<p class="subtitle">${advancedMarker.dataset.subtitle}</p>` : `` }
                                        ${ advancedMarker.dataset.description ? `<p class="description">${advancedMarker.dataset.description}</p>` : `` }
                                    </div>
                                </div>
                            </div>
                        </div>

                    `

                    this.infoWindow = new InfoWindow({
                        content: content
                    });
                    this.infoWindow.open({
                        anchor: advancedMarker
                    });
                });
            });
        }

        var previousPageButton = this.shadow.getElementById("previousPageButton");
        previousPageButton.addEventListener("click", () => {
            triggerEvent(this, {
                action: OuterbaseTableEvent.getPreviousPage,
                value: {}
            })
        });

        var nextPageButton = this.shadow.getElementById("nextPageButton");
        nextPageButton.addEventListener("click", () => {
            triggerEvent(this, {
                action: OuterbaseTableEvent.getNextPage,
                value: {}
            })
        });
    }

Demo Time

You can find the code in my Github repository.

Final Thoughts

I have had lots of fun building this plugin for the Outerbase platform. Web component is a new thing for me and I have got lots of issues in the beginning. But I am happy that I can make it work in the end. Overall, I think the ability to create a plugin and visualize the database tables directly on the platform is a cool feature. As more and more plugins are available, it can be a powerful tool to easily interact with our databases.

I hope this article helps you create a new plugin for Outerbase. If you have any questions or comments, feel free to do so.

Luckily for us, Espressif has provided a Docker image (espressif/idf) for building applications and libraries with specific versions of ESP-IDF. It appears that the image is regularly, making it reasonably safe to employ it for our setup.

Setup

• For this setup, you are using a host system running Windows OS with WSL2 enabled. The WSL2 will be used so that we can access the ESP development board from within the container.

• Docker Desktop installed

• The IDE used here is VSCode with the Dev Containers installed. It lets you use a Docker container as a full-featured development environment.

Constructing compose-dev.yaml

To use the Docker Dev Environment, we need to create the compose-dev.yaml file. It is quite simple with a few additions.

services:
  build-env:
    image: espressif/idf
    command: sleep infinity
    devices:
      - '/dev:/dev'
    init: true
    volumes:
      - ..:/workspaces

I added devices: '/dev:/dev' to map /dev directory from the WSL2 environment to the Docker container. This is useful for the next step when we want to communicate with the ESP board from inside the container.

Open Docker Desktop and go to Dev Environments. Simply click on the Create button and create the new environment. The container will stay running and you will see the following results.

Flashing and Monitoring ESP32

The ESP-IDF Programming Guide page outlines a method to communicate with the development boards from within the container. The idea is to use the remote serial port protocol (RFC2217). So I tried to follow the method described in the document, but it was not successful for me. Luckily there is another way to achieve this.

An alternative way for accessing the ESP development board from inside the container is using WSL2. By default, WSL2 does not support USB devices. Fortunately, there is a project officially supported by Microsoft that enables WSL2 to be able to access connected USB devices on Windows OS via a USB/IP mechanism. The project is called usbipd-win.

Setting up usbipd-win

1. Firstly we need to install usbipd-win. We just need to follow the installation instructions described on its Github repository page. It is pretty straightforward.

2. Open the WSL2 terminal and execute uname -a. If it reports a kernel version of 5.10.60.1 or later, then you don't need to do anything. Otherwise, follow the instructions on installing USB/IP client tools on WSL2 here.

3. Run Powershell as Administrator and execute the following command.

    ### Powershell
    PS C:\Windows\system32> usbipd wsl list
    BUSID  VID:PID    DEVICE                                                        STATE
    2-1    303a:1001  USB Serial Device (COM3), USB JTAG/serial debug unit          Not attached
    2-2    046d:c534  USB Input Device                                              Not attached
    2-3    0bda:4853  Realtek Bluetooth Adapter                                     Not attached
    3-1    04f2:b758  Integrated Camera, Integrated IR Camera, Camera DFU Device    Not attached
   

   Find the USB device of your ESP development board connected. In this example, it is assigned BUSID 2-1

4. Attach the USB device to WSL2 by running this command.

    ### Powershell
    PS C:\Windows\system32> usbipd wsl attach --busid 2-1
    usbipd: info: Using default WSL distribution 'Ubuntu-22.04'; specify the '--distribution' option to select a different one.
   

5. Go back to the WSL2 terminal and run lsusb to see that the ESP32 development can be accessed from WSL2 now via /dev/ttyACM0 (or /dev/ttyUSB0).

    ### WSL2
    > lsusb
    Bus 001 Device 014: ID 303a:1001 Espressif USB JTAG/serial debug unit
   
    > ls -al /dev/tty*
    ...
    crw------- 1 root root 166,  0 Aug 26 17:27 /dev/ttyACM0
    ...
   

6. Now we need to change the permission so that the device can be accessed by non-root users.

    ### WSL2
    > sudo chmod 666 /dev/ttyACM0
   

If you remember, we added an extra field in the compose-dev.yaml file. With this option, we map the /dev directory of WSL2 into the Docker container. Hence, it is possible to access your development board from inside the container.

...
    devices:
      - '/dev:/dev'
...

Automating usbipd

Attaching USB devices using usbipd requires manual steps that need to be executed whenever we plug or re-plug our ESP development board. It can be quite annoying over time. Luckily, there is a convenient GUI that can automate all these steps. This project was created by Andrew Leech. You can just download the package from here and install it.

The GUI is quite self-explanatory. You just need to find your ESP development board and click on 'Auto-Attach'. With this feature enabled, your ESP development board will be attached automatically.

udev

On WSL2, we have to configure the attached USB device so that non-root users can access it. We can use the chmod command to configure it manually, but then we have to do it every time the USB device is re-attached. To avoid all the hassles, we can create a udev rule.

Create a new file called 99-usbftdi.rules on /etc/udev/rules.d/ and add the following information to the file. The {idVendor} and {idProduct} values can be obtained from lsusb

> lsusb
Bus 001 Device 014: ID 303a:1001 Espressif USB JTAG/serial debug unit

> cat /etc/udev/rules.d/99-usbftdi.rules
SUBSYSTEM=='usb', ATTRS{idVendor}=="303a", ATTRS{idProduct}=="1001", MODE="0666"

# Then reload udevadm and restart the udev service
> udevadm control --reload
> sudo service udev restart

Now, every time the development board is plugged in, it will be automatically attached to your WSL2 environment and can be readily accessed from the container.

Final Result

Upon the successful creation of the development environment, the container can be opened in VSCode. The VSCode extensions, such as CMake and C++, will also be automatically installed.

If everything is correctly configured, it is possible to build the application inside the container

Since the container can access the USB device via WSL2, we are also able to directly flash the image to the ESP development board.

In this article, we discuss creating a Docker container using the Docker Dev Environment for building ESP32 applications and flashing them directly to an ESP board from within the Docker container. This streamlined workflow simplifies the development process and ensures a consistent development environment across different systems.

You can see the full setup on my Github repository.