Common Modules

jyablonski Common Modules is an internal Python library hosted on PyPI that is used throughout various services & applications in the project

---

How It Works

The library provides a collection of helper and utility functions that can be reused across scripts and services throughout the project.

It can be installed as a Python package in any environment using the following command:

pip install jyablonski-common-modules

The library currently includes functions for SQL-related connections and upsert operations, general Python logging, and AWS utilities. I plan to expand its functionality and scope as needed in the future.

Code Layout

The project is broken down like below:

• Directoryjyablonski_common_modules

  • Directoryaws/

    • …
  • Directorygeneral/

    • …
  • Directorylogging/

    • …
  • Directorysql/

    • …
• Directorytests

  • confest.py
  • Directoryunit/

    • …
  • Directoryintegration/

    • …

This source code structure allows for clean and concise import statements, such as:

from jyablonski_common_modules.logging import create_logger
from jyablonski_common_modules.sql import create_sql_engine, write_to_sql_upsert

To enable importing any function from subdirectories like sql or logging, the __init__.py for each subdirectory is configured like so:

from .connection import create_sql_engine
from .writers import write_to_sql_upsert


__all__ = ["create_sql_engine", "write_to_sql_upsert"]

CI / CD

For continuous integration (CI), the entire test suite is run on every commit in a pull request using Docker.

After a PR is merged, the continuous deployment (CD) pipeline performs the following steps:

1. Builds the library and generates platform-specific wheels
2. Publishes the new version and wheels to PyPI