From 8097508b361568d7623da0a5fafd43bfe23a0200 Mon Sep 17 00:00:00 2001 From: d33bs Date: Tue, 20 Feb 2024 17:01:26 -0700 Subject: [PATCH 01/16] init a new blog post; outline content --- _posts/2024-02-20-Lockfiles.md | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 _posts/2024-02-20-Lockfiles.md diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md new file mode 100644 index 0000000000..6ecefa6440 --- /dev/null +++ b/_posts/2024-02-20-Lockfiles.md @@ -0,0 +1,83 @@ +--- +title: "Lockfiles" +author: dave-bunten +tags: + - software-education + - python + - lockfiles + - dependency-management + - environment-management + - software-ecosystems +--- + +# Lockfiles + +{% include blog-post-intro.html %} + +## Introduction + + + + + +## What are dependencies? 📦 + +- [Reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel) +- [Not invented here](https://en.wikipedia.org/wiki/Not_invented_here) +- [Bounded context](https://martinfowler.com/bliki/BoundedContext.html) + +## The dependency wilderness 🌲 + + + + +```mermaid +gantt + your code :a, 2022-07-01, 3w + pandas :active ,b, 2022-07-07, 1w + numpy :active, c, 2022-07-09, 10d + scipy :active, d, 2022-07-04, 6d +``` + +_Caption here._ +{:.center} + +- Parallel timelines +- Dependency chaos +- Dependency management + +## Information Entropy + +- [Entropy (information theory)](https://en.wikipedia.org/wiki/Entropy_(information_theory)) +- Dependency entropy +- Systems theory stocks, flows, and uncertainty + +## Local development uncertainty + +- Continuous Integration +- Developer experience (DevEx) + +## Reducing uncertainty + + +## Examples + +- Python: + - [Poetry `poetry.lock` file](https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control) + - [PDM `pdm.lock` file](https://pdm-project.org/latest/usage/dependency/#specify-the-lockfile-to-use) +- Conda: [`conda-lock` `conda-lock.yml`](https://github.com/conda/conda-lock) +- R: [`renv` `renv.lock` file](https://rstudio.github.io/renv/articles/lockfile.html) +- Ruby: [Rubygems Gemfile.lock files](https://bundler.io/guides/rationale.html#checking-your-code-into-version-control) +- JavaScript: + - [NPM `package-lock.json` files](https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json) + - [Yarn `yarn.lock` files](https://classic.yarnpkg.com/lang/en/docs/yarn-lock/) +- Rust: [Cargo `cargo.lock` file](https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html) + +## Concluding Thoughts + From 5f18c40df05ce34e13905e9dab646a1f33198d80 Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 21 Feb 2024 11:52:52 -0700 Subject: [PATCH 02/16] more outlining --- _posts/2024-02-20-Lockfiles.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md index 6ecefa6440..9370f213a7 100644 --- a/_posts/2024-02-20-Lockfiles.md +++ b/_posts/2024-02-20-Lockfiles.md @@ -22,6 +22,7 @@ tags: ## What are dependencies? 📦 +- Example spectrum between including others work vs writing everything yourself. | <--------------> | - [Reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel) - [Not invented here](https://en.wikipedia.org/wiki/Not_invented_here) - [Bounded context](https://martinfowler.com/bliki/BoundedContext.html) @@ -58,13 +59,23 @@ _Caption here._ - Dependency entropy - Systems theory stocks, flows, and uncertainty -## Local development uncertainty +## Dependency uncertainty +- Dependency version ranges - Continuous Integration - Developer experience (DevEx) -## Reducing uncertainty +## Reducing uncertainty with lockfiles +- Snapshots in time with lockfiles +- Layers of depenedency certainty + + +## Using lockfiles + +- +- Dependabot https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/ +- Renovate https://github.com/renovatebot/renovate ## Examples From 8f97729e9b6ad19f229d88c845bc6bfef61a4810 Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 21 Feb 2024 15:43:02 -0700 Subject: [PATCH 03/16] first pass at introduction --- _posts/2024-02-20-Lockfiles.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md index 9370f213a7..cdae322c87 100644 --- a/_posts/2024-02-20-Lockfiles.md +++ b/_posts/2024-02-20-Lockfiles.md @@ -22,6 +22,11 @@ tags: ## What are dependencies? 📦 +Writing software often entails reusing existing software to solve common challenges and avoid ["reinventing the wheel"](https://en.wikipedia.org/wiki/Reinventing_the_wheel). +Using others software "inventions" helps save time and focus but comes with unique challenges like unpredictability. +These challenges can sometimes lead to ["not invented here syndrome"](https://en.wikipedia.org/wiki/Not_invented_here) where there's less trust in external-facing work outside of an individual or group of people (sometimes resulting in self-centric thinking). + + - Example spectrum between including others work vs writing everything yourself. | <--------------> | - [Reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel) - [Not invented here](https://en.wikipedia.org/wiki/Not_invented_here) From 799ff6b536ef3897a25bb0ba4aced0451022ee5c Mon Sep 17 00:00:00 2001 From: d33bs Date: Thu, 22 Feb 2024 08:59:50 -0700 Subject: [PATCH 04/16] further content development --- _posts/2024-02-20-Lockfiles.md | 36 +++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md index cdae322c87..966abe97b5 100644 --- a/_posts/2024-02-20-Lockfiles.md +++ b/_posts/2024-02-20-Lockfiles.md @@ -17,22 +17,27 @@ tags: ## Introduction +Writing software often entails reusing existing software to solve common challenges and take advantage of existing work. +An external piece of software used by a specific project can be called a "dependency" (the software "depends" on that external work to accomplish tasks). +Collections of software are oftentimes made available as "packages" through various platforms. +[Package management](https://en.wikipedia.org/wiki/Package_manager) for dependencies, the task of managing collections of dependencies for a specific project, is a specialized area of software development that can involve the use of unique tools and files. +This article will cover package dependency management through special files generally referred to as "lockfiles". +## Why use dependencies? -## What are dependencies? 📦 +- Example spectrum between including others work vs writing everything yourself. | <--------------> | -Writing software often entails reusing existing software to solve common challenges and avoid ["reinventing the wheel"](https://en.wikipedia.org/wiki/Reinventing_the_wheel). -Using others software "inventions" helps save time and focus but comes with unique challenges like unpredictability. -These challenges can sometimes lead to ["not invented here syndrome"](https://en.wikipedia.org/wiki/Not_invented_here) where there's less trust in external-facing work outside of an individual or group of people (sometimes resulting in self-centric thinking). +There are various advantages to using packaged dependencies in your projects. +Using existing work this way practices a collective ["don't repeat yourself [or ourselves]" (DRY)](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) to avoid [reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel). +Using dependencies allows us to make explicit decisions about the specific focus, or context, which the project will prioritize. +The concept of [bounded context](https://martinfowler.com/bliki/BoundedContext.html) from [domain-driven design](https://en.wikipedia.org/wiki/Domain-driven_design) is sometimes used to help distinguish what is in or out of scope for a particular project. +Bounded context can be used as a way to draw abstract lines around a certain [span of control](https://en.wikipedia.org/wiki/Span_of_control) in order to align available resources (like time and people) with the focus of the project. +Without these considerations and the use of dependencies we might face "endless" software [feature creep](https://en.wikipedia.org/wiki/Feature_creep) by continually adding new bounded contexts that are outside of our span of control (or resources). -- Example spectrum between including others work vs writing everything yourself. | <--------------> | -- [Reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel) -- [Not invented here](https://en.wikipedia.org/wiki/Not_invented_here) -- [Bounded context](https://martinfowler.com/bliki/BoundedContext.html) -## The dependency wilderness 🌲 +## The dependency wilderness -```mermaid +```mermaid! gantt - your code :a, 2022-07-01, 3w - pandas :active ,b, 2022-07-07, 1w - numpy :active, c, 2022-07-09, 10d - scipy :active, d, 2022-07-04, 6d + your project :a, 2024-07-01, 3w + pandas :active ,b, 2024-07-07, 1w + numpy :active, c, 2024-07-09, 10d + scipy :active, d, 2024-07-04, 6d ``` -_Caption here._ +_Dependencies are often on their own unpredictable schedule outside of your project's control._ {:.center} Using existing software package dependencies helps conserve resources but comes with unique challenges related to unpredictability (such as when those dependencies are updated). -This unpredictability can sometimes result in what's colloquially called ["dependency hell"](https://en.wikipedia.org/wiki/Dependency_hell), where for example multiple external dependencies conflict with one another and are unable to be automatically resolved. +This unpredictability can sometimes result in what's colloquially called ["dependency hell"](https://en.wikipedia.org/wiki/Dependency_hell), where for example multiple external dependencies conflict with one another and are unable to be automatically resolved (among other issues). These challenges can be especially frustrating due to when they occur (often outside of our personal schedule awareness) and how long they can take to debug (finding fixes sometimes entails costly [trial-and-error](https://en.wikipedia.org/wiki/Trial_and_error)). +It can feel like walking through a forest at night without a flashlight, constantly tripping over roots or running into stumps and branches! +## Illuminating the dependency thicket -```mermaid +```mermaid! flowchart LR - invention["Overwhelming\nreinvention"] + invention["Overwhelming\ninvention"] dependencychaos["Infinite\ndependency\nchaos"] invention <-......-> | software dependency \n choices |dependencychaos ``` -_Software dependency choices may be understood through careful consideration between the cost of reinvention vs (external) dependency chaos._ +_Software dependency choices may be understood through careful consideration between the cost of internal overwhelming invention vs external dependency chaos._ {:.center} Dependency chaos can sometimes lead to ["not invented here syndrome"](https://en.wikipedia.org/wiki/Not_invented_here) where there's less trust in external-facing work outside of an individual or group of people. -When or if this happens it can be important to reconsider existing knowledge on a topic area through formal or informal [literature review](https://en.wikipedia.org/wiki/Literature_review) (understanding that code within software is a type of literature). +When or if this happens it can be important to understand dependencies as a __scale of choices between overwhelming invention and infinite dependency chaos__. +For example, to accomplish a small project it may not be wise to create a brand new programming language (towards the extreme of overwhelming invention). +On the other hand, if we depended upon all existing work within a certain context the solution may not be specialized, efficient, or resourceful enough to meet the goals within a reasonable amount of time. + +```mermaid! +mindmap + root((Project)) + Data storage + File 1 + Database 2 + Data x processing + Package X + Package Y + Integration + Solution A + Platform B + +``` + +_Dependency awareness and opportunity can be grouped into concerns and documented as part of a literature review (seen here as a [mind map](https://en.wikipedia.org/wiki/Mind_map))._ +{:.center} + + +It can be helpful to reconsider existing knowledge on a topic area through formal or informal [literature review](https://en.wikipedia.org/wiki/Literature_review) (understanding that code within software is a type of literature) when thinking about the scale of decisions mentioned above. Outlining existing work through a literature review can help with [second-order thinking revision](https://en.wikipedia.org/wiki/Revision_(writing)#Reflection_in_the_revision_process) where we might benefit from reflecting on dependency decision-making again after an initial (first-order) creative process. +Each potential dependency discovered through this process can be organized using [separation of concerns (SoC)](https://en.wikipedia.org/wiki/Separation_of_concerns) under specific _concern_ labels, or a general set of information which affects related code. +Include dependencies within your project which will helpfully limit the code produced (or SoC _sections_) thereby reducing the overall amount of _concerns_ the project must maintain. -## Package dependency specification +```mermaid! +flowchart LR + + subgraph contextB["Context A"] + direction TB + processed1["data"] + widget + metadata1["metadata"] + end + + subgraph contextA["Context B"] + direction TB + processed2["data"] + gizmo + metadata2["metadata"] + end + + processed1 <-.-> | looosely coupled\nshared context | processed2 + widget ~~~ | distinct solutions\nwith different names | gizmo + metadata1 ~~~ | distinct solutions\nwith the same names | metadata2 +``` + +_Bounded contexts along with shared or distinct components can be used to help limit the complexity of a project in helpful ways._ +{:.center} + +The concept of [bounded context](https://martinfowler.com/bliki/BoundedContext.html) from [domain-driven design](https://en.wikipedia.org/wiki/Domain-driven_design) can sometimes be used to help distinguish what is in or out of scope for a particular project as a way of reducing complexity. +Bounded context can be used as a way to draw abstract lines around a certain [span of control](https://en.wikipedia.org/wiki/Span_of_control) in order to align available resources (like time and people) with the focus of the project. +It also can help promote [loose coupling](https://en.wikipedia.org/wiki/Loose_coupling) of software components in order to enable flexible design over time. +Without these considerations and the use of dependencies we might face "endless" software [feature creep](https://en.wikipedia.org/wiki/Feature_creep) by continually adding new bounded contexts that are outside of our span of control (or resources). + +## Version constraints as dependency specification control - +
Version constraintNatural language descriptionDescription of the version constraint
@@ -132,9 +185,9 @@ To accomplish this we can use what are sometimes called dependency "version rang These are often specific to the package management system and programming language being used. See the [Python Packaging Authority's Version Specifiers section](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) for an example of how these version constraints work. -## Version constraint flexibility can be chaotic +## Version constraints can still be chaotic -```mermaid +```mermaid! sequenceDiagram autonumber Dependency A (DepA) ->>Release Platform: v1.0.0 Released! @@ -145,34 +198,147 @@ sequenceDiagram Developer D -->> Release Platform: Look for version which matches constraint (>= 1.0.0) Release Platform ->> Developer D: Install DepA v1.1.0 (>= 1.0.0) Developer D->> Project C: Add new feature + rect rgb(254, 226, 226) Project C->> Developer D: Automated tests fail due to DepA changes! + end ``` _Unintentional failures can occur due to timeline variations between internal projects and external dependencies._ {:.center} -In addition to the flexibility of version range specifications we sometimes require repeatable behavior to be productive with a project. -For example, we want for each developer and continuous integration step to have reproducible environments even if a dependency gets updated while internal development takes place. +We sometimes require repeatable behavior to be productive with a project in addition to the flexibility of version range specifications. +For example, we may want for each developer and continuous integration step to have reproducible environments even if a dependency gets updated while internal development takes place. +Dependency version constraints oftentimes aren't enough on their own to prevent reproducibility issues from occurring. +See the above diagram for a timeline depicting how `Developer B` and `Developer D` may have different experiences despite best efforts with version constraints (`Dependency A` may make a release that fits the version constraint but breaks `Project C` when `Developer D` tries to modify unrelated code). + + +## Lockfiles for reproducible version constraint behavior + + +```mermaid! +sequenceDiagram + autonumber + Dependency A (DepA) ->>Release Platform: v1.0.0 Released! + Release Platform->>Developer B: Install DepA >= 1.0.0 + Developer B->>Project C: Add constraint DepA >= 1.0.0 (with lockfile) + Dependency A (DepA) ->>Release Platform: v1.1.0 Released! + Project C ->> Developer D: Install from project + Developer D -->> Release Platform: Look for version which matches constraint (>= 1.0.0) and lockfile data about versions + Release Platform ->> Developer D: Install DepA v1.0.0 (matching >= 1.0.0 and lockfile reference to versions available to previous changes) + Developer D->> Project C: Add new feature + rect rgb(209, 250, 229) + Project C->> Developer D: Automated tests isolate observations to new feature + end +``` + +Version constraint lockfiles provide one way to ensure reproducible behaviors within your projects. + +Lockfiles usually have the following characteristics (this varies by programming language and dependency type): + +- Lockfiles usually capture data about existing available dependencies which match a provided version constraint specification as a single file which can be added to source control. +- Lockfiles are referenced when available to help create reproducible installations of dependencies. +- Lockfiles are often automatically created or changed by a package or environment management tool of some kind. + +See the above modified timeline for `Developer B` and `Developer D` to better understand how their project will benefit from a shared lockfile and reproducible dependency installations. -## Using lockfiles +## Pythonic Example + + + + + + + + + + + + + + + + + + + + + + +
Python Poetry command usedDescription of what occurs
+ +`poetry add pandas` + + + +- Adds a [__caret-based version constraint specification__](https://python-poetry.org/docs/dependency-specification/#caret-requirements) based on the latest release (for example `^2.2.1`) within a `pyproject.toml` file. This version constraint can be understood as `>= 2.2.1, < 2.3.0`. +- Create or update the `poetry.lock` lockfile with known compatible versions of Pandas based on the version constraint mentioned above. +- Installs the version of Pandas which matches the `pyproject.toml` and `poetry.lock` specifications. + +
+ +`poetry install` + + + +Installs the version of Pandas which matches the `pyproject.toml` and `poetry.lock` specifications (for example, within a new environment or for another developer). + +
+ +`poetry update pandas` + + + +- Poetry checks for available Pandas releases which are compatible with the version constraint (for ex. `^2.2.1`). +- If there are new versions available which match the constraint, Poetry will update the `poetry.lock` lockfile and install the matching version. + +
+ +`poetry lock` + + + +- Update all dependencies referenced in the `poetry.lock` lockfile with the latest compatible versions based on the version constraints specified within the `pyproject.toml`. +- Optionally, if the `--no-update` flag is also used, refresh the dependency versions referenced within the `poetry.lock` lockfile based on version constraints specified within the `pyproject.toml` without seeking updated dependency releases. + +
+ +_Use Poetry commands to implement dependency version constraints and lockfiles for reproducible Python project environments._ +{:.center} + +[Poetry](https://python-poetry.org/) is a Python packaging and dependency management tool which implements version constraints and lockfiles to help developers maintain their software projects. +Using commands like `poetry add ...` and `poetry lock` automatically creates `poetry.lock` lockfiles based on specifications which are added either automatically or manually to `pyproject.toml` files. +Similar to other tools, Poetry can operate with or without `poetry.lock` lockfiles ([see here for more information](https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control)). +Another alternative to Poetry which makes use of lockfiles is [PDM (`pdm.lock` files)](https://pdm-project.org/latest/usage/dependency/#specify-the-lockfile-to-use). + +## Avoiding over-constrained dependencies + +```mermaid! +flowchart LR + + project["Project"] + developer["Developer"] + depbot["Dependency\nChecking Tool"] + securityvuln["Security\nVulnerability"] + pr["Pull\nRequest"] + + developer --> | reviews\nchanges | pr + securityvuln --> | announced and\nconsidered by | depbot + depbot --> | opens PR with\nrelated fix | pr + pr --> project +``` + +_Automated dependency checking tools like Dependabot or Renovate can be used to reduce project risk through timely dependency update changes assisted by human reviewers._ +{:.center} -- -- Dependabot https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/ -- Renovate https://github.com/renovatebot/renovate +Using dependency version constraints and lockfiles are helpful for reproducibility but imply a risk of over-constraint. +Two important over-constraint considerations are: -## Examples +- __Bug fixes:__ we may perpetuate an incorrect or failing solution within our project as the result of not installing later releases of a dependency. +- __Security fixes:__ we may unknowingly create security risks for others through the inclusion of known [security vulnerable](https://en.wikipedia.org/wiki/Vulnerability_(computing)) dependency versions. -- Python: - - [Poetry `poetry.lock` file](https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control) - - [PDM `pdm.lock` file](https://pdm-project.org/latest/usage/dependency/#specify-the-lockfile-to-use) -- Conda: [`conda-lock` `conda-lock.yml`](https://github.com/conda/conda-lock) -- R: [`renv` `renv.lock` file](https://rstudio.github.io/renv/articles/lockfile.html) -- Ruby: [Rubygems Gemfile.lock files](https://bundler.io/guides/rationale.html#checking-your-code-into-version-control) -- JavaScript: - - [NPM `package-lock.json` files](https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json) - - [Yarn `yarn.lock` files](https://classic.yarnpkg.com/lang/en/docs/yarn-lock/) -- Rust: [Cargo `cargo.lock` file](https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html) +Make sure to address these risks by routinely considering whether your dependencies need to be updated (manually) or through the use of automated tools like [GitHub's Dependabot](https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/) or [Mend Renovate](https://github.com/renovatebot/renovate). +Tools like Dependabot or Renovate enable scheduled checks and updates to be applied to your project which can lead to a balanced way of ensuring risk reduction and productive future-focused development. ## Concluding Thoughts From 9dcdcc82f871d12f513a1504a0b85c618b1d90d6 Mon Sep 17 00:00:00 2001 From: d33bs Date: Mon, 26 Feb 2024 12:52:07 -0700 Subject: [PATCH 08/16] add concluding thoughts --- _posts/2024-02-20-Lockfiles.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md index a24d28ddae..3bbd40ae53 100644 --- a/_posts/2024-02-20-Lockfiles.md +++ b/_posts/2024-02-20-Lockfiles.md @@ -342,3 +342,6 @@ Tools like Dependabot or Renovate enable scheduled checks and updates to be appl ## Concluding Thoughts +This article covered why dependencies are used, what complications they come with, and some tools to use addressing those challenges. +Every project can vary quite a bit when it comes to dependency management decision making and maintenance. +We hope you find success with dependency management through these and look forward to providing more information on this topic in the future. From 7aadb8c4884127015299301813c3bbbe28eeeee6 Mon Sep 17 00:00:00 2001 From: d33bs Date: Mon, 26 Feb 2024 12:56:00 -0700 Subject: [PATCH 09/16] update title --- _posts/2024-02-20-Lockfiles.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Lockfiles.md index 3bbd40ae53..cf14b48703 100644 --- a/_posts/2024-02-20-Lockfiles.md +++ b/_posts/2024-02-20-Lockfiles.md @@ -1,5 +1,5 @@ --- -title: "Lockfiles" +title: "Navigating Dependency Chaos with Lockfiles" author: dave-bunten tags: - software-education @@ -8,9 +8,10 @@ tags: - dependency-management - environment-management - software-ecosystems + - dependency-chaos --- -# Lockfiles +# Navigating Dependency Chaos with Lockfiles {% include blog-post-intro.html %} @@ -57,7 +58,7 @@ _Dependencies are often on their own unpredictable schedule outside of your proj Using existing software package dependencies helps conserve resources but comes with unique challenges related to unpredictability (such as when those dependencies are updated). -This unpredictability can sometimes result in what's colloquially called ["dependency hell"](https://en.wikipedia.org/wiki/Dependency_hell), where for example multiple external dependencies conflict with one another and are unable to be automatically resolved (among other issues). +This unpredictability can sometimes result in what's colloquially called ["dependency hell" or "dependency chaos"](https://en.wikipedia.org/wiki/Dependency_hell), where for example multiple external dependencies conflict with one another and are unable to be automatically resolved (among other issues). These challenges can be especially frustrating due to when they occur (often outside of our personal schedule awareness) and how long they can take to debug (finding fixes sometimes entails costly [trial-and-error](https://en.wikipedia.org/wiki/Trial_and_error)). It can feel like walking through a forest at night without a flashlight, constantly tripping over roots or running into stumps and branches! From 24c9323184f57fe772e409de5b000579637a0cff Mon Sep 17 00:00:00 2001 From: d33bs Date: Mon, 26 Feb 2024 12:56:42 -0700 Subject: [PATCH 10/16] rename post with filename --- ...d => 2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename _posts/{2024-02-20-Lockfiles.md => 2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md} (100%) diff --git a/_posts/2024-02-20-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md similarity index 100% rename from _posts/2024-02-20-Lockfiles.md rename to _posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md From 04e96da93e595364ac741868995fd03f78114401 Mon Sep 17 00:00:00 2001 From: Dave Bunten Date: Wed, 28 Feb 2024 12:13:26 -0700 Subject: [PATCH 11/16] Apply suggestions from code review Co-authored-by: Faisal Alquaddoomi --- .../2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index cf14b48703..8c0083dfd2 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -30,9 +30,9 @@ This article will cover package dependency management through special files gene {% include figure.html image="https://imgs.xkcd.com/comics/reinvent_the_wheel.png" caption="'Reinvent the Wheel' comic by Randall Munroe, XKCD." %} There are various advantages to using packaged dependencies in your projects. -Using existing work this way practices a collective ["don't repeat yourself [or ourselves]" (DRY)](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) to avoid [reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel). +Using existing work this way practices a collective ["don't repeat yourself [or ourselves]" (DRY)](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) among the global community of software developers to avoid [reinventing the wheel](https://en.wikipedia.org/wiki/Reinventing_the_wheel). Using dependencies allows us to make explicit decisions about the specific focus, or context, which the project will prioritize. -While it's oftentimes easy to include and use dependencies in a project they come with important risks that are important to consider. +While it's oftentimes easy to include and use dependencies in a project they come with risks that are important to consider. ## The dependency wilderness From fff4806ea2ee2cbfc2ef76a4d5d0e341ac09ec7f Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 28 Feb 2024 12:33:30 -0700 Subject: [PATCH 12/16] incorporate Vince's feedback Co-Authored-By: Vincent Rubinetti <8326331+vincerubinetti@users.noreply.github.com> --- ...20-Navigating-Dependency-Chaos-with-Lockfiles.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index 8c0083dfd2..9acc806f52 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -34,6 +34,14 @@ Using existing work this way practices a collective ["don't repeat yourself [or Using dependencies allows us to make explicit decisions about the specific focus, or context, which the project will prioritize. While it's oftentimes easy to include and use dependencies in a project they come with risks that are important to consider. +See below for a rough list of reasons why one might opt to use specific dependencies in a project: + +1. Solutions which entail a lot of [edge cases](https://en.wikipedia.org/wiki/Edge_case) (particularly error prone). +2. Solutions which need constant maintenance, i.e. a "frequently moving targets". +3. Solutions which require special domain knowledge or training to correctly implement. + +A common dependency which demonstrates these aspects are those which assist with datetimes, timezones, and time deltas. + ## The dependency wilderness @@ -181,7 +189,7 @@ _Version constraint specifications provide code-based descriptions for dependenc {:.center} Many aspects of dependency chaos arise from the fact that dependencies are updated at various times. -We often want to make certain we use the most up-to-date version of a dependency because those updates may come with performance, corrective, or other benefits. +We often want to make certain we use the most up-to-date version of a dependency because those updates may come with performance, corrective, security, or other benefits. To accomplish this we can use what are sometimes called dependency "version range constraints" or "compliant version specifications" to provide some flexibility in how packages are installed for our projects. These are often specific to the package management system and programming language being used. See the [Python Packaging Authority's Version Specifiers section](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) for an example of how these version constraints work. @@ -233,10 +241,11 @@ sequenceDiagram ``` Version constraint lockfiles provide one way to ensure reproducible behaviors within your projects. +Lockfiles are usually recommended to be included in source control, so one always has a complete snapshot (short of the literal full source code of the dependencies) of the project's last known working configuration. Lockfiles usually have the following characteristics (this varies by programming language and dependency type): -- Lockfiles usually capture data about existing available dependencies which match a provided version constraint specification as a single file which can be added to source control. +- Lockfiles capture data about existing available dependencies which match a provided version constraint specification as a single file which can be added to source control. - Lockfiles are referenced when available to help create reproducible installations of dependencies. - Lockfiles are often automatically created or changed by a package or environment management tool of some kind. From 3e91d8b0682bbe304e8e68a0c6350403c25590f7 Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 28 Feb 2024 12:39:09 -0700 Subject: [PATCH 13/16] move mindmap to mermaid10 render --- ...4-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index 9acc806f52..01d9ef2d3e 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -90,7 +90,11 @@ When or if this happens it can be important to understand dependencies as a __sc For example, to accomplish a small project it may not be wise to create a brand new programming language (towards the extreme of overwhelming invention). On the other hand, if we depended upon all existing work within a certain context the solution may not be specialized, efficient, or resourceful enough to meet the goals within a reasonable amount of time. -```mermaid! + +
 mindmap
   root((Project))
     Data storage
@@ -102,8 +106,7 @@ mindmap
     Integration
       Solution A
       Platform B
-
-```
+
_Dependency awareness and opportunity can be grouped into concerns and documented as part of a literature review (seen here as a [mind map](https://en.wikipedia.org/wiki/Mind_map))._ {:.center} From 25652ebfb1a6102138396b8ca0ab7398400cc56f Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 28 Feb 2024 13:00:44 -0700 Subject: [PATCH 14/16] add semver details Co-Authored-By: Faisal Alquaddoomi --- ...2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index 01d9ef2d3e..a4689edc98 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -197,6 +197,12 @@ To accomplish this we can use what are sometimes called dependency "version rang These are often specific to the package management system and programming language being used. See the [Python Packaging Authority's Version Specifiers section](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) for an example of how these version constraints work. +Many version specification constraints build upon ideas from [semantic versioning (SemVer)](https://en.wikipedia.org/wiki/Software_versioning#Semantic_versioning). +Generally, SemVer uses a dotted three number syntax which includes a major, minor, and patch version separated by periods. +For example, a SemVer `1.2.3` represents major version 1, minor version 2, patch 3. +Developers may use of this type of specification to help differentiate the various releases of their software and help build user confidence about expected operations. +See the [Semantic Versioning specification at https://semver.org/](https://semver.org/) for more information about how SemVer works. + ## Version constraints can still be chaotic ```mermaid! From 36ab01ab9a60d67f3b5ffb0a58d9e6f7313f52df Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 28 Feb 2024 13:04:57 -0700 Subject: [PATCH 15/16] add note about dependency resolutions Co-Authored-By: Faisal Alquaddoomi --- _posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index a4689edc98..334c14a449 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -194,6 +194,7 @@ _Version constraint specifications provide code-based descriptions for dependenc Many aspects of dependency chaos arise from the fact that dependencies are updated at various times. We often want to make certain we use the most up-to-date version of a dependency because those updates may come with performance, corrective, security, or other benefits. To accomplish this we can use what are sometimes called dependency "version range constraints" or "compliant version specifications" to provide some flexibility in how packages are installed for our projects. +Version ranges are usually preferred to help keep software projects updated and also allow for flexible dependency resolutions (for example, when a single dependency is required by multiple other dependencies). These are often specific to the package management system and programming language being used. See the [Python Packaging Authority's Version Specifiers section](https://packaging.python.org/en/latest/specifications/version-specifiers/#id5) for an example of how these version constraints work. From ab5b187a220426aef122253e561c784e5be2274e Mon Sep 17 00:00:00 2001 From: d33bs Date: Wed, 28 Feb 2024 13:14:06 -0700 Subject: [PATCH 16/16] add greater specificity to lockfile description Co-Authored-By: Faisal Alquaddoomi --- .../2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md index 334c14a449..401779efa5 100644 --- a/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md +++ b/_posts/2024-02-20-Navigating-Dependency-Chaos-with-Lockfiles.md @@ -255,9 +255,11 @@ Lockfiles are usually recommended to be included in source control, so one alway Lockfiles usually have the following characteristics (this varies by programming language and dependency type): -- Lockfiles capture data about existing available dependencies which match a provided version constraint specification as a single file which can be added to source control. +- Lockfiles capture data about existing available and installable dependencies which match a provided version constraint specification as a single file which can be added to source control. - Lockfiles are referenced when available to help create reproducible installations of dependencies. - Lockfiles are often automatically created or changed by a package or environment management tool of some kind. +- Lockfiles focus on reproducibility of dependency installations and don't enable dependency resolution on their own (this is instead a part of version range specification and package management tools). +- Lockfiles are used by developers, automated procedures (as with [CI/CD](https://en.wikipedia.org/wiki/CI/CD) procedures), [production deployment environments](https://en.wikipedia.org/wiki/Deployment_environment#Production), and elsewhere to help ensure reproducibility. See the above modified timeline for `Developer B` and `Developer D` to better understand how their project will benefit from a shared lockfile and reproducible dependency installations.