Hmm... where to start? by nickumia

Disclaimer: The thoughts and ideas conveyed in this blog are those of the author and do not reflect those of GSA, REI, Data.gov or any other organization that the author is part of.

Part 1 - What is this?

When I started, a short six months ago (at time of writing), I thought a month was enough time to start to document my experiences that might help another soul in their data.gov assimilation and their professional work career progression as well. Well... as you can tell from the mostly empty early versions of this page, the task turned out to be much more daunting than originally thought. While there's always much more to learn and future me might look back and think this as inadequate, the point is to capture the journey because future me was old me at one point and was stuck in much of the same startup issues that all projects do have.

[A faint scream: "Enough with the gibberish old-timer! Just tell me what I need to know"]

A good place to start is knowing what is "data.gov". Data.gov is a community interested in appropriate integrity and availability of federal data in the US. It is a team of developers, project managers, project coordinators and humans (I fall into the humans category 😀) dedicated to the creation, maintenance and evolution of the data.gov applications/platforms/principles.

Instead of claiming to answer questions, I will mostly describe different areas, give a brief outline and then leave it up to you to reach out to ask questions about this information 😆

Efforts around the time when I started:

• O&M Work
• Support Work
• Ckanext Upgrade Work
• Cloud.gov/SSB Work

Part 2: Technology Stack

Organization of data.gov:

• Main Applications
  • 🗻 catalog.data.gov (python)
  • 📚 inventory.data.gov (python)
  • 📽️ dashboard.data.gov (php -- soon to be deprecated)
• Static sites
  • 📦 www.data.gov
  • 📖 resources.data.gov
• Extensions
  • 🌊 Current inventory
  • Note: Extensions may come and go 💨
• Supplementary Service Broker (SSB)
  • 👾 Broker Coordination
  • 🖋️ EKS Brokerpak
  • 📇 Solr Brokerpak
  • 📫 SMTP Brokerpak

Supporting technology:

• 🐍 Python
• 📘 PostgreSQL/MySQL
• 🔍 Apache Solr
  • SolrCloud, Solr-Operator
• 🔖 Redis
• ☁️ AWS
  • RDS, Route53, EC2, EKS, LB, S3, VPC, ACM, SSM, CloudWatch, CloudFront, IAM, KMS, EBS
• ☁️ Terraform
• ☁️ CloudFoundry/Cloud.gov
• 🌌 Kubernetes
• 💻 Bash
• 📅 Github Actions
• 🎛️ CKAN
• ☁️ Federalist
• 👤 Front-end
  • React, Jekyll, Yarn, HTML/CSS/JS (Will probably re-visit and add more)

Each Application that data.gov supports has a very large code base, data pipeline, deployment architecture. There is a lot of architecture that I'm also not as familiar with because it predates my existence on the team. I can't hope to document the entire dependency chain and development cycle of everything because it's continuously evolving and the breadth and depth is truly humbling. What I'll choose to talk about instead is the abstract design of our systems and the data.gov principles that would keep me active on the project even if my full-time job were to change.

Part 3 - Mission

There is a very intricate relationship between the stakeholders and users of Data.gov. General groups:

• 🏗️ Data.gov leadership
• 🧑‍💻 Data.gov developers/support team
• 🇺🇸 General Public
• 🏬 City/State/Federal Agencies
• ❔ There's definitely more, I will update when I think about the rest of them.

As the Data.gov team, we support all of these groups who use various of our systems for different purposes.