Tag: programming

From Cron Chaos to Centralized Calm

I’ve spent countless evenings hunting down why one of my dozen automated processes failed—digging through server A’s logs, then B’s, then wondering if timezone math on machine C silently swallowed a reminder. If that sounds familiar, check out the original article now: Replacing cron jobs with a centralized task scheduler. It flipped my whole mindset.

Instead of treating each cron script as a black box, the author models every future action as a row in one ScheduledTasks table. Think of it: every job you’d ever schedule lives in a single, queryable place. Because each task records when it’s due, its priority, retries left, and even an expiration window, you immediately know:

  • What went wrong? Was the row created? Did the status flip to “EXECUTING”?
  • When did it fail? Timestamps are part of the schema.
  • Can I retry it? Built-in retry logic based on expectedExecutionTimeInMinutes handles stuck tasks automatically.

And because the table uses deterministic IDs for editable tasks—upserting instead of piling on duplicates—your reminder for “Event X at 3 PM” never spawns two competing jobs if the event gets rescheduled. It just updates the one, single record.

Applying This to Your Own Stack

  1. Model work as data: Start by designing a simple table (or collection) that captures every scheduled action: due time, status, payload, retries, and expiration.
  2. Use one poller, many workers: Replace your multiple cron scripts with a single poller that enqueues due tasks into your favorite queue (SQS, RabbitMQ, etc.), then let specialized consumers pick up and execute.
  3. Unify logging & monitoring: With everything funneled through one scheduler, you gain a centralized dashboard—no more jumping across machines to trace a failure.

By embracing this pattern, I went from juggling eight Node scripts across three servers to maintaining one tiny service. When something breaks now, I head straight to the ScheduledTasks table, filter by status or timestamp, and—boom—I’ve got my starting point. No more haystack.

Why Project Estimates Fail: Lessons from a Systems Thinker’s Lens

Inspired by The work is never just “the work” by Dave Stewart

“Even a detailed estimate of ‘the work’ can miss the dark matter that makes up the majority of a project’s real effort.”

When it comes to project management—especially in software and creative work—most of us have lived through the agony of missed deadlines and ballooning timelines. It’s tempting to blame bad luck, moving goalposts, or simple optimism. But as Dave Stewart reveals, there’s a more systemic, and ultimately more instructive, explanation.

Let’s step back and see the big picture—the “systems view”—and discover why underestimation isn’t just a personal failing, but a deeply-rooted feature of how complex projects function.

The Invisible System: Why “The Work” is Just the Tip of the Iceberg

Stewart’s article provides a hard-won confession: after a year-long project went wildly off course, he realized the effort spent on “the work” (i.e., coding, designing, building) was just a fraction of the total investment. The majority was spent on what he calls the “work around the work”—from setup and research, to iteration, firefighting, and post-launch support.

From a systems thinker’s standpoint, this is a textbook example of the planning fallacy—a cognitive bias where we underestimate complexity by focusing on visible tasks and ignoring the web of dependencies and uncertainty that surrounds every project.

Mapping the Project Ecosystem

What Stewart does beautifully is name and map the categories of hidden labor:

  • Preparation: Infrastructure, setup, initial research
  • Acquisition: Scoping, pitching, client meetings
  • Iteration: Debugging, refactoring, ongoing improvements
  • Support: Deployment, updates, ongoing fixes
  • The Unexpected: Surprises, scope creep, disasters

By visualizing the project as an ecosystem—where “the work” is only one node among many—he demonstrates a key principle of systems thinking: emergent complexity. Each category adds not just linear effort, but amplifies feedback loops (delays, misunderstandings, unexpected roadblocks) that make estimation so hazardous.

Patterns and Implications

A systems lens reveals several recurring patterns:

  • Invisible Feedback Loops: Tasks outside “the work” (meetings, reviews, firefighting) generate new work, shifting priorities and resource allocation—often without being tracked or acknowledged.
  • Nonlinear Impact: Small “invisible” tasks, left unaccounted for, aggregate into substantial overruns. Like dark matter, their presence is felt even if they remain unseen.
  • Optimism Bias Is Systemic: Most teams and individuals underestimate not out of ignorance, but because our brains and organizational structures reward “happy path” thinking.
  • Every Project Is a Living System: Changing one part (e.g., a delayed client feedback loop) can ripple through the whole system, derailing even the most detailed plan.

Designing for Reality, Not Idealism

The key takeaway for systems thinkers is awareness and intentional design:

  1. Model the Whole System: During estimation, explicitly map out all “nodes”—not just core deliverables but supporting, enabling, and maintaining tasks.
  2. Quantify Uncertainty: Use multipliers, ranges, and postmortems to factor in the “dark matter” of invisible work.
  3. Surface Assumptions: Name and question the implicit beliefs behind every estimate (e.g., “the client will provide feedback within 24 hours”—will they, really?).
  4. Iterate the System: Treat your estimation process itself as a system to be improved, not a static formula.

Actionable Insights for the Systems Thinker

  • Create a “Work Ecosystem Map” for each new project, labeling categories like preparation, acquisition, iteration, support, and surprises.
  • Hold Team Retrospectives focused not just on deliverables but on the “meta-work” that surrounded them—what did we miss? What new loops emerged?
  • Educate Stakeholders: Share frameworks like Stewart’s to align expectations and build organizational literacy around hidden work.
  • Measure, Don’t Assume: Use real project data to tune your own multipliers and assumptions over time.

Final Thought

Projects are living systems, not checklists. By recognizing the invisible forces at play, we empower ourselves (and our teams) to design more resilient processes, set realistic expectations, and—just maybe—find more satisfaction in the work itself.

“The work is never just the work. It’s everything else—unseen, unsung, but absolutely essential.”

Further Reading:
Dive into the original article: The work is never just “the work”
Reflect on the planning fallacy: Wikipedia – Planning Fallacy
Explore systems thinking: Donella Meadows – Thinking in Systems

Amplified, Not Replaced: A Veteran Engineer’s Take on Coding’s Uncertain Future

As someone who’s weathered tech cycles, scaled legacy systems, and mentored more than a few generations of engineers, I find myself returning to a recent essay by Jonathan Hoyt: “The Uncertain Future of Coding Careers and Why I’m Still Hopeful”. Hoyt’s piece feels timely—addressing, with candor and humility, the growing sense of anxiety many in our profession feel as AI rapidly transforms the software landscape.

Hoyt’s narrative opens with a conversation familiar to any experienced lead or architect: a junior developer questioning whether they’ve chosen a doomed career. It’s a concern that echoes through countless engineering Slack channels in the wake of high-profile tech layoffs and the visible rise of AI tools like GitHub Copilot. Even for those of us long in the tooth, Hoyt admits, it’s tempting to wonder if we’re on the verge of obsolescence.

But what makes Hoyt’s perspective refreshing—especially for those further along in their careers—is the pivot from fear to agency. He reframes AI, not as an existential threat, but as an amplifier of human ingenuity. For senior engineers and system architects, this means our most valuable skills are not rote implementation or brute-force debugging, but context-building, system design, and the ability to ask the right questions. As Hoyt puts it, the real work becomes guiding the machines, curating and contextualizing knowledge, and ultimately shepherding both code and colleagues into new creative territory.

The essay’s most resonant point for experienced professionals is the call to continuous reinvention. Hoyt writes about treating obsolescence as a kind of internal challenge—constantly working to automate yourself out of your current role, so you’re always prepared to step into the next. For architects, this means doubling down on mentorship, sharing knowledge freely, and contributing to the collective “shared brain” of the industry—be it through open source, internal documentation, or just helping the next engineer up the ladder.

Hoyt’s post doesn’t sugarcoat the uncertainty ahead. The routine entry points into the field are shifting, and not everyone will find the transition easy. Yet, he argues, the need for creative, context-aware technologists will only grow. If AI takes on the repetitive work, our opportunity is to spend more time on invention, strategy, and the high-leverage decisions that shape not just projects, but organizations.

If you’ve spent your career worrying that you might be automated out of relevance, Hoyt’s essay offers both a challenge and a comfort. It’s a reminder that the future of programming isn’t about competing with machines, but learning to be amplified by them—and ensuring we’re always building, learning, and sharing in ways that move the whole field forward.

For anyone in a senior engineering or system architecture role, Jonathan Hoyt’s original piece is essential reading. It doesn’t just address the fears of those just starting out; it offers a vision of hope and practical action for those of us guiding teams—and the next generation—through the shifting sands of technology.