Ideas on engineering, data platforms, software, AI and complex systems.https://15x1976.neocities.org/enhttps://15x1976.neocities.org/en/writing/ai-augmented-teams-turning-individual-knowledge-into-collective-capability/https://15x1976.neocities.org/en/writing/ai-augmented-teams-turning-individual-knowledge-into-collective-capability/Giving every team member an AI assistant can make individuals faster while fragmenting the team. The real challenge is turning that additional capability into shared context, decisions, and knowledge.Thu, 24 Sep 2026 00:00:00 GMTAIEngineering CultureArchitecturehttps://15x1976.neocities.org/en/writing/ai-native-engineering-from-using-ai-to-designing-with-it/https://15x1976.neocities.org/en/writing/ai-native-engineering-from-using-ai-to-designing-with-it/AI-Native work is not about asking a model for more code. It means redesigning the engineering process so analysis, decisions, artifacts, and construction can evolve with AI without giving up human judgment.Thu, 10 Sep 2026 00:00:00 GMTAISoftware EngineeringEngineering Culturehttps://15x1976.neocities.org/en/writing/snowflake-and-databricks-where-they-converge-and-which-differences-matter-when-designing-a-platform/https://15x1976.neocities.org/en/writing/snowflake-and-databricks-where-they-converge-and-which-differences-matter-when-designing-a-platform/Snowflake and Databricks converge across many capabilities, but differences in execution, storage, operations, and engineering experience still matter when designing a Data Platform.Thu, 27 Aug 2026 00:00:00 GMTSnowflakeDatabricksData Architecturehttps://15x1976.neocities.org/en/writing/from-databricks-to-snowflake-preserving-deployment-discipline-with-cli-and-dcm-projects/https://15x1976.neocities.org/en/writing/from-databricks-to-snowflake-preserving-deployment-discipline-with-cli-and-dcm-projects/The deployment discipline learned with Databricks Asset Bundles can transfer to Snowflake without forcing equivalence, using CLI and DCM Projects to version and deploy resources repeatably.Tue, 11 Aug 2026 00:00:00 GMTSnowflakeDatabricksSoftware Engineeringhttps://15x1976.neocities.org/en/writing/snowpark-using-python-on-snowflake-while-understanding-where-the-work-runs/https://15x1976.neocities.org/en/writing/snowpark-using-python-on-snowflake-while-understanding-where-the-work-runs/Snowpark lets us express transformations in Python while keeping much of the work inside Snowflake; understanding where each piece runs prevents unnecessary movement of data and resources.Fri, 24 Jul 2026 00:00:00 GMTSnowflakeData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/from-databricks-to-snowflake-what-knowledge-to-transfer-and-what-assumptions-to-revisit/https://15x1976.neocities.org/en/writing/from-databricks-to-snowflake-what-knowledge-to-transfer-and-what-assumptions-to-revisit/Moving from Databricks to Snowflake lets us reuse substantial data engineering judgment, but it requires revisiting assumptions about execution, storage, incremental processing, and platform management.Fri, 10 Jul 2026 00:00:00 GMTSnowflakeDatabricksData Architecturehttps://15x1976.neocities.org/en/writing/mastering-databricks-the-decisions-behind-its-advanced-capabilities/https://15x1976.neocities.org/en/writing/mastering-databricks-the-decisions-behind-its-advanced-capabilities/Mastering Databricks is not about knowing every feature, but understanding what problem each capability solves, what it delegates to the platform, and what responsibility remains in the design.Fri, 26 Jun 2026 00:00:00 GMTDatabricksData ArchitectureData Engineeringhttps://15x1976.neocities.org/en/writing/databricks-capabilities-worth-evaluating-before-building-your-own-solution/https://15x1976.neocities.org/en/writing/databricks-capabilities-worth-evaluating-before-building-your-own-solution/Before building custom infrastructure, it is worth reviewing which responsibilities Databricks can already assume and which controls our architecture genuinely needs to retain.Thu, 11 Jun 2026 00:00:00 GMTDatabricksData ArchitectureSoftware Engineeringhttps://15x1976.neocities.org/en/writing/bronze-without-an-extra-copy-what-guarantees-we-need-before-pointing-to-existing-data/https://15x1976.neocities.org/en/writing/bronze-without-an-extra-copy-what-guarantees-we-need-before-pointing-to-existing-data/Avoiding an extra physical copy in Bronze can save work, but it is safe only when existing data provides enough stability, identity, retention, and recovery guarantees.Fri, 22 May 2026 00:00:00 GMTData ArchitectureData EngineeringDatabrickshttps://15x1976.neocities.org/en/writing/serverless-in-databricks-delegating-compute-without-losing-operational-control/https://15x1976.neocities.org/en/writing/serverless-in-databricks-delegating-compute-without-losing-operational-control/Serverless delegates infrastructure decisions to Databricks, but reliable operations still require boundaries, observability, understandable costs, and clear workload contracts.Fri, 08 May 2026 00:00:00 GMTDatabricksCloudData Engineeringhttps://15x1976.neocities.org/en/writing/photon-in-databricks-how-the-vectorized-engine-works-and-when-it-improves-pipeline-cost/https://15x1976.neocities.org/en/writing/photon-in-databricks-how-the-vectorized-engine-works-and-when-it-improves-pipeline-cost/Photon accelerates certain workloads through vectorized execution, but faster performance reduces cost only when we understand what work it accelerates and how compute is billed.Thu, 23 Apr 2026 00:00:00 GMTDatabricksData EngineeringCloudhttps://15x1976.neocities.org/en/writing/liquid-clustering-in-databricks-organizing-data-to-read-less-and-evolve-the-design/https://15x1976.neocities.org/en/writing/liquid-clustering-in-databricks-organizing-data-to-read-less-and-evolve-the-design/Liquid Clustering changes how we think about the physical organization of Delta tables: it aims to reduce data read while allowing the design to evolve beyond rigid partitions.Thu, 09 Apr 2026 00:00:00 GMTDatabricksData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/advanced-engineering-in-databricks-the-judgment-behind-a-reliable-platform/https://15x1976.neocities.org/en/writing/advanced-engineering-in-databricks-the-judgment-behind-a-reliable-platform/Advanced engineering in Databricks depends less on accumulating features and more on applying judgment to reliability, operations, cost, and the boundaries of each platform capability.Tue, 31 Mar 2026 00:00:00 GMTDatabricksData ArchitectureData Engineeringhttps://15x1976.neocities.org/en/writing/refactoring-pipelines-in-databricks-improving-performance-and-operations-without-losing-data-meaning/https://15x1976.neocities.org/en/writing/refactoring-pipelines-in-databricks-improving-performance-and-operations-without-losing-data-meaning/Refactoring a pipeline is not only about making it faster: performance and operations must improve without changing contracts, grain, or data meaning.Fri, 20 Mar 2026 00:00:00 GMTDatabricksData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/from-delta-live-tables-to-lakeflow-when-a-declarative-pipeline-makes-sense/https://15x1976.neocities.org/en/writing/from-delta-live-tables-to-lakeflow-when-a-declarative-pipeline-makes-sense/Delta Live Tables evolved into Lakeflow Declarative Pipelines. The important decision is not the name, but when to delegate dependencies, incremental processing, and operations to a declarative framework.Tue, 10 Mar 2026 00:00:00 GMTDatabricksData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/from-the-broker-to-the-lakehouse-what-changes-with-lakeflow-connect/https://15x1976.neocities.org/en/writing/from-the-broker-to-the-lakehouse-what-changes-with-lakeflow-connect/Lakeflow Connect can simplify ingestion from brokers, but it does not remove decisions about delivery, recovery, publication, and data meaning inside the Lakehouse.Fri, 27 Feb 2026 00:00:00 GMTDatabricksData ArchitectureData Engineeringhttps://15x1976.neocities.org/en/writing/the-cost-of-ingesting-data-in-databricks-discover-process-wait/https://15x1976.neocities.org/en/writing/the-cost-of-ingesting-data-in-databricks-discover-process-wait/Ingestion cost is not only compute: it also includes file discovery, idle capacity, and repeated work. Optimization requires looking at the entire path.Tue, 17 Feb 2026 00:00:00 GMTDatabricksData EngineeringCloudhttps://15x1976.neocities.org/en/writing/asynchrony-in-databricks-from-file-arrival-to-trusted-data/https://15x1976.neocities.org/en/writing/asynchrony-in-databricks-from-file-arrival-to-trusted-data/A pipeline can finish green and still publish incomplete data. Designing asynchrony in Databricks means separating arrival, processing, completeness, and trusted publication.Thu, 05 Feb 2026 00:00:00 GMTDatabricksData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/event-brokers-queues-streams-and-architectural-guarantees/https://15x1976.neocities.org/en/writing/event-brokers-queues-streams-and-architectural-guarantees/Queues and streams solve different problems; sound Event Broker design requires understanding delivery, ordering, replay, acknowledgments, and idempotency before choosing technology.Mon, 26 Jan 2026 00:00:00 GMTArchitectureData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/asynchrony-in-elt-when-data-is-actually-ready/https://15x1976.neocities.org/en/writing/asynchrony-in-elt-when-data-is-actually-ready/Asynchrony can reduce waiting in ELT, but it also requires explicit states, dependencies, and publication conditions to know when data is actually ready.Thu, 08 Jan 2026 00:00:00 GMTData EngineeringArchitectureSoftware Engineeringhttps://15x1976.neocities.org/en/writing/the-champion-is-not-handed-over/https://15x1976.neocities.org/en/writing/the-champion-is-not-handed-over/The blind handoff between Data Science and Data Engineering is replaced by a frozen Champion, contract, joint code review, refactoring, and shared validation.Mon, 22 Dec 2025 00:00:00 GMTAIData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/when-every-ticket-solves-the-same-problem-again/https://15x1976.neocities.org/en/writing/when-every-ticket-solves-the-same-problem-again/Industrializing a data factory means turning repeated decisions into reusable capabilities and reserving engineering for what is genuinely new.Tue, 09 Dec 2025 00:00:00 GMTData ArchitectureDatabricksSoftware Engineeringhttps://15x1976.neocities.org/en/writing/when-every-new-table-means-a-new-pipeline/https://15x1976.neocities.org/en/writing/when-every-new-table-means-a-new-pipeline/An SAP ingestion platform grew by creating a pipeline for every new table. The alternative was changing the unit of growth from tables and pipelines to reusable extraction patterns.Tue, 25 Nov 2025 00:00:00 GMTData ArchitectureData Engineeringhttps://15x1976.neocities.org/en/writing/what-the-hell-is-data-mesh-and-why-is-everyone-talking-about-it/https://15x1976.neocities.org/en/writing/what-the-hell-is-data-mesh-and-why-is-everyone-talking-about-it/Data Mesh is not “putting data into domains.” It redistributes responsibility through domain ownership, data products, self-service and federated governance.Tue, 11 Nov 2025 00:00:00 GMTData ArchitectureEngineering Culturehttps://15x1976.neocities.org/en/writing/everything-is-green-and-the-model-is-wrong/https://15x1976.neocities.org/en/writing/everything-is-green-and-the-model-is-wrong/A green job proves the code finished. It does not prove the data is correct or the model is still useful. ML observability has multiple layers.Wed, 29 Oct 2025 00:00:00 GMTAIData EngineeringEngineering Culturehttps://15x1976.neocities.org/en/writing/training-on-the-past-without-looking-into-the-future/https://15x1976.neocities.org/en/writing/training-on-the-past-without-looking-into-the-future/Training on old records is not enough. We need to reconstruct what information would actually have been available at the moment each prediction was made.Fri, 17 Oct 2025 00:00:00 GMTAIData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/could-you-reproduce-the-model-you-put-into-production-six-months-ago/https://15x1976.neocities.org/en/writing/could-you-reproduce-the-model-you-put-into-production-six-months-ago/Saving the model artifact is not enough to reproduce an experiment. Data, features, parameters, configuration, dependencies and code all participate in the result.Mon, 06 Oct 2025 00:00:00 GMTAIData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/it-works-in-my-notebook-from-experiment-to-product/https://15x1976.neocities.org/en/writing/it-works-in-my-notebook-from-experiment-to-product/A model working in a notebook proves a hypothesis. A product must preserve that meaning while adding repeatability, recovery and operation.Tue, 23 Sep 2025 00:00:00 GMTSoftware EngineeringData EngineeringAIhttps://15x1976.neocities.org/en/writing/from-notebooks-to-software-professionalizing-databricks-with-asset-bundles/https://15x1976.neocities.org/en/writing/from-notebooks-to-software-professionalizing-databricks-with-asset-bundles/As a data platform grows, jobs, configuration and deployment deserve software discipline too. Asset Bundles are one mechanism for that discipline, not the thesis itself.Tue, 09 Sep 2025 00:00:00 GMTDatabricksSoftware EngineeringData Engineeringhttps://15x1976.neocities.org/en/writing/adf-vs-databricks-workflows-the-false-orchestrator-war/https://15x1976.neocities.org/en/writing/adf-vs-databricks-workflows-the-false-orchestrator-war/ADF and Databricks Workflows overlap, but choosing a universal winner is less useful than deciding where each responsibility starts and ends.Wed, 27 Aug 2025 00:00:00 GMTCloudDatabricksData Engineeringhttps://15x1976.neocities.org/en/writing/show-me-how-you-tag-your-workloads-and-i-will-show-you-where-you-lose-money/https://15x1976.neocities.org/en/writing/show-me-how-you-tag-your-workloads-and-i-will-show-you-where-you-lose-money/Cost optimization without attribution is guesswork. Before reducing spend, connect compute to workloads, owners and value.Thu, 14 Aug 2025 00:00:00 GMTCloudData EngineeringEngineering Culturehttps://15x1976.neocities.org/en/writing/your-platform-does-not-need-more-alerts-it-needs-better-alerts/https://15x1976.neocities.org/en/writing/your-platform-does-not-need-more-alerts-it-needs-better-alerts/Hundreds of alerts can make a platform less observable than twenty good ones. The goal is not to detect events; it is to identify what requires action and why.Mon, 04 Aug 2025 00:00:00 GMTEngineering CultureData EngineeringCloudhttps://15x1976.neocities.org/en/writing/data-governance-does-not-mean-setting-permissions/https://15x1976.neocities.org/en/writing/data-governance-does-not-mean-setting-permissions/Access control is security. Governance goes further: what does the data mean, who owns that meaning, and under what conditions can it be trusted?Mon, 21 Jul 2025 00:00:00 GMTData ArchitectureEngineering CultureCloudhttps://15x1976.neocities.org/en/writing/when-lineage-is-missing-everyone-becomes-a-detective/https://15x1976.neocities.org/en/writing/when-lineage-is-missing-everyone-becomes-a-detective/Without lineage, every important change becomes detective work: who produces this table, who consumes it, and what breaks if we change it?Mon, 07 Jul 2025 00:00:00 GMTData ArchitectureData EngineeringEngineering Culturehttps://15x1976.neocities.org/en/writing/credentials-in-code-are-technical-debt-with-an-expiration-date/https://15x1976.neocities.org/en/writing/credentials-in-code-are-technical-debt-with-an-expiration-date/A hardcoded password solves a connection in seconds and can create a problem that lasts for years. Maturity means reducing how much secret material the application needs to know.Wed, 25 Jun 2025 00:00:00 GMTCloudSoftware EngineeringEngineering Culturehttps://15x1976.neocities.org/en/writing/your-sources-will-change-design-for-it/https://15x1976.neocities.org/en/writing/your-sources-will-change-design-for-it/Sources change even when our pipelines do not. Good boundaries make schema, data and semantic drift observable before they become silently wrong data.Tue, 10 Jun 2025 00:00:00 GMTData EngineeringArchitecturehttps://15x1976.neocities.org/en/writing/hard-rest-apis-surviving-pagination-tokens-and-retries/https://15x1976.neocities.org/en/writing/hard-rest-apis-surviving-pagination-tokens-and-retries/An API is not reliable because one request returned 200. Production engineering begins when pagination, token expiry, rate limits and partial recovery become part of the contract.Thu, 29 May 2025 00:00:00 GMTSoftware EngineeringData Engineeringhttps://15x1976.neocities.org/en/writing/how-one-query-can-ruin-everyone-s-morning/https://15x1976.neocities.org/en/writing/how-one-query-can-ruin-everyone-s-morning/A hard-to-maintain query is often hard to optimize for the same reason: nobody can explain where the work is happening. Breaking the problem apart restores visibility before it adds machinery.Fri, 16 May 2025 00:00:00 GMTData EngineeringSoftware Engineeringhttps://15x1976.neocities.org/en/writing/before-scaling-the-cluster-reduce-the-work/https://15x1976.neocities.org/en/writing/before-scaling-the-cluster-reduce-the-work/More compute can accelerate a poor strategy, but rarely turns it into a good one. Before scaling the cluster, remove work that never needed to happen.Mon, 05 May 2025 00:00:00 GMTDatabricksData Engineeringhttps://15x1976.neocities.org/en/writing/small-files-and-partitioning-when-physical-design-sends-the-bill/https://15x1976.neocities.org/en/writing/small-files-and-partitioning-when-physical-design-sends-the-bill/A table can contain the right data and still perform badly. How those bytes are physically distributed can matter as much as the transformation that produced them.Thu, 24 Apr 2025 00:00:00 GMTDatabricksData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/idempotency-execution-is-part-of-the-design/https://15x1976.neocities.org/en/writing/idempotency-execution-is-part-of-the-design/A pipeline that works once is not necessarily finished. The real design shows up when the same unit of work must be executed again without corrupting state.Tue, 08 Apr 2025 00:00:00 GMTData EngineeringSoftware EngineeringArchitecturehttps://15x1976.neocities.org/en/writing/time-travel-nobody-misses-it-until-five-minutes-after-the-wrong-overwrite/https://15x1976.neocities.org/en/writing/time-travel-nobody-misses-it-until-five-minutes-after-the-wrong-overwrite/Time Travel feels optional until the first bad overwrite. Then it becomes a powerful recovery tool—while still not being an infinite backup.Thu, 27 Mar 2025 00:00:00 GMTDatabricksData Engineeringhttps://15x1976.neocities.org/en/writing/streaming-does-not-mean-real-time/https://15x1976.neocities.org/en/writing/streaming-does-not-mean-real-time/Streaming describes a processing model; real time describes a latency expectation. Confusing the two can buy complexity without buying business value.Fri, 14 Mar 2025 00:00:00 GMTData EngineeringArchitecturehttps://15x1976.neocities.org/en/writing/cdc-stop-reading-what-you-know-did-not-change/https://15x1976.neocities.org/en/writing/cdc-stop-reading-what-you-know-did-not-change/Reading everything again can be the right strategy. It becomes questionable when we know what changed and still pay to ignore that information.Mon, 03 Mar 2025 00:00:00 GMTData EngineeringData Architecturehttps://15x1976.neocities.org/en/writing/the-data-warehouse-mistakes-we-keep-repeating-in-the-data-lake/https://15x1976.neocities.org/en/writing/the-data-warehouse-mistakes-we-keep-repeating-in-the-data-lake/Moving from a Data Warehouse to a Lake does not automatically remove old constraints. If we copy every old decision without revisiting its purpose, we only relocate them.Tue, 18 Feb 2025 00:00:00 GMTData ArchitectureData Engineeringhttps://15x1976.neocities.org/en/writing/the-medallion-model-is-not-a-religion/https://15x1976.neocities.org/en/writing/the-medallion-model-is-not-a-religion/Bronze, Silver and Gold are useful when each layer owns a responsibility. When data is copied only to satisfy the diagram, the pattern stops helping us think.Tue, 04 Feb 2025 00:00:00 GMTData ArchitectureData EngineeringDatabrickshttps://15x1976.neocities.org/en/writing/architecture-does-not-end-in-powerpoint/https://15x1976.neocities.org/en/writing/architecture-does-not-end-in-powerpoint/Architecture does not end when the diagram looks convincing. It earns its value through decisions, proofs, standards, implementation and operation.Thu, 23 Jan 2025 00:00:00 GMTArchitectureEngineering Culturehttps://15x1976.neocities.org/en/writing/the-notebook-is-not-the-application/https://15x1976.neocities.org/en/writing/the-notebook-is-not-the-application/A notebook can be an excellent interface without also owning configuration, secrets, business logic, orchestration and recovery.Mon, 06 Jan 2025 00:00:00 GMTSoftware EngineeringData EngineeringDatabricks