Machine Learning Times
Machine Learning Times
AI Success Depends On How You Choose This One Number
 Originally published in Forbes, March 25, 2024. To do...
Elon Musk Predicts Artificial General Intelligence In 2 Years. Here’s Why That’s Hype
 Originally published in Forbes, April 10, 2024 When OpenAI’s...
Survey: Machine Learning Projects Still Routinely Fail to Deploy
 Originally published in KDnuggets. Eric Siegel highlights the chronic...
Three Best Practices for Unilever’s Global Analytics Initiatives
    This article from Morgan Vawter, Global Vice...

4 months ago
2024 Data Engineering Trends

Originally published in Kestra, Jan 24, 2024.

Doing More with Less

The tech industry in 2024 is under pressure to optimize resources. Technology and data leaders are asked to integrate more data to support new AI-driven features while simultaneously being forced to reduce costs and headcount. Judging by the recent layoffs at a.o. Google, Amazon, Meta, Twitch, Spotify, and Discord, even the largest tech companies are not immune to this trend toward increased efficiency.

The Impact of AI on Layoffs vs. Economic Factors

The growing capabilities of LLMs are reshaping the job market, and the data space is no exception. While it’s difficult to estimate to what extent AI progress has contributed to the growing waves of tech layoffs, many companies are cutting costs in their established lines of business and reallocating that budget toward AI development. Dropbox reduced its headcount by 16% last year and reallocated those resources toward hiring AI specialists in order to “stay competitive”.

Economic factors such as a slowdown in VC funding and some (late) post-pandemic adjustments also play a role in the headcount decisions.

Implications for Data Engineering

As organizations seek to do more with less, there’s a growing demand for generalists proficient in cloud-native technologies, data, AI, and platform engineering. This shift is steering the field away from highly specialized roles, such as ETL or BI engineers, in favor of a broader range of engineering skills. Data engineering teams in 2024 start resembling software engineering teams. This happens partially thanks to the growing maturity of data engineering as a discipline and partially out of necessity: data teams are expected to deliver more with less, and this requires building data products faster, often in smaller teams than before.

On the other hand, software engineers working on AI-driven features or data products start taking over many data engineering tasks, such as data cleaning, validation, and governance, because the quality of AI-based products depends on the quality of the underlying data. Tuning an LLM on bad data won’t lead to good outcomes for the business, regardless of how many GPUs we throw at it. You may notice that the boundaries between what software and data teams do are getting blurry in 2024.

To continue reading this article, click here.

6 thoughts on “2024 Data Engineering Trends

  1. The Toyota Matrix, produced from 2003 to 2014, generally earned a reputation for reliability and practicality. However, there are a few model years that potential buyers may want to approach with caution. One such year is 2009, as it saw a recall related to engine control modules that could lead to stalling issues. Another year to consider carefully is 2010, due to reported problems with excessive oil consumption and engine oil leaks. Additionally, the 2013 model year had its share of concerns, with customers noting issues with the water pump and airbag systems. While overall, the Matrix is considered a dependable vehicle, prospective buyers should thoroughly research and inspect individual cars, especially those from the aforementioned years, to ensure they are aware of any potential issues and can make an informed purchasing decision.

  2. While the BMW 7 Series is generally known for its luxurious features and performance, some model years have experienced certain issues that potential buyers may want to be aware of. One notable period to exercise caution is the model years between 2002 and 2008, particularly with the E65/E66 generation. During this time, owners reported various electrical and electronic problems, including issues with the iDrive system, which controls many of the car’s functions. Additionally, there were complaints about the complexity and cost of repairs, particularly for components like the air suspension. It’s worth noting that BMW addressed many of these issues in later model years, so buyers may want to consider vehicles from the subsequent generations for a more reliable ownership experience.

  3. This trend is pushing for a broader skill set among Retro Bowl data engineers, blending roles with software engineers, as AI-driven projects demand high-quality data and versatile expertise. The result is a more dynamic, streamlined approach to data engineering, with a strong emphasis on AI and cloud-native technologies.


Leave a Reply