Rendered at 07:03:37 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
icedchai 8 hours ago [-]
Whenever I encounter a "data lake" it's just a bunch of random CSV and JSON files on S3, and a bunch of half baked Python scripts to query it. And usually somebody tells me it's "big data", and that's why it is has to be that way. So I check the size and it would easily fit on a laptop from 15 years ago.
nullpoint420 59 minutes ago [-]
Bingo. Double points if the infra surrounding this costs more than a maxed out MBP a month in spend
citizenpaul 40 minutes ago [-]
The first time (and all others) I heard "data lake" my brain instantly translated it to "somone elses problem". As in the very name implies a carelessness to just dump whatever and somone else will take care of it.
dejobaan 11 hours ago [-]
Not a data lake expert here, but I think there's a parallel with telemetry/logging for video games. The author talks about mass "ingestion without purpose," which is what I've sometimes flubbed in the past—during playtesting, we'd collect piles of data about where the players won/lost/moved/whatever, and there was so darned much of it that we'd just toss our hands up and not do anything with it.
The article suggests (sensibly!) that the right thing to do is to actually go in and design what's being logged with purpose (i.e., everything should answer a known question). But I don't always know what I'm looking for when I'm testing a thing. At the risk of using LLMs as a hammer that hammers everything, I wonder if they'd help with that problem?
I've pointed Codex at Posthog logs, for instance, and it's been able to say, "yeah, well, look at this pile of residential proxies hitting your site." I wonder if it'd similarly be able to say, "hey, I notice a pattern where everyone who picks the Cleric class stops playing after level 3"?
rolymath 12 minutes ago [-]
> look at this pile of residential proxies hitting your site
How does it know that?
johsole 10 hours ago [-]
I'm building a product and exploring these questions in that domain. I'm thinking about it a little differently. I have a known set of hardware sensors that I'm collecting all the data from. Per sensor the data has a strong schema with a timestamp, what is interesting is the insights that can be gleaned across sensors. So what sensors do you have that you want to record, have a strong schema against that sensor, now have something you can use to tie across sensors, now you can use LLMs to query the data and draw interesting insights
capplexham 3 hours ago [-]
The paradigm shift from ETL to ELT is what went wrong with Data Lakes. An enormous amount of business logic and effort was historically accumulated in the Extract-Transform stage (usually in COBOL or SQL), and then Loaded into a Data Mart for end-users.
When ELT became the corporate "Strategy", Extract-Load was technically easy and had a lot of Big Data evangelism and exuberance behind it. The money flowed easily at the beginning, but budgets aren't unlimited and corporate knowledge isn't easy to retain. So Data Lakes became Swamps of just Extract-Load, and business users were told to transform the data themselves.
As someone who lived through it for 12 years at an Insurance company, I like to think of it as ETL becoming Extract-Load-GoFuckYourself.
analog31 6 hours ago [-]
The people who told us to push all of our data to the data lake are the ones who are now telling us to feed all of our knowledge to AI. In both cases, the dream was to replace us with computation, either ML or LLM's. We survived the first one relatively unscathed, don't know about the second one yet, at least for my own job.
mint5 11 hours ago [-]
It’s hard to take an article seriously that uses a snowflake ceo maligning Hadoop as evidence of something. (Note, I am already a believer that data lake efforts usually ended badly.)
Regardless of whether their criticism is factual, it’s absurd to use a direct competitors assessment of the competing tech as a credible source!
If you do that in the opening paragraphs of your paper, it’s wrecked all credibility for everything that follows.
The article suggests (sensibly!) that the right thing to do is to actually go in and design what's being logged with purpose (i.e., everything should answer a known question). But I don't always know what I'm looking for when I'm testing a thing. At the risk of using LLMs as a hammer that hammers everything, I wonder if they'd help with that problem?
I've pointed Codex at Posthog logs, for instance, and it's been able to say, "yeah, well, look at this pile of residential proxies hitting your site." I wonder if it'd similarly be able to say, "hey, I notice a pattern where everyone who picks the Cleric class stops playing after level 3"?
How does it know that?
When ELT became the corporate "Strategy", Extract-Load was technically easy and had a lot of Big Data evangelism and exuberance behind it. The money flowed easily at the beginning, but budgets aren't unlimited and corporate knowledge isn't easy to retain. So Data Lakes became Swamps of just Extract-Load, and business users were told to transform the data themselves.
As someone who lived through it for 12 years at an Insurance company, I like to think of it as ETL becoming Extract-Load-GoFuckYourself.
Regardless of whether their criticism is factual, it’s absurd to use a direct competitors assessment of the competing tech as a credible source!
If you do that in the opening paragraphs of your paper, it’s wrecked all credibility for everything that follows.