it means apply a transformation to every element of the dataset

To where?

reduce is equivalent to std::accumulate

it can also be used to filter in Hadoop

Filter while transforming?

yes

Why would someone do that?!

Hadoop is many bad things (like mainly not always the best solution) but at the very least it is flexible

Why would someone do that?!

let's say that you have a dataset of customers (like, 10M customers) and want to select all the customers that made purchases higher than 5000$, give them a discount and get the total amount of discount you gave in total

10M is large

We don't call it big data for show

I imagine transforming filter as gather the whole dataset and filter them in their road

that is exactly what it is

So If you wanna filter a dataset in this way You'll have to get them all first

and this is why you need big ass servers with hundreds of GB of RAM

But still I see searching filter is better

Prefiltered data is even better

but prefiltered data is harder to paralellize operations on

It's the same as searching filter, right?

no, it is the same as a Database index

exactly

it allows you to search the data instantaneously without much overhead

and the bloom filter i mentionned earlier is a way to know if a key is or not in a database (oversimplification)

hopefully Cassandra uses a third of that in bloom filters

Just in a quick way Instead of searching the whole database

you try to match you key vs the filter, if it matches, the data may be in the database, if even one point doesn't match the data definitely isn't

Of course it's in the database

Of course it's in the database

no, but it likely is in

it can have false positives but no false negatives

This is encrypted xD

let's get a simple example of the concept

you have 5 vowels a(0) e(0) i(0) o(0) u(0)

What are all these zeros

the filter

you add the word "yes" in the set