--

It does partially. E.g. Merge Sort.

To sort a partition whole partition needs to be loaded into memory. But while partition might not fit in the memory. So it loads the part that fits into memory, sorts it and then write this to disk. Then it loads the remainder of the partition into memory and sorts it. Once the entire partition is split into sorted sections and write to disk (section called spills), all the sections are merged into a single sorted partition

--

--

Amit Singh Rathore
Amit Singh Rathore

Written by Amit Singh Rathore

Staff Data Engineer @ Visa — Writes about Cloud | Big Data | ML

No responses yet