I have a question regarding the understanding of Dask and parallel processing and computing.
Here in this article it says:
Dask can efficiently perform parallel computations on a single machine using multi-core CPUs. For example, if you have a quad core processor, Dask can effectively use all 4 cores of your system simultaneously for processing . In order to use lesser memory during computations, Dask stores the complete data on the disk, and uses chunks of data (smaller parts, rather than the whole data) from the disk for processing. During the processing, the intermediate values generated (if any) are discarded as soon as possible, to save the memory consumption.
my question is that if dask keeps data on the disk and use chunks of data from the disk for processing and hence saving the memory consumption, then where is the parallel computing if data would load in to memory in chunks?
if the process of loading chunks and computing on them is parallel, then how it can save the memory usage?