How to parallelize C or C++ programs

If you wanted to write the parallel program in either C or C++ programming language then you are in right place to get started with it. In this article, I will show how to parallelize C or C++ programs. We can parallelize the program within a process. Further, we can parallelize the program across the processes. The two libraries that we are going to look into are called Message Passing Interface(MPI) and Open Multi-Processing(OpenMP).

Why do we need to write a parallel program?

The main reason to parallelize a program code is to complete the task faster. As a result, it will save time and resources. For instance, if two men can complete a work in one hour. The same work can be completed in half an hour by 4 people. Which is faster. The same task can be carried out by all the processes or each process can do a different task also.

But remember that parallelizing the program does not always guarantee in speed up of the work. It depends on the nature of the task and how well you have done the load balancing of the task across the different processes. If you have not done the load balancing of the task properly, it might slow down the task instead of making it faster.

How to parallelize C or C++ programs within a process?

You can parallelize the C or C++ program code within a process by making use of the OpenMP library. With the help of the OpenMP library, you can define how many threads you want to launch. The internal implementation of it is taken care of by the library. In order to use it, you can include it in your code by “#include<omp.h>”. There are certain specific structures of the OpenMP library to follow. So that you can parallelize the program. Let’s demonstrate how to find the array sum using OpenMP programming.

Simple program using OpenMP programming

We will launch 3 threads. Then print the thread number simultaneously.

#include<iostream>
#include<omp.h>
using namespace std;
int main(){
	omp_set_num_threads(3);
	#pragma omp parallel
	{
		int threadID = omp_get_thread_num();
		cout << "I am thread " <<  threadID << endl;
	}
	return 0;
}

To summarize the above code, the omp_set_num_threads() set the value of the total number of threads created in the program. The “omp_get_thread_num” is used to get the thread number. The “#pragma omp parallel” is the syntax for parallelizing the for a loop. The same can be implemented in C programming.

How to parallelize C or C++ programs across different processes?

You can parallelize C or C++ program code across the different processes using the MPI library. You can define the number of processes that need to be launched. But you have to decide on what kind of task is done by each process. Further, you have to handle the communication between the different processes. It follows the paradigm of master and slave processes. One process will be considered as master and the rest of the processes will be considered as slaves. You can include the MPI library in your program code by “#include<mpi.h>”. Let’s write the simple greeting program code using MPI programming.

Greeting program using MPI Programming

You might be wondering what the greeting program is. In the greeting program, the master process will ask “How are you?” questions and the slave process will reply by saying “I am fine, thank you!”.

#include<stdio.h>
#include<mpi.h>

int main(int argc, char *argv[]){
	int rank, size;
	MPI_Init(&argc, &argv);
	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
	MPI_Comm_size(MPI_COMM_WORLD, &size);
	if(rank == 0){
		printf("Rank %d: How are you?\n", rank);
	}else{
		printf("Rank %d: I am fine, thank you!\n", rank);
	}
	MPI_Finalize();
	return 0;
}

To summarize the above MPI library function used, the MPI_Init() initializes the MPI, MPI_Comm_rank() and MPI_Comm_size() defines the total number of processes and assign the rank to each process respectively. Then finally, MPI_Finalize() binds all the program code. The same can be implemented in C++ programming.

Conclusion

You have learned how to parallelize C or C++ programs. In order to achieve it, we make use of OpenMP and MPI libraries. OpenMP can be used to parallelize the program within a process. Whereas MPI can be used to parallelize the program code across the different processes. Each process can do the same or different kinds of tasks. We can make use of a combination of both. The parallelizing of the program does not always make the program faster. It can make the program slower also if you do not perform load balancing properly.

You may want to read about how to find the sum of the array using MPI programming.