Ghost particles and particle exchange. More...

#include "BoxGeometry.hpp"
#include "ParticleList.hpp"
#include <utils/Vector.hpp>
#include <boost/mpi/communicator.hpp>
#include <cstddef>
#include <utility>
#include <vector>

Include dependency graph for ghosts.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
struct	GhostCommunication

struct	GhostCommunicator
	Properties for a ghost communication. More...

Macros
Transfer types, for \ref GhostCommunicator::type
#define	GHOST_SEND 0
	send to a single node

#define	GHOST_RECV 1
	recv from a single node

#define	GHOST_BCST 2
	broadcast, the node entry gives the sender

#define	GHOST_RDCE 3
	reduce, the node entry gives the receiver

#define	GHOST_LOCL 4
	transfer data from cell to cell on this node

#define	GHOST_JOBMASK 15
	mask to the job area of the transfer type

#define	GHOST_PREFETCH 16
	additional flag for prefetching

#define	GHOST_PSTSTORE 32
	additional flag for poststoring

Enumerations
enum	: unsigned { GHOSTTRANS_NONE = 0u , GHOSTTRANS_PROPRTS = 1u , GHOSTTRANS_POSITION = 2u , GHOSTTRANS_MOMENTUM = 8u , GHOSTTRANS_FORCE = 16u , GHOSTTRANS_RATTLE = 32u , GHOSTTRANS_PARTNUM = 64u , GHOSTTRANS_BONDS = 128u }
	Transfer data classes, for ghost_communicator. More...

Functions
void	ghost_communicator (GhostCommunicator const &gcr, BoxGeometry const &box_geo, unsigned int data_parts)
	Do a ghost communication with the specified data parts.

Detailed Description

Ghost particles and particle exchange.

In this file you find everything concerning the exchange of particle data (particles, ghosts, positions and forces) for short range interactions between the spacial domains of neighbouring nodes.

How does this work

The ghost communication transfers data from cells on one node to cells on another node during the integration process. Note that data can only be transferred from one cell to one other, and the contents of the other cell will be overwritten.

The particle data that has to be transferred, and especially from where to where, heavily depends on the cell system. In ESPResSo, this is abstracted in form of ghost communicators (GhostCommunicator) and ghost communications (GhostCommunication). The ghost communicators represent the four communications above and consist of the data to transfer (which is determined by their type) and a list of ghost communications. The data types are described by the particle data classes:

GHOSTTRANS_PROPRTS transfers the ParticleProperties
GHOSTTRANS_POSITION transfers the ParticlePosition
GHOSTTRANS_MOMENTUM transfers the ParticleMomentum
GHOSTTRANS_FORCE transfers the ParticleForce
GHOSTTRANS_RATTLE transfers the ParticleRattle
GHOSTTRANS_PARTNUM transfers the cell sizes

Each ghost communication describes a single communication of the local with another node (or all other nodes). The data transferred can be any number of cells, there are five communication types:

GHOST_SEND sends data to one other node
GHOST_RECV recvs data from one other node. In the case of forces, they are added up, everything else is overwritten
GHOST_BCST sends data to all nodes
GHOST_RDCE recvs data from all nodes. In the case of forces, they are added up, everything else is overwritten
GHOST_LOCL transfer data from a local cell to another local cell. In this case, the first half of the cells are the sending cells, the other half are the corresponding receivers.

Note that for the first four communications you have to make sure that when one node sends to another, that the sender has GHOST_SEND and the receiver GHOST_RECV. In the case of GHOST_BCST resp. GHOST_RDCE, all nodes have to have the same communication type and the same master sender/receiver (just like the MPI commands).

A special topic are GHOST_PREFETCH and GHOST_PSTSTORE. For example, if all nodes broadcast to the other, the naive implementation will be that n_nodes times a GHOST_BCST is done with different master nodes. But this means that each time n_nodes - 1 nodes wait for the master to construct its send buffer. Therefore there is the prefetch flag which can be set on a pair of recv/send operations. If the ghost communication reaches a recv operation with prefetch, the next send operation (which must have the prefetch set!!) is searched and the send buffer already created. When sending, this precreated send buffer is used. In the scenario above, all nodes create the send buffers simultaneously in the first communication step, thereby reducing the latency a little bit. The pststore is similar and postpones the write back of received data until a send operation (with a precreated send buffer) is finished.

The ghost communicators are created by the cell systems.

Definition in file ghosts.hpp.

Macro Definition Documentation

◆ GHOST_BCST

#define GHOST_BCST 2

broadcast, the node entry gives the sender

Definition at line 108 of file ghosts.hpp.

◆ GHOST_JOBMASK

#define GHOST_JOBMASK 15

mask to the job area of the transfer type

Definition at line 115 of file ghosts.hpp.

◆ GHOST_LOCL

#define GHOST_LOCL 4

transfer data from cell to cell on this node

Definition at line 112 of file ghosts.hpp.

◆ GHOST_PREFETCH

#define GHOST_PREFETCH 16

additional flag for prefetching

Definition at line 117 of file ghosts.hpp.

◆ GHOST_PSTSTORE

#define GHOST_PSTSTORE 32

additional flag for poststoring

Definition at line 119 of file ghosts.hpp.

◆ GHOST_RDCE

#define GHOST_RDCE 3

reduce, the node entry gives the receiver

Definition at line 110 of file ghosts.hpp.

◆ GHOST_RECV

#define GHOST_RECV 1

recv from a single node

Definition at line 106 of file ghosts.hpp.

◆ GHOST_SEND

#define GHOST_SEND 0

send to a single node

Definition at line 104 of file ghosts.hpp.

Enumeration Type Documentation

◆ anonymous enum

anonymous enum : unsigned

Transfer data classes, for ghost_communicator.

Enumerator
GHOSTTRANS_NONE
GHOSTTRANS_PROPRTS	transfer ParticleProperties
GHOSTTRANS_POSITION	transfer ParticlePosition
GHOSTTRANS_MOMENTUM	transfer ParticleMomentum
GHOSTTRANS_FORCE	transfer ParticleForce
GHOSTTRANS_RATTLE	transfer ParticleRattle
GHOSTTRANS_PARTNUM	resize the receiver particle arrays to the size of the senders
GHOSTTRANS_BONDS

Definition at line 123 of file ghosts.hpp.

Function Documentation

◆ ghost_communicator()

void ghost_communicator	(	GhostCommunicator const &	gcr,
		BoxGeometry const &	box_geo,
		unsigned int	data_parts
	)

Do a ghost communication with the specified data parts.

Definition at line 440 of file ghosts.cpp.

References add_forces_from_recv_buffer(), add_rattle_correction_from_recv_buffer(), CommBuf::bonds(), calc_transmit_size(), cell_cell_transfer(), GhostCommunicator::communications, CommBuf::data(), GHOST_BCST, GHOST_JOBMASK, GHOST_LOCL, GHOST_PREFETCH, GHOST_PSTSTORE, GHOST_RDCE, GHOST_RECV, GHOST_SEND, GHOSTTRANS_FORCE, GHOSTTRANS_NONE, GHOSTTRANS_RATTLE, is_recv_op(), is_send_op(), GhostCommunicator::mpi_comm, GhostCommunication::node, node, prepare_recv_buffer(), prepare_send_buffer(), put_recv_buffer(), REQ_GHOST_SEND, CommBuf::size(), this_node, and GhostCommunication::type.

Referenced by CellStructure::ghosts_count(), CellStructure::ghosts_reduce_forces(), CellStructure::ghosts_reduce_rattle_correction(), CellStructure::ghosts_update(), and HybridDecomposition::resort().

Classes

Macros

Enumerations

Functions