Convert user types to MPI_BYTE

[aprokop]

This logic seems problematic to me:

kokkos-comm/src/impl/KokkosComm_types.hpp

Lines 102 to 105 in fc25da1

	else {
	static_assert(std::is_void_v<T>, "mpi_type not implemented");
	return MPI_CHAR; // unreachable
	}

If I have a View<Box*>, with Box being some POD struct, in my opinion that view should be sent with message size view.size() * sizeof(View::value_type) and MPI type MPI_BYTE. Instead, right now it refuses to deal with it.

[cedricchevalier19]

I am not an MPI expert, but I think doing something like that is not semantically sound. Custom MPI datatypes can handle this stuff, but perhaps we can also explicitly serialize the data.

@devreal @hmt23 ?

[devreal]

The problem with opaque types in general is that they may contains parts that we must not copy to other processes (like virtual function tables). Kokkos could check for is_trivially and transfer only those types with MPI_BYTE. We'd still copy padding bytes, which may be acceptable (it isn't for MPI derived types but Kokkos could relax that constraint). Other types need a way of serializing, maybe in the spirit of Boost::serialization (which is slow without some hackery, ask me how I know...)