forked from cornelisnetworks/opa-psm2
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
233 lines (179 loc) · 8.77 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
This file is provided under a dual BSD/GPLv2 license. When using or
redistributing this file, you may do so under either license.
GPL LICENSE SUMMARY
Copyright(c) 2015 Intel Corporation.
This program is free software; you can redistribute it and/or modify
it under the terms of version 2 of the GNU General Public License as
published by the Free Software Foundation.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
Contact Information:
Intel Corporation, www.intel.com
BSD LICENSE
Copyright(c) 2015 Intel Corporation.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
* Neither the name of Intel Corporation nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Copyright (c) 2003-2014 Intel Corporation. All rights reserved.
================================================================================
ABSTRACT
--------
Discusses how to build, install and test the PSM 2 library source code.
Contains the following sections:
- INTRODUCTION
- DEPENDENCIES
- BUILDING
- BUILDING USING MAKEFILE
- BUILDING USING RPMBUILD (CREATING SOURCE AND BINARY RPM'S)
- INSTALLING USING MAKEFILE
- INSTALLING USING RPM
- RELATED SOFTWARE TO PSM
INTRODUCTION
------------
This README file discusses how to build, install and test the PSM 2 library
source code.
The PSM 2 library supports a number of fabric media and stacks, and all of
them run on version 7.X of Red Hat Enterprise Linux (abbreviated: RHEL), and
SuSE SLES.
Only the x86_64 architecture is supported.
As of this writing, (July 21, 2015) OPA is not natively supported in any RHEL
7.X kernel, and building PSM on a host w/o native OPA is not possible unless you
have the correct kernel-devel package; but if you have a modern version of the
IFS (Infiniband Fabric Suite), you should be able to build it. It is easier
if you wait until the hfi1 driver is upstreamed and becomes part of the RHEL 7.X.
There are two mechanisms for building and installing the PSM 2 library:
1. Use provided Makefiles to build and install or
2. Generate a tarball that can be used to build source and binary RPMs using
rpmbuild
DEPENDENCIES
------------
The following packages are required to build the PSM 2 library source code,
all packages are for the x86_64 architecture:
compat-rdma-devel
gcc-4.8.2
glibc-devel
glibc-headers
kernel-headers
libuuid-devel
In addition to depending on these packages, root privileges are required to
install the runtime libraries and development header files into standard
system location.
BUILDING
--------
The instructions below use $PRODUCT and $RELEASE to refer to the product
and release identifiers of the opa-psm RPM.
BUILDING USING MAKEFILES
------------------------
1. Untar the tarball:
$ tar zxvf opa-psm-$PRODUCT-$RELEASE.tar.gz
2. Change directory into the untarred location:
$ cd opa-psm-$PRODUCT-$RELEASE
3. Run make on the command line. This will build the PSM 2 library.
$ make
BUILDING USING RPMBUILD
-----------------------
This is a two step process: Generate a PSM distribution tarball which can then
be used to generate source and binary RPMs using the rpmbuild command with the
-ta option.
1. Untar the tarball:
$ tar zxvf opa-psm-$PRODUCT-$RELEASE.tar.gz
2. Change directory into the untarred location:
$ cd opa-psm-$PRODUCT-$RELEASE
3. Generate distribution tarball
$ make dist
4. Generate source and binary packages
$ rpmbuild -ta hfi1-psm-$PRODUCT-$RELEASE.tar.gz
As of this writing (July 21, 2015), this command results in the following
collection of rpm's and source code rpm's to populate your rpmbuild directory
tree in your home directory:
RPMS/x86_64/hfi1-psm-0.7-95.x86_64.rpm
RPMS/x86_64/hfi1-psm-devel-0.7-95.x86_64.rpm
RPMS/x86_64/hfi1-psm-compat-0.7-95.x86_64.rpm
RPMS/x86_64/hfi1-psm-compat-devel-0.7-95.x86_64.rpm
RPMS/x86_64/hfi1-psm-debuginfo-0.7-95.x86_64.rpm
SRPMS/hfi1-psm-0.7-95.src.rpm
INSTALLING USING MAKEFILE
-------------------------
Install the libraries and header files on the system (as root):
$ make install
The libraries will be installed in /usr/lib64, and the header files will
be installed in /usr/include.
This behavior can be altered by using the "DESTDIR" and "LIBDIR" variables on
the "make install" command line. "DESTDIR" will add a leading path component
to the overall install path and "LIBDIR" will change the path where libraries
will be installed. For example, "make DESTDIR=/tmp/psm-install install" will
install all files (libraries and headers) into "/tmp/psm-install/usr/...",
"make DESTDIR=/tmp/psm-install LIBDIR=/libraries install" will install the
libraries in "/tmp/psm-install/libraries" and the headers in
"/tmp/psm-install/usr/include", and "make LIBDIR=/tmp/libs install" will
install the libraries in "/tmp/libs" and the headers in "/usr/include".
INSTALLING USING RPM
--------------------
You can install the rpm's and source rpm's previously built using rpmbuild using
the rpm command, as the root user. See the rpm man page for details of
installing rpm's.
RELATED SOFTWARE TO PSM
=======================
MPI Libraries supported
-----------------------
A large number of open source (OpenMPI, MVAPICH, MVAPICH2) and Vendor MPI
implementations support PSM for optimized communication on HCAs. Vendor MPI
implementations (HP-MPI, Intel MPI 4.0 with PMI, Platform/Scali MPI)
require that the PSM runtime libraries be installed and available on
each node. Usually a configuration file or a command line switch to mpirun
needs to be specified to utilize the PSM transport.
OpenMPI support
---------------
It is recommended to use the OpenMPI v1.5 development branch. Prior versions
of OpenMPI have an issue with support PSM network transports mixed with standard
Verbs transport (BTL openib). This prevents an OpenMPI installation with
network modules available for PSM and Verbs to work correctly on nodes with
no HFI hardware. This has been fixed in the latest development branch
allowing a single OpenMPI installation to target HFI hardware via PSM or Verbs
as well as alternate transports seamlessly.
PSM header and runtime files need to be installed on a node where the OpenMPI
build is performed. All compute nodes additionally should have the PSM runtime
libraries available on them. OpenMPI provides a standard configure, make and
make install mechanism which will detect and build the relevant PSM network
modules for OpenMPI once the header and runtime files are detected.
MVAPICH and MVAPICH2 support
----------------------------
Both MVAPICH and MVAPICH2 support PSM transport for optimized communication on
HFI hardware. MVAPICH2 1.4 and MVAPICH 1.2 versions are
recommended. PSM header and runtime files need to be installed on a node where
MVAPICH builds are performed. All compute nodes additionally should have the
PSM runtime libraries available on them.
MVAPICH provides a shell script in its top level directory called
make.mvapich.psm to configure, make and install MVAPICH with PSM support.
MVAPICH2 provides a standard configure and make infrastructure. In order to
MVAPICH2 for PSM the following should be performed from the top level directory:
- ./configure --prefix=<path_to_install_mvapich2> --with-device=ch3:psm
- make
- make install
OFED Support
------------
OFED 1.5.1rc2 or above should be installed on the node. Prior versions of OFED
have an older HFI driver (ib_qib) and do not fully support all the PSM
features with this release.