SST with one-side TCP
Created by: jychoi-hpc
Trying to use SST between NERSC DTN (as a writer) and Cori compute node (as a reciever) but failed. I think the problem is caused when one side has no accessible address. I.e.,
while DTN nodes have a public IP address, Cori computer node have no public IP address.
@eisenhauer I remember there was technique called "reverse connection" or something like that to resolve such situation. Will it be difficult to add such function in SST?
Here is some information collected by using sst_conn_tool
which I found very useful:
From the writer (DTN):
jyc@dtn03 delta-fusion (master) $ sst_conn_tool -i
ADIOS2_IP_CONFIG best guess hostname is "dtn03.nersc.gov"
ADIOS2_IP_CONFIG Possible interface lo : IPV4 127.0.0.1
ADIOS2_IP_CONFIG Possible interface eth0 : IPV4 10.55.46.183
ADIOS2_IP_CONFIG Possible interface ib100 : IPV4 10.100.107.103
ADIOS2_IP_CONFIG Possible interface bond0 : IPV4 128.55.150.20
ADIOS2_IP_CONFIG Possible interface bond1.136 : IPV4 128.55.136.20
ADIOS2_IP_CONFIG Possible interface bond2.205 : IPV4 128.55.205.20
ADIOS2_IP_CONFIG best guess IP is "128.55.205.20"
ADIOS2_IP_CONFIG specified port range is "ANY" (unspecified)
The following environment variables can impact ADIOS2_IP_CONFIG operation:
ADIOS2_IP - Publish the specified IP address for contact
ADIOS2_HOSTNAME - Publish the specified hostname for contact
ADIOS2_USE_HOSTNAME - Publish a hostname preferentially over IP address
ADIOS2_INTERFACE - Use the IP address associated with the specified network interface
ADIOS2_PORT_RANGE - Use a port within the specified range "low:high",
or specify "any" to let the OS choose
Sst writer is listening for TCP/IP connection at IP 128.55.205.20, port 40764
Sst connection tool waiting for connection...
Writer 0 (0x10299b0): Connection failed in SstInitWSReader! Contact list was:
Writer 0 (0x10299b0): AAMBAClXsP5pCYAKCF7/GGDqAAANb2leK7MAANWI344DAAAABQBlbmV0AAA=
Writer 0 (0x10299b0): Potential reader registration failed
From the Cori computer node:
sst_conn_tool -c -i
[PE_0]: MPI rank order: Using default aprun rank ordering.
[PE_0]: rank 0 is on nid02390
[PE_0]: cpumask set to 272 cpus on nid02390, cpumask = 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
ADIOS2_IP_CONFIG Possible interface lo : IPV4 127.0.0.1
ADIOS2_IP_CONFIG Possible interface ipogif0 : IPV4 10.128.9.105
ADIOS2_IP_CONFIG best guess hostname is "10.128.9.105"
ADIOS2_IP_CONFIG best guess IP is "10.128.9.105"
ADIOS2_IP_CONFIG specified port range is "ANY" (unspecified)
The following environment variables can impact ADIOS2_IP_CONFIG operation:
ADIOS2_IP - Publish the specified IP address for contact
ADIOS2_HOSTNAME - Publish the specified hostname for contact
ADIOS2_USE_HOSTNAME - Publish a hostname preferentially over IP address
ADIOS2_INTERFACE - Use the IP address associated with the specified network interface
ADIOS2_PORT_RANGE - Use a port within the specified range "low:high",
or specify "any" to let the OS choose
Sst reader at IP 10.128.9.105, listening UDP port 45867
Attempting TCP/IP connection to writer at IP 128.55.205.20, port 40764