SST with one-side TCP

Created by: jychoi-hpc

Trying to use SST between NERSC DTN (as a writer) and Cori compute node (as a reciever) but failed. I think the problem is caused when one side has no accessible address. I.e.,
while DTN nodes have a public IP address, Cori computer node have no public IP address.

@eisenhauer I remember there was technique called "reverse connection" or something like that to resolve such situation. Will it be difficult to add such function in SST?

Here is some information collected by using sst_conn_tool which I found very useful:

From the writer (DTN):

jyc@dtn03 delta-fusion (master) $ sst_conn_tool -i

	ADIOS2_IP_CONFIG best guess hostname is "dtn03.nersc.gov"
	ADIOS2_IP_CONFIG Possible interface lo : IPV4 127.0.0.1
	ADIOS2_IP_CONFIG Possible interface eth0 : IPV4 10.55.46.183
	ADIOS2_IP_CONFIG Possible interface ib100 : IPV4 10.100.107.103
	ADIOS2_IP_CONFIG Possible interface bond0 : IPV4 128.55.150.20
	ADIOS2_IP_CONFIG Possible interface bond1.136 : IPV4 128.55.136.20
	ADIOS2_IP_CONFIG Possible interface bond2.205 : IPV4 128.55.205.20
	ADIOS2_IP_CONFIG best guess IP is "128.55.205.20"
	ADIOS2_IP_CONFIG specified port range is "ANY" (unspecified)

 	The following environment variables can impact ADIOS2_IP_CONFIG operation:
		ADIOS2_IP  		-  Publish the specified IP address for contact
		ADIOS2_HOSTNAME  	-  Publish the specified hostname for contact
		ADIOS2_USE_HOSTNAME 	-  Publish a hostname preferentially over IP address
		ADIOS2_INTERFACE  	-  Use the IP address associated with the specified network interface
		ADIOS2_PORT_RANGE  	-  Use a port within the specified range "low:high",
		                 	   or specify "any" to let the OS choose

	Sst writer is listening for TCP/IP connection at IP 128.55.205.20, port 40764

	Sst connection tool waiting for connection...

Writer 0 (0x10299b0): Connection failed in SstInitWSReader! Contact list was:
Writer 0 (0x10299b0): AAMBAClXsP5pCYAKCF7/GGDqAAANb2leK7MAANWI344DAAAABQBlbmV0AAA=
Writer 0 (0x10299b0): Potential reader registration failed

From the Cori computer node:

sst_conn_tool -c -i
[PE_0]: MPI rank order: Using default aprun rank ordering.
[PE_0]: rank 0 is on nid02390
[PE_0]: cpumask set to 272 cpus on nid02390, cpumask = 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

	ADIOS2_IP_CONFIG Possible interface lo : IPV4 127.0.0.1
	ADIOS2_IP_CONFIG Possible interface ipogif0 : IPV4 10.128.9.105
	ADIOS2_IP_CONFIG best guess hostname is "10.128.9.105"
	ADIOS2_IP_CONFIG best guess IP is "10.128.9.105"
	ADIOS2_IP_CONFIG specified port range is "ANY" (unspecified)

 	The following environment variables can impact ADIOS2_IP_CONFIG operation:
		ADIOS2_IP  		-  Publish the specified IP address for contact
		ADIOS2_HOSTNAME  	-  Publish the specified hostname for contact
		ADIOS2_USE_HOSTNAME 	-  Publish a hostname preferentially over IP address
		ADIOS2_INTERFACE  	-  Use the IP address associated with the specified network interface
		ADIOS2_PORT_RANGE  	-  Use a port within the specified range "low:high",
		                 	   or specify "any" to let the OS choose

	Sst reader at IP 10.128.9.105, listening UDP port 45867


	Attempting TCP/IP connection to writer at IP 128.55.205.20, port 40764