Add training description to report.

#matplotlib.rc("ytick", labelsize=14)

def heat_maps(dataset, depths, widths):

def heat_maps(dataset, depths, widths, save_csv=False):

	dir = os.path.join("figs", dataset, "hm")

	os.makedirs(dir, exist_ok=True)

	quants = ["loss", "v_b", "v_w", "v_n"]

	ims = {q: np.array(v) for (q, v) in ims.items()}

	for (q, title) in zip(quants, titles):

		plt.figure()

		plt.imshow(ims[q], cmap="YlGnBu_r")

		plt.imshow(ims[q], cmap="YlGnBu_r", origin="lower")

		plt.colorbar()

		plt.xticks(np.arange(len(widths))+1, widths)

		plt.yticks(np.arange(len(depths))+1, depths)

		filename = os.path.join(dir, "{}.png".format(q))

		plt.savefig(filename, box_inches="tight")

		plt.close()

		if save_csv:

			data = np.concatenate((np.asarray(widths).reshape((-1, 1)), ims[q].T), axis=1)

			headers = ["width"] + ["depth_{:.0f}".format(d) for d in depths]

			df = pd.DataFrame(data, columns=headers)

			for h in headers:

				if h == "width":

					df[h] = df[h].map(lambda x: str(int(x)))

				else:

					df[h] = df[h].map(lambda x: "{:.3f}".format(x))

			df.to_csv(os.path.join(dir, q+".csv"), index=False)

def learning_curves(dataset, depth, width):

	main_text = main_text.replace("%lcsections%", lc_sections)

	main_text = main_text.replace("%dataset%", dataset)

	main_text = main_text.replace("%datasetname%", dataset_name)

	with open( os.path.join("../results", dataset, "depths1_{}_widths{}.json".format(lc_depths[0], lc_widths[0])), "r") as fp:

		args = json.load(fp)["args"]

	for (k, v) in args.items():

		k = "%{}%".format(k)

		main_text = main_text.replace(k, str(v))

	with open(dataset+".tex", "w") as fp:

		fp.write(main_text)

	dataset_name = "Boston housing prices"

	hm_depths = [10*i for i in range(1, 10)]

	hm_widths = [4, 16, 64, 256]

	heat_maps(dataset, hm_depths, hm_widths)

	hm_widths = [2, 4, 8, 16, 32, 64, 128, 256]

#	heat_maps(dataset, hm_depths, hm_widths, save_csv=True)

	lc_depths = [10, 30, 50, 70, 90]

	lc_widths = [4, 16, 64, 256]

	for d in lc_depths:

		for w in lc_widths:

			learning_curves(dataset, d, w)

#	for d in lc_depths:

#		for w in lc_widths:

#			learning_curves(dataset, d, w)

	tex(dataset, dataset_name, lc_depths, lc_widths)

\newpage

\section*{Depth %depth% width %width%}

\begin{center}

\begin{tabular}{cc}

\includegraphics[width=0.5\linewidth]{figs/%dataset%/lc/d%depth%w%width%/loss.png} & %

\includegraphics[width=0.5\linewidth]{figs/%dataset%/lc/d%depth%w%width%/delta_loss.png} \\

\includegraphics[width=0.5\linewidth]{figs/%dataset%/lc/d%depth%w%width%/params.png} & %

\quad \\

\end{tabular}

\end{center}

\begin{document}

\maketitle

We consider a single-bottleneck NNGP model with normalized ReLU activation, one infinite-width hidden layer before the bottleneck, $D$ infinite-width hidden layers after the bottleneck, and bottleneck width $W$. 

We apply the bottleneck NNGP model to the %datasetname% dataset for various $D$ and $W$ and compare their likelihoods. 

For each $D$ and $W$, we optimize the model hyperparameters $v_b$, $v_w$, and $v_n$ with maximum marginal likelihood. 

We initialize the hyperparameters to $v_b=%vb%$, $v_w=%vw%$, and $vn=%vn%$. 

We train the model for $%iters%$ iterations of Adam with learning rate $%lr%$ and full batches. 

The loss function is the negative marginal log-likelihood divided by the dataset size. 

We note that the loss function is stochastic since we sample the bottleneck layer during the forward pass; 

we draw $%train_samples%$ samples at each iteration of training. 

Due to this stochasticity, it can be difficult to tell if the learning rate is sufficiently small based only on the loss-vs-iterations learning curve. 

Therefore, at each iteration, we record the loss before updating the parameters and the loss after updating the parameters, 

where we make sure to use the same draw from the bottleneck layer in both cases. 

The change between these two loss values must be negative at each iteration; 

whenever the loss increases after the parameter update, we multiply the learning rate by $0.9$.

After training, we compute the final loss using the optimized model hyperparameters, 

this time drawing $%test_samples%$ samples from the bottleneck layer. 

The heat maps on the next page compare the final losses (negative marginal log-likelihood per data example) and the optimal hyperparameters over all bottleneck NNGP architectures that were tested. 

``Depth'' and ``Width'' refer to post-bottleneck depth $D$ and bottleneck width $W$ respectively. 

The remaining pages present learning curves for a sample of the architectures that were tested. 

The ``Loss'' learning curve includes the loss at each iteration before the parameter update (red) and after the parameter update (blue), 

where again the same sample draw from the bottleneck layer was used within each iteration. 

The sample draws are IID for distinct iterations, hence the noise in the learning curves. 

The ``Change in loss'' learning curve gives the difference between the red and blue losses as a percentage of the red loss.

\newpage

\begin{center}

\begin{tabular}{cc}

\includegraphics[width=0.5\linewidth]{figs/%dataset%/hm/loss.png} & %

\includegraphics[width=0.5\linewidth]{figs/%dataset%/hm/v_b.png} \\

\includegraphics[width=0.5\linewidth]{figs/%dataset%/hm/v_w.png} & %

\includegraphics[width=0.5\linewidth]{figs/%dataset%/hm/v_n.png} \\

\end{tabular}

\end{center}

%lcsections%