Label Points in Seaborn lmplot (python) with multiple plots

I'm trying to add labels to each data point in my lmplot. I want to label each data point by an index. Right now my code is the following:

p1=sns.lmplot(x="target", y="source", col="color", hue="color", 
              data=ddf, col_wrap=2, ci=None, palette="muted",
              scatter_kws={"s": 50, "alpha": 1})

def label_point(x, y, val, ax):
    a = pd.concat({'x': x, 'y': y, 'val': val}, axis=1)
    for i, point in a.iterrows():
    ax.text(point['x']+.02, point['y'], str(point['val']))

label_point(, ddf.source, ddf.chip, plt.gca())

This plots all the labels onto the last plot.

lmplot with labels

I tried label_point(, ddf.source, ddf.chip, plt.gcf()) instead to use the whole figure rather than the current axes but then it throws an error.

ValueError: Image size of 163205x147206 pixels is too large. 
It must be less than 2^16 in each direction.


The problem is, how should the labeling function know which plot to label, if the entire dataset is passed to it?!

As an example, you can use pandas' .groupby to loop through the unique colors and create a seaborn.regplot for each of them. Then it's easy to label each axes individually.

import matplotlib.pyplot as plt
import numpy as np; np.random.seed(42)
import pandas as pd
import seaborn as sns

def label_point(df, ax):
    for i, point in df.iterrows():
        ax.annotate("{:.1f}".format(point['val']), xy = (point['x'], point['y']),
                    xytext=(2,-2), textcoords="offset points")

df = pd.DataFrame({"x": np.sort(np.random.rand(50)),
                   "y": np.cumsum(np.random.randn(50)),
                   "val" : np.random.randint(10,31, size=50),
                   "color" : np.random.randint(0,3,size=50 )})
colors = ["crimson", "indigo", "limegreen"]

fig, axes = plt.subplots(2,2, sharex=True, sharey=True)

for (c, grp), ax in zip(df.groupby("color"), axes.flat):
    sns.regplot(x="x", y="y", data=grp, color=colors[c], ax=ax,
                scatter_kws={"s": 25, "alpha": 1})
    label_point(grp, ax)


enter image description here