Cannot train the Embedding layer

[Leo-LiHao]

Hi, I'm trying to train a model with the Embedding layer, however, it does not work. Here is the example code:

// set up the model 
var inputs = keras.Input(shape: (5), name: "text", dtype: TF_DataType.DtInt32Ref);
var embed = keras.layers.Embedding(1000, 64, input_length: 5).Apply(inputs);
var embed_flat = keras.layers.Flatten().Apply(embed);
var output = keras.layers.Dense(10).Apply(embed_flat);
var model = keras.Model(inputs, output, name: "test");

// init input and labels
var input_array = np.random.randint(1000, size: (100, 5)).ravel().ToArray<int>();
var x_train = new NDArray(input_array, (100, 5));
var labels = np.random.randint(10, size: (100, 1)).ravel().ToArray<int>();
var y_train = new NDArray(labels, (100));
            
var opt = keras.optimizers.SGD(5e-2f);
model.compile(opt, keras.losses.SparseCategoricalCrossentropy(from_logits: true), metrics: new[] { "accuracy" });
model.summary();
model.fit(x_train, y_train, 10, 10, 1, 0.0f);

The error information:

Tensorflow.InvalidArgumentError: var and delta do not have the same shape[1000,64] [50,64]

[daz10000]

I just hit a similar problem with this simple code (var and grad, not var and delta!) The dimensions are strange - one is the vocab size x embedding layer size (8x21) and the other is 32x15x8 (480x8), or the batch size x message length x output embedding layer size. The latter is the shape I'd expect to be working through - the vocab size is odd to have show up as a dimension.

var and grad do not have the same shape[21,8] [480,8]

#r "nuget:TensorFlow.Net" 
#r "nuget:TensorFlow.Keras" 
#r "nuget:NumSharp"
#r "nuget:SciSharp.TensorFlow.Redist"

open type Tensorflow.KerasApi
open Tensorflow.Keras.Layers
open Tensorflow.NumPy

let vocab = 21
let messageLength = 15
let layers = LayersApi()
let inputs = layers.Input(messageLength)
let embedding = layers.Embedding(input_dim = vocab, input_length=15,output_dim = 8).Apply(inputs)
let flatten = layers.Flatten().Apply(embedding)

let dense1 = layers.Dense(8, activation = "relu").Apply(flatten)
let dense2 = layers.Dense(1, activation = "relu").Apply(dense1)
let model = keras.Model(inputs, dense2)

model.summary()

model.compile(
        optimizer = keras.optimizers.Adam(),
        loss = keras.losses.BinaryCrossentropy(),
        metrics = [|"accuracy"|]
)

let samples = 1000
let rng = System.Random()
let input = Array2D.init samples messageLength 
                            (fun j i -> rng.Next(vocab) |> float32 )

let output = 
    Array.init 
        samples 
        (fun j -> 
            let values = [| for i in 0..messageLength-1 -> input.[j,i]  |]
            if (values |> Array.average )>= 10.0f then 1.0 else 0.0
        )

model.fit(np.array input,np.array output,epochs=10,batch_size=32)

Tensorflow.InvalidArgumentError: var and grad do not have the same shape[21,8] [480,8]
   at Tensorflow.Eager.EagerRunner.TFE_FastPathExecute(FastPathOpExecInfo op_exec_info)
   at Tensorflow.Contexts.Context.ExecEagerAction(String OpType, String Name, ExecuteOpArgs args)
   at Tensorflow.Contexts.Context.ExecuteOp(String opType, String name, ExecuteOpArgs args)
   at Tensorflow.gen_training_ops.resource_apply_adam(Tensor var, Tensor m, Tensor v, Tensor beta1_power, Tensor beta2_power, Tensor lr, Tensor beta1, Tensor beta2, Tensor epsilon, Tensor grad, Boolean use_locking, Boolean use_nesterov, String name)
   at Tensorflow.Keras.Optimizers.Adam._resource_apply_dense(IVariableV1 var, Tensor grad, Dictionary`2 apply_state)
   at Tensorflow.Keras.Optimizers.OptimizerV2.apply_grad_to_update_var(IVariableV1 var, Tensor grad, Dictionary`2 apply_state)
   at Tensorflow.Keras.Optimizers.OptimizerV2.<>c__DisplayClass25_1.<_distributed_apply>b__1(NameScope <p0>)
   at Tensorflow.Binding.tf_with[T](T py, Action`1 action)
   at Tensorflow.Keras.Engine.Model.train_step(DataHandler data_handler, Tensors x, Tensors y)
   at Tensorflow.Keras.Engine.Model.train_step_function(DataHandler data_handler, OwnedIterator iterator)
   at Tensorflow.Keras.Engine.Model.FitInternal(DataHandler data_handler, Int32 epochs, Int32 verbose, List`1 callbackList, Nullable`1 validation_data, Func`3 train_step_func)
   at Tensorflow.Keras.Engine.Model.fit(NDArray x, NDArray y, Int32 batch_size, Int32 epochs, Int32 verbose, List`1 callbacks, Single validation_split, Nullable`1 validation_data, Boolean shuffle, Int32 initial_epoch, Int32 max_queue_size, Int32 workers, Boolean use_multiprocessing)
   at <StartupCode$FSI_0006>.$FSI_0006.main@() in bug2.fsx:line 89
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Stopped due to error

My only joy so far has been that I can change the message (slightly!) to

Tensorflow.InvalidArgumentError: Incompatible shapes: [21,8] vs. [480,8]

By switching the optimizer! - I don't think that's relevant though.

optimizer = keras.optimizers.RMSprop()

[daz10000]

I was starting to go a little crazy trying to get this to work, so I did a sanity check with (I think!) the same net in python keras and it seems to work. The cool thing about tensorflow.net is that it maintains the naming schema and conventions, so this is easy to flip over. I'm starting to suspect there's a bug in the embedding implementation, or a convention I don't understand. It's odd that the vocab size reemerges as a dimension in the shape since that's implicit in the scale of the integers for the input, but should never be a dimension (unless there's one-hot encoding expected or materialized somewhere). I'll see if I can follow the embedding implementation.

import tensorflow as tf
import tensorflow.keras.layers as layers
import tensorflow.keras.models as models
import numpy as np
import random

vocab = 21
messageLength = 15

inputs = layers.Input(messageLength)
embedding = layers.Embedding(input_dim = vocab, input_length=15,output_dim = 8)(inputs)
flatten = layers.Flatten()(embedding)

dense1 = layers.Dense(32, activation = "relu")(flatten)
dense2 = layers.Dense(1, activation = "sigmoid")(dense1)
model = models.Model(inputs, dense2)

model.summary()

model.compile(
        optimizer = 'adam',
        loss = 'binary_crossentropy',
        metrics = ['accuracy']
)

samples = 1000
input = np.random.randint(low=0,high=vocab-1, size = (samples, messageLength))

output = np.array([ 0 for i in range(0,samples) ] )

for j in range(0,samples):
    t = 0
    for i in range(0,messageLength):
        t = t + input[j,i]
    if t/messageLength >= 10:
        output[j] = 1

model.fit(input,output,epochs=10,batch_size=32)

>>> model.fit(input,output,epochs=10,batch_size=32)
Epoch 1/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0850 - accuracy: 0.9870
Epoch 2/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0751 - accuracy: 0.9920
Epoch 3/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0708 - accuracy: 0.9920
Epoch 4/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0622 - accuracy: 0.9890
Epoch 5/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0574 - accuracy: 0.9930
Epoch 6/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0531 - accuracy: 0.9900
Epoch 7/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0489 - accuracy: 0.9940
Epoch 8/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0461 - accuracy: 0.9950
Epoch 9/10
32/32 [==============================] - 0s 2ms/step - loss: 0.0423 - accuracy: 0.9960
Epoch 10/10
32/32 [==============================] - 0s 3ms/step - loss: 0.0404 - accuracy: 0.9990
<keras.callbacks.History object at 0x000001E574EFA440>

[SanftMonster]

@Wanglongzhi2001 It seems to be a BUG during the shape inference of model build. Would you like to help with this issue?

[Wanglongzhi2001]

@Wanglongzhi2001 It seems to be a BUG during the shape inference of model build. Would you like to help with this issue?

OK.

[daz10000]

Let me know if I can help / test any branches. It seems like it's probably a confusion on the dimensions (it's tricky - took me a while to wrap my head around the 'vocabulary' dimension since it's implicit in the range of values you are feeding in. You're doing a heroic job putting this all together - and I want to get more familiar with code so I can help. Probably most useful thing I can do short term is write some simple examples and then ask dumb questions :) Fixing this embedding problem will also help me to test the RNN stuff more thoroughly, since many tasks involving recurrent nets on text input also end up using embedding as a first step.

[Wanglongzhi2001]

Let me know if I can help / test any branches. It seems like it's probably a confusion on the dimensions (it's tricky - took me a while to wrap my head around the 'vocabulary' dimension since it's implicit in the range of values you are feeding in. You're doing a heroic job putting this all together - and I want to get more familiar with code so I can help. Probably most useful thing I can do short term is write some simple examples and then ask dumb questions :) Fixing this embedding problem will also help me to test the RNN stuff more thoroughly, since many tasks involving recurrent nets on text input also end up using embedding as a first step.

Thank you for your kindness and attention to TensorFlow.NET, the reason cause this problem is that the shape of variables of the Embedding layer and the corresponding gradient are not euqal during training. And you are right, embedding layer is important to RNN. I'm fixing this. BTW, I'll let you know if need to add some Embedding or RNN-related tests later. ^_^

[daz10000]

Hey! - wanted to say thanks, and confirm that the changes related to #1125 appear to resolve this issue ( I just verified with the published nuget libs and my code above). I think it's safe to close the ticket, but I didn't open it originally, so maybe @Leo-LiHao can close, or also verify.