Unable to reduce time in summarization!

Environment info

  • transformers version: t5-large
  • Platform: Amazon Sagemaker
  • Python version: 3.7
  • Tensorflow version (GPU?):2.3
  • Using distributed or parallel set-up in script?: Unable to implement it properly.

Who can help

@LysandreJik, @patil-suraj, @jplu

Problem:

Unable to reduce the summarization time.

The tasks I am working on is:

I am using pretrained transformer of T5(TFT5ForConditionalGeneration) for text summarization.
Brief script:

inputs = tokenizer("summarize: " + text, return_tensors="tf").input_ids

outputs = model.generate(
        inputs, 
        max_length=200, 
        min_length=5,  
        num_beams=5,)

I tried to use distributed strategy of tensorflow. But it doesn’t made any improvement.

strategy = tf.distribute.MirroredStrategy()
  
strategy = tf.distribute.MirroredStrategy(devices=["/gpu:0", "/gpu:1"])

Expected behavior

I am hoping that if we increase the number of GPU, time must be reduced. But It is not happening in this case.

1 possible answer(s) on “Unable to reduce time in summarization!

  1. Hello!

    The generation part of TensorFlow can only be run on eagermode, hence doesn’t matter how you execute it, you will not be able to run it “fast”. It is planned to bring a graph execution for the generation, but no ETA yet. Sorry for the inconvenience.