Skip to content

Problems when running Dola-static decoding #25

@Lotusboat

Description

@Lotusboat

I'm following your work and trying to run your code. However, when I try to run Dola-static mode using the sample "python gsm8k_eval.py --model-name huggyllama/llama-7b --early-exit-layers 16,32 --data-path /path/to/data/folder --output-path output-path.json --num-gpus 1", some errors occured:

Traceback (most recent call last):
File "/mnt/dola/gsm8k_eval.py", line 313, in
model_completion, c_dist = llm.generate(input_text, **generate_kwargs)
File "/mnt/dola/dola.py", line 104, in generate
outputs = self.model.generate(input_ids, max_length=max_len, num_return_sequences=1,
File "/home/miniconda3/envs/dola/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/mnt/dola/transformers-4.28.1/src/transformers/generation/utils.py", line 1244, in generate
self._validate_model_kwargs(model_kwargs.copy())
File "/mnt/dola/transformers-4.28.1/src/transformers/generation/utils.py", line 1115, in _validate_model_kwargs
raise ValueError(
ValueError: The following model_kwargs are not used by the model: ['premature_layer'] (note: typos in the generate arguments will also show up in this list)

My environment has torch=2.7.1 and python=3.10.0. I checked function generate in utils.py but didn't find parameter premature_layers. How can I deal with this? Can you give me some suggestion? Thanks very much!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions