convert torch 7 model and get an error :AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer' #8

Mrxuefei · 2017-05-23T08:51:33Z

wangjian@lhtserver-2:~/wj$ python convert_torch.py -m vgg.t7
Traceback (most recent call last):
File "convert_torch.py", line 286, in
torch_to_pytorch(args.model,args.output)
File "convert_torch.py", line 233, in torch_to_pytorch
model = load_lua(t7_filename,unknown_classes=True)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 599, in load_lua
return reader.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 584, in read
return self.read_object()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 537, in read_object
return reader_registry[cls_name](self, version)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 242, in read_nn_class
attributes = reader.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 586, in read
return self.read_table()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 563, in read_table
v = self.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 586, in read
return self.read_table()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 563, in read_table
v = self.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 584, in read
return self.read_object()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 539, in read_object
return TorchObject(cls_name, self.read())
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 586, in read
return self.read_table()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 563, in read_table
v = self.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 584, in read
return self.read_object()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 537, in read_object
return reader_registry[cls_name](self, version)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 146, in read_tensor
storage = reader.read()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 584, in read
return self.read_object()
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 514, in wrapper
result = fn(self, *args, **kwargs)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 537, in read_object
return reader_registry[cls_name](self, version)
File "/usr/local/anaconda2/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 165, in read_storage
return python_class.from_buffer(reader.f.read(size), 'native')
AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer'

clcarwin · 2017-05-23T10:24:47Z

Where did you download the vgg.t7? You can try check the size of the vgg.t7 and convert torch.cuda.FloatStorage to torch.FloatStorage before using it.

vgg model from https://github.com/jcjohnson/cnn-benchmarks has been tested, you can use this version.

johnny5550822 · 2017-05-30T23:01:46Z

@clcarwin I got the same error. I try to use this library to convert a standalone training model (from my experiments; not vgg). So is that mean I cannot use this library to convert torch.cuda.FloatStorage?

clcarwin · 2017-05-31T03:27:31Z

@johnny5550822 convert GPU to CPU model first.

m = torch.load('xxxx.t7')
m = m:float()
torch.save('xxxx.cpu.t7',m)

johnny5550822 · 2017-05-31T08:10:21Z

Got it. Thanks!

Cadene · 2017-06-03T10:39:10Z

Thanks ;) worked for resnet152 from https://github.com/facebook/fb.resnet.torch

heyalqh · 2017-07-07T09:44:53Z

@johnny5550822 do you get the problem " 'torch.FloatTensor' object has no attribute 'output' ",how should I do?

johnny5550822 · 2017-07-08T00:08:13Z

@heyalqh did not get this problem. Can you share the screenshot of the code and error?

heyalqh · 2017-07-08T02:13:18Z

@johnny5550822

DCurro · 2017-07-19T00:28:12Z

Does this mean that I cannot use the model on the GPU after conversion?

When I do module.cuda() and attempt to evaluate module.forward(data), I get the error:

RuntimeError: expected CPU tensor (got CUDA tensor)

When I put my data on the GPU data = Variable(torch.from_numpy(batch)).cuda()

I end up with this error instead:

Traceback (most recent call last):
  File "thing.py", line 51, in <module>
    module.forward(data.float())
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py", line 64, in forward
    input = module(input)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/home/d2curro/repos/thinslicing_pytorch/cpu_net.py", line 21, in forward
    return self.lambda_func(self.forward_prepare(input))
  File "/opt/home/d2curro/repos/thinslicing_pytorch/cpu_net.py", line 35, in <lambda>
    Lambda(lambda x,lrn=torch.legacy.nn.SpatialCrossMapLRN(*(5, 0.0005, 0.75, 2)): Variable(lrn.forward(x.data))),
  File "/usr/local/lib/python2.7/dist-packages/torch/legacy/nn/Module.py", line 33, in forward
    return self.updateOutput(input)
  File "/usr/local/lib/python2.7/dist-packages/torch/legacy/nn/SpatialCrossMapLRN.py", line 25, in updateOutput
    self._backend.SpatialCrossMapLRN_updateOutput(
  File "/usr/local/lib/python2.7/dist-packages/torch/_thnn/utils.py", line 22, in __getattr__
    raise NotImplementedError
NotImplementedError

Andybert · 2017-08-31T02:18:38Z

when I try to convert the following torch net to pytorch,

require 'torch'
require 'nn'
require 'nnx'
require 'optim'
require 'rnn'

function buildModel_MeanPool_RNN(nFltrs1,nFltrs2,nFltrs3,nPersonsTrain)

local nFilters = {nFltrs1,nFltrs2,nFltrs3}

local filtsize = {5,5,5}
local poolsize = {2,2,2}
local stepSize = {2,2,2}

-- remember this adds padding to ALL SIDES of the image
local padDim = 4

local cnn = nn.Sequential()

local ninputChannels = 5
cnn:add(nn.SpatialZeroPadding(padDim, padDim, padDim, padDim))
cnn:add(nn.SpatialConvolutionMM(ninputChannels, nFilters[1], filtsize[1], filtsize[1], 1, 1))
cnn:add(nn.Tanh())
cnn:add(nn.SpatialMaxPooling(poolsize[1],poolsize[1],stepSize[1],stepSize[1]))

ninputChannels = nFilters[1]
cnn:add(nn.SpatialZeroPadding(padDim, padDim, padDim, padDim))
cnn:add(nn.SpatialConvolutionMM(ninputChannels, nFilters[2], filtsize[2], filtsize[2], 1, 1))
cnn:add(nn.Tanh())
cnn:add(nn.SpatialMaxPooling(poolsize[2],poolsize[2],stepSize[2],stepSize[2]))

ninputChannels = nFilters[2]
cnn:add(nn.SpatialZeroPadding(padDim, padDim, padDim, padDim))
cnn:add(nn.SpatialConvolutionMM(ninputChannels, nFilters[3], filtsize[3], filtsize[3], 1, 1))
cnn:add(nn.Tanh())
cnn:add(nn.SpatialMaxPooling(poolsize[3],poolsize[3],stepSize[3],stepSize[3]))

local nFullyConnected = nFilters[3]108

cnn:add(nn.Reshape(1,nFullyConnected))
cnn:add(nn.Dropout(0.6))
cnn:add(nn.Linear(nFullyConnected,128))
-- cnn:cuda()

local h2h = nn.Sequential()
h2h:add(nn.Tanh())
h2h:add(nn.Dropout(0.6))
h2h:add(nn.Linear(128,128))
-- h2h:cuda()

local r1 = nn.Recurrent(
128,
cnn,
h2h,
nn.Identity(),
16)

local rnn1 = nn.Sequencer(
nn.Sequential()
:add(r1)
)

Combined_CNN_RNN_1 = nn.Sequential()
Combined_CNN_RNN_1:add(rnn1)
Combined_CNN_RNN_1:add(nn.JoinTable(1))
Combined_CNN_RNN_1:add(nn.Mean(1))

local r2 = nn.Recurrent(
128,
cnn:clone('weight','bias','gradWeight','gradBias'),
h2h:clone('weight','bias','gradWeight','gradBias'),
nn.Identity(),
16)

local rnn2 = nn.Sequencer(
nn.Sequential()
:add(r2)
)

Combined_CNN_RNN_2 = nn.Sequential()
Combined_CNN_RNN_2:add(rnn2)
Combined_CNN_RNN_2:add(nn.JoinTable(1))
Combined_CNN_RNN_2:add(nn.Mean(1))

-- Combined_CNN_RNN_2 = Combined_CNN_RNN_1:clone('weight','bias','gradWeight','gradBias')

local mlp2 = nn.ParallelTable()
mlp2:add(Combined_CNN_RNN_1)
mlp2:add(Combined_CNN_RNN_2)
-- mlp2:cuda()

local mlp3 = nn.ConcatTable()
mlp3:add(nn.Identity())
mlp3:add(nn.Identity())
mlp3:add(nn.Identity())
-- mlp3:cuda()

local mlp4 = nn.ParallelTable()
mlp4:add(nn.Identity())
mlp4:add(nn.SelectTable(1))
mlp4:add(nn.SelectTable(2))
-- mlp4:cuda()

-- used to predict the identity of each person
local classifierLayer = nn.Linear(128,nPersonsTrain)

-- identification
local mlp6 = nn.Sequential()
mlp6:add(classifierLayer)
mlp6:add(nn.LogSoftMax())
-- mlp6:cuda()

local mlp7 = nn.Sequential()
mlp7:add(classifierLayer:clone('weight','bias','gradWeight','gradBias'))
mlp7:add(nn.LogSoftMax())
-- mlp7:cuda()

local mlp5 = nn.ParallelTable()
mlp5:add(nn.PairwiseDistance(2))
mlp5:add(mlp6)
mlp5:add(mlp7)
-- mlp5:cuda()

local fullModel = nn.Sequential()
fullModel:add(mlp2)
fullModel:add(mlp3)
fullModel:add(mlp4)
fullModel:add(mlp5)
-- fullModel:cuda()

local crit = nn.SuperCriterion()
crit:add(nn.HingeEmbeddingCriterion(2),1)
crit:add(nn.ClassNLLCriterion(),1)
crit:add(nn.ClassNLLCriterion(),1)

return fullModel, crit, Combined_CNN_RNN_1, cnn
end

fullModel, crit, Combined_CNN_RNN_1, cnn = buildModel_MeanPool_RNN(16,32,32,150)
torch.save('Combined_CNN_RNN.t7',Combined_CNN_RNN_1)

But it is not working:
Not Implement nn.Sequencer
Not Implement JoinTable
Not Implement Mean

stalagmite7 · 2017-09-01T20:30:08Z

@clcarwin , when i try to convert the model to use float tensors (the original model was trained on GPUs and CUDA tensors is not usable in the code), I get this error:

 [string "m = m:float()"]:1: attempt to call method 'float' (a nil value)
stack traceback:
    [string "m = m:float()"]:1: in main chunk
[C]: in function 'xpcall'
/home/dravi/torch/install/share/lua/5.1/trepl/init.lua:679: in function 'repl'
...ravi/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:204: in main chunk
[C]: at 0x00405d50

I'm doing this via the command line 'th', is there any package that is missing and needs to be 'required' for the float tensor conversion?

huntkao · 2018-05-21T08:02:24Z

@stalagmite7, I have the same problem. Did you ever solve this? Thanks.

Spandan-Madan · 2018-08-06T22:12:23Z

@stalagmite7 @huntkao I am having the same issue! Did either of you end up solving this issue?

Thanks a lot!

stalagmite7 · 2018-08-06T22:42:54Z

No, @huntkao and @Spandan-Madan . I finally ended up not going down this route because it was so buggy

dmortem · 2018-08-19T01:10:27Z

@clcarwin
Hi, I have SyntaxError on the line "m = m:float()". How to solve it? Thanks a lot!

Spandan-Madan · 2018-08-19T02:03:32Z

In my specific case, this is what happened:- lua uses tables for all data arrays pretty much. So the only way to find out if the object is a model itself or a table containing model and other things is to look inside the table and see what it contains. In my case, the table object was a list of tables and one of the tables inside it was the model. So, I indexed inside to the right element. And then converted that model to CPU using model = model:float() and it worked perfectly. I know it sounds silly, but give it a shot. You never know you might have the same problem as me!

…

On Sat, Aug 18, 2018 at 6:10 PM, dmortem ***@***.***> wrote: @clcarwin <https://github.com/clcarwin> Hi, I have SyntaxError on the line "m = m:float()". How to solve it? Thanks a lot! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AU8Xo45ExiBYx1z0Y8ItlYXr2WTROPgGks5uSLsDgaJpZM4NjX48> .

LaiPiXiong · 2018-09-06T13:22:11Z

@dmortem Hi, I have the same error with you, do you overcome it and how?

dmortem · 2018-09-06T14:59:15Z

@LaiPiXiong Hi, I solved it by put codes into a lua file, not python.

LaiPiXiong · 2018-09-07T02:43:43Z

@dmortem Thanks, do you change torch to pytorch successfully? I met error as @stalagmite7. And I try to download xnornet.py but it still show error "string m=m:float() attempt to call method 'float' (a nil value) ".

dmortem · 2018-09-07T02:51:36Z

@LaiPiXiong After I configuring torch environment, I put three sentences in a lua file, and execute it. The model has been converted successfully. And whether you can use this model in Pytorch depends on the original torch model since some torch structures cannot be used in Pytorch, so maybe we still cannot use the converted model.

LaiPiXiong · 2018-09-07T02:57:46Z

@dmortem I can't convert it right. Does "convert_torch.py" also based on torch?

dmortem · 2018-09-07T03:13:29Z

@LaiPiXiong I'm sorry, I didn't convert the model successfully, and my error is "torch.utils.serialization.read_lua_file.T7ReaderException: unknown type id 436207616. The file may be corrupted.". This error is because my model has some different layers which are absent in torch.legacy.
And what I do is to run lua file to convert the model of the GPU version into the CPU version, and then run python file to convert it into a pytorch model.

LaiPiXiong · 2018-09-07T06:14:44Z

@dmortem I can't convert model in GPU to CPU version. Do you install luarocks before torch?

LaiPiXiong · 2018-09-07T08:18:50Z

@dmortem Hi, I convert torch model to pytorch successfully. I use 'import 'cudnn'' before "m = torch.load(model-name)". I learn this from rwightman/pytorch-planet-amazon#1

Shubodh · 2018-09-11T12:14:29Z

@LaiPiXiong It's rather

require 'cutorch'
require 'cudnn'

as it is in lua.

LaiPiXiong · 2018-09-11T12:21:46Z

Oh, I see, thanks.

AllenYLJiang · 2019-01-28T03:39:32Z

@johnny5550822
You should download the model from
https://github.com/jcjohnson/cnn-benchmarks
This does not contain the cudnn contents and can be successful.

rafikg · 2019-03-29T13:06:45Z

Somebody resolved the problem ?

lochotan4794 · 2021-01-12T04:34:14Z

I have the same issue. Anyone help me please !

omarongraini · 2023-08-17T11:57:47Z

I am trying to convert this model driveLink from this repo repoLink but it gives the error
return python_class.from_buffer(reader.f.read(size), 'native') AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer'
i tried to convert it to cpu but the model wont load using torch with error magic_number = pickle_module.load(f) _pickle.UnpicklingError: invalid load key, '\x03'.
any help will be appreciated

omarongraini · 2023-08-20T03:00:33Z

m = torch.load('xxxx.t7')

I am trying to use this way to convert the model to cpu but torch.load gives this types of error UnpicklingError: invalid load key, '\x03'.

bhack mentioned this issue Jan 10, 2018

PS model DagnyT/hardnet#6

Closed

YaoQ mentioned this issue Mar 4, 2018

head files missing jonathanmarek1/binarynet-tensorflow#1

Open

danielcranston mentioned this issue Jan 17, 2020

Convert pre-trained models to CPU before saving yindaz/DeepCompletionRelease#36

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

convert torch 7 model and get an error :AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer' #8

convert torch 7 model and get an error :AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer' #8

Mrxuefei commented May 23, 2017 •

edited

Loading

clcarwin commented May 23, 2017

johnny5550822 commented May 30, 2017

clcarwin commented May 31, 2017

johnny5550822 commented May 31, 2017

Cadene commented Jun 3, 2017 •

edited

Loading

heyalqh commented Jul 7, 2017

johnny5550822 commented Jul 8, 2017

heyalqh commented Jul 8, 2017

DCurro commented Jul 19, 2017 •

edited

Loading

Andybert commented Aug 31, 2017

stalagmite7 commented Sep 1, 2017 •

edited

Loading

huntkao commented May 21, 2018

Spandan-Madan commented Aug 6, 2018

stalagmite7 commented Aug 6, 2018

dmortem commented Aug 19, 2018

Spandan-Madan commented Aug 19, 2018 via email

LaiPiXiong commented Sep 6, 2018

dmortem commented Sep 6, 2018

LaiPiXiong commented Sep 7, 2018

dmortem commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

dmortem commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

Shubodh commented Sep 11, 2018

LaiPiXiong commented Sep 11, 2018

AllenYLJiang commented Jan 28, 2019

rafikg commented Mar 29, 2019

lochotan4794 commented Jan 12, 2021

omarongraini commented Aug 17, 2023

omarongraini commented Aug 20, 2023

convert torch 7 model and get an error :AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer' #8

convert torch 7 model and get an error :AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer' #8

Comments

Mrxuefei commented May 23, 2017 • edited Loading

clcarwin commented May 23, 2017

johnny5550822 commented May 30, 2017

clcarwin commented May 31, 2017

johnny5550822 commented May 31, 2017

Cadene commented Jun 3, 2017 • edited Loading

heyalqh commented Jul 7, 2017

johnny5550822 commented Jul 8, 2017

heyalqh commented Jul 8, 2017

DCurro commented Jul 19, 2017 • edited Loading

Andybert commented Aug 31, 2017

stalagmite7 commented Sep 1, 2017 • edited Loading

huntkao commented May 21, 2018

Spandan-Madan commented Aug 6, 2018

stalagmite7 commented Aug 6, 2018

dmortem commented Aug 19, 2018

Spandan-Madan commented Aug 19, 2018 via email

LaiPiXiong commented Sep 6, 2018

dmortem commented Sep 6, 2018

LaiPiXiong commented Sep 7, 2018

dmortem commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

dmortem commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

LaiPiXiong commented Sep 7, 2018

Shubodh commented Sep 11, 2018

LaiPiXiong commented Sep 11, 2018

AllenYLJiang commented Jan 28, 2019

rafikg commented Mar 29, 2019

lochotan4794 commented Jan 12, 2021

omarongraini commented Aug 17, 2023

omarongraini commented Aug 20, 2023

Mrxuefei commented May 23, 2017 •

edited

Loading

Cadene commented Jun 3, 2017 •

edited

Loading

DCurro commented Jul 19, 2017 •

edited

Loading

stalagmite7 commented Sep 1, 2017 •

edited

Loading