Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to insert data into hive-solr external table due to Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: com.lucidworks.hadoop.hive.LWHiveOutputFormat error #46

Open
podiligiri opened this issue Jun 14, 2018 · 1 comment

Comments

@podiligiri
Copy link

step 1) created hive table test

step 2 ) created external table as follows
CREATE EXTERNAL TABLE rams.solr_test (
id string,
application_date string,
first_name string,
middle_name string,
last_name string,
preferred_email string,
address string,
city string,
country string,
candidate_disposition_reason string,
candidate_disposition string,
tag_original_ats string,
hiring_manager string,
job_title string,
is_candidate_active string,
attachments string)
STORED BY 'com.lucidworks.hadoop.hive.LWStorageHandler'
LOCATION '/user/xxx/hiredesk'
TBLPROPERTIES('solr.server.url'='solr URL ',
'solr.zkhost' = 'zookeeper host name',
'solr.collection' = 'inc_hiredesk_docs',
'solr.query' = ':');

note : I created a solr collection ( 'inc_hiredesk_docs') and gave valid solr server url and zookeeper hostnames

step 3) added following jar files

add /opt/lucidworks-hdpsearch/hive/solr-hive-serde-2.2.1.jar

add jar /opt/lucidworks-hdpsearch/hive/solr-hive-serde-0.13-2.2.1.jar

step 4) trying to insert the record but getting the following erro

insert into rams.solr_test values("0001AA28-036A-40F8-A6E8-D0FA39044556","2013-11-12 22:07:16.510","Manish","test","Kumawat","[email protected]","test","Sikar","test","test","test","test","test","test","test","test")

Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork))'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: com.lucidworks.hadoop.hive.LWHiveOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)

@ctargett
Copy link
Contributor

It appears you've added 2 SerDe jars, one for Hive 1.x and one only for Hive 0.13. You only need one of them.

  • If you're not using Hive 0.13, you shouldn't add 'solr-hive-serde-0.13-2.2.1.jar' and only use 'solr-hive-serde-2.2.1.jar';
  • if you are using 0.13, you should only use 'solr-hive-serde-0.13-2.2.1.jar'.

Also note that the Hive SerDe doesn't yet support Hive 2.x or 3.x if that's the version you're trying to use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants